Download in .ogv format
Man! There's a bunch of smug Linuxer's on this website! Why don't you all take what Shawn is sharing here and ADD to it instead of trying to squash him? At least he's trying to educate the community! Everybody can't be the Linux Gods you portray yourselves to be (which is probably a good thing)!
Shawn, keep up the good work with these Linux Minutes, I enjoy and appreciate them, whether they're strictly Linux-related or something more on the Linux periphery , like this tip. 100% of a Sys Admin's job is not at the console!
Been there too. It's normally a PITA to fix, especially if you've got dumb switches and hundreds of drops. STP in the smart switches helps, but can do weird things in other ways.
But you're right, at the end of the day, it happens because somebody plugged two switches together, and until you find that problem, life sucks.
What makes life suck even worse is when you realize it was YOU who plugged the two switches together in a closet with 300 drops in it. hehe. Yeah, I got a plaque for that one. ;)
You could have solved that problem in about 5 seconds from your desk. HP even has a pretty GUI, but being Linux Journal, you can ssh in there as well. :)
In a big environment where you have more than 6-10 downlinks to other switches, I usually use the "bisection method" as a trouble-finding algorithm (http://en.wikipedia.org/wiki/Bisection_method).
That is, for example, you disconnect the first half of the network connections you have on the main switch. If the problem persists, you reconnect them (to make the poor, remaining network service to flow again) and disconnect the first half of the second half and so on, recursively. If the problem disappears, reconnect half-at-a-time (recursively) the disconnected cables.
Since in the clip I see you use a specific vendor for your main switch, I can say that using the same switches configured to run multiple spanning-tree, we had good results in detecting and automatically blocking loops till the first level of unmanaged, non-stp switches (with fast intervention times since mstp is faster than stp).
Keep in mind that enabling stp on your network, loops are less likely to block the entire network, but more difficult to come up to your attention since you need good monitoring tools to query your network equipments about stp state for each port.
The problem isn't with my main switches, it's with the cheapo desktop switches in the classroom. We are a school, and don't have enough network drops into the rooms, so we have to use desktop switches in each room to allow more computer connections.
The cheap switches don't have STP, and so cause a network breakdown. My "good" switches don't see a network loop, they just see millions of multicast packets, and dutifully try to send them to every port in the district.
In a perfect world, I'd have enough drops for each room -- but it's a school, and we're lucky to have computers at all. :(
Shawn Powers is a Linux Journal Associate Editor. You might find him on IRC, Twitter, or training IT pros at CBT Nuggets.
Been there done that.
I do tech work for a private school; have the same issues. Money is tight and when rooms get built or added on they cheap out on the network infrastructure ( cabling and switches ).
Also it doesn't matter if you use STP or not, you still have to deal with the network loopback in that classroom; because someone plugged there cat5 cable into 2 ports on the wall.
It really becomes a pain.
Damn. Back when I was in school, we had to worry about someone taking the 50 ohm resistor off the end of a BNC cable...
Bravo. So gratifying to see a demonstration of old-fashioned deductive reasoning used in a real-world situation, where hands-off, hi-tech equipment aren't being utilized. School techs in my rural county also have to deal with inadequate equipment supported by inadequate budgets. The situation isn't going to change soon, if ever. Any of them would embrace your MacGyver way of troubleshooting in keeping their inadequate computer systems running.
Firstly I would just like to say thank you for your tech tips which I find intuitive and an excellent resource for the linux community!
However with regards to this article in particular I would not call this article fixing loopy network, instead I would call it delaying the inevitable.
As a previous poster has mentioned you should be using spanning Tree Protocol (STP) if you have three or more switches interconnected. In a correctly configured network this situation should never occur.
Out of curiosity how often are you having to do this?
In a large scale environment, you should be using spanning-tree capable switches in order to prevent L2 loops from bringing down your system.
on the plus-side, that means you have the ability to have a hot-standby cable (like via an alternate path) between your switches, and STP will make sure that only one is in use at any time.
I feel so sorry for ur employer.