Tracking Down Blips

You'll need to adjust your interface names if your Ethernet devices don't come up as eth0 and eth1, but the configuration should make sense from looking at my example. You either can restart networking or, even better, reboot your system to make sure it comes up properly on startup. Once up and running, the ifconfig command should look something like mine:


spowers@pooky:~$ ifconfig
br0    Link encap:Ethernet  HWaddr 00:25:90:34:d4:3a
       inet addr:192.168.1.6  Bcast:192.168.1.255  Mask:255.255.255.0
       inet6 addr: fe80::225:90ff:fe34:d43a/64 Scope:Link
       UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
       RX packets:1820381471 errors:0 dropped:0 overruns:0 frame:0
       TX packets:124514207 errors:0 dropped:0 overruns:0 carrier:0
       collisions:0 txqueuelen:1000
       RX bytes:1850742830285 (1.8 TB)  TX bytes:34896441989 (34.8 GB)

eth0  Link encap:Ethernet  HWaddr 00:25:90:34:d4:3a
       inet6 addr: fe80::225:90ff:fe34:d43a/64 Scope:Link
       UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
       RX packets:1040590501 errors:0 dropped:56421 overruns:0 frame:0
       TX packets:782968757 errors:0 dropped:0 overruns:0 carrier:0
       collisions:0 txqueuelen:1000
       RX bytes:1101548247906 (1.1 TB)  TX bytes:493789819966 (493.7 GB)
       Interrupt:16 Memory:fb5e0000-fb600000

eth1  Link encap:Ethernet  HWaddr 00:25:90:34:d4:3b
       UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
       RX packets:1001689311 errors:0 dropped:55961 overruns:0 frame:0
       TX packets:1165557388 errors:0 dropped:0 overruns:0 carrier:0
       collisions:0 txqueuelen:1000
       RX bytes:797026007540 (797.0 GB)  TX bytes:1134936725326 (1.1 TB)
       Interrupt:17 Memory:fb6e0000-fb700000

Note the only interface with an IP address is br0. The other two interfaces are bridged together, so traffic can freely flow through them. It's interesting that this is the way distributions like Untangled work. They create a bridge device and then filter/block/redirect traffic as it passes through.

The Software

So far, I've gotten only to the point where the monitoring computer can listen to the traffic going to and from the internet. I haven't actually installed any software to do the listening. Quite a few different packages exist for capturing traffic and analyzing it. Depending on the type of network trends you're looking for, you might choose a different software package from me. I actually installed a few, but rely most on BandwidthD for analyzing traffic. I'll talk more about BandwidthD, but be sure to check out some others too:

I like BandwidthD because it shows traffic graphs for each device on my network. If you remember my initial problem, I was trying to figure out what device on my network was downloading something every 20 minutes. I figured it was a game system or cell phone stuck in a failed download loop or something.

Installing BandwidthD (or most any of the utilities) is a simple apt-get install away. The software is most likely in your distribution's software repository, and even if the version is a little outdated, it should work perfectly fine. The only thing I needed to do is edit /etc/bandwidthd/bandwidthd.conf and set the network I wanted to monitor and the interface I wanted to listen on. Otherwise, I left everything the default. BandwidthD installs an Apache configuration file, so you should be able to access its interface at http://server.ip.address/bandwidthd/.

After it's running for a while, you should see statistics like those in Figure 6, which shows the top 20 bandwidth users on my network. It's fun information to see, but if you're looking for a specific traffic pattern, you'll need to scroll down a bit to see network graphs for every device on your network. Figure 7 shows the traffic to and from my Plex Media Server. Keep in mind this is only traffic going to and from the internet, but still, you can see when friends and family were watching videos from my server over the internet. It's important to note that although the default page of BandwidthD shows only the top 20 users, you can click on the network address to see every user who accesses the internet. It's an amazing tool for figuring out what's happening on your network.

Figure 6. I love BandwidthD; it's full of so much juicy data.

Figure 7. Having a set of graphs for each device on the network is amazingly convenient. Having it done automatically is just plain amazing.

What about My Blip?

It turns out that I couldn't find anything on my network causing those network usage spikes every 20 minutes. I looked at the graphs for every single device on my network and compared it to the spikes on my Cacti bandwidth graphs. I just couldn't find a match. Then I realized that my total bandwidth graph from BandwidthD should come pretty close to matching my WAN bandwidth graph from Cacti. And, it didn't. My entire network monitoring server setup seemed to be for nothing, because I couldn't track down what was causing the blips.

I decided to troubleshoot my Cacti installation to see if there was something happening every 20 minutes to cause an error. It was then that I noticed that while the WAN interface on my router had the blip every 20 minutes, the LAN side of my router (which I graph, but never really look at because it's just an inverse of the WAN graph) didn't have the blip. It turns out that my UniFi router has a feature that runs a speed test every 20 minutes to graph the health of the connection. I don't remember turning that feature on, but sure enough, it was enabled. When I disabled the periodic speedtest, my network blips stopped.

So in the end, my network monitoring setup didn't find anything, but I don't regret setting it up. Now I can monitor traffic easily and see what sort of bandwidth requirements individual devices need. In fact, the only change I plan to make is to set up my server using Option 2 instead of Option 3, because I recently upgraded my server rack to managed switching hardware. That way if my monitoring computer dies, my internet connection stays up.

______________________

Shawn Powers is a Linux Journal Associate Editor. You might find him on IRC, Twitter, or training IT pros at CBT Nuggets.