Watching real-time server bandwidth, an exercise in the awesomeness of Open Source
Published on
This week we had a significant escalation in traffic on a particular server. It was not unexpected, however we had changed infrastructure between peak periods and I was keen to see how the new network and virtual machine were coping under the load.
It's an easy thing to do to monitor the processing load on a CentOS server in real time, but without access to Webmin or MRTG I was suddenly without a means to easily see what was happening at the TCP/IP connection level.
Google to the rescue with tcptrack...
While I will typically go quite deep through Google results to get an answer I need, the first few in my search yielded the gold I needed in this case, so there's highly likely to be things I missed. If there are, please leave a constructive comment below!
Note - I wasn't looking for logging, I was looking for real-time bandwidth per-connection stats and aggregate. Basically "top" for network.
The quick win: tcptrack
Almost immediately I found tcptrack mentioned, with a screenshot that looked to be exactly what I was after, so a little more digging found me a download location.
This VM is 64-bit CentOS 6, so:
# cd /home
# wget http://pkgs.repoforge.org/tcptrack/tcptrack-1.4.0-1.el6.rf.x86_64.rpm
# rpm -ivh tcptrack-1.4.0-1.el6.rf.x86_64.rpm
I'm generally a believer in using yum on CentOS, or apt-get on Ubuntu/Debian, however I could not find tcptrack on any of the main CentOS 6 yum repos - more on this later.
Using tcptrack
Using tcptrack is brain-dead simple. You need only the -i flag and one parameter, the name of the interface you wish to monitor - eth0, eth1 etc.
# tcptrack -i eth0
tcptrack is supposed to use additional params, but for some reason my install errored on more than just the basic syntax. It's something I will be following up in the future.
What's supposed to work is the addition of additional aguements such as port 80 (tcptrack -i eth1 port 80). It would have been useful, but the only traffic in and out of that server is on ports I'm interested anyway due to the heavily firewalled nature of this VM, so it wasn't of immediate concern.
What tcptrack's doing here is listing all the active and expiring TCP sessions, in near real time, on a fast refresh. The screenshot I took here was at 11Mb/s but I saw rates as high as 18Mb/s, and certainly consistently above 5Mb/s. By the next afternoon (after the entry deadline had passed) the traffic dropped back to a more normal 500-900kb/s. I should point out here that this peak phase of system activity is about entry of an application plus the upload of a small segment of video, hence the smaller number of concurrent high demand connections - but there are a *lot* of submissions.
tcptrack options
The RPM includes the tcptrack man page, but I also found it online here http://linux.die.net/man/1/tcptrack.
The most useful of the options for me was the "interactive command" for sorting:
s - Cycle through the sorting options: unsorted, sorted by rate, sorted by total bytes.
To use this, with tcptrack running, press "s" and it sorts the connections by rate. I needed nothing more... network voyerism connection was perfected for my inner geek.
A drawback and a benefit of the Open Source Movement
A great piece of software like tcptrack is so useful for the specific task it's going into my toolbox and I'm likely to pull it out time and time again. However, while I love the open source movement, the experience with tcptrack and the additional options that didn't work (showing me a compile error) reminded me just how transient it can be.
I tried to go to the developer's site to see if I could find a better RPM (or the authentic sourcecode repos), or find a good forum where users or contributors could help me. The developer's site is offline - not only is the page not there, but the entire parent site is gone, dissapeared into the ether. Yes, I know, the Wayback Machine could probably get me the last known good pages, but that's not the point.
On the positive side, the gorgeous side of the Open Source Movement means even though the develper is gone into the ether, the rpm's, man pages and probably the source code is out there on th enet for a long time to come.
It's likely that if that was commercial or even shareware software the only way I'd be able to get it would be via a warez site, and I certainly wouldn't be able to use it legally even if I could get a copy.
Long live Open Source :)