How to reliably measure the network bandwidth used by a process

Question

I have developed an application and I want to measure how much network bandwidth it consumes in some typical test cases.

I found a few applications like nethog etc. however I am not sure how accurate its reports are!

I would like some way to very accurately measure the same as the results need to go into a report for a conference.

I'm willing to write customized solutions for the same if someone guides me how to!

I want something where I can run the monitoring program and my target application to get network usage statistics - cumulative bytes sent/rcvd. .. maximum usage and average usage etc

Solution

Can the application be isolated on a single machine?
Does anything else have to run on the system?

If a system can be dedicated this way, periodically grab the last line from /proc/net/netstat and subtract the corresponding values of InOctets and OutOctets.

This system, Fedora 15, shows this after 23 days of uptime:

TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps ArpFilter TW TWRecycled TWKilled PAWSPassive PAWSActive PAWSEstab DelayedACKs DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPPrequeued TCPDirectCopyFromBacklog TCPDirectCopyFromPrequeue TCPPrequeueDropped TCPHPHits TCPHPHitsToUser TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging TCPFACKReorder TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo TCPDSACKUndo TCPLossUndo TCPLoss TCPLostRetransmit TCPRenoFailures TCPSackFailures TCPLossFailures TCPFastRetrans TCPForwardRetrans TCPSlowStartRetrans TCPTimeouts TCPRenoRecoveryFail TCPSackRecoveryFail TCPSchedulerFailed TCPRcvCollapsed TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv TCPAbortOnSyn TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures TCPSACKDiscard TCPDSACKIgnoredOld TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD6NotFound TCPMD5Unexpected TCPSackShifted TCPSackMerged TCPSackShiftFallback TCPBacklogDrop TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop
TcpExt: 0 0 0 0 0 0 0 0 10 0 67116 0 0 0 0 8 117271 53 18860 0 0 102295 23352211 87967244 0 16861098 118195 893786 881659 0 29 10 0 0 0 9 10 16 12 2321 21 0 1 156 39 940 13 921 8015 0 1 2 0 18461 22 941 0 0 2974 15422 0 709 0 0 0 1 8 119 3 0 0 0 0 25231 0 0 0 4 0 0 0
IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets OutMcastOctets InBcastOctets OutBcastOctets
IpExt: 0 0 25308 48 725 1 24434248973 4218365129 2181277 13241 365505 65

Of course, that format is unfriendly for here, but fairly nice for scripting languages to deal with. You can see the depth and variety of information! The last line shows that this system has read 24,434,248,973 bytes and written 4,218,365,129. (It is on day nine of scraping a large website.)

Comments

[AnkurVj]

Unfortunately another application using the network will have to run simultaneously on the same machine ! \

[wallyk]

@AnkurVj: Can the application be configured to do net i/o to only one other system? That is, could a dedicated server be set up to use the same technique, but from the other end?

[AnkurVj]

yes I might be able to do that .. I think that should give me what I need .. thanks 1

[AnkurVj]

I checked and found that the i/o stats displayed are not per process ... it is showing the same stats for all processes ...

[scai]

@AnkurVj That's correct, /proc/<pid>/net/dev contains the same information as /proc/net/dev.