Re: network bitrate of a poll of processes

On Mon, May 26, 2008 at 4:27 PM, Mathieu Prevot
<mathieu.prevot@xxxxxxxxx> wrote:
2008/5/26 Marc Spitzer <mspitzer@xxxxxxxxx>:
On Mon, May 26, 2008 at 12:14 PM, Mathieu Prevot
<mathieu.prevot@xxxxxxxxx> wrote:
2008/5/26 Dan Nelson <dnelson@xxxxxxxxxxxxxxx>:
In the last episode (May 26), Mathieu Prevot said:
Hi,

I would like to know the bitrate of a pool of child processes that use
a network connection, how can I have something like netstat -w1
provide but at the process level ?

If you can segregate them to their own UID, you can use an ipfw "count
ip from any to any uid 6666" rule to count the packets. Another option
would be to jail them to a dedicated IP address and count traffic on
that IP.

Both are nice... thank you. And don't you have something more portable
ie. that could run on "any" posix os - like a sh script that use
standard tools - and that doesn't need root user intervention ?

Mathieu

I do not think so, firewall for example is controlled by root. And
users do not generally have detailed access to other users
information. You can look into the BASM framework to track what your
apps are doing and if you know the behavior to expect you can use
netflow/argus/tcpdump to track that and post process the logs in each
case into something useful.

What are you trying to do, details do help?

marc

I have a script that generates many child processes that download
videos (~10MB) in this case, or do heavy data processing/analysing. I
can tune the number of workers as I want, but It's a shared cluster so
I would like to

1) measure 2) optimize/control (by number of workers ? at the script
level ! so I can update thing very dynamically)
a) CPU load b) IO on drive(s) c) network load

and I prefer solutions that don't need root access/intervention. I can
measure indirectly network load using `date +%s` and `du -b |sed
's/\t.*//' ` regularly... but I may have to deal with several 10GB
later - and actually the number of files is more critical than their
size.
Maybe writing a (very) quick and dirty python app that listen/attach
to a group of processes (for the network case) ?

Mathieu


for the CPU stuff take a look at ps:
bob% ps aO%mem,%cpu,ppid
PID %MEM %CPU PPID TT STAT TIME COMMAND
768 0.2 0.0 1 v0 Is+ 0:00.01 /usr/libexec/getty Pc ttyv0
769 0.2 0.0 1 v1 Is+ 0:00.01 /usr/libexec/getty Pc ttyv1
770 0.2 0.0 1 v2 Is+ 0:00.01 /usr/libexec/getty Pc ttyv2
771 0.2 0.0 1 v3 Is+ 0:00.01 /usr/libexec/getty Pc ttyv3
772 0.2 0.0 1 v4 Is+ 0:00.01 /usr/libexec/getty Pc ttyv4
773 0.2 0.0 1 v5 Is+ 0:00.01 /usr/libexec/getty Pc ttyv5
774 0.2 0.0 1 v6 Is+ 0:00.01 /usr/libexec/getty Pc ttyv6
775 0.2 0.0 1 v7 Is+ 0:00.01 /usr/libexec/getty Pc ttyv7
272 0.2 0.0 1 con- I 0:00.04 dhclient: le0 [priv] (dhclient)
951 0.6 0.0 949 p0 Rs 0:00.28 -zsh (zsh)
1056 0.2 0.0 951 p0 R+ 0:00.01 ps aO%mem,%cpu,ppid

add what you need and you have your cpu and such stats, it might do
your IO as well also look for thing in (s)bin that end in stat for
things that might be a better fit.

Network I do not know how to get you the level of access you want
without root/trusted access at least on collection. You will not need
it on processing.

as far as managing thing after you have the information I have nothing
for you there.

good luck,

marc
--
Freedom is nothing but a chance to be better.
Albert Camus
_______________________________________________
freebsd-hackers@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: 2003 Server slowed to a crawl
    ... click Processes then CPU column to sort by ... Investigate a possible Network problem (is there a broadcast storm ... >> method for telling the computer's processor that it needs attention. ... >> which have signed drivers (such hardware is sold with a Microsoft Windows ...
    (microsoft.public.windows.server.general)
  • Re: Game Company- Java Server Thread Priority
    ... > Program takes 0% of the CPU load, and when processing a turn goes up to ... An example of network waits. ... expected it to go *up* (since the server box wasn't running the client too). ... involved a network access which slowed it down massively. ...
    (comp.lang.java.programmer)
  • Re: 2003 Server slowed to a crawl
    ... click Processes then CPU column to sort by processes ... > the network, are backing up large amounts of user data during production ... > method for telling the computer's processor that it needs attention. ... > which have signed drivers (such hardware is sold with a Microsoft Windows ...
    (microsoft.public.windows.server.general)
  • Re: Got idle CPU cycles?
    ... >> freeze a network solid. ... Any host CPU will have its pipes completely filled. ... Increment the radius, do it again. ... crystal structure data that we are exploring. ...
    (comp.os.linux.misc)
  • Re: a proposed callout API
    ... of the network implementation in such different operating systems ... The basic premise is that you can make better use of any available cpu ... the operating systems are so different from each other I don't think ...
    (freebsd-arch)