RE: very busy syslog server
- From: "Imri Zvik" <imriz@xxxxxxxxxxxxxxx>
- Date: Thu, 8 Dec 2005 13:54:51 +0200
Already tried different values, even some really high arbitrary ones...
I think I will try splitting the work load on a couple of instances of
syslog...
I hope that would do the trick
--
Imri Zvik
PGP (2.6.3ia) Public Key: http://mariska.inter.net.il/~imriz/imriz.pgp
-----Original Message-----
From: jaws [mailto:jaws@xxxxxxxxxxx]
Sent: Thursday, December 08, 2005 1:48 PM
To: Imri Zvik
Cc: j_guojun@xxxxxxx; freebsd-performance@xxxxxxxxxxx
Subject: Re: very busy syslog server
What you need is more sockets available and a much larger sockbufs. Try
adjusting the following parameters:
net.inet.udp.maxdgram
net.inet.udp.recvspace
Dont rely on the default values rather set it to a much higher values.
Regards,
jaws
Imri Zvik wrote:
>Hi,
>
>1. The NIC being used is "Intel(R) PRO/1000" (the em(4) driver).
>2. The CPU utilization in average is between 15% and 20%.
>3. This machine is being used _only_ for the sysloging - the database
resides on another server.
>
>Meanwhile, I have added some more memory to the machine, and now it has
3GB of RAM, but I am still seeing packets being dropped due to full
socket buffers.
>
>Thanks,
>
>--
>Imri Zvik
>PGP (2.6.3ia) Public Key: http://mariska.inter.net.il/~imriz/imriz.pgp
>
>________________________________________
>From: Jin Guojun [mailto:jinmtb@xxxxxxxxxxxxx]
>Sent: Wednesday, December 07, 2005 9:56 PM
>To: Sean Chittenden
>Cc: Imri Zvik; freebsd-performance@xxxxxxxxxxx
>Subject: Re: very busy syslog server
>
>Sean Chittenden wrote:
>I'm trying to setup a syslog server to serve a large group of
>servers. For the syslog daemon, I have chosen rsyslogd, and the
>backend is mysql (on a different machine).
>
>The machine has 2 Intel Xeon 2.80GHz CPUs, and 1GB of RAM, and it is
>running FreeBSD 6 (6.0-STABLE).
>
>The problem is, that I see a lot of UDP packets being dropped:
>
>udp:
> 390202 datagrams received
> 0 with incomplete header
> 0 with bad data length field
> 0 with bad checksum
> 6 with no checksum
> 0 dropped due to no socket
> 0 broadcast/multicast datagrams dropped due to no socket
>->>> 123677 dropped due to full socket buffers
> 0 not for hashed pcb
> 266525 delivered
> 133260 datagrams output
>
>I have tried to increase net.inet.udp.recvspace, but it didn't solve
>the problem.
>
>I would appreciate any hint or tips.
>
>
>When you're doing a large number of packets per second, you may want
>to look into enabling device polling(4). Right now, every packet
>results in an interrupt. With device polling, you can handle more
>than one packet per interrupt. See the man page for details.
>Not quite, the interrupt interval depends on the device driver, or
which NIC is used.
>A number NICs are able to to interrupt coalescence, which requires to
increase buffer
>descriptor ring size (just for receiving buffer descriptors). Of
course, polling is a simple thing
>to try.
>
>Before we can come up a better way to alter a better solution for this
case, you also need to
>monitor a few things:
>
>What is NIC on this machine?
>
>What is the CUP utilization in average and in case the packet drops?
You can simply write a
>script to do this instead of instructing kernel to do so (since this
needs no super accurate):
>
>run vmstat to record CPU utilization in every 1 to 3 seconds for use
when following event happens:
>use netstat watch UDP and pipe it to awk "netstat -udp | awk
'$2=="drooped" {print $1; exit}'"
>every 3-5 seconds, and compare the result with previous one to see if
any changes. If so,
>grep the last couple of line from vmstat output records.
>
>>>From your information, it seems that this machine has enough memory
bandwidth for syslog needs,
>since it is not clear what this machine is for rlog daemon or sql
server, or both are on the same machine.
>If the third case is true, then you may run out of memory bandwidth.
Under this circumstance,
>you need to obtain the packet rate and the average packet size in order
to determine the I/O
>and memory bandwidth requirements.
>
> -Jin Guojun
>_______________________________________________
>freebsd-performance@xxxxxxxxxxx mailing list
>http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>To unsubscribe, send any mail to
"freebsd-performance-unsubscribe@xxxxxxxxxxx"
>
>
>
_______________________________________________
freebsd-performance@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"
- Prev by Date: Re: very busy syslog server
- Next by Date: Re: very busy syslog server
- Previous by thread: Re: very busy syslog server
- Index(es):
Relevant Pages
|
|
