Re: inode state

• Previous message: Charles Howse: "Re: How to make printer print faster?"
• Maybe in reply to: Fabian Thylmann: "inode state"
• Next in thread: Robert Watson: "Re: inode state"
• Reply: Robert Watson: "Re: inode state"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

To: "Robert Watson" <rwatson@freebsd.org>
Date: Tue, 9 Dec 2003 20:41:06 +0100

Hi Robert,

thanks for the reply, and yea, totally forgot the version. its 4.9.

Now, the problem is:
1) I can not use linuxthreads since my server is also multiplexing via
kqueue's and I can not find any version of linuxthreads which implements a
thread-safe version of kqueue.
2) I can not use freebsd 5.2 because it fails to boot on a dell poweredge
1750 with two harddisks. The LSILogic SCSI Controller the server uses (mpt
driver) seems to not find any hdds and gives up with an error.
If I remove one of the two disks 5.2 boots but the kernel traps as soon as
it tries to write to the hdd.

Is there some way to keep the number of kernl-level locks as low as
possible? This all seems to be associated to flushing dirty buffers and I
wonder if there is no way to make it flush in way smaller bursts or why
exactly it has to lock the process while doing so.

Fabian

----- Original Message -----
From: "Robert Watson" <rwatson@freebsd.org>
To: "Fabian Thylmann" <fthylmann@stats.net>
Cc: <freebsd-questions@freebsd.org>
Sent: Tuesday, December 09, 2003 7:38 PM
Subject: Re: inode state

> On Tue, 9 Dec 2003, Fabian Thylmann wrote:
>
> > I have a heavily used threaded server program running on one of my Dell
> > Poweredge 1750 servers. Its a statistical analysis package for websites.
> > Currently it analyses over 60 million requests a day, which (because of
> > many different reasons) causes it to handle around 120 million http
> > requests a day. At peaks around 1500 requests a second.
> >
> > The system stores most many statistics in memory which is flushed to
> > disk in circles by a worker thread.
> >
> > Another big part is stored in an on-disk database which is mmap()'d into
> > memory. Because we do not have enough memory to keep everything in
> > memory at one time the mmap() system of course pages data in and out.
> >
> > When I look in systat -v I see that dirtybuf climbs to about 1700 and
> > then they get flushed to disk, causing high disk usage of around 300-400
> > tps whcih renders the disks useless for anything else.
> >
> > When those flushes occure, my apps state as displayed by top(1) gets
> > into inode state, PRI is set to -14 and cpu usage rapidly drops. The
> > program and ALL of its threads are stalled at that time. Those inode
> > states take around 2 oe 3 seconds and happen every 30 seconds or so.
> >
> > In those 3 seconds we lose around 1500 hits at peak times for processing
> > because the app can not handle them fast enough. This results in around
> > 2 million or so hits lost over the day for processing.
> >
> > I am now wondering if anyone can explain to me why ALL threads and not
> > just the threads that actually do I/O work get blocked when dirty
> > buffers are flushed and what to do to fix this problem.
> >
> > I would be very happy if someone could reply and point me into the right
> > direction!
>
> You don't mention which version of FreeBSD you're running -- if 4.x, you
> probably want to relink your application against the "linuxthreads" port.
> This is because libc_r implements threads inside a single process without
> the support of the kernel, which means that if the process is blocked in
> kernel, all threads will be blocked in kernel. The linuxthreads package
> uses a model similar to Linux's threading implementation (hence the name)
> to allow the threads to be scheduled using lightweight versions of
> processes (shared file descriptors, etc). This isn't quite
> POSIX-compliant, but it works quite well for disk-bound applications such
> as databases.
>
> If you're running on 5.x, especially recent 5.1 or 5.2 prereleases, you
> probably want to give libkse a try. It's the new m:n threading
> implementation that will become the default in 5.3, and also permits
> parallelism (only in a more POSIX-compliant way, and in theory offering
> much greater scalability for large numbers of threads). I stick the
> following lines in my /etc/libmap.conf on 5.x boxes to force all
> applications linked against libc_r to use libkse instead:
>
> libc_r.so.5 libkse.so.1
> libc_r.so libkse.so
>
> One particularly nice thing about the m:n thread support is that you can
> run-time plug the thread library between several options (libc_r, libthr,
> libkse) to pick the one that performs best for your application. Another
> benefit of running with a non-libc_r threads package is that if you have
> an SMP box, you'll see real parallelism.
>
> Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
> robert@fledge.watson.org Senior Research Scientist, McAfee Research
>
>
>

_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"

• Previous message: Charles Howse: "Re: How to make printer print faster?"
• Maybe in reply to: Fabian Thylmann: "inode state"
• Next in thread: Robert Watson: "Re: inode state"
• Reply: Robert Watson: "Re: inode state"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]