Re: inode state

From: Robert Watson (rwatson_at_freebsd.org)
Date: 12/09/03

  • Next message: paul beard: "how to make dhclient give up its lease and get a new one with routes, etc. ?" 
    Date: Tue, 9 Dec 2003 13:38:34 -0500 (EST)
To: Fabian Thylmann <fthylmann@stats.net>

    On Tue, 9 Dec 2003, Fabian Thylmann wrote:

    > I have a heavily used threaded server program running on one of my Dell
    > Poweredge 1750 servers. Its a statistical analysis package for websites.
    > Currently it analyses over 60 million requests a day, which (because of
    > many different reasons) causes it to handle around 120 million http
    > requests a day. At peaks around 1500 requests a second.
    >
    > The system stores most many statistics in memory which is flushed to
    > disk in circles by a worker thread.
    >
    > Another big part is stored in an on-disk database which is mmap()'d into
    > memory. Because we do not have enough memory to keep everything in
    > memory at one time the mmap() system of course pages data in and out.
    >
    > When I look in systat -v I see that dirtybuf climbs to about 1700 and
    > then they get flushed to disk, causing high disk usage of around 300-400
    > tps whcih renders the disks useless for anything else.
    >
    > When those flushes occure, my apps state as displayed by top(1) gets
    > into inode state, PRI is set to -14 and cpu usage rapidly drops. The
    > program and ALL of its threads are stalled at that time. Those inode
    > states take around 2 oe 3 seconds and happen every 30 seconds or so.
    >
    > In those 3 seconds we lose around 1500 hits at peak times for processing
    > because the app can not handle them fast enough. This results in around
    > 2 million or so hits lost over the day for processing.
    >
    > I am now wondering if anyone can explain to me why ALL threads and not
    > just the threads that actually do I/O work get blocked when dirty
    > buffers are flushed and what to do to fix this problem.
    >
    > I would be very happy if someone could reply and point me into the right
    > direction!

    You don't mention which version of FreeBSD you're running -- if 4.x, you
    probably want to relink your application against the "linuxthreads" port.
    This is because libc_r implements threads inside a single process without
    the support of the kernel, which means that if the process is blocked in
    kernel, all threads will be blocked in kernel. The linuxthreads package
    uses a model similar to Linux's threading implementation (hence the name)
    to allow the threads to be scheduled using lightweight versions of
    processes (shared file descriptors, etc). This isn't quite
    POSIX-compliant, but it works quite well for disk-bound applications such
    as databases.

    If you're running on 5.x, especially recent 5.1 or 5.2 prereleases, you
    probably want to give libkse a try. It's the new m:n threading
    implementation that will become the default in 5.3, and also permits
    parallelism (only in a more POSIX-compliant way, and in theory offering
    much greater scalability for large numbers of threads). I stick the
    following lines in my /etc/libmap.conf on 5.x boxes to force all
    applications linked against libc_r to use libkse instead:

      libc_r.so.5 libkse.so.1
      libc_r.so libkse.so

    One particularly nice thing about the m:n thread support is that you can
    run-time plug the thread library between several options (libc_r, libthr,
    libkse) to pick the one that performs best for your application. Another
    benefit of running with a non-libc_r threads package is that if you have
    an SMP box, you'll see real parallelism.

    Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
    robert@fledge.watson.org Senior Research Scientist, McAfee Research

    _______________________________________________
    freebsd-questions@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-questions
    To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"

  • Next message: paul beard: "how to make dhclient give up its lease and get a new one with routes, etc. ?"

    Relevant Pages

    • Re: Partition Resizing
      ... You can also try "regular" parted, in the "parted" package. ... partition, then copying it, then deleting the old one and expanding the new one ... The GNU Parted disk partition resizing program ... Note that ReiserFS support is only enabled if you install ...
      (Debian-User)
    • Re: logic analyzer signal tap 2 - writing data
      ... matlab to Quartus Tcl? ... When you click on the insystem_memory_edit package in the ... # Write memory content using the hex memory file ... i am trying to load the quartus_stp package, ...
      (comp.arch.fpga)
    • Re: saving presentation to cd
      ... It was created on my work computer which has 97. ... | and when I insert a disk I get a box full of program choices to use. ... |> Are you using Pack and Go or Package for CD? ... |> contents of the folder to the CD. ...
      (microsoft.public.powerpoint)
    • Re: intermittent library cache pin/lock problems
      ... > Have you tried pinning things into the library cache? ... > PACKAGE, PACKAGE BODY, FUNCTION, PROCEDURE) and kept = 'NO', or the ... > What can happen is that the shared pool gets fragmented, ... there still might be memory leaks... ...
      (comp.databases.oracle.server)
    • Re: logic analyzer signal tap 2 - writing data
      ... i am trying to load the quartus_stp package, ... You can use the TCL interface to automate the memory update. ... If you are using an external logic analyzer, ...
      (comp.arch.fpga)