Re: Very low disk performance on 5.x

From: Bruce Evans (bde_at_zeta.org.au)
Date: 05/09/05

  • Next message: Poul-Henning Kamp: "Re: Very low disk performance on 5.x" 
    Date: Mon, 9 May 2005 21:11:16 +1000 (EST)
To: Scott Long <scottl@samsco.org>

    On Sun, 8 May 2005, Scott Long wrote:

    > Changing MAXPHYS is very dangerous, unfortunately. The root of the
    > problem is that kernel virtual memory (KVA) gets assigned to each I/O
    > buffer as it passes through the kernel. If we allow too much I/O through
    > at once then we have the very real possibility of exhausting the kernel
    > address space and causing a deadlock and/or panic. That is why MAXPHYS
    > is set so low. Your DD test is unlikely to trigger a problem, but try
    > doing a bunch of DD's is parallel and you likely will.
    >
    > The solution is to re-engineer the way that I/O buffers pass through
    > the kernel and only assign KVA when needed (for doing software parity
    > calculations, for example). That way we could make MAXPHYS be any
    > arbitrarily large number and not worry about exhausting KVA. I believe
    > that there is some work in progress in this area, but it's a large
    > project since nearly every single storage driver would need to be
    > changed. Another possibility is to recognise that amd64 doesn't have
    > the same KVA restrictions as i386 and thus can be treated differently.
    > However, doing the KVA work is still attractive since it'll yeild some
    > performance benefits too.

    How would it yield anything except complexity and negative performance
    benefits on machines that don't need it? The most efficient way to map
    kva for disk buffers is to do it only once at boot time using a simple
    sparse mapping technique as in 4.4BSD-Lite1. Simple sparse mapping needs
    lots of kva, and i386's don't have enough for machines with more than
    about 16MB of RAM, but machines with a 64-bit address space can have
    2^32 times as much kva as i386's.

    Bruce
    _______________________________________________
    freebsd-performance@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-performance
    To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

  • Next message: Poul-Henning Kamp: "Re: Very low disk performance on 5.x"

    Relevant Pages

    • Re: Very low disk performance on 5.x
      ... Changing MAXPHYS is very dangerous, ... problem is that kernel virtual memory (KVA) gets assigned to each I/O ...
      (freebsd-performance)
    • Re: ZFS committed to the FreeBSD base.
      ... It's not RAM that ZFS really likes but your KVA (Kernel Virtual Address) ... memory shared between various kernel systems and bounded to be smaller ...
      (freebsd-current)
    • Re: reproducable "panic: pmap_enter: attempted pmap_enter on 4MB page"
      ... Sam Leffler writes: ... > NFS server. ... For me it seemed to always happen when a particular kva was used. ... WITNESS and INVARIANTS kernel. ...
      (freebsd-current)
    • Re: panic: vm_thread_new: kstack allocation failed
      ... On closer inspection this means that we've run out of KVA. ... through the kernel memory allocations to see if a subsystem has gone ... > I just realised there is no effective limit on kernel threads in the system. ... > probably one could cause this with a fork bomb appoach using forks and ...
      (freebsd-stable)
    • Re: Linux disk performance.
      ... the main place I have found O_DIRECT useful is in keeping programs doing large i/o quantities from blowing the buffers and making the other applications run like crap. ... If you application is running alone, unless you are very short of CPU or memory avoiding the copy to an o/s buffer will be down in the measurement noise. ... I had a news server which normally did 120 art/sec (~480 tps), which dropped to about 50 tps when doing large file copies even at low priority. ... If you upgrade to a newer kernel you can try other i/o scheduler options, default cfq or even deadline might be helpful. ...
      (Linux-Kernel)