Re: NBUF limits on all OpenServer versions

From: Bela Lubkin (belal_at_sco.com)
Date: 08/26/03 

Date: Tue, 26 Aug 2003 11:10:43 GMT
To: scomsc@xenitec.ca

Frederico Fonseca wrote:

> After some searches on google, it seems that the value for NBUF in
> these systems is limited to around 450MB.
>
> If the above is correct, is there anyone that knows why this limit
> wasn't increased with 5.0.7?
> Technical explanation appreciated.
>
> If incorrect how do we change it to be above that limit as some of my
> colleagues would like to be able to do it.

The limit remains 450MB.

Buffer cache buffers are allocated out of kernel virtual addresses which
can be "direct mapped". These are addresses in the C0000000-DFFFFFFF
range. Kernel virtual addresses in this range can be converted to
physical addresses by subtracting C0000000; and conversely, physical
addresses below 20000000 can be converted to kernel virtual addresses by
adding C0000000. (Of course there are macros that should be used. You
wanted the technical details...)

The total range in question is 1/2 gigabyte. The actual limit of 450MB
was arbitrarily set to leave some direct-map space for other uses. It
could probably be pushed up to 475MB or something like that, but I
doubt you would find such a small increment helpful.

Why is the direct map important here? HBAs usually need to know the
physical address of a transfer buffer, since the DMA hardware speaks
physical addresses. Converting an virtual address to a physical address
is very cheap for addresses in the direct map. It's expensive for other
addresses, requiring the kernel to walk data structures looking for the
translation. HBA drivers typically use the cheap macros instead of the
expensive data-structure-walking functions; especially since the buffer
cache is designed to always give them a direct-map buffer address.

Expanding beyond the direct map would require one of:

  (1) another bounce-buffer scheme where the buffer cache would always
      pass direct-map addresses to the HBA driver, then copy the data
      elsewhere (as is already done for HBA drivers that do ISA DMA to
      addresses < 16MB); or

  (2) new protocols between HBA drivers and the buffer cache, so the
      buffer cache would know which buffers never to deliver to a
      particular HBA. This would also require updated HBA drivers to
      take advantage of the new scheme.

>Bela<

Relevant Pages

  • Re: Cached file read performance
    ... 32MB fits in the buffer cache, ... time seconds seconds calls ns/call ns/call name ... Allocation involves vm stuff which is very expensive (takes about 1.25 ...
    (freebsd-performance)
  • Odd performance observation for RAID0
    ... building a RAID0 across 2 large hardware ... into a point of serialization of the buffer cache. ... on one RAID card, even though the RAID0 striped across 2 RAID cards. ...
    (Linux-Kernel)
  • Re: buffer cache question
    ... this modification is only for the *dirty* memory buffer. ... If know right, the kernel does buffering, not NBD! ... But the kernel only use ~830MB for buffer cache instead of dinamically use ...
    (Linux-Kernel)
  • Re: Can anyone explain this huge increase in logical I./O....?
    ... you have much more concurrency for these resources. ... It could be that your buffer cache is inadequately sized in PROD to support this to your satisfaction. ... The 'cache buffer chains' latch along with the 'buffer busy waits' indicates that the system is waiting for an available block in the buffer cache. ...
    (comp.databases.oracle.server)