Re: sbrk(2) broken



For performance reasons, malloc(3) will hold on to a number of pages
that theoretically could be given back to the kernel, simply because
it expects to need them shortly.

Although the primary concern is malloc(), I would like to point out that
various programs implementing copying garbage collection could more
efficiently give memory back to the system than malloc(), and could therefor
benefit more than malloc() from some kind of feedback from the kernel.

There was concern over the complexity involved with intelligently doing
something about the memory pressure hints in userspace, but this does not
apply here since the allocator/garbage collection would be the equivalent of
malloc() and complexity there would not affect application code.

The problem with malloc() being that, unless I am missing something, malloc
will never be able to give back memory to the kernel except insofar as the
memory mapped is continuously unused between some location and the break (in
the case of sbrk()) or over the entire range (mmap()). malloc() cannot force
this to be the case, since pointers must remain valid. The possibility of
reclamation is then often going to be limited to completely unused space
being held by malloc() for future use, rather than also applying to areas
already used for allocation.

Programs implementing copying GC, or able to for some other reason to move
allocated memory around, could compact the heap and give back left-over
memory. In some cases this would only entail a temporary improvement due to
defragmentation, but in others (such as a long-running program spiking in
memory use, only then to drop a lot of that memory) it could have a pretty
massive effect on memory use.

Where a malloc() using program might be unable to sbrk() or munmap() because
there happens to be some left-over non-free piece of memory at the top of the
mapped range, a GC could use indications from the system to ensure this is
not the case (depending on details of the implementation; for example,
compactation of tenured generations could be forced early, etc).

(This is not to say I am aware of any implementation that actually supports
this, but on the other hand perhaps that is due to the lack of operating
systems that provide the required feedback.)

--
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <peter.schuller@xxxxxxxxxxxx>'
Key retrieval: Send an E-Mail to getpgpkey@xxxxxxxxx
E-Mail: peter.schuller@xxxxxxxxxxxx Web: http://www.scode.org

Attachment: signature.asc
Description: This is a digitally signed message part.



Relevant Pages

  • Re: NTFS - Kernel memory leak in driver for kernel 2.4.28?
    ... I should say that the malloc() succeeds, but the 16mb I need for the ... buffer are not available. ... memory tied up in the inode and dentry cache. ... kernel attempts to use for the dentry/inode cache, or make it much, ...
    (Linux-Kernel)
  • Re: sbrk(2) broken
    ... How could you hide it inside malloc? ... memory becomes available. ... granular indication from the kernel about how bad things are. ... we could cache vnodes and inodes less agressively, ...
    (freebsd-current)
  • nommu: handling anonymous mmap clearing in userspace rather than kernel
    ... doing malloc() in userspace on no-mmu systems can take up ... mapped memory to zero. ... to signal to the kernel that it should skip the memset. ...
    (Linux-Kernel)
  • Thank You -- Thomas J. Gritzan
    ... Thomas -- Your suggestion to malloc() out a block of memory was the ... Below are some details of my memory issues ... ... As a work around solution I guessed a ram disk would solve the ... persistence will frustrate the off topic police and give them a target ...
    (comp.lang.c)
  • kernel 2.4 inode/dentry cache not clearing on umount?
    ... kernel is reporting only 3-4Mb free because the inode/dentry caches ... are taking up most of the memory, ... I assure you that under these conditions, the malloc() will fail with NULL. ... cache does not shrink and leaves only 3Mb of available free memory. ...
    (Linux-Kernel)