Re: sbrk(2) broken

For performance reasons, malloc(3) will hold on to a number of pages
that theoretically could be given back to the kernel, simply because
it expects to need them shortly.

Although the primary concern is malloc(), I would like to point out that
various programs implementing copying garbage collection could more
efficiently give memory back to the system than malloc(), and could therefor
benefit more than malloc() from some kind of feedback from the kernel.

There was concern over the complexity involved with intelligently doing
something about the memory pressure hints in userspace, but this does not
apply here since the allocator/garbage collection would be the equivalent of
malloc() and complexity there would not affect application code.

The problem with malloc() being that, unless I am missing something, malloc
will never be able to give back memory to the kernel except insofar as the
memory mapped is continuously unused between some location and the break (in
the case of sbrk()) or over the entire range (mmap()). malloc() cannot force
this to be the case, since pointers must remain valid. The possibility of
reclamation is then often going to be limited to completely unused space
being held by malloc() for future use, rather than also applying to areas
already used for allocation.

Programs implementing copying GC, or able to for some other reason to move
allocated memory around, could compact the heap and give back left-over
memory. In some cases this would only entail a temporary improvement due to
defragmentation, but in others (such as a long-running program spiking in
memory use, only then to drop a lot of that memory) it could have a pretty
massive effect on memory use.

Where a malloc() using program might be unable to sbrk() or munmap() because
there happens to be some left-over non-free piece of memory at the top of the
mapped range, a GC could use indications from the system to ensure this is
not the case (depending on details of the implementation; for example,
compactation of tenured generations could be forced early, etc).

(This is not to say I am aware of any implementation that actually supports
this, but on the other hand perhaps that is due to the lack of operating
systems that provide the required feedback.)

/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <peter.schuller@xxxxxxxxxxxx>'
Key retrieval: Send an E-Mail to getpgpkey@xxxxxxxxx
E-Mail: peter.schuller@xxxxxxxxxxxx Web:

Attachment: signature.asc
Description: This is a digitally signed message part.