Re: Looking for speed increases in "make index"
- From: Peter Jeremy <peterjeremy@xxxxxxxxxxxxxxxx>
- Date: Tue, 29 May 2007 18:32:05 +1000
On 2007-May-28 00:15:28 +0200, Michel Talon <talon@xxxxxxxxxxxxxxxx> wrote:
Of course a lot of people have thinked about it, and quickly realized...
that it was not going to work. In the bsd.ports.mk, evaluation of one
variable may be dependent on some conditional, which may itself be
dependant on some other variable, appearing as some target. This
constantly happens in bsd.ports.mk.
To gain some performance, a first idea would be to simplify
bsd.ports.mk. I am convinced that a substantial part of the 4000 lines
are historical crap which serve no useful purpose. There are tons of
variables who have probably purely anecdotical interest. There are
targets which could be happily suppressed.
These two paragraphs are contradictory. If bsd.ports.mk really is
4000 lines of historical crap and full full of unused variables than
an "evaluate on demand" approach would be a big win.
A second idea would be to multithread make, since modern machines will
have a lot of cores.
Not yet. Dual-core machines are still relatively new. Of the seven
hosts that I regularly use, only one is dual core. Threading adds
significant overheads and it's not clear how it would help in this
specific instance.
Anyways, one of the things which cannot be a big factor is reading
bsd.ports.mk from hard disk.
Reading 210KB from a modern disk is quite fast.
It is certainly cached in memory when you
run make index.
Which is the specific case being discussed in this thread.
On the other hand it is so big that it probably doesn't
fit in cache, or probably only fits in caches of machines generously
endowed.
It may not stay in L1 cache but unless your system is extremely memory
stressed, it should stay in RAM. I'd suggest that if bsd.ports.mk is
not cached then you have more serious problems to address before you
worry about the performance of make(1).
(i am thinking Pentium 4 versus Core 2 Duo) is striking. It is less
than 10 minutes on the big cache machine versus 30 minutes on the small
cache one. If the cache effect is indeed dominant, then reducing the
size of bsd.ports.mk by all possible means would be very beneficial.
This is likely to be more a combination of the crippled pentium 4 core
and a largish memory footprint for make than anything to do with
bsd.ports.mk itself.
By the way an alternative system would be to use something other than
make to do the job.
This is a possibility but it needs to be something that can be part
of the base system. This means you are limited to C/C++, awk, sh
and make. In particular, python, ruby, perl and similar scripting
languages are out.
Whilst make seems the obvious choice for the ports infrastructure, in
reality, the infrastructure does not really need or use the sort of
implicit dependency tracking and target transformations that make
excels at. Of course, any alternative to make needs to provide a
language for defining dependencies that is equally powerful and easy
to use.
--
Peter Jeremy
Attachment:
pgpsrHI7QwsPt.pgp
Description: PGP signature
- References:
- Re: Looking for speed increases in "make index"
- From: Michel Talon
- Re: Looking for speed increases in "make index"
- Prev by Date: Re: Looking for speed increases in "make index" and pkg_version for ports
- Next by Date: Re: Looking for speed increases in "make index" and pkg_version for ports
- Previous by thread: Re: Looking for speed increases in "make index"
- Next by thread: Xorg port problem?
- Index(es):
Relevant Pages
|
|
