Re: Looking for speed increases in "make index" and pkg_version for ports

On 2007-May-27 15:52:16 -0500, Stephen Montgomery-Smith <stephen@xxxxxxxxxxxxxxxxx> wrote:
I have been thinking a lot about looking for speed increases for "make
index" and pkg_version and things like that. So for example, in
pkg_version, it calls "make -V PKGNAME" for every installed package. Now
"make -V PKGNAME" should be a speedy operation, but the make has to load in
and analyze bsd.port.mk, a quite complicated file with about 200,000
characters in it, when all it is needing to do is to figure out the value of
the variable PKGNAME.

This would be trivial if some packages didn't change names depending
on options and what was installed. I agree that parsing a 210KB file
17,000 times is not going to be fast. Especially since some ports
include bsd.ports.mk multiple times...

I suggest rewriting "make" so that variables are only evaluated on a "need
to know" basis.

This sounds like a good idea but I suspect it's not going to be
feasible. The biggest problem I see is that the make language allows
variables to be expanded either when they are assigned or when they
are referenced. If a variable expansion is delayed from the
assignment to the first use then the expansion must be performed using
the state of make as it was when the variable was assigned. The cost
of keeping this state probably exceeds the cost of actually evaluating
the variable.

So, for example, if all we need to know is PKGNAME, there
is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of
that particular port has done something like having PORTNAME depend on the
value of _RUN_LIB_DEPENDS.

Rather than trying to develop a tool that can quickly expand PKGNAME
irrespective of what convoluted code the author has used, how about a
partial solution? For most ports, PKGNAME depends solely on 3 or 4
variables that are statically defined in the port Makefile. The
obvious solution would seem to be to develop a script that can handle
the easy cases itself and punt the difficult cases back to make. The
definition of 'easy' can be adjusted over time to increase performance.
This approach would seem to have a relatively low bar to entry whilst
offering good effort/performance tradeoff at the low end.

The various depends lists would seem amenable to the same approach -
though the entry level tool will have far lower coverage due to the
extensive use of USE_GNOME=... and similar 'macro'-style constructs.

--
Peter Jeremy

Attachment: pgpV6zOthx36X.pgp
Description: PGP signature

Relevant Pages

  • Re: Looking for speed increases in "make index" and pkg_version for ports
    ... So for example, in pkg_version, it calls "make -V PKGNAME" for every installed package. ... It basically does what you suggest, except it keeps a list of ports it has already checked, so that it doesn't do them over again. ... I have done profiling tests on make, and in its current form, bsd.ports.mk actually spends rather little time inside of bin/sh. ... functions in GNU make which could help in regards to ports. ...
    (freebsd-hackers)
  • Re: Looking for speed increases in "make index" and pkg_version for ports
    ... "make -V PKGNAME" should be a speedy operation, but the make has to load in ... the variable PKGNAME. ... heavy reliance on /bin/sh is part of why the ports collection is slow. ... functions in GNU make which could help in regards to ports. ...
    (freebsd-hackers)
  • Re: Looking for speed increases in "make index" and pkg_version for ports
    ... Make is a program primarily designed for figuring out which was made first, the target or the source, but in the ports what we really want is a scripting language that presides over "cd WKSRC; ... Mk/*.mk what is is needed is a ports server and a thin CLI ... This server can store dependency data in an efficient manner, ... So for example, in pkg_version, it calls "make -V PKGNAME" for every installed package. ...
    (freebsd-hackers)
  • pkg_create
    ... When you do make install in a ports application /usr/ports/... ... Does it create a pkg in /var/db/pkg/... ... And do you use pkg_create like this pkg_create pkgname ...
    (freebsd-questions)
Quantcast