Re: Looking for speed increases in "make index" and pkg_version for ports

On Sun, May 27, 2007 at 03:52:16PM -0500, Stephen Montgomery-Smith wrote:
I have been thinking a lot about looking for speed increases for "make
index" and pkg_version and things like that. So for example, in
pkg_version, it calls "make -V PKGNAME" for every installed package. Now
"make -V PKGNAME" should be a speedy operation, but the make has to load in
and analyze bsd.port.mk, a quite complicated file with about 200,000
characters in it, when all it is needing to do is to figure out the value of
the variable PKGNAME.

I have a related question, pertaining to "make all-depends-list" and the
utter atrocity that is the make variable ALL-DEPENDS-LIST. If you don't
know what it is, look for ^ALL-DEPENDS-LIST around line 5175, in
bsd.ports.mk.

I call it an atrocity because it's a mix of make variable expansion
combined with sh scripting, and it's nearly impossible to read. It's
not commented either, so folks like myself are left thinking "What IS
this mess?!". It's expanded via $$(${ALL-DEPENDS-LIST}) in for loops,
throughout several places in bsd.port.mk.

I do not entirely understand what ALL-DEPENDS-LIST is about (that should
be apparent), but upon performing some of my benchmarks, I found this to
be a very slow piece of bsd.port.mk. make -V _DEPEND_DIRS is incredibly
fast, but ALL-DEPENDS-LIST is not.

Does it need to be done this way? Can we just iterate through all of
the ports, call make -V _DEPEND_DIRS, then sort | uniq the results? I
suppose that depends on the operation (make vs. make clean vs.
others)...

The port I used for testing some of the benchmarks was net/gacxtool. It
seems to be a good example of a "hefty" port.

I suggest rewriting "make" so that variables are only evaluated on a "need
to know" basis. So, for example, if all we need to know is PKGNAME, there
is no need to evaluate, for example, _RUN_LIB_DEPENDS, unless the writer of
that particular port has done something like having PORTNAME depend on the
value of _RUN_LIB_DEPENDS. So "make" should analyze all the code it is
given, and only figure it out if it is needed to do so. This would include,
for example, figuring out .for and .if directives on a need to know basis as
well.

This sounds like a good solution. In fact, I'm lead to believe that
heavy reliance on /bin/sh is part of why the ports collection is slow.
No, it's not the sole reason, but it's one of many. I'm of the belief
that anything we can do to migrate portions into native make would be
benefitial.

That said, I'll ask this out in the open: am I the only one who sees the
benefit of GNU make in regards to this? There's a lot of built-in
functions in GNU make which could help in regards to ports. I have no
qualms with PMake per se, but if another tool gives us what we need,
then maybe we should consider the pros and cons of adapting that.
There's also CMake, which is incredibly fast.

--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

_______________________________________________
freebsd-hackers@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: Looking for speed increases in "make index" and pkg_version for ports
    ... So for example, in pkg_version, it calls "make -V PKGNAME" for every installed package. ... It basically does what you suggest, except it keeps a list of ports it has already checked, so that it doesn't do them over again. ... I have done profiling tests on make, and in its current form, bsd.ports.mk actually spends rather little time inside of bin/sh. ... functions in GNU make which could help in regards to ports. ...
    (freebsd-hackers)
  • Re: Looking for speed increases in "make index" and pkg_version for ports
    ... the variable PKGNAME. ... assignment to the first use then the expansion must be performed using ... For most ports, PKGNAME depends solely on 3 or 4 ... though the entry level tool will have far lower coverage due to the ...
    (freebsd-hackers)
  • Re: Looking for speed increases in "make index" and pkg_version for ports
    ... Make is a program primarily designed for figuring out which was made first, the target or the source, but in the ports what we really want is a scripting language that presides over "cd WKSRC; ... Mk/*.mk what is is needed is a ports server and a thin CLI ... This server can store dependency data in an efficient manner, ... So for example, in pkg_version, it calls "make -V PKGNAME" for every installed package. ...
    (freebsd-hackers)
  • pkg_create
    ... When you do make install in a ports application /usr/ports/... ... Does it create a pkg in /var/db/pkg/... ... And do you use pkg_create like this pkg_create pkgname ...
    (freebsd-questions)
  • Re: Looking for speed increases in "make index" and pkg_version for ports
    ... "make -V PKGNAME" should be a speedy operation, but the make has to load ... in and analyze bsd.port.mk, a quite complicated file with about 200,000 ... value of the variable PKGNAME. ... Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org ...
    (freebsd-hackers)
Quantcast