Re: DPS Initial Ideas

On Sat, May 12, 2007 at 11:44:22PM +0200, Michel Talon wrote:
On Sat, May 12, 2007 at 03:33:02PM -0400, Kris Kennaway wrote:
On Sat, May 12, 2007 at 11:09:35AM +0200, Michel Talon wrote:

Seriously, the FreeBSD package
system is in great need of a profound overhaul, pretending it works well
is complete denial of reality. I hope that young people working on
summer code projects will infuse *new* ideas, and not spend their
vacations polishing inadequate tools.

I know that this is your belief, but please try to avoid grasping at
straws: there are elements in your argument that are along the lines
of "The FreeBSD package system is broken and needs to be fundamentally
changed. Rewriting it to use SQLite is a fundamental change.
Therefore rewriting it to use SQLite will fix the problems."


Really i don't think at all this way. I think that *perhaps* SQLite
may marginally better than a Berkeley database for solving part of the
problem, not much more. What i reacted to, was the conservatism which
pervades the community as soon as someone emits the idea of using a new tool.

It seems to me that you do not appreciate the reasons behind this
conservatism. A very important one is that we have two students who
have committed to spending their summer working on improving the
existing pkg_tools in ways that will solve some of the real problems
we are facing, and the project we have agreed upon is that they will
be using existing tools rather than rewriting from scratch as part of
a not-yet-defined larger project.

To some extent it is the timing here that is most unfortunate. If
SQLite has been raised as a viable alternative a few months ago it
would have made a great project idea, but instead we have committed to
improving our existing tools, and the barrier for throwing out these
plans is therefore very high. The burden of proof is therefore set
much higher than "SQL is awesome and buzzword-compliant and might be
better".

It may be that borrowing from Debian the idea of "abstract" dependencies
which can be fulfilled by several concrete packages may also simplify
the dependency problem. For example tomcat may depend on "java" and java
my be fulfilled either by diablo-jdk15 or jdk15. This way when you change
from diablo-jdk15 to jdk15 you don't need to change anything to tomcat.

Another feature that Debian has, and which may happily complete the previous
one, is the specification of necessary dependencies with a version number
in a certain range (this obviously requires a reasonable standardisation of
version numbers, so that comparison of <some package>-0.99 to
<some package>-1.0-rc doesn't depend on arcane rules). This way you don't need
to change dependencies which are in the correct range, even if a more recent
version exists. This mechanism has been imported in NetBSD pkgsrc.

We actually have both of these features in ports, but they have not
been pushed down into packages. I think it will be relatively simple
to do so, without requiring a rewrite from scratch.

And a problem which has proven useful in Debian is keeping track of the
packages which have been required by the end user and those which have been
installed as dependencies. This is the difference between apt-get and
aptitude. Apparently people are very happy to be able to remove not only
a package they have required, but also all its dependencies (which are
not required by another program) at one stroke. This also helps in case
some big package requires dependency A, but after upgrade, they have changed
their mind and require alternative dependency B. With this mechanism, after
upgrade A disappears, while without it you will have both an upgraded version
of A and B. I have observed on my machine this is an important cause
of time monotonic bloat of the package tree.

This one could also be added to the existing tools.

To answer the slowness problem in registering installed packages, one may
think about making use of the INDEX file. In fact all the information that
is necessary to fill the dependency entries is contained in INDEX, and
accessible here in milliseconds with any tool such as awk. It so happens that
the ports system doesn't make any use of the INDEX file and systematically
recomputes the dependencies through recursive make invocations which are very
time consuming. Of course this requires up to date INDEX, or a mechanism to
keep INDEX continually up to date.

The problem is that maintaining the INDEX is expensive and/or tricky.
p5-FreeBSD-Portindex comes close but seems to have some wrinkles.

Part of the registration is also filling the +REQUIRED_BY files of the
dependencies of a package when one installs a package. If this package has a
lot of dependencies this means opening, editing and closing a large number of
files. This is expensive. One may imagine using a database containing the
global dependency information, then +REQUIRED_BY files are no more necessary,
since the information can be recomputed in very little time.

Yes, this is one thing that is addressed in the current plan.

Given that this work is happening (or at least will be happening, I am
not sure when the SoC officially starts), the best thing is for
interested people to work with Garrett to help him achieve the goals
of his project.

Sure. I am convinced this is the reason why several people, including myself
present some ideas in the mailing list now, before Garrett begins working on
his project.

I think this is a retcon; the original poster did not appear to have
this motivation and neither do many others who have posted.

Kris

Attachment: pgpJXYXuMfDJ7.pgp
Description: PGP signature

Relevant Pages

  • Re: DPS Initial Ideas
    ... of "The FreeBSD package system is broken and needs to be fundamentally ... Rewriting it to use SQLite is a fundamental change. ... running a perl script to connect to a Berkeley database. ... It may be that borrowing from Debian the idea of "abstract" dependencies ...
    (freebsd-hackers)
  • Re: Separate Compilation in Programming Languages
    ... for the Package P1 to the package body of P2. ... no need to recompile anything, ... This is not true in the traditional Ada way when the dependencies are ... package Stacks is ...
    (comp.lang.ada)
  • debian-user-digest Digest V2005 #2076
    ... Re: Firefox and Debian Testing: Gett [Vincent Lefevre ... > If a package in testing is the same version as in stable, ... Once that happens the dependencies for the ...
    (Debian-User)
  • Re: packages, load paths and environment variables
    ... There is one reason. ... you only need the package definition of the first (you only ... Therefore when you have CL sources with crossed dependencies, ... and add a in the sources (implementation module) of A. ...
    (comp.lang.lisp)