Re: "tinderbox" using stacked unionfs

On Tuesday 26 January 2010 15:04:59 Gary Jennejohn wrote:
On Mon, 25 Jan 2010 20:39:11 +0200

David Naylor <naylor.b.david@xxxxxxxxx> wrote:
On Saturday 23 January 2010 12:33:14 David Naylor wrote:
Hi,

I have been experimenting with using unionfs to build ports in a
"tinderbox" environment. This avoids having to remove and extract
files for the build of each port and it allows the discovery of
installed/modified files by the port.

Please see attached for a (updated) shell script that handles all the
"heavy lifting" of building ports. Of importance is LOCALBASE and
BUILDDIR. If you want to override LOCALBASE please use `env` as the
script needs to know about it. BUILDDIR (/usr/build by default) is
where the script stores everything (including PKG_DBDIR).

Please see attached for an updated script. This no longer uses `sort -u`
but removed duplicates while maintaining dependency order. (See below)

ENOSCRIPT

See attached. Processes are still freezing periodically (requiring a
restart). The mtree problem appears when installing meta ports (x11/xorg-
apps, x11/xorg, etc).
#!/bin/sh

BUILDDIR=${BUILDDIR:-/usr/build}
LOCALBASE=${LOCALBASE:-/usr/local}
PORTSDIR=${PORTSDIR:-/usr/ports}
PORT_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_ports}
PKG_DBDIR=${PKG_DBDIR:-$BUILDDIR/db_pkg}
PACKAGES=${PACKAGES:-$BUILDDIR/packages}

MAKE="env LOCALBASE=$LOCALBASE PORTSDIR=$PORTSDIR PORT_DBDIR=$PORT_DBDIR PKG_DBDIR=$PKG_DBDIR PACKAGES=$PACKAGES make"

set -e

mkdir -p $BUILDDIR $LOCALBASE $PKG_DBDIR $PACKAGES

[ -n "$(kldstat -v | grep unionfs)" ] || kldload unionfs
[ ! -e $BUILDDIR/.installing_port ] || rm -r `cat $BUILDDIR/.installing_port` $BUILDDIR/.installing_port

port2name() {

echo $1 | sed 's|[/.-]|_|g'

}

port2pkg() {

local pkg_name=
local port=

port=$1; shift
eval pkg_name=PKG$(port2name $port)
eval pkg=\$$pkg_name
if [ -z "$pkg" ]
then
pkg=$($MAKE -C $port -V PKGNAME)
eval $pkg_name=$pkg
fi

}

depends() {

local depend=
local depends_name=
local _deps=
local name=
local port=
local type

type=$1
port=$2

eval depends_name=DEPEND_${type}_$(port2name $port)
eval deps=\"\$$depends_name\"

if [ -z "$deps" ]
then
echo "Getting $type dependancies for $port" > /dev/stderr

if [ "$type" = "build" ]
then
depend_list="$($MAKE -C $port -V EXTRACT_DEPENDS -V BUILD_DEPENDS -V LIB_DEPENDS -V RUN_DEPENDS)"
else
depend_list="$($MAKE -C $port -V LIB_DEPENDS -V RUN_DEPENDS)"
fi
for depend in $depend_list
do
name=$(echo $depend | cut -f 2 -d ':')
depends runtime $name
_deps="$_deps $deps $name"
done

deps=" "
for depend in $_deps
do
if [ -z "`echo "$deps" | grep " $depend "`" ]
then
deps="$deps$depend "
fi
done

[ "`echo $deps | tr ' ' '\n' | sort`" = "`echo $deps | tr ' ' '\n' | sort -u`" ]

depends_name=$depends_name
eval $depends_name=\"$deps \"
fi

}

run_make() {

set +e
trap "true" INT TERM EXIT
$MAKE "$@"
status=$?
trap - INT TERM EXIT
set -e

}

build() {

local _deps=
local dep=
local port=

port=$1

depends build $port
_deps="$deps"
for dep in $_deps
do
port2pkg $dep
if [ ! -d $BUILDDIR/$pkg ]
then
if ! build $dep
then
echo "Port $port failed due to dependency $dep" > /dev/stderr
return 255
fi
fi
done

echo "Building port $port..."

for pkg in $_deps
do
port2pkg $pkg
mount -t unionfs -r -o noatime $BUILDDIR/$pkg $LOCALBASE
done

run_make -C $port clean build -DNO_DEPENDS -DBATCH

if [ $status -eq 0 ]
then
port2pkg $port
mkdir -p $BUILDDIR/$pkg
mount -t unionfs -o noatime $BUILDDIR/$pkg $LOCALBASE

echo $BUILDDIR/$pkg > $BUILDDIR/.installing_port
run_make -C $port install package -DNO_DEPENDS -DBATCH
rm $BUILDDIR/.installing_port

[ $status -ne 0 -a -n "$NO_CLEANUP" ] || umount $LOCALBASE
fi

for pkg in $(echo $_deps | sort -r)
do
[ $status -ne 0 -a -n "$NO_CLEANUP" ] || umount $LOCALBASE
done

if [ $status -ne 0 ]
then
echo "Port $port failed to build" > /dev/stderr
port2pkg $port
rm -rf $BUILDDIR/$pkg || (chflags -R 0 $BUILDDIR/$pkg; rm -rf $BUILDDIR/$pkg)
else
$MAKE -C $port clean
fi

return $status

}

build /usr/ports/x11/xorg

Attachment: signature.asc
Description: This is a digitally signed message part.

Relevant Pages

  • Re: stacked unionfs freeze and crash FreeBSD
    ... The attached script, that uses stacked unionfs, causes FreeBSD-9 (cvsup ... eval pkg_name=PKG$(port2name $port) ... deps=$(for depend in $_deps ... echo "Port $port failed due to dependancy $dep" ...
    (freebsd-current)
  • Re: [Proof of Concept] Stacked unionfs based tinderbox
    ... stacked unionfs to implement a 'tinderbox' type build system. ... What I did back then was using a clean base system as the ... So for each port that has perl as ... local _deps= ...
    (freebsd-hackers)
  • "tinderbox" using stacked unionfs
    ... This avoids having to remove and extract files for the build of ... each port and it allows the discovery of installed/modified files by the port. ... Please see attached for a shell script that handles all the "heavy ... deps=$(for depend in $_deps ...
    (freebsd-current)
  • Re: How to Create a TCP/IP PrinterPort on a MS Cluster 2k3 via Scr
    ... XP is not in the cluster so I'm not clear on the failure question. ... the user running the script should be admin on the remote print server. ... thus the creation of the port on the node. ...
    (microsoft.public.windows.server.clustering)
  • Re: How to Create a TCP/IP PrinterPort on a MS Cluster 2k3 via Scr
    ... XP is not in the cluster so I'm not clear on the failure question. ... the user running the script should be admin on the remote print server. ... thus the creation of the port on the node. ...
    (microsoft.public.windows.server.clustering)