Re: Bug in #! processing - One More Time

From: Maxim Sobolev (sobomax_at_portaone.com)
Date: 02/24/05

  • Next message: Garance A Drosihn: "Re: Bug in #! processing - One More Time" 
    Date: Thu, 24 Feb 2005 12:33:51 +0200
To: Garance A Drosihn <drosih@rpi.edu>

    Garance A Drosihn wrote:
    > Sometimes it's the simplest little changes which can suck the
    > life out of you... I am aware that this is a trivial issue,
    > but now that I've figured out what is really going on, I am
    > not sure what the "best" fix would be.
    >
    > To recap some history:
    >
    > a) In Jan 2000, someone sent in a PR that perl documentation
    > (including the famous "Camel" book from O'Reilly) claims
    > that users can start a script with the line:
    >
    > #!/bin/sh -- # -*- perl -*- -p
    >
    > to avoid a variety of issues when writing cross-platform
    > scripts. Ignore the question of "but why?" for the moment,
    > it *is* documented by perl (and in books on some other
    > scripting languages). He proposed a fix, and that was
    > committed to src/sys/kern/imgact_shell.c as revision 1.21
    > back in Feb 15 2000 (predating 4.0-release). It was MFC'ed
    > into release 3.5 on March 20, 2000.
    >
    > The PR is:
    > http://www.FreeBSD.org/cgi/query-pr.cgi?pr=16393
    >
    > NOTE: People *do* use this "feature".
    > Counter: This feature doesn't actually work on recent
    > releases of Redhat Linux. I don't know about
    > other linuxes.
    >
    > b) In 2002, some other user updated that PR saying that the
    > new behavior wasn't quite right either. I assume nothing
    > much was done at the time, but he spent time to collect
    > a lot of details (which will be given below).
    >
    > c) In 2004, after 5.3-release, the issue came up again. I assume
    > that is in another PR, but I haven't checked. In any case,
    > kern/imgact_shell.c was changed to remove that special
    > processing for '#, after discussion in -current. The change
    > was committed to HEAD (6.x) on October 31st as revision 1.27.
    > It was MFC'ed to 5.3-stable on November 8th.
    >
    > This broke scripts which depended on the special-handling of
    > '#', but the conclusion in -current was that /bin/sh should
    > handle such processing (if it wanted to), and not execve().
    >
    > d) In January I was finally bitten by this running 6.x-current,
    > and a friend of mine happened to get hit by it at the same
    > time running 5.3-stable. So I wrote up a quick fix and did
    > some minimal testing. I posted that to -current on Jan 31st,
    > but I didn't want to commit it until I did more testing,
    > which I wanted to do *after* I brought my systems up-to-date.
    >
    > e) On January 29th, sobomax committed an "unrelated" fix to
    > kern/imgact_shell.c, except that it just happened to bring back
    > the special '#' processing which had been removed in October...
    >
    > f) I update my systems, do extensive testing of my patch, and I
    > committed it once I was confident it worked in all situations.
    > However, I didn't notice that the shell was no longer even
    > *seeing* the parameters after '#' (I had tested that part
    > back in #d), so it turns out the key loop I that had added
    > was never actually getting triggered.
    >
    > I committed it to 6.x-current last week.
    >
    > g) On Monday I get ready to MFC the change to 5.3 (ahead of the
    > rush to beat the code-freeze!). But... the damn thing does
    > NOT work right in some common situations!! WTF?!?
    >
    > So, I figure out all the above history, and I locally modify
    > kern/imgact_shell.c to again remove the special '#'-processing.
    > I go to fix my patch to /bin/sh, and I realize...
    >
    > There is no simple, "make everyone happy" fix for it. Sigh.
    >
    > The problem is in the way the execve() system call passes all
    > arguments to the shell. Given a shell named /tmp/list_args.pl,
    > which starts out as:
    > #!/bin/sh -x -- # -*- perl -*- -p
    >
    > and is executed via:
    > /tmp/list_args.pl aaa bbb
    >
    > What /bin/sh sees for arguments are:
    > arg[0] == '-x'
    > arg[1] == '--'
    > arg[2] == '#'
    > arg[3] == '-*-'
    > arg[4] == 'perl'
    > arg[5] == '-*-'
    > arg[6] == '-p'
    > arg[7] == '/tmp/list_args.pl'
    > arg[8] == 'aaa'
    > arg[9] == 'bbb'
    >
    > The problem is that /bin/sh has no way of knowing where the
    > "shebang-line options" end, and the "command-line options" start.
    > (or does it? I couldn't think of any reliable way, given that
    > the '#' could be followed by any totally arbitrary strings).
    >
    > Going back to the follow-up to PR 16393, part of the challenge
    > with fixing this is that many other OS's do *not* break up the
    > options on the shebang line the way FreeBSD does.
    > From the PR:
    >
    > Given a file called '/tmp/x2' with shebang line:
    > #!/tmp/interp -a -b -c #dee eee
    >
    > If /tmp/x2 is exec'd, the operating system runs /tmp/interp
    > with the following arguments:
    >
    > Solaris 8:
    > args: "/tmp/interp" "-a" "/tmp/x2"
    >
    > Tru64 4.0:
    > args: "interp" "-a -b -c #dee eee" "/tmp/x2"
    >
    > FreeBSD 2.2.7:
    > args: "/tmp/interp" "-a" "-b" "-c" "#dee" "eee" "/tmp/x2"
    >
    > FreeBSD 4.0:
    > args: "/tmp/interp" "-a" "-b" "-c" "/tmp/x2"
    >
    > Linux 2.4.12:
    > args: "/tmp/interp" "-a -b -c #dee eee" "/tmp/x2"
    >
    > Linux 2.2.19:
    > args: "interp" "-a -b -c #dee eee" "/tmp/x2"
    >
    > Irix 6.5:
    > args: "/tmp/interp" "-a -b -c #dee eee" "/tmp/x2"
    >
    > HPUX 11.00:
    > args: "/tmp/x2" "-a -b -c #dee eee" "/tmp/x2"
    >
    > AIX 4.3:
    > args: "interp" "-a -b -c #dee eee" "/tmp/x2"
    >
    > Mac OX X:
    > args: "interp" "-a -b -c #dee eee" "/tmp/x2"
    >
    > The most common behavior is:
    > argv[0]: full path of interpreter
    > argv[1]: all remaining args, coalesced into one string
    > argv[2]: The file file exec'd.
    >
    > The change committed back in 2000 made the comment: "This complies
    > to POSIX 1003.2, in that Posix says the implementation is free to
    > choose whatever it likes.". I actually like the idea that FreeBSD
    > splits up the arguments from the shebang-line, but that leaves us
    > with the problem of figuring out shebang-options from user-specified
    > options given on the command-line.
    >
    > As I see it, we have the following choices to fix this:
    >
    > 1) MFC the January 31st change to kern/imgact_shell.c to 5.3-stable,
    > as it is. This means we haven't fixed the problem that people
    > complained about in 2002 and again in 2004. And I still think
    > it is "not appropriate" for the execve() system to be deciding
    > what '#' means on that line. The biggest advantage is that this
    > means 5.4-release will behave exactly the same as 3.5 through
    > 5.3-release have behaved.
    >
    > 2) Remove '#'-processing from kern/imgact_shell.c, and remove my
    > change to bin/sh/options.c (which doesn't work right once we
    > do that). This breaks shell-scripts which use the feature as
    > documented by perl (and other scripting languages), and fixes
    > the problem people complained about in 2002/2004.
    >
    > 3) Change kern/imgact_shell.c to process shebang options the same
    > way other (non-BSD?) operating systems do. By that I mean:
    > send the entire string as arg[1], and let the scripting
    > language sort it out. This is an incompatible change from
    > FreeBSD 5.3 to 5.4, but would put make us "more consistent"
    > with other operating systems.
    >
    > 4) Provide some way for /bin/sh to find out where the shebang
    > options end, and the user-specified options begin. This could
    > make everyone happy, but it's more work and right now (this
    > close to 5.4-release) that wouldn't make me particularly happy...
    >
    > Or we could do #1 for now, and plan to do #4 after 5.4-release.
    > Or do #1 now in 5.3, and go with some incompatible change (#2
    > or #3) only in 6.x-current.
    >
    > What do people think? I know this is a mind-numbingly trivial
    > issue to care about, but I figured that if I just went ahead
    > with any particular solution, someone would be irritated with me
    > and assume I must not have understood "the issues". They will
    > then commit yet *another* change which undoes whatever I did,
    > while they fix something they feel that I broke.
    >
    > And if nothing else, this is proof that one can't just blindly
    > MFC some change, no matter now trivial it seems.

    I would vote for making #3 and respective /bin/sh changes and MFCing
    them into 5.4. We don't have that many shell scripts that rely on the
    previus functionality - ones that in the base system (if any) can be
    easily fixed, while ones in /usr/ports can be conditionalized on
    OSVERSION. Removing yet another superfluous difference between FreeBSD
    and other systems out there is good thing especially considering that
    BSD-way creates serious problems that can't be resolved without changing
    semantics anyway.

    -Maxim
    _______________________________________________
    freebsd-arch@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-arch
    To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"

  • Next message: Garance A Drosihn: "Re: Bug in #! processing - One More Time"

    Relevant Pages

    • Re: FreeBSD Security Advisory FreeBSD-SA-01:40.fts [REVISED]
      ... >> Now this is the only Advisory that is actual for FreeBSD 3.5-STABLE ... >> (e.g. that does not provide fix). ... Will the PR remain open forever if nobody will commit this? ...
      (FreeBSD-Security)
    • Re: Differences between GNU and FreeBSD C library
      ... problems if the code you are porting relies on non-portable ... Even if you're not up to submitting a fix for the problem yet, ... a precise description of where FreeBSD departs from ... bug reports which also include a patch are very ...
      (comp.unix.bsd.freebsd.misc)
    • Re: Internet Explorer Address Bar Problem (Not Responding)
      ... I've actually seen this fix suggested on numerous posts. ... I had been hoping that Microsoft or HP would eventually ... come out with a scripting file of their own to rectify the problems caused by ... A windows deep scan showed a problem with an "AVEEO" but said it could not ...
      (microsoft.public.windows.inetexplorer.ie6.browser)
    • Re: /boot/loader Invalid Format
      ... if someone has freebsd release as you, let him upload it to a site, with ... then boot your computer with the fixit cd of freebsd ... (founded this trick out with a friend of mine who messed up her bootloader ... >> I've got a problem with my bootloader and i dont know how to fix it. ...
      (freebsd-hackers)
    • Re: remaining FreeBSD 4.11-RC3 bugs
      ... as a complete fix and stopped halfway for various reasons, ... If the FreeBSD 4.11-RC tcpdump is essentially the same as the FreeBSD ... The patch quoted in the PR ...
      (freebsd-stable)