Re: Call for input on comp.unix.shell FAQ

From: Joe Halpin (j.p.h_at_comcast.net)
Date: 07/12/03 

Date: Sat, 12 Jul 2003 19:22:12 GMT

Stephane CHAZELAS <stephane_chazelas@yahoo.fr> writes:

> Faux_Pseudo wrote:
> [...]
> > I think that a FAQ for this group would be great and I would love to
> > help put it together so that we can stop answering the same old
> > questions and just tell people to FAQ it.
> [...]
>
> Agreed. So, Joe, did you start something? You could post your
> first draft to this group once you've got something, so that we
> can comment or complete it.

This isn't exactly a first draft. However, I'm pretty busy now, and
things are going more slowly than I anticipated. So here's what I have
to this point. If anyone wants to contribute an answer that isn't
there yet, I'll be happy to take it :-)

Also, please let me know if you see anything that isn't correct.

Archive-name: comp-unix-shell-faq/faq/cus-faq
Version: $Id$
Maintained By: Joe Halpin <j.p.h@comcast.net>

This FAQ contains the answers to some Frequently Asked Questions
often seen in comp.unix.shell. It spells "unix" in lower case letters
to avoid arguments about whether or not Linux, FreeBSD, etc are
unix. That's not the point of this FAQ, and I'm ignoring the issue.

There are two levels of questions about shells.

One is the use of the shell itself as an interface to the operating
system. For example, "how do I run a program in the background, and go
on with other things?". Or "how do I setup environmental variables
when I log in?".

The other level is how to write shell scripts. This often involves
having the shell execute unix utilities to perform part of the work
the shell script needs to accomplish, and requires knowledge of these
utilities, which isn't nominally in the scope of shell
programming. However, unless the question involves something other
than standard unix utilities, it should be included in this FAQ.

Standard unix utilities are defined by either POSIX or the Single Unix
Specification. These are now joined and are normally abbreviated as
"POSIX/SUS". This specification can be found at

http://www.opengroup.org/onlinepubs/007904975/toc.htm

The man pages found on that web page define standard behavior for any
given utility (or the shell itself). However, you should also check
the man page on your system for any utility or shell you need to
use. There isn't always a perfect correspondence between the standard
and a particular implementation (in fact, I'm not sure there's any
case in which they perfectly correspond).

Other good web sites that provide information about shells and shell
programming (including OS utilities) include:

http://www.shelldorado.com/
http://www.faqs.org/faqs/by-newsgroup/comp/comp.unix.shell.html

The answers given in this FAQ are provided with the best intentions,
but they may not be accurate for any particular shell/os. They may be
completely wrong. If you don't test the answers, that's a bug in your
procedures. There are no guarantees for the answers given here.

If you do test the answers and find a problem, please send email to
the maintainer, so it can be corrected. However, under no
circumstances will the maintainer be held liable for mistakes in this
FAQ. If the answers work for you, well and good. If not, please tell
me and I'll modify them appropriately so that this will be more
useful. If you don't agree with this, stop reading here.

A number of people have contributed to this FAQ, knowingly or
unknowingly. Some of the answers were taken from previous postings in
the group, and other people contributed questions and answers
directly to the maintainer, which you are welcome to do as well.

(For the group -- Please let me know if you'd like your name and/or
email address listed here when you contribute something)

CONTENTS:

0. Glossary
1. How can I send e-mails with attached files?
2. How can I generate random numbers in shell scripts?
3. How can I automatically transfer files using FTP with error checking?
4. How can I remove whitespace characters within file names?
5. How can I automate a telnet session?
6. How do I get yesterdays date?
7. Why did someone tell me to RTFM?
8. Why is csh harmful to my health?
9. How do I create a lock file?
10. How to get a date with x <time unit> of difference?
11. What is way to remane files w/ odd charcaters in their name?
12. How can (CRLF|LF|LFCR) be translated into one of the other?
13. How a shell prompt can be set up to change the title of xterm?
14. How to batch a FTP download/upload?
15. How to get the exit code of cmd1 in cmd1|cmd2
16. "script.sh: no found" while script starts with "#!/bin/sh" (^M issue)
17. Why one shouldn't use "echo" in a script?
18. how to loop through files with spaces in their name?
19. how to change my login shell to zsh?
20. when to use shells instead of perl/python/ruby/tcl...
21. why shouldn't I use csh?
22. why is shell programming considered harmful?
23. what shell should I use for scripting?
24. how to reverse a file?
25. how to remove last 10 lines?
26. how to get a file size, a file modification time?
27. how to get a process id by process argv[0]?
28. how to get a script to update my current environment?
29. how to find files less than 15 minute old?
30. how can I know wether my script is portable?
31. how do I rename *.foo to *.bar?
32. how to redirect stdout to file1 stderr to file2 while still
    seeing them on screen?
33. how to loop through *.txt files and treat the case when
    there's no such file (+ dotfile issue).
34. why isn't var set in "cmd | read var"
35. how to use shell variables in awk scripts
36. date manipulations
37. how to input the user with a timeout?
38. how to input user for only one char?
39. why isn't my .profile read?
40. known incompatibilities/limitations in sed/grep/awk/...
41. why do I get "[5" not found in "[$1 -eq 2]"?
42. how to deal with dot files?
43 why is zsh the best shell? ;)
44. How to exactly display the content of $var (with a \n appended).
45. How to exactly display the content of $var (without a \n
    appended).

ANSWERS

0. Glossary

   a. shebang

      This is the first line of a shell script, which indicates to the
      operating system which interpreter (shell) it should invoke to
      interpret the script. It has the form

      #!/path/to/shell [ argument ]

      where /path/to/shell might be /bin/sh, /usr/local/bin/bash, etc.

      This line is only interpreted by the operating system. That is,
      if a shell script (test.sh) is executable and run from the
      command line by typing its name.

      If it's run by typing

      $ sh test.sh

      then sh is run with the argument "test.sh". It then interprets
      test.sh. For sh, the shebang line is simply a comment, and is
      ignored.

   b. race condition

      This is a situation in which two entities (processes, threads,
      etc) are trying to access a shared resource, or perform the same
      action, and the result depends on the order of execution of the
      two entities.

1. How can I send e-mails with attached files?

   a. Use uuencode
   
      This is the simplest way to do this. For example

      $ uuencode surfing.jpeg surfing.jpeg | mail sylvia@home.com

      To send regular text as well

      $ (cat mailtext; uuencode surfing.jpeg surfing.jpeg) |
        mail sylvia@home.com

   b. Use MIME

      $ metasend -b -t john@friends.org -s "Hear our son!" \
            -m audio/basic -f crying.au

      These examples are taken from
      http://www.shelldorado.com/articles/mailattachments.html which
      goes into much more detail about this.

2. How can I generate random numbers in shell scripts?

   This depends on the shell, and the facilities available from the
   OS.

   a. Some shells have a variable called RANDOM, which evaluates to a
      different value every time you dereference it. If your shell has
      this variable,

      $ number=$RANDOM will produce a random number.

   b. Some systems have a /dev/urandom device, which generates a
      stream of bits. This can be accessed using the dd(1) utility. An
      example of this (from a more extensive discussion of different
      techniques at http://www.shelldorado.com/scripts/cmds/rand)

      n=`dd if=/dev/urandom bs=1 count=4 2>/dev/null | od -t u4 | \
      awk 'NR==1 {print $2}'`

   c. Use a utility such as awk(1), which has random number generation
      included. This approach is the most portable between shells.

      awk -v seed="$$" 'BEGIN {srand(seed);print rand()}'

      However, note that if you call this line more than once without
      changing the seed, you'll get the same number you did the
      previous time.

3. How can I automatically transfer files using FTP with error
   checking?

    First, there are tools to do that: curl, wget, lftp, ncftp. But, they
    are generally not part of the base system (you need to install them).
    
    zsh (version 4 and above) provides a FTP facility, see
    "info -f zsh -n 'zsh/zftp Module'"
    
    #! /usr/bin/zsh
    zftp open host user passwd || exit
    zftp get /remote/file > /local/file; r=$?
    zftp close && exit r
    
    With your system "ftp" command, two ways:
    
    1- using "ftp -n". Without the -n option, ftp expects user interaction
    to enter the password, so you'd need to use "expect". With "-n", you
    provide the user and passwd as any other FTP command.
    
    #! /bin/sh
    ftp -n << EOF
    open ftp.domain.org
    user anonymous ${LOGNAME:-`who am i`}@
    binary
    get /remote/file /local/file
    bye
    EOF
    
    The error checking can't be made correctly (if "open" fails, the "user"
    command will be still sent even if it shouldn't).
    
    2- using ~/.netrc
    
    If you put:
    
    <<
    machine ftp.domain.org
    login mylogin
    password mypasswd
    macdef init
      binary
      get /remote/file /local/file
      bye
    
    
>>
    
    (with the trailing empty line) in your ~/.netrc (ensure it's not world
    readable) and then run "ftp ftp.domain.org", ftp will find the matching
    "machine" entry in your ~/.netrc and use the parameters provided there
    to make the ftp transaction.
    
    Those work at least on Linux, FreeBSD, Solaris, HPUX

4. How can I remove whitespace characters within file names?

   a. Use the substitution capabilites of awk, sed, et al.

      f=`echo "$filename" | sed 's/ /_/g'` #replace with '_'

      f=`echo "$filename" | awk '{gsub(" ","_");print $0}'`

   b. Use the substitution capabilities of the shell if it has
      them. Check the man page for your shell (probably under a
      section named something like "Parameter expansion") to see. The
      following works with bash

      f=${filename// /_}

5. How can I automate a telnet session?

   This is outside the realm of shell programing, per se. You need
   a more special purpose scripting language such as expect. See
   http://expect.nist.gov/

   Perl scripts can also do this with the Telnet module from CPAN.

6. How do I get yesterdays date?

   (There are a number of responses to this on Google Groups. However
   there are so many options that I'm not sure how to approach
   this. I'm not sure duplicating the threads on Google is appropriate
   for a FAQ. Suggestions would be appreciated.)

7. Why did someone tell me to RTFM?

   Because you didn't :-)

   RTFM is part of Usenet lingo, and means "Read The F-ing Manual".
   Generally people say this when someone asks a question that is
   asked so often, and is answered plainly in some relevant man page,
   that they're tired of seeing it asked.

   So RTFM, and the FAQs first before asking. Also, if you're new to
   the group, search Google Groups

   http://groups.google.com/advanced_group_search

   before asking questions. And please don't post your homework
   questions to the group unless you've tried to figure them out, and
   have some specific questions. People will be generally be happy to
   help you with your homework if you post what you've got and ask
   specific questions.

8. Why is csh harmful to my health?

9. How do I create a lock file?

   Very carefully :-)

   The scheduler can stop one process in the middle of a non-atomic
   operation, and run another one, which wants to perform the same
   operation. The second one, having a full timeslice, might finish
   the operation. When control returns to the first process, confusion
   will reign.

   The trick is to do something atomic, so that this won't
   happen. There are a couple ways to do this. One is to create a
   directory instead of a file, the other is to create a symbolic
   link. Both operations are defined to be atomic by POSIX/SUS, by
   virtue of the fact that they both require calls to the
   corresponding system calls, which are atomic. Even here though,
   there is a window of opportunity for bad things to happen. There is
   some startup code in the command line version of mkdir(1) and
   ln(1), which could get interrupted before the system call is
   invoked. However, this is as close as we can get to perfection in a
   shell script.

   For the same reason, beware of testing for the existence of the
   lock file before trying to create it. That will just increase the
   window of possibility for a race condition.

   Be even more afraid of trying to create ANY kind of lock file on an
   NFS partition. NFS pretty much eliminates anything like
   atomicity. If you're going to create a lock file, make sure you're
   doing it on a local partition, such as /tmp.

   Netscape/Mozilla uses the symbolic link method for its lockfile (in
   spite of the fact that it creates it in the user's home directory,
   which may be NFS mounted). When it starts up it creates a file
   named for the IP address of the machine it's running on, and the
   pid of the creating process. Then it tries to create a symbolic
   link named "lock", which points to that file. If this symlink
   already exists, link(2) will return an error. In a script this
   would work something like

   touch /tmp/$$
   ln -s /tmp/$$ /tmp/lockfile 2>/dev/null
   if [ $? -ne 0 ];then
     echo lockfile already exists
     rm /tmp/$$
     exit 1
   else
     echo success
     rm /tmp/$$
   fi

   If you have procmail installed, another possiblity is the
   lockfile(1) command that comes with it. Many of the same caveats
   given above apply to that as well though.

10. How to get a date with x <time unit> of difference?

11. What is way to remane files w/ odd charcaters in their name?

12. How can (CRLF|LF|LFCR) be translated into one of the other?
13. How a shell prompt can be set up to change the title of xterm?
14. How to batch a FTP download/upload?
15. How to get the exit code of cmd1 in cmd1|cmd2

    First, note that cmd1 exit code could be non-zero and still don't
    mean an error. This happens for instance in

    cmd | head -1

    you might observe a 141 (or 269 with ksh93) exit status of cmd1,
    but it's because cmd was interrupted by a SIGPIPE signal when
    "head -1" terminated after having read one line.

    To know the exit status of the elements of a pipeline
    cmd1 | cmd2 | cmd3

    a. with zsh:

       The exit codes are provided in the pipestatus special array.
       cmd1 exit code is in $pipestatus[1], cmd3 exit code in
       $pipestatus[3], so that $? is always the same as
       $pipestatus[-1].

    b. with bash:

       The exit codes are provided in the PIPESTATUS special array.
       cmd1 exit code is in ${PIPESTATUS[0]}, cmd3 exit code in
       ${PIPESTATUS[2]}, so that $? is always the same as
       ${PIPESTATUS: -1}.

    c. with any other Bourne like shells

       You need to use a trick to pass the exit codes to the main
       shell. You can do it using a pipe(2). Instead of running
       "cmd1", you run "cmd1; echo $?" and make sure $? makes it way
       to the shell.

       exec 3>&1
       eval `
         # now, inside the `...`, fd4 goes to the pipe
         # whose other end is read and passed to eval;
         # fd1 is the normal standard output preserved
         # the line before with exec 3>&1
         exec 4>&1 >&3 3>&-
         {
           cmd1 4>&-; echo "ec1=$?;" >&4
         } | {
           cmd2 4>&-; echo "ec2=$?;" >&4
         } | cmd3
         echo "ec3=$?;" >&4
       `

    d. with a POSIX shell

       You can use this function to make it easier:

       run() {
         j=1
         while eval "\${pipestatus_$j+:} false"; do
           unset pipestatus_$j
           j=$(($j+1))
         done
         j=1 com= k=1 l=
         for a; do
           if [ "x$a" = 'x|' ]; then
             com="$com { $l "'3>&-
                         echo "pipestatus_'$j'=$?" >&3
                       } 4>&- |'
             j=$(($j+1)) l=
           else
             l="$l \"\$$k\""
           fi
           k=$(($k+1))
         done
         com="$com $l"' 3>&- >&4 4>&-
                    echo "pipestatus_'$j'=$?"'
         exec 4>&1
         eval "$(exec 3>&1; eval "$com")"
         exec 4>&-
         j=1
         while eval "\${pipestatus_$j+:} false"; do
           eval "[ \$pipestatus_$j -eq 0 ]" || return 1
           j=$(($j+1))
         done
         return 0
       }
       
       use it as:
       
       run cmd1 \| cmd2 \| cmd3
       exit codes are in $pipestatus_1, $pipestatus_2, $pipestatus_3

16. "script.sh: not found"

    a. While script starts with "#!/bin/sh" (^M issue)

       That's the kind of error that occurs when you transfer a file
       by FTP from a MS Windows machine. On those systems, the line
       separator is the CRLF sequence, while on unix the line
       separator is LF alone, CR being just another ordinary character
       (the problem is that it is an invisible one on your terminal
       (where it actually moves the cursor to the beginning of the
       line) or in most text editors or pagers).

       So, if a MSDOS line is "#!/bin/sh", when on a Unix system, it
       becomes "#!/bin/sh<CR>" (other names for <CR> are \r, \015, ^M,
       <Ctrl-M>).

       So, if you run the file as a script, the system will look in
       /bin for an interpreter named "sh<CR>", and report it doesn't
       exist.

       $ sed 'l;d;q' < script.sh
       #!/bin/sh\r$

       shows you the problem ($ marks the end of line, \r is the CR
       character).

    b. PATH issue

       Sometimes a shell is installed someplace other than /bin or
       /usr/bin. For example, a shell which was not part of the OS
       installation might be installed into /usr/local/bin. If the
       script was written on a machine which had ksh located in
       /usr/bin, but was run on a machine where ksh was located in
       /usr/local/bin, the shebang line would not resolve correctly.

       This is unlikely to occur when using sh. However, if the shell
       is bash, zsh, et al, it might be installed in different places
       on different machines.

       One way around this is to use the env command in the shebang
       line. So instead of

       #!/bin/sh

       use

       #!/usr/bin/env sh

       Of course, env might itself live in some other directory than
       /usr/bin, but it's not likely.

17. Why one shouldn't use "echo" in a script?

18. how to loop through files with spaces in their name?

    So, you're going to loop through a list of files? How is this list
    stored? If it's stored as text, there probably was already an
    assumption about the characters allowed in a filename. Every
    character except '\0' (NUL) is allowed in a file name on Unix. So
    the only way to store a list of file names in a file is to
    separate them by a '\0' character (if you don't use a quoting
    mechanism as for xargs input).

    Unfortunately most shells (except zsh) and most standard unix text
    utilities (except GNU ones) can't cope with "\0"
    characters. Moreover, many tools, like "ls", "find", "grep -l"
    output a \n separated list of files. So, if you want to
    postprocess this output, the simpler is to assume that the
    filenames don't contain newline characters (but beware that once
    you make that assumption, you can't pretend anymore your code is
    reliable (and thus can't be exploited)).

    So, if you've got a newline separated list of files in a
    list.txt file, Here are two ways to process it:

    1-

    while IFS= read -r file <&3; do
      something with "$file" # be sure to quote "$file"
    done 3< list.txt
    (if your read doesn't have the "-r" option, either make another
    assumption that filenames don't contain backslashes, or use:

    exec 3<&0
    sed 's/\\/&&/g' < list.txt |
    while IFS= read file; do
      something with "$file" <&3 3<&-
    done
    )

    2-

    IFS="
    " # set the internal field separator to the newline character
      # instead of the default "<space><tab><NL>".
    
    set -f # disable filename generation (or make the assumption that
           # filenames don't contain *, [ or ? characters (maybe more
           # depending on your shell)).
    
    for file in $(cat < list.txt); do
      something with "$file" # it's less a problem if you forget to
                             # quote $file here.
    done
    
    Now, beware that there are things you can do before building
    this list.txt. There are other ways to store filenames. For
    instance, you have the positionnal parameters.
    
    with:
    set -- ./*.txt
    
    you have the list of txt files in the current directory, and no
    problem with weird characters. Looping through them is just a
    matter of:
    
    for file
    do something with "$file"
    done
    
    You can also escape the separator. For instance, with
    
    find . -exec sh -c 'printf %s\\n "$1" | sed -n '"':1
      \$!{N;b1
      }
      s/|/|p/g;s/\n/|n/g;p'" '{}' '{}' \;
      
    instead of
    
    find . -print
    
    you have the same list of files except that the \n in filenames
    are changed to "|n" and the "|" to "|p". So that you're sure
    there's one filename per line and you have to convert back "|n"
    to "\n" and "|p" to "|" before refering to the file.

19. how to change my login shell to zsh?

20. when to use shells instead of perl/python/ruby/tcl...

    a. Portability

       In many cases it can't be assumed that perl/python/etc are
       installed on the target machine. Many customer sites do not
       allow installation of such things. In cases like this, writing
       a shell script is more likely to be successful. In the extreme,
       writing a pure Bourne shell script is most likely to succeed.

    b. Maintainability

       If the script is one which serves some important purpose, and
       will need to be maintained after you get promoted, it's more
       likely that a maintainer can be found for a shell script than
       for other scripting languages (especially less used ones such
       as ruby, rexx, etc).

    c. Policy

       Sometimes you're just told what to use :-)

21. why shouldn't I use csh?

   http://www.softlab.ntua.gr/facilities/documentation/unix/grymoire/CshTop10.txt
   http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/
   
22. why is shell programming considered harmful?
23. what shell should I use for scripting?
24. how to reverse a file?

    Non-standard commands to do so are GNU tac and "tail -r". sed
    '1!G;h;$!d' is subject to sed limitation on the size of its hold
    space and is generally slow.
    
    The awk equivalent would be:
    
    awk '{l[n++]=$0}END{while(n--)print l[n]}'
    It stores the whole file in memory.
    
    The best approach in terms of efficiency portability and resource
    cosumption seems to be:
    
    cat -n | sort -rn | cut -f2-

    "cat -n" is not POSIX but appears to be fairly
    portable. Alternatives are "grep -n '^'", "awk '{print NR,$0}'",
    "nl" can't be used as it processes page headings.

25. how to remove last 10 lines?

26. how to get a file size, a file modification time?
27. how to get a process id by process argv[0]?
28. how to get a script to update my current environment?
29. how to find files less than 15 minute old?
30. how can I know wether my script is portable?
31 how do I rename *.foo to *.bar?
32. how to redirect stdout to file1 stderr to file2 while still
  seeing them on screen?
33. how to loop through *.txt files and treat the case when
  there's no such file (+ dotfile issue).
34. why isn't var set in "cmd | read var"
35. how to use shell variables in awk scripts
36. date manipulations
37. how to input the user with a timeout?
38. how to input user for only one char?
39. why isn't my .profile read?
40. known incompatibilities/limitations in sed/grep/awk/...
41. why do I get "[5" not found in "[$1 -eq 2]"?
42. how to deal with dot files?
43 why is zsh the best shell? ;)
44. How to exactly display the content of $var (with a \n appended).

    A: on POSIX systems or with shells with builtin printf (bash2,
    ksh93, zsh4.1, dash...)

    printf '%s\n' "$var"

    (except for memory/environment full errors, should be expected
    to work at least if $var is not longer than LINE_MAX (supposed
    to be at least _POSIX2_LINE_MAX == 2048), no harcoded limits in
    zsh/ksh/bash/dash builtins)

    ksh, zsh:
    print -r -- "$var"

    zsh:
    echo -E - "$var"

    Other bourne like shells:

    cat << EOF
    $var
    EOF
    (creates a temporary file and forks a process)

    expr "x$var" : 'x\(.*\)'
    (limited to 126 characters with some exprs, may return a
    non-null exit code).

    With ash:
    (unset a; ${a?$var}) 2>&1

45. How to exactly display the content of $var (without a \n
    appended).

    printf %s "$var" # posix
    print -rn -- "$var" # zsh/ksh
    echo -nE - "$var" # zsh

    awk 'NR>1{print ""}{printf("%s",$0)}' << EOF
    $var
    EOF

    (with awk limitations [line length and number of fields])

Relevant Pages

  • Re: days difference between two dates
    ... > I think the answers our FAQ gives and the examples it provides ... > one big advantage of shell scripts are that they run on ... > of tools that can be assumed to be installed, but a C compiler, ... > be preferred to a script that is 100% correct but that is hard ...
    (comp.unix.shell)
  • comp.unix.shell FAQ take three
    ... This FAQ contains the answers to some Frequently Asked Questions ... It spells "unix" in lower case letters ... The other level is how to write shell scripts. ... How do I get a script to update my current environment? ...
    (comp.unix.shell)
  • Re: Call for input on comp.unix.shell FAQ
    ... That's not the point of this FAQ, ... > One is the use of the shell itself as an interface to the operating ... > The other level is how to write shell scripts. ... > the shell script needs to accomplish, ...
    (comp.unix.shell)
  • Re: detect shell script language
    ... In the 'old' days, when a user typed in a command, the interactive shell would immediately pass it off to 'exec' to execute. ... So, on return from exec with an error status, the shell would fork a copy of itself to try and run the script. ... As a result of the above, it was hard to tell whether the script was a Bourne shell or C shell, so the convention was introduced of using the Bourne shell no op command, as the first line in a Bourne shell script. ...
    (Debian-User)
  • Re: awk here documents
    ... script called from shell to produce some pre-formatted blocks of code ... awk variable: ... or awk quotes with shell quotes ...
    (comp.unix.shell)