Re: Invoking external processes in threaded program

Hi Barry,

On Jun 22, 4:50 am, Barry Margolin <bar...@xxxxxxxxxxxx> wrote:
In article
<afcfa511-0d53-4ea8-a7c8-954e5b3d8...@xxxxxxxxxxxxxxxxxxxxxxxxxxx>,



Edd <e...@xxxxxxxxxxxxxxxx> wrote:
I'm trying to write a high level library to invoke and communicate
with an external process. On UNIX, this implies a fork-and-exec with
some pipes (or pseudo-terminals as I learned in another thread [1]).

Now, ideally I would like to be able to start multiple processes in
different threads, but the general advice with respect to fork()ing in
a multithreaded program seems to be that one shouldn't do anything in
the child other than exec*() immediately. I can see why after thinking
about it, but I didn't see this particular trap coming... :)

So now I can't see a way to set up my communication channels in the
child e.g. calling dup2() for the standard stream descriptors and
close()ing the un-used ends of the pipes.

I don't think there's any problem with doing these things in the child
process.

When you fork, only the thread that called fork() is duplicated, so you
don't have to worry about other threads continuing to do things before
the exec().


From what I understand, I do not believe this is the case. My
experiments also seem to lock up, but it's just about possible that I
could have goofed somewhere, I suppose :) Anecdotal evidence aside,
the theoretical problem is this:

Consider a threaded program that fork-and-execs. If a particular mutex
is locked before the fork by a thread other than the forking thread,
when the child attempts to acquire the mutex it will end up waiting
forever, because as you say only the forking thread is replicated in
the child and so the thread that took the lock is no longer around to
unlock the mutex.

The problem is somewhat solvable for my own mutexes; I "simply" make
sure that they're all locked by the forking thread when I fork,
allowing me to unlock them all in the same thread in the child. As far
as I understand it, pthread_atfork() is designed to streamline this
process. But one does not have this luxury with locks taken in C
library functions because the mutexes aren't exposed. Indeed, in my
case it is only the library mutexes that are of concern.

For example, in my test program I fire up 10 threads to invoke "ls" at
the same time and collect the output. I call close() on both sides of
the fork to close the ends of the pipes not in use. If one thread is
calling close() while another is forking, this spells trouble if
close() has taken a lock in the C library.

This summarises what is said in David R. Butenhof's Programming with
POSIX Threads (pages 198-200).

I hope I've got something wrong...

Kind regards,

Edd
.

Relevant Pages

  • Re: Invoking external processes in threaded program
    ... some pipes. ... the child other than exec*immediately. ... When you fork, only the thread that called forkis duplicated, so you ... is locked before the fork by a thread other than the forking thread, ...
    (comp.unix.programmer)
  • Re: Invoking external processes in threaded program
    ... is locked before the fork by a thread other than the forking thread, ... the child and so the thread that took the lock is no longer around to ... The problem is somewhat solvable for my own mutexes; ...
    (comp.unix.programmer)
  • Re: Invoking external processes in threaded program
    ... some pipes. ... the child other than exec*() immediately. ... The child is a single-threaded process so that it can fork and exec ...
    (comp.unix.programmer)
  • Re: c++: Run process and get stdout into string-variable
    ... create two pipes with pipe, one for stdin, one for stdout. ... in the child process returns 0) dup ... In the parent process (the one where fork) returns the PID of the ...
    (comp.os.linux.development.apps)
  • Re: Invoking external processes in threaded program
    ... I'm trying to write a high level library to invoke and communicate ... some pipes. ... the child other than exec*immediately. ... When you fork, only the thread that called forkis duplicated, so you ...
    (comp.unix.programmer)