Re: Pipes and fd question. Large amounts of data.
From: Barry Margolin (barmar_at_alum.mit.edu)
Date: 01/31/05
- Next message: Floyd L. Davidson: "Re: Interesting problem with Serial Port"
- Previous message: Kazman: "Re: Interesting problem with Serial Port"
- In reply to: Johan: "Re: Pipes and fd question. Large amounts of data."
- Next in thread: Paul Sheer: "Re: Pipes and fd question. Large amounts of data."
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Sun, 30 Jan 2005 22:18:27 -0500
In article <10vq0pkf5ugou99@corp.supernews.com>,
"Johan" <me@knoware.nl> wrote:
> why not use sockets
You can't change from pipes to sockets without modifying the programs,
since you have to use different system calls to open them: socket() and
connect() in the writer; socket(), bind(), listen(), and accept() in the
reader. The programs presumably currently use open().
>
> John
>
> "Oded Shimon" <ods15@ods15.dyndns.org> schreef in bericht
> news:41fca6c2@news.012.net.il...
> >I have a rather unique situation. I have 2 programs, neither of which
> >have
> > control over.
> > Program A writes into TWO fifo's.
> > Program B reads from two fifo's.
> >
> > My program is the middle step.
> >
> > The problem - neither programs are aware of each other, and write into any
> > of
> > the fifo's at their own free will. They will also block until whatever
> > data
> > moving they did is complete.
> >
> > Meaning, if I were to use the direct approach and have no middle step, the
> > programs would be thrown into a deadlock instantly. as one program will
> > write
> > info fifo 1, and the other will be reading from fifo 2.
> >
> > The amounts of data is very large, GB's of data in total, and at least
> > 10mb
> > a
> > second or possibly as much as 300mb a second. So efficiency in context
> > switching is very important.
> >
> > programs A & B both write and read using large chunks, usually 300k.
> >
> > So far, my solution is using select() and non blocking pipes. I also used
> > large buffers (20mb). In my measurements, at worst case the programs
> > write/read 6mb before switching to the other fifo. so 20mb is safe enough.
> >
> > I have implemented this, but it has a major disadvantage - every 'write()'
> > only write 4k at a time, never more, because of how non-blocking pipes are
> > done. at 20,000 context switches a second, this method reaches barely 10mb
> > a
> > second, if not less.
> >
> > Blocking pipes have an advantage - they can write large chunks at a time.
> > They
> > have a more serious disadvantage though - the amount of data you ask to be
> > written/read, IS the amount of data that will be written or read, and will
> > block until that much data is moved. I cannot know beforehand exactly how
> > much data the programs want, so this could easily fall into a dead lock.
> >
> > Ideally, I could do this:
> > my program: write(20mb);
> > program B: read(300k);
> > my program: write() returns with return value '300,000'
> >
> > I was unable to find anything like this solution or similar.
> > No combination of blocking/non blocking fd's will give this, or any system
> > call.
> > I am looking for alternative/better suggestions.
> >
> > - ods15.
> >
-- Barry Margolin, barmar@alum.mit.edu Arlington, MA *** PLEASE post questions in newsgroups, not directly to me ***
- Next message: Floyd L. Davidson: "Re: Interesting problem with Serial Port"
- Previous message: Kazman: "Re: Interesting problem with Serial Port"
- In reply to: Johan: "Re: Pipes and fd question. Large amounts of data."
- Next in thread: Paul Sheer: "Re: Pipes and fd question. Large amounts of data."
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
