Re: Adding new non-blocking sockets during select-call

Jeremy wrote:

Ok, much clear to me as to waht 'discoverable' is. I can understand the
io thread is doing read-ahead when data arrives, but how about writes?
are you assuming the other arbitrary thread is doing half-write (likely
to get a EWOULDBLOCK), then the other half been done by the io thread
when it discovers space available, or you assuming the whole write
operation is dispatched to the io thread? I had a stress test that
indicated the second approach would degrade perf. by about 10-20%, but
now sure if the first method is best (or elegant).

You can do it either way. I used to prefer having the other thread do
the write, falling back to the I/O thread only on EWOULDBLOCK. The
problem with that occurs in cases where a non-I/O thread might have to
write a small amount of data to a large number of sockets -- it bogs
down with all the user-space/kernel-space transitions. (If you have no
such cases, then it's fine, I think.)

On the flip side, it's also superfluous to call 'select' or 'poll'
before writing to a socket that one has not written to in a long time.
It is almost certain that a large amount of data will be able to be
written before blocking indications are returned.

I settled on a hybrid approach. When a non-I/O thread wants to send
data to a socket, it does the following: If we are already trying to
select/poll for write on the socket, just add the data to the
user-space send queue. If we are not, queue a 'speculative write' job
to the I/O pool. When an I/O thread dequeues a speculative write job,
it keeps calling 'write' until it either empties the user-space send
queue (in which case, it's done) or gets a blocking indication, in
which case it adds the socket to the write set.

Do you assume locks for the job queue and entering the 'poll' state?
what you described is more like how iocp behaves, but I dont know how
it would work without a complicated design (or even thread scheduling

If you held a lock on the job queue while in 'poll', non-I/O threads
couldn't queue I/O jobs while you were selecting or polling, which
would not be good.



Relevant Pages

  • Re: Adding new non-blocking sockets during select-call
    ... Add the listening socket to ... First I/O thread that has no I/O jobs to do calls 'select' so long ... All discovered sockets are added to a queue. ...
  • RE: Socket owner problem?
    ... Once the socket is closed by the user-space application, ... still packets left in the module's queue. ... What object is this queue logically associated with? ...
  • Re: [PATCH 1/3] Kernel interfaces for multiqueue aware socket
    ... Multiqueue and multicore provide packet parallel processing methodology. ... Current kernel and network drivers place one queue on one core. ... Current socket only can receive or send ... Kernel provides several interfaces by which sockets can be bound to rx/tx queues. ...
  • Re: OT: load distribution algorithm
    ... and hence its queue length becomes ... I realized I could eliminate the -1 flag value ... I also arbitrarily start the roundrobin with the worker holding the ... new socket arrives, the dispatcher always makes a simple choice. ...
  • Re: Interrupts handling in ADA
    ... the other three execute a "read" from network function inside ... Queue of entry calls: ... After the "pop" of the first event, can the fist socket continue its ... execution or does it have to wait until the queue is empty? ...