Re: Adding new non-blocking sockets during select-call

Jeremy wrote:

Ok, much clear to me as to waht 'discoverable' is. I can understand the
io thread is doing read-ahead when data arrives, but how about writes?
are you assuming the other arbitrary thread is doing half-write (likely
to get a EWOULDBLOCK), then the other half been done by the io thread
when it discovers space available, or you assuming the whole write
operation is dispatched to the io thread? I had a stress test that
indicated the second approach would degrade perf. by about 10-20%, but
now sure if the first method is best (or elegant).

You can do it either way. I used to prefer having the other thread do
the write, falling back to the I/O thread only on EWOULDBLOCK. The
problem with that occurs in cases where a non-I/O thread might have to
write a small amount of data to a large number of sockets -- it bogs
down with all the user-space/kernel-space transitions. (If you have no
such cases, then it's fine, I think.)

On the flip side, it's also superfluous to call 'select' or 'poll'
before writing to a socket that one has not written to in a long time.
It is almost certain that a large amount of data will be able to be
written before blocking indications are returned.

I settled on a hybrid approach. When a non-I/O thread wants to send
data to a socket, it does the following: If we are already trying to
select/poll for write on the socket, just add the data to the
user-space send queue. If we are not, queue a 'speculative write' job
to the I/O pool. When an I/O thread dequeues a speculative write job,
it keeps calling 'write' until it either empties the user-space send
queue (in which case, it's done) or gets a blocking indication, in
which case it adds the socket to the write set.

Do you assume locks for the job queue and entering the 'poll' state?
what you described is more like how iocp behaves, but I dont know how
it would work without a complicated design (or even thread scheduling

If you held a lock on the job queue while in 'poll', non-I/O threads
couldn't queue I/O jobs while you were selecting or polling, which
would not be good.