Re: Scalable tcp server

On Sat, 07 Jul 2007 17:07:25 +0200, Henrik Goldman wrote:



So far I have not been able to profile it that much but perhaps I should
recompile it with gprof to see how it behaves.
However it's pretty straightforward what goes on. Since I know that most
clients are not doing anything the only thing that goes on is stuff related
to networking.

Profile.
[my hypothesis is that you waste too much time maintaining the linked
lists, or you are convoying on the semaphores that guard them. Or you
poll too much).



On the server side I have 2 linked lists using std::list.



This is a business demand in order to do remote statistics and get a
complete picture of who is connected to the server. It would not be
needed otherwise. However it's needed to take a snapshot of the current
server status.

Business demand does not dictate an implementation.
You could easily get your 'state report' by scanning an array and
counting the various states that the connections happen to be in.


IMHO, for dispatching you need only *one* list/queue. (this could even
be implemented as a bitmap, eg an fd_set)

Very true. fd_set bitmap will likely not be sufficient since each client
has an amount of status information associated. However everything is
wrapped into a connection object on the server side which also includes
the socket.


"normally" (eg without threads/objects), one would just use an array,
indexed by filedescriptor (since fd is guaranteed to be the lowest
available, this can be a fixed-size array). Each entry would contain all
the {state,data, buffers} wrt this connection.
In your case, you could use an array of pointers to the "objects"

A thread can take a task from(the head of) the list and execute
it. If
the task is finished, you are done, otherwise, the task can be re-added
to the work-list.

Thats exactly what happens. There are just more threads doing that.

If you protect the list-operations by a semaphore, ("latch") there is
probably nothing wrong with this.

2. Perform select() on each connection object and select for
readability and writability.

You call select() for every connection ? That would take 2 systemcalls
for every read... Why not let one centralized select() that adds work
to the worklist ?

Because most OS's has a limit of how many fd's can be polled. I know
that it's common that OS's has 64 as a limit. My idea to work around
this is to poll up to 64 at a time and then add them into the queue
again.
So instead of just processing 1 connection it processes up to 64 at
once.

I don't know about other OSses. For UNIX, there will always be a way to
select() on all your available fds. (the same goes for poll()) You may
have to do some tweaking, but is is possible.
If you insist on chopping up your fdset, (which can only be done when
using poll() BTW), there is one problem: you cannot afford to block inside
select/poll, so you are in-fact busy-polling. ( --> calling select/poll N
times, just to discover 1 readable fd)


If nothing needs to be done, the task should not *be* on the worklist
in the first place....

Well thats the issue. You need to find out which connection needs to be
processed before you can add it.



IMHO, "processed" is ambiguous, here.

WRT processing, there are only two states:

NEED_INPUT: this the 'idle' state for a connection.
this connection's fd has to be included in the fd_set for
input.

HAVE_WORK: in this state, enough input has been collected to perform
some useful work. We don't need more input, since we still have work...
( -->> this fd does not have to be included in the read-fd_set)

Note that the transition between NEED_INPUT and HAVE_WORK can be subtle:
eg if your protocol is 'line based', you cannot process before a CR/LF is
seen. You might ad a pointer or flag (or extra states) to handle this.
[ removing/adding a node to a linked list (+latching) is probably a bit
too expensive, just to check for sufficient input ]

HTH,
AvK


.

Relevant Pages

  • Re: OpenNETCF SetWirelessSettings
    ... should get the same thing as when you enter that string in the UI. ... the ActiveSync connection. ... Encoding.Unicode.GetBytesto create a byte array, ... where WEPKey is my 8 character ...
    (microsoft.public.dotnet.framework.compactframework)
  • Re: Implementing a DataReader : where to store the result from DB ?
    ... an array of array of objects seems to lead to unnecessary type ... Datatables are headless and don't have anything to do with a provider nor do ... open connection after you are done filling ... the same as using a DataReader in general. ...
    (microsoft.public.dotnet.framework.adonet)
  • array hp 1000, suse and 2 channels
    ... i was checking if it is compatibile with suse 9. ... the array has to channels, each connected to its san switch. ... and from each san switch there is a separate connection to the host. ... removing any of the cables causes "one array" failure, ...
    (comp.os.linux.hardware)
  • Wrapper for ADOdb_lite -- best way to do it?
    ... The first creates the ADO object and manages the database connection ... preparing a statement that matches the array keys to the table columns. ... I do this by making the ADO object a global. ... # SQL Query ...
    (comp.lang.php)
  • Data from another server....
    ... One is our public web server running linux ... connection and everything works great. ... My problem is that by the time the array that I started with in the ...
    (comp.lang.php)