Re: List of running processes in C++
- From: jbo5112 <jbo5112@xxxxxxxxx>
- Date: Fri, 18 Jul 2008 10:43:21 -0700 (PDT)
On Jul 16, 3:27 pm, Moi <r...@xxxxxxxxxxxxxxxxxxx> wrote:
On Wed, 16 Jul 2008 11:05:13 -0700, jbo5112 wrote:
The programs load data into oracle and are bound by the speed at which
oracle can write the data. The CPU usage is usually negligible.
Which basically means that oracle does the writing for you, and the
process's I/O bandwidth is controlled by oracle's I/O bandwidth. (Oracle
is probably saturating the disk I/O bus anyway.) The 5% CPU usage could be
an indication for this (but it could also be caused by swapping/paging/
trashing) .
There are times when the CPU usage is low because of paging, but I
think it's more around 0.5% (as Windows counts CPU usage, which
doesn't include any disk access). The CPU usage is low, even with
small data sets that only use ~100MB of the 1-3GB reserved for data
processing programs. I have verified that it is writing the data
through Oracle that chokes performance by turning off the insert
statements in my code. Performance goes through the roof.
[There is also the possibility of oracle being blocked in (synchronous)
disk I/O. Adding more I/O slaves / threads (I don't know oracle that well)
could help in such a case]
Currently the Oracle writes are buffered in packs of 250 wide rows.
Performance increases with a larger buffer, at the expense of more
RAM, and I will enlarge the buffers considerably when I implement some
changes and fixes to lower RAM usage.
Our major slowdown is managing indexes, and something major has to be
done fairly soon. We cannot afford a 10x or more (10x per every 4
cores) price premium for Enterprise Edition with the optional
partitioning. It's expensive enough to be worth rewriting the web
interface in PHP and migrating everything to MySQL.
Adding more instances of your process (clients from oracle's point of
view) will not help you. (unless the DBMS is very busy, in that case *you*
could get twice the slices *for your processes*, but that would be
cheating on the users. It may even degrade locking behavior) Writes to the
logs are also a bottleneck, shared by all sessions.
The unix-way would IMHO be to *prevent* multiple invocations of the
program to run at the same time. The easiest way to do that would be to
create a "wrapper" shellscript, which maintains a lockfile for the process
and (maybe) { calls it; removes the lockfile}
You could extend this method by queuing requests, and let the wrapper
inspect the queue before exiting. but since it is run from cron anyway,
this option seems illogical to me.
HTH,
AvK
I'm actually developing a queuing method within C++, since another
customer needs it for work we've been contracted to do, and somehow
extending it for duplicates (same program with same arguments) to
close. For a simple lock, I've found that the Boost interprocess (new
to 1.35.0) supports mutexes that are shared between programs. Since
I'm using the newest Boost libraries anyway, it seems like a good
temporary fix.
.
- Follow-Ups:
- Re: List of running processes in C++
- From: Moi
- Re: List of running processes in C++
- References:
- List of running processes in C++
- From: jbo5112
- Re: List of running processes in C++
- From: David Schwartz
- Re: List of running processes in C++
- From: jbo5112
- Re: List of running processes in C++
- From: David Schwartz
- Re: List of running processes in C++
- From: jbo5112
- Re: List of running processes in C++
- From: Moi
- List of running processes in C++
- Prev by Date: Re: files "." and ".." from readdir()
- Next by Date: difference between send and writev
- Previous by thread: Re: List of running processes in C++
- Next by thread: Re: List of running processes in C++
- Index(es):
Relevant Pages
|
|
