Re: Need Help


John L wrote:

<pankaj_wolfhunter@xxxxxxxxxxx> wrote in message news:1150353842.009608.26310@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Greeting,
I have a set of files, around 15-16, doing a job of
loading huge flat files data into oracle tables.
eg: file1, file2, file3 etc

I have a main file, test_main, that is triggering each of these 16
files one by one and some in parallel.
eg
file1
file2
file3
....
...

Some of these files are using split command to split the flat files
into, max 5-6, chunk files and then loading these chunk files into
oracle table in background.
like file1, file3

I have been told that whenever the files are getting split into chunk
files and getting loaded in background, no other file should be
executed.
This means that if script is executing file1, which is creating chunk
files, then it should wait for this file to get over before proceeding
to file2 execution.

My question is what modification is required in test_main script in
order to execute the files as required.


So at the moment, your script looks something like (pseudocode):

split file1 && load file1.?? &
split file2 && load file2.?? &
split file3 && load file3.?? &

Is that right? Or something like that?

If so, the simplest answer is not to remove the &s so as not to
run each load in the background.

split file1 && load file1.??
split file2 && load file2.??
split file3 && load file3.??

You can run the whole test_main script in the background if you
wish but there does not seem to be much point. By the way,
test_main is not a very good name for the script.

Having got this far, you could then rewrite it as a loop
over file1, file2, and so on, and perhaps as a loop calling
a shell function.

Note that things will be cleaner if you split the files into
a different directory. If the loader removes files once it
has loaded them, then you could consider splitting the next
file as soon as the number of subfiles currently being loaded
falls below a certain level.

--
John.

Thanks John for such a quick response. Actually we thought of removing
the chunk files after their purpose
is done but the ppl here wants to retain the file even after loading
the same for some purpose.
Also we have been told not to touch the code. That means we cant make
any changes to the already existing code.
Its been told to do some check from outside only from where the scripts
are getting triggered, especially in test_main,
in order to make sure that the requirement is met
..
For now I have been doing following, just want to know how is it

content of test_main

file1
## does not create any chunk file

file2
## creates chunk files

pcount=`jobs | wc -l`
while (( $pcount > 0 )) ## WHILE LOOP TO ENSURE ALL BACKGROUND
PROCESSES ARE COMPLETE
do
sleep 5
pcount=`jobs | wc -l`
done

file3
## creates chunk files

pcount=`jobs | wc -l`
while (( $pcount > 0 )) ## WHILE LOOP TO ENSURE ALL BACKGROUND
PROCESSES ARE COMPLETE
do
sleep 5
pcount=`jobs | wc -l`
done

file4 ## does not create chunk files

any inputs for the same will be appreciated

TIA

.

Relevant Pages

  • RE: [SLE] File Counts

    (SuSE)
  • Re: fopen() with full path affecting subsequent fopen calls
    ... Normally I would expect file2 to be written in the same place as it ... did when I didn't use the full path for file1. ... as well as file3 (which ... It's very likely a "current working directory" problem of some sort, ...
    (comp.lang.c)
  • Re: Problems with tar (tcllib)
    ... I have a simple script that uses the tcllibtarcommand. ...
    (comp.lang.tcl)
  • Re: Need Help
    ... eg: file1, file2, file3 etc ... This means that if script is executing file1, ...
    (comp.unix.shell)
  • Re: simple ln question
    ... that makes it so the contents of file1 are the same as of file2, and if you change the contents of 1, the other ... prints the inode number for the file foobar. ...
    (Fedora)