Re: Move large number of files from AIX to Windows
- From: Ed Long <rdhm99a@xxxxxxxxxxx>
- Date: Wed, 5 Dec 2007 06:13:53 -0800
Interesting question and problem.
My only comment, observation, is that while this certainly is non-trivial, I believe it can be done if you write a program or shell script to do the work.
Rough Pseudo Code:
Do while more directories
Mkdir forced lower case version of input directory name.
If failure due to dup, uniquefy the name (Add number to end etc.).
If failure due to name quality, rename in a consistent way.
Write Detail directory trace record.
Do while more files in Directory.
Copy forced lower case filename.
If failure due to dup, uniquefy the name.
If failure due to name, rename in a consistent way.
Write Detail file trace record.
End While.
End While.
At end produce detail report and summary report of all files copied /renamed etc.
While its true that its not an exact clone, it would be an architecturally defensible copy.
Note that some of the renames may invalidate the intended use but they might not work on WIndows as is so this may be moot.
Interesting question: what about MKS Toolkit or similar Unix workalike tools on Windows? Would they allow the Unix naming convention on Windows?
Roger Deschner <rogerd@xxxxxxx> wrote: THIS CANNOT BE DONE. I tried.
Your NIC is not your problem. I routinely exceed 900 megabits/second for
sustained periods on a 1000 megabit NIC on an H80. Be sure the SSA loop
is properly configured. It must be a complete loop, check this with
diag. To make that part faster, connect the loop to two SSA adapters.
The real problem is simply that IT CANNOT BE DONE. The problem is
incompatibilities between the Unix and Windows filesystems, in terms of
path length, allowed and disallowed characters in directory and file
names, allowed directory nesting depth, and worst of all, differences in
case significance. In Unix, my.file, MY.FILE, and My.File are three
different files, which could contain totally different data. Copy those
three files to Windows, and you'll only have one file. Which contents
you'll get is totally random, but the important point is you'll only get
one of them. The other two will be lost into the bitbucket. Now consider
that subdirname, SUBDIRNAME, and SubDirName are three different
subdirectories on Unix, with lots and lots of files in branch
subdirectories under each. Copy that mess to Windows and all bets are
off as to what you get. Put your data into a blender.
You will have exactly the same problem is you use tar, when you attempt
to untar it on Windows. Exact same problem, for the exact same reason,
even if you untar it with a modern Windows program like winzip. All
Winzip can do is to extract my.file, MY.FILE, and My.File from the tar
archive into the same one single Windows file, overwriting the other
two that happened to get extracted first and second.
So, after trying extremely hard to do this in a project where I was
under a court order to produce Unix data on a Windows system, I was
forced to give up and say that not only could it not be done, but that
it should not be done. In my project, there were plenty of instances of
both directories and files whose names differed only in their case. I
eventually provided the data as Unix tar archives, with a disclaimer
that stated that some data may appear to be missing but really it was
there, and with careful instructions to the plantiff as to how to use
WinZip to extract the files ONE-AT-A-TIME as needed, and a caution not
to extract more than one file per Windows subdirectory without careful,
manual inspection of file names.
My advice to you is to give up now, before you produce inaccurate,
incomplete, and invalid results, which result in destroying potentially
valuable data. With a mass move of 1.6 million files, checking for the
collisions is impractical to say the least. The performance of your
file transfer is the least of your worries.
Roger Deschner University of Illinois at Chicago rogerd@xxxxxxx
Academic Computing & Communications Center
On Tue, 4 Dec 2007, F. Even wrote:
Tried looking into cygwin/rsync as suggested above? ftp? Seems long......
On Dec 3, 2007 3:09 PM, Alan R Vidmar wrote:
I don't believe the size of the data is the problem, I believe it's the
quantity of files and folders (and depth of folders) involved that
contribute to the lengthy process. I can easily get ~45MB/s read and
write off the F80 using NFS while testing with ATTO Disk benchmark.
Alan
"In God we trust, all others we virus scan"
Alan R. Vidmar Assistant Director of IT
Office of Financial Aid University of Colorado
Alan.Vidmar@xxxxxxxxxxxx (303)492-3598
*** This message printed with 100% recycled electrons ***
-----Original Message-----
From: IBM AIX Discussion List [mailto:aix-l@xxxxxxxxxxxxx] On
Behalf Of Kurt Gilpin
Sent: Monday, December 03, 2007 1:39 PM
To: aix-l@xxxxxxxxxxxxx
Subject: Re: Move large number of files from AIX to Windows
Without getting the calculator out and figuring how many
seconds it should take; Are you sure the network adapters are
working properly? I've copied larger filespaces than 60GB
before, and it never took anywhere close to 36 hours. I
routinely use cygwin and ssh to copy large filesystems over
the network to a Windows machine, and it works just fine.
Kurt Gilpin
Senior Information Specialist
MoDOT Information Systems
Kurtis.Gilpin@xxxxxxxxxxxx
Phone: 573.522.6022
Mobile: 573.291.9573
"Christopher D. Clausen"
Sent by: IBM AIX Discussion List
12/03/2007 01:19 PM
Please respond to
IBM AIX Discussion List
To
aix-l@xxxxxxxxxxxxx
cc
Subject
Re: Move large number of files from AIX to Windows
Alan R Vidmar wrote:
I've got an F80 with a large number of files (1.6 Millionfiles, 800K
folders, ~60GB) on a JFS2 SSA Raid 5 array that I need toget down to
a Windows server. I've been able to copy the files usingNFS and SSH,
but it just takes too long (~1.5 days) Both servers have Gbthen NFS or
interfaces and are working as expected, but NFS and SSH are just too
slow for this process.
I think the fastest way may be to move the files/folders to a
portable/removable drive that can be attached to each server. But I
just don't think there exists and interface/partition format that
both AIX and Windows2003 both support.
I also thought about TARing up the whole folder structure
SSH it down, but the TAR process seems to take half as long as the
transfer. Thus it's no time saver.
Any thoughts?
tar -c | nc on the AIX side and then nc | tar -x on the Windows side.
You'll of course need to find a version of netcat and tar for Windows.
This should be significantly faster than SSH although there is no
encryption or authentication.
You may also want to mount -noatime on the filesystem, if AIX
supports
such a thing, as writing an updated entry when the file is
copied isn't
exactly useful for you.
<
Edward Long
- References:
- Re: Move large number of files from AIX to Windows
- From: Roger Deschner
- Re: Move large number of files from AIX to Windows
- Prev by Date: Re: Move large number of files from AIX to Windows
- Next by Date: Re: Move large number of files from AIX to Windows
- Previous by thread: Re: Move large number of files from AIX to Windows
- Next by thread: Re: Move large number of files from AIX to Windows
- Index(es):
Relevant Pages
|
|
