You need to transfer a lot of files across a slightly temperamental ssl connection. You want something like a recursive scp command that supports resuming and will keep on trying until it gets the job done.<\/p>\n
rsync is ideal<\/a> for this purpose – however, I find it quite dodgy under cygwin<\/a>, especially when transferring large files.<\/p>\n A sweet alternative is Unison<\/a>, for synchronizing filesets over ssh.<\/p>\n However, I often find myself falling back on a nice script called scp-resume.sh<\/a> designed for resuming the transfer of large files using dd over ssh. We can invoke this script inside a loop to transfer lots of files at a time.<\/p>\n One problem with the script is the use of the construct below to determine file sizes:<\/p>\n This will fail if there are spaces in the username of the file owner. Most likely you’ll get:<\/p>\n where the group owning the file is reported by cygwin as ‘None’. The fix is to replace every instance of this You might be tempted to use Here my 19-byte text file is taking up 1024 bytes of disk space.<\/p>\n Two other possibilities, suggested by Erik Jan Taal, are To use the With a fixed This very nearly works – the only trouble is that it will only transfer the first file in the list, and then inexplicably stops without an error! This is a difficulty that arises whenever you use the localsize=`ls -l \"${localfile}\" | awk '{ print $5 }'`<\/code><\/p>\n\r\nResuming download of [file] at byte None\r\n...\r\ndd: invalid number `None'<\/pre>\n
ls -l<\/code> construct with something like localsize=`ls -g \"${localfile}\" | awk '{ print $4 }'`<\/code>. The -g<\/code> option displays the file size but not the owner name, so you should be safe from spaces confusing awk. I don’t know if the -g option is POSIX, but it’s in GNU ls anyway.<\/p>\nls -s<\/code>, but this reports the amount of disk space used, rather than the actual length of the file (i.e. it will be a multiple of the allocation blocks). You can see the difference using ls -ls<\/code>:<\/p>\n\r\nHugh Denman@gpplap3 ~\r\n$ cat > asd.txt\r\nfre\r\nhschui\r\nhuernui\r\n\r\nHugh Denman@gpplap3 ~\r\n$ ls -ls --block-size=1 .\/asd.txt\r\n1024 -rw-r--r-- 1 Hugh Denman None 19 Mar 28 17:59 .\/asd.txt<\/pre>\n
perl -e \"print -s '$filename'\"<\/code> and ls -l | sed -n 's\/.* [^0-9]*\\([0-9]\\+\\) .*\/\\1\/ p'<\/code>. These will work on FreeBSD, for example, which does not support ls -g<\/code>.<\/p>\nscp-resume<\/code> script, we’ll need a text file containing the filenames to transfer from the remote machine. Here’s one way to generate this list.
\n$ ssh remote-user@remote.machine.ip.addr \"\/bin\/find \/cygdrive\/d -type f\" | grep -vi i386 > .\/filelist.txt<\/code>
\nIn this example, the remote drive contains the OS installation files in \/cygdrive\/d\/I386<\/code>, which we don’t want to transfer.<\/p>\nscp-resume<\/code> script, and the list of files to transfer present, all that’s left to do is iterate over each file in the list and tell scp-resume to download it. We use the cat filelist.txt | while read FILE<\/code> approach because it will preserve spaces in the filename (unlike for file in `cat filelist.txt`<\/code>).<\/p>\n\r\ncat filelist.txt | while read FILE ; do\r\nDIR=`dirname \"$FILE\"`;\r\nmkdir -p \".\/$DIR\" ;\r\n.\/scp-resume.sh -d \"
remote-user@remote.machine.ip.addr<\/code>:$FILE\" \".\/$FILE\" ;\r\ndone<\/pre>\ncat [file] | while read VAR<\/code> idea, with a shell invocation inside the while loop: whenever a shell is started, it gets STDIN, which kills the pipe (I found that out in a Usenet post<\/a>). So we have to modify scp-resume<\/code> one last time, changing the download command<\/p>\n\r\nssh -C -c arcfour \"$userhost\" \"dd bs=1 skip=$localsize \\\"if=${remotefile}\\\"\" >> $localfile < \/dev\/null<\/pre>\n