So, today I had to transfer a large amount of data from a NetBSD box to a new Gentoo server and I ran into some issues. First, I had to move eight gigabytes of data. Well, first thing I did was try a recursive scp but it seems that the scp utility follows symlinks. So the eight gigabytes of data was getting copied multiple times because this filestructure happened to be heavy on the symlinks. Then I thought I could tar it up and send it but it turns out there wasn’t enough disk space locally and the machine was a little slow to wait around to gzip or bz2 compress the data. Well here’s the solution:

tar cf - directory_to_copy | ssh user@destination.machine "tar xf -"

This will tar up the directory and put it to STDOUT which gets piped to ssh. Then, on the remote machine we extract STDIN. Worked like a champ. Pretty fast and much better than setting up a network filesystem just to get a transfer done. So, with two of the most commonly used unix/bsd/linux/whatever commands, you’re able to transfer files securely. Now, if you don’t need a secure connection you could do this without an encrypted connection and it would be even faster, or you -C the ssh command and compress the connection. This could also be done with gzip or bz2 with the tar commands but I have no idea which would yield the most speed. I imagine bz2 would not be the fastest. ;)