Transfer 15TB of tiny files

I have had very good results using tar, pigz (parallel gzip) and nc. Source machine: tar -cf – -C /path/of/small/files . | pigz | nc -l 9876 Destination machine: To extract: nc source_machine_ip 9876 | pigz -d | tar -xf – -C /put/stuff/here To keep archive: nc source_machine_ip 9876 > smallstuff.tar.gz If you want to … Read more