Transfer 15TB of tiny files

I have had very good results using tar, pigz (parallel gzip) and nc.

Source machine:

tar -cf - -C /path/of/small/files . | pigz | nc -l 9876

Destination machine:

To extract:

nc source_machine_ip 9876 | pigz -d | tar -xf - -C /put/stuff/here

To keep archive:

nc source_machine_ip 9876 > smallstuff.tar.gz

If you want to see the transfer rate just pipe through pv after pigz -d!

Related Posts:

  1. Shell command to tar directory excluding certain files/folders
  2. scp from Linux to Windows
  3. Copying a large directory tree locally? cp or rsync?
  4. How to copy a large number of files quickly between two servers
  5. How do I deal with a filename that starts with the hyphen (-) character?
  6. Bash: No such file or directory?
  7. What’s a .sh file?
  8. How can I symlink a file in Linux? [closed]
  9. Kill detached screen session [closed]
  10. How to get the url of the current svn repo?
  11. How to unmount a busy device
  12. How to unmount a busy device
  13. performing HTTP requests with cURL (using PROXY)
  14. No space left on device
  15. How to perform grep operation on all files in a directory?
  16. Amazon Linux: apt-get: command not found
  17. Pass a password to ssh in pure bash
  18. How can I exclude directories from grep -R?
  19. gcc: error: unrecognized command line option
  20. How to set proxy for wget?
  21. How to change permissions for a folder and its subfolders/files in one step
  22. Docker can’t connect to docker daemon
  23. Need a good hex editor for Linux
  24. Pseudo-terminal will not be allocated because stdin is not a terminal
  25. Directory Inode vs Regular File Inode
  26. How to cat <> a file containing code?
  27. Diff command along with Grep gives “Binary file (standard input) matches”
  28. can’t remove python pip
  29. How would I get a cron job to run every 30 minutes?
  30. How to automatically redirect HTTP to HTTPS on Apache servers?
  31. Trying to use bash on Windows and got no installed distributions message
  32. “Unable to find remote helper for ‘https'” during git clone
  33. What is the difference between “#!/usr/bin/env bash” and “#!/usr/bin/bash”?
  34. Is there a “goto” statement in bash?
  35. tar: Cowardly refusing to create an empty archive
  36. How can I delete a user in linux when the system says its currently used in a process
  37. Command to change the default home directory of a user
  38. Post request with Wget?
  39. Read and write to binary files in C?
  40. How to grep and replace
  41. Telnet [Unable to connect to remote host: Connection refused]
  42. Terminal Multiplexer for Microsoft Windows – Installers for GNU Screen or tmux
  43. Can’t remove a directory in Unix
  44. Is there an alternative sleep function in C to milliseconds?
  45. Gedit command not working in kali linux terminal
  46. Syntax error near unexpected token ‘then’
  47. Curl : connection refused
  48. How to install Android SDK on Ubuntu?
  49. C fopen vs open
  50. How can I use grep to show just filenames on Linux?
  51. Bash script prints “Command Not Found” on empty lines
  52. Why does “docker attach” hang?
  53. Bash export command
  54. How to edit gitignore file
  55. Retrieve last 100 lines logs
  56. What are various options / arguments for “./configure” in Linux
  57. Adding a new entry to the PATH variable in ZSH
  58. CronJob not running
  59. Delete .DS_STORE files in current folder and all subfolders from command line on Mac
  60. How to configure WP filesystem access in Linux (Ubuntu Server)?
  61. Can’t upload media to my Raspberry Pi WordPress server
  62. My WordPress is giving me an 403 error when saving post/draft with specific words
  63. Anyone else experiencing high rates of Linux server crashes during a leap second day?
  64. Can I nohup/screen an already-started process?
  65. df in linux not showing correct free space after file removal
  66. In my /etc/hosts/ file on Linux/OSX, how do I do a wildcard subdomain?
  67. Setting the hostname: FQDN or short name?
  68. Shell command to monitor changes in a file
  69. When does /tmp get cleared?
  70. Difference in sites-available vs sites-enabled vs conf.d directories (Nginx)?
  71. How do I sleep for a millisecond in bash or ksh
  72. Disk full, du tells different. How to further investigate?
  73. What does ‘set -e’ do, and why might it be considered dangerous?
  74. Filename length limits on linux?
  75. What useful things can one add to one’s .bashrc? [closed]
  76. How to add a timestamp to bash script log?
  77. best way to clear all iptables rules
  78. How to remove empty/blank lines from a file in Unix (including spaces)?
  79. How can I kill all stopped jobs?
  80. How to copy file preserving directory path in Linux?
  81. Postfix – how to retry delivery of mail in queue?
  82. How can I rename a Unix user?
  83. How to re-order windows, change the scroll shortcut, and modify the status bar contents in GNU Screen?
  84. Is it possible to alias a hostname in Linux?
  85. How do I list loaded Linux module parameter values?
  86. Does getting disconnected from an SSH session kill your programs?
  87. Tips for Securing a LAMP Server
  88. What Linux distribution is the Amazon Linux AMI based on?
  89. How to prevent a user from login in, but allow “su – user” in Linux?
  90. What is this IP address: 169.254.169.254?
  91. Should I install Linux applications in /var or /opt?
  92. I have a keypair. How do I determine the key length?
  93. Why drop caches in Linux?
  94. Heartbleed: how to reliably and portably check the OpenSSL version?
  95. swap partition vs file for performance?
  96. How can I fully log all bash scripts actions?
  97. Best way to disable swap in Linux
  98. How to remove invalid characters from filenames?
  99. How can I zip/compress a symlink?
  100. How to find the physical volume(s) that hold a logical volume in LVM

Leave a Comment