How to copy a large number of files quickly between two servers

I would recommend tar. When the file trees are already similar, rsync performs very well. However, since rsync will do multiple analysis passes on each file, and then copy the changes, it is much slower than tar for the initial copy. This command will likely do what you want. It will copy the files between the machines, as well as preserve both permissions and user/group ownerships.

tar -c /path/to/dir | ssh remote_server 'tar -xvf - -C /absolute/path/to/remotedir'

As per Mackintosh’s comment below this is the command you would use for rsync

rsync -avW -e ssh /path/to/dir/ remote_server:/path/to/remotedir

Related Posts:

  1. scp from Linux to Windows
  2. Copying a large directory tree locally? cp or rsync?
  3. What limits the maximum number of connections on a Linux server?
  4. Why is TCP accept() performance so bad under Xen?
  5. Transfer 15TB of tiny files
  6. Chmod 777 to a folder and all contents [duplicate]
  7. Cannot connect to the Docker daemon at unix:/var/run/docker.sock. Is the docker daemon running?
  8. Amazon Linux: apt-get: command not found
  9. How do I find all files containing specific text on Linux?
  10. Where can I find php.ini?
  11. gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now
  12. How can I recursively find all files in current and subfolders based on wildcard matching?
  13. Linux: ‘Username’ is not in the sudoers file. This incident will be reported
  14. “E: Unable to locate package python-pip” on Ubuntu 18.04 [duplicate]
  15. How can I exclude directories from grep -R?
  16. How to exclude a directory in find . command
  17. mysql_config not found when installing mysqldb python interface
  18. Linux error while loading shared libraries: cannot open shared object file: No such file or directory
  19. Shell command to tar directory excluding certain files/folders
  20. QEMU: /bin/sh: can’t access tty; job control turned off
  21. Explaining the ‘find -mtime’ command
  22. Meaning of exit status 1 returned by linux command
  23. Creating a new directory in C
  24. Extract file basename without path and extension in bash
  25. How to change permissions for a folder and its subfolders/files in one step
  26. mv: cannot stat error : No such file or directory error
  27. mysql_config not found when installing mysqldb python interface
  28. Linux error while loading shared libraries: cannot open shared object file: No such file or directory
  29. Pseudo-terminal will not be allocated because stdin is not a terminal
  30. screen Cannot open your terminal ‘/dev/pts/0’ – please check
  31. TCP congestion control version: HTCP module vs highspeed module in linux kernel
  32. How to use regex with find command?
  33. What is the difference between /etc/rc.local and ~/.bashrc?
  34. Snort Message – WARNING: No preprocessors configured for policy 0
  35. Compile the Fortran program in Windows using gfortran
  36. How can I set the ‘backend’ in matplotlib in Python?
  37. Yum fails with – There are no enabled repos.
  38. Unable to establish SSL connection upon wget on Ubuntu 14.04 LTS
  39. Getting stty: standard input: Inappropriate ioctl for device when using scp through an ssh tunnel
  40. configure: error: cannot run C compiled programs
  41. What does set -e mean in a bash script?
  42. How to make rpm auto install dependencies
  43. How do I install chkconfig on Ubuntu?
  44. “find: paths must precede expression:” How do I specify a recursive search that also finds files in the current directory?
  45. Apache server keeps crashing, “caught SIGTERM, shutting down”
  46. List all mounts in Linux
  47. How to use sed to extract substring
  48. Wait for user input in C?
  49. Difference between exec, execvp, execl, execv?
  50. tar: Error is not recoverable: exiting now
  51. cd into directory without having permission
  52. python-dev installation error: ImportError: No module named apt_pkg
  53. How to enable Bash in Windows 10 developer preview?
  54. WordPress sites being filled with random PHP files
  55. how to properly mount external server directory for wordpress uploads
  56. Is it a good idea to edit WordPress within Eclipse? [closed]
  57. Can scp copy directories recursively?
  58. What are the functional differences between .profile .bash_profile and .bashrc
  59. Check if port is open or closed on a Linux server?
  60. Why does my hostname appear with the address 127.0.1.1 rather than 127.0.0.1 in /etc/hosts?
  61. Linux command to inspect TXT records of a domain [closed]
  62. When does `cron.daily` run?
  63. How to run a command multiple times, using bash shell?
  64. How to reconnect to a disconnected ssh session
  65. What does a + mean at the end of the permissions from ls -l?
  66. Should I quit using Ifconfig?
  67. SSL Certificate Location on UNIX/Linux
  68. What is “-bash: !”: event not found”
  69. What version of RHEL am I using?
  70. Keeping a linux process running after I logout
  71. Force dig to resolve without using cache
  72. How do you make it obvious you are on a production system?
  73. How can I verify if TLS 1.2 is supported on a remote web server from the RHEL/CentOS shell?
  74. How to forcibly close a socket in TIME_WAIT?
  75. How to check if an RSA public / private key pair match
  76. Mount CIFS Host is down
  77. Colors in bash after piping through less?
  78. List of files installed from apt package
  79. How to copy file preserving directory path in Linux?
  80. Postfix – how to retry delivery of mail in queue?
  81. Allow SFTP but disallow SSH?
  82. Is it possible to alias a hostname in Linux?
  83. How do I list loaded Linux module parameter values?
  84. Does getting disconnected from an SSH session kill your programs?
  85. Tips for Securing a LAMP Server
  86. What Linux distribution is the Amazon Linux AMI based on?
  87. What is this IP address: 169.254.169.254?
  88. How can I export the privileges from MySQL and then import to a new server?
  89. Is there a directory equivalent of /dev/null in Linux?
  90. Can I send some text to the STDIN of an active process running in a screen session?
  91. How can I get processor/RAM/disk specs from the Linux command Line? [duplicate]
  92. Should I install Linux applications in /var or /opt?
  93. Is there a way to see the execution tree of systemd?
  94. How to determine the hostname from an IP address in a Windows network?
  95. create home directories after create users
  96. How do I redirect subdomains to a different port on the same server?
  97. Running Cron every 2 hours [duplicate]
  98. How to check the physical status of an ethernet port in Linux?
  99. What does “debconf: delaying package configuration, since apt-utils is not installed” mean?
  100. In Linux, what is the difference between “buffers” and “cache” reported by the free command?

Leave a Comment

error code: 523