Running programs in parallel using xargs

From the xargs man page:

This manual page documents the GNU version of xargs. xargs reads items from the standard input, delimited by blanks (which can be protected with double or single quotes or a backslash) or newlines, and executes the command (default is /bin/echo) one or more times with any initial- arguments followed by items read from standard input. Blank lines on the standard input are ignored.

Which means that for your example xargs is waiting and collecting all of the output from your script and then running echo <that output>. Not exactly all that useful nor what you wanted.

The -n argument is how many items from the input to use with each command that gets run (nothing, by itself, about parallelism here).

To do what you want with xargs you would need to do something more like this (untested):

printf %s\\n {0..99} | xargs -n 1 -P 8 script-to-run.sh input/ output/

Which breaks down like this.

  • printf %s\\n {0..99} – Print one number per-line from 0 to 99.
  • Run xargs
    • taking at most one argument per run command line
    • and run up to eight processes at a time

Related Posts:

  1. How do I pause my shell script for a second before continuing?
  2. Cp: target is not a directory
  3. In the shell, what does ” 2>&1 ” mean?
  4. What is the preferred Bash shebang?
  5. Read a file line by line assigning the value to a variable
  6. How to compare strings in Bash
  7. How to decode Seagate’s hard drive date code in a Bash script
  8. In the shell, what does ” 2>&1 ” mean?
  9. How to grep for case insensitive string in a file?
  10. Difference between wait and sleep
  11. How do I parse command line arguments in Bash?
  12. Multi-line string with extra space (preserved indentation)
  13. How to make “if not true condition”?
  14. ‘\r’: command not found – .bashrc / .bash_profile [duplicate]
  15. Bash script and /bin/bash^M: bad interpreter: No such file or directory
  16. -bash: syntax error near unexpected token `newline’
  17. An “and” operator for an “if” statement in Bash
  18. How can I check if a directory exists in a Bash shell script?
  19. Echo newline in Bash prints literal \n
  20. Bash script and /bin/bash^M: bad interpreter: No such file or directory [duplicate]
  21. Passing parameters to a Bash function
  22. How can I use “:” as an AWK field separator?
  23. An “and” operator for an “if” statement in Bash
  24. “while :” vs. “while true” [duplicate]
  25. How to reload .bashrc settings without logging out and back in again?
  26. What is the meaning of 2 in 2> /dev/null
  27. How to reload .bash_profile from the command line?
  28. What is the purpose of “&&” in a shell command?
  29. Bash scripting missing ‘]’
  30. How do I compare two string variables in an ‘if’ statement in Bash?
  31. Return value in a Bash function
  32. Is bash a programming language?
  33. How can I count all the lines of code in a directory recursively?
  34. How can I kill a process by name instead of PID, on Linux?
  35. Pass variable from one Jenkins stage to others in sh
  36. Which characters need to be escaped when using Bash?
  37. Delete all local git branches
  38. How can I get the source directory of a Bash script from within the script itself?
  39. cp: missing destination file operand after
  40. BASH Syntax error near unexpected token ‘done’
  41. Shell: How to call one shell script from another shell script?
  42. bash assign default value
  43. What does -z mean in Bash?
  44. How can I match a string with a regex in Bash?
  45. How to use gdb with pipes and stdin
  46. echo that outputs to stderr
  47. How to define hash tables in Bash?
  48. How to escape single quotes within single quoted strings
  49. What is cp: cannot stat error in Unix, I get this error when trying to copy thing from one folder to another
  50. Simple logical operators in Bash
  51. Bash syntax error: unexpected end of file
  52. Create timestamp variable in bash script
  53. How to convert a string to lower case in Bash?
  54. How do I edit $PATH (.bash_profile) on OS X?
  55. Binary Data Posting with curl
  56. Check number of arguments passed to a Bash script
  57. How to modify a global variable within a function in bash?
  58. choosing between $0 and BASH_SOURCE
  59. How to specify the private SSH-key to use when executing shell command on Git?
  60. How do I remove the file suffix and path portion from a path string in Bash?
  61. Sorting data based on second column of a file
  62. Fast ping sweep in python
  63. Speed up rsync with Simultaneous/Concurrent File Transfers?
  64. Multi-dimensional arrays in Bash
  65. Variable interpolation in the shell
  66. What’s the meaning of the parameter -e for bash shell command line?
  67. How do I add a newline using printf?
  68. zip error – Nothing to do
  69. How to remove last n characters from a string in Bash?
  70. Meaning of “[: too many arguments” error from if [] (square brackets)
  71. Check existence of input argument in a Bash shell script
  72. How to split one string into multiple strings separated by at least one space in bash shell?
  73. Linux find and grep command together
  74. How to output a multiline string in Bash?
  75. Make a Bash alias that takes a parameter?
  76. Copy multiple files from one directory to another from Linux shell
  77. curl: no URL specified for restful api
  78. firebase-tools “-bash: firebase: command not found”
  79. How can I do a recursive find/replace of a string with awk or sed?
  80. source command not found in sh shell
  81. Running bash script from within python
  82. How to run C program on Mac OS X using Terminal?
  83. bash : cd : too many arguments
  84. Bash mkdir and subfolders 
  85. Linux Bash: Move multiple different files into same directory
  86. How to get the first line of a file in a bash script?
  87. Difference between sh and Bash
  88. What is a simple explanation for how pipes work in Bash?
  89. How to open Emacs inside Bash
  90. Error handling in Bash
  91. bash which OR operator to use – pipe v double pipe
  92. Timeout a command in bash without unnecessary delay
  93. How to determine if a bash variable is empty?
  94. What is the difference between double and single square brackets in bash?
  95. How do I get the current Unix time in milliseconds in Bash?
  96. How to create a UUID in bash?
  97. Run an interactive bash subshell with initial commands without returning to the (“super”) shell immediately
  98. bash: print stderr in red color
  99. How to make bash scripts print out every command before it executes?
  100. What does passing the -xe parameters to /bin/bash do

Leave a Comment

tech