How do I parallelize a simple Python loop?

Using multiple threads on CPython won’t give you better performance for pure-Python code due to the global interpreter lock (GIL). I suggest using the multiprocessing module instead:

pool = multiprocessing.Pool(4)
out1, out2, out3 = zip(*pool.map(calc_stuff, range(0, 10 * offset, offset)))

Note that this won’t work in the interactive interpreter.

To avoid the usual FUD around the GIL: There wouldn’t be any advantage to using threads for this example anyway. You want to use processes here, not threads, because they avoid a whole bunch of problems.

Related Posts:

  1. How do I parallelize a simple Python loop?
  2. Python multiprocessing.Pool: AttributeError
  3. OpenMP and Python
  4. How can I reverse a list in Python?
  5. How to use filter, map, and reduce in Python 3
  6. What does enumerate() mean?
  7. Python Progress Bar
  8. Most efficient way to find mode in numpy array
  9. Python error “ImportError: No module named”
  10. How to groupby based on two columns in pandas?
  11. How do I append one string to another in Python?
  12. Python Progress Bar
  13. How to correct TypeError: Unicode-objects must be encoded before hashing?
  14. Python: list of lists
  15. How can I split and parse a string in Python?
  16. numpy matrix vector multiplication
  17. whitespace in regular expression
  18. How do I read CSV data into a record array in NumPy?
  19. Basic explanation of python functions
  20. TypeError: ‘Series’ objects are mutable, thus they cannot be hashed problemwith column
  21. How to rename a file using Python
  22. Cosine Similarity between 2 Number Lists
  23. How to install python3 version of package via pip on Ubuntu?
  24. django: TypeError: ‘tuple’ object is not callable
  25. ValueError: Length of values does not match length of index | Pandas DataFrame.unique()
  26. unhashable type: ‘dict’ Type Error [duplicate]
  27. Cannot install Beautiful Soup in Anaconda Python 3.6.4 Windows 10
  28. ImportError: cannot import name
  29. How can you profile a Python script?
  30. How to set the current working directory?
  31. heroku: no default language could be detected for this app
  32. ‘DataFrame’ object has no attribute ‘sort’
  33. “VBoxManage not found. Make sure VirtualBox is installed and VBoxManage is in the path”
  34. What is __pycache__?
  35. What does “\r” do in the following script?
  36. How can I remove a key from a Python dictionary?
  37. Matplotlib automatic legend outside plot
  38. Syntax behind sorted(key=lambda: …)
  39. If vs. else if vs. else statements?
  40. Quoting backslashes in Python string literals
  41. AttributeError: Can only use .dt accessor with datetimelike values
  42. Python string class like StringBuilder in C#?
  43. How do I clear all variables in the middle of a Python script?
  44. How to tell if tensorflow is using gpu acceleration from inside python shell?
  45. What is the easiest way to get current GMT time in Unix timestamp format?
  46. xlrd.biffh.XLRDError: Excel xlsx file; not supported
  47. Python/Json:Expecting property name enclosed in double quotes
  48. Python try-else
  49. How to compare two dates?
  50. Create own colormap using matplotlib and plot color scale
  51. The difference between ‘+=’ and ‘=+’?
  52. Python: can’t assign to literal
  53. “Initializing” variables in python?
  54. Difference between except: and except Exception as e:
  55. Flask raises TemplateNotFound error even though template file exists
  56. Python: Declare as integer and character
  57. How to install pytorch in windows?
  58. Python TypeError must be str not int
  59. “Unicode Error “unicodeescape” codec can’t decode bytes… Cannot open text files in Python 3
  60. Pandas join issue: columns overlap but no suffix specified
  61. How to downgrade python from 3.7 to 3.6
  62. How to turn on line numbers in IDLE?
  63. OpenCV NoneType object has no attribute shape
  64. Python socket.error: [Errno 111] Connection refused
  65. Windows- Pyinstaller Error “failed to execute script ” When App Clicked
  66. Python: What is the correct MIME type for attaching html file to email
  67. Linear regression with matplotlib / numpy
  68. Solving “500: Internal Server Error, nbconvert failed: xelatex not found in PATH”
  69. Using Look Up Tables in Python
  70. Parallel Python: What is a callback?
  71. ImportError: libGL.so.1: cannot open shared object file: No such file or directory
  72. how to exit a python script in an if statement
  73. How to add column to numpy array
  74. OpenCV NoneType object has no attribute shape
  75. AttributeError: ‘str’ object has no attribute ‘items’
  76. ImportError: No module named Crypto.Cipher
  77. Not able to pip install pickle in python 3.6
  78. How to print Unicode character in Python?
  79. Coalesce values from 2 columns into a single column in a pandas dataframe
  80. Unknown format code ‘f’ for object of type ‘str’- Folium
  81. Pyinstaller setting icons don’t change
  82. No module named ‘tqdm’
  83. Can not click on a Element: ElementClickInterceptedException in Splinter / Selenium
  84. How to split elements of a list?
  85. How can I find script’s directory?
  86. Porting Perl to Python
  87. Find object in list that has attribute equal to some value (that meets any condition)
  88. What is the purpose of the single underscore “_” variable in Python?
  89. Must have equal len keys and value when setting with an iterable
  90. print(__doc__) in Python 3 script
  91. Convert a 1D array to a 2D array in numpy
  92. Python IOError: File not open for reading
  93. Strange Exception in Tkinter callback
  94. How to resolve “ImportError: DLL load failed:” on Python?
  95. ‘module’ has no attribute ‘urlencode’
  96. Cannot concatenate ‘str’ and ‘float’ objects?
  97. Python Text Menu Infinite Loop
  98. Install pip for python 3.5
  99. How do I represent and work with n-bit vectors in Python?
  100. How to repeat individual characters in strings in Python

Leave a Comment

tech