How to join on multiple columns in Pyspark?

You should use & / | operators and be careful about operator precedence (== has lower precedence than bitwise AND and OR):

df1 = sqlContext.createDataFrame(
    [(1, "a", 2.0), (2, "b", 3.0), (3, "c", 3.0)],
    ("x1", "x2", "x3"))

df2 = sqlContext.createDataFrame(
    [(1, "f", -1.0), (2, "b", 0.0)], ("x1", "x2", "x3"))

df = df1.join(df2, (df1.x1 == df2.x1) & (df1.x2 == df2.x2))
df.show()

## +---+---+---+---+---+---+
## | x1| x2| x3| x1| x2| x3|
## +---+---+---+---+---+---+
## |  2|  b|3.0|  2|  b|0.0|
## +---+---+---+---+---+---+

Related Posts:

  1. How to join on multiple columns in Pyspark?
  2. How to change dataframe column names in pyspark?
  3. Spark RDD to DataFrame python
  4. How to change dataframe column names in pyspark?
  5. How to delete columns in pyspark dataframe
  6. Pyspark: Exception: Java gateway process exited before sending the driver its port number
  7. environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON
  8. Is there a way to create multiline comments in Python?
  9. ‘pip’ is not recognized as an internal or external command
  10. What is the purpose of the word ‘self’?
  11. Python- Robot Framework Rebot Using List
  12. how to reset index pandas dataframe after dropna() pandas dataframe
  13. How to update/upgrade a package using pip?
  14. How can I remove a specific item from an array?
  15. Behaviour of increment and decrement operators in Python
  16. Convert bytes to a string
  17. How do I update\upgrade pip itself from inside my virtual environment?
  18. Python integer incrementing with ++ [duplicate]
  19. Replacing instances of a character in a string
  20. What is the use of “assert” in Python?
  21. IndexError: too many indices for array
  22. IndexError: too many indices for array
  23. numpy array: IndexError: too many indices for array
  24. Python3 – ModuleNotFoundError: No module named ‘numpy’
  25. How do I specify new lines on Python, when writing on files?
  26. What is the purpose of the return statement?
  27. Relative imports – ModuleNotFoundError: No module named x
  28. bash: pip: command not found
  29. What is the difference between importing matplotlib and matplotlib.pyplot?
  30. Using global variables in a function
  31. How do I check what version of Python is running my script?
  32. How to read a large file – line by line?
  33. How to uninstall pip on OSX?
  34. How to delete a file or folder in Python?
  35. Converting integer to string in Python
  36. deleting file if it exists; python
  37. Reverse a string in Python
  38. Why am I seeing “TypeError: string indices must be integers”?
  39. Python for-in loop preceded by a variable
  40. Python Linked List
  41. What is the result of % in Python?
  42. ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
  43. Reading an Excel file in python using pandas
  44. What does if __name__ == “__main__”: do?
  45. How to print without a newline or space
  46. Python’s equivalent of && (logical-and) in an if-statement
  47. Python time.sleep() vs event.wait()
  48. Program to Unjumble Words on Python [closed]
  49. TypeError: ‘module’ object is not callable
  50. “inconsistent use of tabs and spaces in indentation”
  51. Error: ‘int’ object is not subscriptable – Python
  52. Referring to the null object in Python
  53. How to measure elapsed time in Python?
  54. How to uninstall Anaconda completely from macOS
  55. How to overcome TypeError: unhashable type: ‘list’
  56. Python: Start and stop timer [duplicate]
  57. Does Python have a ternary conditional operator?
  58. python .replace() regex [duplicate]
  59. not all arguments converted during string formatting.. NO % variables
  60. How do I install opencv using pip?
  61. TypeError: ‘builtin_function_or_method’ object is not subscriptable
  62. How do I use raw_input in Python 3
  63. Python Variable Declaration
  64. How do I compare two strings in python?
  65. SyntaxError: “can’t assign to function call”
  66. Curve curvature in numpy
  67. ImportError: No module named sklearn.cross_validation
  68. Get statistics for each group (such as count, mean, etc) using pandas GroupBy?
  69. How can I prevent the TypeError: list indices must be integers, not tuple when copying a python list to a numpy array?
  70. What are data classes and how are they different from common classes?
  71. How to write the Fibonacci Sequence?
  72. Tensorflow import error: No module named ‘tensorflow’
  73. Python random function
  74. Python Dictionary Comprehension
  75. Converting NumPy array into Python List structure?
  76. Converting string into datetime
  77. ImportError: No module named matplotlib.pyplot
  78. How do you get the logical xor of two variables in Python?
  79. How do I install pip on macOS or OS X?
  80. ValueError: could not convert string to float: id
  81. Converting string into datetime
  82. How do I list all files of a directory?
  83. Pip freeze vs. pip list
  84. What is the necessity of plt.figure() in matplotlib?
  85. The difference between comparison to np.nan and isnull()
  86. Best way to return multiple values from a function? [closed]
  87. How to kill a running Spark application?
  88. How does createOrReplaceTempView work in Spark?
  89. What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism?
  90. Python pandas – filter rows after groupby
  91. Pandas join issue: columns overlap but no suffix specified
  92. Meaning of @classmethod and @staticmethod for beginner?
  93. What does sys.stdin read?
  94. The value of “spark.yarn.executor.memoryOverhead” setting?
  95. How to read the last line of a file in Python?
  96. urllib2.HTTPError: HTTP Error 403: Forbidden
  97. Python/Django: log to console under runserver, log to file under Apache
  98. TensorFlow not found using pip
  99. How can I find the dimensions of a matrix in Python?
  100. How to convert rdd object to dataframe in spark

Leave a Comment