You need to make sure your pandas dataframe columns are appropriate for the type spark is inferring. If your pandas dataframe lists something like: And you’re getting that error try: Now, make sure .astype(str) is actually the type you want those columns to be. Basically, when the underlying Java code tries to infer the type from an … Read more
See, There are two ways to convert an RDD to DF in Spark. toDF() and createDataFrame(rdd, schema) I will show you how you can do that dynamically. toDF() The toDF() command gives you the way to convert an RDD[Row] to a Dataframe. The point is, the object Row() can receive a **kwargs argument. So, there is an easy way to do that. This way you … Read more
createOrReplaceTempView creates (or replaces if that view name already exists) a lazily evaluated “view” that you can then use like a hive table in Spark SQL. It does not persist to memory unless you cache the dataset that underpins the view. The data is cached fully only after the .count call. Here’s proof it’s been cached: Related SO: spark createOrReplaceTempView vs … Read more