How does createOrReplaceTempView work in Spark?

createOrReplaceTempView creates (or replaces if that view name already exists) a lazily evaluated “view” that you can then use like a hive table in Spark SQL. It does not persist to memory unless you cache the dataset that underpins the view.

scala> val s = Seq(1,2,3).toDF("num")
s: org.apache.spark.sql.DataFrame = [num: int]

scala> s.createOrReplaceTempView("nums")

scala> spark.table("nums")
res22: org.apache.spark.sql.DataFrame = [num: int]

scala> spark.table("nums").cache
res23: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [num: int]

scala> spark.table("nums").count
res24: Long = 3

The data is cached fully only after the .count call. Here’s proof it’s been cached:

Related SO: spark createOrReplaceTempView vs createGlobalTempView

Relevant quote (comparing to persistent table): “Unlike the createOrReplaceTempView command, saveAsTable will materialize the contents of the DataFrame and create a pointer to the data in the Hive metastore.” from https://spark.apache.org/docs/latest/sql-programming-guide.html#saving-to-persistent-tables

Note : createOrReplaceTempView was formerly registerTempTable

Related Posts:

  1. How to kill a running Spark application?
  2. What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism?
  3. Spark RDD to DataFrame python
  4. How to change dataframe column names in pyspark?
  5. How to delete columns in pyspark dataframe
  6. How to join on multiple columns in Pyspark?
  7. How to join on multiple columns in Pyspark?
  8. The value of “spark.yarn.executor.memoryOverhead” setting?
  9. Converting Pandas dataframe into Spark dataframe error
  10. How to convert rdd object to dataframe in spark
  11. are there dictionaries in javascript like python?
  12. TypeError: only integer scalar arrays can be converted to a scalar index with 1D numpy indices array
  13. How to find Google’s IP address?
  14. What is a CSRF token? What is its importance and how does it work?
  15. How to get Twitch watchtime (like streamelements)
  16. What is the difference between functional and non-functional requirements?
  17. Mealy v/s. Moore
  18. Error 6 (net::ERR_FILE_NOT_FOUND): The files c or directory could not be found
  19. C compiler for Windows?
  20. what is gitpod: what does it actually do?
  21. What is JNDI? What is its basic use? When is it used?
  22. ngrok: tcp forwarding to 25565 (minecraft server)
  23. Inkscape– convert text to object, dynamic offset. Shift+Ctrl+C failed?
  24. Floating div positioning
  25. Eclipse CDT project built but “Launch Failed. Binary Not Found”
  26. Context Deadline Exceeded – prometheus
  27. When do I use the PHP constant “PHP_EOL”?
  28. Using $this when not in object context in PHP
  29. Difference between SIP and SIPS URI?
  30. ‘str’ object has no attribute ‘decode’. Python 3 error?
  31. What’s the difference between & and && in MATLAB?
  32. azure error 502 – Web server received an invalid response while acting as a gateway or proxy server
  33. What is a Ray ID (Cloudflare)?
  34. How do I get a list of built-in data sets in R?
  35. java.text.ParseException: Unparseable date
  36. Difference between .o and .ko file
  37. R error: all arguments must have the same length
  38. spring @sqlgroup with multiple datasource
  39. Alternative to ui-grid(doesn’t support angular2/4)
  40. Tool for drawing parse trees?
  41. How do I install the OpenSSL libraries on Ubuntu?
  42. How can I specify a display?
  43. Clearing localStorage in javascript?
  44. Search all the occurrences of a string in the entire project in Android Studio
  45. “docker build” requires exactly 1 argument(s)
  46. declaring a priority_queue in c++ with a custom comparator
  47. R dates “origin” must be supplied
  48. How to one hot encode several categorical variables in R
  49. cannot load such file — bundler/setup (LoadError)
  50. Mongoose.js: remove collection or DB
  51. slideToggle JQuery right to left
  52. Valgrind Invalid free() / delete / delete[] / realloc() in C
  53. Exception in thread “main” java.lang.reflect.InvocationTargetException
  54. map vs mapValues in Spark
  55. “webxml attribute is required” error in Maven
  56. How can I change the color of my prompt in zsh (different from normal text)?
  57. mkdir c++ function
  58. C’s printf and fprintf(stdout,) are not printing
  59. What’s the difference between SoftReference and WeakReference in Java?
  60. Identity matrix in Julia
  61. Letsencrypt renewal fails: Could not bind to IPv4 or IPv6.. Skipping
  62. reversing list in Lisp
  63. Good way to encapsulate Integer.parseInt()
  64. TCP/IP packets and datagrams
  65. ArrayList of int array in java
  66. What are the differences between Mustache.js and Handlebars.js?
  67. Vim Opening File E325 Attention Error
  68. Terminating app due to uncaught exception ‘NSUnknownKeyException’
  69. Rewrite the base-url for single posts
  70. How to parse wordpress options json
  71. A post has a js redirection script. How to not redirect its category?
  72. Cannot understand WordPress error message
  73. Add column(s) to post edit page
  74. How can i hide and display elements automatically?
  75. How use custom image size + ACF + background image
  76. Pass AJAX variable to PHP functions WordPress plugin
  77. Any benefit or detriment from removing a pagefile on an 8 GB RAM machine?
  78. How do I view the details of a digital certificate .cer file?
  79. How do you validate fstab without rebooting?
  80. DNS – NSLOOKUP what is the meaning of the non-authoritative answer?
  81. Why can’t a CNAME record be used at the apex (aka root) of a domain?
  82. How can I upgrade to Java 1.8 on an Amazon Linux Server?
  83. What port should I open to allow remote desktop?
  84. Can you help me with my capacity planning?
  85. Cheat Sheets for System Administrators?
  86. Amazon EC2 terminology – AMI vs. EBS vs. Snapshot vs. Volume
  87. How to change an EC2 instance’s security group
  88. What type of DNS record is needed to make a subdomain?
  89. Multiple SSL domains on the same IP address and same port?
  90. IPTABLES – Limit rate of a specific incoming IP
  91. How to set default Ansible username/password for SSH connection?
  92. Can I make `find` return non-0 when no matching files are found?
  93. Proxy Error 502 “Reason: Error reading from remote server” with Apache 2.2.3 (Debian) mod_proxy and Jetty 6.1.18
  94. What is the difference between service and systemctl?
  95. Rsync creates a directory with the same name inside of destination directory
  96. How Often Do Windows Servers Need to be Restarted?
  97. sudoers: how to disable requiretty per user
  98. dig show only answer
  99. SSD or HDD for server
  100. Do you have any useful awk and grep scripts for parsing apache logs? [closed]

Leave a Comment