distributed-computing - Read For Learn

What is the point of LRPC? Why would anyone want to make Remote Procedure Calls to the same machine?

From what I understand about RPC (Remote Procedure Calls), is that they provide a way to send function calls, invocations, etc to remote machines. The obvious advantage of this is that you can have a single program that runs on a cluster of machines and can handle more requests, more data, on so on. But … Read more

What is spark.driver.maxResultSize?

assuming that a worker wants to send 4G of data to the driver, then having spark.driver.maxResultSize=1G, will cause the worker to send 4 messages (instead of 1 with unlimited spark.driver.maxResultSize). No. If estimated size of the data is larger than maxResultSize given job will be aborted. The goal here is to protect your application from driver loss, … Read more