I’ll do my best to explain it here on simple terms, but be warned that this topic takes my students a couple of months to finally grasp. You can find more information on the Chapter 2 of the Data Structures and Algorithms in Java book. There is no mechanical procedure that can be used to get the BigOh. As … Read more
It depends on what you are intending to do with it. Sets are significantly faster when it comes to determining if an object is present in the set (as in x in s), but are slower than lists when it comes to iterating over their contents. You can use the timeit module to see which is faster for … Read more
From the answer here, spark.sql.shuffle.partitions configures the number of partitions that are used when shuffling data for joins or aggregations. spark.default.parallelism is the default number of partitions in RDDs returned by transformations like join, reduceByKey, and parallelize when not set explicitly by the user. Note that spark.default.parallelism seems to only be working for raw RDD and is ignored when working with dataframes. If the task you are performing … Read more
guppy3 is quite simple to use. At some point in your code, you have to write the following: This gives you some output like this: You can also find out from where objects are referenced and get statistics about that, but somehow the docs on that are a bit sparse. There is a graphical browser as … Read more
like this: Actually, even though I typically do it like this for simple convenience, over 1,000s of iterations it appears for raw speed there is an advantage for .toString() See Performance tests here (not by me, but found when I went to write my own): http://jsben.ch/#/ghQYR Fastest based on the JSPerf test above: str = num.toString(); It should be … Read more
What is the most efficient way to map a function over a numpy array? The way I’ve been doing it in my current project is as follows: However, this seems like it is probably very inefficient, since I am using a list comprehension to construct the new array as a Python list before converting it … Read more
Build a new list with a list comprehension: You can modify the original list in-place if you want, but it doesn’t actually save time: Here are (Python 3.6.3) timings demonstrating the non-timesave: And Python 2.7.6 timings:
Python includes a profiler called cProfile. It not only gives the total running time, but also times each function separately, and tells you how many times each function was called, making it easy to determine where you should make optimizations. You can call it from within your code, or from the interpreter, like this: Even more … Read more
One simplistic approach to measuring the “elapsed time” between events is to just grab the current date and time. In SQL Server Management Studio To calculate elapsed times, you could grab those date values into variables, and use the DATEDIFF function: That’s just one approach. You can also get elapsed times for queries using SQL … Read more
Gcc 7.1 -O3 these are all identical, godbolt.org/z/a-hc1d – jterm Apr 25 at 3:27 Original answer: Did some digging. Below is an example program and the relevant assembly: Code: Assembly: This was compiled with -std=c++11 -O2. You can see that there is quite a lot more work for the return “”; statement and comparably little for return std::string and return {}; (these … Read more