Learning Spark Quotes

“Note that the hash function you pass will be compared by identity to that of other RDDs. If you want to partition multiple RDDs with the same partitioner, pass the same function object (e.g., a global function) instead of creating a new lambda for each one!”
― Holden Karau, Learning Spark: Lightning-Fast Big Data Analysis

“For each input source, Spark Streaming launches receivers, which are tasks running within the application’s executors that collect data from the input source and save it as RDDs. These receive the input data and replicate it (by default) to another executor for fault tolerance. This data is stored in the memory of the executors in the same way as cached RDDs.1”
― Holden Karau, Learning Spark: Lightning-Fast Big Data Analysis