Status Updates From Designing Data-Intensive Ap...

The feature of a batch processing job is that it reads some input data and produces some output data, without modifying the input, in other words, the output is derived from the input. The input data is bounded: it has a fixed size. Because it is bounded, a job knows when it has finished reading the entire input, and so a job eventually completes when it is done.

The CAP Theorem states that distributed data stores can simultaneously provide at most two of three guarantees: Consistency (all nodes see the same data), Availability (every request receives a response), and Partition Tolerance (the system continues to operate despite network failures)

This is a disadvantage because a job's output is only considered valid when the job has completed successfully. Otherwise the temp file contents are discarded -> the next job cannot continue. To manage these dependencies some workflow schedulers have been developed for Hadoop.

MapReduce jobs are usually chained together into a workflow, such that the output of one job becomes the input to the next job, but from the frameworks pov, they are 2 separate independent jobs.

Therefore, these chained MapReduce jobs are not really like robust pipelines, and more like a sequence of commands where each command's output is written to a temp file, and the next command reads from the temp file.

The MapReduce scheduler tries to run each map function (which can be parallelized) on one of the machines that stores a replica of the input file if the machine has enough RAM and CPU resources.

This is called "putting the computation near the data" and it saves copying the input file over the network, reducing network load and increasing locality.

MapReduce has 4 steps.

Step 1 is to READ the set of input files and break them up into records.
Step 2 is where the MAP function is called to extract key value pairs from each record.
Step 3 is SORTING all of the key value pairs by key for further processing
Step 4 is REDUCE where the reducer function iterates over the sorted key-value pairs and produces output records like counts and more.

HDFS basically has a process running on each machine which exposes a network service that allows other nodes from the distributed network to access files stored on that machine. So HDFS creates one big file system that can use the space on the disks of all machines running the process.

Azure Blob Storage and AWS S3 are similar to HDFS.