Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems

by Martin Kleppmann

TCP performs flow control

backpressure), in which a node limits its own rate of sending in order to avoid overloading

additional queueing at the sender before the

TCP considers a packet to be lost if it is not acknowledged within some timeout

and lost packets are automatically retransmitted.

you can only choose timeouts experimentally:

rather than using configured constant timeouts, systems can continually measure response times and their variability (jitter),

TCP retransmission timeouts also work similarly

network is synchronous:

queueing, because the 16 bits of space for the call have already been reserved in the next hop of the network.

bounded delay.

You can give TCP a variable-sized block of data

Why do datacenter networks and the internet use packet switching?

that they are optimized for bursty traffic.

On the other hand, requesting a web page, sending an email, or transferring a file doesn’t have any particular bandwidth requirement

If you guess too low, the transfer is unnecessarily slow, leaving network capacity unused. If you guess too high, the circuit cannot be set up

build hybrid networks

ATM.iii

Variable delays in networks are not a law of nature, but simply the result of a cost/benefit trade-off.

However, such quality of service is currently not enabled in multi-tenant datacenters and public clouds, or when communicating via the internet.iv

variable delays in the network,

This fact sometimes makes it difficult to determine the order in which things happened when multiple machines are involved.

which is an actual hardware device: usually a quartz crystal oscillator.

so each machine has its own notion of time, which may be slightly faster or slower than on other machines.

possible to synchronize clocks to some degree: the most commonly used mechanism is the Ne...

Modern computers have at least two different kinds of clocks: a time-of-day clock and a monotonic clock.

System.currentTimeMillis() in Java return

calendar, not counting leap seconds.

Time-of-day clocks are usually synchronized with NTP, which means that a timestamp from one machine (ideally) means the same as a timestamp on another machine.

These jumps, as well as similar jumps caused by leap seconds, make time-of-day clocks unsuitable for measuring elapsed time

A monotonic clock is suitable for measuring a duration (time interval),

System.nanoTime() in Java are monotonic clocks,

The name comes from the fact that they are guaranteed to always move forward

On a server with multiple CPU sockets, there may be a separate timer per CPU,

Operating systems compensate for any discrepancy and try to present a monotonic view of the clock to application threads,

NTP may adjust the frequency at which the monotonic clock moves forward (this is known as slewing the clock)

should). Clock drift varies depending on the temperature of the machine.

Anecdotal

NTP synchronization can only be as good as the network delay, so there is a limit to its accuracy when you’re on a congested network with variable packet delays.

NTP clients are quite robust, because they query several servers and ignore outliers.

The fact that leap seconds have crashed many large systems [38, 46] shows how easy it is for incorrect assumptions about clocks to sneak into a system.

In virtual machines,

When a CPU core is shared between virtual machines, each VM is paused for tens of milliseconds while another VM is running.

a day may not have exactly 86,400 seconds,

time-of-day clocks may

move backward ...

Thus, if you use software that requires synchronized clocks, it is essential that you also carefully monitor the clock offsets between all the machines.

Database writes can mysteriously disappear: a node with a lagging clock is unable to overwrite values previously written by a node with a fast clock until the clock skew between the nodes has elapsed

Even with tightly NTP-synchronized clocks, you could send a packet at timestamp 100 ms (according to the sender’s clock) and have it arrive at timestamp 99 ms (according to the recipient’s clock)

Probably not, because NTP’s synchronization accuracy is itself limited by the network round-trip time,