More on this book
Community
Kindle Notes & Highlights
Read between
March 7 - March 21, 2021
May your application’s evolution be rapid and your deployments be frequent.
In the end, our task as engineers is to build systems that do their job (i.e., meet the guarantees that users are expecting), in spite of everything going wrong.
This s a great definition of what an engineer shpuld stride to be: someone who builds things to solve problems in an environment riddled ith problems and whithout guarantees. The ultimate remise is that it has to work and exist. Being pretty or perdect violates this premise.
Thus, a supercomputer is more like a single-node computer than a distributed system: it deals with partial failure by letting it escalate into total failure
If we put distributed systems on a scale we can put hpc on one end and cloud computing on the other. Enterprise datacenters lie in the middle. Hpc hardware works similar to a sinvle node machine, even though is comprise of hundreds of cpus. Cloud computing is made by connecting commodity hardware together hrough network in a multi tenant environment.
in order to implement something like a uniqueness constraint for usernames, it’s not sufficient to have a total ordering of operations — you also need to know when that order is finalized. If you have an operation to create a username, and you are sure that no other node can insert a claim for the same username ahead of your operation in the total order, then you can safely declare the operation successful.
You can implement such a linearizable compare-and-set operation
First you write a message that will trigger a write only if the previous value is null. Then you wait until you see your message in the queue. When you do, you sk for all the messages that have the same event you just did. If the first message you receive is yours you can send the success essage back to the client.
Consensus is one of the most important and fundamental problems in distributed computing. On the surface, it seems simple: informally, the goal is simply to get several nodes to agree on something.
Concensus tries o get a group of nodes connected y an unreliable network to agree on something. It is a hard problem to solve because each node is isolated from he rest and cant even trust that ts clock is in sync with the others. It has n way of knowing if itis alone olr if it can connect to other nodez. And this happens for every message it ries to send.
Unix. The philosophy was described in 1978 as follows [12, 13]: Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”. Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input. Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them. Use tools in
...more
The unix philosofy has a lot of resemblance with the agile and devops manifests of today.
In general, a “stream” refers to data that is incrementally made available over time.
The snapshot of the database must correspond to a known position or offset in the change log, so that you know at which point to start applying changes after the snapshot has been processed.
A lient of message queue must keep an offset f their position in the qeue. This value can be saved along a snapshot to know from which point in the queue messages should start replaying in case of a failure or some other need.
Kafka Connect [41] is an effort to integrate change data capture tools for a wide range of database systems with Kafka. Once the stream of change events is in Kafka, it can be used to update derived data systems such as search indexes, and also feed into stream processing systems as discussed later in this chapter.
Kafka connect can be used to get the update stream from a database. Thus, every change to it can be stored on a message stream.
Event sourcing is a powerful technique for data modeling: from an application point of view it is more meaningful to record the user’s actions as immutable events, rather than recording the effect of those actions on a mutable database.
An interesting thought. Instead of recording changes on the stream, record intents. Then have some other service interpret those intents and produce, and potentially store, its effects.
This transformation can use arbitrary logic, but it should be deterministic so that you can run it again and derive the same application state from the event log.
A projection of the event log must be deterministic. Meaning that, given the same log, at the exact position, should produce the same projection every time.
Log compaction is not possible in the same way.
Log compaction is not possible on an event source queue because you need the full history of events on an object to know its state. When recording effects on an event queue you can compress a series of modifications to its last state and no information would be lost. In the other caseyou dont really have an actual state to derive from the intents stored on the log. Different services might interpret those events differently. So you have to keep the whole sequence to recreate each service state.
The event sourcing philosophy is careful to distinguish between events and commands [48]. When a request from a user first arrives, it is initially a command: at this point it may still fail, for example because some integrity condition is violated. The application must first validate that it can execute the command. If the validation is successful and the command is accepted, it becomes an event, which is durable and immutable.
A message inside an event source system its characterized at first asa command. It defines a user intent but at the moment can still fail. After its validagted it becomes an inmutable event.
Alternatively, the user request to reserve a seat could be split into two events: first a tentative reservation, and then a separate confirmation event once the reservation has been validated
When using event sourcing intents should not be stored as facts until they re confirmed. If a check must be made before something is considered a fact it should be constructed as more than one event. One to ask for something to e validated and another that indicates if tht request was granted or not.
We normally think of databases as storing the current state of the application — this representation is optimized for reads, and it is usually the most convenient for serving queries. The nature of state is that it changes, so databases support updating and deleting data as well as inserting it. How does this fit with immutability?
If you are mathematically inclined, you might say that the application state is what you get when you integrate an event stream over time, and a change stream is what you get when you differentiate the state by time,
you gain a lot of flexibility by separating the form in which data is written from the form it is read, and by allowing several different read views. This idea is sometimes known as command query responsibility segregation (CQRS)
I have a name for my idea of creating an architecture that can take advantage of an event source log in aws.
The traditional approach to database and schema design is based on the fallacy that data must be written in the same form as it will be queried.
CEP systems often use a high-level declarative query language like SQL,
CEP is a system that allows the creation of compñex statements to identify vents on a stream. They generate a complex event when a match i found.
To adjust for incorrect device clocks, one approach is to log three timestamps
The client time when it was created and sent plus the time on the server when it was received. The difference between tjehe last two gives the ffset between the client and the server and can be sed to alcjlate the actual timestamp of the event.
For example, when consuming messages from Kafka, every message has a persistent, monotonically increasing offset. When writing a value to an external database, you can include the offset of the message that triggered the last write with the value. Thus, you can tell whether an update has already been applied, and avoid performing the same update again.
With a little metadata we can make an action idempotent.
The lambda architecture proposes running two different systems in parallel: a batch processing system such as Hadoop MapReduce, and a separate stream-processing system such as Storm.
Never heard of the lambda architecture. Need to found out more about it.
time dependence
Tkme dependance joins appear when you have some data that epends on avalue that might have had a different value in ghe past. So your computation will eed to know this value in the future since its current representation might not represent it.
building applications around dataflow ideas
I need to larn more about what this means.
The ideas we discussed around stream processing and messaging are not restricted to running only in a datacenter: we can take the ideas further, and extend them all the way to end-user devices
The idea of extending the write path to the client is great. Meaning that whenever you do a rad on the client you are cgually reading from your local database which was pfeviously updated by the application preemtively.
a consumer of a log-based message broker can reconnect after failing or becoming disconnected, and ensure that it doesn’t miss any messages that arrived while it was disconnected.
The client stores an offset of the event log queue so when it comes back online it can ask for ll the mesaages it didnt atch.
just because an application uses a data system that provides comparatively strong safety properties, such as serializable transactions, that does not mean the application is guaranteed to be free from data loss or corruption. The application itself needs to take end-to-end measures, such as duplicate suppression, as well.
Just because w use dagabase we strong features it doesnt mean that we are excempt for adding sgrong features to ur aplication also.
The idea of using multiple differently partitioned stages
We an achieve the same level of guarantees of a transaction inside an event ueue. For an accounting balance ovement we can first create an evdent dtating that we need t move data from one account to another. His transaction is given a uique d. Another process takes this message and creates a debit event and a credit event with te assigned id. Other swrvices can be listening to this messages and perform the necessary movements. If there was a constraikn on he balance on an account stating that it cant be lower than zero, a similar approach of the one use for keepimg the uniqueness of usernames can be used. A message should be reated askimg for permission to perform the debit. Then we wait for that message to be on the queue. Lastly we check flr a confirmation message. We can track this message through its ransactionid. If everything checks out we confirm the movement.
This works because ll the transactions of a given account are routed to the same partition where they are process in order. This avoids race onditions.
Violations of timeliness can be annoying and confusing, but violations of integrity can be catastrophic.
Meaning that it is ok if it takes a while for the state to converge. What is not ok s ft hthe database to be able to reach an inconsistent or invalid state.
In countries that respect human rights, the criminal justice system presumes innocence until proven guilty; on the other hand, automated systems can systematically and arbitrarily exclude a person from participating in society without any proof of guilt, and with little chance of appeal.
This is a very good counter argvument about using ai to tag a person with a negative atteibute based ln data.
Predictive analytics systems merely extrapolate from the past; if the past is discriminatory, they codify that discrimination. If we want the future to be better than the past, moral imagination is required, and that’s something only humans can provide [87]. Data and models should be our tools, not our masters.
This is notherf gredat point. If the source we are uing has been taken in a time filled with negative attributes towards certain portions of a population, the results extracted from that data will refñect those same bad conclusions.
surveillance infrastructure the world has ever seen.
A grst thought experiment when tLking about it rivacy: changing the word data for surveillance.
Having privacy does not mean keeping everything secret; it means having the freedom to choose which things to reveal to whom, what to make public, and what to keep secret. The right to privacy is a decision right: it enables each person to decide where they want to be on the spectrum between secrecy and transparency in each situation
Privacy is the right to choose which information about our person we share, and with whom.
