Notes on “Release It!”
Notes on “Release It!”#
The book is about best practices in running application in production.
The Exception that Grounded an Airplane
Little error can propagete and bring to the global failure.
Stabilize Your System
Enterprise software must be cynical that expects bad things may happen and never surprised when they do.
The amazing thing is that developing stable system does not cost a lot, it is the same as unstable system’s cost.
Transaction is an abstract unit of work (Do not confuse with database transaction)
System is a complete interdependent set of hardware, application and services require to process transaction for users.
Robust system keeps processing transaction even when transient impulses, persistent stress or component failures distrust.
Impulse is a sudden spike(rapid shock) of the load. A tweet can cause an impulse.
Stress is a force applied to the system over an extended period. Getting slow responses from your component can cause a stress.
The definition of “long time” is deployment interval. If new code is deployed every day, that service does not have work for a year without restart (really?)
Extending Your Lifespan.
Most of the glitches in the software can’t be found during the development. Because of the short lifespan of the service. If you do not create a test env that simulates your production, production becomes your test env.
Sudden impulses or excessive strain can trigger a failure. The trigger and the way how it propagates with the result of the damage is called a failure mode
Create crackstoppers aka crumple zones to stop crack propagation.(no idea what can it be)
No matter what, anyway your software will have variety of failure modes. If you don’t design your failure modes, then you’ll get whatever unpredictable.
Stop crack propagation.
The more the services are tight coupled, the more chance of the crack to propagate.
Use timeouts, do not let service to be intimately close with each other
Chain of failure
Everything starts from a very little issue.
Fault - a condition that creates an incorrect state
Error - incorrect behaviour because of fault.
Failure - unresponsive system
Triggering fault opens the crack. Fault become errors, and errors provoke failures. Thats how the crack propagate.
Think about where the fault can occur. Ask questions or checklist.
The fail of the system may cost billions of dollars to the company. Big systems fail faster than smaller, as the first one has more moving parts and failure modes.
Can be in two types or between them:
These points are number-on killer of systems.
Most of the connection based on sockets.
The bad thing with TCP/IP is that it can take long time to discover that you can’t connect.
The worst place for the packet is the recv queue.
A slow response is worse that no response
A chain reaction occurs when an application has some defect, and as other instances in the homogenous layer, the defect will effect all of them.
A chain reaction in one layer can lead to cascading reaction in other layer
Recognize that on server down jeopardises the rest. - Chain reaction happens because of the death of one server makes the others pick up the slack.
Use bulkhead pattern
A cascading failure occurs when a crack in one layer triggers a crack in a calling layer.
For example a database fail, if so, other services using the database can be down.
The most affective pattern to combat with it is to use Circuit Breaker and Timeouts
Stop carcks from jumping the gap
Scrutinize resource pools
Systems would work much stable If there were no users.
Self denial attacks
Marketing team can bring down your system with discount offers
Scale your system beforehand
Point-to-point communication is bad. Load must be balanced
Shared resources can a bottleneck
Other services that your service is interacting must have the same capacity. e.g: Frond end 100K, but backend or database only 10K.
Partition your large service into small independent parts so that failure of some of them does not halt whole system.
Choose the right granularity. Partitioning can be done in the level of threads, cores, VMs(bad idea), machines, etc.
Fiddling is handling something in wrong way. For example accidentally formatting your drive.destructively
Does something wrong accidentally
Accessing server create opportunities to fiddling
Its best to keep people out of production
The system should be able to run at least one release cycle without human touch.
One can achieve “no fiddling” with immutable infrastructure
Anything that accumulates resources must release them at the end. In other words it must be drained at same rate.
It’s a process of removing old data from database
This process requires human intervention
To be continued…