A really good article on how Stripe ran a game day testing exercise on one of their systems, and what they found.
The two lessons I took from it:
- It’s important to run these destructive tests, so you know exactly how your system will respond.
- Really think about the technology you depend on. Don’t choose a database because you know it, or (worse) because it’s new and shiny. Think about your applications specific requirements and choose a technology that meets them.