blog-post-continuous-deployment-at-instagram

https://engineering.instagram.com/continuous-deployment-at-instagram-1e18548f01d1#.8a9lezwwa

hacker-news-instagram-continuous-deployment

"it now had a record for every commit in master, and knew the test status of each specific one"

"one of the main advantages of continuous deployment, which was deploying very few commits per rollout"

"The oncall would usually step in and deploy the entire backlog at once, which defeats one of the main advantages of continuous deployments."

"We made various optimizations to get tests running in five minutes instead of 12-15 minutes, and fixed the test infrastructure problems that were causing them to be unreliable."

"provide good visibility into what it has done, is doing, and (preferably) is about to do."

"One possibility here is to split the rollout into multiple stages and implement pipelining."

"To improve this, we implemented backlog handling in the commit selection logic, which made the automation deploy multiple commits when there was a backlog. The algorithm is based on setting a time goal in which to deploy every commit (30min). For each commit in the queue, it calculates the time remaining to meet the goal, the number of rollouts that can be done in that time (using a hard-coded value), and the number of commits that would have to be deployed per rollout. It takes the maximum commits/rollout value, but caps it at three. This allows us to do as many rollouts as possible, while getting every commit out in a reasonable time."

blog-post-continuous-deployment-at-instagram#heuristics

• Tests
  • The test suite needs to be fast. It needs to have decent coverage, but doesn't necessarily have to be perfect. The tests need to be run often: during code review, before landing (and ideally blocking lands on failure), and after landing.
• Canary
  • You need an automated canary to prevent the really bad commits from being deployed to the entire fleet. It doesn't need to be perfect, however - even a very simple set of stats and thresholds can be good enough.
• Automate the normal case
  • You don't have to automate every situation; just automate the known, normal situations. If anything is abnormal, make the automation stop and let humans step in.
• Make people comfortable
  • I think that a big barrier to this kind of automation is when people feel disconnected and out of control. To address this, the system needs to provide good visibility into what it has done, is doing, and (preferably) is about to do. It also needs good stop mechanisms.
• Expect bad deploys
  • Bad changes will get out, but that's okay. You just need to detect this quickly, and be able to roll back quickly.

blog-post-continuous-deployment-at-instagram#heuristics

• rollout

  • can contain multiple commits
  • from blog-post-continuous-deployment-at-instagram

• situational awareness

  • TODO
  • from article-the-human-factors-in-aircraft-automation

Referring Pages