Build Stuff 2018

Everything was prepared. The features were in place, the critical bugs were fixed, the environments were up and running and the test were green. We were finally about to go into production with the system we had been working so hard on for the past 18 months.

We had deployed in Azure with redundancy, fail-overs and slots using continuous delivery. The customers had been doing acceptance testing for weeks. We were ready!

Then it happened.

Three days before going into production, the pipeline started halting. The applications crashed one by one. First the development environment, following the test environment. Then the staging environment and last; the production environment. No exceptions or logs pointing us in the direction of the problem.

We had three days to pin down and fix it.

This is a war-story about how we turned our Azure environments upside down, logged, debugged and at last, found the nasty bug and fixed it.