A Scrum Master friend of mine missed Karaoke night on Saturday because she had to stay until midnight at work getting the release out of the door. Briefly, the story is like this:
It was the release week and the new version of the application should have been on UAT environment on Thursday night. Even if the team worked overtime every day, the release wasn’t ready on Friday evening. Everyone had to come on Saturday because the release of other applications depended on it. The release was finally made on Saturday at midnight, with minor configurations left for Sunday and small bugs for Monday.
Does it sound familiar? I am sure many of you have experienced such a black weekend or night. Or both.
When I heard this story, I had a question: what should a Scrum Master do in this situation? Here are some options:
Short term and immediate actions
1. Emergency retrospective
Even if Scrum requires a retrospective after each sprint, sometimes it’s not enough. This situation triggers an emergency retrospective after the release with the purpose of understanding what has slowed down the release and define actions that will avoid similar issues. A focused meeting allows productive collaboration to find solutions and create the improvement actions.
The Scrum Master role mandates that she will make sure the meeting has the best chances of success. Here are some hints for what she could do:
- Set a simple structure for the meeting: state context, gather what went wrong from the team’s members point of view, discuss what corrective actions to take
- Facilitate the meeting according to this structure
- Avoid the blame game through facilitation, by presenting the events in a neutral way.
- Help the team members blow steam before the meeting (as a suggestion, one on one coffee break discussions helps a lot). This comes from understanding of how teams work: it’s rarely someone’s fault for a mistake, usually the system set for the team is not set in a way to avoid those types of mistakes.
2. Change the team policies
Any team respects a set of policies. Scrum teams in particular have things such as: Definition of Done, Definition of Ready, the sprint length, a choice of technical practices they use and so on. These policies should change to reflect what the team has learned from the incident. Here are 4 possible team policies’ changes:
- Review the definition of done: A story shouldn’t be marked as done if its deployment information is not clear and not validated by the deployment team. The Scrum Master involves the team in improving this definition. She makes it visible and enforces it to be followed.
- Review stories against the definition of done: The stories having impact on the deployment scripts are reviewed with the deployment team before marking them as ready for testing. The Scrum Master arranges for a colleague from Operations (Ops) to participate in these reviews.
- Update release scripts every week or when a story is ready for test: For a good release, this practice must be included in the definition of done. After making the definition visible and bringing Ops to collaborate, the Scrum Master checks on the good execution of this practice and takes corrective actions when needed.
- Release to test environment more often: To avoid unexpected situations when releasing to production, deploying often on test environment is a solution. If the test environment is a copy of the production environment, then the risk of having a broken release decreases considerably. The Scrum Master has the responsibility to present this idea to the team and get the support of technical leaders to run an experiment.
Actions for the long term
For the deeper issues, we have to pay even more attention. Here are some long-term actions that can solve more problems at once:
3. More automated acceptance tests
A broken release introduces regressions. Regressions should be caught early so that the team has enough time to fix them. One solution is to implement automated acceptance tests and schedule them to run every day on a test environment where last changes were deployed.
For teams that are not eager to try this practice, the Scrum Master can build a case for running an experiment: how about one person from the team (maybe even the Scrum Master) writes a few automated acceptance tests and presents them to the team at the next retrospective? There are two advantages in this case:
- It validates that automated acceptance tests are the right solution for this problem in the team’s context
- It’s easier to accept the change
- One person learns how to implement the automated acceptance tests and can teach the others at a later time
4. Automate builds
There is an alternative to deploy often on test environment: automated builds. The developers and the Ops team can work together to implement this change. They should continue to collaborate in order to keep the scripts up to date. This long-term action can be a success if the Scrum Master has the skills to organize the collaboration between the two teams.
5. Better communication with the Operations (Ops) team
Sometimes, the developers and the operations’ teams have different types of tasks and most of the time they work separately. Organizing their collaboration can therefore present many challenges.
The Scrum Master can start with improving the communication between the two teams with:
- Raise the frequency of discussions with the Scrum Master or Team Leader from the Ops team. After a while the communication gets better and the two teams keep themselves up to date about the release-related issues
- A developer can team up with an Ops member and release to test environment whenever a story generates deployment changes.
Conclusion
Release management is not easy, but the Scrum Master can have a positive impact upon it by using her skills to obtain the following benefits:
- increase the morale of the team after a difficult moment
- help the team realize the importance of the retrospective
- improve communication with the Ops team
- motivate the team to write acceptance tests
- enforce best practices inside the team: pair-programming, pair-testing, pair-review
- spot bad execution of release practices earlier and take corrective actions right away
- lead the team to collaborate with the Ops team throughout the whole development cycle
After the next successful release, you can celebrate with your team during a karaoke night! Until then, share with us what did you do in similar situations.
