Craig Pardey Coder | Runner | Cyclist | Parent
Me: Why is this report different to the one in production? SysAdmin: Dunno. I guess they must have fixed it production and forgot to check it back into source control.
If I had a dollar for every time I’ve had that conversation I’d be a rich man. It is usually the result of some late-night troubleshooting and finally the system is working as expected and everyone goes to bed. But the next day, nobody remembers what exactly they changed that fixed the problem, nor do they spend the time to figure it out and get it back into source control. Worse yet, they were making untested changes in a live environment - that shouldn’t be possible!.
Gone are the days when source control was just for source code. Everything required to build, package, deploy, migrate and run your application should be kept in source control. That includes source code, app configs, build scripts, deployment jobs, SQL scripts (DDL and data), server settings, smoke tests … everything.
The fact that this team was troubleshooting a release in production tells me that their deployment process is probably abysmal, so it’s easier to just change things in-situ than to go through the whole commit-push-deploy cycle again. It’s tempting to blame the people involved but more often than not this is an organisational problem. If it was easy to push new builds through the test environments and to production then then tech team would do that.
Deep down, people want to Do The Right Thing but in a low-trust organisation that means days or weeks of paperwork, sign-offs, release approval meetings and other forms of Release Management Theatre. A heavy, approval-based release process like that actively discourages the tech team from locking down production because if the only way to get code to production is through source control then they’re screwed if something goes wrong. The system will be offline for days or weeks while they wait for the sign-offs.
And then bugs that were fixed through the heroic efforts on the live servers get re-introduced on the next release because they never made it into source control. Or worse yet, the team does patches instead of full releases so that a few cycles later nobody has any idea if they can re-create the production instance from source control. Spinning up new instances involves copying the whole filesystem.
If this sounds like your environment then take a long, hard look at your release governance process. Does it encourage and facilitate rapid deployments? If not, you should start automating the release process for all environments so that production releases become boring and predictable. Start by measuring how long it currently takes from a “go live” decision to the code running in production (including QA testing and sign-offs). Now automate the longest step. Repeat.
Do this with diligence and you’ll have a reliable, reproducible system that can push fixes on short notice without relying on Brent(s).
There’s nothing that beats the feeling of being part of an exceptional team and yet it is strangely difficult to define exactly what makes one team great and another team not-so-great. Google recently set out to learn how to build the perfect team and came up with some interesting, albeit previously discovered, answers.
To summarize their findings, Google’s Project Aristotle concluded that on good teams:
- All team members spoke roughly the same amount, and
- Team members were empathetic towards each other
Harvard Business School’s professor Amy Edmondson described this as “a sense of confidence that the team will not embarrass, reject or punish someone for speaking up”. This may seem obvious until you find yourself on a team that crucifies anyone for speaking their mind. I have experienced those teams first hand, and believe me, they’re no fun. Conversations become antagonistic and argumentative and the team dynamic rapidly degrades.
… on the good teams, members spoke in roughly the same proportion, a phenomenon the researchers referred to as “equality in distribution of conversational turn-taking.” On some teams, everyone spoke during each task; on others, leadership shifted among teammates from assignment to assignment. But in each case, by the end of the day, everyone had spoken roughly the same amount.
… the good teams all had high “average social sensitivity” - a fancy way of saying they were skilled at intuiting how others felt based on their tone of voice, their expressions and other nonverbal cues.
Ok, great, but how can you use this to fix a broken team? The Google article describes how one manager, Matt Sakaguchi, approached the issue. He held an offsite meeting and opened up his personal issues to the team - by telling them that he has Stage 4 cancer. By making himself vulnerable and dismantling the work/life division he was able to reach the members of his team and get them to see each other as people, rather than just workers.
Trust, respect and empathy are the foundational attributes of a strong team, where people are comfortable being themselves.
The Westrum items are:
- On my team, information is actively sought.
- On my team, failures are learning opportunities, and messengers of them are not punished.
- On my team, responsibilities are shared.
- On my team, cross-functional collaboration is encouraged and rewarded.
- On my team, failure causes enquiry.
- On my team, new ideas are welcomed.
I’ve been pushing large enterprises to adopt agile development practices for years so I was very excited when I began seeing many of them embarking upon ambitious Agile Transformation projects where they attempt to not only change the way they build software, but completely revolutionise the entire IT process from inception, finance, HR, delivery and operations.
Agile has gained a lot of traction in large enterprises in recent years but I’m concerned about the specifics of how teams are implementing it. I’ve seen everything from traditional eXtreme Programming (XP) practices to Scrum, but nothing scares me more than the teams that self-identify as “agile” but do nothing more than hold a daily team meeting.