In my recent post about CI/CD performance, I mentioned that I’d built a shadow pipeline in Ansible to deploy to our on-prem dev environment. Since then, I have been campaigning to use those Ansible playbooks for our pre-Prod and Prod deployments too.
I secured approvals to do a proof-of-concept deployment to pre-Prod using the Ansible playbooks but it failed dismally because our pre-Prod VMs have drastically different sudo restrictions than our dev env. I do not have access to the pre-Prod VMs so I was not aware of these differences.
In dev, sudo is unrestricted. In pre-Prod and Prod, sudo is restricted to a specific set of commands which breaks Ansible because Ansible requires general privilege escalation. The folks on Reddit agreed that Ansible won’t work with restricted sudo.
Different users are also restricted in what they can do. For example, in dev, one user can both run systemctl and copy files. In pre-Prod and Prod, those actions must be executed by two separate userids (same warm body, two logins - no segregation of duties here!).
I went back to drawing board and re-implemented our Ansible playbooks using Python and Fabric which gives me fine-grained control over exactly which commands are executed via sudo. The interaction model is a bit clunkier than Ansible but it does the job perfectly, and the re-implementation only took a few hours and about 250 lines of AI-assisted code.
Last week, we completed a successful service restart in pre-Prod with the Python+Fabric version of the shadow pipeline. It seems like such a small step but it’s a giant leap for my project. Defeating bureacratic processes in a large organization takes time, experience, and a lot of patience.
On Monday, we’ll be testing a pre-Prod deployment. If that works, I can continue my campaign for approval to use these scripts for all of our deployments and liberate my team from the tyranny of the slow, flaky, enterprise deployment pipeline. Onwards!