I was looking for someone on the operations side of things to join us. You know, Ops, running and maintaining Production. I was asked if I was looking for a SysAdmin, Site Reliability Engineer (SRE) or a DevOps Engineer. I did not give it much thought and went through a resume screening exercise.
And then it struck me.
There were so many “DevOps Engineers” resumes out there:
- who knew Chef/Puppet/Ansible
- who could code against the AWS Api
- who knew Docker :)
Surprisingly, “hands-on and troubleshooting experience, on a high traffic website, powered by any-flavor-of-unix” was absent. And what about Architecture, High-Availability, Monitoring, Load-Balancing?
Why does this make me cringe?
I do DevOps is like saying I do Teamwork. The DevOps team is even worse, because you know…
DevOps is a game changing grassroots philosophy based on:
- Empathy
- Don’t toss code over the wall
- Break the Silos
- Culture of sharing
- Quick and constant feedback
This could be achieved with metrics and tooling. Or whatever you do just try to get better at the above :) If you want a proper history, go read Web Operations - Keeping the Data on Time
This all started around 2008-09 and it seeded a new breed of tools, which was great.
Sometime after that “DevOps” got picked up by marketing and the original message started to get lost.
A Career in Technical Operations
Don’t just be an “operator” of a DevOps tool ;)
Writing a Chef recipe which can get a list of nodes “automagically” is a great start. Congratulations, you don’t need to put IP addresses in a wiki anymore.
That’s just the start. You still need to know things under the hood.
- How (not when - because you know Murphy) will your chef recipe fail?
- If Chef server’s index is corrupt will it cause an outage?
- TCP/IP, DNS, HTTPs, SSH - at a protocol level
Aim to make your system more operable. Use bash scripts if that works for your team. Be sure to write good bash scripts (with error handling), have a guideline and make them testable.
You don’t need fancy technology to encourage collaboration, boring will do. Focus on building the house and not playing with a hammer.
#OpsLife
But I do get why some folks want to be “DevOps Engineers”. I have been on the receiving end of many BOFH. The dark times when:
- servers were treated like pets (vs cattle)
- production access was limited
- it took forever to get production-like environments setup
- monitoring used to suck
- deployments were an all night affair with everybody on a conference call
I definitely did not want to end up in a place like that. Practicing DevOps was a breath of fresh air.
Saw this on twitter “Ops is like a game of Tetris, achievements disappear and errors pile up”. Another one here. It doesn’t have to be that way. Track your outages as a metric and then track mean-time-to-detection (MTTD). Improve those and then add mean-time-to-remediation (MTTR).
If your org does not value that and insists on buying Enterprise DevOps 2.0 Professional Edition, then go find some other org.
So what’s the north star for a career in Technical Operations?
Theo Schlossnagle puts it very eloquently
It is an impossible job requirement: “Knows everything about all technologies deployed in Internet architectures.” While no one fills this requirement, what I want is someone whose career goal is to find out how close they can get.
And don’t be a “jerk”.
Developer Productivity/Tools teams
Some organizations have a “tools” or a “developer productivity” team. They maybe in charge of maintaining the CI/CD or the monitoring stack. They may also write tools which make your life easier (or harder). But you still need the plain old Ops team which works very closely with the tools team and breaks the silos.
And they (the tools team) should be part of an “on-call” rotation. There is something about the experience of getting paged at 3 am and troubleshooting an incident by yourself! It will result in way better internal tools/processes than any amount of sprint planning. Although this should be handled properly to avoid burn-outs (that’s a post for another day).
DevOps before DevOps
By the way, look around you, you may already be in a team which practices DevOps
- Do your devs have access to production?
- Do they troubleshoot with you?
- Do the ops and dev teams look at the same set of metrics?
If you have that, you are already ahead of 70% of IT shops out there.