In my organization, I work with a group of NOC staff, budding junior engineers and a handful of senior engineers; all with a focus on Linux. One interesting step in the way the company grows talent is that there's a path from the NOC to the senior engineering ranks. Viewing the talent pool as a relative newcomer, I see that there's a split in the skill sets that tends to grow over time…
- There are engineers who know one or several particular technologies well and are constantly immersed… e.g. MySQL, firewalls, SAN storage, load balancers…
- There are others who are generalists and can navigate multiple technologies.
- All learn enough Linux (commands, processes) to do what they need and use on a daily basis.
A differentiating factor between some of the staff is how well they embrace scripting, automation and configuration management methodologies. For instance, we have two engineers who do the bulk of Amazon AWS CloudFormation work, and another who handles most of the Puppet infrastructure. Perhaps a quarter of the engineers are adept at BASH shell scripting.
Looking at this in the context of the incredibly high demand for DevOps skills in the job market, I'm curious how other organizations foster the development of these skills and grow their internal talent. Scripting doesn't seem like a particularly-teachable concept.
- How does a sysadmin improve their shell scripting?
- Is there still a place for engineers who do not/cannot keep up in the DevOps paradigm?
- Are we simply to assume that some people will be left behind as these technologies evolve? Is that okay?
Best Answer
I have the benefit of understanding the size and complexity of your environment. Seeing as you work for a cloud/hosting provider, it's safe to assume that you have a large number of small-medium sized environments (10-100 servers). There are certainly daily tasks that are done by the jr. engineers and NOC staff that are repetitive (creating user accounts, configuring backup agents, etc). Similarly, there are probably some manual things that are done by the sr. engineers like installing ESXi on new hardware or configuring things like MPIO or installing VMware modules for specific sets of hardware. All of these things can and should be automated.
If your staff is capable of carrying out the bulk of their workload without automating, then you're overstaffed in my opinion. Any IT staff that can work a full day that consists of mostly manual processes has no motivation to automate. Why learn a new skill that isn't viewed as necessary and might even be scary? After all, necessity is the mother if innovation.
So, at some point in your organization, you will grow to a size where you'll flounder and fall apart, or you'll start automating almost everything and excel. Certainly, the senior engineers should be leading the charge here, and maybe even working with the junior engineers and NOC staff to automate some of their workload. This gives the jr. engineers the opportunity to have the framework of many scripts to work with, which they can tweak for each tenant and new hardware revision as necessary. This removes the daunting thought of "Oh my god, where do I even start?" from the equation and gives them a jumpstart to solving a real problem. Which brings me to my final point. Books and examples are well and good, but there's nothing that can replace the feeling of accomplishment of solving an actual problem that they face. Give them a goal, like all new servers for tenant x should have certain ESXi modules installed, and then work with them to accomplish it. Then adapt the script to work in a multitenant environment.
By needing to, as described above.
Sure, there are plenty of organizations that cannot or will not shift to the DevOps methodology. They're seeming to be more and more boring options, but they're options nonetheless.
As with any new technology - yes.
tl;dr You'll never have anyone really invest in learning it until they see the value in it. If they can accomplish their daily tasks manually, then you're overstaffed and there's no incentive.