Why use Chef/Puppet over shell scripts?

Question

New to Puppet and Chef tools. Seems like the job that they are doing can be done with shell scripting. Maybe it was done in shell scripts until these came along.

I would agree they are more readable. But, are there any other advantages over shell scripts besides just being readable?

Solution 1

A domain-specific language makes a big difference in the amount of code you write. For example, you could argue that there's not much difference between:

chmod 640 /my/file

and

file { "/my/file":
    mode => 640,
}

but there's a great deal of difference between these:

FILE=/my/file
chmod 640 $FILE
chown foo $FILE
chgrp bar $FILE
wget -O $FILE "http://my.puppet.server/dist/$FILE"
     # where the URL contains "Hello world"

and

file { "/my/file":
    mode => 640,
    owner => foo,
    group => bar,
    content => "Hello world",
}

What happens if the wget fails? How will your script handle that? And what happens if there's something after that in your script that requires $FILE to be there with the correct contents?

You might argue that one could just put echo "Hello world" > $FILE in the script, except that in the first example the script must be run on the client, whereas puppet compiles all of this on the server. So if you change the content, you only have to change it on the server and it changes it for as many systems as you want to put it on. And puppet handles dependencies and transfer problems for you automatically.

There's just no comparison - proper configuration management tools save you time and complexity. The more you try to do, the more shell scripts seem inadequate, and the more effort you will save by doing it with puppet.

Solution 2

This will be an unpopular opinion, but configuration management systems are not necessarily better. Sometimes simple really is best.

There is a definite learning curve and administrative overhead associated with the configuration system you choose. You are after all introducing a dependency. As with any automation you must also be careful that security is maintained in deployed configurations.

I've only had a couple of instances where we deployed configuration management and it stuck. It was always when there were a large number of systems with repetitive configuration and a need to perform configurable cookie-cutter deployments.

Solution 3

You've answered your own question...

Automation is becoming more scalable and formalized. Puppet and Chef are considered standards these days (check the job ads).

Cobbled-together shell scripts have their place, but they're not scalable in the context of the DevOps movement. Readability is part of that.

Solution 4

Chef makes it a lot easier to manage and version the setup of a complex infrastructure, especially in any type of cloud environment vs. having to manually ftp or scp a bunch of shell scripts organized in a unstandardized fashion. Depending on how many dependencies you need to manage, the size of this win can vary greatly, making the decision to move to a CM solution a non-obvious one for a lot of people.

The real (often unsung) benefit of Chef is idempotence. Being able to be certain of a resource's state regardless of the combination of recipes run with overlapping interests is a huge benefit over shell script configuration. If you have shell scripts for configuration now, ask yourself how many of them can be run multiple times without unintended/undesirable consequences?

A proper CM solution helps to ensure success at scale by simplifying cross-platform automation and team collaboration. While it is possible to accomplish all this with a well organized, properly maintained, versioned group of shell scripts; you've got to ask yourself "why". Chef/Puppet and similar technologies came about because a group of talented SysOps were tired of having to solve these same problems over and over again and set out to give us a better option.

Key pieces:

• Idempotence
• Ease of Dependency Management (versioning)
• Standardized Organization (accepted at an industry level)
• Abstraction to separate server configuration tasks from system level details
• Ability to leverage community knowledge (that is guaranteed to embrace all the above principles)

Solution 5

Modern configuration management tools such as Puppet and Chef allow you to define the state of a system instead of worrying about activities necessary to achieve a configured server.

For example, your chmod command assumes that the file exists, the user owning the file exists, that the directories have been created, and so on. Your script therefore must consider all of these prerequisites.

State-based configuration management tools are simpler: you just care that the file has the correct permissions. How that is achieved is the tool's problem.

View more solutions

Comments

[Oupkar Sandhu]

When the cost of managing the environment is so great that managing the automation is truly cheaper. If it is simpler to script changes managed in Git or set things up in an ssh multiplexer we do that. What's most important for me is that I monitor the systems and verify everything is functioning as expected. As long as I am alerted when something is misconfigured, it is not so important how it gets configured.

[Steve Townsend]

@ewwhite "when do you decide to automate vs. not" xkcd.com/1205

[GomoX]

More modern tools such as Ansible have considerably less deployment/management overhead than Chef or Puppet, requiring little to no dependencies (Ansible in particular is agentless). This really lowers the bar for adoption.

[AdamH]

@Paul Gear, it is an oversimplification to say that configuration management tools are the way to go in the long run. For example, using AWS a shell script can build the server from scratch, run some tests, and once you sure the AWS instaces is ok, make an image. Then you can deploy this image to a preview or staging environment for further tests, once you validate the image is good for your purposes, then you push to production. Configuration management tools do not help at all in this scenario.

[AdamH]

Now if you want to manage 100's of dedicated servers that people are using daily (and you have little control over what they do, mistakenly or not) that's where configuration management tools should be used.

[AdamH]

@ewwhite You automate when you want to for personal productivity. You learn by automating, you do not learn by doing mundane tasks. Learning = productivity increase and iterative improvement. The automation combines with this to produce more positives. It's a positive snowball instead of a negative one. Technical Progress vs Technical Debt.

[Asclepius]

Are shell scripts somehow less formalized? Does quoting job ads make an argument convincing? And a devops is quite a shame, for s/he is underexploited as a developer. The link says nothing really about scalability. Is the claim that shell scripts are not scalable? I think Puppet is interesting, but not necessarily for the reasons in the answer.

[Asclepius]

Actually my chmod does not assume that the file exists because the file was either created by or tested-for-existence by the script.

[ewwhite]

Job ads reflect the real market for specific skills. Yes, using configuration management for its intended purpose is more scalable, more robust and easier to document than shell scripts. I've been in many environments, and most scripts I see are not constructed in a manner that one would deem "production quality" and organized to handle exceptions. See the accepted answer to understand why.

[Asclepius]

Idempotence is hardly impossible with ss. Dependencies are managed well by yum/apt, etc. Acceptance is irrelevant. Community knowledge exists for ss. I accept, however, that not having to copy over a bunch of scripts, and also configuring systems centrally, are both useful. I hear puppet requires a client-side agent.

[AdamH]

@A-B-B No, that is not implied. I do not even mention shell scripts, I mention automation as a general practice. You can automate things with shell scripts and learn the scripting abstraction instead of doing things manually which will not only save you the time of doing that task again, it will prevent mistakes. Also, you should be able to write your next script a bit better then the last one. A Positive snow ball... However, if you are an expert at writing scripts and you're not scripting something you will ever run again, it does not help you learn or save you time in any way.

[Sridhar]

This isn't a very convincing example. I could start with wget and then I wouldn't end up in a weird state. Is file like a transaction that will get rolled back if any of the steps don't succeed?

[Chaim Eliyah]

"And a devops is quite a shame, for s/he is underexploited as a developer." This simply isn't true. DevOps is a development specialization, just like Web Development. The patterns and practices of software development still apply. You should read about DevOps.