How to test provisioning and configuration in Ansible setup?

Question

Looking at trying to build some resilience into our Ansible setup which deals with provisioning and configuration.

I understand a few methods of testing on the configuration side of things but I'm wondering how best to implement testing on the provisioning side of things, and if there are any tools which can help with this type of implementation.

Currently a lot of our testing is done serially during the playbook which makes a lot of sense for stuff like "has service come up; is the vip available; has this async task finished" but what really concerns me is our ability to manage drift of configuration at both the application and provisioning layer (such as VM configuration). I'm aware Ansible isn't the best tool for working with configuration drift but I'm curious to see your own opinions.

If you have something to fully automate the process even better. (we have a few ugly scripts which report back in slack daily).

Note: Right now we have a few conditions where a reprovision might occur (e.g. rebuild from backup, critical systems issue) but typically it just loops through some of the ansible configuring tasks and thinks no more of it.

Solution 1

Some options out there..

Testing tools: Sorted by github stars

• Serverspec - Ruby, most popular tool out there, built on ruby's rspec
• Goss - YAML, simple, <10MB self-contained binary, extremely fast, can generate tests from system state
• Inspec - Ruby, think of it as an improved serverspec, almost same syntax, made by the chef guys. Built to be easier to extend than serverspec
• Testinfra - Python, has the cool feature of being able to use Ansible's inventory/vars

Major differences between them:

Ultimately, I would suggest spending a day experimenting with all of them to get a feel for them before deciding for yourself.

• With the exception of Goss, all the frameworks can run against a remote machine (ex. over ssh). Goss only runs locally or in docker w/ dgoss.
• All frameworks can be run locally on a server, but require Python or Ruby to be installed or embedded. Inspec provides a self-contained <150MB bundle with an embedded version of ruby. Goss is a single <10MB binary with no external dependencies.
• Goss has built in support for nagios/sensu output, this allows for easier integration with monitoring tools.
• Goss tests tend to be simpler, but less flexible since it's based on YAML. Other frameworks allow you to leverage the full power of the underlying language Python/Ruby to write tests or extend the tool's functionality. (simplicity vs flexibility)
• Goss allows you to generate tests from current system state
• Testinfra to my knowledge is the only one that has built-in support for ansible inventory and variables
• Inspec is backed by Chef

Continuous/divergence testing:

• Chef Compliance - works with inspec to continuously test your servers, paid product
• Goss - Can be easily hooked into Nagios or Sensu. Also, supports exposing server tests as an http endpoint.

Testing harnesses for development:

• kitchen - Testing harness tool, launches instance, runs config management code, runs test suite. Made by the chef guys
• Molecule - Similar to test kitchen, but written specifically for ansible

Full Disclosure: I'm the author of goss

UPDATE: InSpec 4.x or above uses a mixed commercial / open source license - see comments.

Solution 2

The two tools I've seen for this are InSpec and ServerSpec. Serverspec is a Ruby-based tool that builds on RSpec. InSpec is inspired by RSpec and ServerSpec.

I've used ServerSpec. It's cool, but maybe not 100% stable. I've had problems with testing for specific versions of software on Ubuntu.

I've read the InSpec docs but haven't dug in deep. It does essentially the same thing as Serverspec.

Judging by the Github commits, it looks like work on ServerSpec has tailed off somewhat, whereas InSpec is just getting ramped up.

UPDATE: InSpec 4.x or above uses a mixed commercial / open source license - see comments.

Solution 3

When using configuration management tools, such as Ansible, the tool itself would be responsible preventing configuration drift. Once you used Ansible to set a certain configuration, repetitive execution of Ansible will ensure your configuration is as you defined it to be. This also requires your Ansible code to be written in a manner which is idempotent.

Given the above, testing provisioning can be achieved by executing your Ansible playbooks in a loop from some server. For example, a cron job, or Jenkins, can execute the playbooks every 30 minutes and report back on any failure. Having no failures means that your configuration is in check, having failures means that there is a problem getting servers into your desired state.

In a case where you cannot trust your code to be written as idempotent, and thus you cannot really execute Ansible repetitively in a loop from an automatic server a workaround exists. You can do the same as above (run Ansible in a loop) but use its dry run mode. Each time Ansible reports that changes are required, the Jenkins job (or cron job) can notify you that your provisioned configuration has been altered and servers are not in their desired state.

To make sure that your Ansible code is actually doing what you think it should be doing, the solutions mentioned by Dave Swersky apply. Both InSpec and Serverspec are tools that verify in a different way that your playbooks actually does what you mean. A great way to execute these kinds of tools in test environments (even docker containers) is to use kitchen.ci which handles all the glue between the various infra unit testing tools, and execution of your playbooks/modules/cookbooks.

Kitchen.ci was originally used to test Chef cookbooks, but plugins exist for Ansible and other CM tools as well.

Solution 4

Test Kitchen has a kitchen-ansible provisioner plugin for testing of Ansible code. It isn't as deep as the Chef integration but it does get the job done for most cases. There is also the more recent Molecule project which is a dedicated Ansible testing system.

View more solutions

Comments

[Admin]

serverspec.org

[Woodland Hunter]

Does Ansible run only once upon provisioning? If not, what frequency is it being run? Just trying to understand the problem before offering a solution.

[Richard Slater]

Hi @Naphta any of the answers has solved your question please consider accepting it by clicking the check-mark. This indicates to the wider community that you've found a solution and gives some reputation to both the answerer and yourself. There is no obligation to do this.

[Admin]

"I've had problems with testing for specific versions of software on Ubuntu." I fixed this after noticing a question about it on SO: stackoverflow.com/questions/42417864/…

[coderanger]

To slightly clarify, InSpec emulates RSpec but does not build on it.

[Dave Swersky]

@MattSchuchard Thanks for that reference!

[ᴳᵁᴵᴰᴼ]

well "not 100% stable" for a testing tool does not sound any good

[Tensibai]

I understand you're a little biased, but inspec doesn't need Compliance to run periodically. And there's no dependencies to manage, all necessary libs and framework (ruby) is bundled in the package which can be installed locally on each node, when running through ssh/winrm, inspec put itself in place. You point about exclusively external command which is not true, a lot of tests are done by internal ruby code. (I think it worth being noted)

[Ahmed Elsabbahy]

I've updated the answer to correct the external command and bundled ruby for inspec. Can you clarify or send me a link explaining how inspec can be used on it's own to detect/report on drift?

[Tensibai]

Well, the reporting part will be at your hand, it's indeed Compliance goal to make a representation. But you can run inspec in a crontab and just rely on the crontab mail when there's a test failing to alert you (for example).

[Tensibai]

All in all, tahnks for the edit, that sounds a fair exposition of your tool and others. I'm afraid this may become outdated quickly, but +1 anyway for the listing and pointers.

[Ahmed Elsabbahy]

Ah, yeah.. all tools can be leveraged in that way. I was trying to provide tools (or integrations with tools) that provide better reporting. To my knowledge there's compliance, or wrapping the tools in a way that makes them nagios/sensu/some-other-monitoring-tool friendly. Ex: slideshare.net/m_richardson/… Apologies if it came off as biased initially, wasn't intended to be.

[Tensibai]

Indeed, all tools can be leveraged that way, it's just the sentence about compliance sounds like Compliance is necessary to run inspec periodically, which it doesn't provide, you still have to manage the inspec run to report back to compliance. You may take advantage of this reporting to send to an elasticsearch server for example, but that's hacky and out of the scope.

[Tensibai]

And no worry for the bias, I just said what I felt 'wrong' at first read. There's no problem being mistaken by the description of the tools and even less when you take it positive and review the post :)

[RichVel]

InSpec is very good as a test tool, and makes it easy to install nothing on the server, whereas ServerSpec can only do this with some extra work. Some work would be needed to have InSpec use an Ansible inventory though - probably easier if inventory is in YAML format (Ansible 2.4+).

[chicks]

Welcome to DevOps.se Alexey. While your tool may help the user in this case, for this to be an answer here there should be a little more to convey how this is relevant to the question. Otherwise it looks like a link-only advertisement.

[Alexey Melezhik]

Hi! Sure, just given more details.

[Tensibai]

From my point of view, this does nothing more than what rundeck can do and is out of scope for this question. Specially as the answer doesn't explain how it can solve auditing a thousand servers and be alerted of a drift or present a human readable report for an auditor by giving some parseable output. From what I just read it does no more than inspec can do (for exemple) and give less usable output at scale while requiring a lot of external tools to do the job and thus being less portable.

[Alexey Melezhik]

"how it can solve auditing a thousand servers" - I could find anything about thousands of servers in the question

[Alexey Melezhik]

"and be alerted of a drift or present a human readable report for an auditor by giving some parseable output." neither I found this in the question. The obvious alerting of drifting is the fact that tests begin to fail ...

[Tensibai]

So you're just missing the point of what those kind of tools are for in a wider point of view than just assessing state. There's not much else to say

[Alexey Melezhik]

Not at all. You just related to the points ( scalability and report format ) that are not originaly touched in the question. However no problem with that - Outthentic is both scalable and provides sensible reports/output.

[RichVel]

InSpec update - current versions (4.x and higher) are now commercial requiring payment, though underlying source code remains open source. If using for personal use, non-commercial use, or 'experimental' in a business, InSpec is still free, but in other cases they require a commercial license. See Chef licensing docs page and EULA. Beware the confusing "Chef is 100% open source" statements from Apr 2019 - technically true but executable tools using Chef trademarks are subject to EULA.

[RichVel]

InSpec update - current versions (4.x and higher) are now commercial requiring payment, though underlying source code remains open source. If using for personal use, non-commercial use, or 'experimental' in a business, InSpec is still free, but in other cases they require a commercial license. See Chef licensing docs page and EULA. Beware the confusing "Chef is 100% open source" statements from Apr 2019 - technically true but executable tools using Chef trademarks are subject to EULA.

[RichVel]

Commented here about InSpec license because I talked to some people using it without realising it now has a commercial license for most business usage.