Why can't chef resolve my cookbooks?

Question

Intro I am learning Chef to automate the server management at work. I downloaded chefdk 3.0 from here and now I am trying to build my first cookbook using chef.

Important I am using this in a Windows environment for testing purpose, I do expect it to fail since Windows does not have iptables, but I do not expect it to fail saying that it can't find the cookbook. I've tried using Windows cookbook and it works.

The problem I am able to create the cookbook and run it, but I am not able to reference dependencies from supermarket.

I have tried two alternatives:

Alternative 1

I used the following command to create the cookbook

chef generate cookbook learn_chef_httpd

(from this tutorial)

I was able to complete the tutorial and now I would like to test referencing another cookbook, so I chose simple_iptables

I added this line

cookbook 'simple_iptables', '~> 0.7.0'

To my Berksfile, as described in the Supermarket.

Then I added these lines to my default.rb file:

include_recipe 'simple_iptables'

# Allow HTTP, HTTPS
simple_iptables_rule "http" do
  rule [ "--proto tcp --dport 80",
         "--proto tcp --dport 443" ]
  jump "ACCEPT"
end

And I run the cookbook using:

chef-client --local-mode --runlist 'recipe[learn_chef_httpd]'

The problem is that Chef doesn't find the cookbook

Chef::Exceptions::CookbookNotFound
----------------------------------
Cookbook simple_iptables not found. If you're loading simple_iptables from anoth er cookbook, make sure you configure the dependency in your metadata

I tried adding it to the metadata:

depends 'simple_iptables', '~> 0.7.0'

But I still get an error:

Error Resolving Cookbooks for Run List:
Missing Cookbooks:

No such cookbook: simple_iptables

Alternative 2

I am still trying to make it work so I also tried making it "the berkshelf way", so I created a new cookbook.

berks cookbook test

And I added this line

cookbook 'simple_iptables', '~> 0.7.0'

To my Berksfile, as described in the Supermarket.

Then I added these lines to my default.rb file:

include_recipe 'simple_iptables'

# Allow HTTP, HTTPS
simple_iptables_rule "http" do
  rule [ "--proto tcp --dport 80",
         "--proto tcp --dport 443" ]
  jump "ACCEPT"
end

Executed berks install:

berks install

and ran it:

chef-client --local-mode --runlist 'recipe[test]'

The same error came back

Chef::Exceptions::CookbookNotFound
----------------------------------
Cookbook simple_iptables not found. If you're loading simple_iptables from anoth er cookbook, make sure you configure the dependency in your metadata

I tried adding it to the metadata:

depends 'simple_iptables', '~> 0.7.0'

But I still get an error:

Error Resolving Cookbooks for Run List:
Missing Cookbooks:

No such cookbook: simple_iptables

I looked at the ~/berkshelf folder and the cookbooks are there.

** Alternative 3 **

I started a CentOS 6.5 EC2 instance on Amazon, installed Ruby 2.1.3 and Chef. created a ~/chef-repo/cookbooks folder

I created a cookbook using berkshelf, ran

bundle install

added the reference/code as in the other alterantives then

berks install

and ran the same way I did last time.

I got the same issues.

What am I missing? What do I need to make it work?

Solution 1

Make sure you have configured your chef_repo_path as described in the docs.

Basically local-mode needs to know where to find your cookbooks, roles, environments, data bags, etc. Sadly, the documentation is not super clear on where/how you set chef_repo_path

Here's what I can tell from the code.

1. if client.rb is found and contains cookbook_path, chef_repo_path = "#{cookbook_path}/.."
2. if knife.rb is found and contains cookbook_path, chef_repo_path = "#{cookbook_path}/.."
3. Chef can try to divine the path. The code will search from pwd upward looking for a directory called cookbooks. If it finds it, then cookbooks/.. will be set as your chef_repo_path.
4. If all else fails, pwd will be used as your chef_repo_path

If you are using Berkshelf, your best bet is to do a berks vendor in order to get a single directory will ALL of the cookbooks you need. You can then point chef_repo_path to the parent of the cookbooks directory which holds your vendored cookbooks.

Mind you, that's from 10 minutes of digging in the source code, so I may not have it quite right.

Solution 2

While looking to run locally for testing I ran into the same "cannot resolve cookbook" error. Since I potentially want to run cookbooks without a Chef server but also without ChefDK installed for non-Chef developers to do a simple one time workstation setup, I put together the following workflow.

Your cookbook needs to live in a folder under a folder called cookbooks, this appears to be hardcoded "magic", but it seems to be able to live anywhere, eg userdir/projects/cookbooks/my_cookbook_name/.

If you have a Berksfile for the dependencies you need to get them locally before you can run chef-client -z -o my_cookbook_name so run berks vendor to pull the dependencies into a cookbooks directory in the current folder.

The berks vendor command pulls your other dependencies into the cookbooks directory so chef-zero/chef-solo can find them. Run the following in an Administrator PowerShell prompt, it may be Command Prompt compatible but I try not to encourage using cmd.exe.

cd /path/to/cookbooks/my_cookbook_name berks vendor . chef-client -z -o my_cookbook_name

The beauty of using the new chef-zero built into the client is you don't necessarily need the full ChefDK to test, though you will need berkshelf for the berks command on at least 1 machine as getting that installed without the ChefDK is a nightmare. Once you have berks you can run berks package and copy the file it creates to a machine with just the chef-client and extra it somewhere and then run the chef-client -z -o your_cookbook in the directory containing the cookbooks folder.

To get just chef use the omnibus installer and you have enough to bootstrap a node or run the recipe.

. { iwr -useb https://omnitruck.chef.io/install.ps1 } | iex; install -channel stable -project chef

If you need your berks and want a quick way to get the ChefDK, this will do the trick.

. { iwr -useb https://omnitruck.chef.io/install.ps1 } | iex; install -channel stable -project chefdk

Solution 3

After few hours of breaking my mind trying a combination of @kilian and @gratzy's answers from here, this helped,

cmd /c E: && cd /d E:\Items_CI Exploration\chef\chef-repo\cookbooks\ && chef-client --local-mode --runlist 'recipe[cisamplecookbooks::web]'

Yes, without the quotes. Also notice the "E:" added before. The number of '&' didn't make a difference though.

"E:\Items_CI Exploration\chef\chef-repo\cookbooks\" this is the path under which I had my cookbooks to be run. I guess last slash ("\") next to cookbooks is also important.

Comments

[Tejay Cardon]

Are you adding 'these lines' to your berksfile or to your default.rb recipe?

[JSBach]

To the default.rb recipe

[Tensibai]

I may be wrong, but you didn't mention doing a berks install after modifying the berksfile. You may also wish to use berks vendor to make a directory conatining all the cookbooks needed, and pointing chef-client repo path to this directory. All in all, your cookbooks are on your disk, but not somewhere chef-zero look to.

[JSBach]

@Tensibai you are right, I didn't mentioned it there, I'm sorry, my bad! I updated the question

[JSBach]

If I used Berkshelf, it should be located in ~/.berkshelf, right? Shouldn't it look for this path? I've tried using other cookbooks (like this one supermarket.getchef.com/cookbooks/windows ) and it works...

[Tejay Cardon]

I can't say for sure, as I don't use local mode. My suggestion is that you use berks vendor to bundle up all your dependencies into a single cookbooks folder, and then assign the cookbook path explicitly. Even if it does default to ~/.berkshelf, you're not going to want to use that location for long. Pretty quickly you'll have multiple version of cookbooks there as well as needing data bags, roles, or environments. You're best approach is a dedicated location for these things.

[JSBach]

I understand, the problem is that I am going to upload this cookbook to Amazon OpsWorks, so I won't be able to set the path and so on. It has to work "out of the box" unfortunately :(

[Tejay Cardon]

Two unrelated things. Your problem is not with your cookbook, it is with your local configuration. When you push it to OpsWorks, your AWS nodes will be configured to pull cookbooks from OpsWorks. Have you tried the cookbook as-is in opsworks? (be sure to leave the dependency in your metadata.rb file)

[JSBach]

Ok, I will try it. I'm sure I am doing something wrong, I'm going to update the question with the attempt I just had

[JSBach]

Trying to update it directly to Opswporks gives me the same issue No such cookbook: simple_iptables This is getting frustrating and my manager is asking me to stop this task :( I have until the end of the day to figure this out or I will have to change my task priorities :(

[JSBach]

This thread solves it: github.com/berkshelf/berkshelf/issues/1297 quoting: "To use local mode, you would need to berks vendor and then point Chef Zero at that vendored directory. Chef does not know how to read from Berkshelf or the Berkshelf shelf."