Chef: Why are resources in an "include_recipe" step being skipped?

28,569

Solution 1

It turns out this is a pretty normal (but under-documented) part of how Chef works: it compiles everything, and then starts running. Except some recipes (like Postgres) jump the queue to actually install at compile-time, using code like this:

  action :nothing
end.run_action(:run)

The solution is that anything that needs to run before Postgres also needs to do this. Fortunately, newer versions of Build-essentials allow this. You set attributes on a role as follows:

name "myapp"
run_list(
  "recipe[build-essential]",
  "recipe[myapp]"
)
default_attributes(
  "build_essential" => {
    "compiletime" => true
  }
)

Solution 2

if writing a cookbook, add the attribute to your attributes file:

node.default['build_essential']['compiletime'] = true

Thanks to Colin above.

Share:
28,569
Steve Bennett
Author by

Steve Bennett

Freelance Mapbox GL JS, vector tile and VueJS guru, available for consulting work. See http://hire.stevebennett.me. I specialise in high performance web map data visualisations, and custom vector tile generation for the Mapbox platform. I use NodeJS for back-end data processing, and VueJS for front end. My blog: http://stevebennett.me My NPM profile: https://www.npmjs.com/~stevage (Please don't email me asking for help with a StackOverflow question.)

Updated on May 04, 2020

Comments

  • Steve Bennett
    Steve Bennett about 4 years

    Chef seems to be processing resources in a strange order, causing my build to fail. My main recipe (mytardis-chef/site-cookbooks/recipes/default.rb) starts like this:

    include_recipe "build-essential"
    include_recipe "mytardis::deps"
    include_recipe "mytardis::nginx"
    include_recipe "mytardis::postgresql"
    

    mytardis-chef/cookbooks/build-essential/recipes/default.rb looks like this:

    case node['platform']
    when "ubuntu","debian"
      %w{build-essential binutils-doc}.each do |pkg|
        package pkg do
          action :install
        end
      end
    when "centos","redhat","fedora","scientific"
      %w{gcc gcc-c++ kernel-devel make}.each do |pkg|
        package pkg do
          action :install
        end
      end
    end
    

    ... (it's an older version of https://github.com/opscode-cookbooks/build-essential/blob/master/recipes/default.rb)

    At runtime, for reasons I don't understand, this build-essential recipe gets loaded but not executed:

    [default] Waiting for VM to boot. This can take a few minutes.
    [default] VM booted and ready for use!
    [default] Mounting shared folders...
    [default] -- v-root: /vagrant
    [default] -- v-csr-3: /tmp/vagrant-chef-1/chef-solo-3/roles
    [default] -- v-csc-2: /tmp/vagrant-chef-1/chef-solo-2/cookbooks
    [default] -- v-csc-1: /tmp/vagrant-chef-1/chef-solo-1/cookbooks
    [default] -- v-csdb-4: /tmp/vagrant-chef-1/chef-solo-4/data_bags
    [default] Running provisioner: Vagrant::Provisioners::ChefSolo...
    [default] Generating chef JSON and uploading...
    [default] Running chef-solo...
    [Sun, 08 Jul 2012 05:14:32 +0200] INFO: *** Chef 10.12.0 ***
    [Sun, 08 Jul 2012 05:14:32 +0200] DEBUG: Building node object for lucid32
    [Sun, 08 Jul 2012 05:14:32 +0200] DEBUG: Extracting run list from JSON attributes provided on command line
    [Sun, 08 Jul 2012 05:14:32 +0200] INFO: Setting the run_list to ["recipe[mytardis]"] from JSON
    [Sun, 08 Jul 2012 05:14:32 +0200] DEBUG: Applying attributes from json file
    [Sun, 08 Jul 2012 05:14:32 +0200] DEBUG: Platform is ubuntu version 10.04
    [Sun, 08 Jul 2012 05:14:32 +0200] INFO: Run List is [recipe[mytardis]]
    [Sun, 08 Jul 2012 05:14:32 +0200] INFO: Run List expands to [mytardis]
    [Sun, 08 Jul 2012 05:14:32 +0200] INFO: Starting Chef Run for lucid32
    [Sun, 08 Jul 2012 05:14:32 +0200] INFO: Running start handlers
    [Sun, 08 Jul 2012 05:14:32 +0200] INFO: Start handlers complete.
    ...
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe mytardis via include_recipe
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe default in cookbook mytardis
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe build-essential via include_recipe
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe default in cookbook build-essential
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe mytardis::deps via include_recipe
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe deps in cookbook mytardis
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe mytardis::nginx via include_recipe
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe nginx in cookbook mytardis
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe iptables via include_recipe
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe default in cookbook iptables
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe mytardis::postgresql via include_recipe
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe postgresql in cookbook mytardis
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe postgresql::server via include_recipe
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe server in cookbook postgresql
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe postgresql::client via include_recipe
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe client in cookbook postgresql
    [Sun, 08 Jul 2012 05:14:33 +0200] INFO: Processing package[postgresql-client] action install (postgresql::client line 37)
    [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: package[postgresql-client] checking package status for postgresql-client
    ....
    [Sun, 08 Jul 2012 05:14:45 +0200] ERROR: gem_package[pg] (postgresql::client line 42) has had an error
    .
    make
    sh: make: not found
    

    That is, the build-essential recipe is "found" and "loaded", but it's the postgres recipe that gets "processed" first. And since build-essential (which installs the C compiler) wasn't run, the latter fails.

    The relevant part of my Vagrantfile looks like this:

      config.vm.provision :chef_solo do |chef|
         chef.log_level = :debug
         chef.cookbooks_path = ["mytardis-chef/site-cookbooks", "mytardis-chef/cookbooks"]
         chef.roles_path = "mytardis-chef/roles"
         chef.data_bags_path = "mytardis-chef/data_bags"
         chef.add_recipe "mytardis"    
      end
    

    I was previously using a slightly earlier version of Chef (perhaps 10.10.0?) In that version, build-essential was also not being run, but mytardis::deps was. Now using Chef 10.12.0. Physical machine is OSX, VM is Ubuntu Lucid.

    So, a few questions:

    1. Why is build-essential not being "processed"?
    2. What's the right way to do this? (I didn't write these recipes, and I understand they do or have worked for their author.)
    3. Should the site-cookbooks and cookbooks 'shadowing' feature still work? It's supposedly deprecated: http://tickets.opscode.com/browse/CHEF-2308 (I tried making a symbolic link under from site-cookbooks/mytardis/recipes/build-essential but no joy).