Ansible - with_dict: dictionary - How to use variables defined in each dictionary which depends upon others

Question

Environment is: Ansible 1.9.2, CentOS 6.5

I have created a role to download JAVA (.tar.gz) artifact files for 3 different JAVA versions from Artifactory. I'm trying to use Ansible's with_dict feature (instead of using with_items).

Created the following files:

$ cat roles/java/defaults/main.yml

---
java_versions:
  java7_60:
    version: 1.7.60
    group_path: com/oracle/jdk
    classifier: linux-x64
    ext: tar.gz
    dist_file: "jdk-{{ version }}-{{ classifier }}-{{ ext }}"
#    dist_file: "jdk-{{item.value.version }}-{{ item.value.classifier }}-{{ item.value.ext }}"
    dist_url: "{{ artifactory_url }}/{{ group_path }}/{{ version }}/{{ dist_file }}"
#    dist_url: "{{ artifactory_url }}/{{ item.value.group_path }}/{{ item.value.version }}/{{ dist_file }}"

  java7_67:
    version: 1.7.67
    group_path: com/oracle/jdk
    classifier: linux-x64
    ext: tar.gz
    dist_file: "jdk-{{item.value.version }}-{{ item.value.classifier }}-{{ item.value.ext }}"
    dist_url: "{{ artifactory_url }}/{{ item.value.group_path }}/{{ item.value.version }}/{{ dist_file }}"
  java8_45:
    version: 1.8.45
    group_path: com/oracle/jdk
    classifier: linux-x64
    ext: tar.gz
    dist_file: "jdk-{{item.value.version }}-{{ item.value.classifier }}-{{ item.value.ext }}"
    dist_url: "{{ artifactory_url }}/{{ item.value.group_path }}/{{ item.value.version }}/{{ dist_file }}"

How can I set or use dist_file or dist_url variables which depends upon other variables defined in the same KEY (lets say in KEY java7_60)?

Right now, when I'm trying either the current dist_file or dist_url variables OR the commented out lines way of setting them (i.e. using item.value.), it's not setting the value of these 2 variables as desired i.e. depending upon other variables version, group_path, classifier, ext and artifactory_url (which is defined in another common role's defaults/main.yml file)).

I saw that for using with_dict: within a playbook/task, I have to use {{ item.value.variable_name }} but how can I define a variable which is dependent upon others within the same KEY section of a dictionary.

The error message I'm getting while using the above dictionary in the following task is:

$ cat roles/java/tasks/main.yml:

- name: Download Java/JDK Versions
  command: wget -q "{{ item.value.dist_url }}"
    chdir="{{ common_download_dir }}"
    creates="{{ common_download_dir }}/{{ item.value.dist_file }}"
  with_dict: "{{ java_versions }}"
  become_user: "{{ build_user }}"

Error message with using dist_file / dist_url (with the current setup in roles/java/defaults/main.yml):

TASK: [java | Download Java/JDK Versions] *************************************
failed: [server01.poc.jenkins] => (item={'key': 'java7_60', 'value': {'dist_file': u'jdk-{# version #}-{# classifier #}-{# ext #}', 'ext': 'tar.gz', 'version': '1.7.60', 'dist_url': u'{# artifactory_ur #}/{# group_path #}/{# version #}/{# dist_file #}', 'group_path': 'com/oracle/jdk', 'classifier': 'linux-x64'}}) => {"changed": true, "cmd": ["wget", "-q", "{# artifactory_url #}/{# group_path #}/{# version }/{# dist_file #}"], "delta": "0:00:00.006081", "end": "2015-11-23 12:50:18.383728", "item": {"key": "java7_60", "value": {"classifier": "linux-x64", "dist_file": "jdk-{# version #}-{# classifier #}-{# ext #}, "dist_url": "{# artifactory_url #}/{# group_path #}/{# version #}/{# dist_file #}", "ext": "tar.gz", "group_path": "com/oracle/jdk", "version": "1.7.60"}}, "rc": 4, "start": "2015-11-23 12:50:18.377647", "wrnings": ["Consider using get_url module rather than running wget"]}

Error message with using dist_file / dist_url (with the lines which are currently commented out in roles/java/defaults/main.yml):

TASK: [java | Download Java/JDK Versions] *************************************
failed: [server01.poc.jenkins] => (item={'key': 'java7_60', 'value': {'dist_file': u'jdk-{#item.value.version #}-{# item.value.classifier #}-{# item.value.ext #}', 'ext': 'tar.gz', 'version': '1.7.60' , 'dist_url': u'{# artifactory_url #}/{# item.value.group_path #}/{# item.value.version #}/{# dist_file #}', 'group_path': 'com/oracle/jdk', 'classifier': 'linux-x64'}}) => {"changed": true, "cmd": ["wget",  "-q", "{# artifactory_url #}/{# item.value.group_path #}/{# item.value.version #}/{# dist_file #}"], "delta": "0:00:00.005900", "end": "2015-11-23 12:36:24.131327", "item": {"key": "java7_60", "value": {"cla ssifier": "linux-x64", "dist_file": "jdk-{#item.value.version #}-{# item.value.classifier #}-{# item.value.ext #}", "dist_url": "{# artifactory_url #}/{# item.value.group_path #}/{# item.value.version #}/{#  dist_file #}", "ext": "tar.gz", "group_path": "com/oracle/jdk", "version": "1.7.60"}}, "rc": 4, "start": "2015-11-23 12:36:24.125427", "warnings": ["Consider using get_url module rather than running wget"]}

Solution 1

I believe the best way to do what you want (since I don't believe loop variables can reference themselves) would be to have your task be:

- name: Download Java/JDK Versions
  command: wget -q "{{ artifactory_url }}/{{ item.value.group_path }}/{{ item.value.version }}/jdk-{{item.value.version }}-{{ item.value.classifier }}-{{ item.value.ext }}"
    chdir="{{ common_download_dir }}"
    creates="{{ common_download_dir }}/jdk-{{item.value.version }}-{{ item.value.classifier }}-{{ item.value.ext }}"
  with_dict: "{{ java_versions }}"
  become_user: "{{ build_user }}"

essentially manually inserting the variables into your task.

It's not pretty if you need to use the variables for multiple tasks, but in ansible 1.x I don't think there's a way to make it pretty. Ansible 2.0 has blocks, with which you would be able to loop multiple tasks together over a dict, and you would be able to define variables for all tasks in that block to use.

Solution 2

I was curious about this so I did a bit of digging, and in doing so I came across this similar answer. That's only part of the solution in your case though.

It appears that Ansible won't let you reference a variable from its own definition, which I guess makes sense since it's not fully defined. So something like this won't work, and will in fact throw a somewhat confusing error when the variable is referenced:

---
myvar:
    param1: foo
    param2: "{{ myvar['foo'] }} bar"

It also appears, from your own example, that Ansible won't let you use item constructs in variables to reference other complex variables. This sort of makes sense to me since it seems Ansible resolves jinja2 constructs in variables at the time the variable is defined, not at runtime when the variable is referenced.

So although this isn't exactly what you were looking for, I think if you break apart your variable into two parts you can get this to work by doing something along these lines:

---
artifactory_url: "http://path.to.jarfile"
java_versions:
  java7_60:
    version: 1.7.60
    group_path: com/oracle/jdk
    classifier: linux-x64
    ext: tar.gz

java_downloads:
  java7_60:
    dist_url: "{{ artifactory_url }}/{{ java_versions['java7_60']['group_path'] }}/{{ java_versions['java7_60']['version'] }}/jdk-{{ java_versions['java7_60']['version'] }}-{{ java_versions['java7_60']['classifier'] }}.{{ java_versions['java7_60']['ext'] }}"

When you debug java_downloads this way you get the full URL you're looking for:

TASK: [debug var=item] ********************************************************
ok: [localhost] => (item={'key': 'java7_60', 'value': {'dist_url': u'http://path.to.jarfile/com/oracle/jdk/1.7.60/jdk-1.7.60-linux-x64.tar.gz'}}) => {
    "item": {
        "key": "java7_60",
        "value": {
            "dist_url": "http://path.to.jarfile/com/oracle/jdk/1.7.60/jdk-1.7.60-linux-x64.tar.gz"
        }
    },
    "var": {
        "item": {
            "key": "java7_60",
            "value": {
                "dist_url": "http://path.to.jarfile/com/oracle/jdk/1.7.60/jdk-1.7.60-linux-x64.tar.gz"
            }
        }
    }
} 

Solution 3

Thanks to Bruce P and Sepehr N. I did the following after taking the solutions/hints from their reply. Now, my task is working for multiple tools (jdk, mvn, gradle, maven, etc) with multiple versions with minimal changes made to the files and without requiring (2nd dictionary definition).

What I did:

In a top level / some common role's default/main.yml I'll have the following:

$ cat roles/some_common_global_role/defaults/main.yml

---
a_var: giga
other_var: fifa

dist_file: "{{ item.value.tool }}-{{item.value.version }}-{{ item.value.classifier }}.{{ item.value.ext }}"
dist_url: "{{ artifactory_url }}/{{ item.value.group_path }}/{{ item.value.version }}/{{ dist_file }}"

In $ cat roles/java/defaults/main.yml

---
tool: jdk

java_versions:
  java7_60:
    version: 1.7.60
    group_path: com/oracle/jdk
    classifier: linux-x64
    ext: tar.gz
  java7_67:
    version: 1.7.67
    group_path: com/oracle/jdk
    classifier: linux-x64
    ext: tar.gz
  java8_45:
    version: 1.8.45
    group_path: com/oracle/jdk
    classifier: linux-x64
    ext: tar.gz

In individual tool's roles/default/main.yml, I'll set the same dictionary and tool variable and it'll download multiple tools/versions. In this case, your some common level role's default/main.yml will look like:

---
dist_file: "{{ tool }}-{{item.value.version }}-{{ item.value.classifier }}.{{ item.value.ext }}"
dist_url: "{{ artifactory_url }}/{{ item.value.group_path }}/{{ item.value.version }}/{{ dist_file }}"

If I use the above approach, I can define "tool" variable at a given roles//defaults/main.yml task level and it'll be available to dist_file / dist_url.

Also instead of using command: wget -q "...", I can use get_url (module) in Ansible.

- name: Download Java/JDK Versions
#  command: wget -q "{{ dist_url }}"
#    chdir="{{ common_download_dir }}"
#    creates="{{ common_download_dir }}/{{ dist_file }}"
  get_url: url="{{ dist_url }}" dest="{{ common_download_dir }}"
  become_user: "{{ build_user }}"
  with_dict: "{{ java_versions }}"

and make sure from the REMOTE machine you are able to ping the Artifactory server (check ping .... or /etc/resolv.conf file for search/nameserver entries) and if all looks good, then the solutions posted here will WORK!

I personally liked the solution for passing tool variable (at role/task) level (thus, negating the use of defining tool: at dictionary level) and not requiring to define individual level _dist_file/dist_url variables.

Comments

[Sepehr Nazari]

Bruce P's answer is an option best for when you have few different Java versions and use the variables frequently. My answer is best for when you have many Java versions but don't use the variables frequently. The only way to get a pretty solution if you have many Java versions and use the variables frequently is with Ansible 2.0

[AKS]

Thanks Bruce. I think this is my best bet. Again, when I'll have Ansible 2.0, I guess, it'll be more easy.

[AKS]

Thanks Sepehr. I agree with you.