Load YAML nested with Jinja2 in Python

Question

I have a YAML file (all.yaml) that looks like:

...
var1: val1
var2: val2
var3: {{var1}}-{{var2}}.txt
...

If I load it in Python like this:

import yaml

f = open('all.yaml')
dataMap = yaml.safe_load(f)
f.close()
print(dataMap["var3"])

the output is {{var1}}-{{var2}}.txt and not val1-val2.txt.

Is it possible to replace the nested vars with the value?

I tried to load it with:

import jinja2
templateLoader = jinja2.FileSystemLoader( searchpath="/path/to/dir" )
templateEnv = jinja2.Environment( loader=templateLoader )
TEMPLATE_FILE = "all.yaml"
template = templateEnv.get_template( TEMPLATE_FILE )

The exception is no longer thrown, now I am stuck and have to research how to proceed.

Solution 1

First define an Undefined class and load yaml to get known values. Then load it again and render with known values.

#!/usr/bin/env python

import yaml
from jinja2 import Template, Undefined

str1 = '''var1: val1
var2: val2
var3: {{var1}}-{{var2}}.txt
'''

class NullUndefined(Undefined):
  def __getattr__(self, key):
    return ''

t = Template(str1, undefined=NullUndefined)
c = yaml.safe_load(t.render())

print t.render(c)

Run it:

$ ./test.py
var1: val1
var2: val2
var3: val1-val2.txt

Solution 2

Here is one possible solution:

1. Parse your YAML document with the yaml module
2. Iterate over the keys in your YAML document, treating each value as a Jinja2 template to which you pass in the keys of the YAML document as parameters.

For example:

import yaml
from jinja2 import Template

with open('sample.yml') as fd:
    data = yaml.load(fd)

for k, v in data.items():
    t = Template(v)
    data[k] = t.render(**data)

print yaml.safe_dump(data, default_flow_style=False)

This will work fine with your particular example, but wouldn't do anything useful for, say, nested data structures (in fact, it would probably just blow up).

Comments

[larsks]

You can absolutely combine the YAML module and the Jinja2 module to do something like what the OP is asking. Ansible is one example of a tool that performs Jinja2 template processing on YAML values. So sure, you can't do this with the yaml module by itself, but I don't think that's what the OP is asking.

[Sandro Koch]

When the YAML file contains something like val: 1.5 it will throw a AttributeError: 'float' object has no attribute 'iter_fields'

[larsks]

Oh, absolutely. This is an example, not a robust solution! :)

[larsks]

Hey, -1 person, any comments on how this could be improved? I know it's an older one but I'm happy to fix it up. Cheers!

[dreftymac]

@larsks You are correct. Another example tool is cookiecutter

[Anthon]

Of course you can always replace something that is not valid YAML with something that is valid YAML and then load it. However the OP states that his first example (all.yaml) is YAML, and it is not. You first have to expand that jinja2 template and then parse the expanded result hoping it by then is replaced by something the YAML parser understands.

[dreftymac]

That is an entirely accurate statement. An unprocessed Jinja template that just happens to look similar to YAML syntax is not the same thing as well-formed YAML. A processed Jinja template is by no means guaranteed to produce well-formed YAML. One challenge is communicating this circumstance to potential YAML users who have yet to use YAML (let alone develop modules for it), while minimizing the chance they will be confused or intimidated away.

[dreftymac]

Note: This can be avoided by quoting the placeholder so that YAML interprets it as a plain scalar value (string). Change {{variable}} to "{{variable}}".