More complex inheritance in YAML?

Question

YAML has inheritance. The most clear example I have ever seen is here: http://blog.101ideas.cz/posts/dry-your-yaml-files.html

I need something more complex: I need to override object's object's property. Here is an example:

database: &default
  server:
    ip: 192.168.1.5
    port: 2000
  db_name: test
  user: 
    name: root
    password: root

# database foo differs from default by only its port and user password
foo_database:
  <<: *default
  server:
    port: 2001
  db_name: foo
  user:
    password: foo_root

I want to get this result:

foo_database.server.ip -> 192.168.1.5
foo_database.server.port -> 2001
foo_database.db_name -> foo
foo_database.user.name -> root
foo_database.user.password -> foo_root

But if you declare like this, you will get these properties incorrect (according to expected values):

foo_database.server.ip -> will be None
foo_database.user.name -> will be None

because new "server" object has only "port" property and it overrides whole old "server" object.

How do I get the kind of inheritance which I want to achieve?

Edit

Here is my exact intention with a working code in LiveScript:

config = 
  default: 
    ip: \192.168.1.5
    port: 2000
    name: \root 
    password: \root 
    db:
      name: \default
      location: \LA

  foo-database:~ -> @default `merge` do 
    ip: \11.11.11.11
    db:
      name: \my-foo 

  bar-database:~ -> @foo-database `merge` do 
    password: \1234 
    db:
      location: \SF

config.default 
# => {"ip":"192.168.1.5","port":2000,"name":"root","password":"root","db":{"name":"default","location":"LA"}}
config.foo-database  
# => {"ip":"11.11.11.11","port":2000,"name":"root","password":"root","db":{"name":"my-foo","location":"LA"}}
config.bar-database  
# => {"ip":"11.11.11.11","port":2000,"name":"root","password":"1234","db":{"name":"my-foo","location":"SF"}}

Solution 1

Unfortunately, you can't get the kind of "inheritance" you want to achieve because YAML's "inheritance" is more like a form of "merging hashes".

Expanding out your configuration at the point you use the *default alias, you have:

foo_database:
  server:
    ip: 192.168.1.5
    port: 2000
  db_name: test
  user: 
    name: root
    password: root

If you use hashes with the same keys afterwards, they will completely overwrite the hashes declared earlier, leaving you with (excuse the formatting):

foo_database:

  server:
    ip: 192.168.1.5
    port: 2000
  db_name: test
  user: 
   name: root
   password: root  

  server:
    port: 2001
  db_name: foo
  user:
    password: foo_root

So, in your case, it would seem that since the config is not exactly the same, DRYing up your configuration using anchors and aliases probably isn't the right approach.

More references on this issue below:

• Rake, YAML and Inherited Build Configuration
• Merging hashes in yaml conf files

Edit

If you really wanted to, I think you could reconfigure your YAML as below to get exactly what you want, but in your case, I would say the extra obfuscation isn't worth it:

server_defaults: &server_defaults
  ip: 192.168.1.5
  port: 2000

user_defaults: &user_defaults
  name: root
  password: root

database: &default
  server:
    <<: *server_defaults
  db_name: test
  user: 
    <<: *user_defaults

foo_database:
  <<: *default
  server:
    <<: *server_defaults
    port: 2001
  db_name: foo
  user:
    <<: *user_defaults
    password: foo_root

Solution 2

How about this? Use multiple anchors.

database: &default
  server: &server
    ip: 192.168.1.5
    port: 2000
  db_name: test
  user: &user
    name: root
    password: root

foo_database:
  <<: *default
  server:
    << : *server
    port: 2001
  db_name: foo
  user:
    << : *user
    password: foo_root

It's just a little extra work, and slightly harder to read than if what you wanted were built into YAML as you suggested (I thought it would work that way too). But overall not bad.

Solution 3

For this sort of problems, I have created a tool: jq-front. By using yq + jq-front, you can achieve it by slightly modifying your input.

in.yaml:

$local: 
  database_default:
    server:
      ip: 192.168.1.5
      port: 2000
    db_name: test
    user: 
      name: root
      password: root

# database foo differs from default by only its port and user password
foo_database:
  $extends: [ database_default ]
  server:
    port: 2001
  db_name: foo
  user:
    password: foo_root

And you can process this file by a following command line.

$ yq . -j in.yaml | jq-front  | yq . -y

And you will get following output that you wanted.

foo_database:
  server:
    ip: 192.168.1.5
    port: 2001
  db_name: foo
  user:
    name: root
    password: foo_root

NOTE: jq-front is very slow. On my machine the command took 2.5s, which did not matter to me too much since system configuration can be read once and only the converted file is used by the rest of my program.

NOTE: If you use docker + bash, it's lot easier to install jq-front by docker. You only need to add following function to your .bashrc or a file that is sourced by it.

function jq-front() {
  docker run --rm -i \
    -v /:/var/lib/jf \
    -e JF_PATH_BASE="/var/lib/jf" \
    -e JF_PATH="${JF_PATH}" \
    -e JF_DEBUG=${JF_DEBUG:-disabled} \
    -e JF_CWD="$(pwd)" \
    dakusui/jq-front:"${JF_DOCKER_TAG:-latest}" "${@}"
}

Comments

[ceremcem]

We are saying more or less the same thing. "Workaround" would work technically but it would not work "practically" because goal of YAML is being human readable. I gave a trivial example. Let's think that we have an object which has about 100 lines and 1 to 6 sub objects. If we need to change 1-2 objects only where one of them is in the deepest node, everywhere would be filled up with anchors. Anchor naming would be more problematic. This is a need. I asked this question because I need to know if I am wrong about approach or not. If not, I will try to implement that kind of inheritance.

[Paul Fioravanti]

I agree that YAML should be human readable, and hence I do not recommend using the solution in my edit (I only wrote it in case you were very adamant that you wanted results in exactly the way you specified in the question). Personally, I have only ever used anchors and aliases in YAML to refactor out any repeated code. I have never aliased repeated code and then overridden some of it (like in the edit above), as I think it makes the code harder to read. If you want to keep things human readable, in your case it will probably mean keeping some of the repetition as well.

[ceremcem]

Thank you for your conscientious answers. Yes, you are right, repeated code would be more clear in these conditions. But now I'm introducing a new inheritance operator (no no, don't stop me! :P) as "<<<: *anchor" which means "this object will inherit the object pointed by the anchor and none of the inherited object's object could be deleted, only overridden if explicitly present". Yes. Now I'm logically using this operator (I don't care what the parser says). When I have a free time, I will offer this operator for adding to YAML specification. Thank you Paul :)

[Paul Fioravanti]

You're welcome, and good luck. If you end up configuring something interesting, please share it with an edit to your question :-)

[Tyler Collier]

What I didn't think about is how you said you have 6 levels deep. This then is kind of annoying. Still better I think than not having any benefit from inheritance.

[Matthias Kuhn]

@ceremcem did you ever (partially) complete the <<<: *anchor implementation?

[ceremcem]

@MatthiasKuhn I started a configuration tool totally scratch called config-md a while ago, but I didn't continue development since I use LiveScript heavily in novadays and it makes handling configuration stuff very easy. In which language would you need to use this?

[ceremcem]

The config-md is already written in Python. If you have a chance to switch from YAML to another configuration tool, we may make config-md continued...