Merging hashes in yaml conf files

July 31, 2009 by · 12 Comments
Filed under: technology 

YAML is quite handy for writing configuration files. Primary advantage is that, it reads like text file. This works really well if your config file is flat(no hierarchy) and has no repetitions.
If your configurations file has repetitions then it makes sense to separate out those elements and reuse them. What I mean is this – let’s say you your config file looks like this :

development:
  input_location: common_input
  output_location: dev_location
  mail:
    smtp_server: your_server
    login: your_login
    password: top_secret
production:
  input_location: common_input
  output_location: dev_location
  mail:
    smtp_server: your_server
    login: your_login
    password: top_secret

Assuming above code in /tmp/test.yml here is how you can read in python and ruby
$cat readyml.py

#!/usr/bin/env python
from pprint import pprint as pp
#in debian need to install python-yaml
from yaml import load,load_all,dump
hash= load(open('/tmp/test.yml'))
pp(hash['development'])


$ cat readyml.rb

#!/usr/bin/env ruby
require 'pp'
hash= YAML::load(File.open('/tmp/test.yml').read)
pp hash['development']

here is a handy one liner for ruby version
$ ruby -rpp -e 'pp YAML::load(File.open("/tmp/a.yml"))["development"]' or you can try the same in irb or python console.

Note that in the above code snippet ,everything is other than output location is same in development and production part. This is where yml node identifier comes to rescue. Idea is simple have a set of default values and override them at different place.
You could pull it apart as follows:

defaults: &defaults
  input_location: common_input
  output_location: dev_location
  mail:
    sender_name: sender
    smtp_server: your_server
    login: your_login
    password: top_secret
development:
  <<: *defaults
production:
  <<: *defaults
  output_location: prod_location


$ ruby -rpp -e 'pp YAML::load(File.open("/tmp/a.yml"))["development"]["mail"]["login"]'
"your_login"
$

Great , it works(tm) !.
Arguably we traded some clarity for a bit of magic. Here is a small explanation : & , * and <<:  & which is  anchor tag can be understood as node identifier, * is node reference and <<: stands for hash merge.

For more details see either yaml specs or wikipedia
So far so good but there is a catch here, these hash merges are not recursive. What it means is this : let’s say you want to have different sender name for mail in two environments, you may be tempted to do the following:

defaults: &defaults
  input_location: common_input
  output_location: dev_location
  mail:
    sender_name: sender
    smtp_server: your_server
    login: your_login
    password: top_secret
development:
  <<: *defaults
  mail:
    sender_name: sender_dev
production:
  <<: *defaults
  output_location: prod_location
  mail:
    sender_name: sender_prod

Lets check

$ ruby -rpp -e 'pp YAML::load(File.open("/tmp/a.yml"))["development"]["mail"]["login"]'
nil
$

Oops, something went wrong, problem as mentioned above is that the hash merge is not recursive and while merging it replaced mail of default by mail of production which has only one key. Solution/work around is to unroll one more level:

common_settings: &common_settings
input_location: common_input
output_location: dev_location
mail_defaults: &mail_defaults
  sender_name: sender
  smtp_server: your_server
  login: your_login
  password: top_secret

defaults: &defaults
  <<: *common_settings
  mail:
    <<: *mail_defaults
development:
  <<: *defaults
production:
  <<: *defaults
  mail:
    <<: *mail_defaults
    sender_name: sender_prod

Lets check again

$ ruby -rpp -e 'pp YAML::load(File.open("/tmp/a.yml"))["development"]["mail"]["login"]'
"your_login"
$

Did you say you have one more level of nesting, well you can definitely unroll one more level, but then it becomes a mess. So, if you are not trying to write solution to towers of hanoi in a conf file, it is better to restucture conf file than digging into yaml or something else. But that is your call anyway.

  • Recent Posts