Merging hashes in yaml conf files

July 31, 2009 by
Filed under: technology 

YAML is quite handy for writing configuration files. Primary advantage is that, it reads like text file. This works really well if your config file is flat(no hierarchy) and has no repetitions.
If your configurations file has repetitions then it makes sense to separate out those elements and reuse them. What I mean is this – let’s say you your config file looks like this :

development:
  input_location: common_input
  output_location: dev_location
  mail:
    smtp_server: your_server
    login: your_login
    password: top_secret
production:
  input_location: common_input
  output_location: dev_location
  mail:
    smtp_server: your_server
    login: your_login
    password: top_secret

Assuming above code in /tmp/test.yml here is how you can read in python and ruby
$cat readyml.py

#!/usr/bin/env python
from pprint import pprint as pp
#in debian need to install python-yaml
from yaml import load,load_all,dump
hash= load(open('/tmp/test.yml'))
pp(hash['development'])


$ cat readyml.rb

#!/usr/bin/env ruby
require 'pp'
hash= YAML::load(File.open('/tmp/test.yml').read)
pp hash['development']

here is a handy one liner for ruby version
$ ruby -rpp -e 'pp YAML::load(File.open("/tmp/a.yml"))["development"]' or you can try the same in irb or python console.

Note that in the above code snippet ,everything is other than output location is same in development and production part. This is where yml node identifier comes to rescue. Idea is simple have a set of default values and override them at different place.
You could pull it apart as follows:

defaults: &defaults
  input_location: common_input
  output_location: dev_location
  mail:
    sender_name: sender
    smtp_server: your_server
    login: your_login
    password: top_secret
development:
  <<: *defaults
production:
  <<: *defaults
  output_location: prod_location


$ ruby -rpp -e 'pp YAML::load(File.open("/tmp/a.yml"))["development"]["mail"]["login"]'
"your_login"
$

Great , it works(tm) !.
Arguably we traded some clarity for a bit of magic. Here is a small explanation : & , * and <<:  & which is  anchor tag can be understood as node identifier, * is node reference and <<: stands for hash merge.

For more details see either yaml specs or wikipedia
So far so good but there is a catch here, these hash merges are not recursive. What it means is this : let’s say you want to have different sender name for mail in two environments, you may be tempted to do the following:

defaults: &defaults
  input_location: common_input
  output_location: dev_location
  mail:
    sender_name: sender
    smtp_server: your_server
    login: your_login
    password: top_secret
development:
  <<: *defaults
  mail:
    sender_name: sender_dev
production:
  <<: *defaults
  output_location: prod_location
  mail:
    sender_name: sender_prod

Lets check

$ ruby -rpp -e 'pp YAML::load(File.open("/tmp/a.yml"))["development"]["mail"]["login"]'
nil
$

Oops, something went wrong, problem as mentioned above is that the hash merge is not recursive and while merging it replaced mail of default by mail of production which has only one key. Solution/work around is to unroll one more level:

common_settings: &common_settings
input_location: common_input
output_location: dev_location
mail_defaults: &mail_defaults
  sender_name: sender
  smtp_server: your_server
  login: your_login
  password: top_secret

defaults: &defaults
  <<: *common_settings
  mail:
    <<: *mail_defaults
development:
  <<: *defaults
production:
  <<: *defaults
  mail:
    <<: *mail_defaults
    sender_name: sender_prod

Lets check again

$ ruby -rpp -e 'pp YAML::load(File.open("/tmp/a.yml"))["development"]["mail"]["login"]'
"your_login"
$

Did you say you have one more level of nesting, well you can definitely unroll one more level, but then it becomes a mess. So, if you are not trying to write solution to towers of hanoi in a conf file, it is better to restucture conf file than digging into yaml or something else. But that is your call anyway.

Comments

12 Comments on Merging hashes in yaml conf files

  1. The Right Stuff on Sat, 30th Jan 2010 7:33 pm
  2. Rake, YAML and Inherited Build Configuration…

  3. Benjamin "balupton" Lupton on Sun, 19th Sep 2010 11:13 am
  4. Yeah annoying it doesn't merge recursively. I was able to make a workaround in my balphp library.

  5. søgemaskineoptimering on Sun, 27th Feb 2011 11:40 pm
  6. A person familiar with the deal said the two sides still are hashing out a name for a merged company. But this person said an idea being considered is the selection of a name that doesn't have the word “Deutsche” or the acronym “NYSE” in it. …

  7. Dating on Wed, 1st Jun 2011 8:15 am
  8. The full 453-page document also lists potential areas for
    consolidation, including privatizing trash pickup, merging the city and
    Wayne County health departments and partnering with Detroit Public Schools on
    recreation. The proposal did not include any …

  9. dogging on Wed, 1st Jun 2011 2:15 pm
  10. The full 453-page document also lists potential areas for
    consolidation, including privatizing trash pickup, merging the city and
    Wayne County health departments and partnering with Detroit Public Schools on
    recreation. The proposal did not include any …
     

  11. Ashley Madison on Wed, 12th Oct 2011 1:38 pm
  12. Very Good information…

    I have Digged this site for future and will keep a eye on your other posts….

  13. Free internet dating on Tue, 18th Oct 2011 4:10 pm
  14. Very Good informatio….

    I have Digged this site for future and will keep a eye on your other posts….

  15. Russian brides on Tue, 25th Oct 2011 1:01 pm
  16. Great article….

    I was just searching for something else on Bing….

  17. house electro music on Thu, 27th Sep 2012 9:48 am
  18. Great information, the links in your post has where
    very useful. I think people claim for every little thing these days and it is
    really no need to.I thing Submit Your Ideas for the Myrtle Public Pedestrian
    Plaza is nice preparation.I really like this post.I will bookmark this site.

     

  19. Austin on Sat, 8th Jun 2013 10:24 am
  20. pyaml can’t handle this kind of merge

  21. ????? on Mon, 10th Jun 2013 7:49 pm
  22. PyYAML 3.10 works like a charm

  23. dergachev on Sat, 26th Oct 2013 12:01 am
  24. Helpful, thanks! Here’s a gist I made to see this first hand. https://gist.github.com/dergachev/7159135

Tell me what you're thinking...
and oh, if you want a pic to show with your comment, go get a gravatar!





  • Recent Posts