How to read a properties file into an associative array?

Question

I'd like to read the properties in a properties file into an associative array. How can I do it?

Specifics on what to parse: hash and equal signs. Everything else is a bonus.

Properties file content example:

# comment
a=value-a
b=http://prefix.suffix:8080/?key=value
c=password_with\\backslash-and=equals

I'd like this bash associative array to be constructed out of that file:

 declare -A props='(
  [a]="value-a"
  [b]="http://prefix.suffix:8080/?key=value"
  [c]="password_with\\backslash-and=equals" )'

(expected output of declare -p on that associative array, note that ${props[c]} contains only one backslash, "\\" is '\').

Solution 1

Here is howto in Bash4+

#!/usr/bin/env bash

declare -A properties

# Read with:
# IFS (Field Separator) =
# -d (Record separator) newline
# first field before separator as k (key)
# second field after separator and reminder of record as v (value)
while IFS='=' read -d $'\n' -r k v; do
  # Skip lines starting with sharp
  # or lines containing only space or empty lines
  [[ "$k" =~ ^([[:space:]]*|[[:space:]]*#.*)$ ]] && continue
  # Store key value into assoc array
  properties[$k]="$v"
  # stdin the properties file
done < file.properties

# display the array for testing
typeset -p properties

file.properties:

# comment
a=value-a
b=http://prefix.suffix:8080/?key=value
c=password_with\\backslash-and=equals

d e=the d e value
  # comment

The output of this script from the supplied data sample:

declare -A properties=(["d e"]="the d e value" [c]="password_with\\\\backslash-and=equals" [b]="http://prefix.suffix:8080/?key=value" [a]="value-a" )

Solution 2

Use a real parser like perl's Config::Properties module. I would do the whole script in perl, but if you have to use bash, you could do something like:

typeset -A props
while IFS= read -rd '' key && IFS= read -rd '' value; do
  props[$key]=$value
done < <(
  perl -MConfig::Properties -l0 -e '
   $p = Config::Properties->new();
   $p->load(STDIN);
   print for $p->properties' < file.properties
)

(also works with zsh).

Implementing a full parser in bash would be a lot of work and mean re-inventing the wheel. You can implement a good subset with a simple while read loop though as the read builtin expects an input format that is very similar to those properties files:

typeset -A props
while IFS=$':= \t' read key value; do
  [[ $key = [#!]* ]] || [[ $key = "" ]] || props[$key]=$value
done < file.properties

(also works with ksh93 and zsh, the two other Bourne-like shell supporting associative arrays).

That handles:

• prop = value
• prop: value
• prop value
• comments at the start of the line (! and # with optional leading blanks)
• backslash escaping (as in foo\:\:bar=value for keys containing delimiters or foo=\ bar or the password_with\\backslash-and=equals in your sample).
• line continuation with backslash

However, if we check against the specification

• That doesn't handle \n, \r, \uXXXX... sequences

• LF is the only recognised line delimiter (not CR nor CRLF).

• FF is not recognised as a whitespace (we can't just add it to $IFS as depending on the shell and version, \f will not necessarily be recognised as an IFS-whitespace character¹).

• for an input like foo: bar = , that stores bar in ${props[foo]} instead of bar = (foo: bar:baz: is OK though). That's only a problem when the value of the property contains one (unescaped) delimiter (: optionally surrounded by SPC/TAB characters, = optionally surrounded by SPC/TAB characters or sequence of one or more SPC/TAB characters) and it is at the end.

• it treats as comments lines that start with \! or \#. Only a problem for properties whose name starts with ! or #.

• in

    prop=1\
     2\
     3
  

we get 1 2 3 instead of 123: the leading spaces are not ignored in the continuation lines as they should be.

---

^{² IFS whitespace characters, per POSIX are the characters classified as [:space:] in the locale (which generally includes \f but doesn't have to) and that happen to be in $IFS though in ksh88 (on which the POSIX specification is based) and in most shells, that's still limited to SPC, TAB and NL. The only POSIX compliant shell in that regard I found was yash. ksh93 and bash (since 5.0) also include other whitespace (such as CR, FF, VT...), but limited to the single-byte ones (beware on some systems like Solaris, that includes the non-breaking-space which is single byte in some locales)}

Solution 3

For the most common subset of that dataformat, you can use a short function, using bash variable expansion and regexp matching.

Note: This expects lines to be in ^key = value$ format, or ^#.*$ and ^!.*$ for comments. Adapt the code, or pre-process your data otherwise

$ cat /tmp/propdata 
k1 = v1
# A comment
k2 = v2and some s=t=u=f=f
! Another comment
k3 = v3

$ unset DATA
$ declare -A DATA

$ props(){ while read line || [[ -n $line ]]; do
[[ "$line" =~ ^#|^! ]] && continue;
if [[ "${line% =*}" ]]; then DATA[${line% =*}]="${line#*= }" ; fi ;
done < $1 ; }

$ props /tmp/propdata

$ echo "${DATA[k3]}"
v3
$ echo "${DATA[k2]}"
v2and some s=t=u=f=f

Edit: Updated to trim the spaces around the "=" for key and value

Edit2: Filters comments now too.

Solution 4

declare -A properties
function readPopertyFile
{
    while read line || [[ -n $line ]]; do
        key=`echo $line | cut -s -d'=' -f1`
        if [ -n "$key" ]; then
            value=`echo $line | cut -d'=' -f2-`
            properties["$key"]="$value"
        fi
    done < $1
}

Usage:

readPopertyFile "file.properties"

Will read the properties into an associative array variable named properties.

* Works in bash. Don't know about other shells.

* Won't handle multi-line properties.

View more solutions

Comments

[Jeff Schaller]

does it need to handle comments (of the form ! and #) as well as the key/value formats: key=value, key = value, key:value, and key value?

[AlikElzin-kilaka]

@jeff - I don't know. Most of the properties files I saw use equals for separation and hash for comments.

[Jeff Schaller]

I try to ask questions that clarify the OP so that we can get good answers. If you link to an example properties file but only want to parse some of them, then you need to clarify that. In either case, I think you should import the linked properties file into your question so that we don't have to leave the site to determine the requirements.

[Stéphane Chazelas]

@JeffSchaller, Q&A are for everybody, not just the OP, The OP may only need to deal with a very limited set of files, but the next person having a similar requirement may not. It's a good idea for the OP to clarify his exact requirements, but IMO, not having them is not ground for closing the question, so I'm requesting it being reopened.

[Jeff Schaller]

Sorry if I was unclear, @StéphaneChazelas -- my main point was that AlikElzin-kilaka linked to a sample file but then "didn't know" what syntax they wanted to support. I don't personally care what syntax that is, just that this Question should be clear about it.

[Jeff Schaller]

Alik - is this question focused on supporting only the simple "key=value" and "#"-for-comment syntax? I'd be happy to reopen the question if we can get a clear direction on the expected syntax.

[AlikElzin-kilaka]

@StéphaneChazelas - it was a mistake. Sorry. Put you addition in. I also added more specifics.

[Emmanuel]

Using bash parameter expansion to set the variables will save you 2 sub shells, 2 echo, 2 pipes and two cut commands with key=${line%=*} and value=${line#*=}. that's less clear but much faster if the file is big.

[AlikElzin-kilaka]

@Emmanuel - Will ${line#*=} "eat" the whole value? Even when there are several = (equal) chars?

[Gilles 'SO- stop being evil']

Always double quote variable expansions. Your script breaks on lines containing spaces and wildcards due to the missing double quotes. But as Emmanuel advises you should use parameter expansion instead of echo | cut, it's easier to get it right and faster. ${line#*=} cuts off the part up to the first =, and ${line%%*=} (note double %) cuts off the part from the first =.

[Emmanuel]

I noted the cut -s that is followed by a test on empty strings. This is to deal with lines containing no "=". Parameter expansion will not help here but you can filter the input at the beginning of the script : grep = $1 | while read line. It adds one grep to the script but as it is outside the loop, that's ok.

[Emmanuel]

@AlikElzin-kilaka I assumed values would not contain an =, if thats the case use the %% (longest matching string) as advised by Giles.

[AlikElzin-kilaka]

Can't locate Config/Properties.pm in @INC (you may need to install the Config::Properties module). Missing perl module :(

[Stéphane Chazelas]

@AlikElzin-kilaka, hence the link to where to find it in my answer. You can do install Config::Properties in cpan.

[Alessio]

installing modules from CPAN isn't hard, but (depending on your unix or linux distribution) you may be able to install it even easier with your system's package manager. e.g. on debian apt-get install libconfig-properties-perl.

[Stéphane Chazelas]

Try on an input like a=+ * + or a\=b=c or a=b c (3 spaces in between b and c, which SE apparently won't let me insert in a comment), -nene=foo

[cherryhitech]

@cas I'm not able to find this package in Ubuntu 16.04 Is there any alternative for this in Xenial ? packages.ubuntu.com/xenial/perl

[Alessio]

there are dozens of perl config file parsers. Config::Simple (packaged as libconfig-simple-perl) will probably work with this file format. See packages.ubuntu.com/xenial/all/perl for a list of all packaged perl modules in xenial, and search that page for libconfig-.

[cherryhitech]

I searched over there and did not find the package which has Config::Properties. Currently, our code repo has packaged Properties.pm (due to previous environment constraint). I'm new to perl and trying to install the package in Ubuntu OS, rather than carrying having pm file in code repo. I could use other parsers, but, this needs lot of code refactoring. Please let me know which Ubuntu package has this file

[Alacritas]

This is in my opinion the best answer. Simple and straightforward, using pure bash (perl is great, but the question wanted bash; plus this is easier to read unless you're really familiar with perl). Plus is skips comments and empty lines, which is a nice perk.