bash - sed query to edit yaml file

Question

I have a config.yaml file which contains among other values the following list of kafka brokers which I want to remove from the config using a bash script.

kafka.brokers:
    - "node003"
    - "node004"

I am doing this currently by invoking vi from inside the script using the command:

vi $CONF_BENCHMARK/config.yaml -c ":%s/kafka.brokers:\(\n\s*-\s".*"\)*/kafka.brokers:/g" -c ':wq!'

I understand that sed is a more appropriate tool to accomplish the same task but when I try to translate the above regex to sed, it does not work.

sed -i -e "s/kafka.brokers:\(\n\s*-\s".*"\)*/kafka.brokers:/g" $CONF_BENCHMARK/config.yaml

I am doing something wrong ?

Solution 1

As other have pointed out, you will need to be explicit to get sed working with multiply lines.

The real answer is to use AWK a beautiful answer is provided by karakfa. But for the educational purpose I will provide an sed answer:

sed  '
  /kafka.brokers/ {
    :a
    $be
    N
    /\n[[:space:]]*-[[:space:]]"[^\n]*"[^\n]*$/ba
    s/\n.*\(\n\)/\1/
    P;D
    :e
    s/\n.*//
  }
' input

Basically sed will keep append lines to the pattern space from when kafka.brokers up until \n[[:space:]]*-[[:space:]]"[^\n]*"[^\n]*$ is not matches.

This will leave the pattern space with one trailing line it in, i.e:

kafka.brokers:\n    - "node003"\n    - "node004"\nother stuff$

Replacing everything \n.*\(\n\) with a newline leaves the following pattern space:

kafka.brokers:\nother stuff$

P;D will print the first line from the pattern space and then restart the cycle with the remaning pattern space. Making the input support:

kafka.brokers:
    - "node003"
    - "node004"
kafka.brokers:
    - "node005"
more_input

Solution 2

awk to the rescue!

sed is line based, this should work...

$ awk 's{if(/\s*-\s*"[^"]*"/) next; else s=0} /kafka.brokers:/{s=1}1' file

Explanation

if(/\s*-\s*"[^"]*"/) next if pattern matches skip to next line
s{if(/\s... check pattern if only s is set
/kafka.brokers:/{s=1} when header seen set s
1 shorthand for print lines (if not skipped)
s{... else s=0} if s was set but pattern not found, reset s

Solution 3

Your Vim pattern matches across multiple lines, but sed works line-by-line. (That is, it first tries to match your pattern against kafka.brokers: and fails, then it tries to match - "node003", and so on.) Your instinct to use something other than Vim was right, but sed probably isn't the best tool for the job here.

This answer addresses the problem of matching multi-line patterns with sed in more detail.

My personal recommendation would be to use a scripting language like Python or Perl to deal with complicated pattern-matching. You can run a Python command with python -c <command>, for instance, just like you did with Vim, or you could write a small Python script that you call from your Bash script. It's a little more complicated than a sed one-liner, but it will probably save you a lot of debugging and make your script easier to maintain and modify.

Comments

[jaywalker]

Can you please explain how this works ? I might have to modify it a bit before I can use it.

[jaywalker]

I tried perl as perl -pi -e "s/kafka.brokers:\(\n\s*-\s".*"\)*/kafka.brokers:/g" config.yaml but that does not seem to work either. I hope python works for me.

[jaywalker]

@andlrc I am trying to replace in-place in the file config.yaml as my code in the comment above shows. Running your suggestion results Can't open perl script "s/kafka....": No such file or directory

[jaywalker]

Btw I tried your solution on the example provided above and it does not work.

[Andreas Louv]

@HaseebJaved perl -p0e '...' will slurp the whole file at one go. Use -pi -0e to do inplace edit.

[karakfa]

it doesn't work is not very descriptive... Nothing printed? Nothing filtered? Deleted the input file?

[jaywalker]

It printed the contents as it is i.e it did not make any changes.

[x-yuri]

Explanation for why 1 prints the line.