Parse HTML using shell
Solution 1
awk -F '[<>]' '/<td / { gsub(/<b>/, ""); sub(/ .*/, "", $3); print $3 } ' file
Output:
54
1
0
0
Another:
awk -F '[<>]' '
/<td><b>Total<\/b><\/td>/ {
while (getline > 0 && /<td /) {
gsub(/<b>/, ""); sub(/ .*/, "", $3)
print $3
}
exit
}' file
Solution 2
awk is not an HTML parser. Use xpath or even xslt for that. xmllint is a commandline tool which is able to execute XPath queries and xsltproc can be used to perform XSL transformations. Both tools belong to the package libxml2-utils.
Also you can use a programming language which is able to parse HTML
Solution 3
$ awk -F'<td[^>]*>(<b>)?|(</?b>)?</td>' '$2~/[0-9]/{print $2+0}' file
54
1
0
0
Solution 4
You really should to use some real HTML parser for this job, like:
perl -Mojo -0777 -nlE 'say [split(/\s/, $_->all_text)]->[0] for x($_)->find("td[align=right]")->each'
prints:
54
1
0
0
But for this you need to have perl, and installed Mojolicious package.
(it is easy to install with:)
curl -L get.mojolicio.us | sh
Solution 5
BSD/GNU grep/ripgrep
For simple extracting, you can use grep, for example:
-
Your example using
grep:$ egrep -o "[0-9][^<]\?\+" file.html 54 1 0 (0/0) 0and using
ripgrep:$ rg -o ">([^>]+)<" -r '$1' <file.html | tail +2 54 1 0 (0/0) 0 -
Extracting outer html of H1:
$ curl -s http://example.com/ | egrep -o '<h1>.*</h1>' <h1>Example Domain</h1>
Other examples:
-
Extracting the body:
$ curl -s http://example.com/ | xargs | egrep -o '<body>.*</body>' <body> <div> <h1>Example Domain</h1> ...Instead of
xargsyou can also usetr '\n' ' '. For multiple tags, see: Text between two tags.
If you're dealing with large datasets, consider using ripgrep which has similar syntax, but it's a way faster since it's written in Rust.
Lenny
Updated on April 05, 2021Comments
-
Lenny about 3 years
I have a HTML with lots of data and part I am interested in:
<tr valign=top> <td><b>Total</b></td> <td align=right><b>54</b></td> <td align=right><b>1</b></td> <td align=right>0 (0/0)</td> <td align=right><b>0</b></td> </tr>I try to use
awkwhich now is:awk -F "</*b>|</td>" '/<[b]>.*[0-9]/ {print $1, $2, $3 }' "index.html"but what I want is to have:
54 1 0 0Right now I am getting:
'<td align=right> 54' '<td align=right> 1' '<td align=right> 0'Any suggestions?