youtube-dl file download progress with zenity progress bar

Question

How to add youtube-dl file download progress percentage to zenity progress bar

sample code (just an example, not a working one)

#!/bin/sh
(
   progress=$(youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E)
per=$(awk '{print perc}' <<<$progress)
time=$(awk '{print time}' <<<$progress)
file_no=$(awk '{print file_no}' <<<$progress) #only for playlist, example=Downloading video 1 of 4 

echo "$per" ; sleep 1
echo "# $file_no \n Time Left: $time" ; sleep 1

) |
zenity --progress \
  --title="Download" \
  --text="Downloading..." \
  --percentage=0

if [ "$?" = -1 ] ; then
        zenity --error \
          --text="Download cancelled."
fi

i have used this code to get download progress

youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E

This is the out put

[youtube:playlist] PL1C815DB73EC2678E: Downloading webpage
[download] Downloading playlist: Less than 1 minute
[youtube:playlist] playlist Less than 1 minute: Collected 4 video ids (downloading 4 of them)
[download] Downloading video 1 of 4
[youtube] KNLwsqzFfNg: Downloading webpage
[youtube] KNLwsqzFfNg: Extracting video information
[youtube] KNLwsqzFfNg: Downloading DASH manifest
download] Destination: _1 min. - Amendes pour les particules du LHC-KNLwsqzFfNg.m4a

[download]   0.4% of 231.51KiB at  6.10KiB/s ETA 00:30
[download]   1.1% of 231.51KiB at 27.07KiB/s ETA 00:10
[download]   4.0% of 231.51KiB at 19.24KiB/s ETA 00:04
[download]   6.5% of 231.51KiB at 75.06KiB/s ETA 00:03
[download]  13.4% of 231.51KiB at 98.22KiB/s ETA 00:03
[download]  28.7% of 231.51KiB at 81.40KiB/s ETA 00:02
[download]  61.7% of 231.51KiB at 91.56KiB/s ETA 00:01
[download]  86.2% of 231.51KiB at 82.96KiB/s ETA 00:00
[download] 100.0% of 231.51KiB at 73.21KiB/s ETA 00:00
[download] 100% of 231.51KiB in 00:02
[ffmpeg] Correcting container in "_1 min. - Amendes pour les particules du LHC-KNLwsqzFfNg.m4a"
WARNING: Your copy of avconv is outdated, update avconv to version 10-0 or newer if you encounter any errors.
[avconv] Destination: _1 min. - Amendes pour les particules du LHC-KNLwsqzFfNg.mp3
WARNING: Your copy of avconv is outdated, update avconv to version 10-0 or newer if you encounter any errors.
Deleting original file _1 min. - Amendes pour les particules du LHC-KNLwsqzFfNg.m4a (pass -k to keep)
[download] Downloading video 2 of 4
[youtube] wTvXkMpJflk: Downloading webpage
[youtube] wTvXkMpJflk: Extracting video information
[youtube] wTvXkMpJflk: Downloading DASH manifest
etc..
etc..
.
.

and i want only

Downloading video 1 of 4 [download] Downloading video 2 of 4

as $files_no

FIRST FILE

file_no= Downloading video 1 of 4

per      time             rate
0.40%   00:30:00    6.10KiB/s
1.10%   00:10:00    27.07KiB/s
4.00%   00:04:00    19.24KiB/s
6.50%   00:03:00    75.06KiB/s
13.40%  00:03:00    98.22KiB/s
28.70%  00:02:00    81.40KiB/s
61.70%  00:01:00    91.56KiB/s
86.20%  00:00:00    82.96KiB/s
100.00% 00:00:00    231.51KiB/s

SECOND, THIRD...FILES

As separate variable $file, $per, $time i know we can use awk but for this complicated output how should i use it. if all parameters are not possible, can at least percentage and file_no be extracted.

Solution

Yes, it is possible. You need to

1. Make sure that output is unbuffered, that it is printed as soon as it's received. Pipes are buffered by default.
2. Parse the downloader's output so that only the percentages are printed;
3. Parse the output so that the file number is printed with a # at the beginning of the line. Zenity will automatically update the text of its dialog with lines starting with #.

Combining the above, and implementing a little regex magic, we get:

#!/bin/bash
youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 \
           https://www.youtube.com/playlist?list=PL1C815DB73EC2678E | 
 grep --line-buffered -oP '^\[download\].*?\K([0-9.]+\%|#\d+ of \d)' |
    zenity --progress \
  --title="Download" \
  --text="Downloading..." \
  --percentage=0 

Explanation

The --line-buffered option makes grep print its output immediately, turning off the default buffering. The -o makes it print only the matched portion of the line and the -P turns on Perl Compatible Regular Expressions.

The regex is a little complex, so let's break it down:

• ^\[download\] : matches lines that start with [download].
• .*? : 0 or more characters, but the ? makes it stop at the shortest possible match.
• \K : this is basically a lookbehind, it means "ignore anything matched so far".
• (...|...) : the | means OR. Therefore, (A|B) wil match either A or B.
• [0-9.]+\% : 1 or more numbers or . followed by a %. This prints the percentage.
• #\d+ of \d : a # followed by one or more digits, of and then one or more digits again. This matches the "Video X of Y" line.

Taken together, that grep command will print:

#1 of 4
0.1%
0.3%
0.8%
1.7%
3.4%
7.0%
14.0%
28.2%
56.5%
99.5%
100.0%
100%
#2 of 4
0.1%
0.3%
0.8%
1.6%
3.4%
6.9%
13.9%
27.8%
55.8%
[...]

etc, and that's precisely the output that zenity needs. Finally, you can make the whole thing more useful by implementing the ability to specify multiple URLs from the command line:

#!/bin/bash
for url in "$@"
do
  youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 \
           https://www.youtube.com/playlist?list=PL1C815DB73EC2678E | 
   grep --line-buffered -oP '^\[download\].*?\K([0-9.]+\%|#\d+ of \d)' |
    zenity --progress \
  --title="Download" \
  --text="Downloading..." \
  --percentage=0 
done

Then, you can call your script like this:

myscript.sh "http://url1.com" "http://url2.com" "http://urlN.com

Comments

[Wilf]

Try this question, you probably could adapt the script provided there for youtube-dl. There are varying ways you can use to get the latest progress indicator using grep, awk, sed, tail, etc. Note there is already a existing youtube-dl GUI program, though it probably does not have this functionality

[potholiday]

@Wilf I know YAD is even better than zenity but i want to use default programmes and codes available in Ubuntu itself

[terdon]

I don't understand what you're trying to do. Your awk commands won't return any output, you're printing undefined variables. WHat exactly do you want in each of those variables and what do you want passed to zenity?

[potholiday]

@terdon The code will not work !!, that just a sample code. awk is just a dummy command in the above code..I want to show an example script other than that it has no function.

[terdon]

OK, so, what exactly do you need extracted from the output? You seem to want to i) save things into variables and ii) pipe things to zenity. I cna't figure out what should go where.

[potholiday]

@terdon Yes, I want to extract percentage of file downloaded in youtube-dl and pipe it to percentage progress in zenity. and also if playlist is added extract current downloading file details and pipe it to the zenity progress text example =Downloading video 2 of 4 . I am really sorry for any confusion as english is not my mother tongue. If helps i followed this link for zenity progress bar help.gnome.org/users/zenity/stable/progress.html.en

[terdon]

You can't pipe both. Could you come into this chat room for a second and explain it there?

[potholiday]

k i will come to the chat

[potholiday]

it worked !! thanks for taking time and doing this script. Is it possible to pipe file no: ie. #1 of 4 to the zenity text which is currently Downloading... to some thing like this Downloading... #1 of 4

[potholiday]

Thanks for the regex script..still now I never understood the logic about regex and always felt a cumbersome job. Very nice explanation

[terdon]

@potholiday I don't think so. I tried a few thing but none worked. The best I could do was use the # feature that's already implemented by zenity. And you're welcome :)

[potholiday]

once more thank you and your script is more than enough for me now.

[Ciasto piekarz]

gives broken pipe error

[terdon]

@Ciastopiekarz sorry, but I'll need more information to help. Please ask a new question, show the exact script you used and the specific youtube URL. I am guessing one of the URLs you gave didn't work since that would explain the error, but I can't know without more information.