How to delete lines starting with certain strings

command-line grep sed awk youtube-dl

38,074

Solution 1

sed -i '/\[youtube\]/d' /path/to/file

will delete lines, containing "[youtube]".

As one command you can combine patterns like

sed -i '/\[youtube\]\|\[ffmpeg\]\|\[avconv\]/d' /path/to/file

Or right from your command

youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 \
 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E | 
    sed '/\[youtube\]\|\[ffmpeg\]\|\[avconv\]/d' > output.txt

This will write the result to a file output.txt.

If you want to delete lines not just containing [youtube], but starting with [youtube], then add ^ to the pattern, like sed '/^\[youtube\]/d'.

But in your case it does not matter.

Solution 2

I suggest using grep -vE like so:

youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E | grep -vE '^\[(youtube|ffmpeg|avconv)\]'

From man grep:

-v, --invert-match
              Invert the sense of matching, to select non-matching lines.  (-v
              is specified by POSIX.)



-E, --extended-regexp
              Interpret  PATTERN  as  an extended regular expression (ERE, see
              below).  (-E is specified by POSIX.)

The -E flag is used to avoid escaping square brackets with slashes. Without -E flag you have to escape the square brackets with a backslash, like so grep -vE '\[youtube\]\|\[ffmpeg\]\|\[avconv\]' Edit:

Since you've requested awk,here's one with awk:

youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E | awk '{if ($0~/^\[youtube\]/||/^\[ffmpeg\]/||/^\[avconv\]/||/^WARNING/) next;print}'

Solution 3

Use grep -v as following:

youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E | grep -v '^[youtube]' | grep -v '^[ffmpeg]' | grep -v '^[avconv]'

Solution 4

Using Perl:

< inputfile perl -pe 's/^\[(youtube|ffmpeg|avconv)\].*$//' > outputfile

To parse the output of youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E directly, pipe its output to the command without redirecting the content of inputfile:

`youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E | perl -pe 's/^\[(youtube|ffmpeg|avconv)\].*$//' > outputfile`

< inputfile: redirects the content of inputfile to perl's stdin
> outputfile: redirects the content of perl's stdout to outputfile
-p: places a while (<>) { [...] } loop around the script and prints each processed line
-e: reads the script from the arguments

Perl script breakdown:

s: asserts to perform a substitution
/: starts the pattern
^: matches the start of the line
\[: matches a [ character
(: starts grouping the allowed strings
youtube: matches a youtube string
|: separates the second allowed string
ffmpeg: matches a ffmpeg string
|: separates the third allowed string
avconv: matches a avconv string
): stops grouping the allowed strings
\]: matches a ] character
.*: matches any number of any character
$: matches the end of the line
/: stops the pattern / starts the replacement string
/: stops the replacement string

View more solutions

38,074

potholiday

Updated on September 18, 2022

Comments

potholiday over 1 year

Is there a way to delete lines starting with certain strings.

I have this youtube-dl code

youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E

and its result is like this

[youtube:playlist] PL1C815DB73EC2678E: Downloading webpage
[download] Downloading playlist: Less than 1 minute
[youtube:playlist] playlist Less than 1 minute: Collected 4 video ids (downloading 4 of them)
[download] Downloading video 1 of 4
[youtube] KNLwsqzFfNg: Downloading webpage
[youtube] KNLwsqzFfNg: Extracting video information
[youtube] KNLwsqzFfNg: Downloading DASH manifest
[download] Destination: _1 min. - Amendes pour les particules du LHC-KNLwsqzFfNg.m4a

[download]   0.4% of 231.51KiB at  6.10KiB/s ETA 00:30
[download]   1.1% of 231.51KiB at 27.07KiB/s ETA 00:10
[download]   4.0% of 231.51KiB at 19.24KiB/s ETA 00:04
[download]   6.5% of 231.51KiB at 75.06KiB/s ETA 00:03
[download]  13.4% of 231.51KiB at 98.22KiB/s ETA 00:03
[download]  28.7% of 231.51KiB at 81.40KiB/s ETA 00:02
[download]  61.7% of 231.51KiB at 91.56KiB/s ETA 00:01
[download]  86.2% of 231.51KiB at 82.96KiB/s ETA 00:00
[download] 100.0% of 231.51KiB at 73.21KiB/s ETA 00:00
[download] 100% of 231.51KiB in 00:02
[ffmpeg] Correcting container in "_1 min. - Amendes pour les particules du LHC-KNLwsqzFfNg.m4a"
WARNING: Your copy of avconv is outdated, update avconv to version 10-0 or newer if you encounter any errors.
[avconv] Destination: _1 min. - Amendes pour les particules du LHC-KNLwsqzFfNg.mp3
WARNING: Your copy of avconv is outdated, update avconv to version 10-0 or newer if you encounter any errors.
Deleting original file _1 min. - Amendes pour les particules du LHC-KNLwsqzFfNg.m4a (pass -k to keep)
[download] Downloading video 2 of 4
[youtube] wTvXkMpJflk: Downloading webpage
[youtube] wTvXkMpJflk: Extracting video information
[youtube] wTvXkMpJflk: Downloading DASH manifest

I want to delete all lines starting with [youtube], [ffmpeg] and [avconv] and get like this

[youtube:playlist] PL1C815DB73EC2678E: Downloading webpage
[download] Downloading playlist: Less than 1 minute
[youtube:playlist] playlist Less than 1 minute: Collected 4 video ids (downloading 4 of them)
[download] Downloading video 1 of 4
[download] Destination: _1 min. - Amendes pour les particules du LHC-KNLwsqzFfNg.m4a

[download]   0.4% of 231.51KiB at  6.10KiB/s ETA 00:30
[download]   1.1% of 231.51KiB at 27.07KiB/s ETA 00:10
[download]   4.0% of 231.51KiB at 19.24KiB/s ETA 00:04
[download]   6.5% of 231.51KiB at 75.06KiB/s ETA 00:03
[download]  13.4% of 231.51KiB at 98.22KiB/s ETA 00:03
[download]  28.7% of 231.51KiB at 81.40KiB/s ETA 00:02
[download]  61.7% of 231.51KiB at 91.56KiB/s ETA 00:01
[download]  86.2% of 231.51KiB at 82.96KiB/s ETA 00:00
[download] 100.0% of 231.51KiB at 73.21KiB/s ETA 00:00
[download] 100% of 231.51KiB in 00:02
WARNING: Your copy of avconv is outdated, update avconv to version 10-0 or newer if you encounter any errors.
WARNING: Your copy of avconv is outdated, update avconv to version 10-0 or newer if you encounter any errors.
Deleting original file _1 min. - Amendes pour les particules du LHC-KNLwsqzFfNg.m4a (pass -k to keep)
[download] Downloading video 2 of 4
etc..
etc..
.
.

i tried this method but its showing error and it only suppose to delete [youtube]

youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E | sed '^/[youtube]/ d'

Pilot6 almost 9 years

Your pattern is wrong. It should be sed '/^\[youtube\]/d'.

Lekensteyn almost 9 years

The [ and ] should be escaped. With the -E option (extended regex option) you can avoid escaping (, ) and | in the command: grep -vE '\[(youtube|ffmpeg|avconv)\]'. As for the "starting requirement", ^ needs to be prepended.
Sergiy Kolodyazhnyy almost 9 years

@Lekensteyn good point. I'll add that
potholiday almost 9 years

I tried this sed -i '/\[youtube\]\|\[ffmpeg\]\|\[avconv\]/d' youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E and not working. may be because its not a file and its a download link
Pilot6 almost 9 years

Try with a file. It will work. Or you can do it right from the link using | pipe.
potholiday almost 9 years

@Serg not working. when executed it shows > and a blinking cursor. may be some syntax problem
Sergiy Kolodyazhnyy almost 9 years

@potholiday it was missing the single quote at the end. Fixed everything and added awk version. please review
potholiday almost 9 years

grep script doesn’t show any difference from the original code, no change at all. awk i tried like this youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E | awk '{if(/[youtube]/||/[ffmpeg]/) $0="";print}' it deletes all except warning lines
Sergiy Kolodyazhnyy almost 9 years

@potholiday you can add ||/WARNING/, too. like if(/[youtube]/||/ffmpeg/||/avconv/||/WARNING/)
potholiday almost 9 years

worked... the edited one worked
Sergiy Kolodyazhnyy almost 9 years

[ffmpeg] and WARNING are on separate lines, so you have to add that to if statement to delete that line
potholiday almost 9 years

@Serg awk command prints only the warning lines it deletes all other the lines . eg it deletes [download] line also
potholiday almost 9 years

and for me > output.txt is not necessary
Sergiy Kolodyazhnyy almost 9 years

@potholiday What about if ($0~/[youtube]/||/WARNING/||/[ffmpeg]/||/[avconv]/) ? does that make difference ?
Sergiy Kolodyazhnyy almost 9 years

@potholiday OK, fixed both grep and awk versions, now should be working as it should. My attention to details isn't at its best. Let me know how this works
kos almost 9 years

Now it seems to be working, however it leaves empty lines in place of the removed lines; change $0="" to next to fix this
Sergiy Kolodyazhnyy almost 9 years

you might want to edit the command to include pipe from the OP's youtube-dl command. Others might not understand. Otherwise, I've tested it with an input file with text from OP's post,seems to work, hence +1
kos almost 9 years

Thanks @Serg, I saw you updated yours as well so +1 on yours either
potholiday almost 9 years

@Serg gerp is working fine but awk has problem
potholiday almost 9 years

how to delete WARNING line also i tried like this \WARNING/d' but its not deleting.
Sergiy Kolodyazhnyy almost 9 years

@potholiday what exactly ?
potholiday almost 9 years

@Serg same problem awk only prints WARNING line.. any way your grep is working fine. But how to delete WARNING also. I tried this youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E | grep -vE '^\[(youtube|ffmpeg|avconv|WARNING)\]' but still it prints WARNING line also
Sergiy Kolodyazhnyy almost 9 years

@potholiday in that code you posted, WARNING is inside brackets [ ]. Try grep -vE '^\[(youtube|ffmpeg|avconv\]\|^WARNING .
Pilot6 almost 9 years

@potholiday You do not need to escape there, just '/^WARNING/d'
potholiday almost 9 years

tried this youtube-dl --extract-audio --audio-quality 0 --newline --audio-format mp3 https://www.youtube.com/playlist?list=PL1C815DB73EC2678E | grep -vE '^\[(youtube|ffmpeg|avconv\]\|^WARNING' but getting error
potholiday almost 9 years

sed '/\[youtube\]\|\[ffmpeg\]\|\[avconv\]|/^WARNING/d' is showing error any idea whats happening
Pilot6 almost 9 years

Do you see that | is also escaped with ``?
potholiday almost 9 years

showing this error sed: -e expression #1, char 40: unknown command: ^' `
Pilot6 almost 9 years

There was a typo. `sed '/[youtube]\|[ffmpeg]\|[avconv]\|^WA‌RNING/d'
potholiday almost 9 years

Let us continue this discussion in chat.
Karn Kumar about 4 years

it's really simple. sed -i '/youtube\|ffmpeg\|avconv\|WARNING/d' file_name you don't need multiple escapes