Using sed to match a pattern and deleting from the line to the end of the file

Question

I'm trying to match a pattern from piped input and/or a file, and then remove from the matched lines to the end of the file, inclusive. I've looked everywhere, but can't seem to find an expression that would fit my needs.

The following expression allows me to remove to the beginning of the stream, including the matched pattern:

sed -e '1,/Files:/d'

Given some sample data:

Blah blah blah
Foobar foo foo
Files:
 somefiles.tar.gz 1 2 3
 somefiles.tar.gz 1 2 3

-----THIS STUFF IS USELESS-----
BLEH BLEH BLEH
BLEH BLEH BLEH

Running the above expression produces:

Files:
 somefiles.tar.gz 1 2 3
 somefiles.tar.gz 1 2 3

-----THIS STUFF IS USELESS-----
BLEH BLEH BLEH
BLEH BLEH BLEH

I would like to achieve a similar effect, but in the opposite direction. Using the output from the previous expression, I want to remove from -----THIS STUFF IS USELESS----- to the end of the file, inclusive. It should produce (after going through the first expression):

Files:
 somefiles.tar.gz 1 2 3
 somefiles.tar.gz 1 2 3

I'm also open to using any other tools, as long as it is available on any other POSIX system and does not use version specific (e.g. GNU-specific) options.

The actual text can be found here: http://pastebin.com/CYBbJ3qr Note the change from -----THIS STUFF IS USELESS----- to -----BEGIN PGP SIGNATURE-----.

Solution 1

why not

 sed '/^-----THIS STUFF IS USELESS-----$/,$d' file

In a range expression like you have used, ',$' will specify "to the end of the file"

1 is first line in file, 
$ is last line in file.

output

Files:
 somefiles.tar.gz 1 2 3
 somefiles.tar.gz 1 2 3

With GNU sed, you can do

sed '/^-----THIS STUFF IS USELESS-----$/Q' file

where Q is similar to q quit command, but doesn't print the matching line

Solution 2

Instead of trying to figure out how to express what what you don't want, just print what you DO want:

awk -v RS= '/Files:/' file

EDIT: Given your modified input:

awk '/^Files:$/{f=1} f; /^$/{f=0}' file

or:

awk '/^Files:$/{f=1} f; /^-----THIS STUFF IS USELESS-----$/{f=0}' file

if you prefer.

You can also use either of these:

awk '/^Files:$/,/^-----THIS STUFF IS USELESS-----$/' file
sed '/^Files:$/,/^-----THIS STUFF IS USELESS-----$/' file

but they are hard to extend later.

Solution 3

sed -e '/^-----THIS STUFF IS USELESS-----$/,$ d'

Solution 4

Dirty utility knife grep version:

cat your_output.txt | grep -B 99999999 "THIS STUFF IS USELESS" | grep -v "THIS STUFF IS USELESS"

View more solutions

Comments

[Firas Dib]

You need to present more information about how the data you want to remove looks like and how the data you want to preserve looks like.

[kdubs]

I agree, show the exact starting data and ending data

[Albert H]

Apologies, my question may have been somewhat confusing. I've edited it to add some more specifics.

[ennox]

No more data was required (not your fault). The solution is simply: sed '/^-----THIS STUFF IS USELESS-----$/,$d' file

[gniourf_gniourf]

And a useless use of cat :-(

[mVChr]

Indeed, dirtier than I even imagined!

[Albert H]

How would I put that in sed form?

[Albert H]

Doesn't seem to work, I'm afraid. If you want the full text to try out, see: pastebin.com/CYBbJ3qr Just replace "THIS STUFF IS USELESS" with "-----BEGIN PGP SIGNATURE-----" (or for your case, just "BEGIN PGP SIGNATURE").

[Albert H]

This kinda works, but it does not exclude the lines above. It includes all of the lines above Files:. (My example above is not accurate, as there are no blank lines until the "THIS STUFF IS USELESS" part.) If you want the full text to play around with, see: pastebin.com/CYBbJ3qr Just replace "THIS STUFF IS USELESS" with "-----BEGIN PGP SIGNATURE-----" to test (though your answer does not require this change).

[Albert H]

This worked... and I understood how the expression worked, at least for a non-sed user. Thanks!

[N mol]

I am using sed -e '/<!--#content end--></div>/,$d' out.txt but it gives error saying : sed: -e expression #1, char 24: extra characters after command Thanks in advance.

[shellter]

because your target expression has a / char in it, you need to use a different reg-exp delimiter and the front and end, i.e. sed '@<!--#content end--></div>@,$' file. Some sed's require that if you don't use / then that you esacpe that char, i.e. `sed '\@.....@' file (2nd one NOT escaped). You should post this sort of thing as a question here. Good luck.

[shellter]

@Nmol : see reply above.

[Klaatu von Schlacker]

sed (GNU sed) 4.2.2 requires a space between the terminating $d, making it $ d

[shellter]

@KlaatuvonSchlacker : Interesting. Seems like a step backwards from the modern seds that don't require ;s at every possible place in a bracketed block (among other places). Thanks for the update! Good luck to all.

[Katie]

I thought you needed -e for sed to use an expression

[shellter]

@KatieS not usually. There are rare cases where having two (or more code fragments) on the cmd-line with b (branch) instructions can benefit from this, but if you're writing sed code with bs then it should be saved in a file (IMHO ;-) . And of course, you can always try it and see what happens. However, it does have the benefit of self-documentation, making the intent of the coding author explicit. The original O'Reilly books for sed never mentioned it, so I never used it. ;-) Good luck to all.