Edit first line of large text file

Question

I have a huge text file, far too big for the whole thing to be paged into memory. All I need to do with this text file is edit the first line (its a CSV file and I need to alter the titles).

Is there a simple way I can do this in bash?

Solution 1

You can use less to see what you want to edit and use sed to make the changes. This way you edit without loading the entire file.

Another way is to split the file, edit and join again:

split -b 10000k <file>

and to join:

cat xa* > <file>

Solution 2

If your modification changes the length of the line, the whole file needs to be re-written, see for example this discussion on SO. You should probably consider saving the data to a database.

Keeping that in mind, you can stream edit the file with sed. To replace the first line, do something like this (GNU sed):

< oldfile sed '1c\new_heading' > newfile

Comments

[J Slick]

Is there some reason you can't just use a text editor?

[lynks]

The file is too large, its over over 20 times larger than my maximum virtual memory on this machine.

[J Slick]

fair enough. I was under the impression that good text editors could deal with very large files because they only load as much as they need to into memory, but after reading some of the other questions it seems that most of them have problems with them.

[laurent]

@dangph True... may be odd but a few old and obsolete "text editors" (or so called) like edlin for ms-dos or ed for Unix didn't appear to have this problem but where only line editors... RAM at these times was a scarce and precious thing and a file size in Gigabytes was something impossible to believe! ;)

[laurent]

To let things clear ;): I do NOT miss ed!!! but it would be a good fit in this case (not sure it could handle a file of this size...)

[atroon]

upvote for sed.

[lynks]

Fantastic, split and join was exactly what I was looking for, thanks.

[laurent]

sure, sed is better as it can search/replace the whole file easily but if he only needs to change the first line, split is not bad and faster too.

[lynks]

sed would have taken hours to run over the whole file (which is just under half a TB), the changes were only on the first line, splitting it off seems sensible.