Modifying or deleting a line from a text file the low-level way?

Question

I'm working with a Text File in Delphi, and I don't wish to use the method of loading/saving with a string list. I intend to maintain an open filestream where I read and write my data there, keeping massive amounts of data on the hard disk instead of in the memory. I have the simple concept of writing new lines to a text file and reading them, but when it comes to modifying and deleting them, I cannot find any good resources.

Each line in this file contains a name, and equals sign, and the rest is data. For example, SOMEUNIQUENAME=SomeStringValue. I intend to keep a file open for a period of time inside of a thread. This thread performs incoming requests to either get, set, or delete certain fields of data. I use WriteLn and ReadLn in a loop, evaluating EOF. Below is an example of how I read the data:

FFile = TextFile;

...

function TFileWrapper.ReadData(const Name: String): String;
var
  S: String; //Temporary line to be parsed
  N: String; //Temporary name of field
begin
  Result:= '';
  Reset(FFile);
  while not EOF(FFile) do begin
    ReadLn(FFile, S);
    N:= UpperCase(Copy(S, 1, Pos('=', S)-1));
    if N = UpperCase(Name) then begin
      Delete(S, 1, Pos('=', S));
      Result:= S;
      Break;
    end;
  end;
end;

...and then I trigger an event which informs sender of result. The requests are inside of a queue, which is sort of a message pump for these requests. The thread simply processes the next request in the queue repeatedly, similar to how typical applications work.

I have procedures ready to be able to write and delete these fields, but I don't know what I have to do to actually perform the action on the file.

procedure TFileWrapper.WriteData(const Name, Value: String);
var
  S: String; //Temporary line to be parsed
  N: String; //Temporary name of field
begin
  Result:= '';
  Reset(FFile);
  while not EOF(FFile) do begin
    ReadLn(FFile, S);
    N:= UpperCase(Copy(S, 1, Pos('=', S)-1));
    if N = UpperCase(Name) then begin
      //How to re-write this line?
      Break;
    end;
  end;
end;

procedure TFileWrapper.DeleteData(const Name: String);
var
  S: String; //Temporary line to be parsed
  N: String; //Temporary name of field
begin
  Result:= '';
  Reset(FFile);
  while not EOF(FFile) do begin
    ReadLn(FFile, S);
    N:= UpperCase(Copy(S, 1, Pos('=', S)-1));
    if N = UpperCase(Name) then begin
      //How to delete this line?
      Break;
    end;
  end;
end;

In the end, I need to avoid loading the entire file into the memory to be able to accomplish this.

Solution 1

I find this an interesting question, so I made a small console app.

I used 3 methods:

• TStringList
• Streamreader/StreamWriter
• Text file

All methods are timed and repeated 100 times with a text file of 10kb in size and a text file 1Mb in size. Here is the program:

program Project16;

{$APPTYPE CONSOLE}

uses
  SysUtils, Classes, StrUtils, Diagnostics, IOUtils;

procedure DeleteLine(StrList: TStringList; SearchPattern: String);

var
  Index : Integer;

begin
 for Index := 0 to StrList.Count-1 do
  begin
   if ContainsText(StrList[Index], SearchPattern) then
    begin
     StrList.Delete(Index);
     Break;
    end;
  end;
end;

procedure DeleteLineWithStringList(Filename : string; SearchPattern : String);

var StrList : TStringList;

begin
 StrList := TStringList.Create;
 try
  StrList.LoadFromFile(Filename);
  DeleteLine(StrList, SearchPattern);
  // don't overwrite our input file so we can test
  StrList.SaveToFile(TPath.ChangeExtension(Filename, '.new'));
 finally
  StrList.Free;
 end;
end;

procedure DeleteLineWithStreamReaderAndWriter(Filename : string; SearchPattern : String);

var
  Reader    : TStreamReader;
  Writer    : TStreamWriter;
  Line      : String;
  DoSearch  : Boolean;
  DoWrite   : Boolean;

begin
 Reader := TStreamReader.Create(Filename);
 Writer := TStreamWriter.Create(TPath.ChangeExtension(Filename, '.new'));
 try
  DoSearch := True;
  DoWrite := True;
  while Reader.Peek >= 0 do
   begin
    Line := Reader.ReadLine;
    if DoSearch then
     begin
      DoSearch := not ContainsText(Line, SearchPattern);
      DoWrite := DoSearch;
     end;
    if DoWrite then
     Writer.WriteLine(Line)
    else
     DoWrite := True;
   end;
 finally
  Reader.Free;
  Writer.Free;
 end;
end;

procedure DeleteLineWithTextFile(Filename : string; SearchPattern : String);

var
 InFile    : TextFile;
 OutFile   : TextFile;
 Line      : String;
 DoSearch  : Boolean;
 DoWrite   : Boolean;


begin
 AssignFile(InFile, Filename);
 AssignFile(OutFile, TPath.ChangeExtension(Filename, '.new'));
 Reset(InFile);
 Rewrite(OutFile);
 try
  DoSearch := True;
  DoWrite := True;
  while not EOF(InFile) do
   begin
    Readln(InFile, Line);
    if DoSearch then
     begin
      DoSearch := not ContainsText(Line, SearchPattern);
      DoWrite := DoSearch;
     end;
    if DoWrite then
     Writeln(OutFile, Line)
    else
     DoWrite := True;
   end;
 finally
  CloseFile(InFile);
  CloseFile(OutFile);
 end;
end;

procedure TimeDeleteLineWithStreamReaderAndWriter(Iterations : Integer);

var
  Count : Integer;
  Sw    : TStopWatch;

begin
 Writeln(Format('Delete line with stream reader/writer - file 10kb, %d iterations', [Iterations]));
 Sw := TStopwatch.StartNew;
 for Count := 1 to Iterations do
  DeleteLineWithStreamReaderAndWriter('c:\temp\text10kb.txt', 'thislinewillbedeleted=');
 Sw.Stop;
 Writeln(Format('Elapsed time : %d milliseconds', [Sw.ElapsedMilliseconds]));
 Writeln(Format('Delete line with stream reader/writer - file 1Mb, %d iterations', [Iterations]));
 Sw := TStopwatch.StartNew;
 for Count := 1 to Iterations do
  DeleteLineWithStreamReaderAndWriter('c:\temp\text1Mb.txt', 'thislinewillbedeleted=');
 Sw.Stop;
 Writeln(Format('Elapsed time : %d milliseconds', [Sw.ElapsedMilliseconds]));
end;

procedure TimeDeleteLineWithStringList(Iterations : Integer);

var
  Count : Integer;
  Sw    : TStopWatch;

begin
 Writeln(Format('Delete line with TStringlist - file 10kb, %d iterations', [Iterations]));
 Sw := TStopwatch.StartNew;
 for Count := 1 to Iterations do
  DeleteLineWithStringList('c:\temp\text10kb.txt', 'thislinewillbedeleted=');
 Sw.Stop;
 Writeln(Format('Elapsed time : %d milliseconds', [Sw.ElapsedMilliseconds]));
 Writeln(Format('Delete line with TStringlist - file 1Mb, %d iterations', [Iterations]));
 Sw := TStopwatch.StartNew;
 for Count := 1 to Iterations do
  DeleteLineWithStringList('c:\temp\text1Mb.txt', 'thislinewillbedeleted=');
 Sw.Stop;
 Writeln(Format('Elapsed time : %d milliseconds', [Sw.ElapsedMilliseconds]));
end;

procedure TimeDeleteLineWithTextFile(Iterations : Integer);

var
  Count : Integer;
  Sw    : TStopWatch;

begin
 Writeln(Format('Delete line with text file - file 10kb, %d iterations', [Iterations]));
 Sw := TStopwatch.StartNew;
 for Count := 1 to Iterations do
  DeleteLineWithTextFile('c:\temp\text10kb.txt', 'thislinewillbedeleted=');
 Sw.Stop;
 Writeln(Format('Elapsed time : %d milliseconds', [Sw.ElapsedMilliseconds]));
 Writeln(Format('Delete line with text file - file 1Mb, %d iterations', [Iterations]));
 Sw := TStopwatch.StartNew;
 for Count := 1 to Iterations do
  DeleteLineWithTextFile('c:\temp\text1Mb.txt', 'thislinewillbedeleted=');
 Sw.Stop;
 Writeln(Format('Elapsed time : %d milliseconds', [Sw.ElapsedMilliseconds]));
end;

begin
  try
    TimeDeleteLineWithStringList(100);
    TimeDeleteLineWithStreamReaderAndWriter(100);
    TimeDeleteLineWithTextFile(100);
    Writeln('Press ENTER to quit');
    Readln;
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.

Output:

Delete line with TStringlist - file 10kb, 100 iterations
Elapsed time : 188 milliseconds
Delete line with TStringlist - file 1Mb, 100 iterations
Elapsed time : 5137 milliseconds
Delete line with stream reader/writer - file 10kb, 100 iterations
Elapsed time : 456 milliseconds
Delete line with stream reader/writer - file 1Mb, 100 iterations
Elapsed time : 22382 milliseconds
Delete line with text file - file 10kb, 100 iterations
Elapsed time : 250 milliseconds
Delete line with text file - file 1Mb, 100 iterations
Elapsed time : 9656 milliseconds
Press ENTER to quit

As you can see is TStringList the winner here. Since you are not able to use TStringList, TextFile is not a bad choice after all...

P.S. : this code omits the part where you have to delete the inputfile and rename the outputfile to the original filename

Solution 2

Without loading the entire file into a container like TStringList, your only option is to:

• Open the file for input
• Open a separate copy for output
• Start a loop
• Read the content line by line from the input file
• Write the content out line by line to the output file until you reach the line you want to change/delete
• Break the loop
• Read the input line from the input file
• Write the changed line (or skip writing the line you want to delete) to the output file
• Start a new loop
• Read the remainder of the input content, line by line
• Write the rest of that input to the output file, line by line
• Break the loop
• Close the files

So to answer your specific questions:

if N = UpperCase(Name) then begin
  //How to re-write this line?
  Break;
end;

WriteLn the new output to the second (output) file.

if N = UpperCase(Name) then begin
  //How to delete this line?
  Break;
end;

Just skip the WriteLn that outputs the indicated line to the second (output) file.

Your artificial limitation of "I don't want to use TStringList" simply complicates the task for you, when you can simply:

• Load the original file into TStringList using LoadFromFile
• Locate the line you want to modify, either by index, iteration, or IndexOf()
• Modify the line by changing it directly, or deleting it from the TStringList
• Write the entire content out to the original file using TStringList.SaveToFile

The only reasons I've found to not use TStringList to perform these kinds of operations have been that the file size exceeds the capacity of a TStringList (never happened) or when dealing with a file that is text but isn't really "line" oriented (for instance, EDI files that are typically one very long single line of text, or XML files that may not contain line feeds and therefore are also one very long single line of text). Even in the case of EDI or XML, though, it's quite frequently to load them into a TStringList, make the conversion to line-based format (inserting line breaks or whatever), and do the retrieval from the stringlist.

Solution 3

Basically, you can't do what you want to do if you treat the files as simple text files. Such files can be read (from the beginning only) or written to (either from the start, thus creating a new file) or from the end (appending to an existing file). They are not random access files.

On the other hand, you might want to consider defining a file of type string: each record in the file would be a string, and you can access this file in a random fashion. The problem then becomes in knowing which record to access for which string.

A third possibility is using INI files which are more structured and sound like a better bet for your purposes. Apart from the section header, they are a series of strings, key=value, and can be accessed on the basis of the key.

Comments

[Jerry Dodge]

Yes, Ini files were my first direction, but I wanted to skip a bit and not require any header section, but I might as well implement it anyway.

[Jerry Dodge]

That does indeed sound like the only solution, I'm just pondering how SQL Server is able to maintain its data feed in the file stream, and any complex database for that matter. I'm just trying to avoid having to use up the memory many times over for massive amounts of data.

[jachguate]

@Jerry SQL Server or any modern database engine does not use text files to store data. The data is usually stored in pages, and each page contains none, one or many records. When you update a record, if, for example, a varchar column does not fit in the same space as the old record, it is moved to the end of the page or to a different page. There are books about how this works. Read for example Understanding SQL Server Storage Structures or search google for "(sql server/oracle/firebird/db2) internal storage"

[jachguate]

Other little thing: by definition a text file is a sequential file, because the nature of different length lines, and that's traditionally have a different treatment from a random access file. In a text file, without traversing the entire file, you have no method to predict at which file position is stored any particular line, and you can't really change any line size directly in the file. In a random access file, you can go to any record, page or byte position, because all the pages, records or bytes have the same length, so you can do some arithmetic to get that position.

[Ken White]

One thing to note, though: IIRC, TextFile is not Unicode-aware, so it won't work with anything but ASCII text.

[Ken White]

Jerry, as @jachguate said, SQL Server (or any other RDBMS, for that matter) doesn't use anything even remotely resembling text files. They use complex structures of binary (very often compressed) data that are maintained in pages. Comparing the two is like comparing a tricycle (the text file) to a Porsche (SQL Server) (Oracle would be the Ferrari <g>). In other words, you can't.

[whosrdaddy]

@KenWhite: That is not correct. I tested some files in UTF-8 format and it works like a charm

[Ken White]

What Delphi version? This wasn't the case in D2009-XE, because it was a common problem on the EMBT forums (which is where I got my info from - I haven't used TextFile in a decade because it's simply old Pascal methodology, and there are better ways now).

[whosrdaddy]

@KenWhite Delphi XE here. Me neither, I just find it surprising that TextFile performs relatively fast, even if it's old :)

[David Heffernan]

Oracle is more like a Delorean in fact. A huge rip off owned by an evil crook.

[Remy Lebeau]

@KenWhite: in the answer you describe, there is a little room for optimization. If the desired line is the last line in the file, you can simply truncate the source file to the appropriate offset, you don't need to copy the file. If the line is in the front or middle of the file, then the second loop should read/write the data in chunk instead of in lines, as you don't have to worry about the actual contents. Lastly, use buffered reading, or a memory mapping, for faster access to the source file.

[Ken White]

@Remy: Great. :-) The question was about using TextFile - how exactly do you do all those things (especially reading/writing in chunks, ignoring lines, and using memory mapping)? I don't see any of those things in the help file for TextFile. (Note the question didn't ask about writing anything new, but specifically about using TextFile and not a container like TStringList.)

[Remy Lebeau]

@KenWhite: I was commenting on your list of required steps, not the actually implementation of them. Obviously, you can't some things with TextFile, you have to ditch it in favor offor more direct I/O, like FileRead() or TFileStream.

[Ken White]

@Remy: My point is that I was referring specifically to using TextFile, which I clearly indicated in my answer; pointing out ways to optimize it via other methods wasn't quite fair. :-) I don't comment on your answers about how to do things in Indy by pointing out easier ways to accomplish them in ICS or Synapse. <g>