Fast read/write from file in delphi

Question

I am loading a file into a array in binary form this seems to take a while is there a better faster more efficent way to do this. i am using a similar method for writing back to the file.

procedure openfile(fname:string);
var
    myfile: file;
    filesizevalue,i:integer;
begin
  assignfile(myfile,fname);
  filesizevalue:=GetFileSize(fname); //my method
  SetLength(dataarray, filesizevalue);
  i:=0;
  Reset(myFile, 1);
  while not Eof(myFile) do
    begin
      BlockRead(myfile,dataarray[i], 1);
      i:=i+1;
    end;
  CloseFile(myfile);
end;

Solution 1

You generally shouldn't read files byte for byte. Use BlockRead with a larger value (512 or 1024 often are best) and use its return value to find out how many bytes were read.

If the size isn't too large (and your use of SetLength seems to support this), you can also use one BlockRead call reading the complete file at once. So, modifying your approach, this would be:

AssignFile(myfile,fname);
filesizevalue := GetFileSize(fname);
Reset(myFile, 1);
SetLength(dataarray, filesizevalue);
BlockRead(myFile, dataarray[0], filesizevalue);
CloseFile(myfile);

Perhaps you could also change the procedure to a boolean function named OpenAndReadFile and return false if the file couldn't be opened or read.

Solution 2

If your really want to read a binary file fast, let windows worry about buffering ;-) by using Memory Mapped Files. Using this you can simple map a file to a memory location an read like it's an array.

Your function would become:

procedure openfile(fname:string);
var
    InputFile: TMappedFile;
begin
  InputFile := TMappedFile.Create;
  try
    InputFile.MapFile(fname);
    SetLength(dataarray, InputFile.Size);
    Move(PByteArray(InputFile.Content)[0], Result[0], InputFile.Size);
  finally
    InputFile.Free;
  end;
end;

But I would suggest not using the global variable dataarray, but either pass it as a var in the parameter, or use a function which returns the resulting array.

procedure ReadBytesFromFile(const AFileName : String; var ADestination : TByteArray);
var
    InputFile : TMappedFile;
begin
  InputFile := TMappedFile.Create;
  try
    InputFile.MapFile(AFileName);
    SetLength(ADestination, InputFile.Size);
    Move(PByteArray(InputFile.Content)[0], ADestination[0], InputFile.Size);
  finally
    InputFile.Free;
  end;
end;

The TMappedFile is from my article Fast reading of files using Memory Mapping, this article also contains an example of how to use it for more "advanced" binary files.

Solution 3

It depends on the file format. If it consists of several identical records, you can decide to create a file of that record type.

For example:

type
  TMyRecord = record
    fieldA: integer;

    ..
  end;
  TMyFile = file of TMyRecord;

  const
    cBufLen = 100 * sizeof(TMyRecord);
  var
    file: TMyFile;
    i : Integer;

  begin
    AssignFile(file, filename);
    Reset(file);
    i := 0;
    try
      while not Eof(file) do begin
        BlockRead(file, dataarray[i], cBufLen);
        Inc(i, cBufLen);
      end;
    finally
      CloseFile(file);
    end;
  end;

Solution 4

If it's a long enough file that reading it this way takes a noticeable amount of time, I'd use a stream instead. The block read will be a lot faster, and there's no loops to worry about. Something like this:

procedure openfile(fname:string);
var
    myfile: TFileStream;
    filesizevalue:integer;
begin
  filesizevalue:=GetFileSize(fname); //my method
  SetLength(dataarray, filesizevalue);
  myFile := TFileStream.Create(fname);
  try
    myFile.seek(0, soFromBeginning);
    myFile.ReadBuffer(dataarray[0], filesizevalue);
  finally
     myFile.free;
  end;
end;

It appears from your code that your record size is 1 byte long. If not, then change the read line to:

  myFile.ReadBuffer(dataarray[0], filesizevalue * SIZE);

or something similar.

View more solutions

Comments

[Blair Houghton]

im sure you just made a typo but this does not work with i replaced as 1 and filesizevalue

[Dijar]

Yes, that's a typo. It should be filesizevalue, also be aware that your array should be an array of byte.

[Dijar]

Fixed now. Seems it has to be dataarray[0], too.

[RobS]

Good answer. I think you need to use the packed keyword when you're writing to files to be sure that you'll be easily able to retrieve the contents in future versions of delphi or from other languages as byte-alignment of the fields could be different.

[maaartinus]

Have you ever measured the performance? I guess, memory mapped files should win hands down, but I'd like to see some figures before I start to rewrite any code.