.NET C# - Random access in text files - no easy way?

Question

I've got a text file that contains several 'records' inside of it. Each record contains a name and a collection of numbers as data.

I'm trying to build a class that will read through the file, present only the names of all the records, and then allow the user to select which record data he/she wants.

The first time I go through the file, I only read header names, but I can keep track of the 'position' in the file where the header is. I need random access to the text file to seek to the beginning of each record after a user asks for it.

I have to do it this way because the file is too large to be read in completely in memory (1GB+) with the other memory demands of the application.

I've tried using the .NET StreamReader class to accomplish this (which provides very easy to use 'ReadLine' functionality, but there is no way to capture the true position of the file (the position in the BaseStream property is skewed due to the buffer the class uses).

Is there no easy way to do this in .NET?

Solution 1

There are some good answers provided, but I couldn't find some source code that would work in my very simplistic case. Here it is, with the hope that it'll save someone else the hour that I spent searching around.

The "very simplistic case" that I refer to is: the text encoding is fixed-width, and the line ending characters are the same throughout the file. This code works well in my case (where I'm parsing a log file, and I sometime have to seek ahead in the file, and then come back. I implemented just enough to do what I needed to do (ex: only one constructor, and only override ReadLine()), so most likely you'll need to add code... but I think it's a reasonable starting point.

public class PositionableStreamReader : StreamReader
{
    public PositionableStreamReader(string path)
        :base(path)
        {}

    private int myLineEndingCharacterLength = Environment.NewLine.Length;
    public int LineEndingCharacterLength
    {
        get { return myLineEndingCharacterLength; }
        set { myLineEndingCharacterLength = value; }
    }

    public override string ReadLine()
    {
        string line = base.ReadLine();
        if (null != line)
            myStreamPosition += line.Length + myLineEndingCharacterLength;
        return line;
    }

    private long myStreamPosition = 0;
    public long Position
    {
        get { return myStreamPosition; }
        set
        {
            myStreamPosition = value;
            this.BaseStream.Position = value;
            this.DiscardBufferedData();
        }
    }
}

Here's an example of how to use the PositionableStreamReader:

PositionableStreamReader sr = new PositionableStreamReader("somepath.txt");

// read some lines
while (something)
    sr.ReadLine();

// bookmark the current position
long streamPosition = sr.Position;

// read some lines
while (something)
    sr.ReadLine();

// go back to the bookmarked position
sr.Position = streamPosition;

// read some lines
while (something)
    sr.ReadLine();

Solution 2

FileStream has the seek() method.

Solution 3

You can use a System.IO.FileStream instead of StreamReader. If you know exactly, what file contains ( the encoding for example ), you can do all operation like with StreamReader.

Solution 4

If you're flexible with how the data file is written and don't mind it being a little less text editor-friendly, you could write your records with a BinaryWriter:

using (BinaryWriter writer = 
    new BinaryWriter(File.Open("data.txt", FileMode.Create)))
{
    writer.Write("one,1,1,1,1");
    writer.Write("two,2,2,2,2");
    writer.Write("three,3,3,3,3");
}

Then, initially reading each record is simple because you can use the BinaryReader's ReadString method:

using (BinaryReader reader = new BinaryReader(File.OpenRead("data.txt")))
{
    string line = null;
    long position = reader.BaseStream.Position;
    while (reader.PeekChar() > -1)
    {
        line = reader.ReadString();

        //parse the name out of the line here...

        Console.WriteLine("{0},{1}", position, line);
        position = reader.BaseStream.Position;
    }
}

The BinaryReader isn't buffered so you get the proper position to store and use later. The only hassle is parsing the name out of the line, which you may have to do with a StreamReader anyway.

Solution 5

Is the encoding a fixed-size one (e.g. ASCII or UCS-2)? If so, you could keep track of the character index (based on the number of characters you've seen) and find the binary index based on that.

Otherwise, no - you'd basically need to write your own StreamReader implementation which lets you peek at the binary index. It's a shame that StreamReader doesn't implement this, I agree.

View more solutions

Comments

[Jon Skeet]

That's not useful when we don't know where to seek to.

[Roger Lipscombe]

If the file's more than 1GB in size, and you're running on 32-bit, you'll probably run out of address space, even if Windows swaps its little heart out.

[Powerlord]

Maybe we're using different definitions of random access. I (as well as Jason apparently) take it to mean a file of records with a specific size in bytes, thus the start of a record is (recnum - 1) * recsize

[Mike Burton]

More importantly, the OP suggests that they can record the stream indices at which individual records begin, so knowing where to seek to is a solved problem in this instance.

[Roland Jee]

@Jon: "The first time I go through the file, I only read header names, but I can keep track of the 'position' in the file where the header is. I need random access to the text file to seek to the beginning of each record after a user asks for it." Sounds like we know where to seek to.

[Kcats]

"the position in the BaseStream property is skewed due to the buffer the class uses". Sounds like we don't know where to seek to.

[mhand]

This is very old but you could use file stream to manually loop through lines, or even pass the file stream to a stream reader and use the reader to go through lines but use the file stream to get the offset (stream.Position). Then later you can use the file stream to seek and then use the reader to read those lines. The trick is to use both. You may need to create a new reader after seeking, not sure of the behavior.

[mhand]

If you have complete control of the file you could write an index at the start of the file as well.