How to read a large (1 GB) txt file in .NET?

Question

I have a 1 GB text file which I need to read line by line. What is the best and fastest way to do this?

private void ReadTxtFile()
{            
    string filePath = string.Empty;
    filePath = openFileDialog1.FileName;
    if (string.IsNullOrEmpty(filePath))
    {
        using (StreamReader sr = new StreamReader(filePath))
        {
            String line;
            while ((line = sr.ReadLine()) != null)
            {
                FormatData(line);                        
            }
        }
    }
}

In FormatData() I check the starting word of line which must be matched with a word and based on that increment an integer variable.

void FormatData(string line)
{
    if (line.StartWith(word))
    {
        globalIntVariable++;
    }
}

Solution 1

If you are using .NET 4.0, try MemoryMappedFile which is a designed class for this scenario.

You can use StreamReader.ReadLine otherwise.

Solution 2

Using StreamReader is probably the way to since you don't want the whole file in memory at once. MemoryMappedFile is more for random access than sequential reading (it's ten times as fast for sequential reading and memory mapping is ten times as fast for random access).

You might also try creating your streamreader from a filestream with FileOptions set to SequentialScan (see FileOptions Enumeration), but I doubt it will make much of a difference.

There are however ways to make your example more effective, since you do your formatting in the same loop as reading. You're wasting clockcycles, so if you want even more performance, it would be better with a multithreaded asynchronous solution where one thread reads data and another formats it as it becomes available. Checkout BlockingColletion that might fit your needs:

Blocking Collection and the Producer-Consumer Problem

If you want the fastest possible performance, in my experience the only way is to read in as large a chunk of binary data sequentially and deserialize it into text in parallel, but the code starts to get complicated at that point.

Solution 3

You can use LINQ:

int result = File.ReadLines(filePath).Count(line => line.StartsWith(word));

File.ReadLines returns an IEnumerable<String> that lazily reads each line from the file without loading the whole file into memory.

Enumerable.Count counts the lines that start with the word.

If you are calling this from an UI thread, use a BackgroundWorker.

Solution 4

Probably to read it line by line.

You should rather not try to force it into memory by reading to end and then processing.

Solution 5

StreamReader.ReadLine should work fine. Let the framework choose the buffering, unless you know by profiling you can do better.

View more solutions

Comments

[Jeevan Bhatt]

StreamReader.ReadLine is fine for small file but when i tried it for large file then it is very slow some time not responding.

[Jeevan Bhatt]

@Mathew: posted code look at it, lines length are not fix some time line contain only 200 word and some time it will be 2000 or greater that it.

[Matthew Flaschen]

2000 isn't a huge amount. That's only 20 KB, if we're talking English words. However, you still may want to call the FileStream constructor manually, specifying the buffer size. I also think FormatData may actually be the issue. That method doesn't keep all the data in memory, does it?

[Homde]

If you're only doing sequential reading you're better of using StreamReader than MemoryMappedFile since it's much faster. Memory mapping is better for random access.

[Jeevan Bhatt]

@Matthew: i have commented FormatData() and it is still slow, no much significant difference with and without FormatData().

[Homde]

Furthermore you probably can't create a ViewAccesor spanning the entire 1 gb so you have to manage that as well as parsing out the linebreaks. FileStreams are 10 times as fast as Memory-Mapped files for sequential reading.

[cspolton]

+1 The limiting factor is going to be the speed of the reads from disk, so to improve performance have different threads reading vs processing the lines.

[Audrius]

@Jeevan can you define "slow"? If you read [small file] in n time, then big file will be read in n * [big file]/[small file]. Maybe you are experiencing what's expected?

[dodgy_coder]

@ konrad - agreed, great comment, FYI there is a bit of a discussion of this in O'Reilly's excellent "C# 4.0 in a Nutshell", page 569. For sequential I/O and a 1GB file size, then MemoryMappedFiles are definitely overkill and may slow things down.

[Royi Namir]

@TimSchmelter do you really expect to load 1 gb file to memory ?memorymappedfile has a lot of usages... i dont think this is one of them...

[Royi Namir]

@dodgy_coder i have tis book also , it doesnt say nothing about 1GB file. - the sample there is 1 million which is MB. the only thing mentiones is sequential vs random access.

[Tim Schmelter]

@RoyiNamir: MemoryMappedFile allow to read views of parts of extremly large files. You don't need to create a view from the whole file at once. So it's very scalable since you can define the portions yourself(f.e. 100MB).msdn.microsoft.com/en-us/library/dd997372.aspx

[dodgy_coder]

@RoyiNamir whether the book (C# 4.0 in a Nutshell) has an example of exactly 1GB in size is irrelevant. There's actually a title on page 569 called "Memory Mapped Files and Random File I/O" I'm looking at it now. Quoted: "Rule of thumb: FileStreams are 10 times faster than MemoryMappedFiles for sequential I/O. MemoryMappedFiles are 10 times faster than FileStreams for random I/O". TL;DR Use the right tool for the right job.

[Tim Schmelter]

@dodgy_coder: I'd be cautitous with such generalizations. Although i agree with your last sentence, but you should better measure it yourself.

[Kiquenet]

any full source code sample -using in real application in production environment, not msdn samples - about it ?

[Ctrl S]

"It will return the bellow text:" What text?

[Ozkan]

@Homde the last part you wrote is actually what StreamReader does internally, so why bother?