Reading file content changes in .NET

Question

In Linux, a lot of IPC is done by appending to a file in 1 process and reading the new content from another process.

I want to do the above in Windows/.NET (Too messy to use normal IPC such as pipes). I'm appending to a file from a Python process, and I want to read the changes and ONLY the changes each time FileSystemWatcher reports an event. I do not want to read the entire file content into memory each time I'm looking for changes (the file will be huge)

Each append operation appends a row of data that starts with a unique incrementing counter (timestamp+key) and ends with a newline.

Solution 1

    using (FileStream fs = new FileStream
       (fileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
    {
        using (StreamReader sr = new StreamReader(fs))
        {
            while (someCondition)
            {
                while (!sr.EndOfStream)
                    ProcessLinr(sr.ReadLine());
                while (sr.EndOfStream)
                    Thread.Sleep(100);
                ProcessLinr(sr.ReadLine());            
            }
        }
    }

this will help you read only appended lines

Solution 2

You can store the offset of the last read operation and seek the file to that offset when you get a changed file notification. An example follows:

Main method:

public static void Main(string[] args)
{
    File.WriteAllLines("test.txt", new string[] { });

    new Thread(() => ReadFromFile()).Start();

    WriteToFile();
}

Read from file method:

private static void ReadFromFile()
{
    long offset = 0;

    FileSystemWatcher fsw = new FileSystemWatcher
    {
        Path = Environment.CurrentDirectory,
        Filter = "test.txt"
    };

    FileStream file = File.Open(
        "test.txt",
        FileMode.Open,
        FileAccess.Read,
        FileShare.Write);

    StreamReader reader = new StreamReader(file);
    while (true)
    {
        fsw.WaitForChanged(WatcherChangeTypes.Changed);

        file.Seek(offset, SeekOrigin.Begin);
        if (!reader.EndOfStream)
        {
            do
            {
                Console.WriteLine(reader.ReadLine());
            } while (!reader.EndOfStream);

            offset = file.Position;
        }
    }
}

Write to file method:

private static void WriteToFile()
{
    for (int i = 0; i < 100; i++)
    {
        FileStream writeFile = File.Open(
            "test.txt",
            FileMode.Append,
            FileAccess.Write,
            FileShare.Read);

        using (FileStream file = writeFile)
        {
            using (StreamWriter sw = new StreamWriter(file))
            {
                sw.WriteLine(i);
                Thread.Sleep(100);
            }
        }
    }
}

Comments

[jameszhao00]

I'm a bit of a noob when it comes to file operations. Does seek load everything into memory or does it only load from offset to offset+length?

[João Angelo]

To my knowledge the seek operation will not load any data and just position the file stream at the specified position. In this case the position will be where the new data starts.

[MaYaN]

A GREAT solution, short and lethal!