Read all contents of memory mapped file or Memory Mapped View Accessor without knowing the size of it

Question

I need something similar to ReadToEnd or ReadAllBytes to read all of the contents of the MemoryMappedFile using the MappedViewAccessor if I don't know the size of it, how can I do it?

I have searched for it, I have seen this question, but it is not the thing I am looking for:

How can I quickly read bytes from a memory mapped file in .NET?

Edit:

There is a problem, the (int)stream.Length is not giving me the correct length, it rather gives the size of the internal buffer used! I need to refresh this question because it is very pressing.

Solution 1

Rather use the Stream:

public static Byte[] ReadMMFAllBytes(string fileName)
{
    using (var mmf = MemoryMappedFile.OpenExisting(fileName))
    {
        using (var stream = mmf.CreateViewStream())
        {
            using (BinaryReader binReader = new BinaryReader(stream))
            {
                return binReader.ReadBytes((int)stream.Length);
            }
        }
    }
}

Solution 2

This is difficult to answer since there are still many details of your application that you haven't specified, but I think both Guffa's and Amer's answers are still partially correct:

• A MemoryMappedFile is more memory than file; it is a sequence of 4Kb pages in memory. So, stream.Length will in fact give you all of the bytes (there is no "internal buffer size"), but it might give you more bytes than you expect since the size will always be rounded up to a 4Kb boundary.
• The "file" semantic comes from associating the MemoryMappedFile to a real filesystem file. Assuming that the process which creates the file always adjusts the file size, then you can get the precise size of the file via the fileSystem.

If all of the above would fit your application, then the following should work:

    static byte[] ReadMemoryMappedFile(string fileName)
    {
        long length = new FileInfo(fileName).Length;
        using (var stream = File.Open(fileName, FileMode.OpenOrCreate, FileAccess.Read, FileShare.ReadWrite))
        {
            using (var mmf = MemoryMappedFile.CreateFromFile(stream, null, length, MemoryMappedFileAccess.Read, null, HandleInheritability.Inheritable, false))
            {
                using (var viewStream = mmf.CreateViewStream(0, length, MemoryMappedFileAccess.Read))
                {
                    using (BinaryReader binReader = new BinaryReader(viewStream))
                    {
                        var result = binReader.ReadBytes((int)length);
                        return result;
                    }
                }
            }
        }
    }

To write the data, you can use this:

    private static void WriteData(string fileName, byte[] data)
    {
        using (var stream = File.Open(fileName, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.ReadWrite))
        {
            using (var mmf = MemoryMappedFile.CreateFromFile(stream, null, data.Length, MemoryMappedFileAccess.ReadWrite, null, HandleInheritability.Inheritable, true))
            {
                using (var view = mmf.CreateViewAccessor())
                {
                    view.WriteArray(0, data, 0, data.Length);
                }
            }

            stream.SetLength(data.Length);  // Make sure the file is the correct length, in case the data got smaller.
        }
    }

But, by the time you do all of the above you might do just as well to use the file directly and avoid the memory mapping. If mapping it to the filesystem isn't acceptable, then Guffa's answer of encoding the length (or an end marker) in the data itself is probably best.

Solution 3

You can't do that.

A view accessor is created with a minimum size of a system page, which means that it may be larger than the actual file. A view stream is just a stream form of an accessor, so it will also give the same behaviour.

"views are provided in units of system pages, and the size of the view is rounded up to the next system page size"

http://msdn.microsoft.com/en-us/library/dd267577.aspx

The accessor will gladly read and write outside the actual file without throwing an exception. When reading, any bytes outside the file will just be zero. When writing, the bytes written outside the file are just ignored.

To read the file from a memory mapped file with the exact size of the original file, you have to already know that size.

Solution 4

Stream created by MemoryMappedFile has a length aligned to file system page size (usually 4096). You have to get the file size from somewhere else. If it is memory mapped file you could use that code:

byte[] ReadAllMemoryMappedFileBytes(string filePath)
{
    var fileInfo = new FileInfo(filePath);
    using (var file = MemoryMappedFile.CreateFromFile(filePath, FileMode.Open))
    using (var stream = file.CreateViewAccessor())
    {
        byte[] bytes = new byte[fileInfo.Length];
        stream.ReadArray(0, bytes, 0, bytes.Length);
        return bytes;
    }
}

Solution 5

Use FileInfo class to get length as shown below

using System.Data;
using System.IO;
using System.IO.Compression;
using System.IO.MemoryMappedFiles;

// ...

public void WriteToMemoryMap(DataSet ds, string key, string fileName)
{
    var bytes = CompressData(ds);
    using (MemoryMappedFile objMf = MemoryMappedFile.CreateFromFile(fileName, FileMode.OpenOrCreate, key, bytes.Length))
    {
        using (MemoryMappedViewAccessor accessor = objMf.CreateViewAccessor())
        {
            accessor.WriteArray(0, bytes, 0, bytes.Length);
        }
    }
}
public DataSet ReadFromMemoryMap(string fileName)
{
    var fi = new FileInfo(fileName);
    var length = (int)fi.Length;
    var newBytes = new byte[length];
    using (MemoryMappedFile objMf = MemoryMappedFile.CreateFromFile(fileName, FileMode.Open))
    {
        using (MemoryMappedViewAccessor accessor = objMf.CreateViewAccessor())
        {
            accessor.ReadArray(0, newBytes, 0, length);
        }
    }
    return DecompressData(newBytes);
}
public byte[] CompressData(DataSet ds)
{
    try
    {
        byte[] data = null;
        var memStream = new MemoryStream();
        var zipStream = new GZipStream(memStream, CompressionMode.Compress);
        ds.WriteXml(zipStream, XmlWriteMode.WriteSchema);
        zipStream.Close();
        data = memStream.ToArray();
        memStream.Close();
        return data;
    }
    catch (Exception)
    {
        return null;
    }
}
public DataSet DecompressData(byte[] data)
{
    try
    {
        var memStream = new MemoryStream(data);
        var unzipStream = new GZipStream(memStream, CompressionMode.Decompress);
        var objDataSet = new DataSet();
        objDataSet.ReadXml(unzipStream, XmlReadMode.ReadSchema);
        unzipStream.Close();
        memStream.Close();
        return objDataSet;
    }
    catch (Exception)
    {
        return null;
    }
}

View more solutions

Comments

[Saw]

What do you suggest for IPC? put add some tail to the file? or what?

[Saw]

I am thinking in putting the size of the file at the beginning.

[Guffa]

@MohamedSakherSawan: Yes, any file structure where the data in the file itself can be used to determine the size would work.

[Saw]

The answer has some problems, please provide a better answer.

[Jay]

For all that you may as well include the current offset / length in the header as well...

[Lorenzo Dematté]

@MohamedSakherSawan, are you using memory mapped files for IPC? Why haven't you written it in your question? It is an important detail!

[ElektroStudios]

amazing, i didn't found any problem reading/writting strings that way, those rounded 4096 bytes go out when getting the string bytes. really very usefull solution.

[SuperBiasedMan]

If possible please explain a bit more how this code works for the asker to understand why it solves their question.

[Admin]

This line is crucial var fi = new FileInfo(fileName); var length = (int)fi.Length; Once he knows the length it will allow him to use to read everything in that file.

[tofutim]

I don't think you can use FileInfo on a shared memory file.

[Brans Ds]

this answer is wrong! Whyle system allocates memory for view in memory pages(true both for start views offset and length), the .NET hide this from the user by preventing them from writing to any memory that user did not request. So not "bytes written outside the file are just ignored." - exception will be thowed.

[Brans Ds]

no need to get FileInfo(fileName).Length CreateFromFile internaly make the capacity of the memory mapped file match the size of the file. Just pass 0 as size.

[Guffa]

@BransDs: Do you know if the behaviour has changed? The behaviour was as I descibed when I wrote it.

[Brans Ds]

@Guffa maybe.. sory, didn't know. Just for someone who find and use this. I had some trouble because of this recently.

[stmax]

@BransDs even when you pass 0 as size then capacity will be equal or greater than the real file size (its rounded up to the next 4096 byte boundary). If you need the real file size there's no way around new FileInfo(path).Length.