How can I quickly read bytes from a memory mapped file in .NET?
Solution 1
This solution requires unsafe code (compile with /unsafe
switch), but grabs a pointer to the memory directly; then Marshal.Copy
can be used. This is much, much faster than the methods provided by the .NET framework.
// assumes part of a class where _view is a MemoryMappedViewAccessor object
public unsafe byte[] ReadBytes(int offset, int num)
{
byte[] arr = new byte[num];
byte *ptr = (byte*)0;
this._view.SafeMemoryMappedViewHandle.AcquirePointer(ref ptr);
Marshal.Copy(IntPtr.Add(new IntPtr(ptr), offset), arr, 0, num);
this._view.SafeMemoryMappedViewHandle.ReleasePointer();
return arr;
}
public unsafe void WriteBytes(int offset, byte[] data)
{
byte* ptr = (byte*)0;
this._view.SafeMemoryMappedViewHandle.AcquirePointer(ref ptr);
Marshal.Copy(data, 0, IntPtr.Add(new IntPtr(ptr), offset), data.Length);
this._view.SafeMemoryMappedViewHandle.ReleasePointer();
}
Solution 2
See this bug report: No way to determine internal offset used by MemoryMappedViewAccessor - Makes SafeMemoryMappedViewHandle property unusable.
From the report:
MemoryMappedViewAccessor has a SafeMemoryMappedViewHandle property, which returns the ViewHandle being used internally by the MemoryMappedView, but does not have any property to return the offset being used by the MemoryMappedView.
As the MemoryMappedView is page aligning the offset requested in MemoryMappedFile.CreateViewAccessor(offset,size) it is impossible to use the SafeMemoryMappedViewHandle for anything useful without knowing the offset.
Note that what we actually want to do is use the AcquirePointer(ref byte* pointer) method to allow some fast pointer based (possibly unmanaged) code to run. We're OK with the pointer being page aligned, but it must be possible to find out what the offset from the originally requested address is.
Solution 3
I know this is an older question which has been answered but I wanted to add my two cents.
I ran a test with both the accepted answer (using the unsafe code) and with the MemoryMappedViewStream approach for reading a 200MB byte array.
MemoryMappedViewStream
const int MMF_MAX_SIZE = 209_715_200;
var buffer = new byte[ MMF_VIEW_SIZE ];
using( var mmf = MemoryMappedFile.OpenExisting( "mmf1" ) )
using( var view = mmf.CreateViewStream( 0, buffer.Length, MemoryMappedFileAccess.ReadWrite ) )
{
if( view.CanRead )
{
Console.WriteLine( "Begin read" );
sw.Start( );
view.Read( buffer, 0, MMF_MAX_SIZE );
sw.Stop( );
Console.WriteLine( $"Read done - {sw.ElapsedMilliseconds}ms" );
}
}
I ran the test 3 times with each approach and received the following times.
MemoryMappedViewStream:
- 483ms
- 501ms
- 490ms
Unsafe method
- 531ms
- 517ms
- 523ms
From the small amount of testing it looks like the MemoryMappedViewStream
has a very slight advantage. With that in mind for anyone reading this post down the road I would go with the MemoryMappedViewStream
.
Solution 4
A safe version of this solution is:
var file = MemoryMappedFile.CreateFromFile(...);
var accessor = file.CreateViewAccessor();
var bytes = new byte[yourLength];
// assuming the string is at the start of the file
// aka position: 0
// https://msdn.microsoft.com/en-us/library/dd267761(v=vs.110).aspx
accessor.ReadArray<byte>(
position: 0, // The number of bytes in the accessor at which to begin reading
array: bytes, // The array to contain the structures read from the accessor
offset: 0, // The index in `array` in which to place the first copied structure
count: yourLength // The number of structures of type T to read from the accessor.
);
var myString = Encoding.UTF8.GetString(bytes);
I have tested this, it does work. I cannot comment on it's performance or if it's the BEST overall solution just that it works.
Related videos on Youtube
Kieren Johnstone
Software developer, company director and guitarist. Experience in C#, C++, Java, SQL, various embedded systems, assembler. Enjoy work with algorithms, distributed computing, anything clever or inventive. Familiar with various frameworks, technologies and methodologies; ASP.NET / MVC, WPF, MVVM, strong OOP, design patterns, DI, XML / XSLT, WCF, low-level network protocols, some COM, MFC, unit testing, code optimization and other low-level fun bits.
Updated on June 13, 2022Comments
-
Kieren Johnstone almost 2 years
In some situations the
MemoryMappedViewAccessor
class just doesn't cut it for reading bytes efficiently; the best we get is the genericReadArray<byte>
which it the route for all structs and involves several unnecessary steps when you just need bytes.It's possible to use a
MemoryMappedViewStream
, but because it's based on aStream
you need to seek to the correct position first, and then the read operation itself has many more unnecessary steps.Is there a quick, high-performance way to read an array of bytes from a memory-mapped file in .NET, given that it should just be a particular area of the address space to read from?
-
Kieren Johnstone about 12 yearsIt seems silly.. if you're in control of the view, you don't need .NET to tell you the offset, since you specified it. (That's what I do:
_view
is an accessor at offset 0) -
Kieren Johnstone about 12 yearsFwiw, this code has also been stress-tested to death [billions of calls, thousands of different MMFs] on several machines
-
Matt Howells over 11 yearsYou should be using a Critical Execution Block and a try-finally to ensure that ReleasePointer runs even if the Marshal.Copy throws an exception.
-
Kieren Johnstone about 11 yearsNow hundreds of billions of calls, and hundreds of thousands of MMFs. This bug does not happen with my code ;)
-
Admin about 11 yearsGood answer =) Indeed, profiling shows the managed wrapper is 30x times slower than using an unsafe pointer to access the mapped memory.
-
LaFleur almost 9 years@MattHowells I agree. I read that CER might affect performance, but it seems negligble (in a controlled test at least). Regardless of performance implications it is the correct usage pattern as described under "remarks" here; msdn.microsoft.com/en-us/library/…
-
Kieren Johnstone almost 7 yearsCool, yes definitely avoids the use of pointers :)
ReadArray<byte>
is much, much slower, however -
RobinG almost 7 yearsSeems to be fixed now - you can access the PointerOffset property of the MemoryMappedViewAccessor to work out the correct pointer address corresponding to the requested offset. Just add the PointerOffset to the address returned by SafeMemoryMappedViewHandle.AcquirePointer()
-
mhand over 2 yearsThis makes sense, the documentation says to use view stream for sequential access, it's optimized for it, whereas the view accessor is intended for random access.