Introduction to Memory Mapped Files

Introduction

The .NET framework provides APIs for file editing, however the APIs offered in versions prior to .NET framework 4.0 did not scale well for working with extremely large files.

Of the various stream classes in System.IO, FileStream does buffering internally whereas MemoryStream does not need buffering, because it is a non-buffered stream whose encapsulated data is directly accessible in memory. With no backup store besides memory, the MemoryStream is as useful as a temporary buffer.

MSDN defined memory mapped file as a representation that allows applications to access files in the same way as they access dynamic memory. Click the following link to read the complete article Managing Memory-Mapped Files.

In .NET framework 4.0, the Base Class Libraries folks in the Common Language Runtime team added managed APIs for supporting memory-mapped files under the System.IO.MemoryMappedFiles namespace.

Types of Memory Mapped Files

Memory mapped files have two variants:

  • Persisted Files– These files have a physical file on disk which they relate to. These types of memory-mapped files are used when working with extremely large files. A portion of these physical files are loaded in memory for accessing their contents.
  • Non-persisted files – These files do not have a corresponding physical file on the disk. When the process terminates, all content is lost. These types of files are used for inter-process communication also called IPC. In such cases, processes can map to the same memory mapped file by using a common name that is assigned by the process to create the file.

Benefits of Memory Mapped Files

One of the primary benefits of using memory-mapped files is increased I/O performance since the contents of the file are loaded in memory. Accessing RAM is faster than disk I/O operation and hence a performance boost is achieved when dealing with extremely large files.

Memory mapped files also offer lazy loading which equated to using a small amount of RAM for even a large file. This works as follows. Usually an application only has to show one page’s worth of data. For such applications, there is no point loading all the contents of the file in memory. Having memory mapped files and their ability to create views allows us to reduce the memory footprint of the application.

Drawbacks of Memory Mapped Files

Since memory mapped files amount to loading data in memory, if the user does not create a proper view, it will amount to using a lot of memory and there will be an initial performance hit since all the contents will have to be read into memory.

Working With Memory Mapped Files

Memory mapped file are used by creating a view of a part or full memory mapped file.

On 32 bit operating systems, since an application is limited to 2 GB of user memory space, multiple views might be needed.

If you are using a memory file stream to read sequential data from a physical file, you should use the MemoryMappedViewStream stream which can be done by calling the CreateViewStream method on MemoryMappedFile object.

For IPC, your memory mapped file will need to have a MemoryMappedViewAccessor view which accesses random memory and that can be created by calling CreateViewAccessor method on the MemoryMappedFile instance.

Sample Application Using Persisted Memory Mapped File

Lets create a sample application which involves using memory mapped file. In the code snippet we are creating a persisted memory mapped file which maps to a physical file which is accessed sequentially. Sample code with the listing is also available.

  using System;
  using System.Collections.Generic;
  using System.Linq;
  using System.Text;
  using System.IO;
  using System.IO.MemoryMappedFiles;

  namespace PersistentMemoryMappedFile
  {
      class Program
      {
          static string LargeString = "On the Insert tab, the galleries include items that are designed to coordinate with the overall look of your document. You can use these galleries to insert tables, headers, footers, lists, cover pages, and other document building blocks. When you create pictures, charts, or diagrams, they also coordinate with your current document look. You can easily change the formatting of selected text in the document text by choosing a look for the selected text from the Quick Styles gallery on the Home tab. You can also format text directly by using the other controls on the Home tab. Most controls offer a choice of using the look from the current theme or using a format that you specify directly. To change the overall look of your document, choose new Theme elements on the Page Layout tab. To change the looks available in the Quick Style gallery, use the Change Current Quick Style Set command. Both the Themes gallery and the Quick Styles gallery provide reset commands so that you can always restore the look of your document to the original contained in your current template. On the Insert tab, the galleries include items that are designed to coordinate with the overall look of your document. You can use these galleries to insert tables, headers, footers, lists, cover pages, and other document building blocks. When you create pictures, charts, or diagrams, they also coordinate with your current document look. You can easily change the formatting of selected text in the document text by choosing a look for the selected text from the Quick Styles gallery on the Home tab. You can also format text directly by using the other controls on the Home tab. Most controls offer a choice of using the look from the current theme or using a format that you specify directly. To change the overall look of your document, choose new Theme elements on the Page Layout tab. To change the looks available in the Quick Style gallery, use the Change Current Quick Style Set command. Both the Themes gallery and the Quick Styles gallery provide reset commands so that you can always restore the look of your document to the original contained in your current template.";
          static void Main(string[] args)
          {
              CreateLargeFile();
              CreateMemoryMappedFile("largetextfile.txt");

          }

          private static void CreateMemoryMappedFile(string fileName)
          {
              FileInfo fInfo = new FileInfo(fileName);
              long length = fInfo.Length;
              MemoryMappedFile mySequentialMMF = MemoryMappedFile.CreateFromFile(fileName, FileMode.Open, "MySequentialMap");
              for (long i = 0; i < length; i = i + 100000)
              {
                  MemoryMappedViewStream myMMFViewStream = mySequentialMMF.CreateViewStream(0, 100000, MemoryMappedFileAccess.ReadWrite);
                  myMMFViewStream.WriteByte((byte)'A');
                  Console.WriteLine("We changed a bit");
              }
          }

          private static void CreateLargeFile()
          {
              using (StreamWriter sw = new StreamWriter("largetextfile.txt", false))
              {
                  for (int i = 0; i < 500000; i++)
                      sw.WriteLine(LargeString);
              }
              File.Copy("largetextfile.txt", "largetextfileoriginal.txt");

          }
      }
  }

Code Listing 1

In the above code snippet we are creating a large file and then using MemoryMappedFile modifying every 100000th byte in the stream to an ‘A’. If you have a file doffing tool which can handle large files, you will notice the difference.

More by Author

Get the Free Newsletter!

Subscribe to Developer Insider for top news, trends & analysis

Must Read