Introduction to Memory Mapped Files

Introduction

The .NET framework provides APIs for file editing, however the APIs offered in versions prior to .NET framework 4.0 did not scale well for working with extremely large files.

Of the various stream classes in System.IO, FileStream does buffering internally whereas MemoryStream does not need buffering, because it is a non-buffered stream whose encapsulated data is directly accessible in memory. With no backup store besides memory, the MemoryStream is as useful as a temporary buffer.

MSDN defined memory mapped file as a representation that allows applications to access files in the same way as they access dynamic memory. Click the following link to read the complete article Managing Memory-Mapped Files.

In .NET framework 4.0, the Base Class Libraries folks in the Common Language Runtime team added managed APIs for supporting memory-mapped files under the System.IO.MemoryMappedFiles namespace.

Types of Memory Mapped Files

Memory mapped files have two variants:

  • Persisted Files- These files have a physical file on disk which they relate to. These types of memory-mapped files are used when working with extremely large files. A portion of these physical files are loaded in memory for accessing their contents.
  • Non-persisted files - These files do not have a corresponding physical file on the disk. When the process terminates, all content is lost. These types of files are used for inter-process communication also called IPC. In such cases, processes can map to the same memory mapped file by using a common name that is assigned by the process to create the file.

Benefits of Memory Mapped Files

One of the primary benefits of using memory-mapped files is increased I/O performance since the contents of the file are loaded in memory. Accessing RAM is faster than disk I/O operation and hence a performance boost is achieved when dealing with extremely large files.

Memory mapped files also offer lazy loading which equated to using a small amount of RAM for even a large file. This works as follows. Usually an application only has to show one page's worth of data. For such applications, there is no point loading all the contents of the file in memory. Having memory mapped files and their ability to create views allows us to reduce the memory footprint of the application.

Drawbacks of Memory Mapped Files

Since memory mapped files amount to loading data in memory, if the user does not create a proper view, it will amount to using a lot of memory and there will be an initial performance hit since all the contents will have to be read into memory.

Working With Memory Mapped Files

Memory mapped file are used by creating a view of a part or full memory mapped file.

On 32 bit operating systems, since an application is limited to 2 GB of user memory space, multiple views might be needed.

If you are using a memory file stream to read sequential data from a physical file, you should use the MemoryMappedViewStream stream which can be done by calling the CreateViewStream method on MemoryMappedFile object.

For IPC, your memory mapped file will need to have a MemoryMappedViewAccessor view which accesses random memory and that can be created by calling CreateViewAccessor method on the MemoryMappedFile instance.

Sample Application Using Persisted Memory Mapped File

Lets create a sample application which involves using memory mapped file. In the code snippet we are creating a persisted memory mapped file which maps to a physical file which is accessed sequentially. Sample code with the listing is also available.

  using System;
  using System.Collections.Generic;
  using System.Linq;
  using System.Text;
  using System.IO;
  using System.IO.MemoryMappedFiles;
  
  namespace PersistentMemoryMappedFile
  {
      class Program
      {
          static string LargeString = "On the Insert tab, the galleries include items that are designed to coordinate with the overall look of your document. You can use these galleries to insert tables, headers, footers, lists, cover pages, and other document building blocks. When you create pictures, charts, or diagrams, they also coordinate with your current document look. You can easily change the formatting of selected text in the document text by choosing a look for the selected text from the Quick Styles gallery on the Home tab. You can also format text directly by using the other controls on the Home tab. Most controls offer a choice of using the look from the current theme or using a format that you specify directly. To change the overall look of your document, choose new Theme elements on the Page Layout tab. To change the looks available in the Quick Style gallery, use the Change Current Quick Style Set command. Both the Themes gallery and the Quick Styles gallery provide reset commands so that you can always restore the look of your document to the original contained in your current template. On the Insert tab, the galleries include items that are designed to coordinate with the overall look of your document. You can use these galleries to insert tables, headers, footers, lists, cover pages, and other document building blocks. When you create pictures, charts, or diagrams, they also coordinate with your current document look. You can easily change the formatting of selected text in the document text by choosing a look for the selected text from the Quick Styles gallery on the Home tab. You can also format text directly by using the other controls on the Home tab. Most controls offer a choice of using the look from the current theme or using a format that you specify directly. To change the overall look of your document, choose new Theme elements on the Page Layout tab. To change the looks available in the Quick Style gallery, use the Change Current Quick Style Set command. Both the Themes gallery and the Quick Styles gallery provide reset commands so that you can always restore the look of your document to the original contained in your current template.";
          static void Main(string[] args)
          {
              CreateLargeFile();
              CreateMemoryMappedFile("largetextfile.txt");
  
          }
  
          private static void CreateMemoryMappedFile(string fileName)
          {
              FileInfo fInfo = new FileInfo(fileName);
              long length = fInfo.Length;
              MemoryMappedFile mySequentialMMF = MemoryMappedFile.CreateFromFile(fileName, FileMode.Open, "MySequentialMap");
              for (long i = 0; i < length; i = i + 100000)
              {
                  MemoryMappedViewStream myMMFViewStream = mySequentialMMF.CreateViewStream(0, 100000, MemoryMappedFileAccess.ReadWrite);
                  myMMFViewStream.WriteByte((byte)'A');
                  Console.WriteLine("We changed a bit");
              }
          }
  
          private static void CreateLargeFile()
          {
              using (StreamWriter sw = new StreamWriter("largetextfile.txt", false))
              {
                  for (int i = 0; i < 500000; i++)
                      sw.WriteLine(LargeString);
              }
              File.Copy("largetextfile.txt", "largetextfileoriginal.txt");
  
          }
      }
  }

Code Listing 1

In the above code snippet we are creating a large file and then using MemoryMappedFile modifying every 100000th byte in the stream to an 'A'. If you have a file doffing tool which can handle large files, you will notice the difference.



Introduction to Memory Mapped Files

Sample Application Using MemoryMappedFile for InterProcess Communication

Let us see an example of using Memory Mapped Files for Inter Process communication.

Here we have two applications, one is the writer and the other is reader. Here is the listing of the writer.

  using System;
  using System.Collections.Generic;
  using System.Linq;
  using System.Text;
  using System.IO.MemoryMappedFiles;
  using System.IO;
  using System.Threading;
  namespace NonPersistedMemoryMappedFileA
  {
      class Program
      {
          static void Main(string[] args)
          {
              MemoryMappedFile myNonPersisterMemoryMappedFile = MemoryMappedFile.CreateNew("myNonPersisterMemoryMappedFile", 100);
              bool IsmutexCreated;
              Mutex mymutex = new Mutex(true, "NonPersisterMemoryMappedFilemutex", out IsmutexCreated);
              Console.WriteLine("Start NonPersisterMemoryMappedFileB now");
              Console.ReadLine();
  
              StreamWriter sw = new StreamWriter(myNonPersisterMemoryMappedFile.CreateViewStream());
              sw.WriteLine("Process A says Hi");
              sw.Close();
              mymutex.ReleaseMutex();
  
          }
      }
  }

Listing 2 - NonPersistedMemoryMappedFileB.cs


Compile the above listing as NonPersistedMemoryMappedFileA.exe using the following command line:

  C:\windows\microsoft.net\frame work\v4.0.30319\csc.exe NonPersistedMemoryMappedFileA.cs

Here is the listing of the second application which takes part in the IPC.

  using System;
  using System.Collections.Generic;
  using System.Linq;
  using System.Text;
  using System.IO.MemoryMappedFiles;
  using System.IO;
  using System.Threading;
  
  namespace NonPersistedMemoryMappedFileB
  {
      class Program
      {
          static void Main(string[] args)
          {
              MemoryMappedFile myNonPersisterMemoryMappedFile = MemoryMappedFile.OpenExisting("myNonPersisterMemoryMappedFile");
              Mutex mymutex = Mutex.OpenExisting("NonPersisterMemoryMappedFilemutex");
              mymutex.WaitOne();
  
              StreamReader sr = new StreamReader(myNonPersisterMemoryMappedFile.CreateViewStream());
              Console.WriteLine(sr.ReadLine());
              sr.Close();
              mymutex.ReleaseMutex();
  
              StreamWriter sw = new StreamWriter(myNonPersisterMemoryMappedFile.CreateViewStream());
              sw.WriteLine("Process B says Hi");
              sw.Close();
  
          }
      }
  }

Listing 3 - NonPersistedMemoryMappedFileB.cs


Compile the above listing as NonPersistedMemoryMappedFileB.exe using the following command line.

  C:\windows\microsoft.net\framework\v4.0.30319\csc.exe NonPersistedMemoryMappedFileB.cs

To observe the use of memory mapped file, open a command prompt and execute NonPersistedMemoryMappedFileA.exe.

When prompted, execute NonPersistedMemoryMappedFileB.exe and click enter on the window running NonPersistedMemoryMappedFileA.exe.

You will notice that as soon as "enter" is pressed, the NonPersistedMemoryMappedFileB.exe application outputs the message sent to it via NonPersistedMemoryMappedFileA.exe

This example shows how IPC works with Memory mapped files.

Conclusion

We have seen how Memory Mapped Files improve performance of applications involving disk IO with very large file. We also see how IPC between processes can be accomplished using memory mapped files.

Related Articles





About the Author

Vipul Vipul Patel

Vipul Patel is a Software Engineer currently working at Microsoft Corporation, working in the Office Communications Group and has worked in the .NET team earlier in the Base Class libraries and the Debugging and Profiling team. He can be reached at vipul_d_patel@hotmail.com

Downloads

Comments

  • There are no comments yet. Be the first to comment!

Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • IBM Worklight is a mobile application development platform that lets you extend your business to mobile devices. It is designed to provide an open, comprehensive platform to build, run and manage HTML5, hybrid and native mobile apps.

  • Companies must routinely transfer files and share data to run their business, work with partners, and speed operations. However, many find the traditional approach to file transfer lacks necessary security, is too complex and difficult to manage, does not support the levels of automation needed, and breaks down when addressing the file transfer requirements of new areas like Big Data analytics and mobile applications. This QuinStreet SmartSelect discusses how the changing business environment is making the use …

Most Popular Programming Stories

More for Developers

Latest Developer Headlines

RSS Feeds