Video overlay using Microsoft DirectShow

Table of contents

1. What does this article cover?
2. What is the intended audience and what are the requirements ?
3. Simple video stream playback sample
4. The overlay paradigm
5. Making the overlay mix
5.1 Double-buffer technique
5.2 Triple-buffer technique
6. hWnd, DC, and the virtual system coordinates
7. Piece of code

 

1. What does this article cover?

 

Though the Microsoft DirectShow SDK provides both high and low level APIs to show video playback in a window or even fullscreen, it provides no sample showing how to perform sprite overlay on top of the video. This article is dedicated to this. I demonstrate how to perform sprite overlay, and more particularly GDI polygon shape overlay, by adding a few lines of code to a sample provided in the DirectShow SDK.

Sprite overlay on top of video means value-added content. The overlay is not meant only to emphasize something appearing in the video, it can be interactive. A routine could pump mouse clicks and perform specific actions, hence hypermedia and multimedia scenariis can clearly become a reality. A routine could also pump mouse moves from the windows message queue and allow the sprite to move, or something else, according to user-action....

 

2. What is the intended audience and what are the requirements ?

 

This article is clearly intended to the developer community. I focus on the DirectShow SDK and hence it may be helpful to developers interested in this API. It may be of interest to programmers working on another multimedia API, and to developers working on similar content concerns for a commercial software or something.

The requirements are to have the DirectShow SDK installed on your computer. Not only the run-time of course. Anyway, it doesn't matter whether you have the 5.2 version or higher, currently 6 version, since the sample program I reuse hasn't changed a single bit...

In order to download the DirectShow 6.0 SDK, check this link http://www.microsoft.com/directX/download.asp

You may also need to download the DirectX runtime in order to be able to run projects using DirectDraw. Check for it at the same address.

 

3. Simple video stream playback sample

 

First you will have to copy/paste the source code from the DirectShow documentation. Starting at the default page, go to the Application Developer's guide, then How to... then Play a Movie in a Window Using DirectDrawEx and Multimedia Streaming. This program is an extension of the simple ShowStream sample provided in the SDK, which is at a higher user-level than what follows.

Make it a VC++ project : create a new workspace, select WIN32 Application target, and then include all .h, .cpp files. Then in the project settings, add the following library links on the Link tab : amstrmid.lib quartz.lib strmbase.lib ddraw.lib . The librairies dedicated to DirectShow are strmbase.lib and amstrmid.lib (multimedia streaming, debug version), and quartz.lib (DirectShow run-time). ddraw.lib is for DirectDraw.

Build the stuff, and run it. From now on, take a look at the code. The playback process consists in a few elementary steps :

  1. Create all DirectDraw surfaces aimed at receiving the video samples, a secondary backbuffer in Triple-buffer mode, and the primary display surface,
  2. Prepare a video media stream and an audio media stream,
  3. Open the video file, and apply a render procedure to create an appropriate Filter Graph,
  4. Run the multimedia stream,
  5. Loop and render any new video sample. Watch out COMPLETE_VIDEO flag to stop the multimedia stream when it has ended. Watch out keyboard, mouse, ... to allow interaction.

 

4. The overlay paradigm

 

While the first four steps are merely initialisation steps, the fifth step is the main loop. Our procedure will be included in this main loop.

Let's take a look at the RenderToSurface procedure. The Update call asks the multimedia stream to update the next video/audio sample, which is blitted in a backbuffer. This backbuffer is attached to the stream sample engine at init-time.

Then the content of the backbuffer is simply blitted into the primary surface, resulting in what you see when you run the program. Note that the stretch blit redimensions the surface so that it fits the whole client area. The redimension is due to the fact that the video sample has a constant (width,height) size though the client area has a stretchable size.

Drawing a sprite after this blit operation is possible. Obtain a DC for the surface through IDirectDrawSurface->GetDC, then apply any desired GDI operations (the hWnd can be known easily), then release the DC. The drawback of doing that ? It flickers since it's not really a good idea to draw directly on the visible surface. You may wonder why the blit operation doesn't provoke flickers too ? It's simple, two successive blits are almost the same since the content almost doesn't change much between two frames.

Dealing with the DC, please remember that you can't debug a program inside a GetDC, ReleaseDC block.

We are about to see solutions for clean sprite overlay in the next paragraph.

 

5. Making the overlay mix

 

We show that the double-buffer technique is a good solution for simple playbacked video (ie always updated, never paused), but a triple-buffer technique will help to solve the problem of non-updated backbuffer.

 

5.1 Double-buffer technique

 

Before the blit, why don't we draw something into the backbuffer ? Afterall the entire content of the backbuffer is copied so it's a good place for a sprite overlay. And furthermore we wouldn't draw directly on the primary surface, preventing flicker.

 

5.2 Triple-buffer technique

 

We know that we have to draw a sprite into a backbuffer. Let us call sample-backbuffer the surface attached to the stream sample engine. As you may allow video pause, and thus the sample-backbuffer wouldn't be updated in that interval time, you would draw again and again into the actual same surface, resulting on an awful visible trace as the sprite moves. The solution is to store the sample-backbuffer in another backbuffer, the copy-backbuffer, and actually draw in that copy-backbuffer.

The update sample is blitted to the copy-backbuffer, we then draw into the copy-backbuffer surface, and finally we blit the copy-backbuffer to the primary surface. It all works fine except in terms of CPU time due to double blit-cake. And when the video is paused ? The sample-backbuffer is not updated, but the blit is still performed thus the copy-backbuffer is updated, with always the same surface, and this is what we wanted.

To optimize CPU performance, perform a draw into the sample-backbuffer when the video is playing, and perform a draw into the copy-backbuffer when the video is pausing. This means a single blit most of the time.

 

6. hWnd, DC, and the virtual system coordinates

 

Everything would be ok if all three surfaces would have the same size and aspect ratio. But it's not the case and may be you don't want what you draw be stretched. In GDI terms, a blit operation is a simple copy operation, while a stretch blit is a scaled copy operation. In DirectDraw surface terms, there is no strictly stretch blit operation, and the blit can be either a copy or a scaled-copy (using the hardware if hardware stretch is available).

I used the word blit in the last paragraph and didn't stress much about stretching, but it is now time to uncover these details.

Generally speaking, the sample-backbuffer is blitted to the copy-backbuffer. The copy-backbuffer thus has the same width and height than the video itself has. Then we draw in the copy-backbuffer. We finally stretch-blit the copy-backbuffer to the primary surface which has a dynamical (width,height) size. In all three surfaces the color-depth is the same. All in all, the drawn sprite is stretched.

To avoid streched sprite, stretch blit the sample-backbuffer to the copy-backbuffer, then draw the sprite, then blit to the primary surface. The DC also has to be affected since it has all physical and virtual system coordinates. To draw consistantly, always draw the sprite using a DC for which the virtual system coordinates is constant, for example (0,0)-(1000,1000), using the SetWindowExt function, while the physical system coordinates is the actual copy-backbuffer size, using SetViewportExt.

 

7. Piece of code

 

I give hereby a piece of code demonstrating what I have explained, which is the most part of the RenderToSurface function. Copy/Paste and build. It should work.

if (!g_bPaused)
{ 
  
 // update each frame
 

 if (g_pSample->Update(0, NULL, NULL, 0) != S_OK)
 {
  g_bAppactive = FALSE; 
	g_pMMStream->SetState(STREAMSTATE_STOP); 
 }
}

 
// blit from the sample-offscreen surface to the copy-offscreen surface 


hr = g_pDDSOffscreen->Blt(&rect, g_pDDSOffscreenSample, &rect, DDBLT_WAIT, NULL);

if (FAILED(hr))
{ 
 AfxMessageBox("Blt failed"); 
 ExitCode(); 
}



// obtain the DC so we can fuck with the GDI into the copy-offscreen surface


HDC my_dc;
g_pDDSOffscreen->GetDC(&my_dc);


// virtual system coordinates / physical system coordinates


::SetMapMode(my_dc,MM_ANISOTROPIC);
SIZE size;


// video_width and video_height are obtained at the Video media stream init-time


::SetViewportExtEx(my_dc, video_width, video_height, &size);
RECT rect2;
GetClientRect(&rect2);  // client region of the primary surface 
::SetWindowExtEx(my_dc, rect2.right-rect2.left, rect2.bottom-rect2.top, &size);

HPEN my_pen=(HPEN)::CreatePen(PS_SOLID,2,RGB(255,0,0));
HPEN old_pen=(HPEN)::SelectObject(my_dc,(HPEN)my_pen);


// ...
// ... place here all GDI operations ...
// ...


::SelectObject(my_dc,(HPEN)old_pen);


// release the DC


g_pDDSOffscreen->ReleaseDC(my_dc)


// clear the Hstack


::DeleteObject((HPEN)my_pen);


// finally perform a blit to the primary surface


hr = g_pPrimarySurface->Blt(&rect2, g_pDDSOffscreen, &rect, DDBLT_WAIT, NULL);

if(FAILED(hr))
{ 
 AfxMessageBox("Blt failed"); ExitCode(); 
}

DirectShow is a product from Microsoft corporation.

Contact the author :arstlg@hotmail.com,arstlg@iname.com

 



Comments

  • Please help me on Directshow +MFC

    Posted by nvnoi76 on 12/01/2004 09:32pm

    I write a program to play Video files using Directshow and MFC ( Visual C++ 6.0 , DirectX 9.0 ) I compile in DEBUG mode, It runs well and correctly. But when I compile in RELEASE mode. It reports error : "Unhandle exception in Playwnd.exe (MFC42.dll)0xC0000005: Access Violation" Is there any problem on MFC42.dll ? Please help me to correct it. Thank you very much. Noi - nvnoi76@yahoo.com

    Reply
  • How to Overlay on Game screens or DirectX Applications

    Posted by OHDev on 05/23/2004 05:50pm

    How to Overlay on Game screens or DirectX Applications Hi there , I would like to know how to Overlay sprites or text ontop of directx Screens ,, the same as NVidia Does with their Logo,Im willing to do this because im developing a Cyber Cafe Application and i would like to Display Machine Name and Time left. Thank you Very Much OHDev

    • overlaying dx app

      Posted by mad_addison on 04/23/2005 10:04pm

      hi i think you can find something around that is this article http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnwmt/html/vmr_d3d.asp the summary is that you can render your scene to 3d surface then if you want to overlay video : render video with VMR (video mixing render) on that surface if you want to overlay text: draw the text on the surface iam working on this right now please if you found any solution please tell me at email : mad_addison@yahoo.com-

      Reply
    Reply
  • Managed DirectX 9 and Firewire Video

    Posted by Legacy on 01/13/2004 12:00am

    Originally posted by: Peter


    Stephane,

    Great article. I enjoyed it greatly.

    I'm new to the Managed DirectX 9.0 functionality. I'm trying my hands at managing DV video captured with an OCHI compliant Firewire card. I wonder if you could comment on how I can use Managed DirectX to capture the video frame from the Firewire card and send it to the VGA buffers.

    Thanks,

    Peter

    Reply
  • How to convert avi file into bmp file

    Posted by Legacy on 09/03/2002 12:00am

    Originally posted by: ZhangGq

    I don't know how to convert avi file into bmp file.
    I want someone give me some VC++ code.

    Reply
  • Direct show

    Posted by Legacy on 12/26/2001 12:00am

    Originally posted by: ramkumar

    How to Play a character Buffer from Media Player and Not from a file.
    1. The content should be buffered and Played.
    2. The content will be either be of mp3,mpeg or wav format,but not in a file.
    3. The file will be in a different extension.
    4. On opening the file the contents alone to be played through buffer instead making it a file run.
    5. The content also be splitted into parts. One by one the content to be copied to the buffer and should be played.

    Reply
  • How to Send the Datas in memory to a client ?

    Posted by Legacy on 12/15/2001 12:00am

    Originally posted by: reaking

    I want to send the datas specified by the m_pPrimarySurface pointer to a client.
    But the sockets can only send a string. How should I do?

    my email: zhanghuajun21cn@263.net

    THANKS !

    Reply
  • Do you know how to put a circular bitmap button on the DVD video stream?

    Posted by Legacy on 11/14/2000 12:00am

    Originally posted by: Hally

    It's not a full screen application. I don't want to use DirectDraw. Do you know how to do it?

    Reply
  • A note to the people sending emails to arstlg@hotmail.com

    Posted by Legacy on 02/22/2000 12:00am

    Originally posted by: arstlg


    A note to the people sending emails to arstlg@hotmail.com : well I am glad to receive so many emails, but I have to say I don't want them anymore. I am just tired of that DirectX stuff, issues, knowledge, samples, and so on. And on top of that, it seems very much that I have worked on a 0$ basis as Microsoft DirectX technical support for over 6 months, and to my opinion it's time to stop this !!

    So please don't email. I won't answer anymore.

    arstlg@hotmail.com

    Reply
  • Video overlay using Microsoft DirectShow

    Posted by Legacy on 05/05/1999 12:00am

    Originally posted by: rOK!

    im missing about 3 seconds vid/sound in your and! the MS dxmediaSDK sample app's...

    someone know to avoid this ?

    except adding
    Sleep(3000);
    to the

    if (m_pSample->Update(0, NULL, NULL, 0) != S_OK) {
    }

    handler in
    void CMovieView::RenderToSurface(BOOL bSingleBlit)

    thanx
    rOK!
    {


    Reply
Leave a Comment
  • Your email address will not be published. All fields are required.

Top White Papers and Webcasts

  • Protecting business operations means shifting the priorities around availability from disaster recovery to business continuity. Enterprises are shifting their focus from recovery from a disaster to preventing the disaster in the first place. With this change in mindset, disaster recovery is no longer the first line of defense; the organizations with a smarter business continuity practice are less impacted when disasters strike. This SmartSelect will provide insight to help guide your enterprise toward better …

  • Java developers know that testing code changes can be a huge pain, and waiting for an application to redeploy after a code fix can take an eternity. Wouldn't it be great if you could see your code changes immediately, fine-tune, debug, explore and deploy code without waiting for ages? In this white paper, find out how that's possible with a Java plugin that drastically changes the way you develop, test and run Java applications. Discover the advantages of this plugin, and the changes you can expect to see …

Most Popular Programming Stories

More for Developers

Latest Developer Headlines

RSS Feeds