Video overlay using Microsoft DirectShow

Table of contents

1. What does this article cover?
2. What is the intended audience and what are the requirements ?
3. Simple video stream playback sample
4. The overlay paradigm
5. Making the overlay mix
5.1 Double-buffer technique
5.2 Triple-buffer technique
6. hWnd, DC, and the virtual system coordinates
7. Piece of code

 

1. What does this article cover?

 

Though the Microsoft DirectShow SDK provides both high and low level APIs to show video playback in a window or even fullscreen, it provides no sample showing how to perform sprite overlay on top of the video. This article is dedicated to this. I demonstrate how to perform sprite overlay, and more particularly GDI polygon shape overlay, by adding a few lines of code to a sample provided in the DirectShow SDK.

Sprite overlay on top of video means value-added content. The overlay is not meant only to emphasize something appearing in the video, it can be interactive. A routine could pump mouse clicks and perform specific actions, hence hypermedia and multimedia scenariis can clearly become a reality. A routine could also pump mouse moves from the windows message queue and allow the sprite to move, or something else, according to user-action....

 

2. What is the intended audience and what are the requirements ?

 

This article is clearly intended to the developer community. I focus on the DirectShow SDK and hence it may be helpful to developers interested in this API. It may be of interest to programmers working on another multimedia API, and to developers working on similar content concerns for a commercial software or something.

The requirements are to have the DirectShow SDK installed on your computer. Not only the run-time of course. Anyway, it doesn't matter whether you have the 5.2 version or higher, currently 6 version, since the sample program I reuse hasn't changed a single bit...

In order to download the DirectShow 6.0 SDK, check this link http://www.microsoft.com/directX/download.asp

You may also need to download the DirectX runtime in order to be able to run projects using DirectDraw. Check for it at the same address.

 

3. Simple video stream playback sample

 

First you will have to copy/paste the source code from the DirectShow documentation. Starting at the default page, go to the Application Developer's guide, then How to... then Play a Movie in a Window Using DirectDrawEx and Multimedia Streaming. This program is an extension of the simple ShowStream sample provided in the SDK, which is at a higher user-level than what follows.

Make it a VC++ project : create a new workspace, select WIN32 Application target, and then include all .h, .cpp files. Then in the project settings, add the following library links on the Link tab : amstrmid.lib quartz.lib strmbase.lib ddraw.lib . The librairies dedicated to DirectShow are strmbase.lib and amstrmid.lib (multimedia streaming, debug version), and quartz.lib (DirectShow run-time). ddraw.lib is for DirectDraw.

Build the stuff, and run it. From now on, take a look at the code. The playback process consists in a few elementary steps :

  1. Create all DirectDraw surfaces aimed at receiving the video samples, a secondary backbuffer in Triple-buffer mode, and the primary display surface,
  2. Prepare a video media stream and an audio media stream,
  3. Open the video file, and apply a render procedure to create an appropriate Filter Graph,
  4. Run the multimedia stream,
  5. Loop and render any new video sample. Watch out COMPLETE_VIDEO flag to stop the multimedia stream when it has ended. Watch out keyboard, mouse, ... to allow interaction.

 

4. The overlay paradigm

 

While the first four steps are merely initialisation steps, the fifth step is the main loop. Our procedure will be included in this main loop.

Let's take a look at the RenderToSurface procedure. The Update call asks the multimedia stream to update the next video/audio sample, which is blitted in a backbuffer. This backbuffer is attached to the stream sample engine at init-time.

Then the content of the backbuffer is simply blitted into the primary surface, resulting in what you see when you run the program. Note that the stretch blit redimensions the surface so that it fits the whole client area. The redimension is due to the fact that the video sample has a constant (width,height) size though the client area has a stretchable size.

Drawing a sprite after this blit operation is possible. Obtain a DC for the surface through IDirectDrawSurface->GetDC, then apply any desired GDI operations (the hWnd can be known easily), then release the DC. The drawback of doing that ? It flickers since it's not really a good idea to draw directly on the visible surface. You may wonder why the blit operation doesn't provoke flickers too ? It's simple, two successive blits are almost the same since the content almost doesn't change much between two frames.

Dealing with the DC, please remember that you can't debug a program inside a GetDC, ReleaseDC block.

We are about to see solutions for clean sprite overlay in the next paragraph.

 

5. Making the overlay mix

 

We show that the double-buffer technique is a good solution for simple playbacked video (ie always updated, never paused), but a triple-buffer technique will help to solve the problem of non-updated backbuffer.

 

5.1 Double-buffer technique

 

Before the blit, why don't we draw something into the backbuffer ? Afterall the entire content of the backbuffer is copied so it's a good place for a sprite overlay. And furthermore we wouldn't draw directly on the primary surface, preventing flicker.

 

5.2 Triple-buffer technique

 

We know that we have to draw a sprite into a backbuffer. Let us call sample-backbuffer the surface attached to the stream sample engine. As you may allow video pause, and thus the sample-backbuffer wouldn't be updated in that interval time, you would draw again and again into the actual same surface, resulting on an awful visible trace as the sprite moves. The solution is to store the sample-backbuffer in another backbuffer, the copy-backbuffer, and actually draw in that copy-backbuffer.

The update sample is blitted to the copy-backbuffer, we then draw into the copy-backbuffer surface, and finally we blit the copy-backbuffer to the primary surface. It all works fine except in terms of CPU time due to double blit-cake. And when the video is paused ? The sample-backbuffer is not updated, but the blit is still performed thus the copy-backbuffer is updated, with always the same surface, and this is what we wanted.

To optimize CPU performance, perform a draw into the sample-backbuffer when the video is playing, and perform a draw into the copy-backbuffer when the video is pausing. This means a single blit most of the time.

 

6. hWnd, DC, and the virtual system coordinates

 

Everything would be ok if all three surfaces would have the same size and aspect ratio. But it's not the case and may be you don't want what you draw be stretched. In GDI terms, a blit operation is a simple copy operation, while a stretch blit is a scaled copy operation. In DirectDraw surface terms, there is no strictly stretch blit operation, and the blit can be either a copy or a scaled-copy (using the hardware if hardware stretch is available).

I used the word blit in the last paragraph and didn't stress much about stretching, but it is now time to uncover these details.

Generally speaking, the sample-backbuffer is blitted to the copy-backbuffer. The copy-backbuffer thus has the same width and height than the video itself has. Then we draw in the copy-backbuffer. We finally stretch-blit the copy-backbuffer to the primary surface which has a dynamical (width,height) size. In all three surfaces the color-depth is the same. All in all, the drawn sprite is stretched.

To avoid streched sprite, stretch blit the sample-backbuffer to the copy-backbuffer, then draw the sprite, then blit to the primary surface. The DC also has to be affected since it has all physical and virtual system coordinates. To draw consistantly, always draw the sprite using a DC for which the virtual system coordinates is constant, for example (0,0)-(1000,1000), using the SetWindowExt function, while the physical system coordinates is the actual copy-backbuffer size, using SetViewportExt.

 

7. Piece of code

 

I give hereby a piece of code demonstrating what I have explained, which is the most part of the RenderToSurface function. Copy/Paste and build. It should work.

if (!g_bPaused)
{ 
  
 // update each frame
 

 if (g_pSample->Update(0, NULL, NULL, 0) != S_OK)
 {
  g_bAppactive = FALSE; 
	g_pMMStream->SetState(STREAMSTATE_STOP); 
 }
}

 
// blit from the sample-offscreen surface to the copy-offscreen surface 


hr = g_pDDSOffscreen->Blt(&rect, g_pDDSOffscreenSample, &rect, DDBLT_WAIT, NULL);

if (FAILED(hr))
{ 
 AfxMessageBox("Blt failed"); 
 ExitCode(); 
}



// obtain the DC so we can fuck with the GDI into the copy-offscreen surface


HDC my_dc;
g_pDDSOffscreen->GetDC(&my_dc);


// virtual system coordinates / physical system coordinates


::SetMapMode(my_dc,MM_ANISOTROPIC);
SIZE size;


// video_width and video_height are obtained at the Video media stream init-time


::SetViewportExtEx(my_dc, video_width, video_height, &size);
RECT rect2;
GetClientRect(&rect2);  // client region of the primary surface 
::SetWindowExtEx(my_dc, rect2.right-rect2.left, rect2.bottom-rect2.top, &size);

HPEN my_pen=(HPEN)::CreatePen(PS_SOLID,2,RGB(255,0,0));
HPEN old_pen=(HPEN)::SelectObject(my_dc,(HPEN)my_pen);


// ...
// ... place here all GDI operations ...
// ...


::SelectObject(my_dc,(HPEN)old_pen);


// release the DC


g_pDDSOffscreen->ReleaseDC(my_dc)


// clear the Hstack


::DeleteObject((HPEN)my_pen);


// finally perform a blit to the primary surface


hr = g_pPrimarySurface->Blt(&rect2, g_pDDSOffscreen, &rect, DDBLT_WAIT, NULL);

if(FAILED(hr))
{ 
 AfxMessageBox("Blt failed"); ExitCode(); 
}

DirectShow is a product from Microsoft corporation.

Contact the author :arstlg@hotmail.com,arstlg@iname.com