Building the Right Environment to Support AI, Machine Learning and Deep Learning
The advent of the TabletPC and ink integration of the Windows operating system occurred at a time when managed development and the .NET Framework were being pushed as the future of development. Because of this focus on managed development, the initial APIs for integrating ink were almost exclusively managed, leaving native developers with the headache of using .NET to COM interoperability as the primary means of integrating ink into their application.
With the release of Windows XP TabletPC Edition 2005 and now expanded with Windows Vista, native developers have a rich library of controls that they can use to integrate with pen and touch digitizers. This article will focus on a simple task—retrofitting an existing MFC application to take advantage of writing and drawing in ink, but the opportunity to integrate with ink for native code is much richer. From native C++, the following development activities are all possible:
- Real-time interaction with the digitizer using the RealTimeStylus class.
- Programmatic access to inked documents stored in Windows Journal through the Journal Reader component.
- Using the Ink Analysis APIs to write a custom recognizer for ink input.
All of this functionality, as well as the InkEdit and InkPicture controls to be covered later in this article, are provided via a set of COM controls that can be consumed like all other COM controls from a standard native project. A lot of the functionality mirrors managed controls of similar names available in the Microsoft.Ink assembly, and developers should be careful when reading documentation related to Ink input because it often can be unclear whether the documentation is referring to managed or native components.
Inking in MFC
Before covering the new Ink controls, it is worth briefly covering the end-user experience for a traditional application that was designed without ink integration in mind. Figure 1 shows a simple MFC console application with a standard Edit control running on Windows Vista Ultimate, which includes ink input support. End users can use a pen or mouse to write in the Input Panel, as shown in Figure 1, and when the desired text has been entered, the ink will be converted to text and moved into the Edit control. This Input Panel allows input across a variety of controls for most applications running under Vista, and for legacy applications that don't use owner-drawn controls, provides a reasonable experience when working with a pen.
Figure 1: Ink Input into a Legacy Application
To provide a more natural interface for ink input, native developers can use one of two COM controls to capture ink. The InkEdit control allows an end user to ink directly onto the controls surface (which is an extended rich edit control), and once the ink input has been completed, the ink is converted to text, as shown in Figure 2. These controls ship with the Windows SDK for Windows Vista, and can be added to an existing project by simply selecting the Choose Items item from the Toolbox context menu and adding the two controls as shown in Figure 3.
Figure 2: Ink Integration on a MFC Dialog
Figure 3: Adding the controls to the Visual Studio Toolbox
If an application using the InkEdit control is being developed on an operating system that does not support pen input, the InkMode property will be set to Disabled, and this can not be changed via the Visual Studio property grid. To overcome this limitation, InkMode needs to be set to IEM_InkAndGesture in the dialog's OnInitDialog method.
For the InkEdit control, the amount of time that needs to pass between the last stoke of ink and the point at which text recognition begins is configured with the RecognitionTimeout property; by default, this value is 2000 milliseconds. The control also supports a Recognize method, so text recognition can be delayed indefinitely until initiated by application logic or the end user.
InkEdit allows ink to be stored as ink as well as text. By setting the InkInsertMode of the control to IEM_InsertInk, ink is stored in the control rather than text. The control does not support a large range of functionality to actually get at the stored ink, and although it is possible to get a collection of InkDisp objects from the control, which represent the actual ink stokes, the strokes in this ink must be recognized as text before the strokes are available.
In contrast to the InkEdit control that is designed to capture either ink or typed text, and store it as either text or ink, the InkPicture control, which is shown at the bottom of the MFC dialog in Figure 2, is designed to capture and store ink directly without converting it to text. This control is suited for situations where capturing drawing from the end-user is required. Long-time MFC users can think of InkPicture as the old Scribble sample application packaged as a COM control and optimized for recognizing the subtleties of pen input, which, depending on the pen hardware in use, is much richer than simple mouse scribbling.
The final control worth mentioning briefly is the InkOverlay COM class, which can be placed over existing UI controls to allow pen-based annotations. This control is fairly easy to integrate into existing MFC applications, and can give a mature and familiar application a welcome make-over for Vista at a low development cost.
Upgrading a MFC to Vista by adding some ink functionality is a simple task, and well worth the effort. As a start, look at replacing existing RichEdit textboxes with the InkEdit control—even if all the properties (other than InkMode) are left at their default values, the application will still be friendlier for pen users. The only UI redesign that may be required is to increase the size of the control to provide a larger area for writing in.