Building the Right Environment to Support AI, Machine Learning and Deep Learning
The Build Process Today
For long-time Visual C++ developers, Microsoft's introduction of MSBuild will look very much like a back-to-the-future move. Going back in time a fair way, MAK files were the traditional way to record the steps required to build a C++ project. The MAK file was essentially a glorified batch file that allowed command line tools to be executed in a particular order. Despite their apparent simplicity, MAK files could support a quite complex workflow; debugging and disentangling complex MAK files became a black art.
Early versions of Visual C++ sat on top of these MAK files. You could use Visual C++ to open, compile, and debug a custom MAK file, depending on how complex it was. Eventually, Visual C++ stopped using MAK files and switches to a custom text file format. This custom text file format lasted all the way up to Visual C++ 6's DSW and DSP files. These files could be hand-edited, but they were fragile and prone to corruption.
One of the other real limitations of the build process in the current version of Visual Studio is that extensibility was limited to the start and end of the build process. During the actual compilation phases, extensibility is extremely limited, and confined to a few small tweaks that had been explicitly made available by a particular compiler.
The final area where the current build and file system was starting to show its age was in the way extensibility was implemented. As developers well know, batching out to a new process and passing configuration parameters via the command line is a lot less robust from an error-notification and event-coordination perspective than using an in-process component. Each command-line utility also tended to use its own interface, making the exercise of building pre- and post-build batch files needlessly complex.
Today's Automated Build Tools
The obvious question then becomes: "if these limitations and difficulties with the build process in today's Visual Studio are so significant, why hasn't someone else been working on a solution?" The answer is equally obvious: "they have." Nant is an open-source .NET port of the popular Java Ant build tool. The fundamentals of NAnt are explained here. To briefly introduce NAnt, it uses an XML build file that has a single project, which would typically be an executable, and various targets, which would be logical steps in a build process such as clean, compile, link, run unit test, or deploy. Each target has a set of tasks, which equates to a specific piece of code being run. NAnt is extremely configurable and extensible, and can even be used to build C++ projects (see John Lam's excellent entries on the subject here and here).
NAnt has two main issues: Crossing over between the NAnt world and the Visual Studio world is not a trivial exercise, and while there are a number of GUI tools available, NAnt remains a text-file-centric approach to creating a build process. These limitations can be overcome with enough effort, but the NAnt experience is still not as polished as the Visual Studio IDE. Whether overcoming NAnt's issues is worth it, especially considering some of the other options available today (covered below), is a point of debate.
NAnt's difficulties (perceived or otherwise) have given rise to two great commercial offerings that allow a build to be fully automated: Visual Build Pro from kinook and FinalBuilder from AtoZed Software. These two products are quite similar. Both basically offer the ability to construct a software production process that spans many more tasks than simply building a DLL or EXE. The products have inbuilt adapters for all of Visual Studio's project types, all the major installation and documentation vendors project types, support for legacy projects still using Visual Studio 6 project files, and a wealth of common tasks like FTP, file, and directory access and e-mail notifications. Both products are very reasonable in cost, and offer the best entry path into the world of automated builds before Visual Studio .NET 2005 ships.
MSBuild is conceptually very similar to NAnt. As with NAnt, the top-level XML element in an MSBuild file is Project, under which there are four key elements:
- ItemGroup—specifies input items into a project, such as references and source code files
- PropertyGroup—specifies key-value pair that provide properties for a project, such as whether compiler optimizations are turned on
- Tasks—components that actually carry out the build process, such as compilers and utility components to write a log file when a project has completed (Custom tasks can be written to complete any computational task, and this feature is extremely important from a C++ perspective, as the next section shows.)
- Targets—a grouping of tasks that results in the production of an output, such as the generation of a compiled assembly or a full MSI file complete with documentation
MSBuild offers some key features not available with today's build tools:
- A great GUI tool for producing MSBuild files in the form of Visual Studio
- The ability to integrate deeply into the compile process
- Seamless extension of the build process
- The ability to build Visual Studio projects on machines without Visual Studio installed (This is particularly beneficial for organizations that have dedicated build machines because it means a Visual Studio license will not be required for these machines.)
- A standard mechanism for the provision of build configuration information rather than ad-hoc command-line parameters
- A standard XML schema defining the build file format, which is an advance over today's Visual Studio .NET project files
- A highly scalable build engine that Microsoft is stress testing with many of its own products
The C++ Story
This is the section of an article that a C++ author hates to write. Visual C++ support for MSBuild is coming, but not as soon as we'd like. For the Visual C++ .NET 2005 release, the project file format will be similar to the XML-based structure used in Visual C++ .NET 2003. The reasons for the lack of inbuilt MSBuild support are varied, but the main issue for Visual C++ will continue to be the need for the Visual C++ team to spread their resources between managed and native compilation, and the need to ensure that technologies such as MSBuild that come from the managed side offer adequate support for all the native features that Visual C++ continues to offer. The amount of work completed by the Visual C++ team during the Whidbey development cycle has been Herculean, and the fact that MSBuild support is not fully there in the next version is certainly understandable.
The lack of direct support for MSBuild is, thankfully, not the end of the story. (I wouldn't have devoted the entire article to MSBuild if it had no relevance to us C++ folk!.) A tool called VC Build is available today from the Visual C++ .NET team site on GotDotNet. It "enables you to build Visual Studio® .NET 2003 solutions that contain Visual C++ projects, as well as stand-alone Visual C++ projects." This tool also has a Visual C++ 2005 version, and will be wrapped by the MSBuild task, which is also called VCBuild. The VCBuild task allows the compilation of a Visual C++ project to be completed as part of a more complete build, and because the maturity of the C++ compiler offers much richer customization ability than the newer C# and VB.NET compilers, most of the benefits of MSBuild are available to a C++ developer using Visual Studio .NET 2005. The major issue with the Visual Studio .NET 2005 story is not the range of flexibility it offers, but rather the difficulty in managing both VCPROJ and MSBUILD files.
Microsoft's long-term plan is to make MSBuild The One-And-Only Build Tool from Microsoft, and to phase out VCPROJ in the version or versions of Visual Studio .NET that follow Visual Studio .NET 2005.
The Build Process Comes a Long Way
MSBuild offers some great functionality and extends the concept of the build process a long way out from the traditional link and compile steps. Source control, unit testing, installation file creation, help file generation, automated deployment, and project management recording can be easily integrated into the build process.
Visual C++ won't be fully integrated with MSBuild in Visual Studio .NET 2005, but the extensible nature of MSBuild and the VCBuild tool mean that C++ developers can take advantage of MSBuild without waiting for Visual Studio .NET (codename Orcas) or beyond.
Next month, I hope to have Beta 2 in hand, which is traditionally the point in the beta cycle where feature sets are trimmed down to size and frozen. I'll check out any changes in Beta 2 as they related to the IDE, and then continue the ongoing IDE tour.