Over twenty years I have worked on a lot of projects. Being an author, columnist, and consultant I have heard about many more projects than I ever could have possibly worked on. The most common, glaring, obvious fact is that a great deal of software that is created from scratch never sees the light of day. Of the software that does get used by real customers, very little of it got to customers on time, on budget, or with the originally promised features.
The conventional wisdom has been that software engineering is hard, and that every application represents something new in the Universe. So it has always been, so it will always be.
There is a secret organization that hardware engineers belong to called the IEEE, and they have a dirty little secret that they haven't shared with software engineers. The secret is that the power, freedom, and choices need to be eliminated. Why haven't they shared the secret? Because most software engineers haven't passed a standardized test, they haven't learned the secret handshake, and they haven't demonstrated a willingness to take a considerable pay cut.
Unlike the Illuminati, the IEEE exists and they hold the key to producing software more cheaply, more reliably, and faster than most software project managers could ever dream.
Twenty Years in the Knowing
I have been consulting, writing, and presenting to companies and for programmers for twenty years now. During some of that time I have worked on projects that directly interact with or reside on devices that aren't exactly computers. To do so I had to actually read hardware specifications for those devices, and I found out hardware's dirty little secret: device manufacturers rarely build anything from scratch. Hardware is assembled only from pre-existing parts and put in a shiny box. No one ever looks inside the box and few ever look at the actual blueprints of the things in the box. All the end user knows is the box is shiny and it does something like wash clothes, play music, or record television. To put it succinctly hardware is the same old parts used over and over with different shiny boxes.
When an electronic engineer (or any engineer not a programmer) graduates from college -and graduate with an engineering degree they must-they join the IEEE. (You can Wikipedia IEEE for the acronym; that's not important.) (Anyone can be a programmer. They let Psychology majors write code.) When one joins the IEEE one takes a pledge to work for moderate pay, never invent anything new when something exists that will do the job, and to never ever start a company from their garage.
Once they have taken the pledge the new member is given a 27 page booklet. The booklet has all of the chips, parts, resistors, capacitors, and acceptable wire formulations deemed by the IEEE as being "acceptable". The parts for everything to be built now and in the future must already exist in the booklet. On a very rare occasion something new is added to the booklet, but that new thing is voted on by secret, senior priests of the IEEE and then everyone gets a new booklet. It is important to note that only senior IEEE priests get to add anything to the list of "acceptable" things and this is done only rarely.
In software anyone with a keyboard can invent something new and with the Internet, email, and version control that something new ends up on other people's desks pretty quickly. So much creativity is confusing. If there were a software booklet it would be 27,000,000 pages long and highly redundant. The difficulty with software engineering is that there are no high priests, everyone gets a say in the creation process, and everyone has a different opinion.
How Hardware is Built
The upshot is that when hardware is built the hardware engineer goes to a cabinet and much like preparing dinner chooses from the equivalent of chicken, pork, beef, or fish and a couple of vegetables and makes dinner. In hardware every meal time there is a meal.
Sorry, where was I? I got lost in my metaphor. Oh yeah!
Hardware engineers know several things. First, they have to assemble something to put in the box. Second, they only have a limited number of things to build from. And third, the thing will be packaged in a shiny plastic case, shrink wrapped, advertised on TV or placed in a Happy Meal, and people will purchase it. Shiny new hardware things can be created so fast and so cheaply this way that the IEEE and the hardware guys don't even care that this system by its very nature is hit or miss.
The key to mass producing shiny hardware things is never invent new parts, even if it means you put in a part that does 100 things and only one is needed, re-organize the parts into different configurations, and change the shiny box (making it smaller if possible). (In 1978 I had a Zenith stereo that played 78 RPM record albums, it played music. In 2009, I have a wafer thin iPod (sorry Zune guys), but it just plays music. Someone just figured out how to use fewer things from the parts catalog, but most of them existed in 1978.)
How Software is Built
Continuing my meal metaphor, when it comes to software everyone misses meals, all kinds of meals. Software is like an Ethiopian famine. When it comes to software and meal time there is no fish, pork, beef, chicken or vegetables. There is no kitchen, no utensils, and no structured meal time. At meal time the average software engineer is talking about how molecules can be assembled to make proteins. (I must be hungry.)
Now, as programmers, this state of affairs is not really our fault. Programmers are smart, creative, have no controlling body or high priests. Programmers labor in a state of lawlessness where the biggest brain or the loudest voice wins the day. Programmers have too many choices, too much power, and we are just too smart for our own good.
Software eventually will be assembled more from existing parts. Individual factions of programmers won't be asking themselves questions like should I use ADO.NET, SQL, LINQ, XPO, or something else to get data to and from the database. There will be one database data-getting thingy and everyone will use it. Our lists of choices will be smaller, we'll accept lower pay, we will swear a secret oath to the high priests of software engineering, and from that day forward software projects won't fail. Right?
In the mean time, try to build applications from code that already exists, from technologies that that you already know how to use, and that are sufficient to do the job, even if you have to mix and match a little or use something that does more than you need it to do. You will get to build more shiny software thingies if you do.
About the Author
Paul Kimmel is the VB Today columnist for www.codeguru.com and has written several books on object-oriented programming and .NET. Check out his upcoming book Professional DevExpress ASP.NET Controls from Wrox Press coming for by PDC 2009. You may contact him for technology questions at email@example.com. Paul Kimmel is a Technical Evangelist for Developer Express, Inc, and you can ask him about Developer Express at firstname.lastname@example.org and read his DX blog at http://community.devexpress.com/blogs/paulk.