Building the Right Environment to Support AI, Machine Learning and Deep Learning
Why doesn't Moore's Law apply to software development? If it did, every 18 months or so applications would be twice as fast and/or be built in half the time. The reason Moore's Law or any ordered progression of improvement doesn't apply to software development is that very few people are working on intellectually improving it and almost no one is applying the results of intellectual studies to it.
For instance, Steve McConnell's Code Complete, Second Edition presents some very reasonable principles. But, as I go through it now, I wonder if any manager has read it, and if so, why none of its ideas are being applied.
Where is the emergence of specialized roles? Where is the upfront design? Where is the storyboarding of the implementation cycle? Where is the consistent use of patterns and refactoring? Why is everyone still trying to figure out how to build a custom persistence (database) layer? Why are programmers, architects, and analysts expected to all start on the same project on the same day? What are the programmers doing while the architects and analysts are architecting and analyzing? How come the promotion path for technologists is always from technologist to manager when these are two different disciplines?
In this editorial, I will try to answer some of these questions by describing a scenario where software can be designed, programmed, and assembled as quickly and inexpensively as possible. How quickly? Well, you will have to try this approach until the basic tenets become second nature and then tell me.
Prerequisites to Building Software Fast
This article is really about the software building process, not necessarily the definition process, but you have to define what you're building before you can build it quickly. Along that vein, here are the prerequisites to building software as quickly as humanly possible:
- Use a team that has experience working together.
- Roles must be highly specialized.
- Do good, advanced design work.
- Construct a solution by risk and importance.
- Storyboard the construction plan.
- Use a macro schedule for the team and a micro schedule for individuals.
- Specialists must work in parallel, building to black boxes.
- Keep black boxes simple.
- Insist on unit and system testing.
- Manage change aggressively.
Depend on Experienced Teams
Building anything with a new team carries a huge risk factor. Try to avoid building anything important with a new team. If you have a new team, build something disposable in isolation. This will allow members to learn to work together before they have to succeed.
If the team members play well together, they will work well together. If one or two talented members do not work well with the others, isolate them geographically. Also, always assume that the boss is a dysfunctional team member and move him or her to another floor or building.
When your team does work well together, try to keep them together. People will leave, however, so carefullybut continuouslyaudition new members for the team. This will prevent a successful team from ever totally dissolving.
A generalist makes a good spouse, but only a marginal technologist. In a spouse, it is useful to have someone who makes a little money, can cook and clean so you get a break, and is a good parent so the kids get good values from both parents. Among your development team, generalists are less useful.
Whereas having solid general skills is good, working in multiple roles at the same time isn't as good. The reason is that people are likely to be very good in some areas but marginal in others and they will work in earnest at the things they really like to do. The result is generalists will appear to be average performers instead of superstars because of the moderate to low performance in their areas of lowest interest. In contrast, when a person is permitted to work with undivided attention in an area that is a labor of love, he or she will perform at a much higher level.
Specialization is not an excuse for C# programmers to be ignorant of SQL, for example; it is permission to focus on doing the C# work intently.
Advanced Design Work Is Critical
Advanced design work means that architects, analysts, and designers get together and do a bunch of analysis and design work before the programmers are invited to the show. Think of the design team as the party planners: Plan the party and hang the streamers before the programmers start swilling beer and eating all the hors d'oeuvres.
I am not advocating one-size-fits-all design. Small projects can use a simple outline with an organized statement of work, and big projects will need UML models and other documentation, which might include prototypes. However, a significant amount of this work should be done before the programmers are summoned.
Construct by Risk and Importance
The hard and simple fact is that projects often run out of time, money, or both before 100 percent of the original feature set is implemented. An overlooked fact is that in many instances a lot of what is built is fluff. Prioritizing by importance and risk gives your team a very good chance of building the critical subset of features before your project runs out of gas.
An important additional concept here is that if the software looks promising and solves most of the business case, management will support funding the project's second wind, which means a greater likelihood of getting the whole feature set in production.
Storyboard the Construction Plan Publicly
Movies are greata story that one or two minds conceive takes on three-dimensional life in celluloid and entertains thousands or millions of people. How does Hollywood do it? Storyboarding. The scenes, cast, characters, location, and equipment use are all planned using something that approximates to a cartoon strip. The same technique works for software too. Let me explain.
Suppose you want to provide good technical documentation for your next killer app. It would be helpful to coordinate the technical writer's time around the project's production schedule. If the writer won't be needed until the end of the project, why pay for the writer to be sitting around in the meantime? Suppose your best DBA works in another city. Wouldn't compiling a list of tasks for him to complete during the two days he's working at your location make sense?
Storyboarding is a good way to allocate resources and manage dependencies. If you are lucky enough to hire a graphics artist for your GUI work, you could create an actual visual storyboard. Besides being cool, it'd be a great way to get team members to notice and pay attention to the schedule. At a minimum, schedule tasks in a very public and visible way.
Define Macro and Micro Schedules
Use a macro schedule of milestones for the team and make a big deal when the team reaches them. Failed projects fail one milestone at a time. It is also important that the team's milestones are big enough to help foster an atmosphere of excitement and success.
If a milestone is missed, make a big push to complete the milestone and then proceed to the next one. It is better to push to a milestone and then go back to a normal effort level than to promote 100-hour workweeks all the time. A culture of overtime is a terminal disease; the cure is better planning and short bursts of extra work only when needed.
Frequently missed milestones are a function of mismanagement. Better estimating procedures are needed, not longer hours.
For technologists, use micro schedules that are no longer than about a day. Have each technologist estimate his or her own subtasks and then hold him or her to it. New technologists on new teams using new tools will miss, but experienced technologists on old stable teams with familiar tools will provide good estimates. Reward good estimates and discourage the age-old habit of padding to death.
Discourage the practice of permitting schedules to slide one day at a time. If a little overtime is needed to catch up, so be it. If a technologist is always working late, you need better scheduling, more training, additional helpers, or a combination of all three. This is a function of good management too.
If programmers are rewarded for shorter turnaround times and real improvement over historical performance, you can build a culture of personal improvement.