Full Text Search: The Key to Better Natural Language Queries for NoSQL in Node.js
How many times have you sat is a design meeting and wondered, "What the heck is this person talking about?" For me, the number is more than a few. Everybody seems to be using the same words, but they're all talking about something different. Eric Evans has a way to solve a problem. In his book, Domain Driven Design, Evans is emphatic that a vital part of the software development process is the creation and continuous use of Ubiquitous Language. Evans is no dope. He's onto something.
Getting Lost in Translation
As I've said above, I've been on more than one project in which business people and developers use the same words, but in reality are talking about completely different things. Or, they use different words to talk about the same thing. As result, a lot of time gets spent in translation. And, as with any translation, things get lost. Translation costs time and money, and can affect the quality of the product being made. To quote Evans,
"The overhead costs of all the translation, plus the risk of misunderstanding, is just too high. A project needs a common language that is more robust than the lowest common denominator. With conscious effort by the team, the domain model can provide the backbone for that common language, while connecting team communication to the software implementation. That language can be ubiquitous in the team's work."
Domain Driven Design, page 25
Evans' solution is to create a Ubiquitous Language that is used all the time, by all members of the software development project, from executives to subject matter experts to developers. A Ubiquitous Language is more than just a set of agreed-upon phrases and anagrams. A Ubiquitous Language is the way that the team communicates with one another. The Ubiquitous Language describes exactly the domain that is being modeled. Creating and using a Ubiquitous Language is not some semantic mumbo jumbo. Software that is made in a domain that is well described by a Ubiquitous Language has a better chance of meeting the expectations at hand than software that is made under nebulous circumstances. Using a Ubiquitous Language counts.
Moving Past the Semantic Mumbo Jumbo
Using a Ubiquitous Language is very real world. Let's look at a case in point. Imagine that we're a software development company that has been contracted by the US Navy to make some software that registers vessels coming into a seaport. You are a developer in a design meeting and a naval officer says the following,
"The boats arrive on a weekly basis."
Simple statement, right? Wrong. I'll wager good money that the developer is thinking of a boat as this thing that travels on water. The naval officer on the other hand is thinking of boat in relationship to a ship, that a boat is a smaller vessel that can be stowed on an ocean-going vessel, a ship (see Figure 1).
Figure 1: Two parties can conceptualize the same work differently
Now, this might seems like an issue of trivial semantics, but it's not. If my assumption is true, and let's pretend it is, the result is that the one word, boat, is referring to two different concepts, one by the sailor, one by the developer. Now, here's the gist. The developer is going to create code that describes his or her concept; at the same time, the sailor is expecting software that conforms to the naval definition. This is only one example of a mismatched conceptualization of a word. A software development project might have thousands of words in play that might have entirely different concepts attached to them among the participating parties. Unless a Ubiquitous Language is used, developers will code to their concepts, Q/A personnel will test to their concepts, and users will expect software that meet their definitions.
Remember, please, this is software. We build software according to the domain model. If the model is well understood and well defined, the software will be accurate. If the model is a collection of amorphous concepts, in which each concept is as different as the mind imagining it, well, you can take the result to the logical conclusion. The software will be fragmented and faulty.
So, what's to be done? How do we create and implement a Ubiquitous Language?
Talking Won't Do
There are two parts to implementing a Ubiquitous Language. The first part is getting all members of the development team—Subject Matter Experts, Project Managers, Business Analysts, and Developers—to agree to use the Ubiquitous Language, always. Evans calls the approach, One Team, One Language. The second part is actually using the Ubiquitous Language.
No doubt about it, getting a team to agree to use the Ubiquitous Language is hard. Most people want to go about doing business the way they've always done it, whether business as usual produces the desires the desired results or not. And, if you ask people to actually talk differently, well, we're asking for world of hurt. Evans' position is that the team needs to use the Ubiquitous Language verbally. Me? I am not so ambitious. I've rarely seen a complete adherence to a common verbal language succeed. The depth of historical entrenchment is too deep. However, language is not just about utterance. Written language can provide us the with the starting point for the communication we need. When it comes to having a written component of a Ubiquitous Language, I am a big supporter of using class, use case, and workflow diagrams (see Figure 2).
Figure 2: Using diagrams as the written part of the Ubiquitous Language is a good start
Later on, as the team takes shape, we can migrate to a more comprehensive diagramming language, such as UML. Then, as the the written language becomes entrenched as the way of communicating, using verbal language becomes easier. Analogically, as a novice speaker, it's a whole lot easier for me to read German with a good dictionary handy than trying to figure out a real-time conversation in a room full of German speakers.
Creating the Domain Dictionary
Before we can use language, we need to establish a vocabulary. That vocabulary is defined in a Domain Dictionary. A Domain Dictionary is a listing of all terms, with definitions, that are used within the domain. The purpose of the Domain Dictionary is to remove any ambiguity about terms that are used to define the parts of the model. A Domain Dictionary is a living document, the accuracy of which is determined by absolute agreement among all members of the team. Will there be conflict when going about the task of defining a term? Yes. Does subject matter expertise matter? Yes. However, conflict when creating a Domain Dictionary is not a bad thing for the team to experience. It's by conflict that a true, accurate description of the domain is defined.
Where do I store my Domain Dictionary?
A Domain Dictionary is a living document that will be subject to a lot of change at the beginning of the given software development cycle. Typically, there is a maintainer of the dictionary, whose job it is to make sure that entries are made and that the document is formatted well for easy reading and understanding.
There are a lot of ways to store a Domain Dictionary. If your Domain Dictionary is a sensitive document, you'll do well to use a tool housed internally behind a firewall and subject to authentication. You can use a tool such as Confluence, SharePoint, or an Open Source product such as WikiMedia. A Domain Dictionary that does not have a stringent security requirement can be a shared Google Doc.
Your Domain Dictionary should not be a series of email exchanges. Not only is such a documentation method unprofessional, it's inaccurate. Items get lost and updates are difficult to make and monitor.
Working with a Written Ubiquitous Language
As mentioned above, my position for the fastest way get a Ubiquitous Language to be a vital part of the team's software development practice is take a written approach and start out by using class, use case, and workflow diagrams. Thus, a certain amount of education is going to have to take place. As with any language, you need to learn how to speak it. There is a formal aspect to class, use case, and workflow diagrams that need to be mastered. So, we come to training. Yes, all members of the team need to be able to write proper diagrams. There are a number of ways to facilitate the learning. You can have formal classes led by a person expert in the diagramming techniques. (Yes, team members can share knowledge about diagramming informally. But in terms for formal education, sessions led by an independent professional are best. Using an independent professional keeps the Mr. Know It All Factor down.)
You can use YouTube videos should your enterprise not have the resources to provided professional training sessions. The important thing is that everybody on the team have professional facility with class, use case, and workflow diagrams and that these diagrams serve as the language that defines the domain.
But, what about people who think this is hogwash, that they have neither the time or desire to learn a new language, albeit a diagramming language? Here is Evan's response:
"A project faces serious problems when its language is fractured. Domain experts use their jargon while technical teams members have their own language tuned for discussing the domain in terms of design.
The terminology of day-to-day discussions is disconnected from the terminology embedded in the code (ultimately the most important product of a software project.)"
That most software projects fail is one of the dirty little secrets in the world of software development. If you find you are faced with personnel who do not want to make a commitment to learn the skills required to create accurate, understandable diagrams, you have a fundamental problem at hand that is not going to be solved later on down the line. You are either on the bus or off the bus. If your team doesn't have the skills to diagram a domain model, the domain is not understood. You cannot code a domain that is not fully understood.
Do all software development teams need to use a Ubiquitous Language?
It depends. If you are on a small team, say a product manager, two or three devs, a Q/A person, and a business analyst, and you all have been working together shipping software successfully for a year or two, you probably have your Ubiquitous Language in play already. You know your domain and are able to talk about it consistently and accurately. Usually, such teams are found in startups or very small businesses.
However, large teams, in large companies, where you might have twenty to a hundred people working in a variety of teams under a single domain will do well to take the time to formally commit to and use a Ubiquitous Language. Large domains, by nature, are hard to understand, with a lot of different people seeing things many different ways. Unifying the understanding of the domain is a key indicator for success. With millions of dollars in play, taking the time to develop, master, and use a Ubiquitous Language is a small upfront cost to incur in the scheme of things.
Should all team members be able to read and write a class, use case, and workflow diagram? Should the team use a common Domain Dictionary? Let me respond with a question. Should all college students, from pre-med to performing arts to hospitality management be able to write a proper term paper? If you are going participate in the discipline of making software successfully, there are certain skills you need just to get through the door.
Short answer: Yes, all team members be able to read and write a class, use, case and workflow diagram.
Stay the Course
Committing to using a Ubiquitous Language is not easy. If it were, most teams would do it. However, please keep in mind that shipping software isn't easy either. The road your team travels will have lots of bumps. Your Ubiquitous Language will change as the domain model is understood more fully. You'll find inconsistencies in the model that will need to addressed. However, just because the model changes doesn't mean that we abandon the use of the Ubiquitous Language. Languages evolve as the world evolves. Keep in mind that up until very recently the word, friend, was used primarily as a noun. Then, Facebook can along. Did we stop using English to talk about that how we create relationship with someone on Facebook? No, we extended the language so that friend can be used as a verb. Today it is perfectly accurate, "I friended him." The same will be true of your team's Ubiquitous Language
Creating, using, and evolving a Ubiquitous Language will go a long way toward making your team known as a team that ships well integrated software that works and meets the expectations at hand. And, just as importantly, your team will serve as a model to other teams that are looking for a better way to create clarity, cohesion, and consistency in the pursuit of making software that matters. Leading by example is the best way to lead. Using a Ubiquitous Language is the type of leadership that will live on, well after your code ships.