How to Write Software Right

I have been working on, testing, developing, breaking, engineering, architecting, designing, and creating software for over twenty years. Seven and a half of them have been professional years. I have been in school, participating in Computer Science related activities for seven of my twenty years. I have ready thousands of books, essays, tutorials, forums, blog posts, and reports on various subjects related to all aspects of the IT industry. I have worked with eight companies, on five open community projects run by others, more than ten open source projects of my own, with hundreds of coworkers both domestically and abroad, and on several contracts both of my own initiation and because others wanted my help. Also, I participate casually in four local programming language and API user groups, a couple of global API user groups, and have attended Microsoft Tech Ed, and Agile Roots conferences. My point is that I have worked a fairly large percentage of my life on software, and I have experienced a lot of different economic situations, team and solo environments, and software communities. And in more than eighty percent of my overall experience, the resulting software has been unacceptably buggy, politics play a greater role as to when software is complete than discrete quality does, programmers don't have time and in many cases don't have the experience to engineer software properly, and management, of both software activities and people, is poorly executed, in part due to lack of understanding the problems being solved, the solutions to those problems, and also and again due to lack of experience.

The Problem, As I See It

The problem as I see it, has a lot to do with management. As projects grow, management becomes a greater necessity. This is well-founded, but projects need the right kind of management. The right kind of management is an aspect of Software Engineering that in my mind has yet to become fully understood. MBA students learn about strategies for being successful in the marketplace. While these strategies work well in supply-demand situations where the product is a physical, tangible commodity, the same strategies do not work as well when the product is abstract and service-based, as is the situation with software. Software does not follow supply-demand rules in the same way as commodities like gasoline, vegetables, and fast food. In software, the commodity isn't what the end-user gets, rather it is what the producer has. In software the commodity is not the software, it is the programmer, the software architect, the technical writer, the software engineer, and the software quality assurance professional. And that commodity needs more than just carnuba wax to preserve it. Programmers need training, motivation, and benefits. Good software engineers need to know that they are trusted and entrusted with the final product. Software architects need to be listened to and adhered, not downplayed and criticized. I know a good number of these commodities that are extremely well-versed and know what they're doing, but when their skill-set is undercut by the bottom line, then their performance suffers, and as a result so does the product that they are working on.

A sub-problem of the management problem seems to me to be the bottom line, and while just like any in other commercial industry, the product must be profitable to survive the market, the commodity, the worker must be able to engage in the highest possible quality work to produce a profitable product. When a consumer purchases an engineered product other than software, such as an HD television, an electric mixer, or an automobile (excluding the software-driven components), it is expected to be of the highest quality.

A Case In Point

Nine years ago, my wife purchased a two-year old 2000 Chevrolet Malibu sedan. Chevrolet is generally a good company that produces high quality vehicles at an astounding rate and at a lower cost to the consumer than many companies. On the other hand a company like BMW has been producing superiorly engineered high quality products for years as well, but their vehicles cost significantly more than a Chevrolet. As an example a 2010 Chevrolet Malibu four door sedan has an invoice cost of $\$$20,733, while a comparable 2010 BMW 3 Series four door sedan has an invoice cost of $\$$30,500, costing the consumer about 47% more at the bank. The BMW 3 Series is a quality vehicle. The Chevrolet Malibu is a quality vehicle. And while I have never driven or owned a BMW 3 Series, somehow I expect that it has never had any of the engineering problems that my wife's 2000 Chevrolet Malibu has experienced: turn signals stop working unless you jimmy the emergency signal button (why? because the connection between the turn signals, emergency signals, the emergency signal button, and the turn signal wand was soldered together), brake pads and rotors need to be replaced every three to six thousand miles (professionals expect to change brake pads and rotors on vehicles around 20,000 to 50,000 miles depending on usage, but on my wife's Malibu they really wore out that fast, and we were purchasing performance grade pads), and finally several hose fittings to the cooling system burst, costing hundreds of dollars in repairs, hundreds of dollars in coolant, and a highly pressurized system that continued to fail. Mechanics were finally able to solve the turn signal issues with my persistent calling to GM and complaining about it. After reading on the NHTSA's web site about how many people had complained about it, and how many had received notice of a recall, I decided to find out why I hadn't received notice of a recall, and their cop-out answer was that my vehicle hadn't been manufactured at the same plant as those vehicles that were recalled for the exact same turn signal problem, same year, make, model combination. After several months I finally received a letter stating that they would repair the problem for free at a certified Chevrolet mechanic, but that my situation was not considered a recall. What was the real reason for the problem? In one word, management. Either somebody didn't manage the quality of the product at its inception, or somebody didn't manage the quality of the product at production, or both. Either way, I'm certain that any electrical system vehicle engineer would tell you that a soldered connection would not last very long under the stress that a motor vehicle would put on it. Whether they were given time to engineer it properly is a good question. But on the front of vehicle manufacturers, quality appears to be suffering more and more. Especially recently were several Toyota recalls and similar recalls and issues by other manufacturers over the years. These types of issues are problems that can be solved by careful engineering, and quality assurance through testing and continuous integration. Motor vehicles should be higher quality than this, especially for as expensive as they are, but also because lives depend on that kind of quality.
Most of the time software isn't so system-critical, but sometimes it is even more system critical. Imagine if a commercial airliner suddenly lost altitude because the autopilot software thought is was nearing time to land, while in fact the airliner was flying in the middle of the ocean with no place to land, or over a jagged mountain range. Imagine if a space shuttle computer suddenly locked up causing a fatal crash due to a software defect. Luckily that didn't happen. Imagine if an automated cannon suddenly discharged on its own military troops due to a software defect, killing some. While most of us in the IT industry don't work on projects of the life-threatening system-critical nature of these incidents, some do, and we shouldn't be blaming anybody but ourselves when these sorts of costly problems occur. If management is getting in the way of quality, then we need to step up and let them know, and for those MBAs out there, you need to know that the world doesn't happen according to a text book.

Acceptable Defect Density

"The software needs to get released at some point,"

your managers are certainly saying. And I completely agree. If you don't release software, then you won't be able to get a return on it. So is there an acceptable number of defects or issues that production software can contain? I don't feel like it's fair to lay down an IT-industry standard, rather each industry using software to control any portion of their products must define their own standard. Better yet each company must strive to the highest standard possible. In my opinion zero defects is the most acceptable number of defects in production ready software. It is pretty unlikely that all defects can be averted in any software project, but I believe that a 99% defect free rate based on the size of the system (however you choose to measure that, i.e. defects per lines of code), is a realistic goal. How do we attain a 99% defect free rate? The answer, though simple in my opinion, is not necessarily easy to implement. The answer is that you need to fully engineer software. "Fully engineering" means that all parties involved in creating the software need to participate in analyzing the problem you are trying to solve and the risks associated with it. Management needs to be there so that they can help make the problem clear. Project management needs to be there so that you can understand what kind of resources are available to solve the problem. Developers, architects, engineers, and technical writers need to be there so that the project is understood by those who are actually solving the problem and doing the work. And the problem and solutions need to be discussed and developed until the end goal is clear in everybody's mind. Now I don't mean have meeting after meeting and fully document every single caveat or problem situation possible in the system, but don't be satisfied with the highest level of explaining what is wanted. Give enough detail that questions can be asked. Answer the questions in a way that satisfactorily resolves any concerns. And always leave your door open in case issues arise during the development process. After that comes the development, and then come the real challenges. It has been said that in order to fully understand how to solve a problem with software, you must begin solving it. Even with tools like programming language knowledge, computer architecture knowledge, design patterns knowledge, and all of the experience and wisdom in the world, every problem solved with software is different. You might even come to solve the same problem in a different way given the nature of the reason for solving the problem.
Back to defect density in the real world, Steve McConnell, author of Code Complete, and his blog IEEE Software, says that [Bibliography-1]

"[A software company's] task is treacherous, treading the line between releasing poor quality software early and high quality software late. A good answer to the question, 'Is the software good enough to release now?'" can be critical to a company’s survival.

Once again the problem with the software industry is fully exposed. Why can't we just write software correctly? I recently read a compilation of reports organized for the United States Air Force pertaining to systems engineering needs by Edward R. Comer of Software Productivity Solutions, Inc. It is not surprising to me that their conclusions are the same as the conclusions of most of the industry. For example in the Needs Survey of the "1975 NRL Navy Software Development Problems Report", the "result of a year-long investigation into Navy software problems...based on interviews with...people associated with Navy software development" the following twelve recommendations were made [Bibliography-2]:

1. Unify life cycle control of software. Development responsibilities for a system should not be split, and maintenance activity should not be independent of development activity.
2. Require the participation of experienced software engineers in all system discussions. This is especially crucial for early decisions such as the determination of the system configuration, assignment of development responsibility, and choice of support software.
3. Require the participation of system users in the development cycle from the time requirements are established until the system is delivered. Changes which are inexpensive and easy at system design time are often extremely expensive and difficult after the software has been written.
4. Write acceptance criteria into software development contracts. This will help avoid unnecessary misunderstandings and delays for negotiation before a system is delivered.
5. Develop software on a system that provides good support facilities. If necessary, consider developing support software prior to or in conjunction with system development.
6. Design software for maximum compatibility and reusability. Premature design decisions should be avoided; logically related systems should have their differences isolated and easily traceable to a few design decisions.
7. Allocate development time properly among design, coding and checkout. Since manpower-allocation estimates are based in part on the time estimates for different phases of development, improper estimation can be quite expensive.
8. List, in advance of design, all areas in which requirements are likely to change. This can be done at the time requirements are stated and will help the designer partition the software to isolate areas most likely to change.
9. Use state-of-the-art design principles, such as information hiding. Principles which optimize reliability, cost reduction, and maintainability should be emphasized.
10. Critical design reviews should be active reviews and not passive tutorials. Sufficient time must be allowed to read design documents before the review, and the documents must be readable.
11. Do not depend on progress reports to know the state of the system. Programmer estimates are typically biased; milestones are a more accurate indication of development progress.
12. Require executable milestones that can be satisfactorily demonstrated. Milestones demonstrating system capabilities that will rest on major design decisions should be written into development contracts.

These twelve recommendations follow suit with what many people in the industry state are needed, and what many software development processes try to provide. Perhaps the problem with software processes however is the same exact problem that generally exists with software, there is no silver bullet! I have worked with several companies and groups who have use Agile as part of their process. One of the attractions of the Agile software development process is that there are several different methods of implementing it. Back when Agile was young, there were many schools of thought pertaining to the Agile development methodology that said, "if you're not doing this, then you aren't Agile". More recently Alistair Cockburn basically told us, at an Agile Roots conference held in Salt Lake City, UT in July of 2010, that "Agile is agile", and there is no wrong way to implement the Agile software development methodology. More importantly, implementing only the pieces of different methods of Agile that are important to your organization is the best way to implement Agile. For example, make daily stand-up meetings part of your day-to-day process, and make visibility into your projects transparent. Those are two pieces of different methods of Agile. But again processes don't solve software problems, people do. And if a process or bureaucratic red tape are causing your project to fall behind, then get rid of them. Process are meant to help teams with little or no direction to make direction in my opinion. But teams outgrow processes on a daily basis. Making a team conform to a standard that isn't working for them holds back the potential of that team.

And Finally There's Quality

Software quality assurance is becoming a larger part of software in the industry. More and more various software development shops are opening up quality assurance departments in their companies. But this trend does not reveal that software is coming out at a higher quality, rather in underlines the issues that most companies have experienced in the past, and will continue to experience with quality. The companies building up their QA departments are just the ones who realize that quality is becoming a bigger issue than they can muster on their own, without dedicated resources.
How many companies would be willing to release their software into the wild even if it had a defect to lines of code ratio of 10%? What if your QA department didn't sign off on it and told you that you would lose more money by releasing it now with the 10% defect ratio than you would by releasing it late with a 1% defect ratio? Would you still release it? In my experience companies tend to release on time with a higher defect ratio than what is internally acceptable more often than not. So where is the problem again? QA is telling you not to release it. But why? Because they know that the risk of needing to refund customers or do non-billable work is higher. Well QA should be listened to now. Understanding assessments made by QA should not be regarded any less than making sure that your developers know how to develop, that your architects have designed a complete and well thought-out product, or that you're implementing time-tested design patterns that have been proven to be correct. If you have a QA department, then you probably realize that you have a problem with your software. It's also probable that you don't know where the problem is. It is estimated that it is far less expensive to design software well and completely in the beginning than to need to rely on customers to find your issues. It is still better if an internal quality assurance pass finds the problems before you release the software. If you have a QA department and you aren't listening to their advice, then you might as well not have a QA department, because now you're probably losing at least one and a half times the money - you're still relying on customers to find the issues before you'll fix them because you're downplaying the idea that what your QA department has to say is valuable. Of course there's always the other side of the coin that says that their utilization of a software quality assurance team is in order to find the defects that their design, unit testing, and code reviews didn't find. Steve McConnell provides us with some insight in his book, Code Complete [Bibliography-3]:

...software testing alone has limited effectiveness -- the average defect detection rate is only 25 percent for unit testing, 35 percent for function testing, and 45 percent for integration testing. In contrast, the average effectiveness of design and code inspections are 55 and 60 percent.

In conclusion engineering software is difficult. There are bad ways to do it, good ways to do it, and better ways to do it. My utopious dream of one day working on the perfect software project, may always remain a dream. But if we don't start working smarter on software than working harder, then there are going to continue to be huge consequences that are bad. Software isn't going away, and it will only continue to become more complex. Software will continue to be used in more and more applications. And software will only become more correct and defect-free if professionals in the software industry become better educated, and try harder to make software the right way.

---

Bibliography

1. Steve McConnell, "IEEE Software", [http://www.stevemcconnell.com/ieeesoftware/bp09.htm]
2. Edward R. Comer, "System Engineering Concept Demonstration, Systems Engineering Needs", Copyright © 1992, Software Productivity Solutions, Inc.[http://www.dtic.mil/cgi-bin/GetTRDoc?Location=U2&doc=GetTRDoc.pdf&AD=ADA265468]
3. Steve McConnell, "Code Complete", third party quotation from book: http://www.codinghorror.com/blog/2006/01/code-reviews-just-do-it.html, Code Complete 2nd Edition Home Page, On Amazon.com

Patterns and Practices: Scope and Type Inference Through Syntactic Sugar

Code Complete

Once again I'm referring to Code Complete. Code Complete taught me a lot of things and coding style is one of those that I hold important to this day. Let me preface this by stating that Code Complete was built around C++ and later C# coding, still I believe that utilizing coding style that is indicative of scope and usage makes programs more maintainable. I also believe that more maintainable code can more easily be proven to work and be proven to be correct. Many programming languages, especially dynamically typed programming languages, which are typically scripting languages, use syntax to identify scope and possibly type. Here are some examples:

From C++

#include <iostream>
#include <string>
using namespace std;

class MyObject
{
private:
 string name;
public:
 MyObject(const string & name);
 string getName();
};

// The :: token indicates the relationship of this function definition.
MyObject::MyObject(const string & name)
{
 this->name = name; // this indicates the relationship of the left name versus the right name.
}

string MyObject::getName()
{
 return this->name;
}

int main()
{
 MyObject object("Object 1");

 cout << "First object's name: " << object.getName() << endl;

 return 0;
}

From Ruby

class MyObject
 attr_reader :name # The : prefix indicates that name is a symbol.
 
 def initialize(name)
  @name = name # The @ prefix indicates the relationship of the left name versus the right name.
 end
end

object = MyObject.new("Object 1")

puts "First object's name: " + object.name

From Python

# Tabs indicate containers in Python - there are no curly braces or end statements.
class MyObject:
    def __init__(self, name):
        self._name = name # self and _ indicates the relationship of the _name to the class.
        
    def getName(self):
        return self._name
        
object = MyObject('Object 1')

print "First object's name: ", object.getName()

From Perl

package MyObject;

sub new {
 my $class = shift; # The $ prefix indicates that the variable is a scalar.
 my $self = {
  _name => shift
 };
 
 bless $self, $class;
 
 return $self;
}

sub getName {
 my($self) = @_; # The @ prefix indicates an array.
 
 return $self->{_name}; # The { and } dereference hash values by key.
}

package main;

$object = new MyObject("Object 1");

print "First object's name: " . $object->getName();
1;

Anyway hopefully you get the point that many languages hint in some way how to use certain keywords. C++, a heavily typed language, requires you to signify who a method belongs to. The main or global context is inferred, but a class context requires ::. In Ruby a single colon, :, indicates a symbol, and you can reference that symbol in a class via @symbolName. Python requires tabs to indicate contained code or code in a container, such as a function or a class. And finally in Perl (where everything is a string) there are several indicators or what kind a variable is or how to use it, for example $ = scalar, % = hash, @ = array. All of these things are essentially syntactic sugar.

...and Scope Inference

In Code Complete there are several hints to infer scope in a program. Some of what I am about to show you were developed somewhat more by a team I worked on recently.

Syntax	Conditions
PascalCasing	Namespaces Class names Method names* Properties and public fields
IInterfaceName	All interfaces
TGenericType	All generic types All template types
_camelCasing	Private class-scoped fields
__camelCasing	Private class-scoped static fields
camelCasing	Method arguments Method-body-scoped variables Method names*

^{* this may vary depending upon language}

Using these scope hints it is very simple to see what the purpose and scope of each object is. For example if I see a variable named _name then I instantly know that it is privately owned by a class, or in the case of the Python example I know that I should not change it manually (Python doesn't have a private class scope).

Code Quality

As I program more and more the quantity of files, classes, enumerations, structs, and fields increases dramatically. With that increase comes the ever increasing possibility of defects. When I am programming I try to follow a set of guidelines known as a programming style guide. Style guides help us to be consistent in whatever we're doing. It is probably more common for style guides to be used in artistic endeavors like in creating a magazine or newspaper, a text book, or even a web site. Somewhere where readability and usability matter. But wait, doesn't readability and usability matter in code as well? I believe that it absolutely matters in code.

IDEs

1. Using an IDE for all of your coding is one good way to increase code quality.

A lot of style is handled easily by using and IDE (Integrated Development Environment). IDEs like Eclipse, NetBeans and Visual Studio .NET take care of indentation, tabs, closing curly braces and many other important semantics while programming. One feature that I really enjoy about most IDEs (even Vim an Emacs do this) is the ability to ensure that tabs are always tab characters, but their length in spaces can be fixed for different developers. This ensures consistency in the way code is formed, but allows each developer to retain his own look, or rather the look of his code for his own purposes.

Code Complete

2. Reading about best and good, sound practices is another great way to boost the quality of your code.

Code Complete[1] is a great book that I read recently. Some programmers will say that Code Complete is good for some programming languages but not for others, but I don't think that is correct. While Code Complete was written around C and C++ programming, the practices described in it are good practices when coding in an language and most of them can be applied to any language.

The practices described in Code Complete entail things like when to write a method, how to formulate field names, how to ensure that you are using fields in the correct scope, how to write self-documenting code, and how to keep your code from being too confusing to other programmers, or you after a year or two away from it.

I advise all people pursuing programming as a career or as a hobby (or both, as in my case) to read this book. If you can't fund the $50 ($40 if you buy it from Barnes and Noble online) to get the book, then check out your city library. They are very likely to have it. If they don't have it, then you can always stop into your favorite major bookseller, sit down, and take a few minutes to read it every couple of days. I recommend saving up and buying it though.

Design Patterns

3. Learn and use design patterns.

For a very long time now I have been aware of design patterns. What I was not aware of until recently is that design patterns are an incredible way for programmers to communicate ideas about projects. For example, instead of saying,

"I only want to have one global instance of this class available at any given time, so I'm going to put it into a settings class, and hopefully nobody will try to make their own instance",

you can say,

"I only want to have one instance of this class available at any given time, so I'll make it into a Singleton."

Singleton is one of those design patterns. But the second statement does two things for the programmer and his colleagues that the first one does not:

1. The first statement conveys concern that somebody will want to make multiple instances of his class, while the second one, by stating that the Singleton design pattern will be used, ensures that nobody will be able to make new instances of his class.
2. The first statement does little to convey intent, while the second one says his class will be a Singleton, so programmers who know that pattern already know how he is going to accomplish ensuring nobody will be able to make multiple instances and that it will be global. See the second statement never used the word "global".

Aside from knowing when to create classes or methods, design patterns are probably the programmer's best toolkit when it comes to communicating intent, and reasoning why something should be written a certain way. A good book about some design patterns that I am currently reading as a refresher is "Head First Design Patterns"[2] from O'Reilly. It hits several of the very common design patterns and highlights design principles, such as

"Program to and interface, not an implementation"

which help you keep the quality of your code in high standing.

Code Reviews

4. Ask for code reviews and do them for your peers.

Very few things can keep a programmer from writing messy code better than peer reviews of code. We did it in high school, we did it in college, and it wasn't just a good idea or for fun, because we should do it professionally also. If we don't do code reviews professionally then we are missing out on a time-tested proven method of increasing code quality. It all boils down to this: if your peers can't understand what's going on, then how can you?

The next time you write a class, pass it on to your neighbor in the next cubicle in an email or link him to your code and let him take a look to see if what you did makes sense. Ideally you probably shouldn't need to document every single line. If it's good and clean then your cube-neighbor will likely be able to see your intent and see the paths through your code.

Likewise ask to view others' code. One way to get better is to see how others program, share ideas, and take an active role in improving your code quality.

Conclusion

Nobody should be afraid to ask for help to ensure that you have high quality code. By following the above four principles in your coding, your code quality will improve drastically. I guarantee it, and if it doesn't then you're not trying hard enough. Here are the four principles again:

1. Using an IDE for all of your coding is one good way to increase code quality.
2. Reading about best and good, sound practices is another great way to boost the quality of your code.
3. Learn and use design patterns.
4. Ask for code reviews and do them for your peers.

If you have some other ideas, please share them.

---

Bibliography

1. "Code Complete", by Steven McConnell, Microsoft Press, (C) 1993-2007 Steven C. McConnell.

2. "Head First Design Patterns", by Eric Freeman, Elizabeth Freeman, Kathy Sierra, and Bert Bates, O'Reilly, (C) 2004 O'Reilly Media, Inc.