|OMIX WHITE PAPER|
The purpose of this paper is to help OMIX better manage its projects through well-defined policies and procedures, as well as sound software engineering practices. Without policies and procedures, there is little direction for individuals on a project. In some cases this may not present a problem, but as project size and scope grows, lack of organization will manifest itself in rather ugly ways.
It is also important to understand the difference between coding or programming, and software engineering. In one case, the developer is merely using the syntax of a given language to solve a known problem. In the other, the developer is designing a solution. This starts with identifying the problem or application, and goes all the way through writing documentation and delivering the finished product. Software engineering is an integral part of good project management.
WHY BOTHER WITH PROJECT MANAGEMENT?
An excellent question. After all, project management just means adding another layer of meetings and bogus organizational hierarchy, right? Wrong. While there is some increased overhead involved in a well-managed project, the rewards are definitely worth the slight inconvenience. Reasons for project management include:
THE NINE STEPS IN A PROJECT'S LIFE CYCLE
The project life cycle, as defined by OMIX, consists of nine (9) distinct steps. These are: the requirement survey, requirement specification, functional specification, internal design specification, implementation, internal testing, beta (external) testing, acceptance, and post-project debriefing. Each is explained in more detail below.
Step 1 (of 9): Requirements Survey
This is the very first step in the project life cycle. Basically, a need has been found, which OMIX would like to take part in fulfilling. This is usually a meeting at the client’s site, which would generally include (in addition to the client, of course) a manager and sales representative. At this meeting, it is determined exactly what problem the client is trying to solve, and possibly the potential technologies to be used. This is a very high-level meeting, meaning that specific costs, timeframes, implementations, etc. are not discussed.
Step 2 (of 9): Requirement Specification
At this point, OMIX should have a feel for what the client is after. An OMIX representative will then draft a requirement specification, basically putting down in writing what OMIX believes the client is after based on the requirement survey. A very rough estimation of timeframe and cost can be included in the specification, but it should be emphasized that this is still a high-level portion of the life cycle.
At this point, OMIX should have a fair idea about how many people will be required, as well as the basic technologies to be utilized. The manager can then appoint a project lead from among the developers, who will assist the manager and sales representative from this point forward. The lead should be selected based on experience, availability, and familiarity of the technologies being used.
Some research may be required to help with the estimation of time and cost. For example, when using technologies OMIX is unfamiliar with, it makes sense to get a feeling for how much work is involved in incorporating those technologies. It should go without saying that when working with new or beta technologies, time estimates should be very conservative; it's much better to deliver a product sooner than the client expects than to be late. OMIX should also make it clear to the client that we may run into problems beyond our control, and that they should be prepared for such a case, however unlikely.
Luckily, as we gain experience developing with new technologies, each successive project will fall into a "well-known" domain, much like HTML and CGI do currently. These projects will pose little risk, and should let us exploit the work we’ve already done on similar projects.
The requirement specification is then brought to the client to verify that everyone is indeed on the same page as far as the scope of the project. If this is the case, the client will sign off on the specification, and discussions will ensue concerning the details.
Step 3 (of 9): The Functional Specification
Based on the discussions brought about by the requirement specification, a functional specification is drafted. The functional specification is a detailed document describing work OMIX is proposing to the client. It should include specific technologies, server setups, hosting details, process interaction, etc. At this time, a schedule of milestones should be included as a rough time line for completion.
If a user interface is required, it should also be described in this document. Remember that this is a functional specification, not an implementation specification; as such, final screen shots need not be provided. The functionality of each screen and its graphical user interface elements should be noted, however.
For all intents and purposes, this is the contract between OMIX and the client. It is important to be specific enough that the client cannot add extra work, and claim it falls under the scope outlined in the functional specification. At the same time, it should not be so specific as to hinder our ability to work around unforeseen problems. When there are any discrepancies about what work is to be done, what technology to be implemented, etc., the functional specification will be the reference.
Final cost information will also be included in the functional specification. Once the client signs off on this document, engineering can begin.
Step 4 (of 9): The Internal Design Specification
The internal design specification is what the developers use as their guide. It is based off the functional specification, but is even more detailed at the technical level. This is where classes. Functions/methods, data structures, interfaces, and other details are laid out. This document is obviously for internal use only.
Here the project lead should distribute development tasks based on ability and availability. Together, it should be determined what previous code can be used to cut development time, what portions of new code can be shared, and how new code can be made modular enough for reuse at a later date. Hopefully all the developers will be familiar with OMIX coding standards (which cover everything from code formatting to comments to source tree structures).
Along with the technical design, which happens at this point, a very high-resolution schedule should also be developed. In this schedule, time lines are set for individual data structures and methods.
This is where the engineering begins in earnest. Much more will be said about this in the section on software engineering below.
Step 5 (of 9): Implementation
This is the actual "programming" portion of the development cycle. It's what people generally think of when they think about programming, but this step is far less critical or demanding than the design portion above. Basically, this is the edit-compile-debug stage of development.
A "to do" list should be kept by each developer, which retains "done" items. This list should be periodically sent to the project lead for review, assistance, etc. Revision control should also be utilized extensively at this point. The general rule of thumb is: if you’re about to undertake a somewhat significant code modification, check your code in and out so you have a "known good" to fall back on when you break everything.
Finally, periodic peer code reviews should take place. This is to ensure good programming practice, as well as to check for conformance with OMIX coding standards. This is also a great learning opportunity for all involved; new programming tricks and techniques, or complete design methodologies can be imparted to the less (and oftentimes, more) experienced.
Step 6 (of 9): Internal Testing
Finally, the developer thinks his or her code is “good enough”. Other OMIX employees are charged with using the software and “beating on it”, looking for bugs, poor bounds/error checking, etc.
Assuming a general-purpose application, an employee not on the project would be the ideal candidate for testing. No instructions beyond what the client will get should be given. This is an excellent test of user interface and usability. If, however, the application is very specific or targets a vertical market, it may be necessary to have someone on the project do the testing, or at the very least, provide some background information to a tester not on the project team. It makes little sense to have the developer test his or her own code (and presumably they were conscientious enough to do this during development anyway).
Any bugs should be reported back to the developer IN WRITTEN FORM. It's not good enough to say, "Hey, this is broken." Be specific. What happens? Core dump? System hang? Incorrect result? Is it reproducible? How? The more information, the better. For many projects we use bug-tracking software such as ClearDDTS.
Step 7 (of 9): Beta (External) Testing
This phase is essentially identical to the internal testing phase, except the client is actually using the software. Developers can generally count on the client not being nearly as specific about how and when problems occur, nor do they have as much control over the hardware/software environment in use. Otherwise, proceed as you would under internal testing, saving any and all correspondence with the client.
It is assumed that once beta testing begins, features are basically frozen. All effort is directed toward fixing those features that exist, not adding new ones. It is not unreasonable, however, to begin beta testing code that has some features missing which were not included because of time constraints, but will be included in the final delivery. Not desirable, but not unreasonable, either.
In terms of scheduling, beta testing should begin somewhere between 2 and 10 days before final delivery, depending on the size of the project.
Step 8 (of 9): Acceptance
At this point, the project has undergone heavy testing, and has been certified by the developers, OMIX testers, and client testers. The client signs off that the deliverables in the functional specification have been developed and delivered, and the project team immediately retires to the local pub for senseless amounts of Guinness extra stout.
Step 9 (of 9): Post-Project Debriefing
As the final step in the project life cycle, this is an internal affair. This meeting is every bit as important as the first; here the project team discusses what went wrong, as well as what went right, with their project. How could problems have been avoided? What portions went smoothly? Why? This is an opportunity for everyone (including those not on this particular project team) to learn something about project management and software engineering. Hopefully something you learn here will save you time, money, and grief on a subsequent project.
THE ROLE OF SOFTWARE ENGINEERING
As pointed out earlier, there is a distinct difference between programming and software engineering. When done properly, the process of software engineering is very complimentary to good project management. As with project management, there is a life cycle associated with software engineering; it consists of eight (8) steps. In fact, software engineering is such an integral part of project management, you’ll find that both endeavors share numerous steps.
Step 1 (of 8): Define the Problem
If you're working on a piece of software, you're presumably trying to solve some problem. Whether this is balancing a checkbook, playing a game, or getting forms data from a web page to a database, the problem or task must be well defined. If you start coding with only a vague idea of how you're going to solve the problem, you’re destined to flounder.
Step 2 (of 8): Develop a Solution
"Develop a solution" does not mean write the solution in C and be happy. It means you should think about the problem, how you would solve it, how you SHOULD solve it, what technologies you’re going to use to implement the solution, what language it should be written in, a time line for its development, resources required, plus a host of other things. This is basically a high-level solution, analogous to the requirement specification in the project life cycle.
It is at this stage that you should generate a very general solution; you should be working on the forest, not the trees. The truth of the matter is, though, that the forest is in many ways more important than the trees. You need to think about how each portion of your program will interact with the others, not the algorithm to be used for each portion.
One way to start on this path is to draw a diagram of the solution's functionality (see diagram 1, below). You should layout the major functional parts, and their interaction with each other.
Step 3 (of 8): Refine Your Solution
This is an iterative process; you should continue adding detail to your original design until you've broken things down to the function/method level. Start with the black boxes in your diagram, and determine the primary function required by that box. Then break that function down into smaller functions, including support functions. (Note, ‘functions’ here do not refer to programming language functions and procedures, but rather general tasks or processes.
When you're finished, you should be able to trace the logic of the program to see which functions could potentially be called by which other functions, what data they need to share or pass back and forth, etc. The major data structures should also be forming, although not necessarily solidified.
The language you're going to use for implementation will begin to have some bearing on the direction you go with your refinements. The more and more detailed your solution specification becomes, the more you will have to consider the final implementation language. The most obvious example of this is whether the language is object oriented or not. The layout will look completely different for a procedural language like C++ (making the huge assumption that you actually implement object oriented design, as opposed to just using C++ constructs like cin and cout, but that's a whole other topic).
Once you’re at this point, you've finished 75 percent of the solution. This is an important concept! Coding itself should be a relatively small part of the overall software engineering picture. Don't be discouraged by the fact that you've spent the past two weeks working on diagrams, thinking about function interdependence, data structures, and algorithms. This is how software engineering is done. Having everything laid out in this fashion will cut coding time down to a very minimum. Think of it this way—the person who can engineer the solution but cannot code it is worth three times the person who knows the language syntax like the back of his or her hand, but couldn't design an elegant solution to save his or her life.
Step 4 (of 8): Code Your Solution
Assuming you've done all the steps leading up to this point, this part should be among the shortest of the entire process. There’s no algorithm design, no deciding between implementation languages (you should have done this in the previous step), etc. The only hang-ups you're likely to have are syntactic.
If you find that you didn't do a thorough enough job in the previous step, or made bad assumptions about how functions would interact, or just plain hadn't considered some obscure case, you'll find out in this stage. If you've tried to design your solution with an eye toward expandability, feature changes, and general modularity, hopefully you'll be able to handle the problem without throwing out all your previous work. How well you do this is a matter of experience.
While doing the actual coding, remember to use comments liberally. In a nutshell, comment everything that isn't trivial. Sure, you know what your code is doing right now. But, six months down the line, you may very well not. Even if you're a programming deity, and there's no chance you'll be confused by your own code somewhere down the line, remember that it's entirely possible other people will have to work on your mess. So, comment everything!
Step 5 (of 8): Testing and Debugging
In the process of coding your solution, you're bound to run into errors of both logic and implementation. Testing on your part will reveal these, as will the compiler. Finding and fixing these bugs is what this stage of software engineering is about. If your chosen language supports a preprocessor, you can make good use of it to include and exclude code for purposes of both testing and debugging.
Testing is more involved than the developer coding and compiling successfully. Even if the developer uses the software, and everything seems to work, an outside source should also be part of the testing process. Whether this is the client doing beta testing, or someone else within OMIX, it's necessary so that matters of design and interface can be verified. How to use a program is obvious to the developer, because he or she actually developed it. Having a computer outsider use the program is oftentimes very enlightening.
Step 6 (of 8): Code Review
At various points throughout your coding endeavor, peer review of your code should be performed. The depth of this review will vary, but of primary interest is that the code meets OMIX coding standards. Allowing other people to see your code and comment on it serves several purposes:
Step 7 (of 8): Documentation
How much documentation is required will vary from project to project. If you're working on a commercial application, you will of course have to document your software in a rather significant way. If you're merely developing a solution for handling the results of a web form query, however, a simple README file may suffice. If the documentation within your code is extensive enough, even that may be enough. Again, this will vary.
How to go about writing a 250-page manual goes beyond the scope of this paper, but suffice it to say that documentation is a good thing, and is definitely a part of software engineering (no matter how loathsome).
Step 8 (of 8): Delivery
This can be seen as the end of a complete cycle of software engineering. You can look forward to going through it again for version 2.0, however.
GOALS OF SOFTWARE ENGINEERING
All of the above is well and good, but what exactly is the purpose of designing software as opposed to hacking code? The end result is the same, right? You have some piece of software that performs some task, whether it's well designed or not. Not really.
Ease of Debugging
If you're writing software that's only a page or two in length, debugging isn't particularly difficult. There's minimal code, so there is a minimal number of things which can go wrong. Consider a much larger project—maybe hundreds of pages. Do you really want to sift through 200 pages of poorly formatted, ill-designed, undocumented code? Even if you don't mind, OMIX may mind that you're spending three weeks tracking down a simple bug that should take 10 minutes.
If you're written things in a modular or object oriented fashion (which are not the same, by the way), and kept function/method sizes reasonable, tracking bugs becomes much easier. The end result is you're more productive. You finish the project sooner, and can get on to bigger and better things.
This is probably the biggest argument for good software engineering. How many times have you written code to parse HTML forms? Or make a connection and query an Oracle database? Or output debugging information intelligently? Anything more than once is too many, and simply a waste of time, unless you're adding functionality or improving the design in some fashion.
Once you've written clean, modular code to perform a task, you should strive to reuse it as much as possible. Package it up in a class, library, or just a directory—whatever option you have available given the language and environment you're working in. Presuming it's been tested and debugged, you can just drop it in to your next project. That's one more black box in your next design you don't have to worry about.
The advantages of this should be obvious: you don’t waste time rewriting the same code over and over again, you know the code being used works and has been debugged, and profits go up accordingly.
By the way, "cut and paste" is NOT reusing code properly. Aside from being a sign of a true hacker (in the poorest sense possible, perhaps "kludger" would be a better term), it doesn't make good software engineering sense. What happens if you discover some obscure bug in that code? Or make an upgrade? Are you going to find all the software you ever incorporated that code into? It would be a complete waste of time.
This should also make clear another benefit of good design—assuming you publish the interface to your class/routines, and stick to those interfaces, you can improve the code without breaking things for everyone. For example, say you write a search class that basically provides grep-like functionality. People use it in their various projects, and life is good. One day, you figure out how to make the search faster by a factor of 5. If people are just using your published interfaces, and you can improve the code without changing them, everyone’s software will suddenly see search speed improvements with no work on their part. This assumes use of a technology such as shared libraries. In static libraries or classes, a recompile would be necessary, but a small price to pay for such an improvement.
Assuming you've taken the time to design your classes well, provided the appropriate interfaces to them, encapsulated the data, etc., you relieve yourself of the need to work side by side with everyone else on the project. You no longer have to worry about how someone else’s code works, what variable names they’re using, etc. All you need to know is you call some function with some given parameters, and you get some end result; specific implementation is not important.
If you've designed things properly, you haven’t coded for specific cases. For example, just because you need a function to open a connection to an Oracle database now, doesn't mean you shouldn't make it general enough to be able to easily open a connection to say, an Informix database instead.
You should attempt to code in a way that lets you extend the functionality of your software without requiring a complete redesign. Much like with magic numbers, if you avoid making assumptions about specifics, you’ll find your code is much easier to modify later without major repercussions.
This document outlines some of the fundamental stages and goals of software engineering and project management. At OMIX, we strive for quality, productivity and efficiency. Our team's commitment to this basic process will benefit our organization from all perspectives. Our combined results are only as good as each and every one of our individual efforts to strive to maintain a winning and effective process.