Finding a Lead Appraiser

My company needs a very inexpensive Lead Appraiser – where can I find one?

I hate to say this, but you are using the wrong criteria for contracting a Lead Appraiser. When it comes to the CMMI and Lead Appraisers, you get what you pay for. And in point of fact, if you go with the lowest price Lead Appraiser that you can find, it is quite possible that this person is either very inexperienced as a Lead Appraiser or may not have the necessary experience or background to provide credible services. The risk you run is that the SEI may not accept the appraisal results in this situation. Then you have spent the money and may need to repeat everything, which can be a very expensive proposition.

Much better criteria to use when considering to hire a Lead Appraiser include:

• number of years experience as a Lead Appraiser
• number of appraisals led (SCAMPI A, B, and C)
• experience in working with small organizations
• ability to interpret the CMMI to appropriately fit your organization's needs
• recommendations/testamonials from clients who have worked with the Lead Appraiser

You should interview the Lead Appraiser. Prepare some questions and scenarios that apply to your organization and learn how they would interpret the CMMI in your context and what they would recommend.

The basic question comes down to why do you want to be appraised? Your internal costs for implementing the CMMI will far outweigh any external costs associated with an SEI-certified Lead Appraiser. You should be implementing the CMMI because there is some business value associated with the CMMI. Lead Appraisers are experts in process improvement. As an analogy, if you are going into the hospital for major surgery, do you go to the lowest price surgeon you can find (possibly someone with questionable credentials) or do you go to a surgeon that has a reputation for quality and success?

Think of it this way. Selecting a Lead Appraiser is a very important decision. There is certainly cost factors as well as risks to consider, and possibly other considerations as well. Therefore, I suggest that you look at the Decision Analysis and Resolution (DAR) Process Area and use the practices described there to select your CMMI services provider.

A Question About DAR Calculations

I have a question regarding DAR calculations. When assigning wieghts to the selected criteria, do I have to relate them to each other? Meaning, do I have to refer to a number that represents my 100% need and relate all criteria to it? For example, I have three Criteria A, B, and C, then I'll assign 3 (30%) to A, 5 (50%) to B, and 2 (20%) to C, and in case one of them was decreased the others will increase. Currently, I define a range for criteria weight (for example from 1 to 5), and I select a number subjectively based on the weight of the criteria.

I think that you may have mixed things up a bit here. The evaluation criteria should be independent and not assigned percentages, the percentages apply to the weighting factors. For each criterion you assign scores between 1 (low) and 5 (high). The weights are derived from team consensus and total to 100% or 1. 20%, 30%, and 50% are typical weights. It is best not to have more than three evaluation criteria because if you have more, then there is the potential for the resulting scores to be too close to each other and it will be very difficult to determine which is the best solution.

With three criteria it is easy to assign 50%, 30%, and 20%. With more criteria you will be forced to assign weights that are closer together. So if you go strictly by the numerical calculations, then there is the potential for lots of ties or very close results thereby making it a challenge to use this technique for selecting an answer. For example, if you had five criteria, then one possible approach, though wrong one, would be to assign 20% to each criteria. Other options would be to assign weights of 30%, 25%, 20%, 15%, and 10% or 50%, 20%, 15%, 10%, and 5%. But you might find it difficult to reach consensus among the team for these values and how to apply them to the criteria. With these assignments, you might calculate the same score for one or more alternative solutions. So with more than three criteria the resulting unique weights could be very close together and not very useful for using this method to calculate scores and use the results to make a decision.

My premise is to limit the number of evaluation criteria to no more than three, not the number of alternative solutions. You can have as many alternative solutions as you want.

And another point is to try to select evaluation criteria that are independent, otherwise you will face challenges as well. For example if you are buying an automobile the independent evaluation criteria might be: 1. Features and price 2. Distance from your home to the dealer and 3. Service reputation.

However, you always have to use the sanity test to determine if the results of the calculation make any sense. You could have chosen the wrong criteria, assigned the wrong weights, and/or assigned the wrong values to each criteria. And yes, management always has the prerogative of making a decision without using the results of the process.

Technical Query vs. Technical Issue

What is the difference between a technical query and a technical issue? And, how do you track them when the projects are maintaning all their queries and issues in email?

In my mind, this is how I distinguish between a technical query and a technical issue. The design team receives a set of requirements to implement. When analyzing the requirements, they encounter a vague or poorly worded requirement such as “the product must be user friendly.” A technical query would be a question back to the requirements provider asking him or her what is meant by this statement. Or a technical query might be a question asking for more detailed technical information so the design team can make the proper design decisions. In contrast a technical issue arises when there is a conflict between requirements, between new or modified requirements and the existing design, between the requirements and the product, several possible design alternatives, etc. A technical issue may require a trade study or research in order to provide the correct information to resolve the issue, basically invoking Decision Analysis and Resolution (DAR).

As far as tracking this information, that is up to the organization to decide on a method that best fits their business practices. Queries could be tracked via email. But I would think a better method would be to include the queries with the requirements so you would capture the history and discussion of any unclear requirements in one location. When you think about it, technical queries are part of the knowledge capture for understanding the requirements. For tracking technical issues, you can perform this action in a variety of ways. Many organizations track all issues the same way in an Issues Log or Issues Tracking System. There are many off the shelf tracking systems that basically have the same functionality.

Process Measures

Can someone please suggest what meaures could be taken for the DAR process area for ML3.

Just like any product, project, or process measure, you should be following the Measurement and Analysis (MA) process of defining the information need and measurement specification so you can collect, analyze, and report on measurements that are relevant and have meaning for your organization. After following the MA process you may find that your resulting measurements are the same as other organizations have used, but NOW you know WHY you need to collect, analyze, and report these data. The justification for these data are now driven by your business and projects, NOT simply because you found some examples in a presentation.

If your organization does not have a history of using data to manage processes and/or projects, then you will need a strong position on why you have chosen the specific measures. Most likely you will receive “push back” from people and if the only justification for the measure is from an external presentation, you may find yourself in a losing situation.

For Decision Analysis and Resolution (DAR) process measures, you may want to look at how much effort is spent conducting a DAR, the number of alternative solutions examined, the number of people involved in the DAR, the effectiveness of the decision (was the right decision made after following the process). BUT you must follow the MA process to determine the RIGHT set of process measures for your organization.

The best place to go to for information about MA and process and product measures is the Practical Software & Systems Measurement web site at http://www.psmsc.com/