Finding a Lead Appraiser

My company needs a very inexpensive Lead Appraiser – where can I find one?

I hate to say this, but you are using the wrong criteria for contracting a Lead Appraiser. When it comes to the CMMI and Lead Appraisers, you get what you pay for. And in point of fact, if you go with the lowest price Lead Appraiser that you can find, it is quite possible that this person is either very inexperienced as a Lead Appraiser or may not have the necessary experience or background to provide credible services. The risk you run is that the SEI may not accept the appraisal results in this situation. Then you have spent the money and may need to repeat everything, which can be a very expensive proposition.

Much better criteria to use when considering to hire a Lead Appraiser include:

• number of years experience as a Lead Appraiser
• number of appraisals led (SCAMPI A, B, and C)
• experience in working with small organizations
• ability to interpret the CMMI to appropriately fit your organization's needs
• recommendations/testamonials from clients who have worked with the Lead Appraiser

You should interview the Lead Appraiser. Prepare some questions and scenarios that apply to your organization and learn how they would interpret the CMMI in your context and what they would recommend.

The basic question comes down to why do you want to be appraised? Your internal costs for implementing the CMMI will far outweigh any external costs associated with an SEI-certified Lead Appraiser. You should be implementing the CMMI because there is some business value associated with the CMMI. Lead Appraisers are experts in process improvement. As an analogy, if you are going into the hospital for major surgery, do you go to the lowest price surgeon you can find (possibly someone with questionable credentials) or do you go to a surgeon that has a reputation for quality and success?

Think of it this way. Selecting a Lead Appraiser is a very important decision. There is certainly cost factors as well as risks to consider, and possibly other considerations as well. Therefore, I suggest that you look at the Decision Analysis and Resolution (DAR) Process Area and use the practices described there to select your CMMI services provider.

A Question About DAR Calculations

I have a question regarding DAR calculations. When assigning wieghts to the selected criteria, do I have to relate them to each other? Meaning, do I have to refer to a number that represents my 100% need and relate all criteria to it? For example, I have three Criteria A, B, and C, then I'll assign 3 (30%) to A, 5 (50%) to B, and 2 (20%) to C, and in case one of them was decreased the others will increase. Currently, I define a range for criteria weight (for example from 1 to 5), and I select a number subjectively based on the weight of the criteria.

I think that you may have mixed things up a bit here. The evaluation criteria should be independent and not assigned percentages, the percentages apply to the weighting factors. For each criterion you assign scores between 1 (low) and 5 (high). The weights are derived from team consensus and total to 100% or 1. 20%, 30%, and 50% are typical weights. It is best not to have more than three evaluation criteria because if you have more, then there is the potential for the resulting scores to be too close to each other and it will be very difficult to determine which is the best solution.

With three criteria it is easy to assign 50%, 30%, and 20%. With more criteria you will be forced to assign weights that are closer together. So if you go strictly by the numerical calculations, then there is the potential for lots of ties or very close results thereby making it a challenge to use this technique for selecting an answer. For example, if you had five criteria, then one possible approach, though wrong one, would be to assign 20% to each criteria. Other options would be to assign weights of 30%, 25%, 20%, 15%, and 10% or 50%, 20%, 15%, 10%, and 5%. But you might find it difficult to reach consensus among the team for these values and how to apply them to the criteria. With these assignments, you might calculate the same score for one or more alternative solutions. So with more than three criteria the resulting unique weights could be very close together and not very useful for using this method to calculate scores and use the results to make a decision.

My premise is to limit the number of evaluation criteria to no more than three, not the number of alternative solutions. You can have as many alternative solutions as you want.

And another point is to try to select evaluation criteria that are independent, otherwise you will face challenges as well. For example if you are buying an automobile the independent evaluation criteria might be: 1. Features and price 2. Distance from your home to the dealer and 3. Service reputation.

However, you always have to use the sanity test to determine if the results of the calculation make any sense. You could have chosen the wrong criteria, assigned the wrong weights, and/or assigned the wrong values to each criteria. And yes, management always has the prerogative of making a decision without using the results of the process.

Stakeholder Invovlement Question

Our organization has project stakeholders whose only project involvement is as a decision maker in an end-of-phase meeting. What is expected of the stakeholders (e.g., training and project involvement) if they are outside of our process, as far as the appraisal interviews are concerned?

In the case described, the stakeholder is NOT outside the process.

The “plan for stakeholder involvement” defines the expected role of the stakeholder in the project, or process. This, in turn, defines the scope of training required. In the case described, the stakeholder would minimally need training in the conduct of end-of-phase meetings. However, if decision making required knowledge of work products produced by the project team, orientation to those products might also be required. Training in Decision Analysis and Resolution (DAR) may also be required, depending upon whether or not the decisions meet the criteria for initiating the DAR process.

Note that training may take many forms, including orientation or briefings on required topics. The expectation is that the stakeholder knows enough to credibly perform the responsibilities described in the project plan or process description.

In the case described, the stakeholder might be selected for an appraisal interview.

Technical Query vs. Technical Issue

What is the difference between a technical query and a technical issue? And, how do you track them when the projects are maintaning all their queries and issues in email?

In my mind, this is how I distinguish between a technical query and a technical issue. The design team receives a set of requirements to implement. When analyzing the requirements, they encounter a vague or poorly worded requirement such as “the product must be user friendly.” A technical query would be a question back to the requirements provider asking him or her what is meant by this statement. Or a technical query might be a question asking for more detailed technical information so the design team can make the proper design decisions. In contrast a technical issue arises when there is a conflict between requirements, between new or modified requirements and the existing design, between the requirements and the product, several possible design alternatives, etc. A technical issue may require a trade study or research in order to provide the correct information to resolve the issue, basically invoking Decision Analysis and Resolution (DAR).

As far as tracking this information, that is up to the organization to decide on a method that best fits their business practices. Queries could be tracked via email. But I would think a better method would be to include the queries with the requirements so you would capture the history and discussion of any unclear requirements in one location. When you think about it, technical queries are part of the knowledge capture for understanding the requirements. For tracking technical issues, you can perform this action in a variety of ways. Many organizations track all issues the same way in an Issues Log or Issues Tracking System. There are many off the shelf tracking systems that basically have the same functionality.

Project Artifact Repository

I have a concern about documentation repository related to CMMI.

We develop projects locally but we store deliveries/documents in a customer online system and use customer clear case to store source code. To improve our process to have CMMI, our consultants said that we must have full control over documents, requirements, etc, because if one day our customer says that we don´t have access to the remote network, we will lose all project information, historical data, etc. Is this a correct interpretation of the CMMI?

But it´s very difficult to us have all artifacts in a local repository because they would be duplicated information and rework, it is not so secure to have confidential information locally, and we would require an infrastructure that we do not need.

What do you think about this solution: to have a contract where the customer affirms that they will permit our access to the documents/sources of our projects for X years even if the business relation ends ? X = ? who should be the signer?

Either your CMMI consultants are giving you some incorrect CMMI compliance advice or you are misunderstanding their message. The CMMI says nothing about the location of the CM repository and the ownership of the repository. What your consultants have told you may be good advice, but not necessary to comply with the CMMI. When considering implementing the CMMI, you have to think about what is best for your organization and how you conduct business.

1. It sounds like you are custom building a system for your customer and they own the repository. It all depends upon the contractual stipulations on ownership of the source code etc. In all likelihood, the customer owns everything and therefore it doesn't matter if the customer denies you access to the repository in the future. You don't own the sourcecode.
2. Now if you are also using the customer's repository for storing project artifacts etc. for different customers, then you do have a problem. In this situation, you have made a bad business decision that has a huge risk to the company's future viability. Now you do need a locally owned and managed repository if your customer elects to deny you access to the repository.

I would NOT try to modify your existing contract per your suggestion below. Instead I would "bite the bullet" and purchase a CM tool and create your own repository if #2 applies to your organization. At a minimum, you may just want to mirror the contents of your customer's repository in the event of network outages just so you can continue to perform work without interruption.

Thank you your reply.

I work in an area that has a lot of projects for one big customer, so all of the artifacts and source code are maintained in the customer repository. There are some "internal" artifacts that we do not deliver, and we maintain in a local repository (metrics, audit reports), but the source code and project plan, requirements spec, etc, are all in the customer repository.

So must we store project plan, requirements spec etc in the local repository too?

Given your explanation, it sounds like you have the proper CM structure established. The customer’s project artifacts are stored in a customer repository. I can only foresee one circumstance where the customer would deny you access to their repository and that would be if they gave the work to a different company. And I am assuming that your contract states that the customer owns all of the project artifacts. Therefore you don’t have the rights to the information if they deny you access to the repository.

Now considering all of the information you are placing in your customer’s repository, there may be some information that you may want to retain for your own records. Depending on how your contract is written, you may have to inform your customer that you are retaining copies for internal purposes. And this subset of information you would place in your local repository. Please keep in mind that the CMMI does not require you to also keep a copy in a local repository and that I am not telling you do to so either. You need to evaluate your business need for maintaining copies in a local repository.

Perhaps it may make sense for you to run through the DAR process to determine which customer artifacts, if any, need to be placed in a local repository.

Process Measures

Can someone please suggest what meaures could be taken for the DAR process area for ML3.

Just like any product, project, or process measure, you should be following the Measurement and Analysis (MA) process of defining the information need and measurement specification so you can collect, analyze, and report on measurements that are relevant and have meaning for your organization. After following the MA process you may find that your resulting measurements are the same as other organizations have used, but NOW you know WHY you need to collect, analyze, and report these data. The justification for these data are now driven by your business and projects, NOT simply because you found some examples in a presentation.

If your organization does not have a history of using data to manage processes and/or projects, then you will need a strong position on why you have chosen the specific measures. Most likely you will receive “push back” from people and if the only justification for the measure is from an external presentation, you may find yourself in a losing situation.

For Decision Analysis and Resolution (DAR) process measures, you may want to look at how much effort is spent conducting a DAR, the number of alternative solutions examined, the number of people involved in the DAR, the effectiveness of the decision (was the right decision made after following the process). BUT you must follow the MA process to determine the RIGHT set of process measures for your organization.

The best place to go to for information about MA and process and product measures is the Practical Software & Systems Measurement web site at http://www.psmsc.com/