Finding a Lead Appraiser

My company needs a very inexpensive Lead Appraiser – where can I find one?

I hate to say this, but you are using the wrong criteria for contracting a Lead Appraiser. When it comes to the CMMI and Lead Appraisers, you get what you pay for. And in point of fact, if you go with the lowest price Lead Appraiser that you can find, it is quite possible that this person is either very inexperienced as a Lead Appraiser or may not have the necessary experience or background to provide credible services. The risk you run is that the SEI may not accept the appraisal results in this situation. Then you have spent the money and may need to repeat everything, which can be a very expensive proposition.

Much better criteria to use when considering to hire a Lead Appraiser include:

• number of years experience as a Lead Appraiser
• number of appraisals led (SCAMPI A, B, and C)
• experience in working with small organizations
• ability to interpret the CMMI to appropriately fit your organization's needs
• recommendations/testamonials from clients who have worked with the Lead Appraiser

You should interview the Lead Appraiser. Prepare some questions and scenarios that apply to your organization and learn how they would interpret the CMMI in your context and what they would recommend.

The basic question comes down to why do you want to be appraised? Your internal costs for implementing the CMMI will far outweigh any external costs associated with an SEI-certified Lead Appraiser. You should be implementing the CMMI because there is some business value associated with the CMMI. Lead Appraisers are experts in process improvement. As an analogy, if you are going into the hospital for major surgery, do you go to the lowest price surgeon you can find (possibly someone with questionable credentials) or do you go to a surgeon that has a reputation for quality and success?

Think of it this way. Selecting a Lead Appraiser is a very important decision. There is certainly cost factors as well as risks to consider, and possibly other considerations as well. Therefore, I suggest that you look at the Decision Analysis and Resolution (DAR) Process Area and use the practices described there to select your CMMI services provider.

Advice on Process Performance Models

We have developed a Process Performance Model (PPM) for system testing defects. In our model we analyzed multiple factors (independent variables) that may influence the number of system test defects. However, the strange part is that the regression analysis has eliminated all of the variables except one, testing effort. Some of the other variables included things like number of test cases, tester's experience, and size (LOC).

Can a valid PPM only have one independent variable? The p-value is less than 0.05, and the R squared value is 0.8, so it seems that it would be reliable. But only one independent variable seems rather limited and weak to me, but I can't think of a justifiable reason why it is invalid considering we have analyzed data from several independent variables and only one was found to be statistically significant.

What do you think?

It does sound a bit odd. Perhaps you have overlooked some other X Factor that you should be evaluating. But it is conceivable to have a model with only one independent variable.

Basically your model looks like Y = mX +b. But given all of the PPMs that I have seen, they all contain multiple X factors.

Several things could be going on here.

1. The regression analysis was faulty

2. The list of X factors evaluated was incomplete

3. One or more of the X factors were a combination of factors that should have been separately evaluated

4. Not enough data were evaluated for the regression analysis to yield proper answers

5. The distribution of the data is not a normal distribution (Gaussian distribution aka Bell Shaped Curve) and analytic techniques for a normal distribution were mistakenly used

You probably should have a statistical expert independently examine your analysis to see if the resulting PPM is in fact a linear relationship between X and Y.

The other option is to use the PPM for several months of analysis and see how accurate it is. If the PPM yields accurate predictions, then it is most likely a good PPM.

One thing that can happen if you have not been careful with storing the raw data and/or not collecting at the proper level of detail is that factors can be inadvertently combined and thus cancel each other out thereby yielding a factor that on the surface looks reasonable, but may in fact not be. For example if one element is high and the other one it is combined with low, then the result will be somewhere in between. Think of two out of phase sine waves. Each one by itself looks like a sine wave, but the combination is something else entirely.

So as an example, suppose you are collecting defect data (requirements defects, input defects, design defects, coding defects, test case defects, etc.) and you have neglected to use these categories to count and distinguish the different types of defects. Then the defect count probably won’t correlate with anything because the individual categories are cancelling each other out. This is the kind of data behavior that caused many of the early HM adopters to throw away their historical data and start over from scratch. So it is very important to follow the Measurement and Analysis PA to properly define all of the base measures and to properly store them so when it comes time to perform the statistical analysis of the data, there is sufficient information stored to allow different types of analyses to be performed to isolate the specific information of interest.

Hope this explanation helps.

Integrated Project Management (IPM) - SP 2.2

Could you please provide some examples of evidence for IPM Specific Practice 2.2 and some examples of a critical dependency?

Integrated Project Management (IPM) Specific Practice 2.2 states “Participate with relevant stakeholders to identify, negotiate, and track critical dependencies.” This practice is one of many compound practices in the CMMI. Therefore, to completely cover the practice, the organization would have to provide evidence that the project is participating with the relevant stakeholders. And that participation would be to identify critical dependencies, negotiate critical dependencies, and track critical dependencies. So the type of Direct Evidence that I would expect to see as a Lead Appraiser or Appraisal Team Member from the organization would be a list of critical dependencies, issues, and action items that were the result of the participation and for Indirect Evidence I would expect to see the meeting minutes or review/inspection reports that resulted in the list provided as Direct Evidence.

Now, if you read the Tips and Hints in the CMMI for IPM SP 2.2, these will provide insight into what a critical dependency is. Each project will have some constraints such as time, quality, budget, etc. that it must meet. By applying these constraints to the project schedule the Project Manager will be able to define the critical path through the project tasks and activities. The critical path contains all of the tasks and activities that have a direct impact on meeting the scheduled end date for the project. If a task on the critical path slips or runs over, then the end date of the project has a corresponding slip or overrun. Tasks that are not on the critical path can move about a bit before they impact the end date. The project tasks and activities on the critical path define the critical dependencies for the project. For example, if both Task A and Task B are on the critical path and Task B cannot begin before Task A completes, that is one critical dependency. Another example of a critical dependency is an external group is supplying a work product to the project and a Task cannot begin until the work product is received and accepted. This type of critical dependency may also be called a critical deliverable.

Hope this explanation helps.

Bi-directional Traceability Question

Would you please explain to me the meaning of bi-directional traceability? If I am not wrong, Horizontal Traceability indicates mapping of a requirement across all the SDLC phases like Requirement->Design->Construction->Testing. But I do not get what is meant by Vertical Traceability. Would you please educate me on this?

Vertical traceability is top-down bottom-up traceability. Vertical traceability is the most common type of traceability. Basically it is tracing from the highest level requirements all the way down to the product component level (could be even to a specific line of code). And then starting at the lowest level and tracing all the way back up to the top. Horizontal traceability is tracing within a document or between peer documents. What you have listed as tracing from requirements to design to construction to test is actually vertical traceability. And when you come right down to it, it doesn’t really matter what type of traceability you label it. All that matters is that you can trace forward from a starting point to an end point and vice versa. The level and complexity of the traceability is determined by the criticality of the product and project. For a program like the Space Shuttle or Space Station, you would need very rigorous requirements traceability. Whereas a small application or a product would only need some simple traceability.

REQM - SP1.1 Query

In REQM – SP1.1 calls for “Criteria for evaluation and acceptance.” Can you tell me what this expected “criteria” would be?

I suggest looking at Requirements Management (REQM) Process Area in your copy of the CMMI. On page 490 in REQM Sub-practice 2 there is a list of example evaluation and acceptance criteria.

REQM does not expect the organization to have evaluation and acceptance criteria. What you are expected to provide is evidence that support the required components (Specific Goal 1 in this case; evidence that demonstrates that your requirements are managed and inconsistencies with project plans and work products are identified) and evidence that supports the expected components (Specific Practice 1.1 in this case; evidence that demonstrates that you have developed an understanding with the requirements providers on the meaning of the requirements). If your Lead Appraiser is telling you that you have to provide these criteria, then he or she should be challenged as to why they are telling you that you need to provide this evidence.

The evaluation and acceptance criteria are one way of developing this understanding. Please bear in mind the section title is TYPICAL Work Products, not EXPECTED Work Products, or REQUIRED Work Products. The list of Typical Work Products is merely a list of example work products.

Selection of Suitable Lifecycle Processes

Selection of a suitable lifecycle for the project/product is more of a technical issue. Then why is it covered in Integrated Project Management (IPM) rather than Technical Solution (TS)?

Selecting a suitable lifecycle is a function of the project requirements and the application domain. The selection is something that should be performed when planning the project activities, not while performing the actual software engineering activities. Selecting a lifecycle model and defining the project’s lifecycle phases are necessary for planning and estimating the project. That is why lifecycles appear in Project Planning (SP 1.3) and Integrated Project Management (sub-practice 1 of SP 1.1) . The selection of the lifecycle is a project management activity.

From another perspective, the purpose of Technical Solution is to design, develop, and implement solutions to requirements. And selecting an appropriate lifecycle model is an activity that needs to be done before you can design, develop, and implement a solution to the requirements.

Process Performance Model and QPM Inquiry

We are a CMMI Maturity Level 5 organization and now we are preparing for our re-appraisal.
We have Process Performance Models (PPMs) with different X- prameters and good R^2, R (adjusted) & P(value).

We perform simulations to calculate the probability of achieving our organizational goal and the determination that the probability % is good enough to achieve the goal.

Now we are in our process of implementing the PPM. My question is: Shall we use the PPM to estimate the X- parameters in my projetct? Or what should I do with this model after that ?

Please advise.

I hate to say this, and I could be wrong, but reading your question it sounds like you are asking what do I need in order to be Maturity Level 5 (ML 5)? Is this equation sufficient? And by asking what do I do with the PPM, I may be misunderstanding you, but if you were really at ML 5 you wouldn’t be asking this question. I get the impression from your question that your organization doesn’t know why it is using a specific technique for the PPMs.

The best solution to your question is to ask your High Maturity Lead Appraiser for help and guidance as he or she should have enough understanding and knowledge of your organization to provide the answer.

But once again, the nature of your question gives me the uneasy feeling that your organization may not even be Maturity Level 4.