SCAMPI Document Review

I have two questions or requests for clarification:

• During the document review in SCAMPI B, wherever we do not have a doc, ppt, or xls as an artifact, we provide screenshots from tools where the planning and tracking are done.  However, our Lead Appraiser (LA) is asking for access to tools for all the ATMs.  But it is not possible to give access to certain tools as they are client specific and only the team working on the project gets access (that too after signing an NDA).  How do we handle this situation?  Even if the tool is internal, access is very much restricted based on role in the project.  Can the LA really require access to tools for all ATMs, which is not allowed as per the policy of an organization?  Is there a guideline on what process to follow in case access to certain tools or application is restricted? 

• Is there a material, which shows a linkage of all the PAs (representing interaction of PAs) and gives a holistic view of CMMI PAs when applied to an organization at Maturity L5. This is more from a training perspective. 

Have you explained these restrictions to your Lead Appraiser?  The Lead Appraiser should be flexible regarding access to restricted tools.  As a Lead Appraiser, I would find the screen shots as acceptable evidence in your situation.  And if I wanted or needed to view additional evidence, I would request that you provide a demonstration of the tool by an authorized user and have this person be directed by the ATMs or Lead Appraiser to view specific information.  I have used screen shots as evidence on numerous appraisals in the past.  The SCAMPI method allows for screen shots and tool demos for just such reasons as yours.  If your Lead Appraiser is unwilling to abide by your restrictions on tool access, then I would strongly urge you to find a new Lead Appraiser and possibly report him or her to the SEI.

Look in Chapter 4 of the CMMI-DEV book and you will find a series of diagrams that show the linkages between the PAs at a very high level.

Is Going Directly for a CMMI ML 5 Appraisal Allowed?

Is a CMMI v1.2 ML 5 appraisal allowed in the following situation?

One of my company's divisions was successfully appraised to CMMI v1.1 ML 5 but the appraisal results expired in 2009. Now my company wants all three divisions, which are located in different cities, appraised to ML 5. Though skipping levels is not recommended, however, is it allowed to go for the appraisal?

There is nothing from the SEI that prevents a company from being appraised to whatever Capability Level or Maturity Level that it chooses. That being said, what does drive the CL or ML is the organization being able to collect, analyze, and correctly use data for statistical and quantitative analysis, as well as being able to institutionalize the behaviors and practices. You should hire an SEI-Certified High Maturity Lead Appraiser and have him or her perform a Class C or Class B appraisal to determine the risks with your current approach and implementation of ML 5. The outcome of this exercise will determine if it is feasible to achieve ML 5 at all three sites.

Query on CMMI for Development v1.3

Our organization was appraised at CMMI ML3 in Oct, 2009 and we'll be going for ML5 in Q1 2011. Although we have been preparing ourselves for CMMI for Development v1.2, we are also aware that the CMMI for Development v1.3 will be out in January, 2011. We wish to get appraised for CMMI for Development v1.3. I have gone through some of the PPTs and PDFs on web but none of them give a clear insight into the changed expectations from the existing PAs or expectations from new PAs to be added.

It'll be a great help if anyone can provide me the draft version of v1.3 or any detailed document about the same.

One item that you may not be aware of is that the SEI would like to see at least 18 months between an ML 3 appraisal and a High Maturity appraisal. If you are planning for an appraisal in Q1 2011, that would be less than 18 months. Therefore you would have a very high probability of your appraisal results being audited by the SEI, which could take a long time before being accepted. I would encourage you to hire a High Maturity Lead Appraiser as soon as possible, if you haven’t already done so, and move your plans for your ML 5 appraisal at least 3 months or more into the future to ensure that you have enough data for performing the High Maturity practices and enough time for institutionalization of HM.

You will have to wait for the v1.3 release in November 2010. The problem with using drafts is that things can change before the release.

Process Performance Model and QPM Inquiry

We are a CMMI Maturity Level 5 organization and now we are preparing for our re-appraisal.
We have Process Performance Models (PPMs) with different X- prameters and good R^2, R (adjusted) & P(value).

We perform simulations to calculate the probability of achieving our organizational goal and the determination that the probability % is good enough to achieve the goal.

Now we are in our process of implementing the PPM. My question is: Shall we use the PPM to estimate the X- parameters in my projetct? Or what should I do with this model after that ?

Please advise.

I hate to say this, and I could be wrong, but reading your question it sounds like you are asking what do I need in order to be Maturity Level 5 (ML 5)? Is this equation sufficient? And by asking what do I do with the PPM, I may be misunderstanding you, but if you were really at ML 5 you wouldn’t be asking this question. I get the impression from your question that your organization doesn’t know why it is using a specific technique for the PPMs.

The best solution to your question is to ask your High Maturity Lead Appraiser for help and guidance as he or she should have enough understanding and knowledge of your organization to provide the answer.

But once again, the nature of your question gives me the uneasy feeling that your organization may not even be Maturity Level 4.

Rationale for Maturity Level 5

Our organization is currently at Maturity Level 3 and is now planning to go for Maturity Level 5. Our CEO is asking for the short term and long term benefits and challenges for implementing Maturity Level 5. How do we provide this information?

The question that needs to be asked is why does your organization want to achieve Maturity Level 5 (ML 5)? If the organization has already achieved Maturity Level 3 (ML 3), what is the motivation for achieveing ML 5? Most likely it is not mandated in a contract or by its customers. But achieving ML 5 is desirable to be competitive in the marketplace. Therefore, the reasons for achieving ML 5 should provide some indication of the short term and long term benefits.

The first step in becoming a High Maturity organization is defining your Quality and Process Performance Objectives (QPPOs) that are based on your business goals and objectives as well as your customer needs. The QPPOs should be stated in a form such that they specify a timeframe. And that information will provide some ideas of the short term and long term benefits.

However, the best advice that I can provide is to hire an SEI-certified High Maturity Lead Appraiser (HMLA) who will work with you to help explain the benefits of ML 5 to the CEO. The HMLA should have experience working with many different organizations at various Maturity Levels and be able to talk about the different challanges that have been faced by other organizations, as well as the challenges and risks within your organization.

At this point, it sounds like your organization is just beginning its journey to ML 5, so I would have to be convinced that your processes are stable enough to provide the data needed to quantify any short term and long term benefits. I think that the best you could do at this point is communicate this information in qualitative terms. Any quantitative information may not be accurate until you have implemented High Maturity.

Applicability of the Informative Material

I have been told that if an organization has to go for CMMI ML 5 the organization has to address all of the sub-practices even though they are informative material and if the company is only going for ML 3, then they can apply appropriate sub-practices. Is this true?

Please allow me to try to explain the model. There are at least two ways to look at the CMMI: 1) implementing the model and 2) appraising the organization against the model.

In addition, there are three CMMI components: Required, Expected, and Informative. These components only have meaning when you are talking about appraisals. The Required components are the Specific and Generic Goals, the Expected components are the Specific and Generic Practices, and everything else is an Informative Component.

When you are implementing the model, you should not be concerned about differentiating between the different types of components. From Chapter Two of the CMMI-ACQ (and this statement applies to ALL CMMI constellations): “All model components are important because the informative material helps you to understand the expected and required material. It is best to take these model components as a whole. If you understand all three types of material, you can then understand all the pieces and how they fit together to form a framework that can benefit your organization.”

When the organization is being appraised against the CMMI in a formal SCAMPI A appraisal, the organization will only be appraised against the Required and Expected components, regardless of the Maturity Level. However, the appraisal team may be evaluating the evidence and perhaps asking questions in the interview sessions at the sub-practice level just to gain a better understanding of how the organization is addressing each of the Required and Expected components. The organization will not be penalized if it is not performing one or more sub-practices. The appraisal team will be identifying and documenting weaknesses with the organization’s implementation of the goals and practices.

Because there has been a misunderstanding of what High Maturity means (ML 4 and ML 5), the SEI has been emphasizing that the proper implementation of the goals and practices for OPP, QPM, OID, and CAR means reading, understanding, and implementing the types of activities described in the Informative material. So, for a ML 4 or ML 5 SCAMPI, the organization will not be evaluated against the OPP. QPM, OID, and CAR sub-practices, but weaknesses will be noted at the goal and practice level if the organization has not properly implemented these Process Areas to meet the intent, which is gained by understanding the Informative material.

Defect Removal Efficiency (DRE)

I am a Defect Prevention team member in our CMMI Maturity Level 5 company. We are trying to plot the Defect Removal Efficiency (DRE) metric in our projects. I have a few questions:

1. Our DRE is based on phase level defect injection and detection concepts. If we go closely by the definition of the phase "testing/QA", bugs should not be injected in testing phase, as this is detection activity. If not then any examples can be useful for me to understand.
2. Can a common DRE format can be utilized for all kinds of projects as in sustaining engineering, maintenance, development and pure testing? Specifically what do to we do for testing or pure QA type of projects?
3. DRE is scoped for defects leaked by QA and reached to customer. How we can address defects in other lifecycle phases i.e. deployment and post production activities?

These are very interesting questions. Here is my two cents worth:

1. Your assumption that testing/QA phase is not a source of defects may or may not be a good assumption. Test cases can be a source of defects as well as the product. A test case that does not find any defects may in fact itself be defective. The purpose of a test case is to find defects in the product. In this case there might be undetected defects slipping through the test cases or erroneous test results. On the other hand testing can sometimes identify defects in the product that may not actually exist. I encountered this situation years ago in a small software development company when they hired a new tester. The company had no documented procedures and testing criteria. So the new tester did the best job he could under these circumstances and based his testing on the published User’s Manual. He proceeded to find tons of defects in the product. That got management’s attention until they realized that all of these “new” defects had been previously addressed. It turned out that the tester was unwittingly using an obsolete version of the User’s Manual. So that mistake invalidated all of his test results. Either way, each of these two situations would contaminate your DRE calculations. Therefore, I would be very careful in making the assumption that the testing/QA phase is not a source of defects, unless you have thoroughly peer reviewed each test case and validated each test result to ensure that what I have described has not happened.
   
2. In the context of the CMMI, QA usually means for most of us Lead Appraisers PPQA – conducting process audits and work product audits, not testing. So I am not clear as to the distinction you are trying to make between testing and "pure QA". It appears that in your organization they may be two aspects of the same activity. Personally, I think that you could use a common DRE format across all types of projects. However, it doesn’t make sense to mix the data for the various projects. So you will have to be very careful to keep the data segregated, otherwise you won’t be able to draw the proper conclusions. For example, the DRE for new development will be different than for sustaining engineering, different from a testing project, etc.
   
3. I think that you answered your question in your statement of the scope. Just change the definition of the scope for your DRE. You could define the DRE scope on a lifecycle phase by phase basis and then you would be able to measure it for deployment and post-production activities. You could also define a DRE for the entire project lifecycle as well.

Understanding How Project Planning Evolves from ML 3 to ML 5

The primary difference between project planning at Maturity Level 3 (ML 3) and Maturity Level 5 (ML 5) is how you construct the project’s defined processes. At ML 3 a new project starts from the organization’s set of standard processes and applies the appropriate tailoring criteria or guidelines for the project, based on the project requirements. At Maturity Level 4 (ML 4) and ML 5 a new project first has to define what its quality and process performance objectives are based on the organization’s quality and process performance objectives (QPPOs). Then the project looks at the organization’s set of standard processes and selects the processes or sub-processes that are capable of being quantitatively and statistically managed to achieve the project’s QPPOs using the established Process Performance Baselines and Process Performance Models. These processes and sub-processes are selected based on historical stability and capability data along with the appropriate tailoring guidelines and criteria to construct the project’s defined process.

Understanding Causal Analysis and Resolution (CAR)

In the context of the CMMI, CAR is a Maturity Level 5 Process Area. You can perform CAR qualitatively at lower Maturity Levels, but you cannot realize the full benefits of CAR until you have stable processes and statistically understand the process capabilities. Let’s step through the practices one-by-one and talk about what they mean from a quantitative perspective;

SP 1.1 Select the defects and other problems for analysis.
You can perform this practice very simply and qualitatively just by having a gut feel that something is wrong and you want to investigate its causes. However, you should be using data and statistical tools such as Pareto analysis, histograms, or process capability analysis to determine which defects to analyze.

SP 1.2 Perform causal analysis of selected defects and other problems and propose actions to address them.
You can perform this practice qualitatively just by meeting with those people responsible for performing the process, using tools like cause-and-effect (fishbone) diagrams or check sheets. From a Maturity Level 5 perspective, you may have criteria that states you perform causal analysis when a stable process does not quantitatively meet its specified quality and process performance objective.

SP 2.1 Implement the selected action proposals that were developed in causal analysis.
Again it is possible to perform this practice qualitatively simply by implementing an action as the result of the cause-and-effect diagram. From a Maturity Level 5 perspective, you may be designing an experiment to see if the proposed action will have the desired quantitative results.

SP 2.2 Evaluate the effect of the changes on process performance.
This practice will be difficult to perform qualitatively. The intent of this practice is to measure the effects of the change on the process performance, which can only be done accurately if you have stable and capable processes in place and you are using quantitative methods to manage these processes.

SP 2.3 Record causal analysis and resolution data for use across the projects and organization.
If you are performing qualitative causal analysis you may not have any data to record other than results from the cause-and-effect diagram and rationale for the change. From a Maturity Level 5 perspective you might have data on the defects and other problems you analyzed and measures of the changes to the process performance resulting from the causal analysis.