Process Performance Model and QPM Inquiry

We are a CMMI Maturity Level 5 organization and now we are preparing for our re-appraisal.
We have Process Performance Models (PPMs) with different X- prameters and good R^2, R (adjusted) & P(value).

We perform simulations to calculate the probability of achieving our organizational goal and the determination that the probability % is good enough to achieve the goal.

Now we are in our process of implementing the PPM. My question is: Shall we use the PPM to estimate the X- parameters in my projetct? Or what should I do with this model after that ?

Please advise.

I hate to say this, and I could be wrong, but reading your question it sounds like you are asking what do I need in order to be Maturity Level 5 (ML 5)? Is this equation sufficient? And by asking what do I do with the PPM, I may be misunderstanding you, but if you were really at ML 5 you wouldn’t be asking this question. I get the impression from your question that your organization doesn’t know why it is using a specific technique for the PPMs.

The best solution to your question is to ask your High Maturity Lead Appraiser for help and guidance as he or she should have enough understanding and knowledge of your organization to provide the answer.

But once again, the nature of your question gives me the uneasy feeling that your organization may not even be Maturity Level 4.

Rationale for Maturity Level 5

Our organization is currently at Maturity Level 3 and is now planning to go for Maturity Level 5. Our CEO is asking for the short term and long term benefits and challenges for implementing Maturity Level 5. How do we provide this information?

The question that needs to be asked is why does your organization want to achieve Maturity Level 5 (ML 5)? If the organization has already achieved Maturity Level 3 (ML 3), what is the motivation for achieveing ML 5? Most likely it is not mandated in a contract or by its customers. But achieving ML 5 is desirable to be competitive in the marketplace. Therefore, the reasons for achieving ML 5 should provide some indication of the short term and long term benefits.

The first step in becoming a High Maturity organization is defining your Quality and Process Performance Objectives (QPPOs) that are based on your business goals and objectives as well as your customer needs. The QPPOs should be stated in a form such that they specify a timeframe. And that information will provide some ideas of the short term and long term benefits.

However, the best advice that I can provide is to hire an SEI-certified High Maturity Lead Appraiser (HMLA) who will work with you to help explain the benefits of ML 5 to the CEO. The HMLA should have experience working with many different organizations at various Maturity Levels and be able to talk about the different challanges that have been faced by other organizations, as well as the challenges and risks within your organization.

At this point, it sounds like your organization is just beginning its journey to ML 5, so I would have to be convinced that your processes are stable enough to provide the data needed to quantify any short term and long term benefits. I think that the best you could do at this point is communicate this information in qualitative terms. Any quantitative information may not be accurate until you have implemented High Maturity.

Defect Removal Efficiency (DRE)

I am a Defect Prevention team member in our CMMI Maturity Level 5 company. We are trying to plot the Defect Removal Efficiency (DRE) metric in our projects. I have a few questions:

1. Our DRE is based on phase level defect injection and detection concepts. If we go closely by the definition of the phase "testing/QA", bugs should not be injected in testing phase, as this is detection activity. If not then any examples can be useful for me to understand.
2. Can a common DRE format can be utilized for all kinds of projects as in sustaining engineering, maintenance, development and pure testing? Specifically what do to we do for testing or pure QA type of projects?
3. DRE is scoped for defects leaked by QA and reached to customer. How we can address defects in other lifecycle phases i.e. deployment and post production activities?

These are very interesting questions. Here is my two cents worth:

1. Your assumption that testing/QA phase is not a source of defects may or may not be a good assumption. Test cases can be a source of defects as well as the product. A test case that does not find any defects may in fact itself be defective. The purpose of a test case is to find defects in the product. In this case there might be undetected defects slipping through the test cases or erroneous test results. On the other hand testing can sometimes identify defects in the product that may not actually exist. I encountered this situation years ago in a small software development company when they hired a new tester. The company had no documented procedures and testing criteria. So the new tester did the best job he could under these circumstances and based his testing on the published User’s Manual. He proceeded to find tons of defects in the product. That got management’s attention until they realized that all of these “new” defects had been previously addressed. It turned out that the tester was unwittingly using an obsolete version of the User’s Manual. So that mistake invalidated all of his test results. Either way, each of these two situations would contaminate your DRE calculations. Therefore, I would be very careful in making the assumption that the testing/QA phase is not a source of defects, unless you have thoroughly peer reviewed each test case and validated each test result to ensure that what I have described has not happened.
   
2. In the context of the CMMI, QA usually means for most of us Lead Appraisers PPQA – conducting process audits and work product audits, not testing. So I am not clear as to the distinction you are trying to make between testing and "pure QA". It appears that in your organization they may be two aspects of the same activity. Personally, I think that you could use a common DRE format across all types of projects. However, it doesn’t make sense to mix the data for the various projects. So you will have to be very careful to keep the data segregated, otherwise you won’t be able to draw the proper conclusions. For example, the DRE for new development will be different than for sustaining engineering, different from a testing project, etc.
   
3. I think that you answered your question in your statement of the scope. Just change the definition of the scope for your DRE. You could define the DRE scope on a lifecycle phase by phase basis and then you would be able to measure it for deployment and post-production activities. You could also define a DRE for the entire project lifecycle as well.

SEI Published Appraisal Results

I have a question related to the published data in SEI web site: http://www.sei.cmu.edu/appraisal-program/profile/pdf/CMMI/2008MarCMMI.pdf Process Maturity Profile by All Reporting Organizations (Page 5):

1. 1.5% or the reporting organizations are (Initial - Maturity Level 1) rating. How is a Maturity Level 1 rating achieved?
2. Also, 8% of the reporting organizations indicate that a Maturity Level is Not Given. What does ‘Not Given’ mean…? Level 0 ?

In my understanding, Maturity Level is related to the Staged representation and can have
rating from ML 2 - 5.

It is great to see and hear that people are actually looking at this information and asking questions about how to interpret the data. You are on your way to understanding Measurement and Analysis!

To answer your first question, how is Maturity Level 1 provided? Obviously, an organization does not conduct a SCAMPI A appraisal to confirm that they are Maturity Level 1. So what happens in this case is the scope of the SCAMPI A appraisal is Maturity Level 2, 3, 4, or 5. But what happens is during the appraisal a serious goal impacting weakness in a Maturity Level 2 Process Area is found by the appraisal team. By the rules of the SCAMPI method then the resulting Maturity Level is 1. Since the SCAMPI A results must be submitted to the SEI, a rating of Maturity Level 1 appears. It doesn’t happen that often because most organizations take the appropriate steps to be prepared to achieve their target Maturity Level. That is why 1.5% of the reported results are for Maturity Level 1.

The Not Given category means that the organization did not report its resulting Maturity Level rating. I would hazard a guess that it probably means that they had successfully achieved Maturity Level 1. There is no Maturity Level 0, despite what many chaotic organizations say they are when first encountering the CMMI.

ML 2/ML 3 Measurement Objectives

For a Maturity Level 3 organization , is it acceptable if the organization sets its process performance objectives qualitatively? For example reduce testing cycle time , improve productivity, etc. and in OPF SG1, one of sub-practices states that process performance objectives may be expressed either quantitatively or qualitatively. Also would you please explain this clarification linking to ML2- MA - SP 1.1 - Establish measurement objective?

These are good questions, but the best way to receive a decent understanding of the concepts it to attend the SEI’s Introduction to the CMMI class. Intro to CMMI course description Brief answers, such as mine in this blog, will probably only raise more questions that would be better addressed in a classroom setting.

1. Only for High Maturity organizations (ML 4 and 5) is there an expectation for quantitatively setting process performance objectives. It is acceptable at ML 3 for qualitative objectives. But to fully understand this distinction you really require a good discussion on the expectations for each Maturity Level, which you would receive in the Intro to CMMI class.
   
2. A measurement objective is different from a process performance objective. The measurement objectives are the purposes for which measurement and analysis is performed and also specify what kinds of actions may be taken as a result of data analyses. In order to determine the measurement objectives, you first have to describe the information needs. An information need is an insight necessary to manage objectives, goals, issues, risks, and problems. The measurement objective is derived from the information need and it is a statement about what should be measured in order to satisfy the information need. For example, the project manager or other decision maker concerned with allocating budget and associated resources to a task may believe that productivity is related to the type of task being performed. Increasing productivity is then the measurement objective that addresses the defined information need. Determining productivity then requires that entities such as the product and the process be measured. There are many ways that productivity can be computed, but the measurement objective is unaffected by the different methods.

To fully understand these concepts you should take a class on Measurement and Analysis or Practical Software Measurement. Trying to explain these concepts in a paragraph or two does not do the subject justice.

Query on ML 3 and ML 4

What is the significance of Maturity Level 3 and Maturity Level 4? And can you explain to me what is Integrated Project Management?

What broad questions! These questions really need a long in depth answer and are addressed very well in the Introduction to CMMI class. So first off I would suggest that you find an opportunity to take this class. Please visit http://www.ppqc.net/training/training.htm for more information about the class.

To briefly answer these two questions, the answer needs to address ML 2 as well. So I will start with some definitions from the CMMI book.
Process Area (PA) – a cluster of related practices in an area that, when implemented collectively, satisfy a set of goals considered important for making improvement in that area.
Maturity Level (ML) – degree of process improvement across a predefined set of process areas in which all goals in the set are attained. An ML is a defined evolutionary plateau for organization process improvement. Each ML matures an important subset of the organization’s processes, preparing it to move to the next ML.
Maturity Level 1: Initial – processes are usually ad hoc and chaotic. The organization usually does not provide a stable environment to support the process. Success in these organizations depends on the competence and heroics of the people in the organization and not on the use of proven processes.
Maturity Level 2: Managed – projects of the organization have ensured that processes are planned and executed in accordance with policy; the projects employ skilled people who have adequate resources to produce controlled outputs; involve relevant stakeholders; are monitored, controlled, and reviewed; and are evaluated for adherence to their process descriptions, The process discipline reflected by ML 2 helps to ensure that existing practices are retained during times of stress. When these practices are in place, projects are performed and managed according to their documented plans.
Maturity Level 3: Defined – processes are well characterized and understood, and are described in standards, procedures, tools, and methods. The organization’s set of standard processes, which is the basis for ML 3, is established and improved over time. These standard processes are used to establish consistency across the organization. Projects establish their defined process by tailoring the organization’s set of standard processes according to tailoring guidelines.
Maturity Level 4: Quantitatively Managed – the organization and projects establish quantitative objectives for quality and process performance and use them as criteria in managing processes. Quantitative objectives are based on the needs of the customer, end users, organization, and process implementers. Quality and process performance is understood in statistical terms and is managed throughout the life of the processes.
Maturity Level 5: Optimizing – an organization continually improves its processes based on a quantitative understanding of the common causes of variation inherent in processes.

Given these definitions and explanations, one of the fundamental differences between ML 3 and ML 4 is that at ML 3 the organization is learning how to use a standard set of processes, tailoring them to the individual project needs, and collecting enough process data such that Process Performance Baselines and Process Performance Models can be built and used at ML 4 to quantitatively manage projects and statistically manage sub-processes to achieve the organization’s quality and process performance objectives.

To answer the second question, you first need to understand the Project Planning (PP), Project Monitoring and Control (PMC), and Integrated Project Management (IPM) PAs. PP and PMC are ML 2 PAs that address the basic project management practices of planning a project, creating a project plan, and using that project plan to track and monitor the project. At ML 2, the organization typically is learning how to create accurate and realistic estimates by building estimation models. It takes time to refine these estimation models, so an ML 2 organization is expected to frequently revise and re-baseline the project plan as the projects get smarter about estimation. At ML 3, one of the project management expectations is that the project estimates are now accurate and realistic. So, rather than constantly update the estimates to match the actuals as done at ML 2, the Project Manager now manages the project to the estimates, meaning that the PM can now fairly accurately predict early on in the lifecycle whether or not the project will hit its downstream targets and take appropriate corrective action to mitigate these risks. The other differences between IPM and PP/PMC include establishing the project’s defined process by applying appropriate tailoring criteria to the organization’s standard processes, establishing the project’s work environment, integrating the various plans that comprise the project plan, managing the project using the integrated plans, and managing the project’s relevant stakeholders. In other words, IPM builds on the project management foundation established by PP and PMC.

This a lengthy explanation but only a surface treatment on these subjects. Again I strongly recommend to anyone interested in this topic that you attend an offering of the Introduction to CMMI v1.2 class. You will go into these concepts in much greater detail and you will come out with a much better understanding of the model, PAs, and MLs than I can convey in this blog.