Appraisal Scope Question for Agile Development

I am preparing for an appraisal coming up in a few months and I have run into an interesting question that I would like to ask you about.

Some of the projects in the Organizational Unit (OU) follow an Agile method for development. It is really a modified combination of SCRUM and XP. One of the questions they asked me was whether they would have to provide evidence for all of the Process Areas (PAs) in the scope of the appraisal? I was perplexed by this question because it seemed rather obvious to me, so I replied "of course." But this question stuck in the back of my mind so I dug into the SCAMPI Method Definition Document (MDD) for more clarification.

From the MDD, Method Assumptions and Design Principles, page I-19 and I-20: "The extent to which an organizational unit has implemented appraisal reference model practices can be determined only by considering, in aggregate, the extent to which those practices are implemented within the organizational unit by project and support groups. This process, in turn, necessitates the consideration of objective evidence for each instantiation, for each model practice within the appraisal scope."

This seems fairly clear to me, we must examine objective evidence for all practices of all instantiations within the scope of the appraisal. However, under the process definition in part 2 of the MDD you can find less convincing information under the Parameters and Limits section of 1.1.3 Determine Appraisal Scope. "Sample projects and support groups selected to form the organizational scope (i.e., the combination of focus and non-focus projects and support functions) must represent all critical factors identified for the organizational unit to which the results will be attributed." So it seems that as long as the combination of all focus and non-focus projects represent all the critical factors it should be sufficient. I know, critical factors are different from the PAs in the model scope. :-)

Also, in Parameters and Limits it clearly states: "Focus projects must provide objective evidence for every PA within the model scope of the appraisal..." "Non-focus projects must provide objective evidence for one or more PAs within the model scope of the appraisal..." So it seems that if an Agile project was a non-focus project they would not have to provide evidence for all PAs. Do you think this is correct?

The MDD goes on to further say in the Parameters and Limits: "In appraisals where the reference model scope includes any project-related PA, the organizational scope must include at least one focus project. If the organizational unit includes more than 3 projects, then the organizational scope must include sufficient focus projects and non-focus projects to generate at least 3 instances of each practice in each project-related PA in the model scope of the appraisal."

So, conceivably, you could have only 1 focus project, and say maybe 6 non-focus projects with each only having a few PAs to "generate at least 3 instances" of the practices that need to be covered in the scope of the appraisal.

I know this may seem a bit extreme, but believe me, I have run into stranger scenarios than this one. lol :-)

Ideally, all projects would cover all PAs. But the role and function of non-focus projects really seems to muddy the waters in my opinion.

What are your thoughts?

There is an SEI Technical Note CMU/SEI-2008-TN-003 published in November 2008 titled CMMI® or Agile: Why Not Embrace Both! by Hillel Glazer, Jeff Dalton, David Anderson, Mike Konrad, and Sandy Shrum that contains a lot of pertinent information about Agile and the CMMI, as well as Hillel's recently published companion article in Crosstalk Love and Marriage: CMMI and Agile Need Each Other http://www.stsc.hill.af.mil/crosstalk/2010/01/1001Glazer.html

In my opinion, if Agile is one of many different development methods in use by the organization and the Agile projects do not cover all of the PAs in scope of the appraisal, then it is not in the spirit of the MDD and non-focus projects to mix and match a number of Agile projects just so there is evidence for all the PAs. The way that I look at non-focus projects is that there may be some projects in the appraisal scope that do not have evidence because the projects have not reached that part of the lifecycle yet, NOT that they are not performing the practice.

And, just because a project is using Agile in a Maturity Level (ML) 3 organization does not mean that they don’t have to perform all of the PAs. There should be evidence available though they may be using terminology that is not obviously mapping to the CMMI. Someone with deep CMMI and Agile knowledge should work with the organization to show them how what they are doing does comply with the CMMI.

High Maturity Reference Books

I would like to read and understand more about High Maturity practices and I am also very keen to develop statistical skills so I will be able to construct a PPM. Can you please recommend some good reference books and/or links?

I would recommend the following books:

“Statistical Methods from the Viewpoint of Quality Control” by Walter A. Shewhart

“Understanding Variation: The Key to Managing Chaos” by Donald J. Wheeler

“Measuring the Software Process” by William A. Florac and Anita D. Carleton

“Building Continual Improvement” by Donald J. Wheeler and Sheila R. Poling

“Metrics and Models in Software Quality Engineering” by Stephen H. Kan

“Practical Software Measurement” by John McGarry, David Card, Cheryl Jones, Beth Layman, Elizabeth Clark, Joseph Dean, and Fred Hall

These are all excellent reference books to better your understanding of statistical and quantitative thinking.

Shortened CMMI Explanation

I have just started CMMI in my organization. I am looking for the shortened explanation for each of the CMMI Process Areas. I then plan to preach CMMI in much simpler, easy to understand language which would be like bulleted points. I also intend to offer seminars in my organization with different departments to make them understand what exactly do they are required to do. If you can help me, I would be very thankful.

I applaud your initiative and your plans. The best advice that I can give you is to take the SEI’s 3-day Introduction to CMMI class. That class will provide you with the necessary information for you to craft a shortened version for your organization. As an alternative, you can read the CMMI. The introduction of each Process Area provides a short explanation of the intent of each Process Area. But if you do not have any background with the CMMI, hire a CMMI consultant and explain to him or her your needs and have them help you prepare the training material you desire. What you are asking for is not free. It takes time and effort to produce and is a consultant's intellectual property.

PPQA After Maturity Level 2

I work in an IT organization that achieved CMMI Maturity Level 2 several years ago (we let the rating lapse) and I was wondering if you had some ideas on the following two questions:

1) What types of activities would PPQA engage in if the org had been Maturity Level 2 (I think they could have pursued Maturity Level 3 and been close)? Please also consider that the company is pursuing other types of improvement methods and models such as lean/6-sigma and ITIL.

2) What strategies should we pursue to show the worth of PPQA? Even in the good old CMM days and SQA one of the issues I had was that it was difficult to show the practical monetary worth of these support functions; one generally had to take it on faith that PPQA/SQA delivered some degree of worth to the company. Any thoughts?

The answer to question 1 is simple. Just read the PPQA Process Area and GP 2.9. The PPQA activities include performing both process and work product audits of the project and organization processes. For Maturity Level 2 that would mean auditing your REQM, PP, PMC, SAM, MA, PPQA, and CM processes.

The answer to question 2 is a bit more difficult. Basically you are asking, what is the cost of quality? One method you can use is to look at the total cost for the project and analyze it using Crosby’s Cost of Quality Model. The total costs break down into two categories: the Cost of Quality and the Cost of Performance.

The Cost of Performance includes such things as: generating plans, documentation, and developing requirements, design, code, and integration.

The Cost of Quality breaks down further into two categories: Cost of Conformance and Cost of Non-Conformance.

The Cost of Non-Conformance includes fixing defects, reworking documents, updating source code, re-reviews, re-tests, patches, engineering changes, CCBs, external failures and fines, Customer Support, and Help Desk.

The Cost of Conformance breaks down to two more categories: Cost of Appraisal and Cost of Prevention.

The Cost of Appraisal includes reviews, walkthroughs, testing (first time), independent V&V, and Audits.

The Cost of Prevention includes training, policies, procedures, tools, planning, quality improvement, data gathering and analysis, root cause analysis, and quality reporting.

The cost of PPQA is included in the Cost of Prevention.

When you consider these definitions and cost break down, the only category that will be affected by PPQA is the Cost of Non-Conformance. When PPQA audits the processes and work products, the audits will reveal non-conformances with people following the documented processes and procedures, which lead to re-work. By addressing these non-conformances, the goal is to reduce or effectively eliminate the rework and that is where you can demonstrate the value of PPQA.

Hope this helps.

Product Vision vs. Product Roadmap

I have a question. Is the Product/ System Solution Vision and the Product/ System Solution roadmap one and the same thing? And which comes first, the Product Vision or the Product Roadmap?

Product Vision and Product Roadmap are not CMMI terms. So, in the context of the CMMI, simply use the standard dictionary definitions of vision and roadmap and you should have your answer.

The way I look at these two concepts is that vision refers to what you are supposed to achieve and roadmap is how you get there. Therefore vision would come first followed by the roadmap.

The bottom line is that you should define these terms for your organization if you think that they apply and are important.

Manage Corrective Actions of PMC

During the analysis of a problem management tool I identified several actions that were open without a deadline and status record or with a deadline expired and without status record. Some actions were without a deadline because they depended on external agents to be resolved (customer, priority addressed by higher level) and others without updating the history and deadline because these didn't have modified status. The monitoring of these actions was performed through interviews. What is the impact of this condition on sub-practice 2.3-1 - Manage Corrective Action of PMC? Does this condition affect the reach of this practice? For me this condition characterizes a non-compliance. I would like to know your opinion.

The correct answer depends upon what you have written in your documented procedures for identifying issues and taking corrective actions. Do your processes and procedures allow you to to conduct verbal reviews of issues and take corrective actions? I do agree with you that you have identified some problems with your process and the proper completion of forms.

If you believe these are true non-conformances, but your documented processes allow this to happen, then it would be a good idea to modify your processes accordingly. Otherwise, you have identified a gap between the documented and practiced processes. These issues should have also been identified through the PPQC process and work product audits.

Also, since the sub-practices are an informative component, they provide additional material to help you understand the intent of the Specific Goals and Practices. There is no expectation that the sub-practices have to be implemented. Therefore, the situation you described really has no impact on PMC SP 2.3 Sub-practice 1.