Appraisal Scope Question for Agile Development

I am preparing for an appraisal coming up in a few months and I have run into an interesting question that I would like to ask you about.

Some of the projects in the Organizational Unit (OU) follow an Agile method for development. It is really a modified combination of SCRUM and XP. One of the questions they asked me was whether they would have to provide evidence for all of the Process Areas (PAs) in the scope of the appraisal? I was perplexed by this question because it seemed rather obvious to me, so I replied "of course." But this question stuck in the back of my mind so I dug into the SCAMPI Method Definition Document (MDD) for more clarification.

From the MDD, Method Assumptions and Design Principles, page I-19 and I-20: "The extent to which an organizational unit has implemented appraisal reference model practices can be determined only by considering, in aggregate, the extent to which those practices are implemented within the organizational unit by project and support groups. This process, in turn, necessitates the consideration of objective evidence for each instantiation, for each model practice within the appraisal scope."

This seems fairly clear to me, we must examine objective evidence for all practices of all instantiations within the scope of the appraisal. However, under the process definition in part 2 of the MDD you can find less convincing information under the Parameters and Limits section of 1.1.3 Determine Appraisal Scope. "Sample projects and support groups selected to form the organizational scope (i.e., the combination of focus and non-focus projects and support functions) must represent all critical factors identified for the organizational unit to which the results will be attributed." So it seems that as long as the combination of all focus and non-focus projects represent all the critical factors it should be sufficient. I know, critical factors are different from the PAs in the model scope. :-)

Also, in Parameters and Limits it clearly states: "Focus projects must provide objective evidence for every PA within the model scope of the appraisal..." "Non-focus projects must provide objective evidence for one or more PAs within the model scope of the appraisal..." So it seems that if an Agile project was a non-focus project they would not have to provide evidence for all PAs. Do you think this is correct?

The MDD goes on to further say in the Parameters and Limits: "In appraisals where the reference model scope includes any project-related PA, the organizational scope must include at least one focus project. If the organizational unit includes more than 3 projects, then the organizational scope must include sufficient focus projects and non-focus projects to generate at least 3 instances of each practice in each project-related PA in the model scope of the appraisal."

So, conceivably, you could have only 1 focus project, and say maybe 6 non-focus projects with each only having a few PAs to "generate at least 3 instances" of the practices that need to be covered in the scope of the appraisal.

I know this may seem a bit extreme, but believe me, I have run into stranger scenarios than this one. lol :-)

Ideally, all projects would cover all PAs. But the role and function of non-focus projects really seems to muddy the waters in my opinion.

What are your thoughts?

There is an SEI Technical Note CMU/SEI-2008-TN-003 published in November 2008 titled CMMI® or Agile: Why Not Embrace Both! by Hillel Glazer, Jeff Dalton, David Anderson, Mike Konrad, and Sandy Shrum that contains a lot of pertinent information about Agile and the CMMI, as well as Hillel's recently published companion article in Crosstalk Love and Marriage: CMMI and Agile Need Each Other http://www.stsc.hill.af.mil/crosstalk/2010/01/1001Glazer.html

In my opinion, if Agile is one of many different development methods in use by the organization and the Agile projects do not cover all of the PAs in scope of the appraisal, then it is not in the spirit of the MDD and non-focus projects to mix and match a number of Agile projects just so there is evidence for all the PAs. The way that I look at non-focus projects is that there may be some projects in the appraisal scope that do not have evidence because the projects have not reached that part of the lifecycle yet, NOT that they are not performing the practice.

And, just because a project is using Agile in a Maturity Level (ML) 3 organization does not mean that they don’t have to perform all of the PAs. There should be evidence available though they may be using terminology that is not obviously mapping to the CMMI. Someone with deep CMMI and Agile knowledge should work with the organization to show them how what they are doing does comply with the CMMI.