Appraisal Scope Question for Agile Development

I am preparing for an appraisal coming up in a few months and I have run into an interesting question that I would like to ask you about.

Some of the projects in the Organizational Unit (OU) follow an Agile method for development. It is really a modified combination of SCRUM and XP. One of the questions they asked me was whether they would have to provide evidence for all of the Process Areas (PAs) in the scope of the appraisal? I was perplexed by this question because it seemed rather obvious to me, so I replied "of course." But this question stuck in the back of my mind so I dug into the SCAMPI Method Definition Document (MDD) for more clarification.

From the MDD, Method Assumptions and Design Principles, page I-19 and I-20: "The extent to which an organizational unit has implemented appraisal reference model practices can be determined only by considering, in aggregate, the extent to which those practices are implemented within the organizational unit by project and support groups. This process, in turn, necessitates the consideration of objective evidence for each instantiation, for each model practice within the appraisal scope."

This seems fairly clear to me, we must examine objective evidence for all practices of all instantiations within the scope of the appraisal. However, under the process definition in part 2 of the MDD you can find less convincing information under the Parameters and Limits section of 1.1.3 Determine Appraisal Scope. "Sample projects and support groups selected to form the organizational scope (i.e., the combination of focus and non-focus projects and support functions) must represent all critical factors identified for the organizational unit to which the results will be attributed." So it seems that as long as the combination of all focus and non-focus projects represent all the critical factors it should be sufficient. I know, critical factors are different from the PAs in the model scope. :-)

Also, in Parameters and Limits it clearly states: "Focus projects must provide objective evidence for every PA within the model scope of the appraisal..." "Non-focus projects must provide objective evidence for one or more PAs within the model scope of the appraisal..." So it seems that if an Agile project was a non-focus project they would not have to provide evidence for all PAs. Do you think this is correct?

The MDD goes on to further say in the Parameters and Limits: "In appraisals where the reference model scope includes any project-related PA, the organizational scope must include at least one focus project. If the organizational unit includes more than 3 projects, then the organizational scope must include sufficient focus projects and non-focus projects to generate at least 3 instances of each practice in each project-related PA in the model scope of the appraisal."

So, conceivably, you could have only 1 focus project, and say maybe 6 non-focus projects with each only having a few PAs to "generate at least 3 instances" of the practices that need to be covered in the scope of the appraisal.

I know this may seem a bit extreme, but believe me, I have run into stranger scenarios than this one. lol :-)

Ideally, all projects would cover all PAs. But the role and function of non-focus projects really seems to muddy the waters in my opinion.

What are your thoughts?

There is an SEI Technical Note CMU/SEI-2008-TN-003 published in November 2008 titled CMMI® or Agile: Why Not Embrace Both! by Hillel Glazer, Jeff Dalton, David Anderson, Mike Konrad, and Sandy Shrum that contains a lot of pertinent information about Agile and the CMMI, as well as Hillel's recently published companion article in Crosstalk Love and Marriage: CMMI and Agile Need Each Other http://www.stsc.hill.af.mil/crosstalk/2010/01/1001Glazer.html

In my opinion, if Agile is one of many different development methods in use by the organization and the Agile projects do not cover all of the PAs in scope of the appraisal, then it is not in the spirit of the MDD and non-focus projects to mix and match a number of Agile projects just so there is evidence for all the PAs. The way that I look at non-focus projects is that there may be some projects in the appraisal scope that do not have evidence because the projects have not reached that part of the lifecycle yet, NOT that they are not performing the practice.

And, just because a project is using Agile in a Maturity Level (ML) 3 organization does not mean that they don’t have to perform all of the PAs. There should be evidence available though they may be using terminology that is not obviously mapping to the CMMI. Someone with deep CMMI and Agile knowledge should work with the organization to show them how what they are doing does comply with the CMMI.

Project Specific Findings

While reading the SCAMPI A Method Definition Document (MDD), I discovered that providing project specific findings is an option that can be requested by the project sponsor. I always believed that the Findings Presentation could only include findings at organizational unit level. So if the sponsor requests project specific findigs as part of the appraisal output, how are they communicated? Included in the final findings? If this is the case, what about the non-attribution of findings? Or via a separate document?

The SCAMPI Method does indeed allow for project specific findings if the Appraisal Sponsor requests them. If requested, then project specific findings could be communicated in the Final Findings Presentation along with the other information, or they could be communicated separately. Reporting will be negotiated and documented in the Appraisal Plan. Keep in mind that even if the Appraisal Sponsor doesn’t request project specific findings, the PIIDs will contain project specific observations, so if someone were interested in finding out about project specific information, all they would have to do would be to read the PIIDs. The organization is required to retain the PIIDs used for the appraisal for three years, the same length of time that the appraisal results are valid.

The non-attribution issue concerns identifying any individual, project, or group as the SOURCE of the information. So by reporting project specific findings, the Lead Appraiser and Appraisal Team are not violating the non-attribution rules unless they reveal the source of the finding(s). When I conduct an appraisal, the final Appraisal Team activity is to scrub the PIIDs of all attribution information (names, interview sessions, etc.) and deleting all previous versions so the organization only retains the scrubbed PIIDs.

Appraisal Team Member Qualification Requirements

What are the requirements to be an Appraisal Team Member (ATM) in SCAMPI A appraisal?

I have heard many opinions on ATM qualifications, such as:

1. Not part of process definition
2. Not involved with Process Implementation
3. Not being an SEPG member.

I am unaware of any criteria set by the SEI on this topic.

The ATM qualifications are spelled out in the SCAMPI Method Definition Document (MDD) Section 1.3.2 SELECT TEAM MEMBERS. I have extracted the pertinent text here.

Parameters and Limits
The minimum acceptable team size for a SCAMPI A appraisal is four people (including the appraisal team leader).
All team members must have previously completed the SEI-licensed Introduction to CMMI course.
With regard to engineering field experience, the team (as a group) must have an average of at least 6 years of experience, and the team total must be at least 25 years of experience in each of the disciplines to be covered in the appraisal.
With regard to management experience, the team (as a group) must have a total of at least 10 years of experience, and at least one team member must have at least 6 years of experience as a manager.
The team must, in aggregate, have representative experience in the lifecycles being appraised.

Optional Practices
Although not required in the Parameters and Limits section above, the following are considered recommended best practices and should be employed whenever feasible:

• Each member should have good written and oral communication skills, the ability to facilitate the free flow of communication, and the ability to perform as team players and negotiate consensus.
• At least half of the team members should have participated in a previous process appraisal.
• Team members should be perceived by the appraisal sponsor as credible.

Additional appraisal team member selection considerations include

• Consider the personal characteristics of individual team members (e.g., communication preferences and personality types) and how these characteristics may affect the dynamics of the team.
• Use one or more authorized SCAMPI Lead Appraisers as team members.

And there is one more ATM requirement that is documented in the MDD errata. The Appraisal Sponsor cannot be an Appraisal Team Member.

What is Meant by a Line of Business?

In a CMMI 1.2 Appraisal there is a requirement that practices in projects and functions within the Organisational Unit must be understood and identified ( e.g. Lines of Business, Disciplines, Effort Types, Project Types etc). What do lines of Business and Effort Types mean?

It might be easier to understand the concept of a LIne of Business through an illustrative example.

A small organization would normally produce one type of product, say a voice recognition software package. This package may be installed on a variety of platforms, but it is the same package. In this situation the company or organization has one Line of Business.

In a large company or corporation, there usually is a number of different types of products produced for different purposes, customer types, etc. For example, an automotive company may have several divisions: car, truck, van, commercial vehicle, etc. Each division represents a different Line of Business. Or a financial company may have several divisions: banking, insurance, investments, etc. Again, each division is a different Line of Business. Typically each Line of Business has different goals and objectives, customers, processes, etc. Another example of a Line of Business is the application domain. It is important to note the different Lines of Business because it is extremely challenging, if not impossible, to conduct a single SCAMPI A appraisal across multiple Lines of Business, especially for Maturity Level 3 and up.

If you read the Method Description Document section 1.1.3 Implementation Guidance, it defines the terms you are asking about:

• application domains (or lines of business)
• geographical breadth
• disciplines (e.g., systems engineering, software engineering, or hardware engineering)
• effort types (e.g., development, maintenance, or services)
• project types (e.g., legacy or new development)
• customer types (e.g., commercial or government agency)
• lifecycle models in use within the organization (e.g., spiral, evolutionary, waterfall, or incremental)

So effort type refers to the type of work: new development, sustaining engineering or maintenance, or services.

Criteria for selection of Focus and Non Focus projects for CMMI 1.2 Assessment

What are the criteria used to determine project selection for CMMI 1.2 Appraisals? I understand they are normally termed as Focus and Non Focus projects.

1. Based on size of organization ( count of projects + resources ) how many projects need to be selected? Can someone specify this as a number of projects or percentage of projects required?
2. We have categorised projects as Development + Maintenance + Testing - so ideally how many projects count or percentage wise under each Project Type would need to be selected?
3. What is the criteria for selecting a project as a focus project? I have read and heard that one of the factors could be presence of all Software Development Lifecycle (SDLC) phases in that project, contribution of data points, it should be an on-going project at time of assessment etc.

Here is the criteria for project selection extracted from section 1.1.3 of the Method Description Document (MDD) v1.2:

Sample projects and support groups selected to form the organizational scope (i.e., the combination of focus and non-focus projects and support functions) must represent all critical factors identified for the organizational unit to which the results will be attributed. The coverage of the organizational critical factors provided by these sample projects and support groups in the organizational scope in relation to the organizational unit must be documented, in quantitative terms, in the appraisal input and ADS.

Each sample project or support group in the planned organizational scope of the appraisal must be one of the three types listed below:

• Focus projects must provide objective evidence for every PA within the model scope of the appraisal which addresses model practices applicable to those projects.
• Non-focus projects must provide objective evidence for one or more PAs within the model scope of the appraisal which address practices performed on projects.
• Support functions must provide objective evidence for practices within the model scope of the appraisal that address organizational infrastructure or functions.

In appraisals where the reference model scope includes any project-related PA, the organizational scope must include at least one focus project. If the organizational unit includes more than 3 projects, then the organizational scope must include sufficient focus projects and non-focus projects to generate at least 3 instances of each practice in each project-related PA in the model scope of the appraisal.

Projects, categories, or groups/functions that are specifically excluded from the appraisal must be identified in the appraisal input and in the ADS as well as the justification for their exclusion. This identification includes legacy projects not using current organizational processes and projects waived from using current organizational processes. As needed, the appraisal team may seek clarification or data from other projects or support functions within the organizational unit. These projects or support functions must also be identified in the ADS.

What this means is that you must have at least one focus project. A focus project is a project that has evidence for every practice and Process Area (PA) in the appraisal scope. In order for that to be the case, that means the project is either already completed or very close to completion. A non-focus project is a project that does NOT have evidence for every practice and PA in the appraisal scope. Meaning a non-focus project is either a project that has recently started or a project that began before the processes were deployed and only has evidence for a subset of the practices.

Keep in mind that some PAs are organization level PAs while others are project-related. For the Organization PAs (e.g., OPF, OPD, OT, etc.) you only have to provide one instance of evidence. So the issue of number of projects only applies to the project-related PAs (e.g., PP, PMC, IPM, REQM, RD, etc.). Therefore, according to the MDD, if the organization has more than three projects, you have provide three instances of evidence for each practice. You could choose to supply three focus projects, two focus projects and multiple non-focus projects, or one focus project and multiple non-focus projects.

Another aspect of evidence that you must comply with is that you have to provide Direct Evidence for every practice and PA in the scope of the appraisal. But you don’t not have to supply Indirect Evidence for every practice and PA. Your Lead Appraiser can explain to you the one row, one column and 50% rules. As a risk mitigation, most of my clients choose to provide 100% coverage for both Direct and Indirect Evidence.