SCAMPI Document Review

I have two questions or requests for clarification:

• During the document review in SCAMPI B, wherever we do not have a doc, ppt, or xls as an artifact, we provide screenshots from tools where the planning and tracking are done.  However, our Lead Appraiser (LA) is asking for access to tools for all the ATMs.  But it is not possible to give access to certain tools as they are client specific and only the team working on the project gets access (that too after signing an NDA).  How do we handle this situation?  Even if the tool is internal, access is very much restricted based on role in the project.  Can the LA really require access to tools for all ATMs, which is not allowed as per the policy of an organization?  Is there a guideline on what process to follow in case access to certain tools or application is restricted? 

• Is there a material, which shows a linkage of all the PAs (representing interaction of PAs) and gives a holistic view of CMMI PAs when applied to an organization at Maturity L5. This is more from a training perspective. 

Have you explained these restrictions to your Lead Appraiser?  The Lead Appraiser should be flexible regarding access to restricted tools.  As a Lead Appraiser, I would find the screen shots as acceptable evidence in your situation.  And if I wanted or needed to view additional evidence, I would request that you provide a demonstration of the tool by an authorized user and have this person be directed by the ATMs or Lead Appraiser to view specific information.  I have used screen shots as evidence on numerous appraisals in the past.  The SCAMPI method allows for screen shots and tool demos for just such reasons as yours.  If your Lead Appraiser is unwilling to abide by your restrictions on tool access, then I would strongly urge you to find a new Lead Appraiser and possibly report him or her to the SEI.

Look in Chapter 4 of the CMMI-DEV book and you will find a series of diagrams that show the linkages between the PAs at a very high level.

Cut Off Time for Updating Documents Prior to a SCAMPI

What is the normal time-frame an organization is allowed to continue making changes to their documentation prior to a SCAMPI?

There is no hard and fast rule for this practice. You need to work this time frame out with your Lead Appraiser to see what he or she is comfortable with. But, think about what you are asking for a minute. Are you talking about process changes? Changes to artifacts? Or both?

If you are talking about process changes, then you need to consider the purpose of the SCAMPI. One of the jobs of the appraisal team is to determine the amount of institutionalization. In order to determine the degree of institutionalization (GGs and GPs), changes to the processes and procedures need to be minimized so there is sufficient time for institutionalization and to collect and present the proper Direct and Indirect Evidence. To be on the safe side and mitigate this risk, organizations may decide to have no process changes for six months before the SCAMPI.

If you are talking about the artifacts, then you need to keep in mind the definition of Focus and Non-Focus Projects and work with your Lead Appraiser to determine your evidence needs. In my experience, my clients have taken the risk mitigation approach of “freezing” the evidence about a month before the Readiness Review to build the PIIDs and then only allow changes after that point if there are weaknesses in the PIIDs that need to be addressed before the SCAMPI.

Please keep in mind that I am not advocating freezing the processes 6 months before an appraisal, it just has been my experience that as a risk mitigation some clients have held off making changes until after their appraisal. This behavior is typical for a first time SCAMPI A in a risk averse organization who wants to do everything possible to have a successful SCAMPI A. After all the CMMI is a set of process improvement guidelines, so I as a Lead Appraiser would expect to see evidence of continuous process improvement. But the org has to take an intelligent approach when rolling out new changes. The workforce gets frustrated with chasing a moving target if the processes and assets are frequently changing, i.e. major updates.

Providing Evidence for a SCAMPI Appraisal

If I am an appraisal team member, is it a problem if I am also helping to prepare the evidence for the appraisal?

V1.2 of the SCAMPI Method Definition Document (MDD) does not specify who should prepare the Process Implementation Indicator Descriptions (PIIDs). In practice the PIIDs can be prepared by the project teams, the appraisal team members, or some combination. There are pros and cons for each approach. If done by the project teams, the PIID preparers may not fully understand how the PIIDs will be used by the appraisal team and therefore it could take many iterations to get the PIIDs in an acceptable shape for the SCAMPI. If prepared by the appraisal team, there is the risk that the team members may get too close to the information and lose some of their appraisal objectivity. The best approach is for the project teams and appraisal team members to collaboratively address the Readiness Review issues and prepare the PIIDs. And as a best practice, try to maintain some independence on the appraisal team by having the team members help prepare the PIIDs for PAs other than the PAs assigned to their mini-teams.

Work Environment Evidence Question

There are two Specific Practices in the CMMI that address the work environment:

• OPD SP 1.6 Establish and maintain work environment standards
• IPM SP 1.3 Establish and maintain the project’s work environment based on the organization’s work environment standards

There are other Specific Practices in the model that address specific environments:

• VER SP 1.2 Establish and maintain the environment needed to support verification
• VAL SP 1.2 Establish and maintain the environment needed to support validation
• PI SP 1.2 Establish and maintain the environment needed to support the integration of the product components

In addition, the model references a number of other project work environments: maintenance, operational, production, and engineering.

Since IPM SP 1.3 contains the phrase “establish and maintain”, the organization needs to provide for a SCAMPI A appraisal evidence of “formulate”, “document”, and “use” of the project’s work environment. What is appropriate evidence to provide for IPM SP 1.3?

The “document” evidence could be the document(s) containing the description of each of the project’s work environments and its relationship to the organization’s work environment standards. The “formulate” evidence might be the inspection/review report(s) of these document(s). And the “use” evidence could be an example of a work product produced using the documented work environment. For example, evidence of use of the development environment might be a product build report demonstrating use of the environment for developing the product.

Since the Verification, Validation, and Integration Environments are covered by other Process Areas, it would be appropriate to provide for IPM SP 1.3 evidence of use of other components of the documented work environment to demonstrate the work environment covers more than testing.

Interpretation of GP 3.2 and Associated Evidence

GP3.2 states "Collect work products, measures, measurement results, and improvement information derived from planning and performing the process to support the future use and improvement of the organization¢s processes and process assets."

Do we need to collect measures and have measurement results for all the Process Areas or would just conducting lessons learned at the end of a milestone/phase/project be enough to satisfy the practice? Lessons learned in our case are mostly qualitative comments only and very few quantitative. We have all the planning and tracking related details available in EPM and other associated templates.

Please keep in mind that GP 3.2 is a generic practice and it applies to EVERY Process Area (PA) from Maturity Level 3 and up. It also happens to be one of those compound practices that require multiple things. In this case four different items to collect and feedback into OPF, OPD, and IPM. The advice that I give my clients when preparing the PIIDs for an appraisal is that the process work products you provide for GP 2.6 should be the same work products you are submitting for GP 3.2; the process measures that you use to monitor and control the process in GP 2.8 and the results reviewed with higher management in GP 2.10 should be the measures and measurement results you are submitting in GP 3.2. Therefore, the only additional piece of information required by GP 3.2 is improvement information from planning and performing the process. Sometimes this takes the form of lessons learned, which is an exercise focused on gathering data on what worked, what didn’t work, and what should be changed for future use. There are other ways of collecting this information without conducting a formal lessons learned meeting. But the bottom line is that you are EXPECTED to collect all of these data items for every PA in scope of your implementation and appraisal.

Another point is that these four items are independent of each other and therefore would be collected at different times. Conducting lessons learned meetings at the end of a milestone, phase, or project is a good practice and they don’t have to be quantitative in nature, unless you are at Maturity Levels 4 or 5. What you are trying to do is surface candidate process improvement suggestions from the people who have just used a process.