Selection criteria for a CMMI Lead Appraiser

My organization has used the same lead appraiser for several appraisls and we are now considering using a different LA for our upcoming ML 3 appraisal. My thoughts are to identify several candidate appraisers and then evaluate them against a set of criteria. Here are some of the criteria that I've identified to date:

• Experience with organization similar to our (kinds of products, size, geographically distributed, etc.)
• Expected time for appraisal of similar scope to ensure that we have similar expectations. We want to check length of appraisal and expected number of hours per day.
• Availability of toolset to support appraisal activities (i.e., PIID collection, ratings, consensus tracking, etc.)
• Availability during dates planned for appraisals
• Costs

Are there any other evaluation criteria I should be considering?

I don’t believe that the criteria you have specified will be able to determine a best fit, some of your criteria might but not the rest.

1. Experience working with other organizations of similar size, etc. is a good choice.
2. Expected duration of the appraisal is not a good choice. If you provide sufficient information for the Lead Appraiser to scope the work, I would expect that all qualified Lead Appraisers would provide the same answer, though there may be some variations. In my experience Readiness Reviews are 5 days and SCAMPI A on-site periods rarely exceed 10 days. It might be better if you presented your expectations for the appraisal duration and see if you got agreement from the Lead Appraiser.
3. Availability of tool set would not be a good choice. Usually qualified Lead Appraisers have a tool set they prefer to use. A better criteria might be the willingness of the Lead Appraiser to use your specified tool set.
4. Availability during your planned appraisal dates is not a good choice. Anyone bidding would state that they were available. Besides, we all know that planning dates are simply planning dates and the actual dates are different.
5. Cost is a good choice and probably the only one where you will be able to discriminate between bids. But I encourage you to understand that a SCAMPI appraisal is NOT a commodity. When you purchase any service, including appraisal services, you must understand the value you are intending to obtain. Requests for Propsal should be structured to obtain the information needed to compare value, not just fees or costs.
   

I recommend that since you do have experience with the CMMI, appraisals, and Maturity Level 2 that you put together a plan for your ML 3 appraisal based on the historical data from your ML 2 efforts. Then ask for bids from several Lead Appraisers. Also ask for recommended optional services. Some Lead Appraisers will bid the job by number of days and others will bid the job by appraisal activity. That should provide you with some very good information to make your decision. And here is a radical thought, since you are talking about ML 3, why don’t you use DAR to help you make your decision. :-)

Finally, I recommend that you interview each candidate Lead Appraiser, either over the phone or in person to see how comfortable you feel with him or her and how each responds to your appraisal needs.

Process Writing vs. Process Definition

What is the difference between Process Writing and Process Definition?

Process Definition is the act of determining or defining the process. That means interviewing people on how they perform their process and then drawing a process flow diagram describing the interactions between all of the process activities. It is basically the type of steps you follow when you perform the second and third exercises in the SEI's Intro to CMMI class. You examine each process activity and answer a series of questions about entry/exit criteria, input/outputs, verification points, measures, sub-activities, etc. Assembling all of this material, including the predecessor/successor relationships between the process activities is Process Definition. Then the final step of putting this information into a usable form by the process practitioners is Process Writing. In other words, Process Writing it simply documenting the process once you have determined all of the necessary pieces and parts.

List of Appraised Companies

How can I see a list of CMMI appraised companies across the globe for all levels and years?

With the release of CMMI v1.2 in August 2006, the SEI instituted a new policy regarding the expiration of appraisal results. All appraisal results expire three years from the anniversary date of the appraisal. The list of published appraisal results is available from the SEI at http://sas.sei.cmu.edu/pars/pars.aspx. Once the appraisal results reach their third anniversary date, they are automatically removed from this list. For example you could check the list today and find the results for an appraisal conducted December 2, 2005. Wait a day or so and you will no longer find the record. You can sort this list based on year, organization name, Maturity Level, etc.

Keep in mind that once the appraisal results have expired, they are no longer valid and the organization should have been re-appraised in order to continue to claim a particular Capability or Maturity Level.

A Question About DAR Calculations

I have a question regarding DAR calculations. When assigning wieghts to the selected criteria, do I have to relate them to each other? Meaning, do I have to refer to a number that represents my 100% need and relate all criteria to it? For example, I have three Criteria A, B, and C, then I'll assign 3 (30%) to A, 5 (50%) to B, and 2 (20%) to C, and in case one of them was decreased the others will increase. Currently, I define a range for criteria weight (for example from 1 to 5), and I select a number subjectively based on the weight of the criteria.

I think that you may have mixed things up a bit here. The evaluation criteria should be independent and not assigned percentages, the percentages apply to the weighting factors. For each criterion you assign scores between 1 (low) and 5 (high). The weights are derived from team consensus and total to 100% or 1. 20%, 30%, and 50% are typical weights. It is best not to have more than three evaluation criteria because if you have more, then there is the potential for the resulting scores to be too close to each other and it will be very difficult to determine which is the best solution.

With three criteria it is easy to assign 50%, 30%, and 20%. With more criteria you will be forced to assign weights that are closer together. So if you go strictly by the numerical calculations, then there is the potential for lots of ties or very close results thereby making it a challenge to use this technique for selecting an answer. For example, if you had five criteria, then one possible approach, though wrong one, would be to assign 20% to each criteria. Other options would be to assign weights of 30%, 25%, 20%, 15%, and 10% or 50%, 20%, 15%, 10%, and 5%. But you might find it difficult to reach consensus among the team for these values and how to apply them to the criteria. With these assignments, you might calculate the same score for one or more alternative solutions. So with more than three criteria the resulting unique weights could be very close together and not very useful for using this method to calculate scores and use the results to make a decision.

My premise is to limit the number of evaluation criteria to no more than three, not the number of alternative solutions. You can have as many alternative solutions as you want.

And another point is to try to select evaluation criteria that are independent, otherwise you will face challenges as well. For example if you are buying an automobile the independent evaluation criteria might be: 1. Features and price 2. Distance from your home to the dealer and 3. Service reputation.

However, you always have to use the sanity test to determine if the results of the calculation make any sense. You could have chosen the wrong criteria, assigned the wrong weights, and/or assigned the wrong values to each criteria. And yes, management always has the prerogative of making a decision without using the results of the process.

Test Case Coverage & Review Effectiveness

Can anyone help me for the definition of Test Case Coverage and Review Effectiveness? How is Review Effectiveness calculated for work products like SRS, SDD &Test Spec?

The term Test Case Coverage means to me a measure of how many of the product requirements are being tested by the defined test cases. It is the testers’ job to define their test cases based on the requirements and the goal should be 100% coverage or close to that. Please bear in mind that 100% coverage is not the same as exhaustive testing. Exhaustive testing means testing every path through the software using all possible values.

The Requirements Traceability Matrix is what you would use to determine the Test Case Coverage. You should be able to map every test case to the requirements and vice versa. Once you have done that you will be able to determine if one or more of the requirements are not being covered by the test cases. So a simple ratio of the number of reqmts not being tested to the total number of requirements would yield a Test Case Coverage measure.

Review Effectiveness is a bit harder to define. When you hold a review, say for the SRS, then you would have a total number of defects discovered and you also know the number of pages in the document. So, to first order, you could divide the total number of defects by the number of pages and derive a defect density measure for the document. But that number by itself doesn’t tell you a whole lot. You need to have a good understanding of the expected number of defects per page coming out of a review. That number would come from your historical data. But this number still doesn’t yield an effectiveness measure. You could be finding lots of minor defects in a review and the major ones are slipping through. So you have to look for defects that were missed by the review in downstream activities such as testing and other reviews. You could also be grouping defects by major and minor categories etc.

You should be asking several questions that would lead you to an effectiveness measure:
1. In what review did you find the defect?
2. In what activity was the defect inserted?
3. What review should have caught the defect but didn’t?

A Question About Measures

Can someone please help me with the following questions:

1. Are there examples of typical or common measurements than an organization can use for every process area? (different examples appear in Generic Practice (GP) 2.8)
2. Where I can look up them?
3. Are there some measurement standards that show or define typical or common measurements? Which?
4. Do they apply to a specific Process Area (PA)? If so, which PA?

Granted there are some common measures such as effort and time spent on an activity. But the underlying principle of the CMMI and more importantly Measurement and Analysis (MA) is to define your goals and objectives and use them to determine your measurement needs. Then you will be able to easily specify those measures that have the most value to your organization and projects. You have to understand why you are collecting a specific piece of data and how you are going to use the information to make a decision, otherwise you will have difficulties and over time people will forget why you are collecting and using the data. They will not see the benefits of these measures. As I have posted to this blog before, start with the MA Process Area and use the practices to define your measures, how you will collect , analyze, report, and use them. MA is based on Practical Software Measurement (PSM). Read the PSM book and visit the PSM web site http://www.psmsc.com/ . The book is very easy to read and understand. It contains several case studies and example measures that will help you define your organization’s measures.

Monitoring Stakeholder Involvement

Project Monitoring and Control (PMC) Specific Practice (SP) 1.5 states "Monitor stakeholder involvement against the project plan." Our organization has chosen to implement this practice by creating a Stakeholder Management Plan (SMP) for each project. The SMP lists each stakeholder, their involvement, the monitoring method, and other pertinent information. Stakeholder involvement is checked during each activity and we do not see any value to having an overt and separate check of the SMP activities by the Project Manager (PM).

1. PMC SP 1.5 specifies that a periodic review of the status of stakeholder involvement. How often should this review occur?
2. What are examples of Direct and Indirect Evidence for PMC SP 1.5 that can be used to help us develop the PMC PIID?

The purpose of PMC is to provide an understanding of the project’s progress so that appropriate corrective actions can be taken when the project’s performance deviates significantly from the plan. Since the Stakeholder Management Plan is one of the components of the project plan, it must be monitored, controlled, and managed on a regular basis in order to take appropriate corrective action. As PMC is normally a PM’s responsibility, it is expected that the PM is regularly monitoring the SMP and its use, but this responsibility can be delegated.

As the stakeholders specified in the SMP are involved in different activities and at different times throughout the project’s life cycle, specifying a fixed monitoring frequency for the entire plan may not provide a lot of value to the PM and the project. It may make more sense to monitor stakeholder involvement against the SMP on an individual activity basis. One method for implementing this type of monitoring would be to specify in the SMP the monitoring frequency for each stakeholder activity, the person responsible for monitoring each activity and stakeholder involvement, and the mechanism for documenting and communicating the monitoring results for each activity. Providing this information in the SMP would then indicate what would be appropriate Direct and Indirect Evidence for a SCAMPI appraisal.

CMMI Use in a Non-Software Organization

The CMMI model is used for improving the maturity of company processes, mainly in software organizations. But the model says that CMMI can be used also in other engineering contexts, such as mechanical, electrical etc. Does anyone know about companies that have been evaluated in contexts other than software?

As you have stated, the CMMI for Development (CMMI-DEV) is useful for systems engineering, hardware engineering, and software engineering. Over the past 20 to 30 years, it has been primarily software organizations that have been implementing maturity models. But these models do apply in other contexts. In point of fact, I have been working with a client for over a year that just performs systems engineering and we are using the CMMI-DEV. We are currently planning their ML 3 SCAMPI A for early 2009. There is no software development involved in the scope of their appraisal. It has been an interesting experience to think outside of the software realm when interpreting the SPs and GPs, but it does work very nicely.

Confusion regarding Project Montoring and Control

I have some confusion regarding Project Monitoring and Control (PMC) Specific Practice (SP) 1.2 Monitor Commitments and SP 1.5 Monitor Stakeholder Involvement. In my mind the intent of both practices are logically the same and the type of evidence produces should also be the same (eg. schedule, minutes of meeting/action file, and status reports). I think these are two duplicate practice. Kindly let me know your view regarding this issue.

This is an excellent question. There could be some overlap between Project Monitoring and Control (PMC) Specific Practice (SP) 1.2 “Monitor commitments against those identified in the project plan” and SP 1.5 “Monitor stakeholder involvement against the project plan.” The primary difference concerns commitment vs. stakeholder involvement. Commitments include such things as delivery dates, specific deliverables, requirements, etc. Commitments are those things that everyone agrees to and they must be documented to ensure a consistent mutual understanding. Documenting the commitments indentifies the responsibilities of those involved with the project (stakeholders).

Monitoring the commitments is not the same thing as monitoring stakeholder involvement. For example group A produces an interface specification needed by group B so they can perform their assigned activities. The commitment that group A has made to group B is that group A will deliver the interface specification on Friday Oct 31, 2008. Monitoring the commitment in this case is periodically checking to see if the interface specification will be delivered early, on time, or late. If the specification will be early or late, then some type of corrective action may be necessary. In contrast, the intent of SP 1.5 is to use a stakeholder involvement plan or matrix to ensure that all of the relevant stakeholders who are supposed to be involved in producing, reviewing, and accepting the interface specification are performing their roles as identified and planned. If not, then some type of corrective action may be necessary.

Granted you may have the same evidence for both practices but the evidence demonstrates different purposes and use.

Monitoring Data Management

Project Monitoring and Control (PMC) Specific Practice (SP) 1.4 states "Monitor the management of project data against the project plan." Our organization has chosen to implement the plan for managing project data by creating a Data Management Plan (DMP) for each project. The DMP lists each item and the method, its storage location, and the monitoring method along with other pertinent information. There are embedded checks and balances for placing items in the proper repository and they are distributed to the lowest level.

1. PMC SP 1.4 specifies that a periodic review of the data management activities against their descriptions in the project plan. How often should this review occur?
2. What are examples of Direct and Indirect Evidence for PMC SP 1.4 that can be used for developing the PMC PIID?

The purpose of PMC is to provide an understanding of the project’s progress so that appropriate corrective actions can be taken when the project’s performance deviates significantly from the plan. Since the DMP is one of the components of the project plan, it must be monitored, controlled, and managed on a regular basis in order to take appropriate corrective action. As PMC is normally the Project Manager's (PM's) responsibility, it is expected that the PM is regularly monitoring the DMP and its use, but this responsibility can be delegated.

As the items in the DMP are created and managed at different times throughout the project’s life cycle, specifying a fixed monitoring frequency for the entire plan may not provide a lot of value to the PM and the project. It may make more sense to monitor each item against the DMP on an individual basis. One method for implementing this type of monitoring would be to specify in the DMP the monitoring frequency for each item, the person responsible for monitoring each item, and the mechanism for documenting and communicating the monitoring results for each item. Providing this information in the DMP would then answer question 2 as well.

Stakeholder Invovlement Question

Our organization has project stakeholders whose only project involvement is as a decision maker in an end-of-phase meeting. What is expected of the stakeholders (e.g., training and project involvement) if they are outside of our process, as far as the appraisal interviews are concerned?

In the case described, the stakeholder is NOT outside the process.

The “plan for stakeholder involvement” defines the expected role of the stakeholder in the project, or process. This, in turn, defines the scope of training required. In the case described, the stakeholder would minimally need training in the conduct of end-of-phase meetings. However, if decision making required knowledge of work products produced by the project team, orientation to those products might also be required. Training in Decision Analysis and Resolution (DAR) may also be required, depending upon whether or not the decisions meet the criteria for initiating the DAR process.

Note that training may take many forms, including orientation or briefings on required topics. The expectation is that the stakeholder knows enough to credibly perform the responsibilities described in the project plan or process description.

In the case described, the stakeholder might be selected for an appraisal interview.

Another Work Environment Question

Both Organizational Process Definition (OPD) and Integrated Project Management (IPM) reference the work environment. What is the difference between OPD SP 1.6 and IPM SP 1.3?

OPD Specific Practice (SP) 1.6 is "Establish and maintain work environment standards" and IPM SP 1.3 is "Establish and maintain the project's work environment based on the organization's work environment standards." OPD is an organizational level Process Area (PA) and IPM is a project level PA.

The expectation for OPD SP 1.6 is that the organization defines work environment standards that allow both the organization and projects to benefit from a common set of tools, training, and maintenance. The intent of OPD SP 1.6 is NOT to define the standard work environment, but to define standards for the work environment. Standards typically include work environment operations, safety, and security procedures, standard hardware and software configurations, standard software application loads, and procedures for requesting waivers or tailoring. In addition, projects may have additional requirements for their work environments.

In contrast, the intent of IPM SP 1.3 is to use the work environment standards defined by OPD SP 1.6 to define the appropriate work environment for the project. The project's work environment might include the development environment, the testing environment, the integration environment, the verification environment, and the validation environment. These environments could be one environment or separate environments and/or facilities.

Work Environment Evidence Question

There are two Specific Practices in the CMMI that address the work environment:

• OPD SP 1.6 Establish and maintain work environment standards
• IPM SP 1.3 Establish and maintain the project’s work environment based on the organization’s work environment standards

There are other Specific Practices in the model that address specific environments:

• VER SP 1.2 Establish and maintain the environment needed to support verification
• VAL SP 1.2 Establish and maintain the environment needed to support validation
• PI SP 1.2 Establish and maintain the environment needed to support the integration of the product components

In addition, the model references a number of other project work environments: maintenance, operational, production, and engineering.

Since IPM SP 1.3 contains the phrase “establish and maintain”, the organization needs to provide for a SCAMPI A appraisal evidence of “formulate”, “document”, and “use” of the project’s work environment. What is appropriate evidence to provide for IPM SP 1.3?

The “document” evidence could be the document(s) containing the description of each of the project’s work environments and its relationship to the organization’s work environment standards. The “formulate” evidence might be the inspection/review report(s) of these document(s). And the “use” evidence could be an example of a work product produced using the documented work environment. For example, evidence of use of the development environment might be a product build report demonstrating use of the environment for developing the product.

Since the Verification, Validation, and Integration Environments are covered by other Process Areas, it would be appropriate to provide for IPM SP 1.3 evidence of use of other components of the documented work environment to demonstrate the work environment covers more than testing.

CM SP 3.2 Evidence Guidance

The release of CMMI v1.2 included a change to the examples for CM SP 3.2, Perform configuration audits to maintain integrity of the configuration baselines.
Examples of audit types include the following:

• Functional Configuration Audits (FCA) – Audits conducted to verify that the as-tested functional characteristics of a configuration item have achieved the requirements specified in its functional baseline documentation and that the operational and support documentation is complete and satisfactory.
• Physical Configuration Audit (PCA) – Audits conducted to verify that the as-built configuration item conforms to the technical documentation that defines it.
• Configuration management audits – Audits conducted to confirm that configuration management records and configuration items are complete, consistent, and accurate.

Are projects now required to provide evidence for each of these examples?

As the Configuration Management (CM) Specific Practice (SP) states, these are examples of Configuration Audits and are part of the informative material of the model. Therefore these examples are neither required nor expected. As long as the organization and projects are performing some kind of configuration audit(s) that assesses the integrity of the baseline(s), that meets the intent of the practice. For a SCAMPI Appraisal projects are not required to provide evidence for each type listed in the examples.

What is the cost of a CMMI v1.2 L3 certification?

Our company is interested to go for CMMI v1.2 ML3 certification. Can you answer the following questions:

1. What are the costs involved (Internal & External)?
2. Are there any SEI cerfication bodies in India?

First of all let me make one clarification to your query. There is no such thing as a CMMI certification. What an organization receives are the results of a SCAMPI A appraisal that indicate the Maturity Level of the organization on the day the appraisal concludes. The organization is not certified. And there is no such thing as an SEI certification body anywhere in the world. What do exist are authorized SEI partners that are allowed to provide CMMI consulting, training, and appraisal services. Visit this web site http://partner-directory.sei.cmu.edu/ and you will be able to find the SEI-authorized partners in India.

The answers to your questions are highly variable depending on the size and scope of your organization and your geographical location. The best place to obtain realistic estimates is to ask several local SEI-authorized CMMI consulting and appraisal providers for their cost proposals, then you will have a handle on the external costs. Internal costs really cannot be determined until you figure out how much work you have to do in order to implement the CMMI and prepare for an appraisal. Suffice it to say, your internal costs will most likely be greater than your external costs.

Can organizations still claim CMM?

If I remember correctly SEI retired the CMM and that organizations can no longer advertize their CMM maturity level. If an organization continues to claim on their web site that they are CMM level 4 - even today - and in addition it is claiming that it will be assessed shortly for CMMi Level 5 for more than a year and half - should we ignore it or bring it to the notice of SEI?If you suggest that we should bring such instances to SEI notice please give me the email id.

The short answer is that any claim of a CMM Maturity Level is no longer valid. As for the ML 5 notice, that is fine as long as the web site states that the organization is working towards ML 5, but I don't see how that is a marketing point. It is similar to a graduate student stating that he or she is in the Doctoral Program and have everything completed except for his or her thesis.

The SEI is not in the business of investigating false claims. Buyer Beware: The only official location to verify an organization's Capability or Maturity Level is the SEI's Published Appraisal Results site http://sas.sei.cmu.edu/pars/pars.aspx. Since the appraisal results are only good for three years, the results are automatically removed on the anniversary date. Just check this site daily for a week or two and you will be able to verify this for yourself. The best approach is to contact the offending organization to inform them of their mistaken claims as they may be unaware of the changes to the SEI's appraisal program. And if you still feel that the SEI should be notified, I would suggest that you contact Jill Diorio who is the SEI Ethics person at jdiorio@sei.cmu.edu.

Query on Process Change Management

Processes are meant for Doing things Right, First time, Every time. This is one of the core concepts I have seen people referring to during process formulation exercises. I presume the whole idea behind this is to make an organisation process driven instead of people driven.In my limited experience, (esp. in small to medium size organisations) I have seen new unit/department heads within a short span of time after their recruitment, devise a series of Continuous Improvement Requests (CIRs) and posted it to the EPG, who on most occasions weigh the CIRs against department and compliance goals, and if they don't see any adverse effect due to these changes, they approve it.One of the premise for approving the changes is to promote positive initiatives in the organisation.

Now, my questions are:

1. Will adopting the practices prescribed by a new department head, which is based on his experiences, affect his subordinates' work( who are the actual practitioners) in an adverse manner?
2. Will organic growth of existing practices (due to collective experience of the practitioners) be ruined due to implementing new practices?
3. Will allowing practice changes from new department heads make an organisation fall behind from process to people driven? (even though this may last until the organisation get accustomed to the new practices) And is this permissible, if we consider the organisation's overall development?

First off, the impacts depend upon the organization’s Maturity Level. A Maturity Level 2 organization doesn’t necessarily have standard procedures for all projects to follow, though I have seen many ML 2 organizations take this approach. Therefore at Maturity Level 2 you can have multiple ways of doing the same thing, from project to project and from manager to manager.

When the organization matures to Maturity Level 3, the premise is that the organization has examined the multiple ways of performing a given practice and determined the Best Practice for the organization and then documents these Best Practices as the set of standard processes for the organization. This examination, coordination, and distribution of the standard processes is typically the responsibility of the Process Group. The Process Group manages the processes and is responsible for coordinating all process changes.

Now having said this, a new department head can make any process changes he or she wants to make. At ML 2 these changes could provide a Best Practice for consideration or additional information on things not to do. However, at ML 3 the organization should have established OPF and OPD processes for making process changes. The new department head would have to follow these change processes to propose his new processes. The Process Group would evaluate his proposals and pilot them as appropriate so the changes could be evaluated in a controlled manner. The outcome of the pilot(s) would determine whether or not the changes are made.

By following these steps, there shouldn’t be any of the problems you allude to in your note. However, if your organization does not have processes in place for making changes to the organization’s processes or the new department head mandates process changes without following the process, then you do have some serious issues to address.

1. Making uncontrolled process changes will impact the practitioners, most likely adversely. People most likely will be frustrated because of the changes.
   
2. Replacing existing procedures with new ones in an uncontrolled manner will adversely disrupt any process improvements and process evolution you have already made.
   
3. Making uncontrolled process changes can cause the organization to regress in Maturity Levels, probably drop from whatever ML you are currently at to ML 1.

CMMi 1.2 Level 3 Query

We have planned to use CMMI 1.2 model for our Organization for process improvement and we are targetting to reach maturity Level 3. The constraint here is we have a very small team in development. E.g. there are only 3 members for database management, 1- 2 team members for front end development (the same is true for middleware development), and a few for production support projects. I am not sure how to manage peer reviews and many other aspects. In all there are around 20 members in the development team. Please let me know the challenges we may face during our journey to maturity Level 3, based on your experience.

Also I am driving the CMMi initiative. Is there any mandate that I should be an SEI-Certified Assessor? I have the required experience in Quality Management System for driving the CMMi initiative, but unfortunately I have not taken any formal Certification on CMMI. Please let me know your thoughts on this point as well.

There are many challenges with process improvement and implementing the CMMI regardless of your size. However, for small organizations, the challenges can be even greater. The biggest challenge is probably budget. Small organizations usually do not have a lot of extra funds for activities not directly associated with producing a product or service. Therefore, you are faced with adding the responsibilities of mapping, defining, and documenting your processes and procedures to your normal 40+ hour/week job. Industry averages over the past two decades show that Process Improvement is roughly 3 – 5% of the organization size. Configuration Management takes another 3 – 5%, as well as PPQA. Worst case the total could be 15% of the organization. So for a development team of about 20 people, you would need 1 person full time on Process Improvement, 1 person full time for CM, and 1 person full time for PPQA. Senior Management is usually reluctant to set aside this much budget when you begin your journey. There is no visible business case for the additional people and what happens is that people have to wear multiple hats to cover these new positions until the burden gets so big that either you stop these activities or management agrees to the additional funding.

Beyond this much it is difficult to provide more information without a better understanding of your organization, management commitment, line of business, etc. There are many ways available to you to scale the CMMI to your organization. You just have to be very careful not to implement something because you think the CMMI requires it, but there is no business value to you. That is the situation that gives process improvement a bad name. Everything that you do with the CMMI must be value-added. About the only way that I know of to avoid mistakes and/or potential obstacles is to work with an experienced CMMI consultant and Lead Appraiser. Over the years we have seen what works and what doesn’t work and there is no need for you to reinvent the “process improvement” wheel one more time.

A third point is that your description implies that you are going directly to Maturity Level 3 without first establishing the foundation of Maturity Level 2. That approach can be a costly one and could jeopardize your program, unless you have a history of process improvement in the organization. The SEI and all Lead Appraisers that I know always recommend that you implement one Maturity Level at a time. The CMMI Staged Representation is a foundational model. You must first establish a firm foundation for process improvement, and that is what you achieve by attaining Maturity Level 2. Then Maturity Level 3 builds upon the foundation of ML 2. Then ML 4 builds upon ML 3. And ML 5 builds upon ML 4. Please keep in mind that I am not saying that you have to be appraised at each level with a SCAMPI A Appraisal. But you need some indication that you have implemented a Maturity Level. The big mistake many organizations make is skipping over ML 2 and going directly to ML 3. There are some fundamental differences in how Project Managers behave at ML 2 vs. ML 3. By skipping ML 2, you run the risk of having ML 3 processes documented but practiced as a ML 2 organization. Therefore, when you are appraised, the results have a high probability of being ML 2, which would NOT be a happy day.

There are no SEI requirements that the person in the organization responsible for implementing the CMMI be an SEI-authorized Lead Appraiser. There really is no benefit to you or your organization for you to become a Lead Appraiser, unless you have lots of internal opportunities to lead appraisals, or your company is in the business of providing CMMI services to other clients. Becoming a Lead Appraiser is an expensive proposition. You have to take at least three classes, which can be a total of $15,000. Before you can be accepted in the classes, you have to be an appraisal team member on at least two appraisals. You have to pass several exams and be observed leading an appraisal. Then you have to lead at least two appraisals over a three year period. In addition, a Lead Appraiser cannot lead an appraisal of his or her own organization.

My best advice to you is to hire a CMMI expert to help you on your journey.

Does anyone have an example of how everything flows together?

I'm looking for a work flow diagram or something of that nature that shows how all of the Level 2 and Level 3 Process areas and associated work products flow together. I'm talking about some sort of diagram or chart that shows a regular waterfall development lifecycle found in a software environment and where each Process area, and it's work products fit into that scenario.

Since the CMMI is a set of guidelines for process improvement, there is no single way of tying everything together. It is highly dependent on how you have implemented the model in your organization. Have you taken the SEI’s Intro to CMMI 3-day class? If so, then you have seen diagrams that sort of address your request. These diagrams also appear in Chapter 4 of the CMMI book. There is a whole series of these diagrams that illustrate how the Process Areas (PAs) are interconnected and the types of information that flow between the PAs. These figures only show possible interactions between the PAs and are not the only way to do things.

Interpreting OPF SP 1.2

While reading and then interpreting SP 1.2 of OPF "Appraise the Organization's Processes" it seems that it is a mini SCAMPI (correct me if I am wrong). Now in this context I have few questions and would like answered.

Organizational Process Focus (OPF) SP 1.2 states “Appraise the organization’s processes periodically and as needed to maintain an understanding of their strengths and weaknesses.”

1. These internal appraisals to satisfy this practice will be performed by the PEG (Process Engineering Group), how much rigor is required? Can we employ the SCAMPI Class C method to satisfy this practice?
SCAMPI A, B, and C appraisals all satisfy this practice as well as any other type of appraisals or assessments that provide an understanding of the organization’s process strengths and weaknesses. There is no level of rigor implied by this practice.

2. Should we check the evidences and/or satisfaction at processes' implementation at project level or its better to keep the scope of these appraisals at OU level only?
OPF is an organizational level process. And the practice states to appraise the organization’s processes. Some of them are performed at the project level and others at the organizational level. But the focus should be at the organizational level.

3. Should we formally present the findings as normally done in SCAMPI Class A appraisals by the LA or that much rigor is not required (as it required lot of stakeholders presence like CEO if he is the sponsor)?
The rigor of the Findings Presentation is up to you. However, you should determine the level of rigor when you are in the planning phase for these evaluations. As a Lead Appraiser, I create an appraisal plan for any appraisal I conduct and I specify in the plan how the practices, projects, organization, etc. will be scored and how the results will be presented.

4. If we are appraising the organizational processes, then should we appraise all the ML 2 and ML 3 process area or we can make selection?
Since this practice covers all process appraisals conducted on the organization’s processes, depending on the appraisal, you have a lot of flexibility on the scope and conduct. It all needs to be specified in the appraisal plan. Of course a SCAMPI A appraisal using the staged representation will dictate which PAs to include depending on the Maturity Level. Other than the SCAMPI A, you have the freedom to pick and choose what you want to appraise. But keep in mind, you do need to have an overall strategy and plan for these appraisals. As a Lead Appraiser appraising this practice, I expect to see a periodic plan for your appraisals, not just the SCAMPI A appraisals I conducted.

5. If we make a checklist for all process areas, then is it a good idea that we include the Subpractices as a questions (may be not all subpractices) in checklist for all ML 2 and ML 3 PAs?
Again, you can use whatever checklists you want for your internal appraisals. In fact, it might work to your advantage to be very rigorous in an internal appraisal. But remember, when it comes time for the SCAMPI A appraisal, you will not be evaluated against the sub-practices.

Who Owns the Process?

A company/organization entirely outsources its software development to another company. That's their business model. The developers working for the company whose name appears on their paycheck work for the client in every way except for who signs the paycheck (so to speak).

The entire development effort (client and developers) implements CMMI.

IF BOTH the leaders of the client company AND the developer company take part in the process efforts, e.g., GP2.1, GP2.7, GP2.8, GP2.10, GP3.2, et al. can the SCAMPI be done such that both OUs are delineated? Can BOTH organizations lay claim to a rating?

I think the key element comes down to the definition of the Organizational Unit. And you imply in your scenario that there are two OUs. One OU has outsourced the engineering work and the other OU is performing the engineering work. Given the OU definitions, I think that this scenario indicates the need for two SCAMPI appraisals. One for the client using the CMMI-ACQ since it has outsourced the development work and one for the development organization using the CMMI-DEV.

Hurricane Ike, Risks, and the CMMI

In a previous blog entry, http://ppqc.blogspot.com/2008/08/help-understanding-process-performance.html, I have used hurricanes to illustrate the differences between Process Performance Baselines and Process Performance Models. But there is a more practical problem with hurricanes that impact organizations, Lead Appraisers, and appraisals at any Maturity Level. This problem manifests itself in the areas of Risk Identification, Tracking, and Mitigation. After the devastation of hurricanes over the past several years, most notably Katrina and now Ike, I recommend to all of my clients along the Gulf Coast and the Atlantic Coast that they include hurricanes or severe weather as a credible risk for the process improvement efforts and appraisal planning, especially during hurricane season June 1 to November 30. Recognizing that hurricanes can cause severe damage and at at minimum, a disruption to work, risk mitigation plans need to be identified. Not only are businesses closed for days, electric power and other essential services are out, homes can be damaged, and people evacuate. Today it is 10 days since Ike came ashore and chewed a swathe through Galveston and Houston. Only about 60% of the power has been restored. Many people have lost everything and most of us are preoccupied with clean up and insurance claims. So even though businesses are back open today, many employees will not be able to return to work. And those that do return to work won't be productive for some time. They will be relating their Hurricane Ike stories.

So, when identifying a hurricane, severer weather, or other act of God or war as a risk, the mitigation plans need to address the worst case scenario and the potential loss of weeks of work, if not data. And from the Lead Appraiser's perspective, a disaster can wreak havoc with the calendar since appraisals tend to stack up at the end of the calendar year. If a disaster strikes in September, it may be a problem to reschedule an appraisal in the same calendar year.

CMMI Certifications & Career Development

I'm a graduate student and starting my degree in 2009. I have over 2 years experience (3 mini assessments) in Process Improvement, CMMI ML 3 and ML 5 evidence collection, and verification. I will be taking the Intro to CMMI class in December 2008.

As I am interested in researching the CMMI and Quality Assurance for my thesis and to shape up my career to be a CMMI consultant position; I contacted SEI. The SEI suggested that I seek the advice and guidance from a few registered Lead Appraisers in different parts of the world on how I should plan my future studies.

My questions are:

1. In the professional world job market is there a value for such research?
2. What other professional qualifications should I gather to be a CMMI consultant?
3. From your experience do you think its worth it (career development wise) to conduct appraisals externally rather than waiting for your organization to provide them free for you?
4. Any other advice you want to provide me in my career planning?

Your guidance in these questions are highly appreciated.

Here is my advice to you to help further your interests and your career.

1. There is always room in the professional arena for research. Research results are regularly presented at professional symposia and conferences. I am sure that the SEI would encourage you to submit an abstract for consideration at an upcoming SEPG Conference. If your research is credible, and you gain visibility in the community, that can only strengthen your credentials as a consultant. It also demonstrates that you are staying current in the field of process improvement.
2. To become a CMMI consultant, you would need practical experience, not necessarily additional professional qualifications. Take a look at the criteria for being on an appraisal team. That criteria also makes for a credible CMMI consultant. So I would recommend that you gain some experience in Project Management, as well as engineering experience, at least 3 to 5 years worth. If you were to start consulting immediately upon graduation, you may be looked upon as providing an academic approach to CMMI implementation.
3. Since implementing the CMMI can take several years and the expenses for appraising an organization are fairly high, there will not be many internal opportunities for appraisals. Especially if you want to become a Lead Appraiser. You may be able to satisfy the minimum requirements to become a Lead Appraiser with internal appraisals, but most likely you won’t be able to conduct enough internal appraisals to maintain your credentials. Therefore, I recommend that you conduct external appraisals. If you were to work for a consulting company specializing in the CMMI, then the company would most likely pay your expenses. Otherwise, you will be faced with financing them yourself.

Difference between measurement objectives and process performance objectives

CMMI uses following terms while talking about measurements:

1. Identified information needs and objectives at organization level (MA)
2. Measurement objectives derived from such information needs and objectives (MA/SP 1.1)
3. Quality and Process Performance Objectives (OPP/SG1)

I want to investigate finer differences between the later two. Are they more or less the same? If not, what are the finer differences?

Your first two questions deal with Measurement and Analysis (MA) which is a Maturity Level 2 Process Area (PA). Your third question is about OPP which is a Maturity Level 4 PA.
MA is a support PA that applies to ALL PAs in the CMMI. Its focus is defining measures and indicators that help answer or address a specific information need. An information need can be articulated using the following format:

<someone> needs to know <what information> in order to make <what decision>.

For example, an issue facing the Project Manager may be that Project Plans are inaccurate causing project teams to routinely miss milestones and deliveries.
Therefore the information need would be: The Project Manager needs to evaluate the schedule, progress, and dependencies of key development activities and events in order to determine necessary corrective actions.
Then the measurement concept or strategy for addressing this information need would be: At the end of each week the Project Manager collects estimates to complete the work for each current task and enters the data into the schedule. The Project Manager and team review progress to determine the necessary corrective actions and modify the Project Plan as needed each Monday morning.
Now it becomes very easy to determine the base and derived measures that support the measurement concept.

So these concepts (information need, measurement concept, and measures) are interrelated concepts that help define what data to collect and analyze to help answer a question.

The Quality and Process Performance Objectives (QPPOs) are entirely different from the measurement concept. The QPPOs are objectives set by the organization based on the organization’s business objectives, past performance of projects, and are constrained by the inherent variability or natural bounds of the process. Now, in order to determine the measures that the organization will use to support the QPPOs, the organization should follow the MA process of articulating the information need, defining the measurement concept, and determining the measures.

Trying to Understand the Differences Between RD SP 2.1 and SP 2.2

I have difficulties understanding the difference between Product Requirements and Product Component Requirements. For me Product Requirements are when I have documented the customer needs and I have performed the next step, interpreting and writing the technical requirements in tree categories, functional, non functional, interface. But what happens with Product Component Requirements? In which category do they apply? When in the project lifecycle do I have to describe this kind of requirement? Is it in design while the developers are writing code and they discover that something is missing? And then in SP2.2 "Allocate product component requirements", I cannot figure out what kind of allocation the CMMI refers to.


The intent of Requirements Development (RD) Specific Goal 2 (SG 2) is to refine and further elaborate on the customer requirements. If you think of the customer requirements being at 50,000 feet, SG 2 is adding the information necessary to bring you down to 10,000 feet. Still pretty high level, but with some detail now.

As I always recommend to people who have model interpretation questions, the first place you should consult is the CMMI Glossary. If you look in the Glossary you will find definitions for product component and product component requirements.

Product Component – In the CMMI Product Suite, a work product that is a lower level component of the product. Product components are integrated to produce the product. There may be multiple levels of product components.

Product Component Requirements – A complete specification of a product component, including fit, form, function, performance, and any other requirement.

Here is an illustrative example to help you better understand the concept of a product component. Let’s consider a mechanical alarm clock. The clock consists of several parts or components: clock face and hands, gears and levers, alarm bell, setting levers/buttons, and a clock housing. There may be others, but these will do for now. Each of these parts is a separate clock component with associated detailed requirements.

So when you consider a software system at a high level, it breaks down into one or more components. And it is possible for each component to further break down into sub-components. This concept is easy to understand for new development. For the maintenance case, you may only be receiving requirements for one component. So it may not be immediately obvious what the difference is between a product and a product component requirement.

Product component requirements apply throughout the lifecycle. However, you are more likely to encounter them in the early phases.

Specific Practice 2.2 “Allocate the requirements for each product component.” If you have a copy of the CMMI book, there are some helpful hints in the margin that should clarify what is meant by this practice. The intent of this practice is to ensure that the requirements for each product component are correctly specified (allocated). At this point in the lifecycle you aren’t addressing implementation, that will be at the 5 foot level. So at the product component level, you may have a mixture of hardware requirements, software requirements, performance requirements, operational requirements, etc. The intent of SP 2.2 is to allocate these different requirements to the proper component so you don’t implement a hardware requirement in software or vice versa by mistake. And once again, in the sustaining maintenance environment, when you receive new requirements, they may have already been allocated for you. So the distinction is not immediately obvious.

What is Meant by a Line of Business?

In a CMMI 1.2 Appraisal there is a requirement that practices in projects and functions within the Organisational Unit must be understood and identified ( e.g. Lines of Business, Disciplines, Effort Types, Project Types etc). What do lines of Business and Effort Types mean?

It might be easier to understand the concept of a LIne of Business through an illustrative example.

A small organization would normally produce one type of product, say a voice recognition software package. This package may be installed on a variety of platforms, but it is the same package. In this situation the company or organization has one Line of Business.

In a large company or corporation, there usually is a number of different types of products produced for different purposes, customer types, etc. For example, an automotive company may have several divisions: car, truck, van, commercial vehicle, etc. Each division represents a different Line of Business. Or a financial company may have several divisions: banking, insurance, investments, etc. Again, each division is a different Line of Business. Typically each Line of Business has different goals and objectives, customers, processes, etc. Another example of a Line of Business is the application domain. It is important to note the different Lines of Business because it is extremely challenging, if not impossible, to conduct a single SCAMPI A appraisal across multiple Lines of Business, especially for Maturity Level 3 and up.

If you read the Method Description Document section 1.1.3 Implementation Guidance, it defines the terms you are asking about:

• application domains (or lines of business)
• geographical breadth
• disciplines (e.g., systems engineering, software engineering, or hardware engineering)
• effort types (e.g., development, maintenance, or services)
• project types (e.g., legacy or new development)
• customer types (e.g., commercial or government agency)
• lifecycle models in use within the organization (e.g., spiral, evolutionary, waterfall, or incremental)

So effort type refers to the type of work: new development, sustaining engineering or maintenance, or services.

Criteria for selection of Focus and Non Focus projects for CMMI 1.2 Assessment

What are the criteria used to determine project selection for CMMI 1.2 Appraisals? I understand they are normally termed as Focus and Non Focus projects.

1. Based on size of organization ( count of projects + resources ) how many projects need to be selected? Can someone specify this as a number of projects or percentage of projects required?
2. We have categorised projects as Development + Maintenance + Testing - so ideally how many projects count or percentage wise under each Project Type would need to be selected?
3. What is the criteria for selecting a project as a focus project? I have read and heard that one of the factors could be presence of all Software Development Lifecycle (SDLC) phases in that project, contribution of data points, it should be an on-going project at time of assessment etc.

Here is the criteria for project selection extracted from section 1.1.3 of the Method Description Document (MDD) v1.2:

Sample projects and support groups selected to form the organizational scope (i.e., the combination of focus and non-focus projects and support functions) must represent all critical factors identified for the organizational unit to which the results will be attributed. The coverage of the organizational critical factors provided by these sample projects and support groups in the organizational scope in relation to the organizational unit must be documented, in quantitative terms, in the appraisal input and ADS.

Each sample project or support group in the planned organizational scope of the appraisal must be one of the three types listed below:

• Focus projects must provide objective evidence for every PA within the model scope of the appraisal which addresses model practices applicable to those projects.
• Non-focus projects must provide objective evidence for one or more PAs within the model scope of the appraisal which address practices performed on projects.
• Support functions must provide objective evidence for practices within the model scope of the appraisal that address organizational infrastructure or functions.

In appraisals where the reference model scope includes any project-related PA, the organizational scope must include at least one focus project. If the organizational unit includes more than 3 projects, then the organizational scope must include sufficient focus projects and non-focus projects to generate at least 3 instances of each practice in each project-related PA in the model scope of the appraisal.

Projects, categories, or groups/functions that are specifically excluded from the appraisal must be identified in the appraisal input and in the ADS as well as the justification for their exclusion. This identification includes legacy projects not using current organizational processes and projects waived from using current organizational processes. As needed, the appraisal team may seek clarification or data from other projects or support functions within the organizational unit. These projects or support functions must also be identified in the ADS.

What this means is that you must have at least one focus project. A focus project is a project that has evidence for every practice and Process Area (PA) in the appraisal scope. In order for that to be the case, that means the project is either already completed or very close to completion. A non-focus project is a project that does NOT have evidence for every practice and PA in the appraisal scope. Meaning a non-focus project is either a project that has recently started or a project that began before the processes were deployed and only has evidence for a subset of the practices.

Keep in mind that some PAs are organization level PAs while others are project-related. For the Organization PAs (e.g., OPF, OPD, OT, etc.) you only have to provide one instance of evidence. So the issue of number of projects only applies to the project-related PAs (e.g., PP, PMC, IPM, REQM, RD, etc.). Therefore, according to the MDD, if the organization has more than three projects, you have provide three instances of evidence for each practice. You could choose to supply three focus projects, two focus projects and multiple non-focus projects, or one focus project and multiple non-focus projects.

Another aspect of evidence that you must comply with is that you have to provide Direct Evidence for every practice and PA in the scope of the appraisal. But you don’t not have to supply Indirect Evidence for every practice and PA. Your Lead Appraiser can explain to you the one row, one column and 50% rules. As a risk mitigation, most of my clients choose to provide 100% coverage for both Direct and Indirect Evidence.

PMI PMBOK - CMMI Process Map

I often meet managers trained in the PMBOK and are certified PMPs (Project Management Professionals) who do not know enough about the CMMI but continue to press for using PMBOK as a guide in improving IT project management processes. I have studied the PMBOK and believe that the CMMI is a better option for managing IT projects. I can help PMPs understand my views better if I can show a map of processes in both the PMBOK and CMMI processes and then showing additional features of the CMMI helpful in managing IT projects. I would appreciate it if you could lead me to any such study where the two are mapped, something similar to the CMMI - ISO 9001:2000 map.

Try this link from the SEI http://www.dtic.mil/ndia/2005cmmi/thursday/sherer.pdf, it will pull up a presentation that compares and contrasts the CMMI and PMBOK. In addition, this page http://www.sei.cmu.edu/cmmi/adoption/comparisons.html provides links to a number of comparisons between the CMMI and other models and standards.

Full Time Resources for Implementing the CMMI

I have been asked to estimate the number of full time resources required by my company to facilitate its drive to Maturity Level 3 PM, SYS, SW, HW and ACQ. Is there any documentation or published information that will help in putting together a robust estimate of resources required?

First and foremost the driving factor for estimating the number of full time resources needed to implement the CMMI is the size of the organization. Over time we have seen that it takes 3 – 5% of the organization to perform PPQA, 3- 5 % of the organization to perform CM, and 3 – 5% of the organization to perform the necessary CMMI implementation activities.

So, if your organization is about 20 – 30 people, then you may only need one full time resource. However, if your organization is about 100 people, then you may need 3 to 5 full time resources.

Other factors contributing to this estimate is how strong a ML 2 foundation is already in place and how much of what you currently have in place is at ML 3. Documenting the processes and procedures is the easy part, and it can be done by a small core group. The larger task is deploying the new processes and process assets and having people use them to change how they approach their jobs.

Interview Questions to Hire a CMMI Expert

We are implementing the CMMI within our organization and are looking to hire someone to help us achieve this goal. We don't necessarily need a certified Lead Appraiser as of yet, but would like to hire someone with CMMI experience and who may beinterested in becoming a Lead Appriaser. I have a couple of internal candidates that are not Lead Appraisers, but have had CMMI experience (according to their resumes anyhow). What would be some good questions to ask them in an interview to gauge how much experience they truly have? I appreciate any help I can get with this.

One word of caution first. It may not be in your best interests to hire someone who wants to become a Lead Appraiser. There usually aren’t enough internal appraisal opportunities for a candidate Lead Appraiser to get the minimum experience or to maintain their Lead Appraiser credentials, so the person would have to look for appraisal work outside of your organization or company.

Here are some questions that I would ask a candidate for a CMMI position:

1. Have you taken the SEI’s 3-day Introduction to CMMI class? If yes, when did you take the class? Who was your instructor?
2. Have you participated as an appraisal team member? If yes, how many times? What were your duties? What was the scope of the appraisal (Maturity Level)? Who was the Lead Appraiser? What would the Lead Appraiser say about your CMMI capabilities and performance on the appraisal team?
3. Have you helped implement the CMMI in an organization?
4. How long have you been working with the CMMI?
5. Please compare and contrast Capability Level vs. Maturity Level
6. What is the only Process Area that can be categorized as Not Applicable? SAM
7. Have you prepared a PIID? If yes, what was the most difficult task and why?
8. How many years of project management experience do you have?
9. How many years of engineering experience do you have?
10. What is your favorite Process Area and why?

Accepting Items for Use in the Process Asset Library

The Configurable Items (CI) in our organization are controlled CIs [Any Project Related Artifact (PRA) received from any source, on which the project team does not exercise any control]. For example, documents obtained from the client or reference technical material. Documents of external origin, such as the ISO Standard, SEI CMMI, etc. are controlled and managed.

Any Project Related Artifact (PRA) is created and managed by the project team and the changing of which does not need any explicit change request, for example, Software Configuration Plan, Review Plan, Audit Plans etc. A baselined CI is any PRA that is managed by the project team and changing it needs an explicit change request for example: Project Plan, SRS ( Software Requirements Specifications) etc.

I am the lead responsible for building our Process Asset Library (PAL) and would appreciate guidelines for what could be the mandatory or acceptable criteria by which the baselined CIs would be updated in the PAL for use by other groups in the organization. I feel, it would certainly help if the criteria could be defined so that people in the organization would know the parameters based on which they could help contribute or share best practices, lessons learned, and any other artifacts in the Process Asset Library for use by anyone in the organization.

I really hate to do this to you, but the answer to your question is it depends upon your needs. Your organization has to decide for itself the acceptance criteria for PAL items. The CMMI does not mandate any criteria for accepting input to the PAL.

From a practical standpoint, you want to encourage the practitioners, project teams, and managers to submit everything of use to you for consideration as candidate PAL items. As the lead responsible for building the PAL, you might consider reviewing each submitted item to determine if you want it in the PAL. I always advocate that in addition to providing a template in the PAL for use by the project teams that you also provide an example of how you want the template to be correctly filled out. Providing a good example can be problematic sometimes. So you would have to review each submitted artifact of the completed template to determine if you want to post that as an example in the PAL. Other project related information may need to be sanitized or normalized by you before you post it to the PAL for use by existing or new projects.

If someone wanted to share a best practice or make a process improvement suggestion, then you should have a process in place for allowing someone to submit a process change. Many companies use their established product/project Change Request system as the tool for submitting Process Change Requests.

The Bottom Line is that you should encourage and accept all data, information, and input as candidate PAL items. Then the Software Engineering Process Group (SEPG), Engineering Process Group (EPG), or Process Group (PG) reviews the input for inclusion in the PAL.

Supplier Agreement Management Question

I have a question related to Supplier Agreement Management (SAM) SP 2.2 - Monitor Selected Supplier Processes. What is the basic intent of this practice and in what scenario does it fit in? It specifies "...situations of tight alignment between processes implemented by the supplier and those of the project..." - which is normally not the case in most (small) projects (as I know). I hope here we are not including "Acceptance" and "Transition" as aligned processes. Also it seems redundant to me with SP 2.1 - Execute the Supplier Agreement because in the contract/ SOW, it is usually mentioned how the supplier needs to monitor his processes (frequency to perform process audits etc.) and the frequency/condition when the customer may ask for a process audit/assessment. So doesn't executing the Supplier agreement covers these two practices?

I can see where you might have some confusion concerning these two SAM practices. SAM SP 2.1 says “Perform activities with the supplier as specified in the supplier agreement.” And your confusion comes about from sub-practice 1 “Monitor supplier progress and performance (schedule, effort, cost, and technical performance) as defined in the supplier agreement.” In essence SAM SP 2.1 is all about performing Project Monitoring and Control (PMC) over the supplier, which should be spelled out in the supplier agreement. You are basically acting as the Project Manager for the supplier by monitoring and controlling their project and technical performance. There is no intent to perform any Process and Product Quality Assurance (PPQA) audits or activities to support this practice.

In contrast SAM SP 2.2 says “Select, monitor, and analyze processes used by the supplier.” This practice is where you perform PPQA activities (process audits and work product audits) on selected supplier processes that are critical to the success of your project and business. Again, this ability to perform PPQA on the supplier must be specified in the supplier agreement. But you want to have the freedom to select any supplier process, so don’t indicate specific processes in the supplier agreement. For example you might decide to monitor and analyze how your supplier performs peer reviews or how they manage their requirements. If you have your supplier doing small projects (less than a month in duration) you may not have many opportunities to perform PPQA on a given supplier project. This situation is the same as when you have small projects done completely in house. There is no hope or expectation that you will be able to perform PPQA activities on every small project.

Look back at my blog on PPQA Audit Frequency http://ppqc.blogspot.com/2008/05/ppqa-audit-frequency.html for a simple way to adjust the frequency of the PPQA audits based on the quality issues discovered. You can use this same approach to determine the frequency of conducting PPQA activities on your supplier, assuming that you supplier regularly performs small projects for you.

Is CMMI only for Software Companies?

Can the CMMI be applied only for software developments companies, or can systems integration companies also derive benefit from it ? What I mean by "systems integration companies" are organizations who engage in selling IT solutions including sofware and hardware as a solution.

The CMMI is not a software only model. By the end of this year, the SEI will have released three CMMI constellations: CMMI for Development, CMMI for Acquisition, and CMMI for Services. The Development and Acquisition constellations are currently available, the Services constellation is due to be released this Fall. The CMMI for Development (CMMI-DEV) covers hardware, software, and systems engineering and so it will apply to systems integration companies as you have defined the term.

Defect Removal Efficiency (DRE)

I am a Defect Prevention team member in our CMMI Maturity Level 5 company. We are trying to plot the Defect Removal Efficiency (DRE) metric in our projects. I have a few questions:

1. Our DRE is based on phase level defect injection and detection concepts. If we go closely by the definition of the phase "testing/QA", bugs should not be injected in testing phase, as this is detection activity. If not then any examples can be useful for me to understand.
2. Can a common DRE format can be utilized for all kinds of projects as in sustaining engineering, maintenance, development and pure testing? Specifically what do to we do for testing or pure QA type of projects?
3. DRE is scoped for defects leaked by QA and reached to customer. How we can address defects in other lifecycle phases i.e. deployment and post production activities?

These are very interesting questions. Here is my two cents worth:

1. Your assumption that testing/QA phase is not a source of defects may or may not be a good assumption. Test cases can be a source of defects as well as the product. A test case that does not find any defects may in fact itself be defective. The purpose of a test case is to find defects in the product. In this case there might be undetected defects slipping through the test cases or erroneous test results. On the other hand testing can sometimes identify defects in the product that may not actually exist. I encountered this situation years ago in a small software development company when they hired a new tester. The company had no documented procedures and testing criteria. So the new tester did the best job he could under these circumstances and based his testing on the published User’s Manual. He proceeded to find tons of defects in the product. That got management’s attention until they realized that all of these “new” defects had been previously addressed. It turned out that the tester was unwittingly using an obsolete version of the User’s Manual. So that mistake invalidated all of his test results. Either way, each of these two situations would contaminate your DRE calculations. Therefore, I would be very careful in making the assumption that the testing/QA phase is not a source of defects, unless you have thoroughly peer reviewed each test case and validated each test result to ensure that what I have described has not happened.
   
2. In the context of the CMMI, QA usually means for most of us Lead Appraisers PPQA – conducting process audits and work product audits, not testing. So I am not clear as to the distinction you are trying to make between testing and "pure QA". It appears that in your organization they may be two aspects of the same activity. Personally, I think that you could use a common DRE format across all types of projects. However, it doesn’t make sense to mix the data for the various projects. So you will have to be very careful to keep the data segregated, otherwise you won’t be able to draw the proper conclusions. For example, the DRE for new development will be different than for sustaining engineering, different from a testing project, etc.
   
3. I think that you answered your question in your statement of the scope. Just change the definition of the scope for your DRE. You could define the DRE scope on a lifecycle phase by phase basis and then you would be able to measure it for deployment and post-production activities. You could also define a DRE for the entire project lifecycle as well.

SEI Published Appraisal Results

I have a question related to the published data in SEI web site: http://www.sei.cmu.edu/appraisal-program/profile/pdf/CMMI/2008MarCMMI.pdf Process Maturity Profile by All Reporting Organizations (Page 5):

1. 1.5% or the reporting organizations are (Initial - Maturity Level 1) rating. How is a Maturity Level 1 rating achieved?
2. Also, 8% of the reporting organizations indicate that a Maturity Level is Not Given. What does ‘Not Given’ mean…? Level 0 ?

In my understanding, Maturity Level is related to the Staged representation and can have
rating from ML 2 - 5.

It is great to see and hear that people are actually looking at this information and asking questions about how to interpret the data. You are on your way to understanding Measurement and Analysis!

To answer your first question, how is Maturity Level 1 provided? Obviously, an organization does not conduct a SCAMPI A appraisal to confirm that they are Maturity Level 1. So what happens in this case is the scope of the SCAMPI A appraisal is Maturity Level 2, 3, 4, or 5. But what happens is during the appraisal a serious goal impacting weakness in a Maturity Level 2 Process Area is found by the appraisal team. By the rules of the SCAMPI method then the resulting Maturity Level is 1. Since the SCAMPI A results must be submitted to the SEI, a rating of Maturity Level 1 appears. It doesn’t happen that often because most organizations take the appropriate steps to be prepared to achieve their target Maturity Level. That is why 1.5% of the reported results are for Maturity Level 1.

The Not Given category means that the organization did not report its resulting Maturity Level rating. I would hazard a guess that it probably means that they had successfully achieved Maturity Level 1. There is no Maturity Level 0, despite what many chaotic organizations say they are when first encountering the CMMI.

A Question About Estimation

I'm a bit confused about what the CMMI requires for estimation. Is it valid for us to define our process as simple as guessing? Could we use something similar to Planning Poker or other similar agile techniques? The problem is that we've been told we need to have a quantitative and repeatable process for estimating in order to meet Project Planning (PP) SP 1.4 "Estimate the project effort and cost for the work products and tasks based on estimation rational." But in reality we don't have historical data and many of us think estimating is subjective anyway.

When you start implementing Maturity Level 2 there is no expectation that you have any historical data for estimating. The best you can do at that point is guess, SWAG, or use engineering judgment. One of the characteristics of being a Maturity Level 2 organization is that you are learning. You are learning what it means to estimate a project. You are learning how to improve your estimates so that over time you will be able to produce accurate estimates to build the foundation to allow the organization to move to Maturity Level 3. So, by the time you are ready for a Maturity Level 2 SCAMPI A appraisal, the expectation is that you have a repeatable estimation process (repeatability is at the core of Maturity Level 2) that is based on historical project data. Estimation is not difficult, it just takes a little bit of brain power and being systematic in your approach. One of the purposes of achieving Maturity Level 2 is moving from being subjective in estimation to being objective. If you have people that believe estimation is subjective, then I am afraid that you are still operating as a Maturity Level 1 organization.

Delivering Process Training

Training plays a major role in the implementation of any process in the organization. The way it is done and the examples/illustrations explained during the session definitely helps the employees understands the need for such model implementation for process improvement. Can some videos, pictures, jokes work out well in a typical CMMI or QMS training session. How can the effectiveness of the training be improved based on these methods or innovative techniques? How do you cover the crowd?

There is no prescribed method in the CMMI for how you train people. Everything you mention in your note sounds like great ideas for getting the message out. The only caution I have is to be sensitive to your audience. A joke that works in one setting may not work in another. A good instructor should be able to read the audience and adjust his or her teaching style appropriately to allow for differences.

So whatever you do to liven up what can be some very dry information is a good idea. You may want to use a video or movie clip that illustrates some important process concept. Go for it! It will make the class much more enjoyable for all concerned.

I recently read a very good book on developing and delivering inspirational messages called “Fire Them Up!” by Carmine Gallo http://www.carminegallo.com/. The book presents seven simple ways to inspire colleagues, customers, and clients. The seven methods can be summarized as igniting your enthusiasm, navigating the way, selling the benefit, painting a picture, inviting participation, reinforcing an optimistic outlook, and encouraging people to reach their potential.

Technical Query vs. Technical Issue

What is the difference between a technical query and a technical issue? And, how do you track them when the projects are maintaning all their queries and issues in email?

In my mind, this is how I distinguish between a technical query and a technical issue. The design team receives a set of requirements to implement. When analyzing the requirements, they encounter a vague or poorly worded requirement such as “the product must be user friendly.” A technical query would be a question back to the requirements provider asking him or her what is meant by this statement. Or a technical query might be a question asking for more detailed technical information so the design team can make the proper design decisions. In contrast a technical issue arises when there is a conflict between requirements, between new or modified requirements and the existing design, between the requirements and the product, several possible design alternatives, etc. A technical issue may require a trade study or research in order to provide the correct information to resolve the issue, basically invoking Decision Analysis and Resolution (DAR).

As far as tracking this information, that is up to the organization to decide on a method that best fits their business practices. Queries could be tracked via email. But I would think a better method would be to include the queries with the requirements so you would capture the history and discussion of any unclear requirements in one location. When you think about it, technical queries are part of the knowledge capture for understanding the requirements. For tracking technical issues, you can perform this action in a variety of ways. Many organizations track all issues the same way in an Issues Log or Issues Tracking System. There are many off the shelf tracking systems that basically have the same functionality.

Understanding How Project Planning Evolves from ML 3 to ML 5

The primary difference between project planning at Maturity Level 3 (ML 3) and Maturity Level 5 (ML 5) is how you construct the project’s defined processes. At ML 3 a new project starts from the organization’s set of standard processes and applies the appropriate tailoring criteria or guidelines for the project, based on the project requirements. At Maturity Level 4 (ML 4) and ML 5 a new project first has to define what its quality and process performance objectives are based on the organization’s quality and process performance objectives (QPPOs). Then the project looks at the organization’s set of standard processes and selects the processes or sub-processes that are capable of being quantitatively and statistically managed to achieve the project’s QPPOs using the established Process Performance Baselines and Process Performance Models. These processes and sub-processes are selected based on historical stability and capability data along with the appropriate tailoring guidelines and criteria to construct the project’s defined process.