Test Case Coverage & Review Effectiveness

Can anyone help me for the definition of Test Case Coverage and Review Effectiveness? How is Review Effectiveness calculated for work products like SRS, SDD &Test Spec?

The term Test Case Coverage means to me a measure of how many of the product requirements are being tested by the defined test cases. It is the testers’ job to define their test cases based on the requirements and the goal should be 100% coverage or close to that. Please bear in mind that 100% coverage is not the same as exhaustive testing. Exhaustive testing means testing every path through the software using all possible values.

The Requirements Traceability Matrix is what you would use to determine the Test Case Coverage. You should be able to map every test case to the requirements and vice versa. Once you have done that you will be able to determine if one or more of the requirements are not being covered by the test cases. So a simple ratio of the number of reqmts not being tested to the total number of requirements would yield a Test Case Coverage measure.

Review Effectiveness is a bit harder to define. When you hold a review, say for the SRS, then you would have a total number of defects discovered and you also know the number of pages in the document. So, to first order, you could divide the total number of defects by the number of pages and derive a defect density measure for the document. But that number by itself doesn’t tell you a whole lot. You need to have a good understanding of the expected number of defects per page coming out of a review. That number would come from your historical data. But this number still doesn’t yield an effectiveness measure. You could be finding lots of minor defects in a review and the major ones are slipping through. So you have to look for defects that were missed by the review in downstream activities such as testing and other reviews. You could also be grouping defects by major and minor categories etc.

You should be asking several questions that would lead you to an effectiveness measure:
1. In what review did you find the defect?
2. In what activity was the defect inserted?
3. What review should have caught the defect but didn’t?

A Question About Measures

Can someone please help me with the following questions:

1. Are there examples of typical or common measurements than an organization can use for every process area? (different examples appear in Generic Practice (GP) 2.8)
2. Where I can look up them?
3. Are there some measurement standards that show or define typical or common measurements? Which?
4. Do they apply to a specific Process Area (PA)? If so, which PA?

Granted there are some common measures such as effort and time spent on an activity. But the underlying principle of the CMMI and more importantly Measurement and Analysis (MA) is to define your goals and objectives and use them to determine your measurement needs. Then you will be able to easily specify those measures that have the most value to your organization and projects. You have to understand why you are collecting a specific piece of data and how you are going to use the information to make a decision, otherwise you will have difficulties and over time people will forget why you are collecting and using the data. They will not see the benefits of these measures. As I have posted to this blog before, start with the MA Process Area and use the practices to define your measures, how you will collect , analyze, report, and use them. MA is based on Practical Software Measurement (PSM). Read the PSM book and visit the PSM web site http://www.psmsc.com/ . The book is very easy to read and understand. It contains several case studies and example measures that will help you define your organization’s measures.

Monitoring Stakeholder Involvement

Project Monitoring and Control (PMC) Specific Practice (SP) 1.5 states "Monitor stakeholder involvement against the project plan." Our organization has chosen to implement this practice by creating a Stakeholder Management Plan (SMP) for each project. The SMP lists each stakeholder, their involvement, the monitoring method, and other pertinent information. Stakeholder involvement is checked during each activity and we do not see any value to having an overt and separate check of the SMP activities by the Project Manager (PM).

1. PMC SP 1.5 specifies that a periodic review of the status of stakeholder involvement. How often should this review occur?
2. What are examples of Direct and Indirect Evidence for PMC SP 1.5 that can be used to help us develop the PMC PIID?

The purpose of PMC is to provide an understanding of the project’s progress so that appropriate corrective actions can be taken when the project’s performance deviates significantly from the plan. Since the Stakeholder Management Plan is one of the components of the project plan, it must be monitored, controlled, and managed on a regular basis in order to take appropriate corrective action. As PMC is normally a PM’s responsibility, it is expected that the PM is regularly monitoring the SMP and its use, but this responsibility can be delegated.

As the stakeholders specified in the SMP are involved in different activities and at different times throughout the project’s life cycle, specifying a fixed monitoring frequency for the entire plan may not provide a lot of value to the PM and the project. It may make more sense to monitor stakeholder involvement against the SMP on an individual activity basis. One method for implementing this type of monitoring would be to specify in the SMP the monitoring frequency for each stakeholder activity, the person responsible for monitoring each activity and stakeholder involvement, and the mechanism for documenting and communicating the monitoring results for each activity. Providing this information in the SMP would then indicate what would be appropriate Direct and Indirect Evidence for a SCAMPI appraisal.

CMMI Use in a Non-Software Organization

The CMMI model is used for improving the maturity of company processes, mainly in software organizations. But the model says that CMMI can be used also in other engineering contexts, such as mechanical, electrical etc. Does anyone know about companies that have been evaluated in contexts other than software?

As you have stated, the CMMI for Development (CMMI-DEV) is useful for systems engineering, hardware engineering, and software engineering. Over the past 20 to 30 years, it has been primarily software organizations that have been implementing maturity models. But these models do apply in other contexts. In point of fact, I have been working with a client for over a year that just performs systems engineering and we are using the CMMI-DEV. We are currently planning their ML 3 SCAMPI A for early 2009. There is no software development involved in the scope of their appraisal. It has been an interesting experience to think outside of the software realm when interpreting the SPs and GPs, but it does work very nicely.

Confusion regarding Project Montoring and Control

I have some confusion regarding Project Monitoring and Control (PMC) Specific Practice (SP) 1.2 Monitor Commitments and SP 1.5 Monitor Stakeholder Involvement. In my mind the intent of both practices are logically the same and the type of evidence produces should also be the same (eg. schedule, minutes of meeting/action file, and status reports). I think these are two duplicate practice. Kindly let me know your view regarding this issue.

This is an excellent question. There could be some overlap between Project Monitoring and Control (PMC) Specific Practice (SP) 1.2 “Monitor commitments against those identified in the project plan” and SP 1.5 “Monitor stakeholder involvement against the project plan.” The primary difference concerns commitment vs. stakeholder involvement. Commitments include such things as delivery dates, specific deliverables, requirements, etc. Commitments are those things that everyone agrees to and they must be documented to ensure a consistent mutual understanding. Documenting the commitments indentifies the responsibilities of those involved with the project (stakeholders).

Monitoring the commitments is not the same thing as monitoring stakeholder involvement. For example group A produces an interface specification needed by group B so they can perform their assigned activities. The commitment that group A has made to group B is that group A will deliver the interface specification on Friday Oct 31, 2008. Monitoring the commitment in this case is periodically checking to see if the interface specification will be delivered early, on time, or late. If the specification will be early or late, then some type of corrective action may be necessary. In contrast, the intent of SP 1.5 is to use a stakeholder involvement plan or matrix to ensure that all of the relevant stakeholders who are supposed to be involved in producing, reviewing, and accepting the interface specification are performing their roles as identified and planned. If not, then some type of corrective action may be necessary.

Granted you may have the same evidence for both practices but the evidence demonstrates different purposes and use.

Monitoring Data Management

Project Monitoring and Control (PMC) Specific Practice (SP) 1.4 states "Monitor the management of project data against the project plan." Our organization has chosen to implement the plan for managing project data by creating a Data Management Plan (DMP) for each project. The DMP lists each item and the method, its storage location, and the monitoring method along with other pertinent information. There are embedded checks and balances for placing items in the proper repository and they are distributed to the lowest level.

1. PMC SP 1.4 specifies that a periodic review of the data management activities against their descriptions in the project plan. How often should this review occur?
2. What are examples of Direct and Indirect Evidence for PMC SP 1.4 that can be used for developing the PMC PIID?

The purpose of PMC is to provide an understanding of the project’s progress so that appropriate corrective actions can be taken when the project’s performance deviates significantly from the plan. Since the DMP is one of the components of the project plan, it must be monitored, controlled, and managed on a regular basis in order to take appropriate corrective action. As PMC is normally the Project Manager's (PM's) responsibility, it is expected that the PM is regularly monitoring the DMP and its use, but this responsibility can be delegated.

As the items in the DMP are created and managed at different times throughout the project’s life cycle, specifying a fixed monitoring frequency for the entire plan may not provide a lot of value to the PM and the project. It may make more sense to monitor each item against the DMP on an individual basis. One method for implementing this type of monitoring would be to specify in the DMP the monitoring frequency for each item, the person responsible for monitoring each item, and the mechanism for documenting and communicating the monitoring results for each item. Providing this information in the DMP would then answer question 2 as well.

Stakeholder Invovlement Question

Our organization has project stakeholders whose only project involvement is as a decision maker in an end-of-phase meeting. What is expected of the stakeholders (e.g., training and project involvement) if they are outside of our process, as far as the appraisal interviews are concerned?

In the case described, the stakeholder is NOT outside the process.

The “plan for stakeholder involvement” defines the expected role of the stakeholder in the project, or process. This, in turn, defines the scope of training required. In the case described, the stakeholder would minimally need training in the conduct of end-of-phase meetings. However, if decision making required knowledge of work products produced by the project team, orientation to those products might also be required. Training in Decision Analysis and Resolution (DAR) may also be required, depending upon whether or not the decisions meet the criteria for initiating the DAR process.

Note that training may take many forms, including orientation or briefings on required topics. The expectation is that the stakeholder knows enough to credibly perform the responsibilities described in the project plan or process description.

In the case described, the stakeholder might be selected for an appraisal interview.

Another Work Environment Question

Both Organizational Process Definition (OPD) and Integrated Project Management (IPM) reference the work environment. What is the difference between OPD SP 1.6 and IPM SP 1.3?

OPD Specific Practice (SP) 1.6 is "Establish and maintain work environment standards" and IPM SP 1.3 is "Establish and maintain the project's work environment based on the organization's work environment standards." OPD is an organizational level Process Area (PA) and IPM is a project level PA.

The expectation for OPD SP 1.6 is that the organization defines work environment standards that allow both the organization and projects to benefit from a common set of tools, training, and maintenance. The intent of OPD SP 1.6 is NOT to define the standard work environment, but to define standards for the work environment. Standards typically include work environment operations, safety, and security procedures, standard hardware and software configurations, standard software application loads, and procedures for requesting waivers or tailoring. In addition, projects may have additional requirements for their work environments.

In contrast, the intent of IPM SP 1.3 is to use the work environment standards defined by OPD SP 1.6 to define the appropriate work environment for the project. The project's work environment might include the development environment, the testing environment, the integration environment, the verification environment, and the validation environment. These environments could be one environment or separate environments and/or facilities.

Work Environment Evidence Question

There are two Specific Practices in the CMMI that address the work environment:

• OPD SP 1.6 Establish and maintain work environment standards
• IPM SP 1.3 Establish and maintain the project’s work environment based on the organization’s work environment standards

There are other Specific Practices in the model that address specific environments:

• VER SP 1.2 Establish and maintain the environment needed to support verification
• VAL SP 1.2 Establish and maintain the environment needed to support validation
• PI SP 1.2 Establish and maintain the environment needed to support the integration of the product components

In addition, the model references a number of other project work environments: maintenance, operational, production, and engineering.

Since IPM SP 1.3 contains the phrase “establish and maintain”, the organization needs to provide for a SCAMPI A appraisal evidence of “formulate”, “document”, and “use” of the project’s work environment. What is appropriate evidence to provide for IPM SP 1.3?

The “document” evidence could be the document(s) containing the description of each of the project’s work environments and its relationship to the organization’s work environment standards. The “formulate” evidence might be the inspection/review report(s) of these document(s). And the “use” evidence could be an example of a work product produced using the documented work environment. For example, evidence of use of the development environment might be a product build report demonstrating use of the environment for developing the product.

Since the Verification, Validation, and Integration Environments are covered by other Process Areas, it would be appropriate to provide for IPM SP 1.3 evidence of use of other components of the documented work environment to demonstrate the work environment covers more than testing.

CM SP 3.2 Evidence Guidance

The release of CMMI v1.2 included a change to the examples for CM SP 3.2, Perform configuration audits to maintain integrity of the configuration baselines.
Examples of audit types include the following:

• Functional Configuration Audits (FCA) – Audits conducted to verify that the as-tested functional characteristics of a configuration item have achieved the requirements specified in its functional baseline documentation and that the operational and support documentation is complete and satisfactory.
• Physical Configuration Audit (PCA) – Audits conducted to verify that the as-built configuration item conforms to the technical documentation that defines it.
• Configuration management audits – Audits conducted to confirm that configuration management records and configuration items are complete, consistent, and accurate.

Are projects now required to provide evidence for each of these examples?

As the Configuration Management (CM) Specific Practice (SP) states, these are examples of Configuration Audits and are part of the informative material of the model. Therefore these examples are neither required nor expected. As long as the organization and projects are performing some kind of configuration audit(s) that assesses the integrity of the baseline(s), that meets the intent of the practice. For a SCAMPI Appraisal projects are not required to provide evidence for each type listed in the examples.

What is the cost of a CMMI v1.2 L3 certification?

Our company is interested to go for CMMI v1.2 ML3 certification. Can you answer the following questions:

1. What are the costs involved (Internal & External)?
2. Are there any SEI cerfication bodies in India?

First of all let me make one clarification to your query. There is no such thing as a CMMI certification. What an organization receives are the results of a SCAMPI A appraisal that indicate the Maturity Level of the organization on the day the appraisal concludes. The organization is not certified. And there is no such thing as an SEI certification body anywhere in the world. What do exist are authorized SEI partners that are allowed to provide CMMI consulting, training, and appraisal services. Visit this web site http://partner-directory.sei.cmu.edu/ and you will be able to find the SEI-authorized partners in India.

The answers to your questions are highly variable depending on the size and scope of your organization and your geographical location. The best place to obtain realistic estimates is to ask several local SEI-authorized CMMI consulting and appraisal providers for their cost proposals, then you will have a handle on the external costs. Internal costs really cannot be determined until you figure out how much work you have to do in order to implement the CMMI and prepare for an appraisal. Suffice it to say, your internal costs will most likely be greater than your external costs.

Can organizations still claim CMM?

If I remember correctly SEI retired the CMM and that organizations can no longer advertize their CMM maturity level. If an organization continues to claim on their web site that they are CMM level 4 - even today - and in addition it is claiming that it will be assessed shortly for CMMi Level 5 for more than a year and half - should we ignore it or bring it to the notice of SEI?If you suggest that we should bring such instances to SEI notice please give me the email id.

The short answer is that any claim of a CMM Maturity Level is no longer valid. As for the ML 5 notice, that is fine as long as the web site states that the organization is working towards ML 5, but I don't see how that is a marketing point. It is similar to a graduate student stating that he or she is in the Doctoral Program and have everything completed except for his or her thesis.

The SEI is not in the business of investigating false claims. Buyer Beware: The only official location to verify an organization's Capability or Maturity Level is the SEI's Published Appraisal Results site http://sas.sei.cmu.edu/pars/pars.aspx. Since the appraisal results are only good for three years, the results are automatically removed on the anniversary date. Just check this site daily for a week or two and you will be able to verify this for yourself. The best approach is to contact the offending organization to inform them of their mistaken claims as they may be unaware of the changes to the SEI's appraisal program. And if you still feel that the SEI should be notified, I would suggest that you contact Jill Diorio who is the SEI Ethics person at jdiorio@sei.cmu.edu.