Test Case Coverage & Review Effectiveness

Can anyone help me for the definition of Test Case Coverage and Review Effectiveness? How is Review Effectiveness calculated for work products like SRS, SDD &Test Spec?

The term Test Case Coverage means to me a measure of how many of the product requirements are being tested by the defined test cases. It is the testers’ job to define their test cases based on the requirements and the goal should be 100% coverage or close to that. Please bear in mind that 100% coverage is not the same as exhaustive testing. Exhaustive testing means testing every path through the software using all possible values.

The Requirements Traceability Matrix is what you would use to determine the Test Case Coverage. You should be able to map every test case to the requirements and vice versa. Once you have done that you will be able to determine if one or more of the requirements are not being covered by the test cases. So a simple ratio of the number of reqmts not being tested to the total number of requirements would yield a Test Case Coverage measure.

Review Effectiveness is a bit harder to define. When you hold a review, say for the SRS, then you would have a total number of defects discovered and you also know the number of pages in the document. So, to first order, you could divide the total number of defects by the number of pages and derive a defect density measure for the document. But that number by itself doesn’t tell you a whole lot. You need to have a good understanding of the expected number of defects per page coming out of a review. That number would come from your historical data. But this number still doesn’t yield an effectiveness measure. You could be finding lots of minor defects in a review and the major ones are slipping through. So you have to look for defects that were missed by the review in downstream activities such as testing and other reviews. You could also be grouping defects by major and minor categories etc.

You should be asking several questions that would lead you to an effectiveness measure:
1. In what review did you find the defect?
2. In what activity was the defect inserted?
3. What review should have caught the defect but didn’t?

A Question About Measures

Can someone please help me with the following questions:

1. Are there examples of typical or common measurements than an organization can use for every process area? (different examples appear in Generic Practice (GP) 2.8)
2. Where I can look up them?
3. Are there some measurement standards that show or define typical or common measurements? Which?
4. Do they apply to a specific Process Area (PA)? If so, which PA?

Granted there are some common measures such as effort and time spent on an activity. But the underlying principle of the CMMI and more importantly Measurement and Analysis (MA) is to define your goals and objectives and use them to determine your measurement needs. Then you will be able to easily specify those measures that have the most value to your organization and projects. You have to understand why you are collecting a specific piece of data and how you are going to use the information to make a decision, otherwise you will have difficulties and over time people will forget why you are collecting and using the data. They will not see the benefits of these measures. As I have posted to this blog before, start with the MA Process Area and use the practices to define your measures, how you will collect , analyze, report, and use them. MA is based on Practical Software Measurement (PSM). Read the PSM book and visit the PSM web site http://www.psmsc.com/ . The book is very easy to read and understand. It contains several case studies and example measures that will help you define your organization’s measures.

Monitoring Stakeholder Involvement

Project Monitoring and Control (PMC) Specific Practice (SP) 1.5 states "Monitor stakeholder involvement against the project plan." Our organization has chosen to implement this practice by creating a Stakeholder Management Plan (SMP) for each project. The SMP lists each stakeholder, their involvement, the monitoring method, and other pertinent information. Stakeholder involvement is checked during each activity and we do not see any value to having an overt and separate check of the SMP activities by the Project Manager (PM).

1. PMC SP 1.5 specifies that a periodic review of the status of stakeholder involvement. How often should this review occur?
2. What are examples of Direct and Indirect Evidence for PMC SP 1.5 that can be used to help us develop the PMC PIID?

The purpose of PMC is to provide an understanding of the project’s progress so that appropriate corrective actions can be taken when the project’s performance deviates significantly from the plan. Since the Stakeholder Management Plan is one of the components of the project plan, it must be monitored, controlled, and managed on a regular basis in order to take appropriate corrective action. As PMC is normally a PM’s responsibility, it is expected that the PM is regularly monitoring the SMP and its use, but this responsibility can be delegated.

As the stakeholders specified in the SMP are involved in different activities and at different times throughout the project’s life cycle, specifying a fixed monitoring frequency for the entire plan may not provide a lot of value to the PM and the project. It may make more sense to monitor stakeholder involvement against the SMP on an individual activity basis. One method for implementing this type of monitoring would be to specify in the SMP the monitoring frequency for each stakeholder activity, the person responsible for monitoring each activity and stakeholder involvement, and the mechanism for documenting and communicating the monitoring results for each activity. Providing this information in the SMP would then indicate what would be appropriate Direct and Indirect Evidence for a SCAMPI appraisal.

CMMI Use in a Non-Software Organization

The CMMI model is used for improving the maturity of company processes, mainly in software organizations. But the model says that CMMI can be used also in other engineering contexts, such as mechanical, electrical etc. Does anyone know about companies that have been evaluated in contexts other than software?

As you have stated, the CMMI for Development (CMMI-DEV) is useful for systems engineering, hardware engineering, and software engineering. Over the past 20 to 30 years, it has been primarily software organizations that have been implementing maturity models. But these models do apply in other contexts. In point of fact, I have been working with a client for over a year that just performs systems engineering and we are using the CMMI-DEV. We are currently planning their ML 3 SCAMPI A for early 2009. There is no software development involved in the scope of their appraisal. It has been an interesting experience to think outside of the software realm when interpreting the SPs and GPs, but it does work very nicely.

Confusion regarding Project Montoring and Control

I have some confusion regarding Project Monitoring and Control (PMC) Specific Practice (SP) 1.2 Monitor Commitments and SP 1.5 Monitor Stakeholder Involvement. In my mind the intent of both practices are logically the same and the type of evidence produces should also be the same (eg. schedule, minutes of meeting/action file, and status reports). I think these are two duplicate practice. Kindly let me know your view regarding this issue.

This is an excellent question. There could be some overlap between Project Monitoring and Control (PMC) Specific Practice (SP) 1.2 “Monitor commitments against those identified in the project plan” and SP 1.5 “Monitor stakeholder involvement against the project plan.” The primary difference concerns commitment vs. stakeholder involvement. Commitments include such things as delivery dates, specific deliverables, requirements, etc. Commitments are those things that everyone agrees to and they must be documented to ensure a consistent mutual understanding. Documenting the commitments indentifies the responsibilities of those involved with the project (stakeholders).

Monitoring the commitments is not the same thing as monitoring stakeholder involvement. For example group A produces an interface specification needed by group B so they can perform their assigned activities. The commitment that group A has made to group B is that group A will deliver the interface specification on Friday Oct 31, 2008. Monitoring the commitment in this case is periodically checking to see if the interface specification will be delivered early, on time, or late. If the specification will be early or late, then some type of corrective action may be necessary. In contrast, the intent of SP 1.5 is to use a stakeholder involvement plan or matrix to ensure that all of the relevant stakeholders who are supposed to be involved in producing, reviewing, and accepting the interface specification are performing their roles as identified and planned. If not, then some type of corrective action may be necessary.

Granted you may have the same evidence for both practices but the evidence demonstrates different purposes and use.

Monitoring Data Management

Project Monitoring and Control (PMC) Specific Practice (SP) 1.4 states "Monitor the management of project data against the project plan." Our organization has chosen to implement the plan for managing project data by creating a Data Management Plan (DMP) for each project. The DMP lists each item and the method, its storage location, and the monitoring method along with other pertinent information. There are embedded checks and balances for placing items in the proper repository and they are distributed to the lowest level.

1. PMC SP 1.4 specifies that a periodic review of the data management activities against their descriptions in the project plan. How often should this review occur?
2. What are examples of Direct and Indirect Evidence for PMC SP 1.4 that can be used for developing the PMC PIID?

The purpose of PMC is to provide an understanding of the project’s progress so that appropriate corrective actions can be taken when the project’s performance deviates significantly from the plan. Since the DMP is one of the components of the project plan, it must be monitored, controlled, and managed on a regular basis in order to take appropriate corrective action. As PMC is normally the Project Manager's (PM's) responsibility, it is expected that the PM is regularly monitoring the DMP and its use, but this responsibility can be delegated.

As the items in the DMP are created and managed at different times throughout the project’s life cycle, specifying a fixed monitoring frequency for the entire plan may not provide a lot of value to the PM and the project. It may make more sense to monitor each item against the DMP on an individual basis. One method for implementing this type of monitoring would be to specify in the DMP the monitoring frequency for each item, the person responsible for monitoring each item, and the mechanism for documenting and communicating the monitoring results for each item. Providing this information in the DMP would then answer question 2 as well.

Stakeholder Invovlement Question

Our organization has project stakeholders whose only project involvement is as a decision maker in an end-of-phase meeting. What is expected of the stakeholders (e.g., training and project involvement) if they are outside of our process, as far as the appraisal interviews are concerned?

In the case described, the stakeholder is NOT outside the process.

The “plan for stakeholder involvement” defines the expected role of the stakeholder in the project, or process. This, in turn, defines the scope of training required. In the case described, the stakeholder would minimally need training in the conduct of end-of-phase meetings. However, if decision making required knowledge of work products produced by the project team, orientation to those products might also be required. Training in Decision Analysis and Resolution (DAR) may also be required, depending upon whether or not the decisions meet the criteria for initiating the DAR process.

Note that training may take many forms, including orientation or briefings on required topics. The expectation is that the stakeholder knows enough to credibly perform the responsibilities described in the project plan or process description.

In the case described, the stakeholder might be selected for an appraisal interview.