How can I see a list of CMMI appraised companies across the globe for all levels and years?
With the release of CMMI v1.2 in August 2006, the SEI instituted a new policy regarding the expiration of appraisal results. All appraisal results expire three years from the anniversary date of the appraisal. The list of published appraisal results is available from the SEI at http://sas.sei.cmu.edu/pars/pars.aspx. Once the appraisal results reach their third anniversary date, they are automatically removed from this list. For example you could check the list today and find the results for an appraisal conducted December 2, 2005. Wait a day or so and you will no longer find the record. You can sort this list based on year, organization name, Maturity Level, etc.
Keep in mind that once the appraisal results have expired, they are no longer valid and the organization should have been re-appraised in order to continue to claim a particular Capability or Maturity Level.
Monday, December 1, 2008
Tuesday, November 11, 2008
A Question About DAR Calculations
I have a question regarding DAR calculations. When assigning wieghts to the selected criteria, do I have to relate them to each other? Meaning, do I have to refer to a number that represents my 100% need and relate all criteria to it? For example, I have three Criteria A, B, and C, then I'll assign 3 (30%) to A, 5 (50%) to B, and 2 (20%) to C, and in case one of them was decreased the others will increase. Currently, I define a range for criteria weight (for example from 1 to 5), and I select a number subjectively based on the weight of the criteria.
I think that you may have mixed things up a bit here. The evaluation criteria should be independent and not assigned percentages, the percentages apply to the weighting factors. For each criterion you assign scores between 1 (low) and 5 (high). The weights are derived from team consensus and total to 100% or 1. 20%, 30%, and 50% are typical weights. It is best not to have more than three evaluation criteria because if you have more, then there is the potential for the resulting scores to be too close to each other and it will be very difficult to determine which is the best solution.
With three criteria it is easy to assign 50%, 30%, and 20%. With more criteria you will be forced to assign weights that are closer together. So if you go strictly by the numerical calculations, then there is the potential for lots of ties or very close results thereby making it a challenge to use this technique for selecting an answer. For example, if you had five criteria, then one possible approach, though wrong one, would be to assign 20% to each criteria. Other options would be to assign weights of 30%, 25%, 20%, 15%, and 10% or 50%, 20%, 15%, 10%, and 5%. But you might find it difficult to reach consensus among the team for these values and how to apply them to the criteria. With these assignments, you might calculate the same score for one or more alternative solutions. So with more than three criteria the resulting unique weights could be very close together and not very useful for using this method to calculate scores and use the results to make a decision.
My premise is to limit the number of evaluation criteria to no more than three, not the number of alternative solutions. You can have as many alternative solutions as you want.
And another point is to try to select evaluation criteria that are independent, otherwise you will face challenges as well. For example if you are buying an automobile the independent evaluation criteria might be: 1. Features and price 2. Distance from your home to the dealer and 3. Service reputation.
However, you always have to use the sanity test to determine if the results of the calculation make any sense. You could have chosen the wrong criteria, assigned the wrong weights, and/or assigned the wrong values to each criteria. And yes, management always has the prerogative of making a decision without using the results of the process.
I think that you may have mixed things up a bit here. The evaluation criteria should be independent and not assigned percentages, the percentages apply to the weighting factors. For each criterion you assign scores between 1 (low) and 5 (high). The weights are derived from team consensus and total to 100% or 1. 20%, 30%, and 50% are typical weights. It is best not to have more than three evaluation criteria because if you have more, then there is the potential for the resulting scores to be too close to each other and it will be very difficult to determine which is the best solution.
With three criteria it is easy to assign 50%, 30%, and 20%. With more criteria you will be forced to assign weights that are closer together. So if you go strictly by the numerical calculations, then there is the potential for lots of ties or very close results thereby making it a challenge to use this technique for selecting an answer. For example, if you had five criteria, then one possible approach, though wrong one, would be to assign 20% to each criteria. Other options would be to assign weights of 30%, 25%, 20%, 15%, and 10% or 50%, 20%, 15%, 10%, and 5%. But you might find it difficult to reach consensus among the team for these values and how to apply them to the criteria. With these assignments, you might calculate the same score for one or more alternative solutions. So with more than three criteria the resulting unique weights could be very close together and not very useful for using this method to calculate scores and use the results to make a decision.
My premise is to limit the number of evaluation criteria to no more than three, not the number of alternative solutions. You can have as many alternative solutions as you want.
And another point is to try to select evaluation criteria that are independent, otherwise you will face challenges as well. For example if you are buying an automobile the independent evaluation criteria might be: 1. Features and price 2. Distance from your home to the dealer and 3. Service reputation.
However, you always have to use the sanity test to determine if the results of the calculation make any sense. You could have chosen the wrong criteria, assigned the wrong weights, and/or assigned the wrong values to each criteria. And yes, management always has the prerogative of making a decision without using the results of the process.
Tuesday, October 28, 2008
Test Case Coverage & Review Effectiveness
Can anyone help me for the definition of Test Case Coverage and Review Effectiveness? How is Review Effectiveness calculated for work products like SRS, SDD &Test Spec?
The term Test Case Coverage means to me a measure of how many of the product requirements are being tested by the defined test cases. It is the testers’ job to define their test cases based on the requirements and the goal should be 100% coverage or close to that. Please bear in mind that 100% coverage is not the same as exhaustive testing. Exhaustive testing means testing every path through the software using all possible values.
The Requirements Traceability Matrix is what you would use to determine the Test Case Coverage. You should be able to map every test case to the requirements and vice versa. Once you have done that you will be able to determine if one or more of the requirements are not being covered by the test cases. So a simple ratio of the number of reqmts not being tested to the total number of requirements would yield a Test Case Coverage measure.
Review Effectiveness is a bit harder to define. When you hold a review, say for the SRS, then you would have a total number of defects discovered and you also know the number of pages in the document. So, to first order, you could divide the total number of defects by the number of pages and derive a defect density measure for the document. But that number by itself doesn’t tell you a whole lot. You need to have a good understanding of the expected number of defects per page coming out of a review. That number would come from your historical data. But this number still doesn’t yield an effectiveness measure. You could be finding lots of minor defects in a review and the major ones are slipping through. So you have to look for defects that were missed by the review in downstream activities such as testing and other reviews. You could also be grouping defects by major and minor categories etc.
You should be asking several questions that would lead you to an effectiveness measure:
1. In what review did you find the defect?
2. In what activity was the defect inserted?
3. What review should have caught the defect but didn’t?
The term Test Case Coverage means to me a measure of how many of the product requirements are being tested by the defined test cases. It is the testers’ job to define their test cases based on the requirements and the goal should be 100% coverage or close to that. Please bear in mind that 100% coverage is not the same as exhaustive testing. Exhaustive testing means testing every path through the software using all possible values.
The Requirements Traceability Matrix is what you would use to determine the Test Case Coverage. You should be able to map every test case to the requirements and vice versa. Once you have done that you will be able to determine if one or more of the requirements are not being covered by the test cases. So a simple ratio of the number of reqmts not being tested to the total number of requirements would yield a Test Case Coverage measure.
Review Effectiveness is a bit harder to define. When you hold a review, say for the SRS, then you would have a total number of defects discovered and you also know the number of pages in the document. So, to first order, you could divide the total number of defects by the number of pages and derive a defect density measure for the document. But that number by itself doesn’t tell you a whole lot. You need to have a good understanding of the expected number of defects per page coming out of a review. That number would come from your historical data. But this number still doesn’t yield an effectiveness measure. You could be finding lots of minor defects in a review and the major ones are slipping through. So you have to look for defects that were missed by the review in downstream activities such as testing and other reviews. You could also be grouping defects by major and minor categories etc.
You should be asking several questions that would lead you to an effectiveness measure:
1. In what review did you find the defect?
2. In what activity was the defect inserted?
3. What review should have caught the defect but didn’t?
Wednesday, October 22, 2008
A Question About Measures
Can someone please help me with the following questions:
- Are there examples of typical or common measurements than an organization can use for every process area? (different examples appear in Generic Practice (GP) 2.8)
- Where I can look up them?
- Are there some measurement standards that show or define typical or common measurements? Which?
- Do they apply to a specific Process Area (PA)? If so, which PA?
Granted there are some common measures such as effort and time spent on an activity. But the underlying principle of the CMMI and more importantly Measurement and Analysis (MA) is to define your goals and objectives and use them to determine your measurement needs. Then you will be able to easily specify those measures that have the most value to your organization and projects. You have to understand why you are collecting a specific piece of data and how you are going to use the information to make a decision, otherwise you will have difficulties and over time people will forget why you are collecting and using the data. They will not see the benefits of these measures. As I have posted to this blog before, start with the MA Process Area and use the practices to define your measures, how you will collect , analyze, report, and use them. MA is based on Practical Software Measurement (PSM). Read the PSM book and visit the PSM web site http://www.psmsc.com/ . The book is very easy to read and understand. It contains several case studies and example measures that will help you define your organization’s measures.
Tuesday, October 21, 2008
Monitoring Stakeholder Involvement
Project Monitoring and Control (PMC) Specific Practice (SP) 1.5 states "Monitor stakeholder involvement against the project plan." Our organization has chosen to implement this practice by creating a Stakeholder Management Plan (SMP) for each project. The SMP lists each stakeholder, their involvement, the monitoring method, and other pertinent information. Stakeholder involvement is checked during each activity and we do not see any value to having an overt and separate check of the SMP activities by the Project Manager (PM).
- PMC SP 1.5 specifies that a periodic review of the status of stakeholder involvement. How often should this review occur?
- What are examples of Direct and Indirect Evidence for PMC SP 1.5 that can be used to help us develop the PMC PIID?
The purpose of PMC is to provide an understanding of the project’s progress so that appropriate corrective actions can be taken when the project’s performance deviates significantly from the plan. Since the Stakeholder Management Plan is one of the components of the project plan, it must be monitored, controlled, and managed on a regular basis in order to take appropriate corrective action. As PMC is normally a PM’s responsibility, it is expected that the PM is regularly monitoring the SMP and its use, but this responsibility can be delegated.
As the stakeholders specified in the SMP are involved in different activities and at different times throughout the project’s life cycle, specifying a fixed monitoring frequency for the entire plan may not provide a lot of value to the PM and the project. It may make more sense to monitor stakeholder involvement against the SMP on an individual activity basis. One method for implementing this type of monitoring would be to specify in the SMP the monitoring frequency for each stakeholder activity, the person responsible for monitoring each activity and stakeholder involvement, and the mechanism for documenting and communicating the monitoring results for each activity. Providing this information in the SMP would then indicate what would be appropriate Direct and Indirect Evidence for a SCAMPI appraisal.
Monday, October 20, 2008
CMMI Use in a Non-Software Organization
The CMMI model is used for improving the maturity of company processes, mainly in software organizations. But the model says that CMMI can be used also in other engineering contexts, such as mechanical, electrical etc. Does anyone know about companies that have been evaluated in contexts other than software?
As you have stated, the CMMI for Development (CMMI-DEV) is useful for systems engineering, hardware engineering, and software engineering. Over the past 20 to 30 years, it has been primarily software organizations that have been implementing maturity models. But these models do apply in other contexts. In point of fact, I have been working with a client for over a year that just performs systems engineering and we are using the CMMI-DEV. We are currently planning their ML 3 SCAMPI A for early 2009. There is no software development involved in the scope of their appraisal. It has been an interesting experience to think outside of the software realm when interpreting the SPs and GPs, but it does work very nicely.
As you have stated, the CMMI for Development (CMMI-DEV) is useful for systems engineering, hardware engineering, and software engineering. Over the past 20 to 30 years, it has been primarily software organizations that have been implementing maturity models. But these models do apply in other contexts. In point of fact, I have been working with a client for over a year that just performs systems engineering and we are using the CMMI-DEV. We are currently planning their ML 3 SCAMPI A for early 2009. There is no software development involved in the scope of their appraisal. It has been an interesting experience to think outside of the software realm when interpreting the SPs and GPs, but it does work very nicely.
Confusion regarding Project Montoring and Control
I have some confusion regarding Project Monitoring and Control (PMC) Specific Practice (SP) 1.2 Monitor Commitments and SP 1.5 Monitor Stakeholder Involvement. In my mind the intent of both practices are logically the same and the type of evidence produces should also be the same (eg. schedule, minutes of meeting/action file, and status reports). I think these are two duplicate practice. Kindly let me know your view regarding this issue.
This is an excellent question. There could be some overlap between Project Monitoring and Control (PMC) Specific Practice (SP) 1.2 “Monitor commitments against those identified in the project plan” and SP 1.5 “Monitor stakeholder involvement against the project plan.” The primary difference concerns commitment vs. stakeholder involvement. Commitments include such things as delivery dates, specific deliverables, requirements, etc. Commitments are those things that everyone agrees to and they must be documented to ensure a consistent mutual understanding. Documenting the commitments indentifies the responsibilities of those involved with the project (stakeholders).
Monitoring the commitments is not the same thing as monitoring stakeholder involvement. For example group A produces an interface specification needed by group B so they can perform their assigned activities. The commitment that group A has made to group B is that group A will deliver the interface specification on Friday Oct 31, 2008. Monitoring the commitment in this case is periodically checking to see if the interface specification will be delivered early, on time, or late. If the specification will be early or late, then some type of corrective action may be necessary. In contrast, the intent of SP 1.5 is to use a stakeholder involvement plan or matrix to ensure that all of the relevant stakeholders who are supposed to be involved in producing, reviewing, and accepting the interface specification are performing their roles as identified and planned. If not, then some type of corrective action may be necessary.
Granted you may have the same evidence for both practices but the evidence demonstrates different purposes and use.
This is an excellent question. There could be some overlap between Project Monitoring and Control (PMC) Specific Practice (SP) 1.2 “Monitor commitments against those identified in the project plan” and SP 1.5 “Monitor stakeholder involvement against the project plan.” The primary difference concerns commitment vs. stakeholder involvement. Commitments include such things as delivery dates, specific deliverables, requirements, etc. Commitments are those things that everyone agrees to and they must be documented to ensure a consistent mutual understanding. Documenting the commitments indentifies the responsibilities of those involved with the project (stakeholders).
Monitoring the commitments is not the same thing as monitoring stakeholder involvement. For example group A produces an interface specification needed by group B so they can perform their assigned activities. The commitment that group A has made to group B is that group A will deliver the interface specification on Friday Oct 31, 2008. Monitoring the commitment in this case is periodically checking to see if the interface specification will be delivered early, on time, or late. If the specification will be early or late, then some type of corrective action may be necessary. In contrast, the intent of SP 1.5 is to use a stakeholder involvement plan or matrix to ensure that all of the relevant stakeholders who are supposed to be involved in producing, reviewing, and accepting the interface specification are performing their roles as identified and planned. If not, then some type of corrective action may be necessary.
Granted you may have the same evidence for both practices but the evidence demonstrates different purposes and use.
Subscribe to:
Posts (Atom)