Measurement and Analysis vs. Generic Practice 2.8

Would you expect that measures being reported for Generic Practice (GP) 2.8 on all Maturity Level (ML) 2 and ML 3 Process Areas (PAs) would follow the complete Measurement and Analysis (MA) PA? That is these GP 2.8 measures would have written objectives, operational definitions, storage, collection and analysis procedures in line with MA Specific Goal (SG) 1 and SG 2. Or can the PAs be monitored (measured) and controlled without the full breath of the MA PA?

This is a good question. So let’s take a step backwards and look at the CMMI and Generic Practice – PA Relationships. The summary table in the Generic Practice section of the model clearly states that Project Monitoring and Control (PMC) can implement GP 2.8 for all project-related processes. And MA provides general guidance about measuring, analyzing, and recording information that can be used in establishing measures for monitoring actual performance of the process. Please note that this information is GUIDANCE and part of the INFORMATIVE material. Therefore, it is not required that the org use MA for GP 2.8. HOWEVER, from a practical point of view, why would MA be part of the model if there wasn’t a requirement and expectation that it would be implemented? Since it is a ML 2 PA and you are asking about ML 3, as a Lead Appraiser I would expect to see that MA was used for defining, collecting, analyzing, and reporting both the project and process measures. Without implementing MA for the process measures, the org would be receiving little to no benefit from GP 2.8. And as I have seen GP 2.8 implemented, sometimes the process measures are embedded in the project measures that have been defined using MA.

MA and PPQA Questions

I have the following two basic queries about CMMI ML 2:

1. While writing a Metrics and measurement process, should we address the organization level metrics data consolidation and review. As ML 2 is project specific, is it proper to also document the organization level data consolidation? Also can anyone tell me, the right site for definition of metrics like requirement stability index, schedule variance, effort variance etc.?
   
2. Similarly while documenting PPQA process, is it proper to start with defining an organization level PPQA plan? I am looking for boundaries where to limit writing processes compliant to ML 2. I know that G.P 2.1 to G.P 2.10 must be in place to achieve CMMI ML 2, but the organization specific plans/areas must not be mentioned/documented at CMMI ML 2.

You sound like you are focusing on CMMI compliance rather than on your business goals and objectives. One of the basic tenets of the model is your business objectives. That is where your focus belongs. And if done properly, you will have the side benefit of being CMMI compliant. So, to address your questions:

1. When documenting your Measurement and Analysis process, you should focus on those measures that are important to you. Remember, the first MA practice SP 1.1 states “Establish and maintain measurement objectives that are derived from identified information needs and objectives.” So whatever you have identified as information needs and objectives, that should be your MA focus. At ML 2, for many organizations that are just doing this for the first time, I recommend the org take baby steps and begin with a project focus. But you don’t have to be restricted to the project, an ML 2 org may have also identified some org level measures as well. Go to the Practical Software and Systems Measurement web site for the specific measurement information you need www.psmsc.com
   
2. There are NO CMMI-imposed restrictions on the limits of PPQA. Your organization must define its own limits for the processes you are going to audit. Since GP 2.9 applies to all Process Areas, at a minimum for ML 2, PPQA applies to all of the ML 2 Process Areas you have implemented in your organization. But, if there are other processes that are critical and/or important to the success of your business, then it makes perfect sense to have PPQA audit them as well. Again, do what is right for your business.

CMMI Implementation

I recently joined a company where there is no process and management recruited me to implement the CMMI. The organization has different business units. Though everyone sits together, they work very differently. I conducted a Gap analysis based on the CMMI Level 2 processes and here are the findings:

1. Project Planning & PMC -- they create project plans and they have the regular project team meetings and they share the minutes. Each team has their own format and templates. The projects don't really do estimations. Can we satisfy the PP & PMC PA'ss without doing any estimation? I know it can't be that way, but can it be tailored?
2. Requirements Management: Some of the business units have CCBs to discuss change requests and other business units discuss requirements changes in their project team meetings. The most significant gap I found is in Requirements Traceability. Traceability of Customer requirements to the Functional Requirements and Traceability of Use cases to the Test cases are missing. Is this reason enough to fail the RM PA? One of the Engineering directors asked me how much traceability you need to satisfy this condition. At that time I said 100% of all the requirements. Then I also read from somewhere that it is OK to define that we maintain traceability for at least the MUST BE CUSTOMER REQUIREMENTS. Traceability of other requirements can be made optional. Can it be possible like that?
3. Configuration Management: The projects thought they have a CMP which is embedded in the project plan. What I found missing are the Configuration Audits ( PCA & FCA). Is it possible to satisfy the Configuration Management PA without doing Configuration Audits to check the document status, builds, and backup strategy?
4. PPQA: One of the business units has a Software Quality Assurance plan, but it is done by one of the testers from another project. As a part of PPQA the SQA will do some spot checks based on a pre-defined checklist, which includes Project Planning, Risk Management, Project Monitoring and Control, Integration and releases. But I guess this can be improved by my role as a independent software quality engineer.
5. M&A: I am very much worried with this process area, as of now the organization status is nil with respect to metrics collection, they don't have a metrics database and no metrics have been defined yet. Is it OK to start now to form a team to do some reasearch and come up with metrics definitions, deploy them and start collecting data? My question is how much data do we need to collect to satisfy this PA? And do we need to provide evidence of analyzing these collected data and show some improvements steps taken at the time of SCAMPI apprisals? How long will it take in general to satisfy the Measurement & Analysis PA?
6. SAM: can we tailor this PA if we are not dealing with suppliers? If yes how can it be possible?
   

The directive from the Leadership team is to acheive CMMI Level 3 by end of 2009. I was baffled to hear this. Under these circumstances, what are the chances of getting CMMI Level 3 or my traget is at least CMMI Level 2? That is what my initial target. Can I acheive CMMI Level 2 by the end of 2009? If so, what are the things I need to address?

Here are some of the things I have already started:

1. CMMI Overview training to all the teams
2. Dailogue session on metrics identification
3. Looking into some Requirements tools which provide Traceability
4. Need to push the project team to have configuration audits.

In addition we already have established a process data base and the processes are defined and templates are being used from the parent organization. Since our company is a multi-site company, one of our counter parts has already acheived CMMI Level 3. We will be using the same process database and their templates. I thinking of providing their training on each Process Area as well. Is this a correct way to use the processes and templates of our parent organization? If not, do we need to establish our own local process data base?

First of all, I sympathize with you and the challenges you face. I applaud the fact that you had the foresight to conduct a Gap Analysis of the organization. You have highlighted a number of key weaknesses within the organization. However, to provide you meaningful feedback on all of your points would require working directly with you and your company.

1. What is your CMMI experience? Have you taken the SEI’s Introduction to CMMI class? If not, I strongly recommend that you and possibly those others in your company who are responsible for your processes take the three day class. The class should provide you a more thorough understanding of the CMMI, its interpretations, and material for constructing an internal Overview class.
2. You have identified some serious deficiencies within the organization in all of the ML 2 Process Areas. These need to be analyzed, addressed, corrected, solutions implemented, and then re-evaluated some months into the future before you can consider a formal SCAMPI at any Maturity Level. The length of time before the next evaluation is a function of a number of factors: number of people in the organization, number of projects, typical project duration, how much time and other resources are dedicated to process improvement, etc.
3. The organization needs to first implement Maturity Level 2 to form a firm foundation before considering moving to Maturity Level 3. The issues you have identified are fairly typical. Basically, it sounds like your organization does not perform PPQA or MA and is challenged with Project Management, Requirements Management, and Configuration Management. The first steps here should be to identify the necessary skills-based training classes you need to bring in-house and train your staff on these concepts. If you just purchase tools and push for audits, you most likely will not achieve the desired effect. You have to understand your process first and the reasons for why it is important to perform each of the steps.
4. Since another division has already achieved ML 3, it is a good idea to learn from their mistakes. But be very careful of the temptation to “clone” their processes and procedures. You have to implement the processes and procedures that match the way you conduct business today.
5. Based on what you have outlined, and given how much time and effort it could take just to address the ML 2 issues, I would say that ML 3 is out of the question for 2009. You could conduct a ML 3 SCAMPI A by the end of this year, but in all likelihood it would not be successful.
6. The best suggestion I have for you is to hire a CMMI consultant and Lead Appraiser to provide you with the proper advice and guidance. Otherwise, you could be spending a lot more time and effort than originally anticipated.

Difference between measurement objectives and process performance objectives

CMMI uses following terms while talking about measurements:

1. Identified information needs and objectives at organization level (MA)
2. Measurement objectives derived from such information needs and objectives (MA/SP 1.1)
3. Quality and Process Performance Objectives (OPP/SG1)

I want to investigate finer differences between the later two. Are they more or less the same? If not, what are the finer differences?

Your first two questions deal with Measurement and Analysis (MA) which is a Maturity Level 2 Process Area (PA). Your third question is about OPP which is a Maturity Level 4 PA.
MA is a support PA that applies to ALL PAs in the CMMI. Its focus is defining measures and indicators that help answer or address a specific information need. An information need can be articulated using the following format:

<someone> needs to know <what information> in order to make <what decision>.

For example, an issue facing the Project Manager may be that Project Plans are inaccurate causing project teams to routinely miss milestones and deliveries.
Therefore the information need would be: The Project Manager needs to evaluate the schedule, progress, and dependencies of key development activities and events in order to determine necessary corrective actions.
Then the measurement concept or strategy for addressing this information need would be: At the end of each week the Project Manager collects estimates to complete the work for each current task and enters the data into the schedule. The Project Manager and team review progress to determine the necessary corrective actions and modify the Project Plan as needed each Monday morning.
Now it becomes very easy to determine the base and derived measures that support the measurement concept.

So these concepts (information need, measurement concept, and measures) are interrelated concepts that help define what data to collect and analyze to help answer a question.

The Quality and Process Performance Objectives (QPPOs) are entirely different from the measurement concept. The QPPOs are objectives set by the organization based on the organization’s business objectives, past performance of projects, and are constrained by the inherent variability or natural bounds of the process. Now, in order to determine the measures that the organization will use to support the QPPOs, the organization should follow the MA process of articulating the information need, defining the measurement concept, and determining the measures.

Definition of Complexity

While going through some SEI Presentations, I came across a term Design Complexity which was one of the contributing factors for the Delivered Defect Density. As we have chosen this as one of the PPMs in our organization, I would like know how to define the "Design Complexity". I have been searching in Google for the definition, but unable to find it.

I see a problem right from the start in your approach. You have elected to use a term and build a Process Performance Model that has no definition within your organization. Don’t do a literature search to determine what Process Performance Baselines (PPBs) and Process Performance Models (PPMs) to build for your organization. You need to start from the basics. What are your organization’s Quality and Process Performance Objectives (QPPOs)? You need to determine these first. Then you analyze the data in your measurement repository to build PPBs and PPMs that support the QPPOs.

The next step to use Measurement and Analysis (MA) to take the QPPOs, derive the information needs for the QPPOs, and then the measurement specifications that support the information needs and QPPOs. Once you have performed the MA process for your QPPOs, then you will be able to determine if you in fact need to look at Delivered Defect Density and/or Design Complexity. And keep in mind these two terms are just labels. The data you collect and analyze for Delivered Defect Density in one organization may be different from the data for Delivered Defect Density that are collected and analyzed for a different organization. The concepts are important, but going through the MA process will help you define what you need for your organization. Then you can apply whatever label makes sense for you. The MA process will also focus you on what data to analyze to build the PPBs and PPMs that will support your QPPOs.

Obviously Delivered Defect Density and Design Complexity are interesting things to look at, but do they support your organization’s QPPOs? If not, then there is no point in investigating them any further.

Now having said all that, I am surprised that you haven’t been able to locate any information on Design Complexity on the internet. In fact here is an SEI link on this subject http://www.sei.cmu.edu/str/descriptions/cyclomatic_body.html and a Wikipedia definition http://en.wikipedia.org/wiki/Cyclomatic_complexity

Tom McCabe has developed a whole collection of complexity measures, which you find out more about here http://www.mccabe.com/

Bottom line, work through the OPP and MA processes first, THEN look for help with definitions and building PPBs and PPMs. Don’t adopt someone else’s PPBs and PPMs just because these worked for them.

Process Measures

Can someone please suggest what meaures could be taken for the DAR process area for ML3.

Just like any product, project, or process measure, you should be following the Measurement and Analysis (MA) process of defining the information need and measurement specification so you can collect, analyze, and report on measurements that are relevant and have meaning for your organization. After following the MA process you may find that your resulting measurements are the same as other organizations have used, but NOW you know WHY you need to collect, analyze, and report these data. The justification for these data are now driven by your business and projects, NOT simply because you found some examples in a presentation.

If your organization does not have a history of using data to manage processes and/or projects, then you will need a strong position on why you have chosen the specific measures. Most likely you will receive “push back” from people and if the only justification for the measure is from an external presentation, you may find yourself in a losing situation.

For Decision Analysis and Resolution (DAR) process measures, you may want to look at how much effort is spent conducting a DAR, the number of alternative solutions examined, the number of people involved in the DAR, the effectiveness of the decision (was the right decision made after following the process). BUT you must follow the MA process to determine the RIGHT set of process measures for your organization.

The best place to go to for information about MA and process and product measures is the Practical Software & Systems Measurement web site at http://www.psmsc.com/

How Should I Charge Paid and Unpaid Labor Hours?

I'm developing a process to charge labour costs to internal projects, hopefully supporting our CMMI level 3 objective and specifically Project Monitoring and Control.

For CMMI, I understand that all hours must be captured even unpaid.

My question is: what hourly rate is charged/estimated for the unpaid hours to the internal project. In two large (CMMI) companies I have asked, they render the unpaid hours to the internal Project Manager, at a rate of zero; in essence capturing it as an important metric. But this requires the employee to separately enter the distribution of his/her unpaid hours by project on their timesheet.

My accounting colleagues insist that all hours must have an internal $ charge and this would eliminate the need for the employee to enter a separate "unpaid" line on his timesheet. But I'm concerned that this approach will cause Project Managers to discourage employees entering any unpaid hours as it will eat into their Project budgets without any personal gain to the employee.

There is nothing in the CMMI that I am aware of that requires anyone to track time. What is required is that the organization uses Measurement and Analysis (MA) to examine its information needs and measurement objectives to determine what base measures to collect and what derived measures to compute to develop measurement indicators that the decision makers can use to make decisions. Now when you crank through the MA process, the organization usually realizes that it has to collect some kind of effort information in order to make a decision.

Part of the process for determining the measurement indicator is specifying the units on the indicator: hours, dollars, number of defects, etc. So my first question back to you is why do you need an hourly rate for tracking your ML 3 efforts? Is just tracking hours spent insufficient? Many times the hourly rate is competition sensitive, so project managers do not have access to that information, but they do have access to effort. Also, how does using an hourly rate help with estimating new process improvement efforts and projects? I submit to you that what you really should be tracking is effort and not dollars. Then you don’t have to be concerned about unpaid vs. paid labor costs, just time spent on a task. When the data are all rolled up, then you could apply an average labor rate to convert hours to $ if you are interested in cost numbers.

If you don’t track unpaid hours, then you really are not getting the true picture of how much effort it took to do a job. For example, if the organization states that an employee will only get paid for working a 40 hour week regardless of how much overtime, no one tracks how much time is spent in excess of 40 hours. Therefore, it appears that it only takes 40 hours /week to produce what is actually may be taking 60+ hours/week to produce. Then management may dump more work on top of the employee because it looks like they can handle the additional burden. So it becomes difficult to justify adding staff because you do not have a true picture of how much it costs. And you really don’t have any accurate estimates for the project.

What Does GP 2.8 Monitor and Control the Process Mean?

In my experience as a Lead Appraiser I find that most organizations struggle with understanding the intent of this Generic Practice. I believe that part of the confusion with this practice is that people may read the practice title instead of the practice.

"Monitor and control the process against the plan for performing the process and take appropriate corrective action."

The intent of GP 2.8 is not just to monitor and control process, as indicated by the title, but to directly monitor and control the process against the plan for the process (GP 2.2) as you PERFORM the process. Many organizations think that by checking some process measures on a monthly or quarterly basis, after the fact, is meeting the intent of GP 2.8. The frequency of monitoring and controlling the process depends on how often you perform the process. Some process occur weekly and others may be only once or twice during the project's lifecycle. So, once again, the monitoring and controlling of the process must occur as you execute the process. Keeping in mind that reporting the results of the monitoring and controlling effort can happen on a monthly or quarterly basis.
The other misconception about GP 2.8 is that you can read the practice and interpret the words to mean that you can monitor and control the process without using data. That is true if you just read the words of the practice title. However, the practice states that the monitoring and controlling of the process is based on the plan for performing the process. Once you are using a "baseline" for comparison, then the implication is that you have some means for measuring and determining if there is a need for taking appropriate corrective action. In fact, the GP 2.8 discussion in the CMMI refers to the Measurement and Analysis (MA) Process Area for more information about measurements. So the intent of GP 2.8 is to use MA to define one or more appropriate process measures and indicators that will be used to measure the process' performance against its plan, as the process is being executed. The GP 2.8 sub-practices communicate the intent of this Generic Practice. Taking corrective action should not be because an arbitrary threshold was exceeded, but instead the criteria for taking corrective action should be when the process requirements and/or objectives are not being met, process issues have been identified, or when process progress deviates significantly from the plan for the process.