Measurement and Analysis vs. Generic Practice 2.8

Would you expect that measures being reported for Generic Practice (GP) 2.8 on all Maturity Level (ML) 2 and ML 3 Process Areas (PAs) would follow the complete Measurement and Analysis (MA) PA? That is these GP 2.8 measures would have written objectives, operational definitions, storage, collection and analysis procedures in line with MA Specific Goal (SG) 1 and SG 2. Or can the PAs be monitored (measured) and controlled without the full breath of the MA PA?

This is a good question. So let’s take a step backwards and look at the CMMI and Generic Practice – PA Relationships. The summary table in the Generic Practice section of the model clearly states that Project Monitoring and Control (PMC) can implement GP 2.8 for all project-related processes. And MA provides general guidance about measuring, analyzing, and recording information that can be used in establishing measures for monitoring actual performance of the process. Please note that this information is GUIDANCE and part of the INFORMATIVE material. Therefore, it is not required that the org use MA for GP 2.8. HOWEVER, from a practical point of view, why would MA be part of the model if there wasn’t a requirement and expectation that it would be implemented? Since it is a ML 2 PA and you are asking about ML 3, as a Lead Appraiser I would expect to see that MA was used for defining, collecting, analyzing, and reporting both the project and process measures. Without implementing MA for the process measures, the org would be receiving little to no benefit from GP 2.8. And as I have seen GP 2.8 implemented, sometimes the process measures are embedded in the project measures that have been defined using MA.

MA and PPQA Questions

I have the following two basic queries about CMMI ML 2:

1. While writing a Metrics and measurement process, should we address the organization level metrics data consolidation and review. As ML 2 is project specific, is it proper to also document the organization level data consolidation? Also can anyone tell me, the right site for definition of metrics like requirement stability index, schedule variance, effort variance etc.?
   
2. Similarly while documenting PPQA process, is it proper to start with defining an organization level PPQA plan? I am looking for boundaries where to limit writing processes compliant to ML 2. I know that G.P 2.1 to G.P 2.10 must be in place to achieve CMMI ML 2, but the organization specific plans/areas must not be mentioned/documented at CMMI ML 2.

You sound like you are focusing on CMMI compliance rather than on your business goals and objectives. One of the basic tenets of the model is your business objectives. That is where your focus belongs. And if done properly, you will have the side benefit of being CMMI compliant. So, to address your questions:

1. When documenting your Measurement and Analysis process, you should focus on those measures that are important to you. Remember, the first MA practice SP 1.1 states “Establish and maintain measurement objectives that are derived from identified information needs and objectives.” So whatever you have identified as information needs and objectives, that should be your MA focus. At ML 2, for many organizations that are just doing this for the first time, I recommend the org take baby steps and begin with a project focus. But you don’t have to be restricted to the project, an ML 2 org may have also identified some org level measures as well. Go to the Practical Software and Systems Measurement web site for the specific measurement information you need www.psmsc.com
   
2. There are NO CMMI-imposed restrictions on the limits of PPQA. Your organization must define its own limits for the processes you are going to audit. Since GP 2.9 applies to all Process Areas, at a minimum for ML 2, PPQA applies to all of the ML 2 Process Areas you have implemented in your organization. But, if there are other processes that are critical and/or important to the success of your business, then it makes perfect sense to have PPQA audit them as well. Again, do what is right for your business.

Software Productivity Resources

In today's economic scenario, everyone in my organization is talking about "productivity improvement" but no one has a clue how to measure it. Different people percieve productivity in their own way. As per the metrics definition it is "size of work product/ effort spent", while in my experience I have rarely seen organizations measuring size in Function Points (FP) or Source Lines of Code (SLOC) very accurately. Measuring size itself is a distant reality.

Could you share some of your thoughts or ideas or resources on "improving software productivity" or "measuring developers productivity" in simple terms?

The first thing that you have to do is define what productivity improvement means for your organization. As you have stated, everyone has a different definition. So by asking for help you will receive additional definitions to what you have already encountered in your organization, and these answers may be as unsatisfactory as what you have already discussed internally. Instead I recommend that you hold a meeting with your management team and come to consensus on what productivity means for your organization. What does management care about?

What I am suggesting that you do is perform Measurement and Analysis SP 1.1 Establish and maintain measurement objectives that are derived from identified information needs and objectives.

Again, what is your management concerned about? Do they want to reduce time to market? Do they want to reduce total lifecycle cost? Do they want to improve product quality? etc. The answers to these and similar questions will provide you with the information you need to determine what productivity improvement means for your company.

And if you are still struggling with this concept, I suggest that you hire a CMMI consultant with a Measurement and Analysis background who can help you and your organization define the specific measures you need for your company.

Upcoming Events of Interest

This Spring has a number of events of interest to readers of the PPQC Blog.

In March 2009, I will be presenting a paper titled Challenges with Implementing the CMMI in Small Settings at the 2009 SEPG North America Conference in San Jose, CA. http://www.sei.cmu.edu/sepgna/2009/

I will be presenting Session ID 2075 on Thursday morning, March 26, 2009. If you will be attending the Conference, please make plans to attend my presentation.


If you can't make it to San Jose in March, perhaps you will be able to attend the CMMI Made Practical conference in London April 28 - 29, 2009. I will be making a presentation titled A Practical Approach to Defining Useful Measures on April 29. http://www.cmminews.com/


And lastly, if all else fails, I will be teaching a public offering the SEI's Introduction to CMMI class, plus PPQC's Overview of Requirements Management in Stuttgart, Germany May 18 - 21, 2009. If you need to take the Introduction to CMMI class to help you implement the model in your organization or if you will be on an appraisal team, here is your opportunity to take the class in Germany's sixth largest city while enjoying its many hills, valleys, and parks in the evening. You can register for this class at PPQC's German partner KT-BITS' web site.

http://www.kt-bits.net/ If you attend one of these events, please be sure to introduce yourself. I certainly look forward to meeting you at one of these events.

A Question About Measures

Can someone please help me with the following questions:

1. Are there examples of typical or common measurements than an organization can use for every process area? (different examples appear in Generic Practice (GP) 2.8)
2. Where I can look up them?
3. Are there some measurement standards that show or define typical or common measurements? Which?
4. Do they apply to a specific Process Area (PA)? If so, which PA?

Granted there are some common measures such as effort and time spent on an activity. But the underlying principle of the CMMI and more importantly Measurement and Analysis (MA) is to define your goals and objectives and use them to determine your measurement needs. Then you will be able to easily specify those measures that have the most value to your organization and projects. You have to understand why you are collecting a specific piece of data and how you are going to use the information to make a decision, otherwise you will have difficulties and over time people will forget why you are collecting and using the data. They will not see the benefits of these measures. As I have posted to this blog before, start with the MA Process Area and use the practices to define your measures, how you will collect , analyze, report, and use them. MA is based on Practical Software Measurement (PSM). Read the PSM book and visit the PSM web site http://www.psmsc.com/ . The book is very easy to read and understand. It contains several case studies and example measures that will help you define your organization’s measures.

Difference between measurement objectives and process performance objectives

CMMI uses following terms while talking about measurements:

1. Identified information needs and objectives at organization level (MA)
2. Measurement objectives derived from such information needs and objectives (MA/SP 1.1)
3. Quality and Process Performance Objectives (OPP/SG1)

I want to investigate finer differences between the later two. Are they more or less the same? If not, what are the finer differences?

Your first two questions deal with Measurement and Analysis (MA) which is a Maturity Level 2 Process Area (PA). Your third question is about OPP which is a Maturity Level 4 PA.
MA is a support PA that applies to ALL PAs in the CMMI. Its focus is defining measures and indicators that help answer or address a specific information need. An information need can be articulated using the following format:

<someone> needs to know <what information> in order to make <what decision>.

For example, an issue facing the Project Manager may be that Project Plans are inaccurate causing project teams to routinely miss milestones and deliveries.
Therefore the information need would be: The Project Manager needs to evaluate the schedule, progress, and dependencies of key development activities and events in order to determine necessary corrective actions.
Then the measurement concept or strategy for addressing this information need would be: At the end of each week the Project Manager collects estimates to complete the work for each current task and enters the data into the schedule. The Project Manager and team review progress to determine the necessary corrective actions and modify the Project Plan as needed each Monday morning.
Now it becomes very easy to determine the base and derived measures that support the measurement concept.

So these concepts (information need, measurement concept, and measures) are interrelated concepts that help define what data to collect and analyze to help answer a question.

The Quality and Process Performance Objectives (QPPOs) are entirely different from the measurement concept. The QPPOs are objectives set by the organization based on the organization’s business objectives, past performance of projects, and are constrained by the inherent variability or natural bounds of the process. Now, in order to determine the measures that the organization will use to support the QPPOs, the organization should follow the MA process of articulating the information need, defining the measurement concept, and determining the measures.

Peer Review Defect Categories

During peer reviews and testing we find defects and record them in defect log sheet. These defects can be used further for defect data analysis or other purposes (defect prevention). Keeping all these in mind, what defect categories should the organization use for better and easy analysis?

Rather than ask what others are using as defect categories, you should be analyzing your defect data and determine the categories that your defects naturally fall into. These categories also need to have meaning for your organization. If you were to use someone else’s categories and these categories do not relate to your products, projects, or organization, then they will be meaningless. Just like the exercise you should be going through to define your measures (MA), you should also perform a similar exercise to decide on the defect categories. In point of fact, this decision is one of the Measurement and Analysis (MA) steps you should be doing to define your peer review measures. You are really talking about what the CMMI calls derived measures. One set of defect categories MIGHT be process defects, requirements defects, design defects, and interface defects. However, once again, you have to determine which categories support the information needs you should be documenting by following the MA process. Basically, what problem(s) are you trying to solve by performing peer reviews? Once you can clearly answer that question, then you will be able to determine your necessary defect categories.

Satish Kumar provided some addtional information to my response:

Typically IBMs Orthogonal Defect classification is widely used in software projects with minor additions / modifications to defect types. You can get more details about the same from the following link: http://www.research.ibm.com/softeng/ODC/ODC.HTM

The link takes you to IBM's "center for software engineering"site. Browse through the publications to find good information about defect classification.

Note: In your organization's defect types categories, avoid using a type named "Miscellaneous / others". My observation is when we included this type, people had a higher tendency to classify using this category and this is not useful to do any defect analysis.

Thanks & Regards,
Tumu Satish Kumar
SCAMPI Lead Appraiser
CMMI Instructor
Concept QA Labs Pvt. ltd.
www.conceptqalabs.com

Definition of Complexity

While going through some SEI Presentations, I came across a term Design Complexity which was one of the contributing factors for the Delivered Defect Density. As we have chosen this as one of the PPMs in our organization, I would like know how to define the "Design Complexity". I have been searching in Google for the definition, but unable to find it.

I see a problem right from the start in your approach. You have elected to use a term and build a Process Performance Model that has no definition within your organization. Don’t do a literature search to determine what Process Performance Baselines (PPBs) and Process Performance Models (PPMs) to build for your organization. You need to start from the basics. What are your organization’s Quality and Process Performance Objectives (QPPOs)? You need to determine these first. Then you analyze the data in your measurement repository to build PPBs and PPMs that support the QPPOs.

The next step to use Measurement and Analysis (MA) to take the QPPOs, derive the information needs for the QPPOs, and then the measurement specifications that support the information needs and QPPOs. Once you have performed the MA process for your QPPOs, then you will be able to determine if you in fact need to look at Delivered Defect Density and/or Design Complexity. And keep in mind these two terms are just labels. The data you collect and analyze for Delivered Defect Density in one organization may be different from the data for Delivered Defect Density that are collected and analyzed for a different organization. The concepts are important, but going through the MA process will help you define what you need for your organization. Then you can apply whatever label makes sense for you. The MA process will also focus you on what data to analyze to build the PPBs and PPMs that will support your QPPOs.

Obviously Delivered Defect Density and Design Complexity are interesting things to look at, but do they support your organization’s QPPOs? If not, then there is no point in investigating them any further.

Now having said all that, I am surprised that you haven’t been able to locate any information on Design Complexity on the internet. In fact here is an SEI link on this subject http://www.sei.cmu.edu/str/descriptions/cyclomatic_body.html and a Wikipedia definition http://en.wikipedia.org/wiki/Cyclomatic_complexity

Tom McCabe has developed a whole collection of complexity measures, which you find out more about here http://www.mccabe.com/

Bottom line, work through the OPP and MA processes first, THEN look for help with definitions and building PPBs and PPMs. Don’t adopt someone else’s PPBs and PPMs just because these worked for them.

Process Measures

Can someone please suggest what meaures could be taken for the DAR process area for ML3.

Just like any product, project, or process measure, you should be following the Measurement and Analysis (MA) process of defining the information need and measurement specification so you can collect, analyze, and report on measurements that are relevant and have meaning for your organization. After following the MA process you may find that your resulting measurements are the same as other organizations have used, but NOW you know WHY you need to collect, analyze, and report these data. The justification for these data are now driven by your business and projects, NOT simply because you found some examples in a presentation.

If your organization does not have a history of using data to manage processes and/or projects, then you will need a strong position on why you have chosen the specific measures. Most likely you will receive “push back” from people and if the only justification for the measure is from an external presentation, you may find yourself in a losing situation.

For Decision Analysis and Resolution (DAR) process measures, you may want to look at how much effort is spent conducting a DAR, the number of alternative solutions examined, the number of people involved in the DAR, the effectiveness of the decision (was the right decision made after following the process). BUT you must follow the MA process to determine the RIGHT set of process measures for your organization.

The best place to go to for information about MA and process and product measures is the Practical Software & Systems Measurement web site at http://www.psmsc.com/

How Should I Charge Paid and Unpaid Labor Hours?

I'm developing a process to charge labour costs to internal projects, hopefully supporting our CMMI level 3 objective and specifically Project Monitoring and Control.

For CMMI, I understand that all hours must be captured even unpaid.

My question is: what hourly rate is charged/estimated for the unpaid hours to the internal project. In two large (CMMI) companies I have asked, they render the unpaid hours to the internal Project Manager, at a rate of zero; in essence capturing it as an important metric. But this requires the employee to separately enter the distribution of his/her unpaid hours by project on their timesheet.

My accounting colleagues insist that all hours must have an internal $ charge and this would eliminate the need for the employee to enter a separate "unpaid" line on his timesheet. But I'm concerned that this approach will cause Project Managers to discourage employees entering any unpaid hours as it will eat into their Project budgets without any personal gain to the employee.

There is nothing in the CMMI that I am aware of that requires anyone to track time. What is required is that the organization uses Measurement and Analysis (MA) to examine its information needs and measurement objectives to determine what base measures to collect and what derived measures to compute to develop measurement indicators that the decision makers can use to make decisions. Now when you crank through the MA process, the organization usually realizes that it has to collect some kind of effort information in order to make a decision.

Part of the process for determining the measurement indicator is specifying the units on the indicator: hours, dollars, number of defects, etc. So my first question back to you is why do you need an hourly rate for tracking your ML 3 efforts? Is just tracking hours spent insufficient? Many times the hourly rate is competition sensitive, so project managers do not have access to that information, but they do have access to effort. Also, how does using an hourly rate help with estimating new process improvement efforts and projects? I submit to you that what you really should be tracking is effort and not dollars. Then you don’t have to be concerned about unpaid vs. paid labor costs, just time spent on a task. When the data are all rolled up, then you could apply an average labor rate to convert hours to $ if you are interested in cost numbers.

If you don’t track unpaid hours, then you really are not getting the true picture of how much effort it took to do a job. For example, if the organization states that an employee will only get paid for working a 40 hour week regardless of how much overtime, no one tracks how much time is spent in excess of 40 hours. Therefore, it appears that it only takes 40 hours /week to produce what is actually may be taking 60+ hours/week to produce. Then management may dump more work on top of the employee because it looks like they can handle the additional burden. So it becomes difficult to justify adding staff because you do not have a true picture of how much it costs. And you really don’t have any accurate estimates for the project.

ML 2/ML 3 Measurement Objectives

For a Maturity Level 3 organization , is it acceptable if the organization sets its process performance objectives qualitatively? For example reduce testing cycle time , improve productivity, etc. and in OPF SG1, one of sub-practices states that process performance objectives may be expressed either quantitatively or qualitatively. Also would you please explain this clarification linking to ML2- MA - SP 1.1 - Establish measurement objective?

These are good questions, but the best way to receive a decent understanding of the concepts it to attend the SEI’s Introduction to the CMMI class. Intro to CMMI course description Brief answers, such as mine in this blog, will probably only raise more questions that would be better addressed in a classroom setting.

1. Only for High Maturity organizations (ML 4 and 5) is there an expectation for quantitatively setting process performance objectives. It is acceptable at ML 3 for qualitative objectives. But to fully understand this distinction you really require a good discussion on the expectations for each Maturity Level, which you would receive in the Intro to CMMI class.
   
2. A measurement objective is different from a process performance objective. The measurement objectives are the purposes for which measurement and analysis is performed and also specify what kinds of actions may be taken as a result of data analyses. In order to determine the measurement objectives, you first have to describe the information needs. An information need is an insight necessary to manage objectives, goals, issues, risks, and problems. The measurement objective is derived from the information need and it is a statement about what should be measured in order to satisfy the information need. For example, the project manager or other decision maker concerned with allocating budget and associated resources to a task may believe that productivity is related to the type of task being performed. Increasing productivity is then the measurement objective that addresses the defined information need. Determining productivity then requires that entities such as the product and the process be measured. There are many ways that productivity can be computed, but the measurement objective is unaffected by the different methods.

To fully understand these concepts you should take a class on Measurement and Analysis or Practical Software Measurement. Trying to explain these concepts in a paragraph or two does not do the subject justice.

What Does GP 2.8 Monitor and Control the Process Mean?

In my experience as a Lead Appraiser I find that most organizations struggle with understanding the intent of this Generic Practice. I believe that part of the confusion with this practice is that people may read the practice title instead of the practice.

"Monitor and control the process against the plan for performing the process and take appropriate corrective action."

The intent of GP 2.8 is not just to monitor and control process, as indicated by the title, but to directly monitor and control the process against the plan for the process (GP 2.2) as you PERFORM the process. Many organizations think that by checking some process measures on a monthly or quarterly basis, after the fact, is meeting the intent of GP 2.8. The frequency of monitoring and controlling the process depends on how often you perform the process. Some process occur weekly and others may be only once or twice during the project's lifecycle. So, once again, the monitoring and controlling of the process must occur as you execute the process. Keeping in mind that reporting the results of the monitoring and controlling effort can happen on a monthly or quarterly basis.
The other misconception about GP 2.8 is that you can read the practice and interpret the words to mean that you can monitor and control the process without using data. That is true if you just read the words of the practice title. However, the practice states that the monitoring and controlling of the process is based on the plan for performing the process. Once you are using a "baseline" for comparison, then the implication is that you have some means for measuring and determining if there is a need for taking appropriate corrective action. In fact, the GP 2.8 discussion in the CMMI refers to the Measurement and Analysis (MA) Process Area for more information about measurements. So the intent of GP 2.8 is to use MA to define one or more appropriate process measures and indicators that will be used to measure the process' performance against its plan, as the process is being executed. The GP 2.8 sub-practices communicate the intent of this Generic Practice. Taking corrective action should not be because an arbitrary threshold was exceeded, but instead the criteria for taking corrective action should be when the process requirements and/or objectives are not being met, process issues have been identified, or when process progress deviates significantly from the plan for the process.

2008 SEPG Conference - Measurement and Analysis Tutorials

The 2008 SEPG Conference kicked off today in Tampa, Florida. http://www.sei.cmu.edu/sepg/2008/program.html I was the session moderator for the Measurement and Analysis tutorials: Satisfying Organizational Needs With A Concise Measurment Program and Metrics Based Process Improvement. Both tutorials emphasized the need to tie measures to specific business objectives, to keep the measurement program simple and "parsimonious", and to avoid identifying, collecting, and analyzing measures for every single Process Area in the CMMI. Your measurement program should be built around how you conduct business, not a specific process improvement model. Start small with the easy measures first that people and managers care about. People have to be motivated to use the measures. So you have to be aware of "what is it it for me (WIIFM)" when considering other's motivations. Gain some experience and knowledge from this initial set of measures and then add, modify, and delete measures to grow your measurement program. It all sounds simple and is a common sense approach. Though in my experience, organizations still need some coaching and mentoring in order to correctly focus their attention on specifying meaningful measurements.