Appraisals: Practice or Subpractice level?

For successful SCAMPI appraisals, is there any reason to prepare process compliance at the sub-practice level? Would appraisers be looking for evidence at that level?

This is a question answered by taking the 3-day Introduction to CMMI class and also by your Lead Appraiser. There are three CMMI components: Required, Expected, and Informative. An appraisal only covers the Required (Goals) and Expected (Practices) components. Your Lead Appraiser should also be providing some training or guidance on how to build the PIIDs, which contain the objective evidence for an appraisal. And the whole appraisal team is involved in reviewing the PIIDs during the Readiness Review to determine if the evidence is proper for a SCAMPI appraisal.

If a Lead Appraiser or the appraisal team is appraising you to the sub-practice level, they have gone too far. The SCAMPI method is only concerned with appraising the organization to the Goals and Practices.

How Do We Select a High Maturity Consultant?

My friend is a Quality Manager in a company who has reached CMMI Maturity Level 3. They now want to achieve Maturity Level 5. They started taking quotes from different companies. In the selection process they found that there are 3 or 4 major players in our country who have up to 3 High Maturity Lead Appraisers. Most of these companies have submitted proposals for consulting and appraisal in a single quote. Now my friend fears that:

1. Most of them already have at least 40 High Maturity clients and at least 30 Maturity Level 3 clients. Will they have the capacity do lead the appraisal on time for my friend's company ( considering 12 SCAMPI appraisals per year per LA) ?
2. Most of them deliver the consulting and training activities , which is 70 % of the contract value and sometimes they break the contracts and not deliver the SCAMPI, which is still highly profitable, since only 30 % value is lost, and no need to deal with High Maturity appraisal needs.

In order to address this issue, does the SEI publish a list of contracts for SEI Partner? Does the SEI have a specific committee or group to look in to the capacity management and availability management of their SEI partners, so that companies will not have such concerns?

All that the SEI does is maintain a list of SEI Partners and certified High Maturity Lead Appraisers. If there is indeed a problem as you have stated, then you or your friend should contact the SEI about the SEI Partner in question as this certainly sounds like unethical behavior.

Another issue is that an SEI-certified High Maturity Lead Appraiser cannot appraise the organization if he or she has provided the consulting to the organization, unless the SEI approves the potential Conflict of Interest.

The steps taken by SEI in this area are impressive. Also I understand that the control on appraisals/per year is established by SEI. Out of curiosity i would like to ask follow up questions.

I belive the critical part of this entire process is that consulting and apprisal services cannot be performed by the same Lead appriser. If there is a need of separate contract for SCAMPI A appraisal activities, and it cannot be included in a consulting contract, then it can have more credibility. Also like ISO where the certification agencies are audited annually (correct me if am wrong), will SEI do an onsite audit on SEI partners? Or do they have a databse of all the contracts established by SEI partners around the world (considering 800 to 1000 appraisals per year)? Because the user community trusts the SEI more than the SEI Partners ( for most of the users it may be the first time to contract with an SEI Partner and they might not be sure of the guidelines provided by the SEI or about the SEI Ethics commitee). All this can be prevented if the SEI takes a copy of all contracts established for SCAMPI A across the countries. What are your views on it? .

Note:The intent of the question is to increase the user communities' trust on SEI to increase, but not to reduce the credibility of SEI partners/Lead Appraisers.

The SEI does not have the time or resources to perform annual on-site audits of the SEI Partners. And as an SEI Partner, I would not welcome an on-site audit by the SEI. It would be additional expense for me.

What the SEI does provide that may help with your concerns is that they perform a QA audit of the results of every appraisal submitted by a Lead Appraiser. If the appraisal results do not meet the evaluation criteria, then a more in-depth audit occurs. What can then happen is that if the problems are serious enough, the Lead Appraiser can lose his or her CMMI credentials. This has happened to a number of Lead Appraisers since this policy was put in place.

In addition, each Lead Appraiser must be certified by the SEI, which provides another layer of credibility.

The SEI Partners provide the service and the certified-Lead Appraisers deliver the service. As a buyer of CMMI services, it is your responsibility to learn about the SEI policies regarding Ethics and Conflict of Interest, as well as the credibility of the different SEI Partners and Lead Appraisers. Otherwise, you get what you pay for. In other words, buyer beware!

And depending upon what country you are in, the SEI Partners are trusted as much or more than the SEI by the user community.

Achieving a Maturity Level Without a Consultant or Training

I was wondering if it is possible to go it alone with CMMI Maturity Level 2. I have been told by many that attempting CMMI Maturity Level 2 without a consultant or highly trained staff would be somewhat challenging. What are your thoughts?

Also, having limited resources for process improvement being a strong possibility, do you have any recommendations for online sources that can help offset the costs?

Though it is entirely possible to implement the CMMI without hiring a CMMI consultant, that is a high risk approach. Though I don’t see how you could avoid taking the Introduction to CMMI training class. That, in my opinion, would be a huge mistake. There are some areas in the CMMI that are open to interpretation when trying to go it alone and you can end up doing things in the spirit of achieving Maturity Level 2 that have no business value to you. In addition, since your organization has to provide 4 to 8 appraisal team members, each appraisal team member must take the SEI’s Introduction to CMMI class.

These are all worthwhile expenditures of your process improvement budget. And when you compare these expenses to your internal costs for process improvement, these are usually negligible in comparison.

But if you are trying to do things as cheaply as possible, that begs the question, why are you even considering implementing the CMMI and trying to achieve Maturity Level 2? Basically you get what you pay for. Going the cheap route doesn’t demonstrate management commitment to process improvement and can result in wasted effort, wasted money, and an aborted process improvement initiative.

Bi-directional Traceability

Our organization is in the process of preparing for a CMMI Maturity Level 2 SCAMPI A appraisal. We are concerned about our approach for bi-directional traceability REQM SP 1.4. We maintain traceability is follows:

1. High Level Requirements <--> Use Cases (Includes GUIs and Database Interactions) <--> Test Cases

2. Use Cases <--> Source Code

Note: One can trace from Test Cases to Source Code through the Use Cases and Vice Versa, but the traceability is not direct. The reason behind this is, test cases are generated from use cases and are tested against the application (black box testing). Source code does not have associated test cases.

Is this kind of traceability considered bi-directional and is satisfactory for Maturity Level 2?

What you describe is one of many ways to implement bi-directional traceability and meet the intent of the CMMI. If your method supports your business goals and objectives and there are no quality issues, your approach should be acceptable for a Maturity Level 2 appraisal.

It is interesting that you think tracing from Test Cases to Source Code via Use Cases may not be acceptable. Traceability is a multi-dimensional mapping that can have one-to-many and many-to-one relationships. As long as you trace from the top all the way to the bottom and vice versa, you should be fine no matter how many links there are in the chain, and the chain can have branches as well.

Please note that bi-directional traceability does not mean tracing one whole document to another whole document. What it means is that a given item in one document (a specific requirement for example) can trace to multiple items in another document, multiple items in one document can trace to one item in another document, one item can trace to one item, etc.

Some Appraisal Questions

1. Why does the SEI ask for focus projects instead of all the projects done by the company?
2. Usually companies can select consultants and the Lead Appraiser (LA). Why is the SEI giving the right to chose the LA by the company or consultant?
3. Why do the appraisal results expire after 3 years?
4. Why doesn't the SEI have compliance appraisals every 6 months or 1 year similar to ISO?

1. The SCAMPI method is a sampling method to determine the degree of institutionalization of the processes on the projects. Therefore the use of focus projects. For organizations where there are only 1 or 2 projects, then all of the projects are usually included in the appraisal scope. But for organizations with many projects, it would be prohibitive to evaluate all of the projects. That is why it is the responsibility of the Lead Appraiser to select the focus projects, along with input from the organization. The principle here is that if the processes are truly institutionalized throughout the organization, then it doesn’t matter which projects are selected for the appraisal. Any set of selected projects should be representative of how all projects in the organization behave.
   
   
2. If I understand your statement, you are incorrect. The organization does select the CMMI consultant and Lead Appraiser. However, only SEI-certified Lead Appraisers are allowed to lead and report SCAMPI appraisal results. If a Lead Appraiser is NOT SEI-certified and he or she leads a SCAMPI appraisal, then the appraisal results are NOT valid.
   
   
3. The appraisal results expire after three years because in the past many organizations tended to backslide in their process maturity after having their appraisal. The three year period is long enough to address the findings from the SCAMPI A appraisal and prepare for a re-appraisal at the same or higher Maturity Level. If there is no expiration date, then there could be less motivation to continue with Process Improvement.
   
   
4. In addition, there is no such thing as a compliance appraisal at this time. There has been some discussion along these lines, but nothing has been settled. There is a fundamental difference between ISO audits and CMMI appraisals. ISO is a standard and the result of the audit is certification. CMMI is a set of guidelines for process improvement and the result is Maturity Level or Capability Level that is valid for three years.

Is Going Directly for a CMMI ML 5 Appraisal Allowed?

Is a CMMI v1.2 ML 5 appraisal allowed in the following situation?

One of my company's divisions was successfully appraised to CMMI v1.1 ML 5 but the appraisal results expired in 2009. Now my company wants all three divisions, which are located in different cities, appraised to ML 5. Though skipping levels is not recommended, however, is it allowed to go for the appraisal?

There is nothing from the SEI that prevents a company from being appraised to whatever Capability Level or Maturity Level that it chooses. That being said, what does drive the CL or ML is the organization being able to collect, analyze, and correctly use data for statistical and quantitative analysis, as well as being able to institutionalize the behaviors and practices. You should hire an SEI-Certified High Maturity Lead Appraiser and have him or her perform a Class C or Class B appraisal to determine the risks with your current approach and implementation of ML 5. The outcome of this exercise will determine if it is feasible to achieve ML 5 at all three sites.

How Do I Become A Lead Appraiser?

Would you please tell me step by step the procedure to become a SCAMPI Lead Appraiser (LA)? I am currently working with a Maturity Level 5, company in India with 5.5 years of relevant experience in Process and Quality Consulting. It would be really great if you could provide me with some references, emails, and sites as a roadmap.

Here are the steps and requirements for becoming a SCAMPI Lead Appraiser. This information is directly from the SEI’s web site.

To become an instructor or Lead Appraiser, you must successfully complete authentic SEI courses. The first prerequisite course for becoming either an instructor or Lead Appraiser is the Introduction to CMMI Version 1.2 course. This courses is available from SEI Partners (see SEI Partner Network Directory and Guide to Services) or from the SEI. The second prerequisite is the Intermediate Concepts of CMMI Version 1.2 course that is available only from the SEI.

CMMI Version 1.2 Instructor Training is the final course leading to candidacy as an Introduction to CMMI Instructor. After successfully completing this course, the candidate instructor must also be observed, by an authorized SEI Observer, teaching the SEI's Introduction to CMMI V1.2 course. Upon successful completion of the observation, the instructor is then authorized as an Introduction to CMMI Instructor.

SCAMPI Lead Appraiser Training is the final course leading to candidacy as a SCAMPI Lead Appraiser. After successfully completing this course, the candidate Lead Appraiser must be observed leading a SCAMPI A appraisal using a CMMI model and be approved by an authorized SEI Observer.

You can find more information on the SEI’s site by searching for SCAMPI Lead Appraiser. Of course you will have to either upgrade to v1.3 or take the v1.3 classes when the new versions are released later this year.

Query on CMMI for Development v1.3

Our organization was appraised at CMMI ML3 in Oct, 2009 and we'll be going for ML5 in Q1 2011. Although we have been preparing ourselves for CMMI for Development v1.2, we are also aware that the CMMI for Development v1.3 will be out in January, 2011. We wish to get appraised for CMMI for Development v1.3. I have gone through some of the PPTs and PDFs on web but none of them give a clear insight into the changed expectations from the existing PAs or expectations from new PAs to be added.

It'll be a great help if anyone can provide me the draft version of v1.3 or any detailed document about the same.

One item that you may not be aware of is that the SEI would like to see at least 18 months between an ML 3 appraisal and a High Maturity appraisal. If you are planning for an appraisal in Q1 2011, that would be less than 18 months. Therefore you would have a very high probability of your appraisal results being audited by the SEI, which could take a long time before being accepted. I would encourage you to hire a High Maturity Lead Appraiser as soon as possible, if you haven’t already done so, and move your plans for your ML 5 appraisal at least 3 months or more into the future to ensure that you have enough data for performing the High Maturity practices and enough time for institutionalization of HM.

You will have to wait for the v1.3 release in November 2010. The problem with using drafts is that things can change before the release.

REQM and RD in the CMMI

Why is REQM Management at Maturity Level 2 and Requirement Development at Maturity Level 3? We develop the requirements first and then manage them in the project.

There reason for the placement is due to the meaning of ML 2 vs. ML 3. ML 2 is all about stabilizing projects and gaining control over project estimates. Once the organization has achieved this, then it can begin to evaluate how to improve the engineering areas.

Since you need to have a baseline upon which to plan a project and the other ML 2 Process Areas, that is why REQM is the first Process Area in ML 2. The intent is to manage the collection of project requirements: good, bad, or indifferent. And use this collection to plan the project, etc. Then when you have achieved ML 2 and move to ML 3, then you can address how to improve the Requirements Elicitation to obtain better requirements.

Please keep in mind that the CMMI is a collection of guidelines and best practices for doing process improvement. The CMMI is not a roadmap for how to do software engineering.

Implementing CMMI Along With ISO 9001

Suppose a company already has an ISO 9001 certification. Then they decide to achieve an appraisal at level 3. Suppose further that their plan is to add additional process assets according to their gap analysis. But then they discover that certain ISO 9001 work instructions or templates require changes to meet the requirements of CMMI level 3. If these assets are changed, would that necessarily invalidate the ISO 9001 certification? Under what conditions would the certification be invalidated, and what needs to be done under those circumstances? Is there a way to avoid this issue?

Without having any details as to your situation, I find it hard to believe that if you are ISO 9001 certified that the CMMI is causing you to make changes to your quality system that would invalidate your ISO 9001 certification. The ISO 9001 standard and the CMMI-DEV model are compatible. Are you working with an SEI-certified Lead Appraiser/consultant? If not, you may be making some decisions to change processes and process assets that are not necessary.

To properly address your concerns, you should have a Lead Appraiser conduct a gap analysis of your organization to determine what you currently have in place that is compatible with the CMMI and identify those gaps that need to be addressed in order for your organization to achieve Maturity Level 3. And any updates to existing processes and process assets should be compatible with both the ISO standard and the CMMI-DEV.

Project Planning SP 1.2 - Task Attribute: Effort or Size?

Is it possible to establish estimates of work product and task attributes by means of task time estimates? Can the task effort be similar to the size of a task?

By going directly to task time estimates you have effectively skipped performing Project Planning Specific Practice 1.2. The intent of SP 1.2 is for you to perform some sort of basis of estimate for the project’s tasks and activities. This is a bottom-up approach. If you are not used to this approach, it can be a struggle at first to take a step backwards from task time estimates and really understand the underlying assumptions that people are making in their heads about the factors that are driving the task time estimates. Some very basic task attributes include estimating the number of pages in a document that is being produced or updated, the number of technical drawings being produced or updated for a hardware item, the number of new or modified interfaces, the number of new or modified screens , etc. Then based on your historical data from previous projects, it is possible for you to empirically determine a set of productivity factors that will convert these sizing parameters into effort and arrive at the task time estimates. The bottom line is effort of a task is not the same as the size of a task.

Do you think this practice would be Fully, Largely, Partially or Not Implemented? Would this be this a problem in a SCAMPI A appraisal? What do you think about that?

Taking this example out of context with everything else your organization is doing makes this a difficult question to answer. The appraisal team is the only group that would be qualified to make that judgment based on documented evidence and the interviews. However, as a Lead Appraiser, I would have to say that you have a problem that needs to be addressed before you conduct a SCAMPI. The SCAMPI rules state that if a Process Area is in the scope of the appraisal, then all of its Specific and Generic practices are applicable. And if you are not performing a practice, which may or may not be the case, then there could be issues in Project Planning that impact Goal Satisfaction and result in a Maturity Level 1 rating.

To provide you the best answer, you should be talking to your Lead Appraiser and have him or her give you the proper guidance on this issue. As a risk mitigation, I would recommend that you put a process in place to estimate sizing parameters that are then used to calculate effort. Your estimators are already doing this, but it sounds like they are doing it in their heads. You just have to break the process down into smaller steps to allow the sizing estimates to be captured first. There is benefit to doing this.

Appraisal Sponsor Roles and Responsibilities

I am looking for information regarding the roles and responsibilities of the appraisal sponsor.

My organization is preparing for an external appraisal and I'd like to provide the sponsor with a summary of her role and what she will be expected to do for the appraisal such as review and sign the appraisal plan, attending the opening and findings briefings etc.

It sounds like you have a pretty good understanding already of the appraisal sponsor’s duties. It would be helpful to you to ask this question of your Lead Appraiser to see if he or she has any specific requests for the appraisal sponsor. As a Lead Appraiser, these are my expectations. The appraisal sponsor:

1. Provides the funding and senior management commitment for process improvement and the appraisal
2. Does not serve on the appraisal team
3. Meets at least once with the Lead Appraiser to discuss the appraisal
4. Provides the business objectives for the organization
5. Signs the Appraisal Input Document, Appraisal Plan, and Appraisal Disclosure Statement
6. Attends the Opening Briefing to reinforce why the appraisal is being conducted and the importance of everyone to support the appraisal team
7. Meet with the appraisal team before the Final Findings presentation to privately receive the results and prepare the proper message for the organization at the Final Findings Presentation
8. Attend the Final Findings Presentation and at the conclusion of the presentation thank the appraisal team for their efforts and thank the organization for their efforts regardless of the outcome. If the results were bad news, turn it into a positive statement of encouragement, etc.
9. Complete the feedback form in SAS for the appraisal results
10. Receive the appraisal results and maintain the appraisal record for three years
11. Do not publicly disclose the appraisal results until the SEI has completed its quality review and announced the results to the Lead Appraiser and appraisal sponsor

How to Start With the CMMI

If a private software company wants to do get a certificate of quality for CMMI Level 2, what must this company do in order to obtain Level 2? And also what are the steps that must be followed from the start (initial) to achieve CMMI Level 2 (Managed)?

First let me state that there is no certification of any type for the CMMI. Individual Lead Appraisers will issue something looks like a certificate to the organization that was appraised indicating the SCAMPI A results. But this is not a certification. The only things that the SEI certifies are the CMMI instructors and Lead Appraisers.

Here are the necessary steps for achieving Maturity Level 2 for the CMMI-DEV. These steps are not necessarily sequential, some can occur concurrently.

1. Hire an SEI-certified Lead Appraiser to conduct a Gap Analysis of the company to determine the current process strengths and weaknesses and help the company construct a Process Improvement Plan (PIP).
2. Obtain executive management sponsorship for the process improvement effort.
3. Train the people responsible for the company’s processes and for addressing the action items in the PIP on the 3-day SEI Introduction to CMMI v1.2 class.
4. Address the issues from the Gap Analysis, document the necessary processes and procedures, and begin conducting the PPQA process and work product audits.
5. Allow time for the new and/or modified processes to get some use on various projects.
6. Conduct a SCAMPI B appraisal as a dress rehearsal for the SCAMPI A. Identify any issues and weaknesses that are potential risks to achieving Maturity Level 2.
7. Create a new PIP to address the SCAMPI B identified weaknesses and risks.
8. Address these issues
9. Conduct the Maturity Level 2 SCAMPI A appraisal.

Using the CMMI-SVC to Transform an Organization into a High-Functioning, Customer-Driven Profit Center

For those of you who were not able to attend the 2010 North American SEPG Conference in Savannah, GA in March, here is a copy of my presentation on the CMMI-SVC.

As a company grows and matures from a startup entrepreneurial venture to a sustainable corporation, the departments and company services that begin as good ideas expand and evolve to support the company’s growing business. Many times these services simply develop without any strategic vision resulting in institutionalized behaviors that are incompatible with the company’s business goals and objectives. Consequently, the transition to a larger corporation becomes a challenge. A notable example is a company’s Engineering Services Department.

When people think of Engineering Services, the Customer Support or Help Desk team is what first comes to mind. However, other services such as Product Training, Field Services (product installation and troubleshooting), and Engineering Sales Support may be provided as well.

As a product development company begins selling product, the Customer Support function becomes one of its first service offerings whether or not it recognizes it as such. In addition, it is natural for the focus of the Customer Support function to be on pleasing their customer base, as many sales are contingent upon repeat business and word of mouth until the company and its product line become established in the marketplace. Nevertheless, without a clear idea of its charter and strategic direction to support business growth and identify new markets and service offerings, the Customer Support Specialists focus instead on supporting their customer base on non-company and non-product issues and questions that consume internal resources without any tangible benefit to the company. Once a company starts banging its head on the “glass ceiling” as it attempts to grow, the leadership may recognize that its current Engineering Services approach does not support its strategic business goals and objectives.

In these circumstances, the company is not necessarily interested in implementing the CMMI for Services (CMMI-SVC) and becoming appraised to either Maturity Level 2 or Maturity Level 3. However, by using the Continuous Representation, the CMMI-SVC can provide the needed guidance to help a company restructure and reorganize its Engineering Services approach in order to become a profit center or revenue generating function.

In this presentation, we will present a case study for OMNI Flow Computers, Inc., a company that specializes in the design, development, and manufacture of panel-mount multi-run, multi-tasking liquid and gas flow computers, and field-mount, hazardous area controllers/RTUs for liquid and gas custody transfer metering systems. The challenge facing OMNI was to develop its Engineering Services Department into a high-functioning, customer-driven profit center. OMNI’s Engineering Services Department consists of three groups: Customer Support, Training, and Engineering Field Services. Customer Support handles customer questions, concerns, and issues. The Training group provides training on the OMNI product line to its customers and users. Engineering Field Services provides on-site troubleshooting services on an as-needed basis.

As the Training and Engineering Field Services groups were recent additional capabilities, Customer Support presented the biggest obstacle to overcome. Noted management consultant Peter Drucker declared several years ago that Quality in a service or product is not what you put into it. It is what the client or customer gets out of it. Moreover, an obstacle to achieving this objective was one of the core challenges faced by the department: developing an appropriate customer focus and developing new service offerings. A major reason for these challenges is the nature of OMNI products. OMNI's customers integrate their products into custody transfer systems that involve a wide variety of large-scale hardware and electronic equipment from other manufacturers. OMNI’s customers usually develop and commission these systems for their clients and end users. Therefore, when calls come in to OMNI’s Customer Support group, the first challenge they had to overcome was determining if the customer’s issue was actually an OMNI product issue or the result of an external issue. The next challenge was to determine the root cause of the issue, so that the customer would receive a timely resolution of their issue.

Fortunately, the release of the CMMI-SVC came at the right time for OMNI. Of the seven new services Process Areas (PAs), many of the Specific Practices and associated informative material proved useful in guiding the transformation of OMNI’s Engineering Services Department. The Service Delivery PA provided excellent guidance for establishing and documenting Engineering Services’ existing service offerings, ensuring that each group was prepared to deliver the defined service offerings, and delivering the service offerings. The Incident Resolution and Prevention PA provided excellent guidance for identifying, documenting, tracking, reporting, and resolving customer complaints, issues, and other service interruptions. The Service Continuity PA helped focus the Engineering Services Department manager to identify and prioritize the department’s essential functions and necessary resources. The Strategic Service Management PA brought the needed focus to establish the Engineering Services strategic needs and plans for its standard services.

The OMNI Engineering Services Department’s journey is not yet over. They are still growing, maturing, and learning what it means to become a high-functioning customer-driven profit center. However, along the way they learned some valuable lessons. This presentation will discuss some of the pitfalls they encountered, what strategies worked and what did not work, as well as provide some practical advice to aid other organizations facing similar challenges.

If you make customers unhappy in the physical world, they might each tell six friends. If you make customers unhappy on the Internet, they can each tell 6000 friends. - Jeff Bezos

Customer service is not a department. It is an attitude. – Unknown

This presentation provides a case study of a computer manufacturer that used the CMMI for Services to help transform its Engineering Services department (Customer Support, Training, and Engineering Field Services groups) into a high-functioning, customer-driven profit center. Challenges, successful approaches, lessons learned, and practical advice to aid other organizations facing similar challenges will be presented.

Using the CMMI-SVC to Transform an Organization into a High-Functioning, Customer-Driven Profit Center

View more presentations from Henry Schneider.

Process Improvement vs. Process Maintenance

An organization on its process improvement journey may include process maintenance activities under the heading of process improvement. Most organizations do that. As far as I know, CMMI does not talk of process maintenance activities. To me, a few examples of process maintenance activities are those required for metrics data collection, project monitoring, process definition, making plans, etc. On the other hand, process improvement will include activities like metrics data analysis, identifying significant variations from planned arrangement, process re-definition based on improvement suggestions and metrics analysis, revising the plans in line to changing requirements, and taking appropriate preventive and/or corrective actions.

Are there any benchmarks for healthy process maintenance vs. improvement activities for software organizations?

These two concepts in inextricably intertwined. I would find it hard to believe that an organization does not identify any process improvement suggestions by just doing process maintenance. Even if you are just maintaining your Maturity Level, you will still identify improvements to the status quo. And I contend that even for process maintenance you have to perform data analysis, identify variations, and identify appropriate corrective actions, otherwise how would you be able to determine that you are maintaining what you have already achieved? I do not understand the need to define these two items as separate activities. What would be the point of merely maintaining your processes?

Configuration Management - Change Request Number Traceability

I would like your help with understanding the following subject:

A change request is processed in a change control tool. Each change request has a number. If a proposed change is accepted, a schedule is created for making the change.

When it is necessary to store the configuration items modified in the configuration system, we have the option of describing what we have done. We record the implemented changes and the reasons for the changes, however we don't track the change request number.

Will the lack of tracing the change request number create a weakness in the Configuration Management Process Area? Is just recording the historical information about the changes in the comments and using the change control tool sufficient?

How important to your projects and business is it to track the change request number? Have you encountered issues with change control on the projects because you have not maintained this information? Recording and tracking the change request number is a very easy thing to do and I am wondering why you haven’t done that.

What also concerns me is that you imply that recording a description of the change made to a baselined item may be optional. If this is true, then it is then possible for someone to decide not to record this information. Recording the information should be mandatory.

Not tracing the change request number and the possibility of not recording a description of the change sound like high risk items that can cause you difficulties in the future. As a risk mitigation, I would record all change numbers and require that all changes made are clearly described in the event that someone else in the future needs to understand the nature and reasons for the change.

And since you are using a tool, which I assume is an off-the-shelf product, you may in fact be tracing the change request number and not even realize it. But from an appraisal point of view, it sounds like you might have most of Configuration Management covered and the problem you are asking about might be viewed by the appraisal team as a weakness. But is it a Goal breaking weakness? This a question that must be answered by the appraisal team.

Configuration Management - SP 1.3-1 , SP 2.2.-1

I would like your opinion in relation to situation below.

Once a change was been verbally approved, is there some problem in relation to CM SP 1.3 (Create or Release Baselines) - sub-practice 1 and CM SP 2.2 (Control configuration items) - sub-practice 2, if the record of the modification in the schedule and the change control tool are done in subsequent periods until a maximum limit of time of the defined period for monitoring of the project?

Example:

Verbal approval of the change - Monday

Beginning of the work - Tuesday, with storage of the configuration items in the configuration system (without baseline).

Register of the change in the change control tool and record of the activities in the project schedule - Friday

Collection of the progress and effort - Next Monday (weekly Monitoring, all monday)

In advance, thank you very much.

The first problem I see is the verbal approval. Verbal approvals are difficult to document or provide as evidence and over time can be forgotten or misremembered, not to downplay the risks associated by not documenting decisions.

Secondly, since you are asking about the sub-practices, that makes me wonder if you have taken the SEI’s Introduction to CMMI class. If you have taken the class, your instructor should have made clear to you the distinction between the required, expected, and informative components of the model. The sub-practices are informative components and are therefore provided only as information to help you understand the intent of the Specific and Generic Practices and Goals. To be clear, you are neither required nor expected to implement the sub-practices.

If the weekly schedule you described works for your projects, then you should be able to map it to the Configuration Management (CM) Specific Practices (SPs). However, the described weekly schedule sounds like it may have some gaps.

I strongly suggest that you have an SEI-certified Lead Appraiser conduct a Gap Analysis (Class C appraisal) of your organization, especially CM, to determine the gaps between your implementation and the CMMI.

Appraisal Scope Question for Agile Development

I am preparing for an appraisal coming up in a few months and I have run into an interesting question that I would like to ask you about.

Some of the projects in the Organizational Unit (OU) follow an Agile method for development. It is really a modified combination of SCRUM and XP. One of the questions they asked me was whether they would have to provide evidence for all of the Process Areas (PAs) in the scope of the appraisal? I was perplexed by this question because it seemed rather obvious to me, so I replied "of course." But this question stuck in the back of my mind so I dug into the SCAMPI Method Definition Document (MDD) for more clarification.

From the MDD, Method Assumptions and Design Principles, page I-19 and I-20: "The extent to which an organizational unit has implemented appraisal reference model practices can be determined only by considering, in aggregate, the extent to which those practices are implemented within the organizational unit by project and support groups. This process, in turn, necessitates the consideration of objective evidence for each instantiation, for each model practice within the appraisal scope."

This seems fairly clear to me, we must examine objective evidence for all practices of all instantiations within the scope of the appraisal. However, under the process definition in part 2 of the MDD you can find less convincing information under the Parameters and Limits section of 1.1.3 Determine Appraisal Scope. "Sample projects and support groups selected to form the organizational scope (i.e., the combination of focus and non-focus projects and support functions) must represent all critical factors identified for the organizational unit to which the results will be attributed." So it seems that as long as the combination of all focus and non-focus projects represent all the critical factors it should be sufficient. I know, critical factors are different from the PAs in the model scope. :-)

Also, in Parameters and Limits it clearly states: "Focus projects must provide objective evidence for every PA within the model scope of the appraisal..." "Non-focus projects must provide objective evidence for one or more PAs within the model scope of the appraisal..." So it seems that if an Agile project was a non-focus project they would not have to provide evidence for all PAs. Do you think this is correct?

The MDD goes on to further say in the Parameters and Limits: "In appraisals where the reference model scope includes any project-related PA, the organizational scope must include at least one focus project. If the organizational unit includes more than 3 projects, then the organizational scope must include sufficient focus projects and non-focus projects to generate at least 3 instances of each practice in each project-related PA in the model scope of the appraisal."

So, conceivably, you could have only 1 focus project, and say maybe 6 non-focus projects with each only having a few PAs to "generate at least 3 instances" of the practices that need to be covered in the scope of the appraisal.

I know this may seem a bit extreme, but believe me, I have run into stranger scenarios than this one. lol :-)

Ideally, all projects would cover all PAs. But the role and function of non-focus projects really seems to muddy the waters in my opinion.

What are your thoughts?

There is an SEI Technical Note CMU/SEI-2008-TN-003 published in November 2008 titled CMMI® or Agile: Why Not Embrace Both! by Hillel Glazer, Jeff Dalton, David Anderson, Mike Konrad, and Sandy Shrum that contains a lot of pertinent information about Agile and the CMMI, as well as Hillel's recently published companion article in Crosstalk Love and Marriage: CMMI and Agile Need Each Other http://www.stsc.hill.af.mil/crosstalk/2010/01/1001Glazer.html

In my opinion, if Agile is one of many different development methods in use by the organization and the Agile projects do not cover all of the PAs in scope of the appraisal, then it is not in the spirit of the MDD and non-focus projects to mix and match a number of Agile projects just so there is evidence for all the PAs. The way that I look at non-focus projects is that there may be some projects in the appraisal scope that do not have evidence because the projects have not reached that part of the lifecycle yet, NOT that they are not performing the practice.

And, just because a project is using Agile in a Maturity Level (ML) 3 organization does not mean that they don’t have to perform all of the PAs. There should be evidence available though they may be using terminology that is not obviously mapping to the CMMI. Someone with deep CMMI and Agile knowledge should work with the organization to show them how what they are doing does comply with the CMMI.

High Maturity Reference Books

I would like to read and understand more about High Maturity practices and I am also very keen to develop statistical skills so I will be able to construct a PPM. Can you please recommend some good reference books and/or links?

I would recommend the following books:

“Statistical Methods from the Viewpoint of Quality Control” by Walter A. Shewhart

“Understanding Variation: The Key to Managing Chaos” by Donald J. Wheeler

“Measuring the Software Process” by William A. Florac and Anita D. Carleton

“Building Continual Improvement” by Donald J. Wheeler and Sheila R. Poling

“Metrics and Models in Software Quality Engineering” by Stephen H. Kan

“Practical Software Measurement” by John McGarry, David Card, Cheryl Jones, Beth Layman, Elizabeth Clark, Joseph Dean, and Fred Hall

These are all excellent reference books to better your understanding of statistical and quantitative thinking.

Shortened CMMI Explanation

I have just started CMMI in my organization. I am looking for the shortened explanation for each of the CMMI Process Areas. I then plan to preach CMMI in much simpler, easy to understand language which would be like bulleted points. I also intend to offer seminars in my organization with different departments to make them understand what exactly do they are required to do. If you can help me, I would be very thankful.

I applaud your initiative and your plans. The best advice that I can give you is to take the SEI’s 3-day Introduction to CMMI class. That class will provide you with the necessary information for you to craft a shortened version for your organization. As an alternative, you can read the CMMI. The introduction of each Process Area provides a short explanation of the intent of each Process Area. But if you do not have any background with the CMMI, hire a CMMI consultant and explain to him or her your needs and have them help you prepare the training material you desire. What you are asking for is not free. It takes time and effort to produce and is a consultant's intellectual property.

PPQA After Maturity Level 2

I work in an IT organization that achieved CMMI Maturity Level 2 several years ago (we let the rating lapse) and I was wondering if you had some ideas on the following two questions:

1) What types of activities would PPQA engage in if the org had been Maturity Level 2 (I think they could have pursued Maturity Level 3 and been close)? Please also consider that the company is pursuing other types of improvement methods and models such as lean/6-sigma and ITIL.

2) What strategies should we pursue to show the worth of PPQA? Even in the good old CMM days and SQA one of the issues I had was that it was difficult to show the practical monetary worth of these support functions; one generally had to take it on faith that PPQA/SQA delivered some degree of worth to the company. Any thoughts?

The answer to question 1 is simple. Just read the PPQA Process Area and GP 2.9. The PPQA activities include performing both process and work product audits of the project and organization processes. For Maturity Level 2 that would mean auditing your REQM, PP, PMC, SAM, MA, PPQA, and CM processes.

The answer to question 2 is a bit more difficult. Basically you are asking, what is the cost of quality? One method you can use is to look at the total cost for the project and analyze it using Crosby’s Cost of Quality Model. The total costs break down into two categories: the Cost of Quality and the Cost of Performance.

The Cost of Performance includes such things as: generating plans, documentation, and developing requirements, design, code, and integration.

The Cost of Quality breaks down further into two categories: Cost of Conformance and Cost of Non-Conformance.

The Cost of Non-Conformance includes fixing defects, reworking documents, updating source code, re-reviews, re-tests, patches, engineering changes, CCBs, external failures and fines, Customer Support, and Help Desk.

The Cost of Conformance breaks down to two more categories: Cost of Appraisal and Cost of Prevention.

The Cost of Appraisal includes reviews, walkthroughs, testing (first time), independent V&V, and Audits.

The Cost of Prevention includes training, policies, procedures, tools, planning, quality improvement, data gathering and analysis, root cause analysis, and quality reporting.

The cost of PPQA is included in the Cost of Prevention.

When you consider these definitions and cost break down, the only category that will be affected by PPQA is the Cost of Non-Conformance. When PPQA audits the processes and work products, the audits will reveal non-conformances with people following the documented processes and procedures, which lead to re-work. By addressing these non-conformances, the goal is to reduce or effectively eliminate the rework and that is where you can demonstrate the value of PPQA.

Hope this helps.

Product Vision vs. Product Roadmap

I have a question. Is the Product/ System Solution Vision and the Product/ System Solution roadmap one and the same thing? And which comes first, the Product Vision or the Product Roadmap?

Product Vision and Product Roadmap are not CMMI terms. So, in the context of the CMMI, simply use the standard dictionary definitions of vision and roadmap and you should have your answer.

The way I look at these two concepts is that vision refers to what you are supposed to achieve and roadmap is how you get there. Therefore vision would come first followed by the roadmap.

The bottom line is that you should define these terms for your organization if you think that they apply and are important.

Manage Corrective Actions of PMC

During the analysis of a problem management tool I identified several actions that were open without a deadline and status record or with a deadline expired and without status record. Some actions were without a deadline because they depended on external agents to be resolved (customer, priority addressed by higher level) and others without updating the history and deadline because these didn't have modified status. The monitoring of these actions was performed through interviews. What is the impact of this condition on sub-practice 2.3-1 - Manage Corrective Action of PMC? Does this condition affect the reach of this practice? For me this condition characterizes a non-compliance. I would like to know your opinion.

The correct answer depends upon what you have written in your documented procedures for identifying issues and taking corrective actions. Do your processes and procedures allow you to to conduct verbal reviews of issues and take corrective actions? I do agree with you that you have identified some problems with your process and the proper completion of forms.

If you believe these are true non-conformances, but your documented processes allow this to happen, then it would be a good idea to modify your processes accordingly. Otherwise, you have identified a gap between the documented and practiced processes. These issues should have also been identified through the PPQC process and work product audits.

Also, since the sub-practices are an informative component, they provide additional material to help you understand the intent of the Specific Goals and Practices. There is no expectation that the sub-practices have to be implemented. Therefore, the situation you described really has no impact on PMC SP 2.3 Sub-practice 1.

CMMI for Modeling and Simulation Efforts

I am interested in applying CMMI techniques to modeling and simulation efforts. I do not mean the modeling and simulation of the software engineering process (I have plenty of information, models etc. on that).

We do modeling and simulation work (e.g. growth of the biofuel industry, migration issues, watershed management) to inform policy makers. We have a grassroots 'best practices' movement and are looking for suggestions, ideas, etc. to improve our process. We do not use 'popular' tools and the niche software we do use is not conducive to configuration management tools etc. In addition, we rarely have a requirements document. We are essentially doing research.

Do you have any suggestions?

In my experience, a pure R&D shop has difficulty implementing the CMMI-DEV because of the free form nature of the environment and work. However, what makes more sense for you to investigate is the CMMI-SVC. It sounds like you are performing engineering or research and analysis services for policy makers.

As you may not have a requirements document for your niche software, you most surely have requirements for the research you are performing for your clients. If not, then you run the risk of having your results called into question.

Therefore, I think that you would be better served by the CMMI-SVC than the CMMI-DEV as it sounds like you may not be doing a lot of development, but instead providing research services.

Project Tracking Through Milestones

We have a Project management process defined for covering Project Planning (PP) and Project Monitoring and Control (PMC). It includes a Work Breakdown Structure (WBS) (task effort size must be about 20 hours), estimation process, assigned resources, etc. We are using MS-Project for tracking the schedule, and each month there is an Excel report with a project status summary.

People think it is very heavy to track the all of the fine-grained tasks in MS-Project. They would prefer tracking the project through milestones in Excel.

I’ve heard about using the burn down chart in Scrum, and I know some organizations use agility within CMMI model.

It is possible to do that? Could we be more “agile”?

The short answer is yes. The CMMI does NOT prescribe any project management tool or level of tracking. These decisions are left up to the organization to make. Sounds like what you need is a process that fits your organization. If people are complaining that the tracking process is too cumbersome to use, then you should definitely examine other methods. Take a top down approach from the business goals and objectives. What project tracking information does a project manager need in order to determine if the business goals and objectives are being met? Once you answer this question, that will help you decide the proper level of project tracking and monitoring. If you are still having difficulty figuring out the best method, work with a CMMI consultant to help you define a process that will be the best fit for your organization.

How Do I Determine Estimates of the Size Attributes?

CMMI-DEV Project Planning (PP) Specific Practice (SP) 1.2 states 'Establish and maintain estimates of the attributes of the work products and tasks.'

During appraisals it has been noted that most of the time we are relying on expert knowledge to determine effort and cost rather than estimating the sizing attributes. Just using expert knowledge makes it difficult to improve our effort estimation, hence having a dedicated set of attributes is essential in order for us to improve our estimations. We have evaluated some comprehensive methods (e.g. Function Points), but these are not considered as beneficial to our projects. We are now looking for an approach that allows us to improve our estimates with a simpler method which could then be refined based on our needs. Would you please give me advice how to tackle this challenge?

What PP SP 1.2 is looking for is the Basis of Estimate for the work product. If you are estimating a document for example, what aspects or attributes of the document can you estimate that when combined with a productivity factor will yield effort and cost? You determine these attributes by analyzing the historical data from your organization’s past projects. So based on the project’s requirements, you could estimate the number of pages that would be written for a document, the number of figures or drawings that need to be created or modified, etc. These items are then sizing parameters.

For source code, you need to analyze historical data from past projects to give an empirical estimate. As you say, function points are comprehensive. A simpler method may be to classify your software requirements in categories such as interface requirements, display requirements, processing requirements, reporting requirements, etc. and then using your historical data determine the correlations between these requirements categories and code size. Many organizations start with this simple kind of estimation/prediction model and continually refine it by incorporating data from each new project until a fairly accurate estimation model emerges. This estimation model is then very specific to the organization and type of software development it performs.

Hope these ideas help.

GP 2.9 in PPQA process area - what do I need?

I'm responsible for coordinating CMMI ML2 implementation in my organization. For PPQA GP 2.9 "Objectively evaluate adherence of the process against its process description, standards, and procedures, and address noncompliance" who can be an internal auditor of my organization?

And about tools and forms for this evaluation: Could I elaborate a checklist for this evaluation? Do you have any examples of this?

An objective evaluation implies some independence from the people performing the process activities. That means the people who perform the PPQA activities do not evaluate their own work. Organizations perform GP 2.9 of PPQA in a variety of ways.

1. Someone else in the organization who is not performing the PPQA audits of REQM, PP, PMC, etc. audits the PPQA activities

2. If the company is large and has several divisions, someone who performs the PPQA activities in another division audits the PPQA activities in your division

3. An external auditor (e.g., ISO 9000) audits the PPQA activities

4. An external consultant audits the PPQA activities

5. If your company is a government supplier, then the government may be auditing the PPQA activities

6. Etc.

So in your case, you could use an internal auditor to audit the PPQA activities as long as that person is not auditing his or her own work.

As far as a checklist, that needs to be developed by the person who is auditing the PPQA activities, just like the PPQA audits develop the checklists for the other Process Areas. The checklists need to cover both a process audit and a work product audit, and it may be easier to have two checklists instead of one. The checklist needs to be based on your documented PPQA process and process assets, not the CMMI.

If you are having difficulty understanding how to write a checklist, then I strongly encourage you and your organization to have someone come and train you how to perform PPQA audits. It is very important to conduct these properly; otherwise you could face difficulties with your process improvement efforts.