SCAMPI A Appraisal Questions for FAR Groups

Can you share the Questionnaires for a ML 3 SCAMPI A appraisal? Currently my organization is preparing for SCAMPI A and I want to trained the interviewees as this is their first experience an appraisal.

I am looking for some questions based on each Process Area, so the FAR Group Members will be prepared to answer them.

I have searched on Google but not able to find any information. :(

I am not surprised that you have not been able to find what you are looking for on the internet. A Lead Appraiser will not share his or her questions with you. Simply because the Lead Appraiser does not want the interviewees to be trained on what answers to provide, thereby biasing the results of any SCAMPI appraisal. If you and the organization are nervous about the appraisal, don’t be. There is no preparation for the interviews necessary. All that is expected by the Lead Appraiser is that any interviewee should be able to talk about HOW they perform their job duties. If people cannot do that, then the organization is not ready for an appraisal.

If you still feel uneasy, talk to your Lead Appraiser and have him or her conduct a Class B appraisal with interviews. A Class B or SCAMPI B appraisal can function as a dress rehearsal for a SCAMPI A without having the organization and the people worry about “flunking” or “passing” the appraisal. The interviewees will have the experience of being interviewed.

If you still want to perform an internal set of interviews, simply take the CMMI and step through the applicable practices with different groups of people (project managers, configuration managers, developers, testers, etc.) and ask them HOW they perform the Specific and Generic Practices. There is no set of standard questions. Each Lead Appraiser has their own style of questioning interviewees.

REQM SP 1.2 Question

Can you please help me?

I am developing a study based on CMMI-DEV and I am having some difficulty with the interpretation of Specific Practice 1.2. My concern is how to apply SP 1.2 in practice. Who are the members of the project that must be committed to the requirements (business analysts, systems analysts, test analysts, developers, systems architects, project manager, and project leader)?

How should their commitment be documented? For example, for each new or changed requirement the members must sign a document that means that they are aware of the requirement and are committed to it.

Do you have a copy of the Addison Wesley published CMMI-DEV version 1.2? If so, there are some very helpful tips in the margins for REQM SP 1.2.

When you have a set of requirements for a project team, everyone on the team impacted by the requirements needs to share a common understanding of the requirements. In addition, since you typically do not have the full set of requirements available at project start, the project team needs to continually evaluate and reevaluate their ability to meet the requirements as the requirements evolve over the life of the project. One way of evaluating requirements is an impact assessment on the existing requirements, design, documents, test cases, current and downstream tasks and activities, cost, schedule, etc. for a new or changed requirement. By going through an impact analysis, that will provide a way for the project participants to communicate their ability to meet the requirements and associated commitments. The resulting impact analysis report can be one way of documenting the commitment. Other methods for documenting commitments include meeting minutes, document signatures, or email. However, just signing a document indicating that the participants are aware of a requirement really does not indicate that they are committed to the requirements and you might run into difficulties later on in the lifecycle.

Please keep in mind that commitments include both the resources involved (people, tools, and facilities) and the schedule for completion.

Covering SAM SP2.2 and SP2.3

I want to ask you about CMMI Supplier Agreement Management (SAM) SP 2.2 Monitor Selected Supplier Processes and SP 2.3 Evaluate Selected Supplier Work Products. I have difficulty in understanding these practices and how to apply them in my company. We have a project that needs to buy new hardware and deliver it to customer.

How can I select and monitor supplier processes in this case?

Can you please explain that to me and give me an example of a supplier process that is applicable in our situation?

The best way to answer this question is to attend the SEI’s Introduction to CMMI 3-day class. During the SAM topic the instructor will explain the difference between these two practices for you. I would also suggest that you read the informative material in the SAM Process Area, for not just these two practices, but all of the Process Areas. The informative material is extremely rich in information, hints, tips, etc. and it provides guidance for understanding the intent of each practice and goal. I would also recommend that you not use the goal and practice titles for anything other than labels. The titles are shortened statements and do not always communicate the correct meaning of the goal or practice.

If you read the informative material for SAM SP 2.2 and SP 2.3 it will be immediately clear to you the differences. And if you still have questions regarding your specific circumstances, I suggest that you ask your SEI-certified Lead Appraiser. Without having a good understanding of your company and how it conducts business, asking specific implementation questions will most likely not provide the correct advice to you.

However, here is a very brief description of the differences between these two practices.

SAM SP 2.2 “Select, monitor, and analyze processes used by the supplier.” You as the acquirer of the supplier’s products or services must define in the supplier’s contract or agreement those specific supplier processes that are critical to your success. You have to state how and when you will be monitoring these critical processes. This practice is done as a risk mitigation to avoid surprises at the end of the contract when the supplier delivers the end product or service. Think of this practice as the acquirer performing PPQA process audits of the supplier’s processes.

SAM SP 2.3 “Select and evaluate work products from the supplier of custom-made products.” If your supplier is only delivering off-the-shelf product to you without any customization, this practice does not apply. However, if your supplier is providing custom-made or modifications to your specifications, then you need to decide which of these products are critical to your success and define those in the supplier’s contract or agreement. You have to state how and when you will be evaluating the products. Think of this practice as the acquirer performing PPQA work product audits of the supplier’s work products.

CMMI Practices for Documentation Teams

Our organization is CMMI Maturity Level 3 Ver 1.2 certified. Our delivery teams are going to implement CMMI practices soon. I did see the processes of the organization and though all the roles appeared from project leader to developer to manager, except for documentation teams. In the same context, I am very curious to see if there are any set of practices to be followed for the documentation department in CMMI as all the processes at CMMI ML2, ML3 are specific to project management, engineering, support, organization areas.

Your question actually raises some other questions:

1. What are the roles and responsibilities of your documentation teams and documentation department?
2. If this group of people is responsible for the technical documentation (e.g. requirements, design, etc.) and the user documentation, how and why were they excluded from your ML 3 SCAMPI A?
3. How are the delivery teams different from the organization that achieved Maturity Level 3?

What is puzzling with your question is that REQM, RD, TS, PI, and CM cover the different aspects of writing and controlling the various documents associated with designing, developing, maintaining, operating, using, and deploying products. And then VER covers the inspection/review of the documents before placing them in the baseline and controlling them with CM.

It does appear from your description that your organization omitted the documentation people as a process role from your process documentation. In my opinion, at a minimum, your PPQA audits should have identified this omission long before your SCAMPI A appraisal. Then your Lead Appraiser should have identified this gap during the appraisal planning process before the SCAMPI A and should have taken steps to address the gap or postponed your appraisal until the documentation group was included in the scope. Since your documentation group was apparently not included in the scope of your appraisal, this oversight also calls into question your Lead Appraiser’s credentials and quite possibly the validity of your SCAMPI A results.

The bottom line, in my opinion, and based on only what you stated, your documentation group should have been included in the scope of your ML 3 SCAMPI A. Even if all they do is Document Configuration Control (which would be covered under CM) or Document Quality Assurance (which would be covered under PPQA).

The answers to my above questions could provide additional information that would change my opinion.

Finding a Lead Appraiser

My company needs a very inexpensive Lead Appraiser – where can I find one?

I hate to say this, but you are using the wrong criteria for contracting a Lead Appraiser. When it comes to the CMMI and Lead Appraisers, you get what you pay for. And in point of fact, if you go with the lowest price Lead Appraiser that you can find, it is quite possible that this person is either very inexperienced as a Lead Appraiser or may not have the necessary experience or background to provide credible services. The risk you run is that the SEI may not accept the appraisal results in this situation. Then you have spent the money and may need to repeat everything, which can be a very expensive proposition.

Much better criteria to use when considering to hire a Lead Appraiser include:

• number of years experience as a Lead Appraiser
• number of appraisals led (SCAMPI A, B, and C)
• experience in working with small organizations
• ability to interpret the CMMI to appropriately fit your organization's needs
• recommendations/testamonials from clients who have worked with the Lead Appraiser

You should interview the Lead Appraiser. Prepare some questions and scenarios that apply to your organization and learn how they would interpret the CMMI in your context and what they would recommend.

The basic question comes down to why do you want to be appraised? Your internal costs for implementing the CMMI will far outweigh any external costs associated with an SEI-certified Lead Appraiser. You should be implementing the CMMI because there is some business value associated with the CMMI. Lead Appraisers are experts in process improvement. As an analogy, if you are going into the hospital for major surgery, do you go to the lowest price surgeon you can find (possibly someone with questionable credentials) or do you go to a surgeon that has a reputation for quality and success?

Think of it this way. Selecting a Lead Appraiser is a very important decision. There is certainly cost factors as well as risks to consider, and possibly other considerations as well. Therefore, I suggest that you look at the Decision Analysis and Resolution (DAR) Process Area and use the practices described there to select your CMMI services provider.

Advice on Process Performance Models

We have developed a Process Performance Model (PPM) for system testing defects. In our model we analyzed multiple factors (independent variables) that may influence the number of system test defects. However, the strange part is that the regression analysis has eliminated all of the variables except one, testing effort. Some of the other variables included things like number of test cases, tester's experience, and size (LOC).

Can a valid PPM only have one independent variable? The p-value is less than 0.05, and the R squared value is 0.8, so it seems that it would be reliable. But only one independent variable seems rather limited and weak to me, but I can't think of a justifiable reason why it is invalid considering we have analyzed data from several independent variables and only one was found to be statistically significant.

What do you think?

It does sound a bit odd. Perhaps you have overlooked some other X Factor that you should be evaluating. But it is conceivable to have a model with only one independent variable.

Basically your model looks like Y = mX +b. But given all of the PPMs that I have seen, they all contain multiple X factors.

Several things could be going on here.

1. The regression analysis was faulty

2. The list of X factors evaluated was incomplete

3. One or more of the X factors were a combination of factors that should have been separately evaluated

4. Not enough data were evaluated for the regression analysis to yield proper answers

5. The distribution of the data is not a normal distribution (Gaussian distribution aka Bell Shaped Curve) and analytic techniques for a normal distribution were mistakenly used

You probably should have a statistical expert independently examine your analysis to see if the resulting PPM is in fact a linear relationship between X and Y.

The other option is to use the PPM for several months of analysis and see how accurate it is. If the PPM yields accurate predictions, then it is most likely a good PPM.

One thing that can happen if you have not been careful with storing the raw data and/or not collecting at the proper level of detail is that factors can be inadvertently combined and thus cancel each other out thereby yielding a factor that on the surface looks reasonable, but may in fact not be. For example if one element is high and the other one it is combined with low, then the result will be somewhere in between. Think of two out of phase sine waves. Each one by itself looks like a sine wave, but the combination is something else entirely.

So as an example, suppose you are collecting defect data (requirements defects, input defects, design defects, coding defects, test case defects, etc.) and you have neglected to use these categories to count and distinguish the different types of defects. Then the defect count probably won’t correlate with anything because the individual categories are cancelling each other out. This is the kind of data behavior that caused many of the early HM adopters to throw away their historical data and start over from scratch. So it is very important to follow the Measurement and Analysis PA to properly define all of the base measures and to properly store them so when it comes time to perform the statistical analysis of the data, there is sufficient information stored to allow different types of analyses to be performed to isolate the specific information of interest.

Hope this explanation helps.

Integrated Project Management (IPM) - SP 2.2

Could you please provide some examples of evidence for IPM Specific Practice 2.2 and some examples of a critical dependency?

Integrated Project Management (IPM) Specific Practice 2.2 states “Participate with relevant stakeholders to identify, negotiate, and track critical dependencies.” This practice is one of many compound practices in the CMMI. Therefore, to completely cover the practice, the organization would have to provide evidence that the project is participating with the relevant stakeholders. And that participation would be to identify critical dependencies, negotiate critical dependencies, and track critical dependencies. So the type of Direct Evidence that I would expect to see as a Lead Appraiser or Appraisal Team Member from the organization would be a list of critical dependencies, issues, and action items that were the result of the participation and for Indirect Evidence I would expect to see the meeting minutes or review/inspection reports that resulted in the list provided as Direct Evidence.

Now, if you read the Tips and Hints in the CMMI for IPM SP 2.2, these will provide insight into what a critical dependency is. Each project will have some constraints such as time, quality, budget, etc. that it must meet. By applying these constraints to the project schedule the Project Manager will be able to define the critical path through the project tasks and activities. The critical path contains all of the tasks and activities that have a direct impact on meeting the scheduled end date for the project. If a task on the critical path slips or runs over, then the end date of the project has a corresponding slip or overrun. Tasks that are not on the critical path can move about a bit before they impact the end date. The project tasks and activities on the critical path define the critical dependencies for the project. For example, if both Task A and Task B are on the critical path and Task B cannot begin before Task A completes, that is one critical dependency. Another example of a critical dependency is an external group is supplying a work product to the project and a Task cannot begin until the work product is received and accepted. This type of critical dependency may also be called a critical deliverable.

Hope this explanation helps.

Bi-directional Traceability Question

Would you please explain to me the meaning of bi-directional traceability? If I am not wrong, Horizontal Traceability indicates mapping of a requirement across all the SDLC phases like Requirement->Design->Construction->Testing. But I do not get what is meant by Vertical Traceability. Would you please educate me on this?

Vertical traceability is top-down bottom-up traceability. Vertical traceability is the most common type of traceability. Basically it is tracing from the highest level requirements all the way down to the product component level (could be even to a specific line of code). And then starting at the lowest level and tracing all the way back up to the top. Horizontal traceability is tracing within a document or between peer documents. What you have listed as tracing from requirements to design to construction to test is actually vertical traceability. And when you come right down to it, it doesn’t really matter what type of traceability you label it. All that matters is that you can trace forward from a starting point to an end point and vice versa. The level and complexity of the traceability is determined by the criticality of the product and project. For a program like the Space Shuttle or Space Station, you would need very rigorous requirements traceability. Whereas a small application or a product would only need some simple traceability.

REQM - SP1.1 Query

In REQM – SP1.1 calls for “Criteria for evaluation and acceptance.” Can you tell me what this expected “criteria” would be?

I suggest looking at Requirements Management (REQM) Process Area in your copy of the CMMI. On page 490 in REQM Sub-practice 2 there is a list of example evaluation and acceptance criteria.

REQM does not expect the organization to have evaluation and acceptance criteria. What you are expected to provide is evidence that support the required components (Specific Goal 1 in this case; evidence that demonstrates that your requirements are managed and inconsistencies with project plans and work products are identified) and evidence that supports the expected components (Specific Practice 1.1 in this case; evidence that demonstrates that you have developed an understanding with the requirements providers on the meaning of the requirements). If your Lead Appraiser is telling you that you have to provide these criteria, then he or she should be challenged as to why they are telling you that you need to provide this evidence.

The evaluation and acceptance criteria are one way of developing this understanding. Please bear in mind the section title is TYPICAL Work Products, not EXPECTED Work Products, or REQUIRED Work Products. The list of Typical Work Products is merely a list of example work products.

Selection of Suitable Lifecycle Processes

Selection of a suitable lifecycle for the project/product is more of a technical issue. Then why is it covered in Integrated Project Management (IPM) rather than Technical Solution (TS)?

Selecting a suitable lifecycle is a function of the project requirements and the application domain. The selection is something that should be performed when planning the project activities, not while performing the actual software engineering activities. Selecting a lifecycle model and defining the project’s lifecycle phases are necessary for planning and estimating the project. That is why lifecycles appear in Project Planning (SP 1.3) and Integrated Project Management (sub-practice 1 of SP 1.1) . The selection of the lifecycle is a project management activity.

From another perspective, the purpose of Technical Solution is to design, develop, and implement solutions to requirements. And selecting an appropriate lifecycle model is an activity that needs to be done before you can design, develop, and implement a solution to the requirements.

Process Performance Model and QPM Inquiry

We are a CMMI Maturity Level 5 organization and now we are preparing for our re-appraisal.
We have Process Performance Models (PPMs) with different X- prameters and good R^2, R (adjusted) & P(value).

We perform simulations to calculate the probability of achieving our organizational goal and the determination that the probability % is good enough to achieve the goal.

Now we are in our process of implementing the PPM. My question is: Shall we use the PPM to estimate the X- parameters in my projetct? Or what should I do with this model after that ?

Please advise.

I hate to say this, and I could be wrong, but reading your question it sounds like you are asking what do I need in order to be Maturity Level 5 (ML 5)? Is this equation sufficient? And by asking what do I do with the PPM, I may be misunderstanding you, but if you were really at ML 5 you wouldn’t be asking this question. I get the impression from your question that your organization doesn’t know why it is using a specific technique for the PPMs.

The best solution to your question is to ask your High Maturity Lead Appraiser for help and guidance as he or she should have enough understanding and knowledge of your organization to provide the answer.

But once again, the nature of your question gives me the uneasy feeling that your organization may not even be Maturity Level 4.

Applicability of SAM

Does Supplier Agreement Management (SAM) apply to the supplier? Suppose you're an organization that is supplying to another organization , for example to develop a software package. This organization is not using suppliers. I assume that SAM is not applicable to this organization. Is my assumption correct?

Your assumption is correct. But allow me to restate your scenario to avoid any confusion by using the word organization mulitple times in one sentence. Organization A develops a software package and delivers it to Organization B. Organization A does not use any suppliers or outsource any work. In this case Organization B is the acquiring organization and Organization A is the supplier organization. Therefore, SAM would apply to Organization B, but not to Organization A.

Rationale for Maturity Level 5

Our organization is currently at Maturity Level 3 and is now planning to go for Maturity Level 5. Our CEO is asking for the short term and long term benefits and challenges for implementing Maturity Level 5. How do we provide this information?

The question that needs to be asked is why does your organization want to achieve Maturity Level 5 (ML 5)? If the organization has already achieved Maturity Level 3 (ML 3), what is the motivation for achieveing ML 5? Most likely it is not mandated in a contract or by its customers. But achieving ML 5 is desirable to be competitive in the marketplace. Therefore, the reasons for achieving ML 5 should provide some indication of the short term and long term benefits.

The first step in becoming a High Maturity organization is defining your Quality and Process Performance Objectives (QPPOs) that are based on your business goals and objectives as well as your customer needs. The QPPOs should be stated in a form such that they specify a timeframe. And that information will provide some ideas of the short term and long term benefits.

However, the best advice that I can provide is to hire an SEI-certified High Maturity Lead Appraiser (HMLA) who will work with you to help explain the benefits of ML 5 to the CEO. The HMLA should have experience working with many different organizations at various Maturity Levels and be able to talk about the different challanges that have been faced by other organizations, as well as the challenges and risks within your organization.

At this point, it sounds like your organization is just beginning its journey to ML 5, so I would have to be convinced that your processes are stable enough to provide the data needed to quantify any short term and long term benefits. I think that the best you could do at this point is communicate this information in qualitative terms. Any quantitative information may not be accurate until you have implemented High Maturity.

The Right Model to Follow

Our operations involve providing reporting solutions & implementing them based on tools like SAP BOBJ, MSTR, MS, ORCL primarily setting up datamarts & ETL processes and also providing technical support to customers as a line of business (Java/.net support for third party tools based on bugs/additional requirements given by customers at random)

Question 1: For such an organization , would it be right to follow CMMI for Development or CMMI for Services or both?
Question 2: Are appraisals made on CMMI V1.3 or are we still in CMMI V1.2. If still on 1.2, when are we likely to move to V1.3?
Question 3: If only a few projects or one division of an organization is appraised, will the appraisal rating stand for the company?

Question 1 – This question is difficult to answer based upon your description. Are you developing a product or delivering a service? I really cannot tell from your description, it could be either or both. If you are developing a product, then the CMMI-DEV might be applicable. If you are delivering a service, then the CMMI-SVC might be applicable. And you could blend the two. Another aspect to consider is are you planning to have a SCAMPI A appraisal? If so, then your Lead Appraiser will be able to help you decide which constellation is applicable to your organization. If you are not planning on being appraised, then I would suggest that you look at both constellations and use the Process Areas that apply best to your organization.
Question 2 – CMMI v1.2 is the current version. The SEI recently announced that v1.3 will be released in November 2010. And based on past experience, the SEI will most likely allow appraisals against v1.2 to be conducted up until October 31, 2011.
Question 3 – The short answer is no. The appraisal results ONLY apply to the organization that was appraised, not the entire company.

Examples of Applying CMMI or CMMI-SVC to Government Organizations

I'm looking for cases where CMMI or CMMI-SVC have been applied (successfully or otherwise) within a government organization. I'm trying to identify lessons learned that would be useful to a US federal government department that has considered applying CMMI practices for process improvement.

There have been multiple US Government and Military organizations that have implemented and been appraised to the CMMI. Just look at the SEI's list of appraisal results http://sas.sei.cmu.edu/pars/pars.aspx and you will find these organizations. As far as the CMMI-SVC goes, this constellation was only released in late Feb 2009 and organizations are just beginning to use the model. So I seriously doubt that there are any examples of use available at this point in time. My guess is that the first time anyone will see examples of the CMMI-SVC being used will be at the 2010 SEPG Conference in Savannah, Georgia March 2010.

Auditor Directing a PPM

We are trying for CMMI ML 5 Ver 1.2, and our auditor has asked us to come up with a PPM for predicting the outcome of CAR and OID. Can you help me on how to go about it?

When you say auditor, I assume that you mean your Lead Appraiser (LA) is asking you for a PPM for predicting the outcome of Causal Analysis and Resolution (CAR) and Organizational Innovation and Deployment (OID). Is your Lead Appraiser an SEI-certified High Maturity Lead Appraiser? Has your organization identified the need for a Process Performance Model (PPM) to predict the CAR and OID outcomes, or is this solely a request from your LA? Your LA is not the person to tell you which PPMs you need. Do you have Process and Product Quality Objectives (QPPOs) that require PPM(s) to predict CAR and OID outcomes? If the answer is no, then you don't need a PPM for CAR and OID.

What I find odd is that you do not mention a Process Performance Baseline (PPB) for CAR and OID. If you are going to define and develop a PPM, then you really need to develop the CAR and OID PPBs first before you can determine the PPMs. From your brief description, it sounds like your Lead Appraiser may have overstepped his boundaries in asking for the CAR and OID PPM.

Identifying Risks

What is the difference between PP SP 2.2 Identify Project Risk and RSKM SP 2.1 Identify Risks?

What you are asking about is one of the basic differences between Maturity Level 2 (ML 2) and Maturity Level 3 (ML 3). Project Planning (PP) is a ML 2 Process Area (PA) and Risk Management (RSKM) is a ML 3 PA. At ML 2, the project only needs to be able to identify risks and that is what PP Specific Practice (SP) 2.2 addresses. At ML 3, RSKM builds upon the foundation of identifying and tracking risks put in place by PP and Project Monitoring and Control (PMC). RSKM SP 2.1 therefore builds upon PP SP 2.2 by adding more rigor for risk identification. Just read the informative material and sub-practices for both SPs and you will immediately see and understand the difference.

Project Specific Findings

While reading the SCAMPI A Method Definition Document (MDD), I discovered that providing project specific findings is an option that can be requested by the project sponsor. I always believed that the Findings Presentation could only include findings at organizational unit level. So if the sponsor requests project specific findigs as part of the appraisal output, how are they communicated? Included in the final findings? If this is the case, what about the non-attribution of findings? Or via a separate document?

The SCAMPI Method does indeed allow for project specific findings if the Appraisal Sponsor requests them. If requested, then project specific findings could be communicated in the Final Findings Presentation along with the other information, or they could be communicated separately. Reporting will be negotiated and documented in the Appraisal Plan. Keep in mind that even if the Appraisal Sponsor doesn’t request project specific findings, the PIIDs will contain project specific observations, so if someone were interested in finding out about project specific information, all they would have to do would be to read the PIIDs. The organization is required to retain the PIIDs used for the appraisal for three years, the same length of time that the appraisal results are valid.

The non-attribution issue concerns identifying any individual, project, or group as the SOURCE of the information. So by reporting project specific findings, the Lead Appraiser and Appraisal Team are not violating the non-attribution rules unless they reveal the source of the finding(s). When I conduct an appraisal, the final Appraisal Team activity is to scrub the PIIDs of all attribution information (names, interview sessions, etc.) and deleting all previous versions so the organization only retains the scrubbed PIIDs.

Selecting Projects for a SCAMPI Appraisal

One of my client organisations works on hardware and software design projects related to locomotive design for a manufacturer. They want to adopt the CMMI-DEV and scope their SCAMPI appraisals only for their software projects. Both, hardware and software design projects are undertaken by the same organisation, under the same management, same company name and at the same location. Is it be proper for the organization to exclude the hardware projects from their CMMI journey and appraisals? I feel it is not proper and it violates the principle of institutionalization of processes across the organization. Moreover, if the Lead Appraiser agrees to conduct an appraisal for software projects only, he or she will violate the principle of randomly selecting the projects for the appraisal.

The way I look at this situation depends upon on how the company is organized. If there are separate hardware and software development groups, departments, or divisions that deliver products to a program office (for example), then the software group could be appraised on its own. The same could be true for the hardware group. In fact, this situation occurs quite frequently in my experience here in the United States.

However, if the hardware and software groups are tightly intertwined in building and delivering a product (meaning you cannot separate the two), then I would say that both the hardware and software groups had to be appraised together.

The correct decision requires the Lead Appraiser to have a very good understanding of the organization and its business, as well as being a function of how the organization defines a project and what its process documentation states.

CMMI Documentation Request

I have read your blog. Can you provide me some documented CMMI processes (Draft)? I would like to learn and implement the process. I have good experience using ISO 9001 for the past 10 years.

There is no such thing as a set of documented CMMI processes that can be provided for a company, though some unscrupulous consultants will try to sell you a set. Rather, there are industry accepted standards for documenting processes, such as ETVX. These process documentation standards are easily available by searching on the internet. The actual processes for REQM , PP, PMC, SAM, MA, PPQA, CM, etc. are a function of the organization’s business and its practices, though there is some commonality across companies and organizations. However, this commonality exists at the CMMI level. The CMMI is a set of guidelines for process improvement. The implementation of these guidelines will differ from organization to organization. The best advice that I can give you is to document all of your current business practices using an industry accepted process documentation standard, avoiding the temptation to improve the process. By simply documenting your existing processes, you will discover opportunities for improvement, but don’t make them at this point. Just document the process as it is currently practiced. Once you have all of your processes documented, then compare the results to the CMMI, add the missing practices, and address any improvement opportunities. Then you will have a set of processes that your employees will have ownership of and will also comply with the CMMI.

Estimating Cost vs. Establishing the Project's Budget

What is the difference between Estimation of Project Cost (PP SP 1.4) and Establishing Project Budget (PP SP 2.1)?

Project Planning (PP) Specific Practice (SP) 1.4 addresses estimating the effort and cost for the project’s work products and tasks based on specified estimation rationale, models, and/or historical data. SP 2.1 takes this information for the individual work products and tasks, along with the major milestones identified in the project’s defined life cycle, any scheduling assumptions and constraints, and task predecessor/successor relationships to construct the project’s schedule and overall budget. Simply put, SP 1.4 concerns individual items in the project and SP 2.1 concerns the project as a whole.

Requirements Traceability, To What Level?

It certainly makes sense to include relations between requirements and source code in the requirements traceability matrix. If requirements change, we've a direct view on the impact on source level. But can this be managed and maintained, even with a tool? Is it wortwhile to put a huge effort in maintaining relations between requirements and the source code? Isn't it sufficient to define these relations at the level of design elements and user acceptance test cases? In this case, we've a means to verify that each requirement is covered by design and test. I believe this strongly reduces the risk of uncovered requirements in the final delivery, which is one of the main purposes of requirements traceability.

The answer is, you do whatever is necessary to meet your business goals and objectives. It depends upon the criticality of the product or service you are delivering. If your product is highly complex and someone could lose their life if there was a missed requirement, then it is necessary to put a huge amount of effort into requirements traceability. Just consider the Space Shuttle program, the amount of requirements etc. The Space Shuttle software is as close to zero defects as you will ever find. And at the other end of the spectrum, you would be justified limiting the amount of effort for traceability.

PPQA or VER?

I have a question. When people performa a review to assure than a coding standard is being used, is it considered a PPQA audit or a verification activity (VER)?

The correct answer is, it depends upon the nature of the review. If your documented software development process states that the coding standard is used to write code. Then a process audit of the software development process would be looking at the coding standard and determining if it was indeed being used by the developers. That would be a Process and Product Quality Assurance (PPQA) audit activity. If your documented verification process stated that a code peer review involves comparing the code to the coding standard, then that would be a Verification (VER) activity. And if your documented processes specified both of these conditions, then the answer to your question is both a PPQA audit activity and a VER activity. How you view the code review against the coding standard is therefore context dependant.

If you are asking this question because you are preparing your Direct and Indirect Evidence for your PIIDs and a SCAMPI A appraisal, then you will need to explain the context so the appraisal team will be able to correctly evaluate the evidence.

Software Sizing

We are trying to achieve CMMI Maturity Level (ML) 3 in my company and we have decided to skip ML 2. So, now, one of our problems is related to software size estimation. We defined a proprietary method, based on Use Case Points and Function Points, but the practitioners are struggling with it. From your experience, what other methods have you seen or implemented in the companies with this same problem? Or, if a proprietary method was defined, what were the main aspects to take in account?

I would strongly urge you to forget the ML 3 Process Areas until you have mastered ML 2. There is a fundamental difference between how a ML 2 Project Manager approaches Project Planning (PP) and Project Monitoring and Control (PMC) vs. a ML 3 Project Manager. Estimation being one of the differences. Use Case Points and Functions Points are fairly sophisticated concepts and there are challenges with getting consistency in determining what each of these things are. I would recommend that you take a step back from the model and the projects and look at your historical project data. Use the actual effort, costs, etc. from previous projects to estimate a new project. Forget about Use Case Points and Function Points for now. Once you have mastered being able to use historical information to build an empirical estimation model, then it might make sense to add a layer of sophistication by considering Use Case Points or Function Points.

Another recommendation is let the Project Manager create the project estimates and then review them with the practitioners as a sanity check rather than ask the practitioners to create the estimates. Over time as the organization gains experience estimating projects etc., then it makes sense to involve the practitioners up front in the estimation process. You have to learn to crawl first with estimation before you can sprint with the big boys.

Application of the CMMI for Services

I am in the process of preparing a "justification" presentation that talks about why the CMMI is important and how it can be applied. Though I have enough material and data to substantiate this, I don't have data that is contextual. I am looking for data, links, or any input that would point me to where the CMMI was applied to a Staff Augmentation organization or a comparison where I can see how each CMMI constellation (DEV, ACQ, and SVC) can be applied (Development Orgs, Support and Maintenance Org, Outsourcing Org, Staff Augmentation Org).

Though I could be wrong here, but I seriously doubt that the kind of information you seek exists at this point in time. Keep in mind that the CMMI for Services (CMMI-SVC) was only released in late February 2009 and the first appraisals to this constellation cannot occur at least for another 2 or 3 months. So there won’t be any anecdotal evidence or case studies available to provide the information you seek. I would imagine that the first opportunity to see any information of the type will be at the 2010 North American SEPG Conference. And since you are asking about applying the CMMI to a Staff Augmentation organization, the applicable CMMI constellation would the CMMI-SVC. There just isn’t a lot of information about its benefits right now, not enough time has elapsed since its release.

Is There a Difference Between Software and Firmware Development wrt the CMMI?

Would you kindly let me know the difference between the development of software and firmware and what CMMI practices are useful for firmware development? Is there a need to have Configuration plan for firmware development projects to control the code version?

Though the location of where the software or firmware resides is different as well as the code, there is no difference in the development practices and activities that occur. The CMMI for Development (CMMI-DEV) applies equally well to any type of product development. All projects have requirements, the work needs to be planned and managed, the work products and deliverables need to be managed, designs have to be developed, etc. The specifics in the processes most likely will vary from hardware development to software development to firmware development, but all of the Process Areas (Engineering, Support, Project Management, and Process Management) apply.

Product Planning and Configuration

It is common knowledge that Maturity Level 2 is project specific, and still I find at times Lead Appraisers asking funny questions during SCAMPI A appraisals. Quite recently, one of my friends told me that his Lead Appraiser is looking for planning at the product level, as well as Configuratuion Management at the product level. I was a bit amazed, thinking that Maturity Level 2 focuses on Project Planning, not product planning. What do you think about this situation? Have you been faced with this situation before? Is there a workaround for it?

From what you describe, it sounds like this Lead Appraiser could be misinterpreting the CMMI and possibly misleading the organization. The CMMI is quite clear that the Project Planning (PP) Process Area (PA) is for project planning purposes, not product planning.

"The purpose of PP is to establish and maintain plans that define project activities."

However, sometimes the difference between project and product can be blurred. By not knowing the context of the situation you described, the Lead Appraiser may have been trying a different approach to draw project planning information out in the interview sessions.

In one respect, it really doesn’t matter the line of questioning in a SCAMPI interview session. The Lead Appraiser could really ask about any topic. However, once he or she starts deviating from the CMMI, they are on shaky ground and could lose credibility. What does matter however, is the set of findings produced by the Lead Appraiser and the Appraisal Team. If there are findings associated with product planning that cannot be tied to the satisfaction of a CMMI Specific Goal or a Specific Practice, then these would be non-model findings and should have no impact on the resulting appraisal rating. However, if these non-model findings do impact the appraisal rating and the Lead Appraiser and Appraisal Team fail to demonstrate the linkage to Goal and Practice satisfaction/implementation, then the Lead Appraiser has not correctly performed his or her Lead Appraiser duties and the SEI should be informed about this issue so it can be investigated.

Why SAM is excluded ?

Why is SAM removed from CMMI L3?

Supplier Agreement Management (SAM) is not excluded from the CMMI-DEV or CMMI-SVC. When you say L3, I assume you mean Maturity Level 3 and SAM is definitely NOT excluded from ML 3. If, however, the Lead Appraiser in working with the organization determines that SAM is not applicable to the work performed by the organization, SAM will be considered Not Applicable to the scope of the appraisal. And that could be at any Maturity Level. Please read previous my posts regarding SAM for more information. http://ppqc.blogspot.com/2009/04/excluding-supplier-agreement-management.html and http://ppqc.blogspot.com/2009/07/cmmi-novice-question.html

Criteria For Selecting and Qualifying an Instructor

What are the important criteria that we should use to qualify and select internal and external instructors?

First of all you need to specify what kind of instructor you want to select and qualify, as well as the topic(s) you wish this person to teach. Then you would need to identify the core competencies for instructors in your organization. Coincidentally, I just created a job description for a Trainer position at my client based on some core competencies I found by searching the Internet. Here is a list of candidate competencies for your consideration:

Establishes and maintains instructor credibility

• Demonstrates content expertise
• States the learning objectives
• Provides a model of professional and inter-personal behavior

Demonstrates effective communication skills

• Uses appropriate verbal and non-verbal language
• Uses frames of reference, language, and technical terms familiar to the students
• Demonstrates that students have understood the message

Demonstrates effective presentation skills

• Uses appropriate modulation of voice to keep students engaged and interested
• Makes eye contact with individual students in the class
• Uses appropriate gestures to communicate
• Make effective use of moving around the room while facilitating discussions
• Uses anecdotes, stories, analogies, real world examples, and humor effectively to illustrate key concepts
• Uses relevant and short “war stories”
• Encourages students to share their experiences

Demonstrates effective questioning skills

• Asks students open ended questions
• Appropriately directs questions to students
• Uses active listening techniques to draw information from students
• Paraphrases student questions to ensure that everyone understands

Responds appropriately to student’s need for clarification or feedback

• Identifies students with clarification and feedback needs
• Provides prompt, timely, and specific feedback focused on the performance, not the person
• Welcomes questions from the students

Provides positive reinforcement and motivational incentives

• Matches learning outcomes to student needs and goals
• Determines if the student has mastered the material by the end of the class
• Uses introductory activities to develop learning motivation
• Makes the learning experience relevant and fun
• Uses feedback and reinforcement during instruction

Uses instructional methods appropriately

• Demonstrates proficiency with a variety of teaching methods
• Limits the amount of lecture time
• Uses demonstrations
• Uses group and individual exercises
• Facilitates group discussions
• Uses instructional techniques that fit the situation
• Encourages group dynamics associated with the media selected
• Uses slides or other media as an outline for class discussion rather than lecture

Uses media effectively

• Demonstrates proficiency with the training equipment (projector, laptop, hardware, etc.)
• Troubleshoots problems associated with training equipment
• Makes effective use of flip charts, presentation slides, hardware, etc.
• Faces the audience when presenting and avoids reading the slides

Demonstrates understanding of the principles of adult learning

• Makes the training topics relevant to the student’s needs
• Makes the students partners in the learning experience

Need Some Clarification About Process Improvement

What types of activities should be considered process improvement? If we modify the templates (any major changes), do we categorize these changes as process improvement?

Merely modifying a template may or may not constitute process improvement. Your template may have changed because of external reasons (your customer wants you to use a different template) that have nothing to do with process improvement. You should explain the rationale for modification, then you would have a stronger case to demonstrate process improvement.

Process improvement suggestions can come from any number of sources:

1. Appraisal findings
2. PPQA audit findings
3. Lessons learned
4. Employee suggestions

If you couple the source of the process improvement suggestion with the actual change, usually spelled out in the Process Improvement Plan, then you have the information you need.

Process Improvement Measures

Do you have any suggestions for a size and quality measure for a Process Improvement organization?

There are other questions that need to be answered first in order to provide a proper response to your question. What does your Process Improvement process say? Examine your process to see what type of activities you have documented, where are the spots in the process you can collect data, and what data you can collect. You also have to consider your business goals and objectives for your Process Improvement organization. Then you will have some idea of the measures you can use for size and quality of your Process Improvement effort. Experts can offer you all different ideas, but if these ideas don’t support your goals and/or your process is written in such a way that it doesn’t support these measures, then you will have to consider something else.

Measurement and Analysis vs. Generic Practice 2.8

Would you expect that measures being reported for Generic Practice (GP) 2.8 on all Maturity Level (ML) 2 and ML 3 Process Areas (PAs) would follow the complete Measurement and Analysis (MA) PA? That is these GP 2.8 measures would have written objectives, operational definitions, storage, collection and analysis procedures in line with MA Specific Goal (SG) 1 and SG 2. Or can the PAs be monitored (measured) and controlled without the full breath of the MA PA?

This is a good question. So let’s take a step backwards and look at the CMMI and Generic Practice – PA Relationships. The summary table in the Generic Practice section of the model clearly states that Project Monitoring and Control (PMC) can implement GP 2.8 for all project-related processes. And MA provides general guidance about measuring, analyzing, and recording information that can be used in establishing measures for monitoring actual performance of the process. Please note that this information is GUIDANCE and part of the INFORMATIVE material. Therefore, it is not required that the org use MA for GP 2.8. HOWEVER, from a practical point of view, why would MA be part of the model if there wasn’t a requirement and expectation that it would be implemented? Since it is a ML 2 PA and you are asking about ML 3, as a Lead Appraiser I would expect to see that MA was used for defining, collecting, analyzing, and reporting both the project and process measures. Without implementing MA for the process measures, the org would be receiving little to no benefit from GP 2.8. And as I have seen GP 2.8 implemented, sometimes the process measures are embedded in the project measures that have been defined using MA.

Requirements Traceability Matrix Question

Is Requirements Traceability Matrix (RTM) a configurable item (CI) or not? Do we need to maintain its version history?

This is a question you really have to answer for yourself by first addressing some more basic questions. How critical is the RTM to the success of your project, organization, and business? What do you use the RTM for? Tracing requirements? Verifying and validating requirements? Regression testing? What value to you, the project, and the organization is the change history of the RTM? If you identify strong business needs for the RTM, then the answer to original question will become obvious.

From my perspective, the RTM is not a CI per se as it is a tool for managing your requirements and not necessarily a product component. But it is a very important, if not essential tool, and maintaining its change history could be necessary for project and organizational success.

Appraisal Team Member Qualification Requirements

What are the requirements to be an Appraisal Team Member (ATM) in SCAMPI A appraisal?

I have heard many opinions on ATM qualifications, such as:

1. Not part of process definition
2. Not involved with Process Implementation
3. Not being an SEPG member.

I am unaware of any criteria set by the SEI on this topic.

The ATM qualifications are spelled out in the SCAMPI Method Definition Document (MDD) Section 1.3.2 SELECT TEAM MEMBERS. I have extracted the pertinent text here.

Parameters and Limits
The minimum acceptable team size for a SCAMPI A appraisal is four people (including the appraisal team leader).
All team members must have previously completed the SEI-licensed Introduction to CMMI course.
With regard to engineering field experience, the team (as a group) must have an average of at least 6 years of experience, and the team total must be at least 25 years of experience in each of the disciplines to be covered in the appraisal.
With regard to management experience, the team (as a group) must have a total of at least 10 years of experience, and at least one team member must have at least 6 years of experience as a manager.
The team must, in aggregate, have representative experience in the lifecycles being appraised.

Optional Practices
Although not required in the Parameters and Limits section above, the following are considered recommended best practices and should be employed whenever feasible:

• Each member should have good written and oral communication skills, the ability to facilitate the free flow of communication, and the ability to perform as team players and negotiate consensus.
• At least half of the team members should have participated in a previous process appraisal.
• Team members should be perceived by the appraisal sponsor as credible.

Additional appraisal team member selection considerations include

• Consider the personal characteristics of individual team members (e.g., communication preferences and personality types) and how these characteristics may affect the dynamics of the team.
• Use one or more authorized SCAMPI Lead Appraisers as team members.

And there is one more ATM requirement that is documented in the MDD errata. The Appraisal Sponsor cannot be an Appraisal Team Member.

Importance of the Sub-Practices

I am trying to explain to a organization that they need to look at the sub-practice areas and make sure that the artifacts for the SCAMPI A also answer the subpractice areas. I understand that the subpractice area are a detailed description that provides guidance.

Has the organization been trained on the SEI’s 3-day Intro to CMMI class? The CMMI instructor should have explained and emphasized the role of the informative material (e. g., sub-practices). And in the words of Rusty Young, this material is “informative” NOT “ignorative.” Another way to look at the informative material is if it has no value to the model, there is no point in including it. Then the model would only consist of goal and practice statements, which would only take about 10 pages to document. The sub-practices are provided to help the reader understand the intent of the practice and goal statements.

HOWEVER, in a SCAMPI A appraisal the appraisal team will only be evaluating the required (goals) and expected (practices) components of the model, NOT the informative material (sub-practices et. al.) So you would be mistaken if you required the organization to provide evidence (Direct and Indirect) for the sub-practices. The organization only provides evidence for the goals and practices in a SCAMPI A.

CMMI Novice Question

I would like to know if all the Maturity Level 2 Process Areas must be completed for a Maturity Level 2 appraisal? There are some process areas which are not applicable to our organization.

For the CMMI for Development (CMMI-DEV) there is only one Process Area (PA) that can be designated Not Applicable (N/A), SAM. Therefore, at a minimum, REQM, PP, PMC, MA, PPQA, and CM are required for a Maturity Level 2 (ML 2) appraisal. And if the organization has outsourced some work, then SAM is also applicable.

I find it hard to believe that you can state that one or more of these PAs are not applicable to your organization. Every project has requirements to manage from the janitor to the President. Everyone works on a project. You just have to define what a project is. And then you manage the project. Everyone can define specific measures that can be used to manage the project. Everyone has some sort of configuration items or documents that have to be managed. And everyone needs some sort of objective evaluation of the process and project compliance. For a small organization, you may have combined one or more of these PAs under one person. But that does not mean these PAs are not applicable.

Now if you said that you had problems with the engineering PAs (RD, TS, PI, VER, and VAL), then I would suggest that the CMMI-DEV may not be the appropriate model constellation for your use and you should look at the CMMI for Services (CMMI-SVC) or CMMI for Acquisition (CMMI-ACQ).