Audit Findings

My personal experience shows that when audits are planned monthly or at milestones, it is very difficult to take any proactive quality measures. Let's say that SQA is conducting a review at the end of the design phase just before the milestone review, and during the audit they identify that a particular design option has been selected without applying DAR, then how can they close this type of reported non-compliance by having evidence that the project team is fixing the issue? What I have seen is that sometimes the project team considers the same non-compliance as an oversight like other types of mistakes and they close the non-compliance by labeling it as a lessons learned. Although as SQA I know that there might be a chance that this same issue can occur again in the future. But apart from presenting the findings to the milestone review meeting, we have nothing to do. And the SQA group does not have insight into most of the organization's processes where this type of event occurs so we can ensure every project is following the process per the plan. So please shed some light on this topic and suggest that what type of postmortem we can do as a reactive response and what type of proactive measure we can take?

It sounds like from your description that all that SQA does is flag a problem and then the project team declares what they are going to do and makes the final decision. In other words SQA has no control over the non-compliance after identifying the problem. This is an incorrect implementation of SQA. The SQA or PPQA people are the “eyes and ears” of senior management and if there is a disagreement between the Project Team and SQA about an audit finding, that must be escalated to Senior Management for resolution. The Project Team does not have the authority to declare that an audit finding has been correctly resolved. SQA has the responsibility and authority to decide if the non-compliance is being properly identified and worked. If SQA feels that the Project Team’s action to address the non-compliance is inadequate, then SQA should not accept the closure and insist that the Project Team take appropriate corrective actions. If SQA meets resistance, then SQA should escalate the issue to top management for resolution. Resolution may involve doing nothing, training or re-training the people following the process, modifying the process, or some combination.
Hope this explanation helps.

Review Activity for a Short Term Project

Our organization will be going through CMMI Maturity Level 2 Appraisal in a couple of months. I have a PPQA question. As per the PPQA Process Area (PA), we require a review of the work products (content/template) and procedures required at Maturity Level2 during the project life cycle. We have one project that is 3 months long. There are many work products that will be produced during the project development life cycle.

• Requirement documents such as SRS, Use cases, Bidirectionally traceability matrix document, change log, etc;
• Plans for all the PAs, e.g. requirements management plan, project plan, configuration plan, etc;
• Development artifacts, such as ERD, Code, UML diagrams, etc;
• QC artifacts, such as test cases, test reports, etc.
• Monitoring/controlling artifacts, such as Issue list, MoMs, Risks, etc.

How is it possible to review the work products for a 3 month project when we don't have a separate QA department and the stakeholders involved in development do the work product reviews one way or the other.

This same question holds true for reviewing procedures.

Of course, we review high priority documents, such as Project Plan, Use Cases, ERD, Application; but not all of them.

Can you help me understand what should be done for a short duration project, such that the PPQA PA requirements are met and we don't have to hire separate people just to fulfill the requirement?

The first thing that I would do is postpone your ML 2 SCAMPI A appraisal as apparently you have a major risk to achieving ML 2 since PPQA does not appear to be in place in your organization. And even if you could put PPQA in place for a 3 month project between now and your appraisals, that may still not be enough time to demonstrate institutionalization, meaning that you have a repeatable process. Essentially you will have one project using PPQA, which is one data point. And it is not possible to determine institutionalization from one data point. Your organization will be at serious risk of not achieving ML 2.

Industry average shows that PPQA is 3 – 5% of your organization. You haven’t told me how large your organization is. But if your organization is 25 people, than 1 person should be assigned to perform the PPQA practices.

I think that you are misunderstanding the differences between reviewing a work product and objectively evaluating a work product. It sounds like your project teams are already reviewing the work products. The role of PPQA is not to review the work products, but to audit the work products and processes to ensure that the work products follow the specific standards and are products according to your documented processes.

I highly recommend that you, or someone you select in your organization, take a training class on how to perform PPQA. I cannot adequately explain how to perform PPQA and answer your specific questions in this blog. The person you select for the training needs to be taught how to conduct a work product audit, how to conduct a process audit, how to plan PPQA audits, how to communicate audit results, and how to track audit non-compliances to resolution. If you don’t already have this capability in house, it will take some time to develop it internally. And I strongly advise against using an external consultant to provide this service. PPQA is for the benefit of your organization and management. It is essentially the eyes and ears of your senior management. And an external consultant may be motivated by other considerations than your best business interests if asked to provide PPQA services.

PPQA After Maturity Level 2

I work in an IT organization that achieved CMMI Maturity Level 2 several years ago (we let the rating lapse) and I was wondering if you had some ideas on the following two questions:

1) What types of activities would PPQA engage in if the org had been Maturity Level 2 (I think they could have pursued Maturity Level 3 and been close)? Please also consider that the company is pursuing other types of improvement methods and models such as lean/6-sigma and ITIL.

2) What strategies should we pursue to show the worth of PPQA? Even in the good old CMM days and SQA one of the issues I had was that it was difficult to show the practical monetary worth of these support functions; one generally had to take it on faith that PPQA/SQA delivered some degree of worth to the company. Any thoughts?

The answer to question 1 is simple. Just read the PPQA Process Area and GP 2.9. The PPQA activities include performing both process and work product audits of the project and organization processes. For Maturity Level 2 that would mean auditing your REQM, PP, PMC, SAM, MA, PPQA, and CM processes.

The answer to question 2 is a bit more difficult. Basically you are asking, what is the cost of quality? One method you can use is to look at the total cost for the project and analyze it using Crosby’s Cost of Quality Model. The total costs break down into two categories: the Cost of Quality and the Cost of Performance.

The Cost of Performance includes such things as: generating plans, documentation, and developing requirements, design, code, and integration.

The Cost of Quality breaks down further into two categories: Cost of Conformance and Cost of Non-Conformance.

The Cost of Non-Conformance includes fixing defects, reworking documents, updating source code, re-reviews, re-tests, patches, engineering changes, CCBs, external failures and fines, Customer Support, and Help Desk.

The Cost of Conformance breaks down to two more categories: Cost of Appraisal and Cost of Prevention.

The Cost of Appraisal includes reviews, walkthroughs, testing (first time), independent V&V, and Audits.

The Cost of Prevention includes training, policies, procedures, tools, planning, quality improvement, data gathering and analysis, root cause analysis, and quality reporting.

The cost of PPQA is included in the Cost of Prevention.

When you consider these definitions and cost break down, the only category that will be affected by PPQA is the Cost of Non-Conformance. When PPQA audits the processes and work products, the audits will reveal non-conformances with people following the documented processes and procedures, which lead to re-work. By addressing these non-conformances, the goal is to reduce or effectively eliminate the rework and that is where you can demonstrate the value of PPQA.

Hope this helps.

GP 2.9 in PPQA process area - what do I need?

I'm responsible for coordinating CMMI ML2 implementation in my organization. For PPQA GP 2.9 "Objectively evaluate adherence of the process against its process description, standards, and procedures, and address noncompliance" who can be an internal auditor of my organization?

And about tools and forms for this evaluation: Could I elaborate a checklist for this evaluation? Do you have any examples of this?

An objective evaluation implies some independence from the people performing the process activities. That means the people who perform the PPQA activities do not evaluate their own work. Organizations perform GP 2.9 of PPQA in a variety of ways.

1. Someone else in the organization who is not performing the PPQA audits of REQM, PP, PMC, etc. audits the PPQA activities

2. If the company is large and has several divisions, someone who performs the PPQA activities in another division audits the PPQA activities in your division

3. An external auditor (e.g., ISO 9000) audits the PPQA activities

4. An external consultant audits the PPQA activities

5. If your company is a government supplier, then the government may be auditing the PPQA activities

6. Etc.

So in your case, you could use an internal auditor to audit the PPQA activities as long as that person is not auditing his or her own work.

As far as a checklist, that needs to be developed by the person who is auditing the PPQA activities, just like the PPQA audits develop the checklists for the other Process Areas. The checklists need to cover both a process audit and a work product audit, and it may be easier to have two checklists instead of one. The checklist needs to be based on your documented PPQA process and process assets, not the CMMI.

If you are having difficulty understanding how to write a checklist, then I strongly encourage you and your organization to have someone come and train you how to perform PPQA audits. It is very important to conduct these properly; otherwise you could face difficulties with your process improvement efforts.

PPQA or VER?

I have a question. When people performa a review to assure than a coding standard is being used, is it considered a PPQA audit or a verification activity (VER)?

The correct answer is, it depends upon the nature of the review. If your documented software development process states that the coding standard is used to write code. Then a process audit of the software development process would be looking at the coding standard and determining if it was indeed being used by the developers. That would be a Process and Product Quality Assurance (PPQA) audit activity. If your documented verification process stated that a code peer review involves comparing the code to the coding standard, then that would be a Verification (VER) activity. And if your documented processes specified both of these conditions, then the answer to your question is both a PPQA audit activity and a VER activity. How you view the code review against the coding standard is therefore context dependant.

If you are asking this question because you are preparing your Direct and Indirect Evidence for your PIIDs and a SCAMPI A appraisal, then you will need to explain the context so the appraisal team will be able to correctly evaluate the evidence.

Need Some Clarification About Process Improvement

What types of activities should be considered process improvement? If we modify the templates (any major changes), do we categorize these changes as process improvement?

Merely modifying a template may or may not constitute process improvement. Your template may have changed because of external reasons (your customer wants you to use a different template) that have nothing to do with process improvement. You should explain the rationale for modification, then you would have a stronger case to demonstrate process improvement.

Process improvement suggestions can come from any number of sources:

1. Appraisal findings
2. PPQA audit findings
3. Lessons learned
4. Employee suggestions

If you couple the source of the process improvement suggestion with the actual change, usually spelled out in the Process Improvement Plan, then you have the information you need.

PPQA Audits

Would you please distinguish the different types of audits 1) Projects, 2) Process and 3) products? Does PPQA audit the Project, Process, or Product? Or all the three? And from which area do we need to collect improvements, 1, 2, or 3? I'm confused, can you help?

You say that you are confused. I Let me try to provide an explanation for what I think you are asking about PPQA. The intent of PPQA is to act as the eyes and ears of senior management to ensure that the practitioners are following the documented processes to produce the work products. So PPQA performs two types of audits: process audits and work product audits. Now the processes being audited can be at the individual level, project level, or the organization level. And the processes being audited are not restricted to the CMMI Process Areas. The organization has to determine which processes to audit based on its business goals and objectives, so there may be processes audited in addition to the processes covered by the CMMI.

A process audit is conducted by first studying the documented process and then interviewing the practitioners to determine if they are following the process as documented.

Each process has one or more work products that are produced by following the process. These work products can be at the individual, project, or organizational level as well. The work products can be audited by sitting at a desk and reviewing the work product against the documented requirements for the work product. Is the work product produced correctly? Does it contain the proper level of information? Etc.

Both process and work product audits will identify non-compliances. By analyzing the non-compliance issues, PPQA should be able to identify the underlying causes for the issues and recommend one or more process improvement suggestions.

MA and PPQA Questions

I have the following two basic queries about CMMI ML 2:

1. While writing a Metrics and measurement process, should we address the organization level metrics data consolidation and review. As ML 2 is project specific, is it proper to also document the organization level data consolidation? Also can anyone tell me, the right site for definition of metrics like requirement stability index, schedule variance, effort variance etc.?
   
2. Similarly while documenting PPQA process, is it proper to start with defining an organization level PPQA plan? I am looking for boundaries where to limit writing processes compliant to ML 2. I know that G.P 2.1 to G.P 2.10 must be in place to achieve CMMI ML 2, but the organization specific plans/areas must not be mentioned/documented at CMMI ML 2.

You sound like you are focusing on CMMI compliance rather than on your business goals and objectives. One of the basic tenets of the model is your business objectives. That is where your focus belongs. And if done properly, you will have the side benefit of being CMMI compliant. So, to address your questions:

1. When documenting your Measurement and Analysis process, you should focus on those measures that are important to you. Remember, the first MA practice SP 1.1 states “Establish and maintain measurement objectives that are derived from identified information needs and objectives.” So whatever you have identified as information needs and objectives, that should be your MA focus. At ML 2, for many organizations that are just doing this for the first time, I recommend the org take baby steps and begin with a project focus. But you don’t have to be restricted to the project, an ML 2 org may have also identified some org level measures as well. Go to the Practical Software and Systems Measurement web site for the specific measurement information you need www.psmsc.com
   
2. There are NO CMMI-imposed restrictions on the limits of PPQA. Your organization must define its own limits for the processes you are going to audit. Since GP 2.9 applies to all Process Areas, at a minimum for ML 2, PPQA applies to all of the ML 2 Process Areas you have implemented in your organization. But, if there are other processes that are critical and/or important to the success of your business, then it makes perfect sense to have PPQA audit them as well. Again, do what is right for your business.

CMMI Implementation

I recently joined a company where there is no process and management recruited me to implement the CMMI. The organization has different business units. Though everyone sits together, they work very differently. I conducted a Gap analysis based on the CMMI Level 2 processes and here are the findings:

1. Project Planning & PMC -- they create project plans and they have the regular project team meetings and they share the minutes. Each team has their own format and templates. The projects don't really do estimations. Can we satisfy the PP & PMC PA'ss without doing any estimation? I know it can't be that way, but can it be tailored?
2. Requirements Management: Some of the business units have CCBs to discuss change requests and other business units discuss requirements changes in their project team meetings. The most significant gap I found is in Requirements Traceability. Traceability of Customer requirements to the Functional Requirements and Traceability of Use cases to the Test cases are missing. Is this reason enough to fail the RM PA? One of the Engineering directors asked me how much traceability you need to satisfy this condition. At that time I said 100% of all the requirements. Then I also read from somewhere that it is OK to define that we maintain traceability for at least the MUST BE CUSTOMER REQUIREMENTS. Traceability of other requirements can be made optional. Can it be possible like that?
3. Configuration Management: The projects thought they have a CMP which is embedded in the project plan. What I found missing are the Configuration Audits ( PCA & FCA). Is it possible to satisfy the Configuration Management PA without doing Configuration Audits to check the document status, builds, and backup strategy?
4. PPQA: One of the business units has a Software Quality Assurance plan, but it is done by one of the testers from another project. As a part of PPQA the SQA will do some spot checks based on a pre-defined checklist, which includes Project Planning, Risk Management, Project Monitoring and Control, Integration and releases. But I guess this can be improved by my role as a independent software quality engineer.
5. M&A: I am very much worried with this process area, as of now the organization status is nil with respect to metrics collection, they don't have a metrics database and no metrics have been defined yet. Is it OK to start now to form a team to do some reasearch and come up with metrics definitions, deploy them and start collecting data? My question is how much data do we need to collect to satisfy this PA? And do we need to provide evidence of analyzing these collected data and show some improvements steps taken at the time of SCAMPI apprisals? How long will it take in general to satisfy the Measurement & Analysis PA?
6. SAM: can we tailor this PA if we are not dealing with suppliers? If yes how can it be possible?
   

The directive from the Leadership team is to acheive CMMI Level 3 by end of 2009. I was baffled to hear this. Under these circumstances, what are the chances of getting CMMI Level 3 or my traget is at least CMMI Level 2? That is what my initial target. Can I acheive CMMI Level 2 by the end of 2009? If so, what are the things I need to address?

Here are some of the things I have already started:

1. CMMI Overview training to all the teams
2. Dailogue session on metrics identification
3. Looking into some Requirements tools which provide Traceability
4. Need to push the project team to have configuration audits.

In addition we already have established a process data base and the processes are defined and templates are being used from the parent organization. Since our company is a multi-site company, one of our counter parts has already acheived CMMI Level 3. We will be using the same process database and their templates. I thinking of providing their training on each Process Area as well. Is this a correct way to use the processes and templates of our parent organization? If not, do we need to establish our own local process data base?

First of all, I sympathize with you and the challenges you face. I applaud the fact that you had the foresight to conduct a Gap Analysis of the organization. You have highlighted a number of key weaknesses within the organization. However, to provide you meaningful feedback on all of your points would require working directly with you and your company.

1. What is your CMMI experience? Have you taken the SEI’s Introduction to CMMI class? If not, I strongly recommend that you and possibly those others in your company who are responsible for your processes take the three day class. The class should provide you a more thorough understanding of the CMMI, its interpretations, and material for constructing an internal Overview class.
2. You have identified some serious deficiencies within the organization in all of the ML 2 Process Areas. These need to be analyzed, addressed, corrected, solutions implemented, and then re-evaluated some months into the future before you can consider a formal SCAMPI at any Maturity Level. The length of time before the next evaluation is a function of a number of factors: number of people in the organization, number of projects, typical project duration, how much time and other resources are dedicated to process improvement, etc.
3. The organization needs to first implement Maturity Level 2 to form a firm foundation before considering moving to Maturity Level 3. The issues you have identified are fairly typical. Basically, it sounds like your organization does not perform PPQA or MA and is challenged with Project Management, Requirements Management, and Configuration Management. The first steps here should be to identify the necessary skills-based training classes you need to bring in-house and train your staff on these concepts. If you just purchase tools and push for audits, you most likely will not achieve the desired effect. You have to understand your process first and the reasons for why it is important to perform each of the steps.
4. Since another division has already achieved ML 3, it is a good idea to learn from their mistakes. But be very careful of the temptation to “clone” their processes and procedures. You have to implement the processes and procedures that match the way you conduct business today.
5. Based on what you have outlined, and given how much time and effort it could take just to address the ML 2 issues, I would say that ML 3 is out of the question for 2009. You could conduct a ML 3 SCAMPI A by the end of this year, but in all likelihood it would not be successful.
6. The best suggestion I have for you is to hire a CMMI consultant and Lead Appraiser to provide you with the proper advice and guidance. Otherwise, you could be spending a lot more time and effort than originally anticipated.

Supplier Agreement Management Question

I have a question related to Supplier Agreement Management (SAM) SP 2.2 - Monitor Selected Supplier Processes. What is the basic intent of this practice and in what scenario does it fit in? It specifies "...situations of tight alignment between processes implemented by the supplier and those of the project..." - which is normally not the case in most (small) projects (as I know). I hope here we are not including "Acceptance" and "Transition" as aligned processes. Also it seems redundant to me with SP 2.1 - Execute the Supplier Agreement because in the contract/ SOW, it is usually mentioned how the supplier needs to monitor his processes (frequency to perform process audits etc.) and the frequency/condition when the customer may ask for a process audit/assessment. So doesn't executing the Supplier agreement covers these two practices?

I can see where you might have some confusion concerning these two SAM practices. SAM SP 2.1 says “Perform activities with the supplier as specified in the supplier agreement.” And your confusion comes about from sub-practice 1 “Monitor supplier progress and performance (schedule, effort, cost, and technical performance) as defined in the supplier agreement.” In essence SAM SP 2.1 is all about performing Project Monitoring and Control (PMC) over the supplier, which should be spelled out in the supplier agreement. You are basically acting as the Project Manager for the supplier by monitoring and controlling their project and technical performance. There is no intent to perform any Process and Product Quality Assurance (PPQA) audits or activities to support this practice.

In contrast SAM SP 2.2 says “Select, monitor, and analyze processes used by the supplier.” This practice is where you perform PPQA activities (process audits and work product audits) on selected supplier processes that are critical to the success of your project and business. Again, this ability to perform PPQA on the supplier must be specified in the supplier agreement. But you want to have the freedom to select any supplier process, so don’t indicate specific processes in the supplier agreement. For example you might decide to monitor and analyze how your supplier performs peer reviews or how they manage their requirements. If you have your supplier doing small projects (less than a month in duration) you may not have many opportunities to perform PPQA on a given supplier project. This situation is the same as when you have small projects done completely in house. There is no hope or expectation that you will be able to perform PPQA activities on every small project.

Look back at my blog on PPQA Audit Frequency http://ppqc.blogspot.com/2008/05/ppqa-audit-frequency.html for a simple way to adjust the frequency of the PPQA audits based on the quality issues discovered. You can use this same approach to determine the frequency of conducting PPQA activities on your supplier, assuming that you supplier regularly performs small projects for you.

Improvement Checkpoints for PPQA

The PPQA Process Area deals with the quality audits of the projects as well as the support groups. The main responsibility of PPQA person is to check for process compliance and help the project team in implementing the Quality Management System (QMS) as best for the projects.

Suppose in an organization all the projects are following the organization's standard processes correctly. What additional things can PPQA can suggest to the project? For example can PPQA suggest the best tailoring for the project scope etc.?

Please let me know what types of value added tasks can be done by PPQA. OR in other words, please describe possible checkpoints that PPQA can check during an audit.

I look at PPQA and auditing the same way I look at the testers and tests. If the testers are using a specific test case and over time the test case is executing properly and no longer finding defects, then the test case needs to be examined to determine if it is still needed. Perhaps the test case is no longer valid. Perhaps it needs to be enhanced to make it more useful. Perhaps it needs to be kept and used for regression testing. Etc. The same is true for PPQA and the audits. If an audit is no longer identifying issues or non-compliances, then you have to question the audit’s effectiveness. You also have to question the frequency of conducting the audit.

Personally, I would be highly suspicious if all of your PPQA audits came out clean with no findings. Over time as your processes and procedures become institutionalized I would expect that you would find less and less non-compliances, but not zero. People make mistakes and when someone new joins the organization it will take some time before they have personally institutionalized their processes.

PPQA has two roles in the organization:

1. Process consultants
2. Process police
   

PPQA is there is enforce the organization’s processes and procedures and identify when they are not being followed. When an issue is discovered in an audit, PPQA needs to determine the root cause. Is the process broken/inadequate? Is it a training issue? Is it a personnel issue? Etc. And PPQA is also there to help people understand why and how to use the organization’s processes. So it is perfectly reasonable to have PPQA suggest process changes, tailoring options, improvement suggestions to the project, etc. while coordinating with the SEPG or EPG.

PPQA Audit Frequency

Organizations that have no history with peforming Process and Product Quality Assurance (PPQA) audits usually ask me how often should they be auditing their processes and work products. And the correct answer is "it depends", but that is usually not satisfactory. The frequency of audits depends upon the nature and severity of the quality issues associated with following the organization's processes. If there are minor findings or quality issues, then the audits don't have to occur very often, may be only once a year. But if there are major findings or issues, then they should be occuring at a higher rate until the issues go away and the processes stabilize.

Yesterday Pat O'Toole posted a message on the CMMI discussion group that take this approach one step further.

When consulting with a client on PPQA Pat suggests that PPQA use a "compliance scale" similar to that used in a SCAMPI appraisal: Fully Compliant, Largely Compliant, Partially Compliant, and Not Compliant.

This approach avoids the game playing of "just doing enough to get a 'Yes' in the audit." It also allows for a finer grading of compliance metrics and trends. And turns the audit feedback sessions into more of an internal consulting discussion than merely a "did we pass or not" exercise.

To "score" an audit, award 100 points for Fully Compliant, 75 points for Largely Compliant, 25 points for Partially Compliant, and 0 points for Not Compliant. Average the score over all of the audit items and you get the score for that particular audit.

You can average the scores of all PPQA audits conducted on a particular project to get the project-level compliance score. Hopefully you find that there is a positive correlation between projects with high compliance scores and the "success" of the project. (If there is a negative correlation you have serious cause of concern!)

I also recommend that you maintain a database (or Excel spreadsheet) with the audit items and their scores across projects and time. You can use the same scoring mechanism described above to show the average score for each audit item.

Audit items that average 90+ for 3 months are candidates for sampling - people appear to "get it" for these items. Audit items that average below some minimum threshold (60?) are probably candidates for reworking the process infrastructure - whatever you've provided isn't being used anyway, so perhaps it's time to give them something that they CAN use (and/or DO find value added).

Pat's quantitative approach makes it very clear which processes and/or projects need to be audited more frequently than others. So when a process or project scores above 90% (or so), you can reduce the audit frequency for that process or project. The default audit frequency needs to be set by the organization. Auditing once a month may be too frequent for some organizations and just right for others. The frequency should match the normal durations of your project lifecycles. Assuming a monthly frequency, if the audit score is 90+% then the frequency for that audit can go to a bi-monthly frequency. If the next time that particular audit again achieves 90+%, the audit can go to on a six month cycle. If on the other hand the score drops below 90, then audit frequency should drop back to the previous frequency. Now you have a variable audit frequency that you can tie directly to the audit results. Pretty cool!

Are external audits covered by PPQA PA?

PPQA and the CMMI do not preclude the use of external resources to perform PPQA process and work product audits. You can “outsource” this activity and still be compliant with the CMMI. However, from a practical implementation approach, this is not a good practice. Do you really want a group/company/consultant who does not have a vested interest in the success of your organization/company to be the eyes and ears of senior management? Just as you want to avoid internal bias and filtering by management internal to the organization when reporting PPQA results, you want to avoid other kinds of bias and filtering by outside factors. For example, there is the risk that an outsourced PPQA group may filter the results so they can grow their presence in your company when it is not justified. When an external company has an active sales and marketing force, there is always pressure to “grow” the account. In addition, what if the external PPQA provider has to pull their resources from you to use on a different project? That will not benefit your organization.

Here is a analogy that might be a bit of a stretch. We have a cleaning service for our home. Every three weeks or so the agency sends some cleaning ladies to our house. Sometimes there are two ladies and sometimes three. There have been occasions when the same ladies come multiple times. Whereas at other times they send new ladies. Therefore, there is no consistency from cleaning visit to cleaning visit. Some ladies do a better job than others. And despite the feedback evaluations we send in after each cleaning visit asking that they continue to send the ladies we are pleased with, there is constant churn. The churn may be due to attrition and hiring of new staff or growing business where the excellent cleaners have to support a larger client base and train new ladies. At any rate, this type of behavior could happen to you if you outsource your PPQA function. That is why I strongly recommend building an internal PPQA capability. Who better to objectively evaluate your processes, procedures, and work products than internal people? And besides, I have found that internal PPQA people are much harder on the organization, thus driving greater benefits, than external people.