Audit Findings

My personal experience shows that when audits are planned monthly or at milestones, it is very difficult to take any proactive quality measures. Let's say that SQA is conducting a review at the end of the design phase just before the milestone review, and during the audit they identify that a particular design option has been selected without applying DAR, then how can they close this type of reported non-compliance by having evidence that the project team is fixing the issue? What I have seen is that sometimes the project team considers the same non-compliance as an oversight like other types of mistakes and they close the non-compliance by labeling it as a lessons learned. Although as SQA I know that there might be a chance that this same issue can occur again in the future. But apart from presenting the findings to the milestone review meeting, we have nothing to do. And the SQA group does not have insight into most of the organization's processes where this type of event occurs so we can ensure every project is following the process per the plan. So please shed some light on this topic and suggest that what type of postmortem we can do as a reactive response and what type of proactive measure we can take?

It sounds like from your description that all that SQA does is flag a problem and then the project team declares what they are going to do and makes the final decision. In other words SQA has no control over the non-compliance after identifying the problem. This is an incorrect implementation of SQA. The SQA or PPQA people are the “eyes and ears” of senior management and if there is a disagreement between the Project Team and SQA about an audit finding, that must be escalated to Senior Management for resolution. The Project Team does not have the authority to declare that an audit finding has been correctly resolved. SQA has the responsibility and authority to decide if the non-compliance is being properly identified and worked. If SQA feels that the Project Team’s action to address the non-compliance is inadequate, then SQA should not accept the closure and insist that the Project Team take appropriate corrective actions. If SQA meets resistance, then SQA should escalate the issue to top management for resolution. Resolution may involve doing nothing, training or re-training the people following the process, modifying the process, or some combination.
Hope this explanation helps.

GP 2.9 in PPQA process area - what do I need?

I'm responsible for coordinating CMMI ML2 implementation in my organization. For PPQA GP 2.9 "Objectively evaluate adherence of the process against its process description, standards, and procedures, and address noncompliance" who can be an internal auditor of my organization?

And about tools and forms for this evaluation: Could I elaborate a checklist for this evaluation? Do you have any examples of this?

An objective evaluation implies some independence from the people performing the process activities. That means the people who perform the PPQA activities do not evaluate their own work. Organizations perform GP 2.9 of PPQA in a variety of ways.

1. Someone else in the organization who is not performing the PPQA audits of REQM, PP, PMC, etc. audits the PPQA activities

2. If the company is large and has several divisions, someone who performs the PPQA activities in another division audits the PPQA activities in your division

3. An external auditor (e.g., ISO 9000) audits the PPQA activities

4. An external consultant audits the PPQA activities

5. If your company is a government supplier, then the government may be auditing the PPQA activities

6. Etc.

So in your case, you could use an internal auditor to audit the PPQA activities as long as that person is not auditing his or her own work.

As far as a checklist, that needs to be developed by the person who is auditing the PPQA activities, just like the PPQA audits develop the checklists for the other Process Areas. The checklists need to cover both a process audit and a work product audit, and it may be easier to have two checklists instead of one. The checklist needs to be based on your documented PPQA process and process assets, not the CMMI.

If you are having difficulty understanding how to write a checklist, then I strongly encourage you and your organization to have someone come and train you how to perform PPQA audits. It is very important to conduct these properly; otherwise you could face difficulties with your process improvement efforts.

PPQA Audits

Would you please distinguish the different types of audits 1) Projects, 2) Process and 3) products? Does PPQA audit the Project, Process, or Product? Or all the three? And from which area do we need to collect improvements, 1, 2, or 3? I'm confused, can you help?

You say that you are confused. I Let me try to provide an explanation for what I think you are asking about PPQA. The intent of PPQA is to act as the eyes and ears of senior management to ensure that the practitioners are following the documented processes to produce the work products. So PPQA performs two types of audits: process audits and work product audits. Now the processes being audited can be at the individual level, project level, or the organization level. And the processes being audited are not restricted to the CMMI Process Areas. The organization has to determine which processes to audit based on its business goals and objectives, so there may be processes audited in addition to the processes covered by the CMMI.

A process audit is conducted by first studying the documented process and then interviewing the practitioners to determine if they are following the process as documented.

Each process has one or more work products that are produced by following the process. These work products can be at the individual, project, or organizational level as well. The work products can be audited by sitting at a desk and reviewing the work product against the documented requirements for the work product. Is the work product produced correctly? Does it contain the proper level of information? Etc.

Both process and work product audits will identify non-compliances. By analyzing the non-compliance issues, PPQA should be able to identify the underlying causes for the issues and recommend one or more process improvement suggestions.

Improvement Checkpoints for PPQA

The PPQA Process Area deals with the quality audits of the projects as well as the support groups. The main responsibility of PPQA person is to check for process compliance and help the project team in implementing the Quality Management System (QMS) as best for the projects.

Suppose in an organization all the projects are following the organization's standard processes correctly. What additional things can PPQA can suggest to the project? For example can PPQA suggest the best tailoring for the project scope etc.?

Please let me know what types of value added tasks can be done by PPQA. OR in other words, please describe possible checkpoints that PPQA can check during an audit.

I look at PPQA and auditing the same way I look at the testers and tests. If the testers are using a specific test case and over time the test case is executing properly and no longer finding defects, then the test case needs to be examined to determine if it is still needed. Perhaps the test case is no longer valid. Perhaps it needs to be enhanced to make it more useful. Perhaps it needs to be kept and used for regression testing. Etc. The same is true for PPQA and the audits. If an audit is no longer identifying issues or non-compliances, then you have to question the audit’s effectiveness. You also have to question the frequency of conducting the audit.

Personally, I would be highly suspicious if all of your PPQA audits came out clean with no findings. Over time as your processes and procedures become institutionalized I would expect that you would find less and less non-compliances, but not zero. People make mistakes and when someone new joins the organization it will take some time before they have personally institutionalized their processes.

PPQA has two roles in the organization:

1. Process consultants
2. Process police
   

PPQA is there is enforce the organization’s processes and procedures and identify when they are not being followed. When an issue is discovered in an audit, PPQA needs to determine the root cause. Is the process broken/inadequate? Is it a training issue? Is it a personnel issue? Etc. And PPQA is also there to help people understand why and how to use the organization’s processes. So it is perfectly reasonable to have PPQA suggest process changes, tailoring options, improvement suggestions to the project, etc. while coordinating with the SEPG or EPG.

PPQA Audit Frequency

Organizations that have no history with peforming Process and Product Quality Assurance (PPQA) audits usually ask me how often should they be auditing their processes and work products. And the correct answer is "it depends", but that is usually not satisfactory. The frequency of audits depends upon the nature and severity of the quality issues associated with following the organization's processes. If there are minor findings or quality issues, then the audits don't have to occur very often, may be only once a year. But if there are major findings or issues, then they should be occuring at a higher rate until the issues go away and the processes stabilize.

Yesterday Pat O'Toole posted a message on the CMMI discussion group that take this approach one step further.

When consulting with a client on PPQA Pat suggests that PPQA use a "compliance scale" similar to that used in a SCAMPI appraisal: Fully Compliant, Largely Compliant, Partially Compliant, and Not Compliant.

This approach avoids the game playing of "just doing enough to get a 'Yes' in the audit." It also allows for a finer grading of compliance metrics and trends. And turns the audit feedback sessions into more of an internal consulting discussion than merely a "did we pass or not" exercise.

To "score" an audit, award 100 points for Fully Compliant, 75 points for Largely Compliant, 25 points for Partially Compliant, and 0 points for Not Compliant. Average the score over all of the audit items and you get the score for that particular audit.

You can average the scores of all PPQA audits conducted on a particular project to get the project-level compliance score. Hopefully you find that there is a positive correlation between projects with high compliance scores and the "success" of the project. (If there is a negative correlation you have serious cause of concern!)

I also recommend that you maintain a database (or Excel spreadsheet) with the audit items and their scores across projects and time. You can use the same scoring mechanism described above to show the average score for each audit item.

Audit items that average 90+ for 3 months are candidates for sampling - people appear to "get it" for these items. Audit items that average below some minimum threshold (60?) are probably candidates for reworking the process infrastructure - whatever you've provided isn't being used anyway, so perhaps it's time to give them something that they CAN use (and/or DO find value added).

Pat's quantitative approach makes it very clear which processes and/or projects need to be audited more frequently than others. So when a process or project scores above 90% (or so), you can reduce the audit frequency for that process or project. The default audit frequency needs to be set by the organization. Auditing once a month may be too frequent for some organizations and just right for others. The frequency should match the normal durations of your project lifecycles. Assuming a monthly frequency, if the audit score is 90+% then the frequency for that audit can go to a bi-monthly frequency. If the next time that particular audit again achieves 90+%, the audit can go to on a six month cycle. If on the other hand the score drops below 90, then audit frequency should drop back to the previous frequency. Now you have a variable audit frequency that you can tie directly to the audit results. Pretty cool!

Are external audits covered by PPQA PA?

PPQA and the CMMI do not preclude the use of external resources to perform PPQA process and work product audits. You can “outsource” this activity and still be compliant with the CMMI. However, from a practical implementation approach, this is not a good practice. Do you really want a group/company/consultant who does not have a vested interest in the success of your organization/company to be the eyes and ears of senior management? Just as you want to avoid internal bias and filtering by management internal to the organization when reporting PPQA results, you want to avoid other kinds of bias and filtering by outside factors. For example, there is the risk that an outsourced PPQA group may filter the results so they can grow their presence in your company when it is not justified. When an external company has an active sales and marketing force, there is always pressure to “grow” the account. In addition, what if the external PPQA provider has to pull their resources from you to use on a different project? That will not benefit your organization.

Here is a analogy that might be a bit of a stretch. We have a cleaning service for our home. Every three weeks or so the agency sends some cleaning ladies to our house. Sometimes there are two ladies and sometimes three. There have been occasions when the same ladies come multiple times. Whereas at other times they send new ladies. Therefore, there is no consistency from cleaning visit to cleaning visit. Some ladies do a better job than others. And despite the feedback evaluations we send in after each cleaning visit asking that they continue to send the ladies we are pleased with, there is constant churn. The churn may be due to attrition and hiring of new staff or growing business where the excellent cleaners have to support a larger client base and train new ladies. At any rate, this type of behavior could happen to you if you outsource your PPQA function. That is why I strongly recommend building an internal PPQA capability. Who better to objectively evaluate your processes, procedures, and work products than internal people? And besides, I have found that internal PPQA people are much harder on the organization, thus driving greater benefits, than external people.