Tuesday, August 12, 2008

Defect Removal Efficiency (DRE)

I am a Defect Prevention team member in our CMMI Maturity Level 5 company. We are trying to plot the Defect Removal Efficiency (DRE) metric in our projects. I have a few questions:

  1. Our DRE is based on phase level defect injection and detection concepts. If we go closely by the definition of the phase "testing/QA", bugs should not be injected in testing phase, as this is detection activity. If not then any examples can be useful for me to understand.
  2. Can a common DRE format can be utilized for all kinds of projects as in sustaining engineering, maintenance, development and pure testing? Specifically what do to we do for testing or pure QA type of projects?
  3. DRE is scoped for defects leaked by QA and reached to customer. How we can address defects in other lifecycle phases i.e. deployment and post production activities?

These are very interesting questions. Here is my two cents worth:

  1. Your assumption that testing/QA phase is not a source of defects may or may not be a good assumption. Test cases can be a source of defects as well as the product. A test case that does not find any defects may in fact itself be defective. The purpose of a test case is to find defects in the product. In this case there might be undetected defects slipping through the test cases or erroneous test results. On the other hand testing can sometimes identify defects in the product that may not actually exist. I encountered this situation years ago in a small software development company when they hired a new tester. The company had no documented procedures and testing criteria. So the new tester did the best job he could under these circumstances and based his testing on the published User’s Manual. He proceeded to find tons of defects in the product. That got management’s attention until they realized that all of these “new” defects had been previously addressed. It turned out that the tester was unwittingly using an obsolete version of the User’s Manual. So that mistake invalidated all of his test results. Either way, each of these two situations would contaminate your DRE calculations. Therefore, I would be very careful in making the assumption that the testing/QA phase is not a source of defects, unless you have thoroughly peer reviewed each test case and validated each test result to ensure that what I have described has not happened.
  2. In the context of the CMMI, QA usually means for most of us Lead Appraisers PPQA – conducting process audits and work product audits, not testing. So I am not clear as to the distinction you are trying to make between testing and "pure QA". It appears that in your organization they may be two aspects of the same activity. Personally, I think that you could use a common DRE format across all types of projects. However, it doesn’t make sense to mix the data for the various projects. So you will have to be very careful to keep the data segregated, otherwise you won’t be able to draw the proper conclusions. For example, the DRE for new development will be different than for sustaining engineering, different from a testing project, etc.
  3. I think that you answered your question in your statement of the scope. Just change the definition of the scope for your DRE. You could define the DRE scope on a lifecycle phase by phase basis and then you would be able to measure it for deployment and post-production activities. You could also define a DRE for the entire project lifecycle as well.

6 comments:

QA_Tester said...

If we use DRE for the whole project can we take review defect as defects and then calculate the DRE for other phases?

But can we really compare a design phase DRE to Requirement phase DRE?

Henry Schneider said...

The approach for using DRE would be a function of the organization's Maturity Level. If you are talking about a ML 3 organization, then the organization might consider a Defect Removal Efficiency measure for the entire project. Though this measure would tell the organization something about its defect removal activities, this measure is an aggregate measure that probably won't provide any insight into which phases or processes are introducing defects and which are removing them. In one case, the process introducing a defect could be cancelled out by the process removing the defect and the net DRE could indicate no problem at all. But it is a place to start and become knowledgeable about DRE.

So for ML 4 and ML 5, you would be better served by looking at the DRE for a particular process or sub-process or by looking at the DRE for a lifecycle phase. Then you will have some interesting data that can help improve your processes. Without having a good understanding of your processes and organization, I can't determine if you could compare a Design Phase DRE to a Requirements Phase DRE. They may or may not be comparable. The best practice is to keep them separate and clearly labeled so there is no confusion. A High Maturity organization may want to perform some correlation analyses between these two measures to determine if they can be compared.

QA_Tester said...

Thanks Henry for that valuable input.

Our organization is going for ML3 and we have a confusion here whether we have to go for Phase wise DRE for the whole project(comparing Req,Design,Coding,Testing) or just use DRE for testing phase comparing the rounds of testing.

Henry Schneider said...

Dear QA_Tester,
What you have to consider is your business and not the CMMI or a Maturity Level. Go back to basics and talk about why you are considering DRE. What information about your projects and your organization does DRE provide? Right now for your organization, is it important for you to know the DRE for the full project lifecycle? Or is it only important for testing.

You must start with your business goals and objectives. These goals and objectives should provide you the information you need to determine the coverage for DRE.

Perhaps you could start just looking at DRE for the testing phase and determine what this measure tells you. Based on the results, you may need to expand DRE to more phases or the entire lifecycle. However, you must have higher management involved and let your business needs determine what lifecycle phase(s) you cover with DRE.

Have you already achieved Maturity Level 2? I suspect that you may not have. A proper implementation of Measurement and Analysis should provide the answer to your questions about DRE.

Satyarth said...

For data fixing Task How to calculate DRE

Is there threshold for DRE

Henry Schneider said...

Dear merimaakaliya,
This blog http://geekswithblogs.net/dthakur/archive/2004/07/05/7617.aspx has a short definition of DRE and a link to a DRE calculator. If there is a threshold for DRE, that would be determined by your organization's goals and objectives.