Showing posts with label Coverage. Show all posts
Showing posts with label Coverage. Show all posts

Friday, 14 December 2007

Are You Reporting Wrong Coverage Number?

If you are not giving weight to your items in your coverage groups, the overall functional coverage number you get on Specman coverage report might be wrong. Let me explain it using an example. Take a look at following code.

<'
struct coverage_example
{
event cover_e;
a : bool;
b : uint (bits: 2);
cover cover_e is
{
item a;
item b;
cross a, b;
};
run() is also
{
emit cover_e;
};
};
extend sys
{
cov_ex : coverage_example;
keep soft cov_ex.a == TRUE;
keep soft cov_ex.b == 0;
};
'>

If you load above code, issue a test command and see the coverage report, it will show you 28%, which is wrong in fact. Specman calculates the overall coverage numbers hierarchical. For the above code, it calculates as following.

For item a:
total bucket = 2
total hit = 1
So, coverage number = 50%

For item b:
total bucket = 4
total hit = 1
So, coverage number = 25%

For cross a, b:
total bucket = 4*2 = 8
total hit = 1
So, coverage number = 12.5%

For Overall:
total bucket = 3 (item a, item b and cross a,b)
total hit = (50 + 25 + 12.5) = 87.5
So, coverage number = 29.166% (87.5/3)

But, in fact, it should be calculated as following.

Total bucket in the coverage group = 2 + 4 + 8 = 14
Total hit = 1 + 1 + 1 = 3
So, overall coverage should be (3*100/14 = 21.4285%)

This wrong calculation can be avoided by doing following modification in the code. See highlighted "green" code below.

<'
struct coverage_example
{
event cover_e;
a : bool;
b : uint (bits: 4);
cover cover_e is
{
//Weight is total number of buckets
item a using weight=2;
item b using weight=4;
cross a,b using weight=8;
};
run() is also
{
emit cover_e;
};
};
extend sys
{
cov_ex : coverage_example;
keep soft cov_ex.a == TRUE;
keep soft cov_ex.b == 0;
};
'>

But, all the time, applying weight to every item is not possible, as some code may be imported from third party vendor or some code might be read-only to the user. Also whenever the number of buckets change, user needs to update the weight of particular item. This might not be feasible sometimes.

To avoid this, there is a utility called bucket_counter which was delivered by Clemens Mueller from Cadence. This e code will report total number of buckets in each item and total number of bucket hits. This simple utility will help you calculate the exact number of functional coverage you have.

Please click here to download the code.

Just load the ecode along with your environment and read the coverage file. Then issue "sys.report_buckets()". It will report total number of buckets in each item, and total number of hits, as shown below.

Valid Space size for item a is 2 buckets
Valid Space size for item b is 4 buckets
Valid Space size for item cross__a__b is 8 buckets
total coverage buckets: 14
total coverage buckets hit: 3

Wednesday, 12 December 2007

Cover "check that" and "expect" statement

If you want to write a checker coverage grid, Cadence Specman team has come up with a solution to put "check that" and "expect" statement in the coverage grid.

Please find this shareware at following link. This works for Specman 6.0 onwards.

http://sandipgor.googlepages.com/shr_chkcov_1.1.tar.gz

Friday, 7 December 2007

100% functional coverage is not enough

Let me address this age old question once again. As we all know that 100% functional coverage does not mean bug free design, but it surely adds to the confidence of the designer. It is one of the criteria for verification completion. So, what are the other criteria people use for their verification completion. Most of us will answer that 100% code coverage achievement and few of us will also add that closing all the pending review items derived from verification review. But is it enough? We still feel that it is not enough. Deep inside our hearth, we still feel the need of some more testing/verification.

Well, this feeling will be there because of the nature of the verification task. But, let me come to the technical aspect of the 100% functional coverage. If we ask our self, what is the functional coverage, we will be scratching our head to find exact wordings to explain the functionality. Let me try to explain it in my words. For me, Functional Coverage Points are nothing but the measure of all possible combinations of inputs which can be applied to the design. So, in our functional coverage grid, we always cover all the input stimulus parameters and we assume that the output behaviour will be checked somewhere else in the verification environment. So, 100% functional coverage means we have stimulated the design with all possible stimulus. But this does not guarantee that all outputs and behaviors of the design are checked. So, 100% functional coverage does not mean bug free design. But, if we also have one more coverage grid which could be checker coverage grid, then, we can definitely say that we have applied all possible combinations of inputs and we have also checked all possible combinations of outputs. Even we need co-relation between these two grids. I mean, for each functional coverage point (one possible combination input), there should be at lease one checker coverage point (one possible output checker). And for each checker coverage point, there should be at least one functional coverage point. Thus co-relating these two grids will definitely help to improve the confidence in the design.

So, in my opinion, verification completion declaration should be based on following criteria.

1) All test cases passed.
2) 100% Code Coverage (of course with some waivers)
3) 100% Functional Coverage (and again, with some waivers)
4) 100% Checker Coverage
5) Correlation between Functional Coverage Grid and Checker Coverage Grid.