183
edits
CodeCoverage/Firefly: Difference between revisions
→FAQs
Mnandigama (talk | contribs) (→FAQs) |
|||
| Line 80: | Line 80: | ||
== FAQs == | == FAQs == | ||
Q. Why are you providing line coverage | '''Q'''. Why are you providing line coverage? | ||
*Line coverage data is the most basic information which helps us to quickly identify the coverage gaps and to come up with suitable test case scenarios. | *Line coverage data is the most basic information which helps us to quickly identify the coverage gaps and to come up with suitable test case scenarios. | ||
<br>Q. What about Functional coverage ? Why is functional coverage less than 100% for files that have 100% line coverage ? Look at the following scenario to prove my point!! . There is 100% line coverage but only 49.3% functional coverage. | <br>'''Q'''. What about Functional coverage ? Why is functional coverage less than 100% for files that have 100% line coverage ? Look at the following scenario to prove my point!! . There is 100% line coverage but only 49.3% functional coverage. | ||
*As you can see the details from here CLICK, this is an artifact of multi threaded applications running along with runtime generated functions. | *As you can see the details from here CLICK, this is an artifact of multi threaded applications running along with runtime generated functions. | ||
| Line 90: | Line 90: | ||
* Function coverage is the least reliable of all modes of code coverage. For instance, if there is a function with 1000 lines of code [ this is an exaggeration ] & with an if condition in the first 10 lines which throws the execution pointer out of the function block for not meeting a condition ... any such ejection from the 'if' branch also shows 100% function coverage where as we have barely scratched the surface of that function in reality. | * Function coverage is the least reliable of all modes of code coverage. For instance, if there is a function with 1000 lines of code [ this is an exaggeration ] & with an if condition in the first 10 lines which throws the execution pointer out of the function block for not meeting a condition ... any such ejection from the 'if' branch also shows 100% function coverage where as we have barely scratched the surface of that function in reality. | ||
Q. OK smart pants !! what about branch coverage ? Is it not the most meaningful of all modes of coverage ? | '''Q'''. OK smart pants !! what about branch coverage ? Is it not the most meaningful of all modes of coverage ? | ||
*You are right that branch coverage provides true sense of coverage. | *You are right that branch coverage provides true sense of coverage. | ||
| Line 98: | Line 98: | ||
*I do not have plans in place to implement a general availability of 'branch coverage' results in this Quarter. | *I do not have plans in place to implement a general availability of 'branch coverage' results in this Quarter. | ||
Q. What are the known limitations of line coverage data ? | '''Q'''. What are the known limitations of line coverage data ? | ||
*If you have a multi conditional line like if(test)?dothis:dothat, even if both conditions are not tested, line coverage counts this line as covered. | *If you have a multi conditional line like if(test)?dothis:dothat, even if both conditions are not tested, line coverage counts this line as covered. | ||
*If you have if(this&that||whatever) , the line would be shown as covered for testing one condition even if the rest are not covered. | *If you have if(this&that||whatever) , the line would be shown as covered for testing one condition even if the rest are not covered. | ||
*Inlined functions may appear as not being covered. | *Inlined functions may appear as not being covered. | ||
*But the '''important point''' is these discrepancies would generate a statistically insignificant variations in the coverage %. | *But the '''important point''' is these discrepancies would generate a statistically insignificant variations in the coverage %. | ||
*When we pick up critical files we should not only pay attention to uncovered lines, but also to multi branch lines as well. | *When we pick up critical files we should not only pay attention to uncovered lines, but also to multi branch lines as well. | ||
<br> '''Q'''. What is the format in which you are presenting the code coverage results analysis ? | |||
Q. What is the format in which you are presenting the code coverage results analysis ? | |||
*The data would look like this | *The data would look like this | ||
| Line 118: | Line 117: | ||
*Mozilla community can work together to identify suitable scenarios to write test cases to fill the coverage gaps efficiently and effectively. | *Mozilla community can work together to identify suitable scenarios to write test cases to fill the coverage gaps efficiently and effectively. | ||
Q. What components that you recommend for first round of study ? | '''Q'''. What components that you recommend for first round of study ? | ||
*Please check the following PDF files | *Please check the following PDF files | ||
| Line 127: | Line 126: | ||
*Our goal is to identify 10 files in each of these components to improve code coverage. | *Our goal is to identify 10 files in each of these components to improve code coverage. | ||
Q. You wrote in previous posts that code coverage runs on the instrumented executable do not complete test runs due to hangs and crashes. What exactly is crashing? I don't see any inherent reason why an instrumented build would be "more sensitive to problems", unless the instrumentation itself is buggy !! | '''Q'''. You wrote in previous posts that code coverage runs on the instrumented executable do not complete test runs due to hangs and crashes. What exactly is crashing? I don't see any inherent reason why an instrumented build would be "more sensitive to problems", unless the instrumentation itself is buggy !! | ||
*one example: instrumentation changes the timing, which can change thread scheduling, which can expose latent threading bugs.We see it with Valgrind relatively often -- bugs that occur natively but not under Valgrind, or vice versa.<br> | *one example: instrumentation changes the timing, which can change thread scheduling, which can expose latent threading bugs.We see it with Valgrind relatively often -- bugs that occur natively but not under Valgrind, or vice versa.<br> | ||
| Line 133: | Line 132: | ||
*You can pick up any dynamic instrumentation tool of your choice. You can find more bugs with it as opposed to optimized build. | *You can pick up any dynamic instrumentation tool of your choice. You can find more bugs with it as opposed to optimized build. | ||
Q. Ok!! Show me how did you arrive at these conclusions. How can I do it by myself ? | '''Q'''. Ok!! Show me how did you arrive at these conclusions. How can I do it by myself ? | ||
*I would be glad to explain that. Buy me a beer :)<br> | |||
'''Q'''. Why are we doing this ? | |||
* My best answer is actually provided by Richard Feynman. | |||
* | ** http://mkngama.blogspot.com/2009/08/richard-feynmanthinking.html | ||
== Contact == | == Contact == | ||