Valentina (Cupać) Jemuović’s Post

NEWS UPDATE: I wrote a follow-up post that compares the presence of "holes" in test suites if we use Test First vs Test Last approaches https://www.linkedin.com/feed/update/urn:li:activity:6970277577198305280/

The coverage is like checking that you were walking through any street in the city, meaning traversing any line of code you wrote. At the end of the day, it can tell you: "Heyy great, you went everywhere today!" The caveat is that the expectation was: "Check out any street and count homeless men/women". YOUR code was just "walking" through the streets, without checking...meaning without fulfil the main expectation! When you look at the test, it was: "should check whether I traverse any street in the city"... Where is THE expectation about counting? Humm...we haven't it. So you can have 100% coverage if your code match your poor test by your poor developer! To detect that, run a mutation testing process, and it will REMOVE the lines where you count homeless people, and ... the tests will still pass indicating your poor tests quality and your full illusion of 100% coverage! To succeed mutation testing challenge in one shot, trust TDD, real TDD; because that discipline won't let you write any line/letter of production code that doesn't pass a failing test.

I used coverage goals as a forcing function to help people adopt TDD. The target was quite simple 100%, and the meaningfulness of this coverage was evaluated during code review stage If it wasn't 100% - build would fail. It is extremely painful to add coverage at this point, especially when code reviews are quite rigorous. So people would inevitably adopt TDD. Of course it is not guaranteed to work, but it did work at least once.

Valentina Cupać (Валентина Цупаћ) In Safety Software Testing this is a fully known and understood problem. Therefore you are never asked to provide "just" Code Coverage, you are always asked to provide Code Coverage derived from Requirement based tests applying Boundary and Equivalence Class Testing. Unfortunately many test books still define White Box Tests as tests with only focus on reaching the code coverage. Exactly this effect leads to the bad suprises desribed in the article. But, why do we no re-write our test books?

I disagree with this. 1. This lacks differentiation between code execution coverage and code mutation coverage. 2. Everyone can make a chart. No offense. Charts are useful to illustrate information. But without a published study, it's not scientific but hearsay. Of course that's a generic problem, despite the size of our industry, solid scientific data is hard to come buy. Still, without link to data and method of study, the chart is just this: a chart. 3. We coaches have a bias towards these negative, both of which are inherent to our job. One bias is that we focus onc the negative, as it's our job to replace it with something better. We'll see the tests without meaning. We'll not see all the good tests written as they're not troubling. And we coaches are usually called into places where our help is needed a lot. Places where stuff is going well enough are less likely to call for our help. 4. Goodhart's law isn't a law, it's an adage. It's useful, but just a model. I don't believe that as a general rule, given coverage goals, developers will automatically write bad tests. That's not a law of nature. It's a matter of priority and the overall situation. (Continued in reply.)

Ideally you want to achieve 100% *use case* coverage, though in all but the simplest application this is likely not practical. But the important point is that increasing *code* coverage does not necessarily help you increase use case coverage. And setting code coverage metrics means people will inevitably pick the easiest possible tests to write that increase code coverage - which are always going to be ones targeted at class level calls, the least useful and most brittle kind of test. I worked on a project before where there had been code coverage targets, and all the property getters had unit tests. Literally this kind of stuff void test_can_read_email() { instance.email = 'user@domain.net'; AssertEqual('user@domain.net', instance.email); } Using code coverage stats as an overall indicator of the health of a project can be ok (a project with 20% coverage probably is in trouble), but it needs to be done in a way that doesn't encourage developers to write easy but pointless tests, and that's tough.

Uncle bob seems to disagree, according to him, we should push towards 100% coverage but he admits that this is not really achievable and that it gives us false feedback, a false sense of security... this is why he recommends combining high coverage with mutation tests... Got all this from this post: https://blog.cleancoder.com/uncle-bob/2016/06/10/MutationTesting.html

Honestly this a represented as a false dichotomy. Arbitrary targets without meaningful execution and value centric metrics can lead to gaming the system, however if the proper quality controls and support is given, analysis of code coverage does provide insight for developers. That being said ,one can easily make the argument that the teams with the culture and mindset to create good testing capabilities don’t require the arbitrary target in the first place, because they write the test to catch problems early as a means of assisting themselves and their fellow team members and not because they are a tedious external requirement.🙃

In my trainings I usually bring this point across with a simple thought experiment: Let's assume the test suite follows a nice arrange-act-assert structure for all tests and you have 100% coverage. Now remove all the assertions and measure coverage again. Maybe the goal isn't the coverage after all. (I've also seen projects that were required to reach 100% coverage as a goal, and this sort of pointless testing is what you'll get for that goal when the deadline approaches.)