Unit Testing - The Merit of Writing Tests First is Questioned

[Robert Cramer]

So I'm looking at this test-driven development (TDD) paradigm, and I like
the idea of having automated tests. That idea is not new.

But according to the TDD approach, we are to write our unit tests FIRST -
even before writing the code that the test is ultimately going to
test/verify.

I do see some merits of TDD - and doing the testing first - but what I think
is a practically guaranteed artifact of this approach is that many if not
most unit tests will become obsolete [and therefore have to be rewritten]
with any non trivial refactoring effort (which is also part of the whole TDD
paradigm). So, if I understand correctly, we are to write unit tests, then
write our code, then refactor the code almost immediately - and bam - just
like that, many of the unit test will also have to be rewritten or
refactored because the code being tested has been refactored [perhaps
refactored out of existance as we remove duplication, encapsulate stuff that
changes and recompose it with its original pre-refactoring class, etc].

Am I missing something?

Your thoughts and opinions are appreciated - specifically on the idea that
if I decide to write my unit tests first (per mainstream TDD), then many of
those tests won't be around [or will, themselves, need to be refactored] by
the time version 1of the system goes live.

Thanks!

 

[Bill Woodruff]

I hope it will not seem patronizing (or off topic) for me to say how much I
appreciate the eloquence, technical depth, and wisdom I find in your
responses here, as in your response to R Cramer's OP on this topic.

best, Bill Woodruff (fossil relic who was once a PostScript guru)
dotScience
Chiang Mai, Thailand

 

[Cowboy \(Gregory A. Beamer\)]

So I'm looking at this test-driven development (TDD) paradigm, and I like
the idea of having automated tests. That idea is not new.

But according to the TDD approach, we are to write our unit tests FIRST -
even before writing the code that the test is ultimately going to
test/verify.

This is the purist approach and it has both merits and detractors. Most of
the detractors are paradigm shifts rather than actual impediments.

I do see some merits of TDD - and doing the testing first - but what I
think is a practically guaranteed artifact of this approach is that many
if not most unit tests will become obsolete [and therefore have to be
rewritten] with any non trivial refactoring effort (which is also part of
the whole TDD paradigm).

If you do a code first, design second approach, this is a risk. If you take
time to brainstorm your application and figure out use cases, flows, etc.,
there is very little danger. Organization is the key, but then again, you
should not be building an application that you do not understand.
Unfortunately, this is VERY common in the industry, as managers try to limit
design time, throw out specs that are worthless, etc.

BTW, refactoring, for the most part, does not break tests, as the majority
of it is either a) refactoring out duplicate code or b) refining current
code. If you are changing interfaces, etc., you are not really refactoring,
you are altering. Let's take an example:

if(x=1) {}
elseif (x=2) {}
elseif (x=3) {}
elseif (x=4) {}
else {}

Each of the conditions has to be walked, which is inefficient. A common
refactor would be to change to a switch (Select ... Case) in VB.

switch(x)
{
case 1: {}
case 2: {}
case 3: {}
case 4: {}
default: {}
}

This creates a state machine. That is a refactor. But, note, that there is
nothing happening here that changes my public interface, so the test still
works.

In fact, since most refactoring is trying to improve, not alter, code, NOT
having tests is a major detriment, as your tests, when written properly,
protect you against new bugs in old code.

Does refactoring ever break tests? Certainly. But, more often than not, it
is because your refactoring revealed a flaw in your test. Either you were
running the test for the wrong behavior or you have set up an overly
optimistic test, or similar.

So, if I understand correctly, we are to write unit tests, then write our
code, then refactor the code almost immediately - and bam - just like
that, many of the unit test will also have to be rewritten or refactored
because the code being tested has been refactored [perhaps refactored out
of existance as we remove duplication, encapsulate stuff that changes and
recompose it with its original pre-refactoring class, etc].

Am I missing something?

Yes. You are missing the core of TDD.

1. Design - Determine what you are building
2. Decide - Agree on functionality to be built
3. Divide - Break work down into bite sized chunks
4. Write Tests - write a test on the expected behavior
5. Write code to pass the test - red to green
6. Refactor - remove any "code smells" (repeat code being the most common)

If you start with step 4, you will probably end up rewriting your tests as
you really have no clue what you are building. In that case, return to step
one. I can even add a pre-step, which is brainstorm.

If you want to get a good idea of how to start design for customer facing
apps (and I include internally facing apps here), you can watch the free
user experience videos at http://sessions.visitmix.com (need to have
SIlverlight installed). Some really nice sessions on design are UX04, UX05,
UX06 and UX07 (in that order), which are videos from Adaptive Path. The
middle of each is boring, as people are doing exercises, so fast forward to
the last 10-15 minutes when they start that.

Another thing to consider is unit tests are just that. They test a single
routine. You can expand this a bit, but if you are system testing, your
tests are too broad.

Let's take an example, which you might be thinking fits your "problem" with
TDD and why it does not.

You have a test that tests a User object. In your application, you will pull
the information from the database and diplay on a web page. Something like:

ASP.NET page
Welcome <asp:Label id="nameLabel" runat="server"/>

Code behind
nameLabel.Text = user.UserName;

You have a user object that is filled by some facade method. A quick stub
would be something like:

public static User GetUserFromId(string userID)
{
//Stuff to test id and pull user object from database here
return user;
}

So, you set up the following test (using Visual Studio unit test format):

[TestMethod]
public void TestGetUserFromIdSuccess()
{
string userID = "G5F765";
string expectedName = "Greg Beamer";

User actual = UserFactory.GetUserFromId(userID);

Assert.AreEqual(expectedName, actual.Name, "Name is different");
}

Suppose, however, that the test data no longer contains the same name for
that user. It fails. But, you are not supposed to be testing the database
retrieval at the business layer. You should be injecting the dependency, so
it always returns the right name. This may sound like "cheating", but you
are only testing the code in THAT routine.

Now, if you change the entire business layer and how you retrieve data, you
will break the application. But that is more likely to happen if you have
not designed it up front, at least in most instances.

Your thoughts and opinions are appreciated - specifically on the idea that
if I decide to write my unit tests first (per mainstream TDD), then many
of those tests won't be around [or will, themselves, need to be
refactored] by the time version 1of the system goes live.

I disagree. In fact, I have used TDD for years now and find that 90%+ of my
tests are still around. There are a few that have changed, but if you design
first and then code, you find that you eliminate a lot of useless rabbit
holes before you even start coding.

--
Gregory A. Beamer
MVP, MCP: +I, SE, SD, DBA

Subscribe to my blog
http://gregorybeamer.spaces.live.com/lists/feed.rss

or just read it:
http://gregorybeamer.spaces.live.com/

*************************************************
| Think outside the box!
|
*************************************************

 

[Robert Cramer]

Thank you Gregory and Peter for your thoughtful and helpful responses.

RE:
<<
1. Design - Determine what you are building
2. Decide - Agree on functionality to be built
3. Divide - Break work down into bite sized chunks
4. Write Tests - write a test on the expected behavior
5. Write code to pass the test - red to green
6. Refactor - remove any "code smells" (repeat code being the most
common)

If you start with step 4, you will probably end up rewriting your tests
as you really have no clue what you are building.
This is exactly my concern [starting with step 4]. I have been developing
non trivial business apps professionally for more than 12 years (complete
lifecycle) and recently attended a TDD seminar in which the unit test was
presented as driving everything. When questions were raised about design
(step 1, according to you -- and I agree with you), that was downplayed with
the explanation that the unit tests will inform your design decisions. Yes -
the creation of unit tests are to inform our design decisions according to
the guys leading the session. In fact, according to the presenters, that's
the "driven" part of test-driven development" -- the tests DRIVE the
Development. That's where they lost credibility with me. But I do like the
idea of having automated tests so that I can periodically run the tests to
quickly verify that life is good with the system after modifications have
been made. But we don't need TDD in order to have automated tests. And I
prefer to drive my designs from, well, a good focused design effort (not
some suite of tests). So rather than dismissing TDD outright, I decided that
the leaders of the sessions were confused, incompetent, or both. So I'm
still wanting to know the value of TDD. The resources I've found seem to
center on the "almighty unit test" as if that should drive everything else -
rather than the automated set of unit tests being part of a more
comprehensive and standard development cycle (as promoted, for example in
the book, Code Complete).

Thank you again for your thoughtful responses. I'll try to take another
serious look at TDD - or at least borrow some ideas from it even if I don't
fully embrace it.

-RC

 

[Jon Skeet [C# MVP]]

This is exactly my concern [starting with step 4]. I have been developing
non trivial business apps professionally for more than 12 years (complete
lifecycle) and recently attended a TDD seminar in which the unit test was
presented as driving everything. When questions were raised about design
(step 1, according to you -- and I agree with you), that was downplayed with
the explanation that the unit tests will inform your design decisions. Yes -
the creation of unit tests are to inform our design decisions according to
the guys leading the session.

I'd agree with that, actually.

Tests shouldn't necessarily drive *architecture* but I find it's
reasonable that they drive *design*.

There's a simple reason here - it generally means you end up with a
class which is easy to use as well as easy to write. You're immediately
thinking from the *caller's* point of view instead of from an
implementation side. You think "what do I want to do with this class"
rather than "what do I want this class to provide".

I know that sounds like a subtle distinction, but I've found it really
makes a big difference. Classes which have been designed by writing
tests *tend* to be easier to use later on. They also *tend* to
naturally express their dependencies in nice ways (e.g. interfaces).

That's just in my limited experience though.

 

[Robert Cramer]

Responses inline:

Tests shouldn't necessarily drive *architecture* but I find it's
reasonable that they drive *design*.

There's a simple reason here - it generally means you end up with a
class which is easy to use as well as easy to write. You're immediately
thinking from the *caller's* point of view instead of from an
implementation side. You think "what do I want to do with this class"
rather than "what do I want this class to provide".

I know that sounds like a subtle distinction, but I've found it really
makes a big difference.

Interesting - I can easily see how that only *sounds* subtle, but I can
easily recognize how changing the perspective ([what can I do with this
class?] as opposed to [what can this class do?]) could result in some very
different, perhaps better, design decisions.

I was thinking architecture all along. My experience is that the term
"architecture" is so heavily overloaded or outright misused that I
completely missed the distinction between design and architecture. Great
catch.

Classes which have been designed by writing
tests *tend* to be easier to use later on. They also *tend* to
naturally express their dependencies in nice ways (e.g. interfaces).

Okay, that's believeable, and makes TDD worth a very close 2nd look on my
part. I guess I need to really get past the presentation.

That's just in my limited experience though.

I appreciate your input. Do you have any suggested readings? Yes, I know I
could Google this, and I have - but I'd like a recommendation from someone I
respect.

-RC

 

[Jon Skeet [C# MVP]]

I know that sounds like a subtle distinction, but I've found it really
makes a big difference.

Interesting - I can easily see how that only *sounds* subtle, but I can
easily recognize how changing the perspective ([what can I do with this
class?] as opposed to [what can this class do?]) could result in some very
different, perhaps better, design decisions.

That's handy, because I suspect I wouldn't be able to really explain it
if you didn't "get it" so easily

I was thinking architecture all along. My experience is that the term
"architecture" is so heavily overloaded or outright misused that I
completely missed the distinction between design and architecture. Great
catch.

No problem. Of course, due to that overloading it's quite possible that
my understanding of the words "architecture" and "design" aren't the
same as yours either!

Okay, that's believeable, and makes TDD worth a very close 2nd look on my
part. I guess I need to really get past the presentation.


I appreciate your input. Do you have any suggested readings? Yes, I know I
could Google this, and I have - but I'd like a recommendation from someone I
respect.

Well, I haven't read it since the release (must get round to it some
time) but "Test-Driven" (http://www.manning.com/koskela/) was excellent
when I read a preview.

Unfortunately it's now been a few years since I first learned about
TDD, so I can't remember what the process was like... I do remember
that learning about mocking was a lightbulb moment. Otherwise
dependencies kill you...

 

[news.microsoft.com]

I worked with Thoughtworks on a project for a year, and though I grokked
TDD, I think that this: http://dannorth.net/introducing-bdd is a much better
way of thinking about it.

 

[Lasse Vågsæther Karlsen]

I know that sounds like a subtle distinction, but I've found it really
makes a big difference.

Interesting - I can easily see how that only *sounds* subtle, but I can
easily recognize how changing the perspective ([what can I do with this
class?] as opposed to [what can this class do?]) could result in some very
different, perhaps better, design decisions.

That's handy, because I suspect I wouldn't be able to really explain it
if you didn't "get it" so easily

I've had a similar discussion recently and the best example a collegue
came up with was: take a bike, and describe how it works, then... as an
alternative, take the cyclist, and describe what he wants to do.

Turns out there are many solutions to the cyclists problem, and it's
actually very hard to describe a bike to someone that hasn't seen it. In
other words, the chances you'll make a bike for someone who doesn't know
what a bike is, and actually solve his problem, is going to be smaller
than if you ask the cyclist first, what do you want to do and how do you
want to do it, and design the bike (or whatever) to fit those needs and
usage patterns.

No problem. Of course, due to that overloading it's quite possible that
my understanding of the words "architecture" and "design" aren't the
same as yours either!

I can second that. I have rewritten one big class for a pet project
three times and the last time I did the time to actually write about 50
tests first that demoed how I would use it, and just hardcode my way
through all the tests to that they compiled and ran. The final resulting
class was way different than the first two iterations, much easier to
use and turned out to be easier to do internally as well. Just changing
the mindset from the API creater (or is that *creator*?) to the API user
made quite a difference.

<snip>

 

[Arne Vajhøj]

So I'm looking at this test-driven development (TDD) paradigm, and I like
the idea of having automated tests. That idea is not new.

But according to the TDD approach, we are to write our unit tests FIRST -
even before writing the code that the test is ultimately going to
test/verify.

Correct. Or written by someone other than the person writing the code.

Otherwise:
implementation problems => tests => design
for TDD or:
tests will tent to test what the code does instead what it should do
for more traditional unit tests.

I do see some merits of TDD - and doing the testing first - but what I think
is a practically guaranteed artifact of this approach is that many if not
most unit tests will become obsolete [and therefore have to be rewritten]
with any non trivial refactoring effort (which is also part of the whole TDD
paradigm). So, if I understand correctly, we are to write unit tests, then
write our code, then refactor the code almost immediately - and bam - just
like that, many of the unit test will also have to be rewritten or
refactored because the code being tested has been refactored [perhaps
refactored out of existance as we remove duplication, encapsulate stuff that
changes and recompose it with its original pre-refactoring class, etc].

A refactoring in the low level sense should not require unit tests to
be rewritten.

A change in design (which you can call refactoring at the high level)
will require a change in unit tests.

It should not happen that often.

And if it does happen, then I would expect rewriting unit tests
to be a very small fraction of the work added.

I don't see it as a problem.

Arne

PS: I am not particular TDD oriented. I see the main advantage of
extensive unit tests to show up after the code is developed
and enter maintenance mode.