Introducing Testing on the Toilet

Introducing "Testing on the Toilet"

We want you to write more tests. Yes, you. You've already been told that tests are the safety net that protects you when you need to refactor your code, or when another developer adds features. You even know that tests can help with the design of your code.

But, although you've read the books and heard the lectures, maybe you need a little more inspiration, tips, and prodding. And you need it to be in a place where when you see it, you can't ignore it.

That's where we can help. We're the "Google Testing Grouplet," a small band of volunteers who are passionate about software testing.

We're unveiling the public release of "Testing on the Toilet": one of Google's little secrets that has helped us to inspire our developers to write well-tested code. We write flyers about everything from dependency injection to code coverage, and then regularly plaster the bathrooms all over Google with each episode, almost 500 stalls worldwide. We've received a lot of feedback about it. Some favorable ("This is great because I'm always forgetting to bring my copy of Linux Nerd 2000 to the bathroom!") and some not ("I'm trying to use the bathroom, can you folks please just LEAVE ME ALONE?"). Even the Washington Post noticed.

We've decided to share this secret weapon with the rest of the world to help spread our passion for testing, and to provide a fun and easy way for you to educate yourself and the rest of your company about these important tricks and techniques.

We'll be putting episodes on this blog on a regular basis and providing PDFs so you can print them out and put them up in your own bathrooms, hallways, kitchens, moon bases, secret underground fortresses, billionaire founders' Priuses, wherever. Send your photos and stories to TotT@google.com and let us know how Testing on the Toilet is received at your company.

And meanwhile, keep writing those tests.

TotT Better Stubbing in Python

TotT Better Stubbing in Python

So you've learned all about method stubs, mock objects, and fakes. You might be tempted to stub out slow or I/O-dependent built-ins. For example:

 def Foo(path):
   if os.path.exists(path):
     return DoSomething()
   else:
     return DoSomethingElse()

 def testFoo(self):         # Somewhere in your unit test class
   old_exists = os.path.exists
   try:
     os.path.exists = lambda x: True
     self.assertEqual(Foo('bar'), something)
     os.path.exists = lambda x: False
     self.assertEqual(Foo('bar'), something_else)
   finally:
     # Remember to clean-up after yourself!
     os.path.exists = old_exists

Congratulations, you just achieved 100% coverage! Unfortunately, you might find that this test fails in strange ways. For example, given the following DoSomethingElse which checks the existence of a different file:

 def DoSomethingElse():
   assert os.path.exists(some_other_file)
   return some_other_file

Foo will now throw an exception in its second invocation because os.path.exists returns False so the assertion fails.

You could avoid this problem by stubbing or mocking out DoSomethingElse, but the task might be daunting in a real-life situation. Instead, it is safer and faster to parameterize the built-in:

 def Foo(path, path_checker=os.path.exists):
   if path_checker(path):
     return DoSomething()
   else:
     return DoSomethingElse()

 def testFoo(self):
   self.assertEqual(Foo('bar', lambda x: True), something)
   self.assertEqual(Foo('bar', lambda x: False), something_else)

Remember to download this episode of Testing on the Toilet, print it, and flyer your office.

TotT Naming Unit Tests Responsibly

TotT Naming Unit Tests Responsibly

For a class, try having a corresponding set of test methods, where each one describes a responsibility of the object, with the first word implicitly the name of the class under test. For example, in Java:

class HtmlLinkRewriterTest ... {
    void testAppendsAdditionalParameterToUrlsInHrefAttributes(){?}
    void testDoesNotRewriteImageOrJavascriptLinks(){?}
    void testThrowsExceptionIfHrefContainsSessionId(){?}
    void testEncodesParameterValue(){?}
  }

This can be read as:

HtmlLinkRewriter appends additional param to URLs in href attrs.
   HtmlLinkRewriter does not rewrite image or JavaScript links.
   HtmlLinkRewriter throws exception if href contains session ID.
   HtmlLinkRewriter encodes parameter value.   

Benefits
The tests emphasizes the object's responsibilities (or features) rather than public methods and inputs/output. This makes it easier for future engineers who want to know what it does without having to delve into the code.
These naming conventions can help point out smells. For example, when it's hard to construct a sentence where the first word is the class under test, it suggests the test may be in the wrong place. And classes that are hard to describe in general often need to be broken down into smaller classes with clearer responsibilities.
Additionally, tools can be used to help understand code quicker:


(This example shows a class in IntelliJ with the TestDox plugin giving an overview of the test.)

Remember to download this episode of Testing on the Toilet, print it, and flyer your office.

TotT JavaScript: Simulating Time in jsUnit Tests

TotT JavaScript: Simulating Time in jsUnit Tests

Sometimes you need to test client-side JavaScript code that uses setTimeout() to do some work in the future. jsUnit contains the Clock.tick() method, which simulates time passing without causing the test to sleep. For example, this function will set up some callbacks to update a status message over the course of four seconds:

function showProgress(status) {
  status.message = "Loading";
  for (var time = 1000; time <= 3000; time += 1000) {
    // Append a '.' to the message every second for 3 secs.
    setTimeout(function() {
      status.message += ".";
    }, time);
  }
  setTimeout(function() {
    // Special case for the 4th second.
    status.message = "Done";
  }, 4000);
}

The jsUnit test for this function would look like this:

function testUpdatesStatusMessageOverFourSeconds() {
  Clock.reset(); // Clear any existing timeout functions on the event queue.
  var  status = {};
  showProgress(status); // Call our function.
  assertEquals("Loading", status.message);
  Clock.tick(2000); // Call any functions on the event queue that have
                    // been scheduled for the first two seconds.
  assertEquals("Loading..",  status.message);
  Clock.tick(2000); // Same thing again, for the next two seconds.
  assertEquals("Done", status.message);
}

This test will run very quickly - it does not require four seconds to run.

Clock supports the functions setTimeout(),
setInterval(), clearTimeout(), and
clearInterval(). The Clock object is defined in
jsUnitMockTimeout.js, which is in the same directory as
jsUnitCore.js.

TotT-Stubs Speed up Your Unit Tests

TotT: Stubs Speed up Your Unit Tests

Michael Feathers defines the qualities of a good unit test as: “they run fast, they help us localize problems.” This can be hard to accomplish when your code accesses a database, hits another server, is time-dependent, etc.
By substituting custom objects for some of your module's dependencies, you can thoroughly test your code, increase your coverage, and still run in less than a second. You can even simulate rare scenarios like database failures and test your error handling code.
A variety of different terms are used to refer to these “custom objects”. In an effort to clarify the vocabulary, Gerard Meszaros provides the following definitions:

• Test Double is a generic term for any test object that replaces a production object.
• Dummy objects are passed around but not actually used. They are usually fillers for parameter lists.
• Fakes have working implementations, but take some shortcut (e.g., InMemoryDatabase).
• Stubs provide canned answers to calls made during a test.
• Mocks have expectations which form a specification of the calls they do and do not receive.

For example, to test a simple method like getIdPrefix() in the IdGetter class:

public class IdGetter {  // Constructor omitted.
  public String getIdPrefix() {
    try {
      String s = db.selectString("select id from foo");
      return s.substring(0, 5);
    } catch (SQLException e) { return ""; }
  }
}

You could write:

db.execute("create table foo (id varchar(40))");  // db created in setUp().
  db.execute("insert into foo (id) values ('hello world!')");
  IdGetter getter = new IdGetter(db);
  assertEquals("hello", getter.getIdPrefix());

The test above works but takes a relatively long time to run (network access), can be unreliable (db machine might be down), and makes it hard to test for errors. You can avoid these pitfalls by using stubs:

public class StubDbThatReturnsId extends Database {
    public String selectString(String query) { return "hello world"; }
  }
  public class StubDbThatFails extends Database {
    public String selectString(String query) throws SQLException {
      throw new SQLException("Fake DB failure");
    }
  }
  public void testReturnsFirstFiveCharsOfId() throws Exception {
    IdGetter getter = new IdGetter(new StubDbThatReturnsId());
    assertEquals("hello", getter.getIdPrefix());
  }
  public void testReturnsEmptyStringIfIdNotFound() throws Exception {
    IdGetter getter = new IdGetter(new StubDbThatFails());
    assertEquals("", getter.getIdPrefix());
  }

Remember to download this episode of Testing on the Toilet and post it in your office.

TotT-The Stroop Effect

TotT: The Stroop Effect

How quickly can you...

1. ...read all 25 words out loud: RED, GREEN, BLUE, ... (Try it now!)
2. ...say all 25 colors out loud: GREEN, YELLOW, WHITE... (Try it now!)

Did the second task require more time and effort? If so, you're experiencing the Stroop Effect, which roughly says that when a label (in this case, the word) is in the same domain as its content (the color) with a conflicting meaning, the label interferes with your ability to comprehend the content.

What does this have to do with testing? Consider the following code:

public void testProtanopiaColorMatcherIsDistinguishable() {
  ColorMatcher colorMatcher = new ColorMatcher(PROTANOPIA);
  assertFalse(“BLUE and VIOLET are indistinguishable”,
    colorMatcher.isDistinguishable(Color.BLUE, Color.VIOLET));
}

When this test fails, it produces a message like this:

Failure: testProtanopiaColorMatcherIsDistinguishable:
Message: BLUE and VIOLET are indistinguishable

Quick: what caused this error? Were BLUE and VIOLET indistinguishable, or not? If you're hesitating, that's the Stroop Effect at work! The label (the message) expresses a truth condition, but the content (in assertFalse) expresses a false condition. Is the ColorMatcher doing the wrong thing, or is the test condition bogus? This message is wasting your valuable time! Now consider this slight alteration to the test name and test message:

Failure: testProtanopiaColorMatcherCannotDistinguishBetweenCertainPairsOfColors
Message: BLUE and VIOLET should be indistinguishable

Do you find this clearer? Protanopia (reduced sensitivity to the red spectrum) causes certain pairs of colors to be indistinguishable. BLUE and VIOLET should have been indistinguishable, but weren't.

Here are some things to keep in mind when writing your tests:

• When someone breaks your test – will your test name and message be useful to them?
• Opinionated test names like testMethodDoesSomething can be more helpful than testMethod.
• Great test messages not only identify the actual behavior,but also the expected behavior.
• Should is a handy word to use in messages – it clarifies what expected behavior didn't actually happen.

Remember to download this episode of Testing on the Toilet and post it in your office. Permalink | Links to this post | 2 comments  

TotT: Refactoring Tests in the Red

Thursday, April 26, 2007 3:28 AM

With a good set of tests in place, refactoring code is much easier, as you can quickly gain a lot of confidence by running the tests again and making sure the code still passes.
As suites of tests grow, it's common to see duplication emerge. Like any code, tests should ideally be kept in a state that's easy to understand and maintain. So, you'll want to refactor your tests, too.
However, refactoring tests can be hard because you don't have tests for the tests.
How do you know that your refactoring of the tests was safe and you didn't accidentally remove one of the assertions?
If you intentionally break the code under test, the failing test can show you that your assertions are still working. For example, if you were refactoring methods in CombineHarvesterTest, you would alter CombineHarvester, making it return the wrong results.
Check that the reason the tests are failing is because the assertions are failing as you'd expect them to. You can then (carefully) refactor the failing tests. If at any step they start passing, it immediately lets you know that the test is broken – undo! When you're done, remember to fix the code under test and make sure the tests pass again.
(revert is your friend, but don't revert the tests!)
Let's repeat that important point:
When you're done...remember to fix the code under test!
Summary

• Refactor production code with the tests passing. This helps you determine that the production code still does what it is meant to.
• Refactor test code with the tests failing. This helps you determine that the test code still does what it is meant to.

Remember to download this episode of Testing on the Toilet and post it in your office.

TotT Be an MVP of GUI Testing

TotT: Be an MVP of GUI Testing

With all the sport drug scandals of late, it's difficult to find good role models these days. However, when your role model is a Domain Model (object model of the business entities), you don't need to cheat to be an MVP--Use Model-View-Presenter!

MVP is very similar to MVC (Model-View-Controller). In MVC, the presentation logic is shared by Controller and View, as shown in the diagram below. The View is usually derived directly from visible GUI framework component, observing the Model and presenting it visually to the user. The Controller is responsible for deciding how to translate user events into Model changes. In MVP, presentation logic is taken over entirely by a Supervising Controller, also known as a Presenter.

MVC

MVP

The View becomes passive, delegating to the Presenter.

public CongressionalHearingView() {
testimonyWidget.addModifyListener(
new ModifyListener() {
public void modifyText(ModifyEvent e) {
presenter.onModifyTestimony(); // presenter decides action to take
}});
}

The Presenter fetches data from the Model and updates the View.

public class CongressionalHearingPresenter {
public void onModifyTestimony() {
model.parseTestimony(view.getTestimonyText()); // manipulate model
}
public void setWitness(Witness w) {
view.setTestimonyText(w.getTestimony()); // update view
}
}

This separation of duties allows for more modular code, and also enables easy unit testing of the Presenter and the View.

public void testSetWitness() {
spyView = new SpyCongressionalHearingView();
presenter = new CongressionalHearingPresenter(spyView);
presenter.setWitness(new Witness(“Mark McGwire”, “I didn't do it”));
assertEquals( “I didn't do it”, spyView.getTestimonyText());
}

Note that this makes use of a perfectly legal injection -- Dependency Injection.

Remember to download this episode of Testing on the Toilet and post it in your office.

TotT Partial Mocks using Forwarding Objects

TotT: Partial Mocks using Forwarding Objects

A Partial Mock is a mock that uses some behavior from a real object and some from a mock object. It is useful when you need bits of both. One way to implement this is often a Forwarding Object (or wrapper) which forwards calls to a delegate.

For example, when writing an Olympic swimming event for ducks, you could create a simple forwarding object to be used by multiple tests:

interface Duck {
Point getLocation();
void quack();
void swimTo(Point p);
}

class ForwardingDuck implements Duck {
private final Duck d;
ForwardingDuck(Duck delegate) {
this.d = delegate;
}
public Point getLocation() {
return d.getLocation();
}
public void quack() {
d.quack();
}
public void swimTo(Point p) {
d.swimTo(p);
}
}

And then create a test that uses all of the real OlympicDuck class's behavior except quacking.

public void testDuckCrossesPoolAndQuacks() {
final Duck mock = EasyMock.createStrictMock(Duck.class);
mock.swimTo(FAR_SIDE);
mock.quack(); // quack after the race
EasyMock.replay(mock);
Duck duck = OlympicDuck.createInstance();
Duck partialDuck = new ForwardingDuck(duck) {
@Override public void quack() {
mock.quack();
}
@Override public void swimTo(Point p) {
mock.swimTo(p);
super.swimTo(p);
}
// no need to @Override “Point getLocation()”
}

OlympicSwimmingEvent.createEventForDucks()
.withDistance(ONE_LENGTH)
.sponsoredBy(QUACKERS_CRACKERS)
.addParticipant(partialDuck)
.doRace();
MatcherAssert.assertThat(duck.getLocation(), is(FAR_SIDE));
EasyMock.verify(mock);

partialDuck is a complex example of a partial mock – it combines real and mock objects in three different ways:

• quack() calls the mock object. It verifies that the duck doesn't promote the sponsor (by quacking) until after the race. (We skip the real quack() method so that our continuous build doesn't drive us crazy.)
• getLocation() calls the real object. It allows us to use the OlympicDuck's location logic instead of rewriting/simulating the logic from that implementation.
• swimTo(point) calls both objects. It allows us to verify the call to the real duck before executing it.

There is some debate about whether you should forward to the real or mock Duck by default. If you use the mock duck by default, any new calls to the mock will break the test, making them brittle. If you use the real duck, some very sensitive calls like submitToDrugTest() might get called by your test if your duck happens to win.

Consider using a Partial Mock in tests when you need to leverage the implementation of the real object, but want to limit, simulate or verify method calls using the power of a mock object.

Remember to download this episode of Testing on the Toilet and post it in your office.

TotT Testing GWT without GwtTestCase

TotT: Testing GWT without GwtTestCase

Because GWT (Google Web Toolkit) is new and exciting it's easy to forget the lessons on clean GUI code structure that have been accumulated over nearly thirty years.

GwtTestCase is good for testing UI-specific code in JavaScript. If you find yourself using GwtTestCase for testing non-ui client-side logic you may not have a clear View/Presenter separation. Separating the View and the Presenter allows for more modular, more easily tested code with shorter test times. Model View Presenter was introduced in another episode back in February. Here's how to apply it to a GWT app.

Defining terms:

• Server – a completely standard backend with no dependency on GWT.

• Model – the data model. May be shared between the client and server side, or if appropriate you might have a different model for the client side. It has no dependency on GWT.

• View – the display. Classes in the view wrap GWT widgets, hiding them from the rest of your code. They contain no logic, no state, and are easy to mock.

• Presenter – all the client side logic and state; it talks to the server and tells the view what to do. It uses RPC mechanisms from GWT but no widgets.

The Presenter, which contains all the interesting client-side code is fully testable in Java!

public void testRefreshPersonListButtonWasClicked() {
IMocksControl easyMockContext = EasyMock.createControl()
mockServer = easyMockContext.createMock(Server.class);
mockView = easyMockContext.createMock(View.class);
List franz = Lists.newArrayList(new Person("Franz", "Mayer"));
mockServer.getPersonList(AsyncCallbackSuccessMatcher<list<person>>reportSuccess(franz)));
mockView.clearPersonList());
mockView.addPerson(“Franz”, “Mayer”);
easyMockContext.replay();
presenter.refreshPersonListButtonClicked();
easyMockContext.verify();
}

Testing failure cases is now as easy as changing expectations. By swapping in the following expectations, the above test goes from testing success to testing that after two server failures, we show an error message.

mockServer.getPersonList(AsyncCallbackFailureMatcher<list<person>>reportFailure(failedExpn))
expectLastCall().times(2); // Ensure the presenter tries twice
mockView.showErrorMessage(“Sorry, please try again later”));

You'll still need an end-to-end test. But all your logic can be tested in small and fast tests.

The Source Code for the Matchers is open-sourced and can be downloaded here: AsyncCallbackSuccessMatcher.java - AsyncCallbackFailureMatcher.java.

Consider using Test Driven Development (TDD) to develop the presenter. It tends to result in higher test coverage, faster and more relevant tests, as well as a better code structure.

This week's episode by David Morgan, Christopher Semturs and Nicolas Wettstein based in Zürich, Switzerland – having a real Mountain View

Toilet-friendly version

AsyncCallbackSuccessMatcher.java

AsyncCallbackFailureMatcher.java.

TotT: Literate Testing With Matchers

TotT: Literate Testing With Matchers

By Zhanyong G. Mock Wan in Google Kirkland

Alright, it sounds like a good idea to verify that matchmakers can read and write. How does this concern us programmers, though?
Actually, we are talking about a way of writing tests here – a way that makes both the test code and its output read like English (hence “literate”). The key to this technique is matchers, which are predicates that know how to describe themselves. For example, in Google C++ Mocking Framework, ContainsRegex("Ahcho+!") is a matcher that matches any string that has the regular expression "Ahcho+!" in it. Therefore, it matches "Ahchoo!" and "Ahchoooo! Sorry.", but not "Aha!".
What's this to do with test readability, anyway? It turns out that matchers, whose names are usually verb phrases, lend themselves easily to an assertion style that resembles natural languages. Namely, the assertion

EXPECT_THAT(value, matcher);

succeeds if value matches matcher. For example,

#include <gmock/gmock.h>
using ::testing::Contains;
...
EXPECT_THAT(GetUserList(), Contains(admin_id));

verifies that the result of GetUserList() contains the administrator.

Now, pretend the punctuations aren't there in the last C++ statement and read it. See what I mean?

Better yet, when an EXPECT_THAT assertion fails, it will print an informative message that includes the expression being validated, its value, and the property we expect it to have – thanks to a matcher's ability to describe itself in human-friendly language. Therefore, not only is the test code readable, the test output it generates is readable too. For instance, the above example might produce:

Value of: GetUserList()
Expected: contains "yoko"
  Actual: { "john", "paul", "george", "ringo" }

This message contains relevant information for diagnosing the problem, often without having to use a debugger.
To get the same effect without using a matcher, you'd have to write something like:

std::vector<std::string> users = GetUserList();
EXPECT_TRUE(VectorContains(users, admin_id))
    << " GetUserList() returns " << users
    << " and admin_id is " << admin_id;

which is harder to write and less clear than the one-liner we saw earlier.

Google C++ Mocking Framework (http://code.google.com/p/googlemock/) provides dozens of matchers for validating many kinds of values: numbers, strings, STL containers, structs, etc. They all produce friendly and informative messages. See http://code.google.com/p/googlemock/wiki/CheatSheet to learn more. If you cannot
find one that matches (pun intended) your need, you can either combine existing matchers, or define your own from scratch. Both are quite easy to do. We'll show you how in another episode. Stay tuned!

TotT: Making a Perfect Matcher

TotT: Making a Perfect Matcher

by Zhanyong G. Mock Wan in Google Kirkland
In the previous episode, we showed how Google C++ Mocking Framework matchers can make both your test code and your test output readable. What if you cannot find the right matcher for the task?

Don't settle for anything less than perfect. It's easy to create a matcher that does exactly what you want, either by composing from existing matchers or by writing one from scratch.

The simplest composite matcher is Not(m), which negates matcher m as you may have guessed. We also have AnyOf(m1, ..., mn) for OR-ing and AllOf(m1, ..., mn) for AND-ing. Combining them wisely and you can get a lot done. For example,

EXPECT_THAT(new_code, AnyOf(StartsWith(“// Tests”)),
              Not(ContainsRegex(“TODO.*intern”))));

could generate a message like:

Expected: (starts with “// Tests”) or
          (doesn't contain regular expression “TODO.*intern”)
Actual: “/* TODO: hire an intern. */ int main() {}”

If the matcher expression gets too complex, or your matcher logic cannot be expressed in terms of existing matchers, you can use plain C++. The MATCHER macro lets you define a named matcher:

MATCHER(IsEven, “”) { return (arg % 2) == 0; }

allows you to write EXPECT_THAT(paren_num, IsEven()) to verify that paren_num is divisible by two. The special variable arg refers to the value being validated (paren_num in this case) – it is not a global variable.

You can put any code between {} to validate arg, as long as it returns a bool value.

The empty string “” tells Google C++ Mocking Framework to automatically generate the matcher's description from its name (therefore you'll see “Expected: is even” when the match fails). As long as you pick a descriptive name, you get a good description for free.

You can also give multiple parameters to a matcher, or customize its description. The code:

// P2 means the matcher has 2 parameters. Their names are low and high.

MATCHER_P2(InClosedRange, low, high, “is in range [%(low)s, %(high)s]”) {
  return low <= arg && arg <= high;
}
...
EXPECT_THAT(my_age, InClosedRange(adult_min, penalty_to_withdraw_401k));

may print:

Expected: is in range [18, 60]
  Actual: 2

(No, that's not my real age.) Note how you can use Python-style interpolation in the description string to print the matcher parameters.
You may wonder why we haven't seen any types in the examples. Rest assured that all the code we showed you is type-safe. Google C++ Mocking Framework uses compiler type inference to “write” the matcher parameter types for you, so that you can spend the time on actually writing tests – or finding your perfect match.