19. The SUnit Framework
SUnit is a minimal yet powerful framework that supports the creation of automated unit tests. This chapter discusses the importance of repeatable unit tests and illustrates the ease of writing them using SUnit.1
19.1 Why SUnit?
Writing tests is an important way of investing in the future reliability and maintainability of your code. Tests should be repeatable, automated, and cover a precise functionality to maximize their potential.
SUnit was developed originally by Kent Beck and was extended by Joseph Pelrine and others. The interest in SUnit is not limited to the Smalltalk community. Indeed, legions of developers understand the power of unit testing and versions of XUnit (as the general framework is called) exist in many other languages.
Testing and building regression test suites is not new; it is common knowledge that regression tests are a good way to catch errors. Extreme Programming has brought a new emphasis to this somewhat neglected discipline by making testing a foundation of its methodology. The Smalltalk community has a long tradition of testing, due to the incremental development supported by its programming environment. However, once you write tests in a workspace or as example methods, there is no easy way to keep track of them and to automatically run them. Unfortunately, tests that you cannot automatically run are less likely to be run. Moreover, having a code snippet to run in isolation often does not readily indicate the expected result. That’s why SUnit is interesting—it provides a code framework to describe the context of your tests and to run them automatically. In less than two minutes, you can write tests using SUnit that become part of an automated test suite. This represents a vast improvement over writing small code snippets in an ephemeral workspace.
19.2 Testing and Tests
Many traditional development methodologies include testing as a step that follows coding, and this step is often cut short when time pressures arise. Yet development of automated tests can save time, since having a suite of tests is extremely useful and allows one to make application changes with much higher confidence.
Automated tests play several roles. First, they are an active and always synchronized documentation of the functionality they cover. Second, they represent the developer’s confidence in a piece of code. Tests help you quickly find defects introduced by changes to your code. Finally, writing tests at the same time or even before writing code forces you to think about the functionality you want to design. By writing tests first, you have to clearly state the context in which your functionality will run, the way it will interact with other code, and, more important, the expected results. Moreover, when you are writing tests, you are your first client and your code will naturally improve.
The culture of tests has always been present in the Smalltalk community; a typical practice is to compile a method and then, from a workspace, write a small expression to test it. This practice supports the extremely tight incremental development cycle promoted by Smalltalk. However, because workspace expressions are not as persistent as the tested code and cannot be run automatically, this approach does not yield the maximum benefit from testing. Moreover, the context of the test is left unspecified so the reader has to interpret the obtained result and assess whether it is right or wrong.
It is clear that we cannot tests all the aspects of an application. Covering a complete application is simply impossible and should not be the goal of testing. Even with a good test suite, some defect can creep into the application and be left hidden waiting for an opportunity to damage your system. While there are a variety of test practices that can address these issues, the goal of regression tests is to ensure that a previously discovered and fixed defect is not reintroduced into a later release of the product.
Writing good tests is a technique that can be easily learned by practice. Let us look at the properties that tests should have to get a maximum benefit:
- Repeatable. We should be able to easily repeat a test and get the same result each time.
- Automated. Tests should be run without human intervention. You should be able to run them during the night.
- Tell a story. A test should cover one aspect of a piece of code. A test should act as a specification for a unit of code.
- Resilient. Changing the internal implementation of a module should not break a test. One way to achieve this property is to write tests based on the interfaces of the tested functionality.
In addition, for test suites, the number of tests should be somehow proportional to the bulk of the tested functionality. For example, changing one aspect of the system might break some tests, but it should not break all the tests. This is important because having 100 tests broken should be a much more important message for you than having 10 tests failing.
By using “test-first” or “test-driven” development, eXtreme Programming proposes to write tests even before writing code. While this is counter-intuitive to the traditional “design-code-test” mindset, it can have a powerful impact on the overall result. Test-driven development can improve the design by helping you to discover the needed interface for a class and by clarifying when you are done (the tests pass!).
19.3 SUnit by Example
Before going into the details of SUnit, let’s look at a step-by-step example. The example in this section tests the class Set, and is included in the SUnit distribution so that you can read the code directly in the image.
Step 1: Define the Class ExampleSetTest
Example 19.1 defines the class ExampleSetTest, a subclass of TestCase.
Step 2: Define the Method setUp
Example 19.2 presents the method setUp, which acts as a context definer method or as an initialize method. It is invoked before the execution of any test method defined in this class. Here we initialize the empty instance variable to refer to an empty set, and the full instance variable to refer to a set containing two elements.
Step 3: Define Three Test Methods
Example 19.3 defines three methods on the class ExampleSetTest. Each method represents one test. If your test method names begin with test, as shown here, the framework will collect them automatically for you into test suites ready to be executed.
Example 19.3
ExampleSetTest>>testIncludes
self assert: (full includes: 5).
self assert: (full includes: #abc).
ExampleSetTest>>testOccurrences
self assert: (empty occurrencesOf: 0) = 0.
self assert: (full occurrencesOf: 5) = 1.
full add: 5.
self assert: (full occurrencesOf: 5) = 1.
ExampleSetTest>>testRemove
full remove: 5.
self assert: (full includes: #abc).
self deny: (full includes: 5).
The testIncludes method tests the includes: method of a Set. After running the setUp method in Example 19.2, sending the message includes: 5 to a set containing 5 should return true.
Next, testOccurrences verifies that there is exactly one occurrence of 5 in the full set, even if we add another element 5 to the set.
Finally, testRemove verifies that if we remove the element 5 from a set, that element is no longer present in the set.
Step 4: Execute the Tests
Now we can execute the tests, using either Topaz or one of the GemBuilder interfaces. To run your tests, execute the following code:
(ExampleSetTest selector: #testRemove) run.
Alternatively, you can execute this expression:
ExampleSetTest run: #testRemove.
Developers often include such an expression as a comment, to be able to run them while browsing. See Example 19.4.
Examining the Value of a Tested Expression
The method TestCase>>assert: requires a single argument, a boolean that represents the value of a tested expression. When the argument is true, the expression is considered to be correct, and we say that the test is valid. When the argument is false, then the test failed. The method deny: is the negation of assert:. Hence
aTest deny: anExpression.
aTest assert: anExpression not.
Finding Out If an Exception Was Raised
SUnit recognizes two kinds of defects: not getting the correct answer (a failure) and not completing the test (an error). If it is anticipated that a test will not complete, then the test should raise an exception. To test that exceptions have been raised during the execution of an expression, SUnit offers two methods, should:raise: and shouldnt:raise:. See Example 19.5.
Example 19.5
ExampleSetTest>>testIllegal
self should: [empty at: 5] raise: Error.
self should: [empty at: 5 put: #abc] raise: Error.
In the example provided by SUnit, the exception is provided via the TestResult class (Example 19.6). Because SUnit runs on a variety of Smalltalk dialects, the SUnit framework factors out the variant parts (such as the name of the exception). If you plan to write tests that are intended to be cross-dialect, look at the class TestResult.
Example 19.6
ExampleSetTest>>testIllegal
self should: [empty at: 5] raise: TestResult error.
self should: [empty at: 5 put: #abc] raise: TestResult error.
Because GemStone Smalltalk has a legacy exception framework that uses numbers to identify exceptions, a subclass of TestCase is provided, GSTestCase, which overrides should:raise: to allow a number argument for the expected error type.
19.4 The SUnit Framework
SUnit is implemented by four main classes: TestSuite, TestCase, TestResult, and TestResource. See Figure 19.1. (Note that this is an object composition diagram, not a class hierarchy diagram.)
TestSuite
The class TestSuite represents a collection of tests. An instance of TestSuite contains zero or more instances of subclasses of TestCase and zero or more instances of TestSuite. The classes TestSuite and TestCase form a composite pattern in which TestSuite is the composite and TestCase is the leaf.
TestCase
The class TestCase represents a family of tests that share a common context. The context is specified by instance variables on a subclass of TestCase and by the specialization method setUp, which initializes the context in which the test will be executed. The class TestCase also defines the method tearDown, which is responsible for cleanup, including releasing the objects allocated by setUp. The method tearDown is invoked after the execution of every test.
TestResult
The class TestResult represents the results of a TestSuite execution. This includes a description of which tests passed, which failed, and which had errors.
TestResource
Recall that the setUp method is used to create a context in which the test will run. Often that context is quite inexpensive to establish, as in Example 19.2 seen earlier, which creates two instances of Set and adds two objects to one of those instances.
At times, however, the context may be comparatively expensive to establish. In such cases, the prospect of re-establishing the context for each run of each test might discourage frequent running of the tests. To address this problem, SUnit introduces the notion of a resource that is shared by multiple tests.
The class TestResource represents a resource that is used by one or more tests in a suite, but instead of being set up and torn down for each test, it is established once before the first test and reset once after the last test. By default, an instance of TestSuite defines as its resources the list of resources for the TestCase instances that compose it.
As shown in Example 19.8, a resource is identified by overriding the class method resources. Here, we define a subclass of TestResource called MyTestResource. We associate it with MyTestCase by overriding the class method resources to return an array of the test classes to which it is associated.
19.5 Understanding the SUnit Implementation
Let’s now look at some key aspects of the implementation by following the execution of a test. Although this understanding is not necessary to use SUnit, it can help you to customize SUnit.
Running a Single Test
To execute a single test, we evaluate the expression
(TestCase selector: aSymbol) run.
The method TestCase>>run creates an instance of TestResult to contain the result of the executed tests, and then invokes the method TestCase>>run:, which in turn invokes the method TestResult>>runCase:. See Figure 19.2.
Figure 19.2 TestCase instance methods run and run: (source code)
TestCase>>run
| result |
result := TestResult new.
self run: result.
ensure: [TestResource resetResources: self resources].
^result.
TestCase>>run: aResult
aResult runCase: self.
The runCase: method (Figure 19.9) invokes the method TestCase>>runCase, which executes a test. Without going into the details, TestCase>>runCase pays attention to the possible exception that may be raised during the execution of the test, invokes the execution of a TestCase by calling the method runCase, and counts the errors, failures, and passed tests.
Example 19.9 TestResult instance method runCase: (source code)
TestResult>>runCase: aTestCase
[aTestCase runCase.
self addPass: aTestCase]
on: self class failure , self class error
do: [:ex | ex sunitAnnounce: aTestCase toResult: self]
As shown in Figure 19.10, the method TestCase>>runCase calls the methods setUp and tearDown.
Example 19.10 TestCase instance method runCase (source code)
TestCase>>runCase
self resources do: [:each | each availableFor: self].
[self setUp.
self performTest]
ensure: [self tearDown]
Running a TestSuite
To execute more than a single test, we invoke the method TestSuite>>run on a TestSuite (see Figure 19.11). The class TestCase provides the functionality to build a test suite from its methods. The expression MyTestCase suite returns a suite containing all the tests defined in the class MyTestCase.
The method TestSuite>>run creates an instance of TestResult, verifies that all the resource are available, then invokes the method TestSuite>>run: to run all the tests that compose the test suite. All the resources are then reset.
Example 19.11 TestSuite instance methods run and run: (source code)
TestSuite>>run
| result |
result := TestResult new.
[self run: result]
ensure: [TestResource resetResources: self resources].
^result
TestSuite>>run: aResult
self tests do: [:each |
self sunitChanged: each.
each run: aResult]
The class TestResource and its subclasses use the class method current to keep track of their currently created instances (one per class) that can be accessed and created. This instance is cleared when the tests have finished running and the resources are reset. The resources are created as needed. See Figure 19.12.
Example 19.12 TestResource class methods isAvailable and current (source code)
TestResource class>>isAvailable
^self current notNil
TestResource class>>current
current isNil ifTrue: [current := self new].
^current
19.6 For More Information
To continue your exploration of repeatable unit testing, visit the Camp Smalltalk SUnit site (http://sunit.sourceforge.net). The SUnit site provides information about SUnit development efforts, along with downloads, documentation, and other materials of interest.
You may also find these books helpful:
Beck, Kent. Test-Driven Development: By Example. Addison-Wesley, 2003.
Beck, Kent, and Cynthia Andres. Extreme Programming Explained: Embrace Change. Addison-Wesley, 2004.
Fowler, Martin, and Kent Beck. Refactoring: Improving the Design of Existing Code. Addison-Wesley, 1999.