19. The SUnit Framework

Previous chapter

Next chapter

SUnit is a minimal yet powerful framework that supports the creation of automated unit tests. This chapter discusses the importance of repeatable unit tests and illustrates the ease of writing them using SUnit.1

Why SUnit?
introduces the SUnit framework and its benefit to the application developer.

Testing and Tests
describes the general goals of automated testing.

SUnit by Example
presents a step-by-step example that illustrates the use of SUnit.

The SUnit Framework
describes the core classes of the SUnit framework.

Understanding the SUnit Implementation
explores key aspects of the implementation by following the execution of a test and test suite.

19.1 Why SUnit?

Writing tests is an important way of investing in the future reliability and maintainability of your code. Tests should be repeatable, automated, and cover a precise functionality to maximize their potential.

SUnit was developed originally by Kent Beck and was extended by Joseph Pelrine and others. The interest in SUnit is not limited to the Smalltalk community. Indeed, legions of developers understand the power of unit testing and versions of XUnit (as the general framework is called) exist in many other languages.

Testing and building regression test suites is not new; it is common knowledge that regression tests are a good way to catch errors. Extreme Programming has brought a new emphasis to this somewhat neglected discipline by making testing a foundation of its methodology. The Smalltalk community has a long tradition of testing, due to the incremental development supported by its programming environment. However, once you write tests in a workspace or as example methods, there is no easy way to keep track of them and to automatically run them. Unfortunately, tests that you cannot automatically run are less likely to be run. Moreover, having a code snippet to run in isolation often does not readily indicate the expected result. That’s why SUnit is interesting—it provides a code framework to describe the context of your tests and to run them automatically. In less than two minutes, you can write tests using SUnit that become part of an automated test suite. This represents a vast improvement over writing small code snippets in an ephemeral workspace.

19.2 Testing and Tests

Many traditional development methodologies include testing as a step that follows coding, and this step is often cut short when time pressures arise. Yet development of automated tests can save time, since having a suite of tests is extremely useful and allows one to make application changes with much higher confidence.

Automated tests play several roles. First, they are an active and always synchronized documentation of the functionality they cover. Second, they represent the developer’s confidence in a piece of code. Tests help you quickly find defects introduced by changes to your code. Finally, writing tests at the same time or even before writing code forces you to think about the functionality you want to design. By writing tests first, you have to clearly state the context in which your functionality will run, the way it will interact with other code, and, more important, the expected results. Moreover, when you are writing tests, you are your first client and your code will naturally improve.

The culture of tests has always been present in the Smalltalk community; a typical practice is to compile a method and then, from a workspace, write a small expression to test it. This practice supports the extremely tight incremental development cycle promoted by Smalltalk. However, because workspace expressions are not as persistent as the tested code and cannot be run automatically, this approach does not yield the maximum benefit from testing. Moreover, the context of the test is left unspecified so the reader has to interpret the obtained result and assess whether it is right or wrong.

It is clear that we cannot tests all the aspects of an application. Covering a complete application is simply impossible and should not be the goal of testing. Even with a good test suite, some defect can creep into the application and be left hidden waiting for an opportunity to damage your system. While there are a variety of test practices that can address these issues, the goal of regression tests is to ensure that a previously discovered and fixed defect is not reintroduced into a later release of the product.

Writing good tests is a technique that can be easily learned by practice. Let us look at the properties that tests should have to get a maximum benefit:

In addition, for test suites, the number of tests should be somehow proportional to the bulk of the tested functionality. For example, changing one aspect of the system might break some tests, but it should not break all the tests. This is important because having 100 tests broken should be a much more important message for you than having 10 tests failing.

By using “test-first” or “test-driven” development, eXtreme Programming proposes to write tests even before writing code. While this is counter-intuitive to the traditional “design-code-test” mindset, it can have a powerful impact on the overall result. Test-driven development can improve the design by helping you to discover the needed interface for a class and by clarifying when you are done (the tests pass!).

The next section provides an example of an SUnit test.

19.3 SUnit by Example

Before going into the details of SUnit, let’s look at a step-by-step example. The example in this section tests the class Set, and is included in the SUnit distribution so that you can read the code directly in the image.

Step 1: Define the Class ExampleSetTest

Example 19.1 defines the class ExampleSetTest, a subclass of TestCase.

Example 19.1
TestCase subclass: 'ExampleSetTest'
	instVarNames: #( full empty)
	classVars: #()
	classInstVars: #()
	poolDictionaries: #()
	inDictionary: Globals
 

The class ExampleSetTest groups all tests related to the class Set. It establishes the context of all the tests that we will specify. Here the context is described by specifying two instance variables, full and empty, that represent a full and empty set, respectively.

Step 2: Define the Method setUp

Example 19.2 presents the method setUp, which acts as a context definer method or as an initialize method. It is invoked before the execution of any test method defined in this class. Here we initialize the empty instance variable to refer to an empty set, and the full instance variable to refer to a set containing two elements.

Example 19.2
ExampleSetTest>>setUp
empty := Set new.
full := Set with: 5 with: #abc.
 

This method defines the context of any tests defined in the class. In testing jargon, it is called the fixture of the test.

Step 3: Define Three Test Methods

Example 19.3 defines three methods on the class ExampleSetTest. Each method represents one test. If your test method names begin with test, as shown here, the framework will collect them automatically for you into test suites ready to be executed.

Example 19.3
ExampleSetTest>>testIncludes
self assert: (full includes: 5).
self assert: (full includes: #abc).
 
ExampleSetTest>>testOccurrences
self assert: (empty occurrencesOf: 0) = 0.
self assert: (full occurrencesOf: 5) = 1.
full add: 5.
self assert: (full occurrencesOf: 5) = 1.
 
ExampleSetTest>>testRemove
full remove: 5.
self assert: (full includes: #abc).
self deny: (full includes: 5).
 

The testIncludes method tests the includes: method of a Set. After running the setUp method in Example 19.2, sending the message includes: 5 to a set containing 5 should return true.

Next, testOccurrences verifies that there is exactly one occurrence of 5 in the full set, even if we add another element 5 to the set.

Finally, testRemove verifies that if we remove the element 5 from a set, that element is no longer present in the set.

Step 4: Execute the Tests

Now we can execute the tests, using either Topaz or one of the GemBuilder interfaces. To run your tests, execute the following code:

(ExampleSetTest selector: #testRemove) run.

Alternatively, you can execute this expression:

ExampleSetTest run: #testRemove. 

Developers often include such an expression as a comment, to be able to run them while browsing. See Example 19.4.

Example 19.4
ExampleSetTest>>testRemove
"self run: #testRemove"
full remove: 5.
self assert: (full includes: #abc).
self deny: (full includes: 5).
 

To debug a test, use one of the following expressions:

(ExampleSetTest selector: #testRemove) debug. 

or

ExampleSetTest debug: #testRemove.

Examining the Value of a Tested Expression

The method TestCase>>assert: requires a single argument, a boolean that represents the value of a tested expression. When the argument is true, the expression is considered to be correct, and we say that the test is valid. When the argument is false, then the test failed. The method deny: is the negation of assert:. Hence

aTest deny: anExpression. 

is equal to

aTest assert: anExpression not. 

Finding Out If an Exception Was Raised

SUnit recognizes two kinds of defects: not getting the correct answer (a failure) and not completing the test (an error). If it is anticipated that a test will not complete, then the test should raise an exception. To test that exceptions have been raised during the execution of an expression, SUnit offers two methods, should:raise: and shouldnt:raise:. See Example 19.5.

Example 19.5
ExampleSetTest>>testIllegal
self should: [empty at: 5] raise: Error.
self should: [empty at: 5 put: #abc] raise: Error.
 

In the example provided by SUnit, the exception is provided via the TestResult class (Example 19.6). Because SUnit runs on a variety of Smalltalk dialects, the SUnit framework factors out the variant parts (such as the name of the exception). If you plan to write tests that are intended to be cross-dialect, look at the class TestResult.

Example 19.6
ExampleSetTest>>testIllegal
self should: [empty at: 5] raise: TestResult error.
self should: [empty at: 5 put: #abc] raise: TestResult error.
 

Because GemStone Smalltalk has a legacy exception framework that uses numbers to identify exceptions, a subclass of TestCase is provided, GSTestCase, which overrides should:raise: to allow a number argument for the expected error type.

Example 19.7
GSExampleSetTest>>testIllegal
self should: [empty at: 5] raise: 2007.
self should: [empty at: 5 put: #abc] raise: 2007.
 

Having provided an example of writing and running a test, we now turn to an investigation of the framework itself.

19.4 The SUnit Framework

SUnit is implemented by four main classes: TestSuite, TestCase, TestResult, and TestResource. See Figure 19.1. (Note that this is an object composition diagram, not a class hierarchy diagram.)

Figure 19.1 The SUnit Core Classes

 

 
TestSuite

The class TestSuite represents a collection of tests. An instance of TestSuite contains zero or more instances of subclasses of TestCase and zero or more instances of TestSuite. The classes TestSuite and TestCase form a composite pattern in which TestSuite is the composite and TestCase is the leaf.

TestCase

The class TestCase represents a family of tests that share a common context. The context is specified by instance variables on a subclass of TestCase and by the specialization method setUp, which initializes the context in which the test will be executed. The class TestCase also defines the method tearDown, which is responsible for cleanup, including releasing the objects allocated by setUp. The method tearDown is invoked after the execution of every test.

TestResult

The class TestResult represents the results of a TestSuite execution. This includes a description of which tests passed, which failed, and which had errors.

TestResource

Recall that the setUp method is used to create a context in which the test will run. Often that context is quite inexpensive to establish, as in Example 19.2 seen earlier, which creates two instances of Set and adds two objects to one of those instances.

At times, however, the context may be comparatively expensive to establish. In such cases, the prospect of re-establishing the context for each run of each test might discourage frequent running of the tests. To address this problem, SUnit introduces the notion of a resource that is shared by multiple tests.

The class TestResource represents a resource that is used by one or more tests in a suite, but instead of being set up and torn down for each test, it is established once before the first test and reset once after the last test. By default, an instance of TestSuite defines as its resources the list of resources for the TestCase instances that compose it.

As shown in Example 19.8, a resource is identified by overriding the class method resources. Here, we define a subclass of TestResource called MyTestResource. We associate it with MyTestCase by overriding the class method resources to return an array of the test classes to which it is associated.

Example 19.8
MyTestCase class>>resources
"associate a resource with a testcase"
^ Array with: MyTestResource.
 

As with a TestCase, we use the method setUp to define the actions that will be run during the setup of the resource.

19.5 Understanding the SUnit Implementation

Let’s now look at some key aspects of the implementation by following the execution of a test. Although this understanding is not necessary to use SUnit, it can help you to customize SUnit.

Running a Single Test

To execute a single test, we evaluate the expression

(TestCase selector: aSymbol) run.

The method TestCase>>run creates an instance of TestResult to contain the result of the executed tests, and then invokes the method TestCase>>run:, which in turn invokes the method TestResult>>runCase:. See Figure 19.2.

Figure 19.2 TestCase instance methods run and run: (source code)
TestCase>>run
| result |
result := TestResult new.
self run: result.
	ensure: [TestResource resetResources: self resources].
^result.
 
TestCase>>run: aResult
aResult runCase: self.
 

The runCase: method (Figure 19.9) invokes the method TestCase>>runCase, which executes a test. Without going into the details, TestCase>>runCase pays attention to the possible exception that may be raised during the execution of the test, invokes the execution of a TestCase by calling the method runCase, and counts the errors, failures, and passed tests.

Example 19.9 TestResult instance method runCase: (source code)
TestResult>>runCase: aTestCase
[aTestCase runCase.
self addPass: aTestCase]
	on: self class failure , self class error
	do: [:ex | ex sunitAnnounce: aTestCase toResult: self]
 

As shown in Figure 19.10, the method TestCase>>runCase calls the methods setUp and tearDown.

Example 19.10 TestCase instance method runCase (source code)
TestCase>>runCase
self resources do: [:each | each availableFor: self].
[self setUp.
self performTest] 
ensure: [self tearDown] 
 

Running a TestSuite

To execute more than a single test, we invoke the method TestSuite>>run on a TestSuite (see Figure 19.11). The class TestCase provides the functionality to build a test suite from its methods. The expression MyTestCase suite returns a suite containing all the tests defined in the class MyTestCase.

The method TestSuite>>run creates an instance of TestResult, verifies that all the resource are available, then invokes the method TestSuite>>run: to run all the tests that compose the test suite. All the resources are then reset.

Example 19.11 TestSuite instance methods run and run: (source code)
TestSuite>>run
| result |
result := TestResult new.
[self run: result]
	ensure: [TestResource resetResources: self resources].
^result
 
TestSuite>>run: aResult
self tests do: [:each | 
	self sunitChanged: each.
	each run: aResult]
 

The class TestResource and its subclasses use the class method current to keep track of their currently created instances (one per class) that can be accessed and created. This instance is cleared when the tests have finished running and the resources are reset. The resources are created as needed. See Figure 19.12.

Example 19.12 TestResource class methods isAvailable and current (source code)
TestResource class>>isAvailable
^self current notNil
 
TestResource class>>current
current isNil ifTrue: [current := self new].
^current
 

19.6 For More Information

To continue your exploration of repeatable unit testing, visit the Camp Smalltalk SUnit site (http://sunit.sourceforge.net). The SUnit site provides information about SUnit development efforts, along with downloads, documentation, and other materials of interest.

You may also find these books helpful:

Beck, Kent. Test-Driven Development: By Example. Addison-Wesley, 2003.

Beck, Kent, and Cynthia Andres. Extreme Programming Explained: Embrace Change. Addison-Wesley, 2004.

Fowler, Martin, and Kent Beck. Refactoring: Improving the Design of Existing Code. Addison-Wesley, 1999.


1. This chapter is adapted from “SUnit Explained” by St├ęphane Ducasse (http://www.iam.unibe.ch/~ducasse/Programmez/OnTheWeb/Eng-Art8-SUnit-V1.pdf) and is used by permission.

Previous chapter

Next chapter