July 2007

Monthly Archive

Managing Unit and Acceptance Tests Effectively

Posted by on 23 Jul 2007 | Tagged as: Acceptance Testing, Agile, Architecture, DotNet, Java, Scrum, TDD

In my experience, the use of Test-Driven Development (TDD) and automated acceptance testing on software projects makes for a powerful tool for flexible code and architectural management. When coaching teams on the use of TDD and acceptance testing there are some foundational test management techniques which I believe are essential for successful adoption. As a project progresses the amount of unit and acceptance tests grow tremendously and this can cause teams to become less effective with their test first strategy. A couple of these test management techniques are categorizing types of unit tests for running on different intervals and environments and structuring acceptance tests for isolation, iterative, and regression usage.

In 2003, I was working on a team developing features for a legacy J2EE application with extensive use of session and entity beans on IBM WebSphere. The existing code base lacked automated unit tests and had many performance issues. In order to tackle these performance issues the underlying architecture had to be reworked. I had been doing TDD for some time now and had become quite proficient with JUnit. The team discussed and agreed to use JUnit as an effective manner to incrementally migrate the application to our new architecture. After a few weeks the unit tests started to take too long to run for some of the developers including myself.

One night I decided to figure out if there would be a way to get these tests to run more quickly. I drew a picture of the current and proposed architecture on the white board and it hit me. We could separate concerns between interfacing layers by categorizing unit tests for business logic and integration points. Upon realizing this basic idea, I came up with the following naming convention that would be used by our Ant scripts for running different categories of unit tests:

  • *UnitTest.java – These would be fast running tests that did not need database, JNDI, EJB, J2EE container configuration, or any other external connectivity. In order to support this ideal we would need to stub out foundational interfaces such as the session and entity bean implementations.
  • *PersistanceTest.java – These unit tests would need access to a the database for testing configuration of entity beans to schema mappings.
  • *ContainerTest.java – These unit tests would run inside the container using a library called JUnitEE and test the container mappings for controller access to session beans and JNDI.

In our development environments we could run all of the tests ending with UnitTest.java when saving a new component implementation. These tests would run fast; anywhere from 3-5 seconds for the entire project. The persistance and container unit tests were run on an individual basis in a team member’s environment and the entire suite of these tests would be run by our Continuous Integration server each time we checked in code. These took a few minutes to run and our build server had to be configured with an existing WebSphere application server instance and DB2 relational database configured to work with the application.

In the “Psychology of Build Times”, Jeff Nielsen presented on the maximum amount of time builds, unit tests, and integration tests should take for a project. If builds and tests took too long than a team will be less likely to continue the discipline of TDD and Continuous Integration best practices. At Digital Focus, where Jeff worked, they had a similar unit test naming convention as the one I described above:

  • *Test.java – unit tests
  • *TestDB.java – database integration tests
  • *TestSRV.java – container integration tests

Another good source of information by Michael Feathers set out unit testing rules to help developers understand what unit tests are and are not. Here is a list of  “a test is not a unit test if”:

  • It talks to the database
  • It communicates across the network
  • It touches the file system
  • It can’t run at the same time as any of your other unit tests
  • You have to do special things to your environment (such as editing config files) to run it

Automated acceptance tests also have effective structures based upon the tools that you are using to capture and run them. For those using Fit I have found that structuring my tests into two categories, regression and iteration, supports near and long term development needs. The Fit tests which reside in the iteration directory can be ran each time code is updated to meet acceptance criteria for functionality being developed in the iteration. The regression Fit tests are ran in the Continuous Integration environment to give feedback to the team on any existing functionality which has broke with recent changes.

If you are using StoryTestIQ, a best practices structure for your automated acceptance tests has been defined by Paul Dupuy, creator of the tool. That structure looks like the following:

  • Integration Tests Tag Suite
  • Iteration Tests
    • Iteration 1 Tag Suite
    • Iteration 2 Tag Suite
  • Story Tests
    • User Story 1 Suite
    • User Story 2 Suite
  • Test Utilities
    • Login
      • Login As Steve
      • Login As Bob
    • Shopping Cart

You might have noticed a couple of descriptions above; Tag Suite and Suite. A tag suite is a collection of all test cases which have been tagged with a particular tag. This is helpful to allow for multiple views of the test cases for different environments such as development, QA, and Continuous Integration. The suite is usually used to collect a user stories acceptance tests together. There are other acceptance testing tools such as Selenium, Watir, and Canoo Web Test each with their own best practices on structuring.

Teams can effectively use TDD and automated acceptance tests without accruing overhead as the implemented functionality gets larger. It takes high levels of discipline to arrange your tests into effective categories for daily development, feedback, and regression. Tests should be treated as first class citizens along with the deliverable code. In order to do this they must be coded and refactored with care. The payoff for the effective use of automated unit, integration, and acceptance tests are tremendous in the quest for zero bugs and a flexible codebase. Happy coding.