Software Testing

During Software Construction

Preamble

overview testing
Figure 1. Where are we?

Plan

  • Introduction

  • Dynamic Testing

  • Test Cases

  • Unit Testing

  • Integration Testing

  • Test Automation

  • Conclusion

Definition

Software Testing is the activity of finding errors in a software application, i.e. that the application functions according to the user’s requirements.

Test Classification

Different criteria according to:
The quality factor

functional, performance, etc.

The level

Unit, Component, Integration, System, Acceptance

The development phase

no-regression, acceptation, etc.

The execution type

dynamic, static

Testing During Software Construction

  1. Unit (functional, dynamic)

  2. Integration (functional, dynamic)

  3. Static Analysis (functional, static)

Dynamic Testing

In a nutshell:

  • Dynamic Testing relies on the execution of the Software Under Test to verify whether it complies with its specification.

  • This requires executing the same program a large amount of times to cover all relevant cases from the specification.

In practical terms

  • Dynamic Testing consists in:

    1. executing a Script that drives the Software into a given state

    2. providing Data to the Software

    3. executing an Oracle that tells whether the test fails or succeeds

  • The Script, the Data, and the Oracle form a Test Case

  • A suite of Test Cases is called a Test Suite

cd test suite

Test Cases

Dynamic Test Cases

dynamic test case
Figure 2. Test Case Execution

Example: The ArrayList class

cd array list
How to test class ArrayList?
  • What should be tested?

  • How many Test Cases?

How to…​
  • describe the Test Cases?

  • name them?

  • code the Test Scripts?

  • code the Oracles?

What should be tested?

Test behaviors, not methods!

  • Single responsibility: one test should be responsible for one scenario only.

    • One method, multiple behaviors: multiple tests

    • One behavior, multiple methods: one test

  • A method: add()

  • Behaviors:

    1. Add to an empty list

    2. Add several times the same element

    3. Add a null element

    4. etc.

How many Test Cases?

Less is more!

  • Test cases are also code: they should be maintainable

  • If there are too many tests for a single behavior, changing this behavior will broke too many tests

  • Tests should simple and small: when then fail, it should be easy to localize the error

How to name test cases?

There are several naming conventions

Most popular:
[MethodName]_[StateUnderTest]_[ExpectedBehavior]
  • Example:

size_emptylist_isZero

Test Case Naming Guidelines

Allow freedom of expression!

  • No rigid naming policy: a complex behavior cannot fit in a narrow name!

  • Name the test as if you were describing the scenario to a non-programmer domain-expert.

The_size_of_an_empty_list_is_zero
A_list_contains_an_element_added_previously
Remove_an_element_doesnt_remove_others

How to Describe a Test Case?

The Given-When-Then Style:
  • GIVEN a context

  • WHEN some condition

  • THEN expect some output

Test Cases:
  1. Given an empty list, When an element is added, Expect list to contain element

  2. Given an empty list, When three elements are added Expect size to be 3

  3. Given a list with elements a, b, and c, When b is removed, Expect list should only contain a and b

How to Code a Test Script?

The script should contain 3 blocks, separated by comments or one empty lines:
  1. Given (Input): Test preparation like creating objects or configuring mocks

  2. When (Action): Call the method under test with the prepared objects

  3. Then (Output): The Oracle, usually an assertion that verifies the correct behavior of the method.

// Given:
List<Integer> list = new List<>();
int expectedSize = 0;

// When:
int actualSize = list.size()

// Then:
assert actualSize == expectedSize;

If the blocks are too complex, use auxiliary (private) methods!

Test Script Guidelines

Keep it Simple, Stupid!

Write readable tests
  • Test scripts must maintainable.

  • Variable, method, and class names should be self-descriptive

  • Use actual and expect to name the value returned by the method under test and the expected value.

Test Script Guidelines (Cont.)

Write single flow code:
  • No conditional logic or loops

  • Split in to two tests rather than using “If” or “Case”

  • Tests should not contain “While”, “Do While” or “For” loops.

  • If test logic has to be repeated, it probably means the test is too complicated.

  • Call method multiple times rather than looping inside of method.

Single Flow Example

ArrayList<String> expected = new ArrayList<>();
ArrayList<String> actual = new ArrayList<>();
expected.add("one");
expected.add("three");
actual.add("one");
actual.add("two");
actual.add("three");

actual.remove("two");

assert actual.equals(expected);

Test Script Guidelines (Cont.)

Test should have no uncertainty:
  • All inputs should be known

  • Method behavior should be predictable

  • Expected output should be strictly defined

Do not handle exceptions
  • Indicate expected exception with attribute.

  • Catch only the expected type of exception.

  • Fail test if expected exception is not caught.

  • Let other exceptions go uncaught.

How to Code an Oracle?

  • Use assertions to describe the expected behavior

assert actual.equals(expected) : "After a remove, other elements remain unaltered"
Test case must contain at most one assert:
  • When the test fails, you will know why.

Oracle Guidelines

Use informative assertion messages
  • By reading the assertion message, one should know why the test failed and what to do.

  • Include business logic information in the assertion message (such as input values, etc.)

Oracle Guidelines (Cont.)

Good assertion messages:
  • Improve documentation of the code

  • Inform developers about the problem if the test fails

  • Helps to localize the error

A bad assertion messages makes developers spend more time diagnosing!

What are Assertions?

  • An Assertion is a statement that detects an unexpected behavior in the code.

  • In other terms, a boolean expression that must be true when the assertion executes.

Assertions ≠ Exceptions:
  • Exceptions handle conditions that are expected to occur

  • Assertions handle conditions that should never occur

Assertions Can Replace Tests

  • Some Test Cases can easily be turned into assertions.

  • Consider for instance the ArrayList::add() method, from OpenJDK:

public boolean add(E e) {
    ensureCapacityInternal(size + 1);  // Increments modCount!!
    elementData[size++] = e;
    return true;
}

Assertions Can Replace Tests (Cont.)

The same method, with an Assertion:
public boolean add(E e) {
    ensureCapacityInternal(size + 1);  // Increments modCount!!
    elementData[size++] = e;

    assert this.contains(e) : "An element added to the list is " +
            "contained by the same list"
    return true;
}
  • The behavior is verified by every test case that uses the add() method:

    • unit, integration, system, etc.

Final Test Case Guidelines

Ensure Isolation
  • To ensure testing robustness and simplify maintenance, tests should never rely on other tests nor should they depend on the ordering in which tests are executed.

Ensure reproducibility
  • Multiple test executions must consistently yield the same result, provided no changes were made on the software under test.

Ensure atomicity
  • Tests must either pass or fail, they cannot be partially successful.

Unit Testing

Definition

Unit testing is a software testing approach that consists in testing each unit of the software under test individually.

Units

  • Intuitively, a unit is the smallest testable part of a software.

  • In procedural programming: a function or a procedure.

  • In object-oriented programming: a method, a class, or a component.

unit

Goals

  • Check that each unit works as designed

  • Isolate each software part and show they are corrects

Rationale

  • The reliability of a system is equal to its less reliable part.

system

A trustworthy system is made of trustworthy units.

Tested properties

  • Functional behavior

  • Error handling

  • Input values check

  • Performance

Benefits

Avoid the «Developer’s Block»:
  • Developers do not fear to change existing code

  • Improve developer’s confidence on his code

Are a living documentation
  • Tests are usage examples of each unit

  • Work as the unit specification

Software construction simplification

Unit tests
  • Simplify debugging:

    • Reduce the search space

    • Avoid useless "prints"

  • Simplify evolution:

    • Avoid regression

    • Work as a «safety net»

  • Simplify integration:

    • If developers trust on each unit, integration errors are easier to find

Integration Testing

Definition

Integration testing verifies whether many separately developed units work together as expected.

integration

In object-oriented programming, integration happens at different levels: method, class, and component.

Goals

  • Test the interaction between units

  • Determine in which order the units must be tested

  • Identify dependencies between units

Rationale

In the real word:
  • different developers are assigned to different modules of the same application

  • each developer has its own logic, which is passed to the code

  • integration testing checks whether the different modules is in accordance with the specification

Modules may interact with third party tools:
  • integration testings checks whether the data send to that tool and the generated response are correct.

Challenges

  • Test management is complex: integration testing often involves interacting with the environment: operation systems, hardware, database, middleware

  • Integration with legacy system my require lot of changes and testing efforts.

  • Integration with third party software is even harder.

Benefits

  • Increases confidence on the developed code

  • Detects higher level issues: broken database, integration mistakes, etc.

  • Helpful when there are frequent requirement changes.

Integration Approaches

Diagram
Figure 3. Dependency Tree
Big Bang

all units are tested in the same time

Bottom-up Integration

build a dependency tree, start with the leaves and go up until the root

Top-down Integration

the inverse; start with the root and go down until the leaves

Integration Testing Best Practices

  • Don’t wait for a complete unit testing to start coding integration tests: be flexible

  • Separate unit tests from integration tests: execute unit before integration tests

  • Log as much as possible: diagnostic is more difficult than in unit testing.

Test Automation

With Maven, JUnit, and Truth

What is Test Automation?

Roughly, the execution of a suite of test cases:
  • Automates some repetitive yet necessary tasks in a testing process already in place

  • Performs additional testing that cannot be done manually

Critical for continuous integration!

Why?

  • Software projects are often complex: small changes tend to affect multiple parts of the software

  • When adding new features, developers usually only check whether the code works as intended

  • The same applies to correcting errors (maintenance)

Benefices

  • Test cases become easy to run: every developer is able to run all the tests in their development environment.

  • Test cases become a part of the build: in an ideal situation, every build automatically launches all the tests and displays their results.

  • Every change is checked by testing the entire system.

An investment in the future

Automatic tests
  • Will be reused countless times during the software lifetime

    • During corrective and evolutive maintenance

  • Force developers to write testable code

  • Improve design

    • Tests are clients of the unit under test, helping developers to write simple interfaces

Truth

A library for simplifying assertion coding

Truth

  • A library for coding assertions in tests (or elsewhere)

  • Owned and maintained by the Guava team.

  • Used in the majority of the tests in Google’s own codebase.

  • Web site: https://truth.dev

Examples
assertThat(text).contains("testuser@google.com");
assertThat(aList)
      .containsExactlyElementsIn(anotherList)
      .inOrder();
assertThat(aMap).containsEntry("one", 1L);
assertThat(aComparable).isIn(Range.closed(1, 10));

Why ?

  • Prevents writing complex assertion logic

  • More descriptive failure messages

Failure messages

Assertions:
assert "one".equals("two") : "One and two are different";

assertThat("one").isEqualTo("two"); // Truth
Java Failure Message:
Exception in thread "main" java.lang.AssertionError: One and two are different
	at fr.unantes.sce.Main.main(Main.java:13)
Truth Failure Message:
Exception in thread "main" expected: two
but was : one
	at fr.unantes.sce.Main.main(Main.java:14)

JUnit

JUnit

  • Java Framework for writing dynamic unit tests

  • Open source, available at http://www.junit.org

  • Origins:

    • eXtreme Programming (XP)

    • Smalltalk Test Framework (Kent Beck)

    • First implementation by Erich Gamma 1997

Usage

JUnit is not only for Unit Testing!

Beyond its name:
  • JUnit is a framework for automatically executing methods (test cases), with several features

  • JUnit can be used to perform system, integration, unit, etc.

Example: The Interval Class

public class Interval<T extends Comparable> {
    private final T begin;
    private final T end;

    protected Interval(T begin, T end) {
        this.begin = begin;
        this.end = end;
    }

    public boolean includes(T i) {
        return i.compareTo(begin) >= 0 && i.compareTo(end) <= 0;
    }
}
  • Use JUnit to test the Interval class.

  • The Interval class represents simple intervals

    • Method includes() checks if a value belongs to the interval

Testing the Interval class

  1. Create a test class

  2. Implement a test method that:

    • Instantiates the Interval 1-10

    • Check if 5 belongs to this interval

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

class IntervalTest {

    @Test
    @DisplayName(value = "An interval contains a number between its boundaries")
    void inverval_contains_number_between_boundaries() {
        Interval<Integer> interval = new Interval<>(0, 10);

        assertTrue(interval.includes(5));
    }
}

Some Implementation details

  • JUnit automatically executes all methods adorned with @Test

  • Assertions.assertTrue(<exp>) makes the test fail if <exp> evaluates to false.

Improving the tests

  • Check if -1 and 11 do not belong to the interval

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

class IntervalTest {

    @Test
    @DisplayName(value = "An interval contains a number between its boundaries")
    void includes() {
        Interval<Integer> interval = new Interval<>(0, 10);

        assertTrue(interval.includes(5));
    }

    @Test
    @DisplayName(value = "An interval doesnt contain a number greater thans its upper boundary")
    void doesntIncludesGreaterThanUpperBoundary() {
        Interval<Integer> interval = new Interval<>(0, 10);

        assertFalse(interval.includes(11));
    }

    @Test
    @DisplayName(value = "An interval doesnt contain a number lesser than its lower boundary")
    void doesntIncludesLesserThanLowerBoundary() {
        Interval<Integer> interval = new Interval<>(0, 10);

        assertFalse(interval.includes(-1));
    }
}

Assembling common code

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

class IntervalTest {
    private Interval<Integer> interval_1_10;

    @BeforeEach
    void setup() {
        interval_1_10 = new Interval<>(0, 10);
    }

    @Test
    @DisplayName(value = "An interval contains a number between its boundaries")
    void includes() {
        assertTrue(interval_1_10.includes(5));
    }

    @Test
    @DisplayName(value = "An interval doesnt contain a number greater thans its upper boundary")
    void doesntIncludesGreaterThanUpperBoundary() {
        assertFalse(interval_1_10.includes(11));
    }

    @Test
    @DisplayName(value = "An interval doesnt contain a number lesser than its lower boundary")
    void doesntIncludesLesserThanLowerBoundary() {
        assertFalse(interval_1_10.includes(-1));
    }
}

More implementation details

  • JUnit automatically executes the method adorned with @BeforeEach before each @Test method.

  • In the example, setup() is executed 3 times.

junit execution

Parameterized tests

  • JUnit automatically executes methods adorned with @ParameterizedTest several times: one time for each value

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;

import static org.junit.jupiter.api.Assertions.*;

class IntervalTest {
    private Interval<Integer> interval_1_10;

    @BeforeEach
    void setup() {
        interval_1_10 = new Interval<>(0, 10);
    }

    @ParameterizedTest
    @ValueSource(ints = {1, 5, 10})
    @DisplayName(value = "An interval contains a number between its boundaries")
    void includes(int value) {
        assertTrue(interval_1_10.includes(value));
    }

    @ParameterizedTest
    @ValueSource(ints = {Integer.MIN_VALUE, -1, 11, Integer.MAX_VALUE})
    @DisplayName(value = "An interval doesnt contain a number outside its boundaries")
    void outsideBoundaries(int value) {
        assertFalse(interval_1_10.includes(value));
    }
}

Running Unit and Integration Tests with Maven

Build Lifecycle

Diagram
Figure 4. Maven Build Lifecycle
Plugins:
Surefire

manages the Test phase

Failsafe

manages the Integration Test phase

Surefire

  • Executes all unit tests included in the src/main/test directory

  • Automatically include all test classes with the following wildcard patterns:

**/Test*.java, **/*Test.java
  • but not:

**/*Abstract.java, **/Abstract*.java
  • Test classes must be named

  • Test classes can be any combination of the following:

    • TestNG

    • JUnit (3.8, 4.x or 5.x)

    • POJO

Failsafe

  • Executes all integration tests included in the src/main/test directory

    • but this can be changed for src/main/it

  • Automatically include all test classes with the following wildcard patterns:

**/IT*.java, **/*IT.java, and **/*ITCase.java
  • Test classes can be any combination of the following:

    • TestNG

    • JUnit (3.8, 4.x or 5.x)

    • POJO

Conclusion

Conclusion

  • It is impossible to test a program completely.

  • Testing cannot prove the absence of bugs.

  • Specifications are never final: software is always evolving

Conclusion (Cont.)

  • Test cases are also software: use software engineering best practices:

    • modularity, factorization, reuse, etc.

  • Maintaining old tests up to date is as important as writing new ones.

Final Word

Tests don’t improve quality: developers do!