System Test execution is done in different domains. The domains contain the operational work.

Test Planning#

Input: reasons to test (Test Trigger) → Doing: convert into needed test activities → Output: Test Execution Plans

Test Trigger

  • Generic Test Trigger: following a pre-defined plan
  • Erratic Test Trigger: due to unplanned or random events

Planning Dimensions

  • What to test? test object, context, version, scope
  • Where to test? test infrastructure, test environment
  • How to test? test cases, test sets, test instructions, automation
  • Who to test? individual, team, or automation
  • When to test? relevant start/end date, time constraints

Test Order

Prioritize certain tests. Always test from general/major features to specific/minor features. Corner cases and negative use cases should be tested afterwards.

Definition of Ready & Done

Definition of Ready: preconditions for test execution beyond test case specifics.

Definition of Done: criteria to end tests — completeness, handling of aborted/failed tests, time-based closure.

Test Specification#

Input: requirements → Doing: convert into test cases/test sets → Output: executable test cases

Test Basis

Anything that influences how the product must, should or may behave: requirements, use cases, expert discussions, KPIs, regulations.

Test Case Structure

  • Precondition: state in which the system must be
  • Actions: sequential steps to perform the test
  • Expectation: specified observable outcome

Test Case Types

Smoketests, bug retests, functional tests, use case tests, performance tests, usability tests, acceptance tests, interoperability tests, regression tests, assertions testing.

Test Case Creation Guidelines

  • Scope should test one specific behavior
  • Clear and concise content, unambiguous terms
  • Executable actions, measurable expectations
  • Action-based language (e.g. 'press the START button')
  • Avoid implicit knowledge requirements

Test Infrastructure#

Input: project environment → Doing: create up-to-date test infrastructure → Output: usable test environment

Test Infrastructure Types

  • Full-system: all components available, end-to-end, closest to customer experience
  • Partial-systems: features/components under test, efficient and flexible
  • Simulation-based: partly/fully simulated, scalable, enables early testing

Hardware-in-the-Loop (HiL)

Simulates the physical world around the system. Used for autonomous driving tests to simulate vehicle dynamics and environment.

Software-in-the-Loop (SiL)

Tests software components in a simulated environment rather than on real hardware. Usually a predecessor to System Test.

Test Execution#

Input: test execution plans → Doing: execute tests → Output: test results

Verification and Validation

Verification checks if the behavior matches the specification. Validation has a broader view and also reviews the specification itself. Holistic System Test incorporates both.

Test Result

If the system behaves as expected the test passed, if not the test failed. For each failed test case an issue must be raised.

Issue Report

A well-documented issue is the foundation for smooth issue resolution. Essential information includes:

  • Timestamp, tester name, test case reference
  • Precondition, expected behavior, actual behavior
  • Reproducibility, priority and severity
  • Supporting data: traces, logs, pictures, media

Problem Management#

Input: issue reports → Doing: handling issues → Output: issue resolution

Issue Status Model

Simplest form: New → Progress → Closed. Can expand with: Triage, Pre-Analysis, Rejected, Review, Accepted, Fixed, Resolved, Retest.

Issue Prioritization

Priority groups: Blocker, High, Medium, Low

Severity groups: Critical, Major, Moderate, Minor, Trivial

Issue Workflow

Screening → Pre-Analysis → Triage → Tracking → Monitoring → Resolution → Retest

Reporting#

Input: test results → Doing: data analysis and visualization → Output: report

Reporting Types

  • Issue Reporting: current issues, blockers, statistics
  • Test Progress Reporting: test plan and execution status
  • Release Related Reporting: milestone-focused, KPI-based
  • Quality Reporting: high-level view incorporating quality topics beyond testing

Status Colors

  • Green: all ok, on track
  • Yellow: there is an issue, but it is handled
  • Red: there is an issue without resolution