8 Min. read

Automated Testing With Specs in Unreal Engine

Unreal provides a few different ways of writing tests as part of its automation framework. My personal favorite are spec tests. I’ve been using them extensively as part of my work at Lumeto as well as in my personal projects.

Specs

Spec (short for specification) tests are based on the Behavior-Driven Development (BDD) methodology. Specs provide an intuitive, human-readable syntax for writing tests that focus on validating the expectations of a public API rather than its implementation.

Specs can be used to implement unit, integration and functional tests and can optionally be written to match user stories — such as “As a user I should be able to…” or “As a developer I should be able to…” — allowing to test slices of functionality.

Writing tests

By convention specs in Unreal are implemented in files called <FeatureName>.spec.cpp. No header file is required. Tests can be stored anywhere in the project. Personally I like to group them in a Tests folder in the same module where features are implemented.

A screenshot of the Tests folder, showing some automated test files
Tests for a new plugin I’m working on… ✌

A minimal test file looks like this:

#if WITH_AUTOMATION_TESTS
 
#include "Misc/AutomationTest.h"
 
BEGIN_DEFINE_SPEC(FExampleSpec, "Example",
    EAutomationTestFlags::ApplicationContextMask | EAutomationTestFlags::ProductFilter);
 
    // Variables and functions defined here will end up being member of 
    // the FExampleSpec class and will be accessible in the tests
 
END_DEFINE_SPEC(FExampleSpec);
 
void FExampleSpec::Define()
{
    // Test cases are defined here
}
 
#endif

Under the hood the macros take care of creating a FExampleSpec class that registers our tests with the engine so we run them (more on this in a bit). EAutomationTestFlags work the same as with other automated tests and help the engine understand the type of test that is being implemented.

Everything is wrapped in the #if WITH_AUTOMATION_TESTS conditional to prevent tests from being included in Shipping builds. Individual test cases (AKA expectations) can be defined inside the Define() function.

Specs provide an expressive, lambda-based interface to define expectations. The following is a simple unit test:

void FExampleSpec::Define()
{
    Describe("FExampleFeature", [this]()
    {
        It("Should return true when input is valid", [this]()
        {
            FExampleFeature Feature;
            TestTrue("MakeSomethingHappen", Feature.MakeSomethingHappen(TEXT("Valid input")));
        });
 
        It("Should return false when input is empty", [this]()
        {
            FExampleFeature Feature;
            TestFalse("MakeSomethingHappen", Feature.MakeSomethingHappen(FString()));
        });
    });
}

Expectations are defined using the It() function and should contain logic that validates specific behavior. The Describe() function can be used to group multiple expectations together. Grouping expectations is optional, but very useful as it makes tests easier to organize, locate and run.

Running tests

Specs can be run just like other tests in Unreal, including:

  1. From the Session Frontend (found in the Tools menu) in the editor:
A screenshot of the Session Frontend in the Unreal Editor, showing all available automated tests for the current project
  1. From the command line:
"<path to engine folder>\Engine\Binaries\Win64\UnrealEditor-Cmd.exe" "<path to project>.uproject" -execcmds="Automation RunTests Example;Quit" -stdout -unattended -NOSPLASH -NullRHI
  1. From Rider (or — I think — recent versions of Visual Studio):
A screenshot of the Tests panel in Rider, showing all availabe automated tests for the current project

Fixtures

It’s good practice to make sure each test runs in a controlled environment that provides everything needed, but nothing more. The objects that define the environment and data for a test are usually called fixtures.

Fixtures are normally created before running a test and destroyed afterwards to make sure each test runs in isolation — preventing false positives and side effects. Chances are, however, that groups of tests that validate similar functionality need to run in a similar environment.

Specs provide two handy functions called BeforeEach() and AfterEach() that run before and after each expectation and that can be used to create and destroy fixtures:

BEGIN_DEFINE_SPEC(FExampleSpec, "ExampleSpecTest",
    EAutomationTestFlags::ApplicationContextMask | EAutomationTestFlags::ProductFilter);
 
    // Defined as a class member to make it accessible from each test
    TUniquePtr<FExampleFeature> FeatureFixture;
 
END_DEFINE_SPEC(FExampleSpec);
 
void FExampleSpec::Define()
{
    Describe("As a user I should be able to do something", [this]()
    {
        BeforeEach([this]()
        {
            // Set up fixtures
            FeatureFixture = MakeUnique<FExampleFeature>();
 
            // If the following assertion fails all tests in the current Describe block won't run
            // This is useful in case fixture setup fails for any reason
            TestTrue("FeatureFixture valid", FeatureFixture.IsValid());
        });
 
        AfterEach([this]()
        {
            // Destroy fixtures
            FeatureFixture.Reset();
        });
 
        // Access fixtures inside expectations
        It("Should return true when input is valid", [this]()
        {
            TestTrue("MakeSomethingHappen", Feature->MakeSomethingHappen(TEXT("Valid input")));
        });
 
        It("Should return false when input isn't valid", [this]()
        {
            TestTrue("MakeSomethingHappen", Feature->MakeSomethingHappen(FString()));
        });
    });
}

BeforeEach() and AfterEach() are relative to their scope. In the example above, they will run only for expectations defined inside the parent Describe() block, making it possible to set up different environments for different groups of tests.

Async tests

Specs support async execution out-of-the-box, allowing to test asynchronous game logic, integration with remote services, etc. This is done by using latent versions of the functions we’ve seen above, such as LatentIt() or LatentBeforeEach():

LatentIt("Should return 5 items", [this](const FDoneDelegate& Done)
{
    FMyLatentInterface LatentInterface;
    LatentInterface.OnLatentOperationDone.AddLambda([this, Done](int32 Result)
    {
        TestEqual("Result", Result, 5);
        Done.Broadcast();
    })
});

Testing gameplay features

With some additional work Specs can be used to write tests for many gameplay features directly from C++, without having to rely exclusively on Blueprint functional tests (which in my experience are slower to implement and run).

As an example, let’s create some tests for an actor that should explode when it receives damage. Since actors can only exist inside a world, we need to create one as part of our test fixtures.

UWorld fixture

The following is a fixture class that creates a temporary UWorld suitable for running tests:

class FWorldFixture
{
public:
 
    explicit FWorldFixture(const FURL& URL = FURL())
    {
        if (GEngine != nullptr)
        {
            static uint32 WorldCounter = 0;
            const FString WorldName = FString::Printf(TEXT("WorldFixture_%d"), WorldCounter++);
 
            if (UWorld* World = UWorld::CreateWorld(EWorldType::Game, false, *WorldName, GetTransientPackage()))
            {
                FWorldContext& WorldContext = GEngine->CreateNewWorldContext(EWorldType::Game);
                WorldContext.SetCurrentWorld(World);
 
                World->InitializeActorsForPlay(URL);
                if (IsValid(World->GetWorldSettings()))
                {
                    // Need to do this manually since world doesn't have a game mode
                    World->GetWorldSettings()->NotifyBeginPlay();
                    World->GetWorldSettings()->NotifyMatchStarted();
                }
                World->BeginPlay();
 
                WeakWorld = MakeWeakObjectPtr(World);
            }
        }
    }
 
    UWorld* GetWorld() const { return WeakWorld.Get(); }
 
    ~FWorldFixture()
    {
        UWorld* World = WeakWorld.Get();
        if (World != nullptr && GEngine != nullptr)
        {
            World->BeginTearingDown();
 
            // Make sure to cleanup all actors immediately
            // DestroyWorld doesn't do this and instead waits for GC to clear everything up
            for (auto It = TActorIterator<AActor>(World); It; ++It)
            {
                It->Destroy();
            }
 
            GEngine->DestroyWorldContext(World);
            World->DestroyWorld(false);
        }
    }
 
private:
 
    TWeakObjectPtr<UWorld> WeakWorld;
};

The world created by the fixture is completely empty, which is ideal to make sure tests run in isolation and to avoid the overhead of loading a real level with meshes, textures, etc. The world and everything it contains are automatically destroyed when the fixture is destructed.

The same approach can be used to write fixtures that spawn actors, components or, generally, set up complex environments or data before running a test, and tear them down when the test is over.

Testing the actor

The following is a barebone implementation of the actor and its tests:

MyDestructible.h
UCLASS()
class AMyDestructible : public AActor
{
    GENERATED_BODY()
 
private:
 
    bool bExploded = false;
 
public:
 
    void Damage(float DamageAmount)
    {
        if (DamageAmount > 0.0f)
        {
            bExploded = true;
        }
    }
 
    bool HasExploded() const { return bExploded; }
};
MyDestructible.spec.cpp
BEGIN_DEFINE_SPEC(FMyDestructibleSpec, "MyDestructible",
    EAutomationTestFlags::ApplicationContextMask | EAutomationTestFlags::ProductFilter);
 
    TUniquePtr<FWorldFixture> WorldFixture;
    TWeakObjectPtr<AMyDestructible> DestructibleFixture;
 
END_DEFINE_SPEC(FExampleSpec);
 
void FExampleSpec::Define()
{
    Describe("AMyDestructible", [this]()
    {
        BeforeEach([this]()
        {
            // Create the world
            WorldFixture = MakeUnique<FWorldFixture>();
 
            if (TestNotNull("World", WorldFixture->GetWorld()))
            {
                // Spawn the actor
                DestructibleFixture = WorldFixture->GetWorld()->SpawnActor<AMyDestructible>();
                TestNotNull("MyDestructible", DestructibleFixture.Get());
            }
        });
 
        AfterEach([this]()
        {
            // Tear down the world and the actor
            WorldFixture.Reset();
        });
 
        It("Should not explode when dealt zero damage", [this]()
        {
            MyDestructible->Damage(0.0);
            TestFalse("HasExploded", MyDestructible->HasExploded());
        });
 
        It("Should explode when dealt damage", [this]()
        {
            MyDestructible->Damage(1.0);
            TestTrue("HasExploded", MyDestructible->HasExploded());
        });
    });
}

Limitations

  1. Specs, just like other automated tests (excluding Gauntlet tests), can’t be used to implement multiplayer tests at least out-of-the-box, although engineers at Rare were able to overcome this limitation.
  2. Specs can only be implemented in C++, which might be a good or bad thing depending on context and constraints.

Further reading

Updates

  • 26 Jan. 2024: improved01 code listings based on this Mastodon conversation.
  • 15 Feb. 2024: added link to Unreal Fest 2023 presentation.