Running in the Dark

TLDR: In UE 5.8, a Level Sequence with a Control Rig or Skeletal Animation track forces every mesh it binds to animate every frame, on screen or off. Pausing or stopping the sequence doesn't undo it. A handful of background characters were costing us around 50 ms of worker time a frame with nobody looking at them - plus a workaround you can ship today.

The last two posts were crashes. This one isn't. Nothing falls over, nothing gets logged, and the game thread looks perfectly healthy. The cost is all on the worker threads, which is exactly why it had gone unnoticed ;p

The Symptom #

A profiling pass on a UE 5.8.4 project. Standing still, nothing animated in view, and the CSV profiler's AnimationParallelEvaluation/TotalTaskTime was sitting at 45 to 55 ms a frame, summed across the worker threads.

The culprits were a handful of background characters in a streamed level, all bound to Control Rig tracks in one looping, auto-playing Level Sequence. They're only visible from one part of the map, but their level stays loaded for much longer. (All but one of them showed the forced tick option when I listed them at runtime. I never found out why that one didn't.)

The obvious first move is to pause the sequence when nobody can see it. That removed the sequence's 3 ms or so of game thread evaluation .. and left the worker time exactly where it was.

Where It Comes From #

AnimCustomInstanceHelper.h, in FAnimCustomInstanceHelper::BindToSkeletalMeshComponent:-

// make sure to tick and refresh all the time when ticks
// @TODO: this needs restoring post-binding
InSkeletalMeshComponent->VisibilityBasedAnimTickOption = EVisibilityBasedAnimTickOption::AlwaysTickPoseAndRefreshBones;

A few lines further down it does the same to every child skeletal mesh component. The TODO rather gives the game away.

• Both the Control Rig track and the Skeletal Animation track bind through this helper, so both force the option.
• The matching unbind function doesn't put the old value back.
• With AlwaysTickPoseAndRefreshBones, USkeletalMeshComponent::ShouldTickPose returns true whether the mesh is rendered or not. The anim graph runs every frame, and so does the Control Rig inside it. Cloth on those meshes keeps simulating too.
• The bind happens when the track is instantiated, not every frame. Setting the placed actors to OnlyTickPoseWhenRendered in the level does nothing, as the bind overwrites it when the sequence starts.

What Puts It Back #

The only restore path I could find is Sequencer's pre-animated state, which only applies if restore state was captured in the first place:-

The two "Yes" rows are from reading the source, not from testing - our sequence loops.

In real-life terms: a film set. The director tells the extras "keep going, whatever happens, even if the camera isn't on you". Then the director calls a break and wanders off for a cup of tea. Nobody tells the extras. They're still out there, miming conversations in an empty street, and they're all on the clock.

The sequence is the director, and Pause() is the tea break. The standing instruction to the extras is the tick option, and it stays in force until someone goes out and tells them otherwise.

The Numbers #

Development build, 1920x1080, standing where none of those characters is visible. Two alternated runs per variant on a high-end desktop PC. Figures are rounded.

First, changing only the sequence state:-

The worker figures move by several ms between runs. The point is that they don't fall.

Then pausing the sequence and resetting the tick option on their meshes, switched by a console variable on the same build:-

Frame time fell by less than the game thread did, because the render thread became the limit in this build.

The Workaround #

A tickable world subsystem, checking four times a second:-

1. For each watched sequence, collect the skeletal mesh components it has bound.
2. Set each one back to OnlyTickPoseWhenRendered, on every check.
3. If none was rendered in the last 0.5 s, and none has been within a set distance of the camera for 2 s, Pause() the sequence.
4. When one is rendered or near again, Play().

The part that matters is small:-

// Called four times a second for each watched sequence.
void USequenceParkingSubsystem::Update(FWatchedSequence& Watched, ULevelSequencePlayer* Player, bool bRelevant, double Now)
{
	// Re-apply every check: Sequencer forces AlwaysTickPoseAndRefreshBones again if it re-binds.
	for (const TWeakObjectPtr<USkeletalMeshComponent>& WeakMesh : Watched.Meshes)
	{
		if (USkeletalMeshComponent* Mesh = WeakMesh.Get())
		{
			Mesh->VisibilityBasedAnimTickOption = EVisibilityBasedAnimTickOption::OnlyTickPoseWhenRendered;
		}
	}

	if (bRelevant) // a mesh was rendered recently, or is near the camera
	{
		Watched.LastRelevantTime = Now;
		if (Watched.bPausedByUs)
		{
			Watched.bPausedByUs = false;
			Player->Play(); // only resume what we paused ourselves
		}
	}
	else if (Player->IsPlaying() && Now - Watched.LastRelevantTime > HoldSeconds)
	{
		Watched.bPausedByUs = true;
		Player->Pause(); // keeps the bindings, so resuming doesn't re-initialise the rigs
	}
}

The bind only happens when the track is instantiated, so a value you set afterwards sticks while the sequence is playing or paused. Re-applying it on every check covers a re-bind.

A mesh that's near but not yet on screen resumes the sequence, but still doesn't tick its pose until it's drawn. That's intended.

Two things to know. A character first seen from beyond the distance can show its paused pose for up to a quarter of a second. And the sequence resumes from where it was paused, not from where it would have got to.

Other Options #

Bake the rigs. If the background motion doesn't need a rig at runtime, bake the Control Rig tracks to animation sequences and play those on Skeletal Animation tracks instead. Measured near the characters with the sequence playing, worker animation time went from around 50 ms to under 3 ms and the game thread dropped by about 3.5 ms. The tick option is still forced (the animation track sets it too) - the forced work is just cheap now.

It also got rid of a frame of well over 200 ms when the level holding the sequence was added to the world, which was the rigs initialising inside BeginPlay. After the bake that was 50 to 80 ms, and no longer on the game thread.

For a sequence that ends, set its sections to Restore State, or set the player's completion override to force restore state before Play(). From the source this restores the option. Not tested here, for the same reason as above.

What Epic Could Do #

Honour the TODO: remember the component's previous tick option at bind, and restore it at unbind whatever the section's completion mode. A narrower version would be to stop forcing the option while the player is paused or stopped. Either needs care in the editor, where Sequencer relies on the mesh updating while you scrub.

I haven't written or tested an engine fix for this one.

Check Your Own Project #

• In a packaged Development build, run a CSV profile somewhere the sequence's characters aren't visible. Look at AnimationParallelEvaluation/TotalTaskTime and Exclusive/GameThread/MovieSceneEval.
• List the skeletal mesh components whose tick option is AlwaysTickPoseAndRefreshBones at runtime, and compare with what the level has saved. Meshes bound by a sequence will differ.
• Pause the sequence and profile again. If the worker time doesn't fall, this is the cause.

I've only checked 5.8.4. The TODO suggests it's been this way for a long time.

Takeaways #

• A Level Sequence with a Control Rig or Skeletal Animation track changes the tick option on every mesh it binds, and on the common paths nothing changes it back.
• Pausing or stopping a sequence saves its game thread evaluation and none of the worker cost.
• Worker thread cost doesn't show up as game thread time. Profile AnimationParallelEvaluation/TotalTaskTime as well, somewhere with nothing animated on screen.

• Previous: Waiting for Navmesh