Using AI to test our game, not just build it

One of our AI testers once reported that a freighter had defeated a heavily defended enemy planet. At first glance, that sounded impossible.

It turned out not to be a bug. The freighter was cloaked, and the defending forces had nothing they could actually target.

That small moment captures something important about how we test Exodus-Protocol. AI agents are extremely useful for exploring the game, validating expected behavior, and spotting suspicious outcomes. But they still need human review. They are testers, not final judges.

We are building Exodus-Protocol, a persistent multiplayer space strategy game played through an AI assistant. That means testing is not only about whether code works, but also whether a conversational interface, a persistent galaxy, and a complex combat system behave as intended over time.

The normal testing layers still matter

We do not rely on AI alone.

For the traditional development side, we still use the testing layers most teams would expect. We use unit tests for backend logic and API behavior. On the frontend side, we use Playwright for browser-based end-to-end testing for both the management portal and the client.

A large part of our UI work is built and tested through Storybook. That works especially well in combination with atomic design. We build atoms, molecules, and larger components in isolation, often against mocked API responses, and validate them there before combining them into full screens. That gives us fast iteration and predictable frontend behavior, while the API itself is validated separately through direct tests.

What makes the project unusual is what happens after those layers.

Testing a game that keeps running

Exodus-Protocol is not a game where every result is visible immediately. Fleets travel. Structures take time to build. Research progresses in the background. Colonization is not instant. You can speed up the galaxy clock during testing, but even then, many important behaviors still unfold over time.

As a small team, I do not want to spend a large part of the week manually waiting for timers to finish and then checking whether the result matches the specification.

So we built testing agents.

These agents have their own in-game accounts and play through our application's text interface, using Model Context Protocol (MCP) to request information and perform actions. They expand their empires, explore systems, colonize planets, gather resources, build structures and fight battles.

Because conversation is already the game's interface, the agents do not need to navigate it through browser automation. They ask questions, give instructions and check the resulting game state. The Playwright tests described earlier cover a separate part of our development process.

Agents that verify instead of only act

The agents are not just issuing random commands. They are aware of the functional specification of the game and use it as a reference for expected behavior.

If an agent starts constructing a building and the server returns an estimated completion time, the agent schedules a follow-up check at or shortly after that time. It then validates whether the action completed correctly and whether the timing was in line with what was promised.

The same principle applies to many other features. The agent is not only trying to progress, but also validating the outcome of that progress.

When something looks wrong, the agent logs it as a potential issue. That report includes reproduction steps, timestamps, and relevant IDs so we can inspect what happened on the server.

Some of those reports are genuine bugs. For example, one issue the agent found was a level 1 module being more expensive than the level 2 version. Another was a case where aborting a mission could create a fuel problem during the fleet's return path because the fleet might no longer be able to reach its next refuel stop.

Some reports are false positives. The cloaked freighter is a good example. The agent saw an unexpected result, flagged it, and we then determined that the game behaved correctly.

That is why we review the reported issues weekly. We decide whether they are valid, already solved or the result of a misunderstanding. When an agent misinterprets something, we improve its instructions.

But improvements do not only come from us. The agents also update their own Markdown files with what they learn about the platform. These files become a growing reference for future sessions, allowing the agents to build on previous discoveries and develop their approach to testing.

In other words, they are not only reporting what might be wrong with the game. They are also recording what they have learned about how to play and test it.

That does not make their conclusions automatically correct. The functional specification remains the reference for expected behavior, and potential bugs still go through our review.

Once a month, the agent also goes back through the list of open issues and tries to reproduce them again. That matters because the game is constantly evolving. Some issues disappear naturally when an unfinished feature becomes complete. The monthly pass helps us separate still-relevant problems from issues that have quietly gone away.

From testing the game to balancing the game

Gameplay validation is one side of the story. Balance is the other.

Exodus-Protocol includes ships, defenses, structures, modules, and other systems with many interdependent statistics. Every unit has a role. The aim is not simply to make everything equally strong, but to make different choices viable.

A large part of our combat model follows a rock-paper-scissors or Stratego-like philosophy. Units and defensive structures are designed to have strengths and weaknesses. A counter should feel meaningful, but not so absolute that one answer always decides the fight. That principle is one of the foundations of our balance work.

At first, we used a battle simulator manually. It showed not just logs, but also a visual representation of what happened in battle: which ship attacked, in what order, and when units were destroyed. It was useful, but also slow. Setting up a scenario, running it, and reviewing the outcome quickly turned into minutes per battle. We still use the visual simulator in some cases to inspect how a battle unfolds.

So we exposed the simulator through an administrative MCP interface and let agents use it.

This changed the process completely. The agents can run large batches of simulations much faster than we can do manually. So far, we have already run around 5,000 simulations to help balance the base statistics of ships and structures.

The metrics we look at

One important balance reference is damage per thousand resources. That helps us compare output relative to cost.

But that is only one lens. We also look at damage per action point, because action economy matters. On top of that, we look at eDPA, expected damage per action point, where hit chance and dodge chance are taken into account. That gives us a more realistic picture than raw theoretical output alone.

There are also additional balance checks we take into consideration, but we are not going to share every detail here. Some things players should discover for themselves.

What matters is that we do not evaluate a unit in isolation. We evaluate whether it performs its intended role, whether that role survives when modules are applied, and whether the result is still in line with its price and construction time.

The bomber is a good example. It is supposed to be strong against capital ships and structures while remaining vulnerable to lighter ships. That sounds straightforward, but it is not. If you give the bomber enough armor or shielding to survive long enough to fulfill its anti-capital role, you may accidentally make it too resilient against the very units that are supposed to counter it.

That is where repeated simulation becomes essential. You are not just asking whether a ship wins. You are asking what it wins against, under which circumstances, at what cost, and whether its role drifts when the surrounding system changes. We repeat scenarios with different random seeds to see how much outcomes vary, rather than judging a matchup from a single battle.

Letting the agents tune the numbers too

More recently, we extended the administrative tooling further. The agents are now able not only to run simulations, but also to change values themselves.

In practice, that means we can define a desired outcome, and the agent can explore different stat combinations, run battles, compare the results, and work toward a better fit. Instead of manually tweaking every number ourselves, we let the agent do a large part of the repetitive exploration.

This does not mean the agent designs the game. It means the agent helps us search the balancing space much faster than we could on our own.

For us, that is the real promise here.

AI is not only helping us build Exodus-Protocol. It is also helping us test it, challenge it, and tune it. The final decisions still belong to us. But as a small team, these systems allow us to validate more, simulate more, and learn faster than we otherwise could.

That does not guarantee success.

But it does make the experiment much more interesting.