SCIFILIZATION

Testing for a Civilization project

I started this project doing all the testing myself through playing the game. This went on for about six weeks as I combed through all the 'hard crashes', the freezes, and the creeping corruption that existed in the code base that had been abandoned since 2017. It was fun, but it wasn't as productive as it could be, and I came to realize that I was finding the same sort of bugs over and over again. I would need some automation.

So I asked Claude to create an auto-play mode. This remains a feature of the game today, you can find it in the Options menu along with a "watch mode" that lets you see all the enemy moves at the same (slow) speed as your own units move.

But with more testing I came to realize that the game behaved differently with different numbers of civilizations. It was at this point that I suggested to Claude that we needed to automate testing for such variables. I had written a CLAUDE.md file that asked it to be strict with what it was doing. With this ask, it new it needed a full test suite.

Tests that play the game

Scifilization went from no automated tests in June 2026 to 2,355 in November. They catch AI and rules bugs that playtesting finds weeks later or never, and they let us change a 900-line AI planner without fear. I'd like each of our projects to have the same safety net, starting small.

2,355
automated tests, built since June 2026

35 min
full run on a laptop

~15 min
for a 320-turn game with no screen

338
test files, one per topic or bug

Why

Civ bugs are slow failures

A change that looks fine on turn 1 can ruin turn 400. Nobody playtests 400 turns after every change, but a machine can. These three came out of Scifilization in the past week alone.

What players saw

The real cause

How it was found

The world ended 94.7% ocean

Global warming had no cap. Each flood turned the coast to swamp, the next drowned the swamp. One game had 8 warming events.

Map statistics from a played game, then a test per rule

The alien defence project finished after the invasion landed

Its parts went to civs 52–71 turns away from the technology they needed

Headless games across 6 seeds; after the fix all 6 finished in time

Some civs built Explorers endlessly

A fallback rule built them after the map was charted; the AI then disbanded them. One civ built 16 and disbanded all 16.

One headless game with unit losses logged; fixed in one line

What we built

Three layers, from one decision to a whole world

seconds · one decision

Scenario tests

Build a tiny world in a few lines of code: an 80×50 map, paint the terrain, place two cities, give a civ its advances. Then ask the AI one question and check the answer.

Example

“A charted civ's fallback never builds an Explorer”, checked over 100 random picks

minutes · a whole game

Headless autoplay

Every civ played by the AI, no screen, on a fixed random seed. Run the same seeds before and after a change and compare.

Example

Six Epic Earth games to turn 600 to check one change to the alien defence race

after the fact · every choice

Decision log

When switched on, the game writes one line per AI choice: what each city built, who declared war, how long each turn took. Small scripts answer questions from it.

Example

“Which civs built Explorers, and when?” answered in seconds from a 10 MB log

The fast layer runs on every change. The slow layers run when a change touches the AI or the economy.

The one rule

Watch every test fail before trusting it

Switch the fix off, confirm the test fails, switch it back on.

A test that passes proves nothing until you have seen it fail. Ours have passed against broken code many times because something unrelated happened to give the right answer.

It happened again in early November. The new Explorer test passed with the fix switched off. At the easiest difficulty level the city always asked for a Militia first, so it never reached the rule under test. One level up, the test failed as it should. Without the rule we would have shipped a test that checked nothing.

Working with an AI coding agent

Tests are what make an agent trustworthy

I'm not a developer. Most of this suite was written by Claude Code, an AI coding agent, working to a short rules file in the repository. The suite is what lets me accept its changes: every fix arrives with a test it has shown to fail without the fix, and the full run must pass before I commit. If any of you use an agent, the same rules file would carry over.

What it costs

The costs are real but manageable

How to start

A first step for each project

  1. Make the game run without a screen. Everything else depends on this. If game rules call the display directly, that is the first thing to untangle.
  2. Write a helper that builds a small world. Map size, terrain, a city, a unit, a civ's advances. Ours is about 470 lines and every test uses it.
  3. Add a test with every bug fix from now on. No backfilling. Watch each one fail first.
  4. Add a seeded headless game that runs a number of turns and prints a summary: cities, sizes, advances, wars.
  5. Log AI decisions as one JSON line each, off by default so players don't fill their disks.

Assets

What you can read and borrow

All of it is in the public repository, mwerneburg/Scifilization. It's written in C#, so a CivOne-based project can adapt the code directly; for anything else it shows the approach.

File

What it shows

CLAUDE.md

The testing rules, including “watch it fail”, in under a page. Readable by people and by AI agents.

tests/Sim.cs

The small-world helper: new game, run turns, reset state between tests.

tests/DomeBuilderTests.cs

A typical scenario test file: a two-city fixture and six one-question tests.

tests/SleepingExplorerTests.cs

A short file with a random-pick test that runs 100 times to be sure.

tests/AutoplayHarness.cs

The headless game runner, set up with environment variables for seed, map and turn count.

src/DecisionLogger.cs

The decision log: what it records and how it stays off for players.

tools/decisions.py

Quick questions answered from the log: turn timing, production choices, settlers.

tools/run-report.py

A summary of any saved game, read straight from the save file.

Licence: Scifilization is GPL-3.0-or-later. The ideas are free to use anywhere. Copying the code into a project under a different licence brings GPL terms with it, so check before you paste.

Figures as of 8 November 2026, from the Scifilization repository and its test runs.