Spread benches

gdharness spread plays a bench's runs over several headless engines at once and prints the same table the bench prints when it runs alone, to the last digit. A bench whose runs share nothing (run i depends on i alone, its seed worked out from the index) is cut into one share per worker, and the outcomes of every run are gathered back in run order before the bench folds them.

gdharness spread res://bench/balance.tscn
gdharness spread res://bench/balance.tscn --workers 4
gdharness spread res://bench/balance.tscn -- --seed=3

The first takes as many workers as the machine has room for, and the last hands --seed=3 to every engine.

It exits 0 when every worker played its whole share and the gathering run printed no error, 1 when anything went wrong, and 2 when the command line itself is wrong.

• --workers N: how many workers. By default as many as the engine slots allow, held lower when the machine has too few processors or too little memory, as gdharness test holds its engines.
• --slots DIR, --slot-count N: the engine slots, the machine's own by default, shared with gdharness test and described in Test runs. Each worker and the gathering engine hold one, so a bench and a gate running at once start no more engines between them than there are slots.
• --silence S: the shortest silence that ends an engine, in seconds; 300 by default. A worker prints a line every tenth of each arm's share, so a long arm should print well inside it.
• --deadline S, --godot PATH, --godot-version V, --cores N: as for gdharness test.
• Everything after -- is handed to every engine as the bench's own arguments.

Writing the bench

The bench is a scene. For each arm, it asks GdhShare.outcomes for the outcomes of that arm's runs, handing it the arm's name, the number of runs and a callable that plays run i and answers what the run came to. GdhShare is in the tests addon, addons/gdharness_tests, which gdharness setup installs.

extends Node


func _ready() -> void:
 var balance: Array = GdhShare.outcomes("balance", 120, _played)
 if not GdhShare.working():
  var totals: Dictionary[String, int] = GdhShare.added(balance, ["most_turns"])
  print("won %d of 120, longest %d turns" % [totals["won"], totals["most_turns"]])
 get_tree().quit()


func _played(run: int) -> Dictionary:
 var duel: Duel = Duel.new(SEED + run)
 duel.play()
 return {"won": 1 if duel.won else 0, "most_turns": duel.turns}

The same scene runs three ways, and GdhShare.outcomes answers in each:

• Alone, as godot res://bench/balance.tscn runs it, it plays every run and answers every outcome.
• In a worker, it plays every Nth run from the worker's own index, so each worker draws long runs and short ones alike. It leaves the outcomes for the gathering engine and answers an empty array. GdhShare.working() is true there, and the bench prints no table.
• In the gathering engine, started once every worker has finished, it plays nothing and answers every run's outcome in run order, each read back as the type it was written as.

The bench folds the outcomes the same way in all three, so a float summed in run order comes out the same as when the bench runs alone. GdhShare.added adds up outcomes that are dictionaries of whole numbers key by key, and keeps the largest for the keys it is given.

What a bench has to keep to:

• An outcome is a plain value: a number, a string, a vector, or an array or dictionary of them. An object cannot be handed from one engine to another; GdhShare refuses one with an error, which fails the run.
• Which arms a bench plays, and how many runs each has, depend on its command line alone. Every engine runs the whole scene, so an arm chosen by the clock or by a file one worker wrote is a different arm in another engine. A worker that plays other arms than worker 0, or a gathering run that skips an arm the workers played, fails the run.
• Each arm has its own name. An arm asked for twice in one run is refused.
• The scene quits when it is done. An engine that never quits is ended after its silence and fails the run.
• Each engine has its own user://, as test engines do, so two workers never write one save or settings file. A bench that reads something from user:// finds it empty.

No engine of a spread run announces itself to the gdharness server: the runtime stays off in an engine handed the spread's arguments, whatever folder its scene is in.

What fails a run

A spread run fails before anything is gathered when a worker:

• was ended for its silence or its deadline, or exited with a code other than 0;
• printed an engine error, the first of which is named;
• played no arm, which is a scene that never asked GdhShare.outcomes, or quit before it did;
• played other arms, or other numbers of runs, than worker 0.

It fails after the gathering when the gathering engine printed an error or exited with a code other than 0, or never asked for an arm the workers played. The gathering engine's output is printed either way, followed by the faults, so a table printed by a run that failed is marked as one. Each engine's output is in .godot/gdharness/spread-runs/<start>-<pid>/<engine>/engine.log, which a failed run names and keeps until five later runs have failed.