Add an option to rerun failed executions

[thymikee]

SkillGym suites can include nondeterministic agent runs where a single case x runner execution fails transiently. It would be useful to have a built-in way to retry only failed executions before the final result is reported.

Potential shape:

• config: run.retryFailed?: number
• CLI: skillgym run ... --retry-failed <n>
• behavior: rerun only failed case x runner executions up to n additional attempts
• reporting: preserve all attempt artifacts and show whether the final pass came from a retry

This would avoid custom wrapper scripts around --case / --runner, and would make broad benchmark runs easier to use when one agent response is flaky.

[thymikee]

One adjacent observation from testing v0.6 with full available parallelism: a 48-run suite group using schedule: parallel and default Parallel 12 completed successfully, but Node emitted MaxListenersExceededWarning for SIGINT/SIGTERM after more than 10 active listeners were added. It did not fail the run, but retry-failed support would likely be used together with high parallelism, so it may be worth accounting for listener cleanup or increasing the listener limit internally.

[V3RON]

Implementation plan

1. Add retry configuration and CLI plumbing.

• Add run.retryFailed?: number to SkillGymConfig in src/config.ts.
• Parse and validate retryFailed like maxParallel: integer >= 0.
• Add CLI flag --retry-failed <n> in:
  • src/cli.ts
  • src/cli/run.ts
  • src/cli/help.ts
• CLI value should override config value.

2. Keep retries inside one logical case x runner execution.

• Do not change src/runner/scheduler.ts.
• Implement retries inside src/runner/execute-suite.ts, around the current single execution flow.
• Preserve existing lifecycle semantics:
  • one onRunnerStart
  • one final onRunnerFinish
  • one CaseResult.runnerResults[] entry per runner
• This avoids churn in scheduling, concurrency, and case-completion logic.

3. Extract a small retry policy abstraction.

• Add a minimal internal interface/function used by execute-suite.ts to decide whether a failed result should retry.
• Default implementation:
  • retry any failed run
  • never retry model rejection
  • stop when attempts exceed configured limit
• Keep it simple and swappable so future policies can narrow retries by failure type/origin without redesigning the executor.

4. Make artifacts attempt-aware.

• Change artifact layout from:
  • <outputDir>/<case>/<runner>
• To:
  • <outputDir>/<case>/<runner>/attempt-1
  • <outputDir>/<case>/<runner>/attempt-2
• Keep RunnerResult.artifactDir pointing to the final attempt directory.
• This preserves existing reporter assumptions while retaining all failed-attempt logs.

5. Make isolated workspaces attempt-aware too.

• Change preserved workspace path from:
  • <outputDir>/workspaces/<case>/<runner>
• To:
  • <outputDir>/workspaces/<case>/<runner>/attempt-1
  • etc.
• This is required so a later retry does not delete a failed earlier attempt’s preserved workspace.

6. Extend the final result model minimally.

• Add small retry metadata to RunnerResult in src/domain/result.ts, such as:
  • attempt
  • attempts
  • retried
• Keep the final attempt as the only primary result.
• Do not expose full attempt history in results.json yet.

7. Update reporting to surface retry outcome clearly.

• standard reporter:
  • keep current success/failure display
  • add a short indicator when the final result came from retry, like retry 2/3
  • add a visible rerun section in the final summary
• json / json-summary:
  • include the new retry metadata fields in the serialized final result
  • stay pure JSON, with no prose outside the JSON payload
• github-actions:
  • include retry metadata in annotations and job summaries
  • add a reruns section in the job summary
• Only actual reruns should appear in rerun-specific reporting. Non-retriable failures like model-rejected stay only in normal failure reporting.

8. Preserve current model-rejection short-circuit behavior.

• In src/runner/execute-suite.ts:
  • if a result is identified as model-rejected, stop retrying immediately
  • mark the runner as rejected only from that final non-retriable outcome
• Later cases for that runner should still be skipped as they are today.
• These skipped runs should not be listed as reruns.

9. Add tests for the full retry behavior.

• Config/CLI:
  • parse run.retryFailed
  • CLI override
  • invalid values
• Executor:
  • retries until pass
  • stops after limit
  • no retry when model is rejected
  • preserves final ordering and case lifecycle
  • keeps per-attempt artifact dirs
  • preserves isolated failed-attempt workspaces
  • emits only one runner lifecycle pair per logical execution
• Reporters:
  • retry indicator appears in standard output
  • rerun section includes only actual reruns
  • retry metadata appears in JSON-style outputs
  • GitHub Actions summary and annotations reflect retries

10. Update docs.

• README.md
• docs/reporters.md
• likely docs/workspaces.md
• Document:
  • run.retryFailed
  • --retry-failed
  • retries are additional attempts
  • only failed executions retry
  • final result is primary
  • all attempt artifacts are preserved
  • final artifactDir points to the last attempt
  • rerun reporting only covers executions that actually retried

11. Apply the adjacent listener-limit fix separately but in the same change if desired.

• Raise process max listeners once at startup to suppress MaxListenersExceededWarning during high parallelism.
• Prefer finite cap like 64, not unlimited.
• Keep existing listener cleanup in src/utils/process.ts unchanged.

Why this plan:

• Smallest safe design.
• No scheduler rewrite.
• No broad reporter contract change.
• Only unavoidable refactor is centralizing attempt-aware artifact/workspace path generation.

Open implementation detail:

• For the listener cap, I recommend 64 unless you expect routinely higher than ~30 concurrent child runs.