Benchmarking MOOS-IvP Skills

Study setup

The MOOS-IvP Skills were benchmarked across 440 runs: 4 models × 11 tasks × (5 baseline runs + 5 skills runs).

Task

A MOOS-IvP-related task like building an app, behavior, mission, test harness, or analyzing mission logs.

Run

One agent attempt on one task. It runs in a fresh sandboxed workspace, isolated from other attempts, and ends with the agent's final response and output files if applicable.

Model

The language model making the attempt. The study uses GPT-5.6 Luna, GPT-5.6 Terra, GPT-5.6 Sol, and GPT-6 Astra, all at high reasoning effort.

Baseline / skills

The two conditions. Baseline means the plugin is unavailable. Skills means the plugin is available. For each task and model, both conditions are run five times.

Repeating each task five times in each condition shows how consistently a model succeeds on the same request. Reviewers judge the delivered work and its retained evidence after each run ends.

How results were judged

Reviewers checked whether the result was functionally complete and how well it followed MOOS-IvP engineering practices.

A result counts as complete only if it satisfies every functional requirement in the request. If something required is missing, only partly works, or cannot be verified, the attempt counts as incomplete. For the log tasks, the requested result is an explanation or list of events supported by the logs. Completion rate is the percentage of attempts that satisfy the whole request.

Conformance refers to the type of criteria that, if omitted, would make an experienced MOOS-IvP developer pause and say “something isn’t right here”. A result can therefore be complete while losing conformance points: it does what the request asks but misses an important MOOS-IvP convention. Reviewers assess both conditions against the same criteria.

A concrete example: Task 2

Task 2 asks the agent to build pContactWaypoint, an app that directs a vehicle toward a selected contact. The same app is judged in two different ways.

Completion criteria. All five must be satisfied:

• pContactWaypoint builds as a MOOS app.
• The configured contact determines which reported position is used.
• Valid moving-contact reports produce configurable active-waypoint updates.
• Missing or invalid contact data never produces an invalid waypoint.
• Operator-facing status identifies the selected contact and current waypoint.

Example conformance criteria. These are five of the engineering criteria applied to the task:

• Configuration is read in OnStartUp(), with invalid or unknown settings handled clearly.
• Incoming contact reports are type-checked before use.
• Recurring waypoint calculations and publications happen in Iterate().
• The standard --help, --example, and --interface commands work and accurately describe the app.
• The app can be launched and exercised through a normal MOOS mission using pAntler and a ProcessConfig block.

Completion is recorded as complete or incomplete. Conformance is scored as the share of applicable criteria fully met. A partly met criterion means the mechanism exists but is incomplete or inconsistent. Partial and unmet criteria receive no credit. Eight fully met criteria, one partly met, and one not met produce an 80% conformance score.

We average each run’s conformance score across all runs, including incomplete ones. Each run and each applicable criterion receives equal weight. The percentage measures adherence to the tested practices. Later examples show what was missed. The chart-data download also includes a half-credit alternative.

Tasks and conditions

Prompts and workspaces

Every model receives the same written request for a given task. The request describes the desired result but does not include the grading criteria.

For Tasks 1–9, each run begins with the same clean repository based on the standard moos-ivp-extend template, with its usual bin, lib, missions, scripts, and src layout and the files needed to build it. It contains no task examples or grading material. For Tasks 10–11, the inputs are three original mission logs.

Every attempt gets a fresh sandboxed workspace, isolated from other attempts. Both conditions have the standard MOOS-IvP toolchain and common development tools such as CMake, Python, Bash, Git, and a C++ compiler. They also have public internet access, which approximates the environment of a real user working in the project. Apart from plugin availability, the two conditions share this environment.

Each attempt has a 180-minute ceiling and no token ceiling. Tasks 6–9 include components similar to earlier tasks, but each starts from the starter repository.

Skills runs receive the plugin. Baseline runs can still inspect the project and available MOOS-IvP resources.

Eleven tasks, five kinds of work

The benchmark covers several of the plugin’s central workflows and their handoffs: applications, IvP behaviors, ordinary missions, evaluation and harnesses, and log analysis. It does not separately benchmark every installed skill. Map building, installation, and repository setup are not independent tasks here. Skills runs had access to the full plugin rather than a single selected skill.

Results by model

Across all four models, agents with skills completed 166 of 220 attempts (75.5%), compared with 131 of 220 (59.5%) without the plugin. Mean engineering conformance rose from 56.2% to 85.7%.

Completion improves for Luna, Terra, and Sol. Average engineering conformance improves for all four models, including Astra, whose baseline already completes almost every task.

Terra makes the largest observed completion gain, moving from 19 to 36 complete runs out of 55. Across all four models, skills produce 35 additional complete attempts, a 15.9 percentage-point increase in completion. This is an absolute difference between rates, rather than a relative percentage increase.

Astra’s baseline completes 52 of 55 attempts, compared with 50 with skills. Its overall baseline advantage is concentrated in the harness task. Skills do not improve Astra’s aggregate completion in this sample. They do raise its average conformance from 65.1% to 92.2%, a difference that completion alone would miss.

How individual conformance scores shifted

The average gain reflects a broad shift across individual attempts. Median conformance rises from 56.3% in the baseline to 92.0% with skills. Only 3 of 220 baseline attempts reach at least 90%, compared with 129 skills attempts. Scores below 60% fall from 116 to 20.

Across all 440 runs, skills improve both aggregate completion and average conformance. The measures still answer different questions. Of the 132 attempts that meet at least 90% of the conformance criteria, 25 are incomplete because they miss at least one required function.

With only 55 attempts per condition for each model, small differences should be read cautiously. The 95% uncertainty intervals for Sol’s and Astra’s completion differences include zero, so these samples do not clearly separate a skills advantage from a baseline advantage for those models. Average conformance rises by roughly 27–32 percentage points in every model, making it the most consistent measured improvement.

How completed results differ

In each example below, baseline and skills complete all five attempts. The conformance scores show how closely each result follows the specified engineering practices.

Astra’s BatteryWatch application moves from 44.0% to 95.0% conformance. The skills runs fully meet the startup-configuration, recurring-computation, mail-registration, and AppCasting-diagnostics criteria in all five attempts. The baseline runs only partly meet the criterion requiring recurring computation in Iterate() in all five attempts. Sol’s contact-waypoint application provides another example, rising from 79.0% to 98.0% conformance while retaining full completion.

Patrol and shadow mission

Task 5 asks for a two-vehicle patrol and shadow mission with operator controls. Astra completes all five attempts in both conditions. Its mean conformance, however, rises from 56.3% to 95.6% with skills.

The selected conformance criteria cover separate community launchers, configurable MOOSDB and pShare ports, and the components needed to run and supervise the mission: simulation, helm, communications, monitoring, logging, and a viewer. They also cover mission documentation:

All five baseline runs include documentation, but none fully meet the documentation criterion. Four baseline runs also partly meet the runtime-and-operator criterion.

The functional result is tied at 5/5, but skills meet these integration and operation criteria more consistently. Those differences remain visible in conformance after completion has reached its ceiling.

Conformance gains are not uniform across every task and model, and the model-level averages combine those gains and losses. The overall conformance advantage also depends on partial-credit rules: it is 29.6 percentage points under the headline definition and 20.8 points when partially met criteria receive half credit. It remains positive under both scoring rules.

Results by task

The two application tasks are already close to 100% completion: 38 of 40 baseline runs and 39 of 40 skills runs are complete. On these tasks, binary completion offers little room to show a benefit, even when implementation practices differ.

The behavior tasks show a larger separation: 17 of 40 baseline runs and 30 of 40 skills runs are complete. The mission tasks move from 27 of 60 baseline runs to 44 of 60 skills runs. Those are differences of 32.5 and 28.3 percentage points, respectively. A reasonable interpretation is that the package is especially useful where the agent must coordinate several MOOS-IvP components and conventions. This study does not isolate which instructions account for the difference.

The conformance panel shows why a completion-only task ranking would be incomplete. The contact-waypoint app ties at 19/20 complete while mean conformance rises from 74.8% to 89.5%. The three ordinary mission tasks improve by about 32–46 percentage points in conformance. The harness has the largest conformance increase, from 33.8% to 84.9%, alongside a completion decline. Those attempts follow most of the specified practices while still failing the required verdict.

Application versus integrated mission

Tasks 2 and 6 form a useful contrast. Task 2 asks for the contact-waypoint application by itself, and both conditions complete 19 of 20 runs. Task 6 asks for a two-vehicle mission incorporating that application, including vehicle movement and collision avoidance. Completion falls to 11 of 20 in baseline and 16 of 20 with skills.

Task 6 adds contact-report routing, active waypoint updates, vehicle motion, and collision avoidance around the app. Task 2 and Task 6 were separate benchmark tasks with separate fresh runs. Task 6 did not continue from a Task 2 implementation, so it does not measure how much extra work it takes to combine the app with a mission. It does show that near-perfect results on the standalone application do not settle the mission question.

Task 5 adds another perspective. It asks for patrol, release, timed shadowing, return, and hold controls across two vehicles. Sol completes two of five baseline attempts and all five skills attempts. Astra completes all five in both conditions. Combining the models captures an overall gain, while the task-by-model view shows how differently each model arrives at it.

Differences between models and repeated attempts

Across the 44 task–model combinations, skills produce more complete attempts in 21, the same number in 20, and fewer in three. Luna moves from 0/5 to 4/5 on both the standalone intercept behavior and the mission that includes it. Terra moves from 0/5 to 4/5 on the mission with the contact-waypoint app. These gains are visible within individual task–model comparisons as well as in the pooled totals.

In the harness task, Luna completes 1/5 baseline attempts and 0/5 with skills. Terra and Sol complete none in either condition. Astra completes 5/5 baseline attempts and 1/5 with skills. The other declining combination is Luna’s tight-loop enumeration, from 1/5 to 0/5.

The repeated attempts distinguish an occasional success from a result that holds across all five tries. Sol completes every attempt on eight of eleven tasks with skills, compared with four without. Terra moves from two such tasks to four, and Luna from none to two. Astra remains at nine in both conditions. Five successes are still a small sample, but the shift is useful: the aggregate gain includes more consistently completed tasks, as well as tasks that become possible in some attempts.

Log analysis

The two log tasks require analysis rather than new software. Task 10 asks for every tight loop in both vehicles’ logs, with start and end timestamps, and completion rises from 13 to 15 of 20. Task 11 asks why Henry makes excessive tight maneuvers compared with Gilda. Under the accepted grading, 18 baseline and 20 skills answers are complete, leaving little room to distinguish models.

Conformance still separates the tasks: tight-loop enumeration rises from 53.2% to 73.9%, while diagnosis rises from 45.4% to 48.5%. An answer can give a plausible explanation and still fall short on timestamped support, complete log coverage, or enough detail for another reviewer to repeat the analysis.

Skill use and validation

Each task has one or more required skills. For example, Task 6 integrates the pContactWaypoint app into a two-vehicle mission, so it requires both app-building and mission-building guidance. The evaluation and harness tasks additionally prescribe a companion mission skill. All 220 skills runs show access to their required skills, and 216 show access to the full set including prescribed companions. Luna and Terra each miss that full set in two runs. Sol and Astra access it in every run.

An access record means that the agent opened or otherwise accessed a relevant skill file during the run. It does not prove that the agent read the guidance in full or used it effectively. Since all 220 skills runs accessed their required skills, finding the right skill was rarely the obstacle here.

The study also records live validation: an attempt to launch the built app, behavior, or mission in the benchmark simulation and check it in execution, rather than inferring from the source and build that it should satisfy the request. A recorded launch attempt need not succeed.

Skills runs also record more live validation in every model. This is consistent with workflows that ask the agent to exercise the app, behavior, or mission it has built. The counts are observational: they cannot establish how much of the completion improvement came from testing. The counts also include analysis tasks that require no new mission. An app or behavior task can use a temporary mission or test harness to exercise its deliverable.

Time and cost

With skills, median elapsed time is about seven to eleven minutes, an increase of roughly 0.7–2 minutes depending on the model. These figures include complete and incomplete attempts.

Cost is estimated from recorded token usage using API rates as of August 10, 2026 for Luna, Terra, and Sol, and September 4, 2026 for Astra. It is an API-equivalent estimate, not a subscription charge or invoice. Those rates stay fixed so later price changes do not alter the comparison.

Cost moves differently from elapsed time. Luna’s median estimate decreases slightly, from $0.149 to $0.141. Terra’s increases from $0.597 to $0.791. Sol’s is nearly unchanged at $2.352 and $2.367, while Astra’s rises from $1.864 to $2.468. One Astra skills run on Task 9 ended without a token-usage record. Its cost is unknown and is excluded from the cost median, leaving 54 observations for that group. The run remains in completion and elapsed-time analyses.

The benchmark measures participant effort, not the later cost of reviewing, integrating, or maintaining the result. It cannot yet put a dollar value on the conformance improvement or estimate total project savings.

Failures and next steps

Task 9 identifies a specific weakness in the harness workflow. Six of 20 baseline runs and one of 20 skills runs are complete, despite substantially higher conformance with skills. Completion does not require every simulated route to succeed. It requires the harness verdict to match what happened: a collision reported as FAIL is valid, while a collision reported as PASS is not.

In one Astra skills run, the log places the vehicle inside the obstacle while the result says grade=pass, hit=false, and collisions=0. Other harnesses report success after collision monitoring or test-case setup fails. These false passes show a harness reliability problem, not evidence that skills generally worsen vehicle behavior.

One possible next step is to strengthen the harness guidance. It could suggest a known-failure case, compare each verdict with the recorded mission events, and treat missing monitoring or failed setup as a failure rather than a pass. Repeating Task 9 would show whether that guidance helps.

Would different attempts change the results?

The 15.9-point completion advantage and 29.6-point conformance advantage come from five baseline and five skills attempts for every task and model. Another set of attempts would produce slightly different values. This analysis estimates how much each difference could move because of run-to-run variation.

For every task and model, we randomly redraw five baseline attempts and five skills attempts from those already collected. The same attempt can be drawn more than once. We then recalculate the completion and mean conformance advantages and repeat the process 20,000 times. This standard uncertainty check is called bootstrap resampling. Keeping each group separate preserves the benchmark’s original mix of tasks, models, and conditions.

The middle 95% of the recalculated advantages range from +10.9% to +21.4% for completion and +28.3% to +30.9% for conformance. Neither range includes zero. This supports positive skills advantages on both measures for these tasks and models, but it does not predict the differences for new tasks or future models.

Both advantages stay positive when we exclude each task, model, or task category in turn. The five categories are applications, behaviors, missions, evaluation and harnesses, and log analysis. No single part of the benchmark creates either overall advantage.

Submitted files were reviewed under anonymous identifiers. Reviewers did not see which model produced the work or whether skills were available. Functionality and conformance were scored separately, and uncertain cases received further review.

The scope still matters. The plugin’s developer selected the tasks, conformance reflects the skills’ practices, and model-based reviewers can make mistakes. A broader task set and independent replication would test how far these findings generalize.

Conclusion

Skills improved conformance to MOOS-IvP practice across all four models, including tasks that agents already completed reliably. Completion gains were largest in behaviors and missions.

These results support using domain skills to help coding agents produce more complete MOOS-IvP software that follows the practices experienced developers rely on for real-world deployment.

The design follows ideas used in software-agent benchmarks such as SWE-bench, where an agent works from a request inside a real repository, and robotics research such as Code as Policies and GenSim, where generated robot code is exercised in simulation. This study applies those ideas to complete MOOS-IvP development workflows and separately measures functional completion and domain-specific engineering practice.

The private benchmark archive contains a detailed report for each of the eleven tasks, including model-level results, criterion breakdowns, and failure analysis. This article presents the main findings across the study.

Future work could strengthen the harness skill and test multi-model workflows, such as a Luna run followed by an Astra run that reviews and improves the result. This would better represent development beyond a single attempt and measure how much effort it takes to integrate generated work into a larger project and maintain it as the project changes.

Data availability: The chart-data downloads contain the aggregate results and exact task prompts. Participant transcripts, reviewer workspaces, and runtime evidence are private.