In the challenging world of maritime operations, every decision carries significant weight. A miscalculation can lead to wasted resources, environmental harm, or even endanger lives, especially for critical tasks like protecting our oceans. This reality drove the creation of Shippy, a specialized AI agent designed for high-stakes maritime domain awareness.
Our journey to build Shippy wasn’t just about developing a powerful AI model. It was fundamentally about establishing an AI system we could trust implicitly, one that could deliver accurate information, understand its limitations, and consistently perform across diverse tasks. This entire system needed to be rigorously verified against Skylight’s live, continuously updated data, ensuring its reliability in a dynamic environment.
Crafting a Trustworthy AI Agent
We conceptualize an AI agent like Shippy as having three core components: its soul, its skills, and its config. This modular architecture allows for robust development and easy iteration, ensuring Shippy remains adaptable and reliable.
The soul is essentially Shippy’s guiding system prompt, defining its persona and establishing crucial behavioral boundaries. Complementing this, skills are specific instruction sets that detail how Shippy should handle particular types of requests. These skills are meticulously crafted as plain markdown files, ensuring they are comprehensible, versioned, and simple to update.
For instance, the Skylight API query skill outlines a complete workflow for answering questions about specific areas, like “show me fishing activity in Panama’s EEZ last month.” This skill directs Shippy to precisely resolve geographical boundaries via Skylight’s API, then query relevant data, format results with deep links to the map, and correctly attribute any vessel metadata. Complex inquiries can even draw upon multiple skills simultaneously, such as combining data queries with external databases and vessel track interpretation.
Crucially, Shippy’s soul also dictates what the agent will and will not do. It will not make legal determinations about vessel compliance, as that remains a human responsibility. Similarly, it will never speculate beyond the data it has access to, with these boundaries explicitly encoded in its system prompt rather than implicitly learned, making them easily auditable and revisable.
The third component, config, encompasses everything else required for Shippy to operate. This includes specifying the agent harness (currently OpenClaw, an open-source framework) and the underlying large language model (LLM), which is presently Claude Opus 4.6. Runtime settings and secrets like API keys are injected at execution, allowing for flexible changes to the LLM or harness without requiring a full system rebuild.
Building with Predictability in Mind
AI agents, by their nature, are nondeterministic; you can’t always predict an LLM’s exact decision-making process. However, we can ensure the tools Shippy uses are entirely predictable and reliable. To achieve this, Shippy communicates with Skylight not through raw API calls, but via a purpose-built command-line interface (CLI).
Early prototypes, which allowed Shippy to construct API calls directly, often led to subtle yet critical bugs, such as malformed pagination or geometry encoding errors. The Skylight CLI effectively abstracts this complexity, presenting a predictable interface where Shippy issues simple commands like skylight events search with typed filter flags. The CLI then manages authentication, pagination, and structured output, drastically reducing potential errors.
This CLI is also self-documenting, offering extensive --help text and detailed error messages that enable both the agent and human developers to recover from mistakes effectively. Its output is always written to a local JSON file, circumventing issues like pipe buffer limits encountered with large result sets and facilitating programmatic access across multi-step analyses.
Underpinning the CLI is a standardized API, featuring multiple resource types like events, vessels, and regions, all accessible through common search and aggregate operations. The inputs and outputs of these APIs are defined with typed schemas and detailed field descriptions. This deliberate layering – typed API, deterministic CLI, and agent skills referencing CLI commands – ensures each component of Shippy can be tested independently, with each layer narrowing the scope for errors in the subsequent layer.
Ensuring data privacy and isolation for hundreds of government agencies and NGOs across 70 countries was another significant engineering challenge. Each user interacts with Shippy within their own ephemeral, isolated session, meaning their data and conversation history remain entirely private. This level of isolation is paramount for maintaining trust and security in such a sensitive domain.
To achieve this, we developed Mothership, a sophisticated agent hosting platform that provisions a dedicated Kubernetes deployment for every user session. When a conversation begins, the system spins up a unique set of pods containing the agent runtime, its skills, and the Skylight CLI. The user’s Skylight JWT is securely injected at this stage, ensuring all the agent’s API calls are strictly scoped to that user’s specific data.
Rigorous Evaluation for Real-World Impact
Traditional AI benchmarks, typically designed for static questions, fall short when evaluating an agent’s real-world behavior. They don’t capture how an agent selects tools, queries live data, acts on results, or knows when to stop. Therefore, we developed a bespoke evaluation system tailored to how Shippy operates, scoring the entire agent—model, skills, and sandbox—against live data.
Our evaluation framework empowers subject-matter experts to craft detailed scenarios and rubrics, assigning weights to criteria most relevant for each task. For example, a fishing-events query heavily weights data accuracy, followed by boundary resolution and timeframe, with source attribution and response style carrying less emphasis. Experts also provide ground truth by annotating individual responses as correct or incorrect, allowing for objective scoring.
This pipeline is straightforward yet powerful: a natural-language prompt is executed within the sandbox, and an LLM judge evaluates each rubric criterion on a scale of 0 to 1, providing written explanations for its reasoning. The weighted aggregate is then checked against a fixed pass threshold. This process ensures transparent and actionable feedback.
Utilizing Harbor, an open evaluation framework, we developed a plugin that spins up a real Shippy session on the exact version being tested, interacting with the same live data a user would encounter. This suite runs in parallel, producing timestamped results and a report of score changes against previous runs. We rerun this suite whenever skills, the model, or underlying data change, preventing any regressing version of Shippy from reaching end users.
Shippy consistently achieves high scores in data retrieval and guardrail tasks, adeptly refusing military intelligence requests and accurately attributing sources. Recent evaluations highlighted areas for improvement, such as Shippy overstepping into tactical recommendations for patrol planning, geometry-sensitive queries where boundary simplification caused missed events, and one instance where the agent incorrectly invented a non-existent CLI command. These insights directly inform our ongoing skill refinements.
The Future of AI for Environmental Protection
We are currently opening Shippy to early adopters on a rolling basis, inviting them to stress-test the agent and identify areas for improvement in its responses and guardrails. Their real-world usage will be invaluable in further honing its capabilities. Moving forward, we are focused on enhancing Shippy’s multi-turn reasoning, deepening its integration with advanced geospatial analysis tools, and evolving its ability to offer more proactive assistance.
The profound lessons learned from building Shippy are already influencing our approach to AI agents across Ai2’s other environmental platforms, including EarthRanger for wildlife conservation and OlmoEarth, our open suite of Earth observation tools. Mothership, our robust hosting platform, was deliberately designed to be general-purpose, making it suitable for hosting a diverse array of future agents beyond the maritime domain. Shippy is proudly built by the Skylight team at Ai2, whose free maritime domain awareness platform is trusted by over 300 partners across more than 70 countries.
Source: Hugging Face Blog
