ExoHub Data Service: Agent-Facing Project Gateway

1. Overview

The ExoHub Data Service is a standalone HTTP service that gives AI agents and automated pipelines a simple, secure interface for consuming and producing ExoHub projects. It sits between the compute environment and ExoHub, handling the git-level mechanics so that agents never need direct git or git-annex access.

Four operations are supported:

Standalone Deployments

Each Data Service instance is deployed per compute environment and registered under a deployment_id. A central /mcp dispatcher routes agent requests to the correct deployment based on the deployment_id in the request path.

Agent  ──POST /mcp/research──▶  Data Service (research deployment)
Agent  ──POST /mcp/staging──▶   Data Service (staging deployment)

A single Data Service process can host multiple deployments; each deployment is configured independently with its own data root, workspace paths, and git remotes.

Security Model

Each deployment runs with a service account used for git-level operations (cloning project repositories). Data access is per-user, not service-account-bounded: the Data Service exchanges the requesting user’s identity token for that user’s own short-lived cloud credentials, so an agent can only read the object-storage data the requesting user is entitled to — never more. In addition, per-project access checks are always enforced against the requesting user’s JWT.

2. REST API

For integrations that prefer plain HTTP over the MCP protocol, the Data Service exposes a REST API. It shares the MCP tools’ handlers and returns the same asynchronous operation model.

An interactive OpenAPI reference is served by every deployment at /api/docs.

Endpoints

The three POST endpoints return 202 Accepted immediately — they start background work and never block on the transfer.

The former paths under /api/datasets/ still answer, but are no longer listed; use the paths above.

Authentication

All /api/* endpoints require an ExoHub JWT:

Authorization: Bearer <jwt>

GET /api/operations/{id} additionally accepts a capability token in place of a JWT, which is what makes operation_url shareable — see Capability Tokens .

The examples below assume two shell variables:

EDS=https://eds.example.com     # your deployment's Data Service base URL
TOKEN=<your ExoHub JWT>

POST /api/projects/download

Starts a download. The project is identified in one of two ways — supply either shape, never both. A body mixing them (or carrying any unknown key) is rejected with 422.

Shape A — by repository:

Shape B — by catalog project:

Response — 202 Accepted:

Download by repository:

curl -X POST "$EDS/api/projects/download" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "repository": "https://github.com/genentech/exohub-data-tutorials/mnist.git",
        "ref": "v0.1.0",
        "scope": "user"
      }'

Download by catalog project (latest version, since version is omitted):

curl -X POST "$EDS/api/projects/download" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "project_id": "mnist", "scope": "user" }'

Both return:

{
  "operation_id": "ds-download-dev-mnist-1ad7be7e5b6d17ac",
  "operation_url": "https://eds.example.com/api/operations/ds-download-dev-mnist-1ad7be7e5b6d17ac?t=Ux3n_S9pQ2t7bK1v"
}

The catalog is the source of truth. Both shapes are resolved against the ExoHub catalog before anything is dispatched, and a project that is not catalogued cannot be downloaded — you get 404 even when the git URL is valid and reachable. Resolution also means latest is pinned to a concrete version at request time, and that a repository request and a project request for the same project converge on the same operation.

POST /api/projects/upload

Uploads a folder from the compute filesystem in place: the source directory itself becomes the project. It must already be inside the requesting user’s own workspace. Uploads are user-scope only — there is no scope field. Files are left as ordinary writable files (not locked into read-only symlinks), so the same directory can be changed and uploaded again to publish an updated version.

Returns the same 202 shape as download: operation_id and operation_url.

By default the published project is not private: the repository is created with internal visibility and the project is readable by any authenticated ExoHub user. Narrow it with read_access and viewers, or widen it to anyone with read_access: "public".

curl -X POST "$EDS/api/projects/upload" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "source_folder": "/data/users/alice/workspace/mnist-analysis-alice-20260601",
        "project_id": "mnist-analysis-alice-20260601",
        "read_access": "viewers",
        "viewers": ["bob"],
        "commit_message": "add MNIST classification results"
      }'

POST /api/projects/prune

Deletes a project’s local copy — the whole project, or one downloaded version of it — from your workspace or the shared path. There is no scope field: it follows from where path is.

Returns the same 202 shape as download: operation_id and operation_url.

What the check looks for without force. Before deleting, the prune looks for work no remote holds — in each version folder of a download (or only the one named by version), or in a repository such as an upload’s: uncommitted or untracked changes, commits that no remote-tracking branch or tag contains, and annexed content no configured remote of the repository has (a copy in someone’s private clone does not count). In a version folder the commits looked at are those reachable from its checked-out commit; in a repository, those reachable from its checked-out commit or any local branch other than git-annex and synced/*. If it finds any, the operation fails and names each kind of change with a count and a few example paths:

…/workspace/mnist/v0.1.0 holds 3 unpublished change(s): 2 modified (e.g. v0.1.0/train.csv,
v0.1.0/labels.csv); 1 untracked (e.g. v0.1.0/notes.md). Prune again with force to delete them.

Not covered by the check — these are deleted without warning, even without force: files ignored by .gitignore, stashes, commits reachable only from a tag you never pushed, and commits your local remote-tracking branches wrongly show as pushed (the check does not ask the server). When a whole download is pruned, also: anything in the project folder outside its version folders and .git-store/ (such as files you added next to them), uncommitted or untracked changes in the working tree of .git-store/ itself, and commits reachable only from local branches that no version folder has checked out. In a repository, commits reachable only from git-annex or synced/* branches are not checked either. Move anything you need out of the project first.

Without force, it then drops annexed content only once git-annex has verified it on a remote; anything it cannot verify is kept and the directory is left in place. With force nothing is verified: a whole-project prune deletes the directory as is, and a version prune drops its content unchecked. If your credentials expire part-way, send the same request again — it continues where it stopped.

• Shared path: pruning a global copy requires being an owner of the project in the catalog (by username or group). It is checked once the operation runs, so a non-owner gets 202 and then a failed operation; a project the catalog records no permissions for cannot be pruned there, not even with force.
• One prune at a time: a prune of the same directory and version while one runs returns the running operation — whoever started it, and whatever its force, because force is not part of the operation’s identity. A prune is refused while a download or upload of the project runs (see Project lock ).

curl -X POST "$EDS/api/projects/prune" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"path": "/data/users/alice/workspace/mnist", "version": "v0.1.0"}'

GET /api/projects/describe

Reports, synchronously, what a project’s local copy holds. It reads the copy on the API itself and never waits for a worker, so it answers even while long downloads keep every worker busy. It changes nothing in your files; it only reads the repositories (with git and git-annex) and the project’s lock file.

Each local_versions entry: version, commit, detached (true for a download worktree; false for a branch you can change and upload), catalogued (whether commit is the one the catalog records for that version), and unpublished_changes — grouped by kind, each kind once with its count and at most five examples, so the response stays small however many files changed. These come from the same check a prune runs, with the same blind spots (see Not covered by the check under prune ): an empty list does not mean a prune would lose nothing.

{
  "project_root": "/data/users/alice/workspace/mnist",
  "project_id": "mnist",
  "layout": "worktrees",
  "running_operations": [],
  "local_versions": [
    {"version": "v0.1.0", "commit": "7ec78f52da961f6aa954759900518a5aabde05ae", "detached": true,
     "catalogued": true, "unpublished_changes": []},
    {"version": "v0.2.0", "commit": "20e7e8c534dca71c4e050dce14aaea91bdd013fa", "detached": true,
     "catalogued": true,
     "unpublished_changes": [{"change": "untracked", "count": 2, "examples": ["notes.md", "scratch/run.log"]}]}
  ],
  "catalog_versions": ["v0.1.0", "v0.2.0"]
}

Reading every version of a large project takes time: if reading the local copy exceeds the time limit (60 s by default) describe answers 504. The limit covers only that read; the catalog lookup and the running-operations lookup come after it, so allow a client timeout somewhat above it.

GET /api/operations/{id}

Returns the current state of an operation. Authorize it either with your JWT or with the capability token from operation_url — not both.

# With your JWT
curl -H "Authorization: Bearer $TOKEN" \
  "$EDS/api/operations/ds-download-dev-mnist-1ad7be7e5b6d17ac"

# With the capability token from operation_url (no login required)
curl "$EDS/api/operations/ds-download-dev-mnist-1ad7be7e5b6d17ac?t=Ux3n_S9pQ2t7bK1v"

Response envelope:

Note that a failed operation still returns HTTP 200 — the HTTP status describes the poll, not the work. Always branch on the status field.

The progress object

The *_total fields come from the catalog and are available as soon as the operation starts, so files_synced / files_total can be rendered as a progress ratio from the very first poll.

Each entry in activities describes one in-flight unit of work:

A running download:

{
  "status": "running",
  "progress": {
    "step": "syncing",
    "files_synced": 3,
    "bytes_synced": 12058624,
    "files_total": 8,
    "bytes_total": 33576840,
    "annex_files_total": 6,
    "files_added": 0,
    "ts": "2026-09-11T12:55:48.543299+00:00",
    "activities": [
      {
        "activity_type": "project_exo_get",
        "state": "started",
        "attempt": 1,
        "last_failure": null,
        "stdout_tail": "get train-images-idx3-ubyte.gz (from s3-annex...) ok",
        "stderr_tail": null,
        "last_heartbeat_time": "2026-09-11T12:55:48.550314Z",
        "step": "syncing",
        "files_synced": 3,
        "bytes_synced": 12058624,
        "files_added": 0
      }
    ]
  },
  "result": null,
  "error": null,
  "reason": null,
  "message": null
}

The result object

A completed download:

{
  "status": "completed",
  "progress": null,
  "result": {
    "status": "ok",
    "error": "",
    "scope": "user",
    "repo_dir": "/data/users/alice/workspace/mnist/v0.1.0",
    "repo_url": "",
    "files_synced": 6,
    "bytes_synced": 33576840,
    "files_total": 8,
    "bytes_total": 33576840,
    "annex_files_total": 6,
    "files_added": 0,
    "commit_sha": "",
    "permission": {
      "granted": true,
      "username": "alice",
      "read_access": "public",
      "matched": "public",
      "reason": "read_access=public grants every user."
    },
    "public_access": null
  },
  "error": null,
  "reason": null,
  "message": null
}

files_synced counts annexed files, so it converges on annex_files_total (6) rather than files_total (8) — the difference is plain git files, which arrive with the clone.

Errors

Request-level problems are returned as standard HTTP errors with a {"detail": "..."} body:

Failures that happen during the operation are not HTTP errors — they surface on the poll as status: "failed" with error populated, and reason / message set for recognised classes:

{
  "status": "failed",
  "progress": null,
  "result": null,
  "error": "activity 'project_exo_get' failed [MAXIMUM_ATTEMPTS_REACHED]: Code: 1: get: 12 failed",
  "reason": "credentials_expired",
  "message": "Your credentials expired during the download and could not be refreshed. Please submit a new download request with a fresh token."
}

failed_files is a bounded sample; message states the true total.

Example: end-to-end walkthrough

EDS=https://eds.example.com     # your deployment's Data Service base URL
TOKEN=<your ExoHub JWT>

# 1. Start the download; capture the operation id
OP=$(curl -s -X POST "$EDS/api/projects/download" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "project_id": "mnist", "version": "v0.1.0", "scope": "user" }' \
  | jq -r .operation_id)

# 2. Poll until the operation reaches a terminal state
while true; do
  BODY=$(curl -s -H "Authorization: Bearer $TOKEN" "$EDS/api/operations/$OP")
  STATUS=$(echo "$BODY" | jq -r .status)
  echo "$BODY" | jq -r '"\(.status) \(.progress.bytes_synced // 0)/\(.progress.bytes_total // 0) bytes"'
  case "$STATUS" in completed|failed|canceled|terminated) break ;; esac
  sleep 10
done

# 3. Read where the data landed
REPO_DIR=$(echo "$BODY" | jq -r .result.repo_dir)
ls "$REPO_DIR"

# 4. Upload results back as a new project (source_folder must be inside your own workspace)
curl -s -X POST "$EDS/api/projects/upload" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "source_folder": "/data/users/alice/workspace/mnist-analysis-alice-20260601", "project_id": "mnist-analysis-alice-20260601" }' \
  | jq .

3. MCP Interface

The Data Service exposes a true Model Context Protocol (MCP) server over HTTP. Agents connect using any MCP-compatible client.

Endpoint

POST /mcp/{deployment_id}

Examples:

• POST /mcp/research
• POST /mcp/staging

Authentication

Pass a valid ExoHub JWT in the Authorization header:

Authorization: Bearer <jwt>

An optional header can override the deployment resolved from the URL path:

X-ExoHub-Deployment: research

Tools

The former tool names request_dataset, upload_dataset, prune_dataset and describe_dataset are still accepted, but are no longer listed; use the names below.

download_project — Download a project

Downloads an ExoHub project into the compute environment for the requesting user.

Returns: an operation_id and an operation_url — poll with get_operation (or long-poll with wait_for_operation) for progress.

¹ Give repository (with optional ref) or project_id (with optional version), never both — the same two shapes as the REST endpoint .

upload_project — Upload a folder as a new project version

Uploads a local folder in the compute environment in place as a new ExoHub project version: the source directory itself becomes the project. It must be inside the requester’s own workspace — there is no scope parameter, and uploads never target the global path. By default the published project is not private: the repository is created with internal visibility and is readable by any authenticated ExoHub user. Narrow it with read_access and viewers, or widen it to anyone with read_access: "public".

Returns: an operation_id and an operation_url — poll with get_operation (or long-poll with wait_for_operation) for progress.

prune_project — Delete a project’s local copy

Same parameters and behaviour as POST /api/projects/prune : path (required), version, force.

Returns: an operation_id and an operation_url — poll with get_operation (or long-poll with wait_for_operation).

describe_project — Describe a project’s local copy

Same parameter and response as GET /api/projects/describe : path (required). Answers at once — no operation to poll.

get_operation — Poll async operation status

Returns: the same envelope as GET /api/operations/{id} — status, progress, result, and failure fields.

wait_for_operation — Long-poll an operation to completion

Long-polls an existing operation (from download_project, upload_project or prune_project) until it reaches a terminal state or timeout_seconds elapses, instead of repeatedly calling get_operation yourself.

Returns: the same terminal payload as get_operation if the operation finishes in time (status: completed/failed/canceled/terminated plus result or error). Otherwise returns a status: "running" payload carrying next_action — a ready-to-call {tool: "wait_for_operation", arguments: {operation_id, timeout_seconds}} — so you can call it again to keep waiting; the workflow keeps running server-side even if the connection drops.

Example: Full Agent Workflow

// 1. Download a project into the user workspace
{
  "tool": "download_project",
  "arguments": {
    "dataset": "datasets/reference-genome-hg38",
    "ref": "v2.1.0",
    "scope": "user"
  }
}
// → { "operation_id": "op_01HXYZ..." }

// 2. Poll until complete
{
  "tool": "get_operation",
  "arguments": { "operation_id": "op_01HXYZ..." }
}
// → { "status": "completed", "path": "/data/users/<you>/datasets/reference-genome-hg38" }

// 3. ... agent processes data, writes results into its own workspace at
//    /data/users/<you>/workspace/hg38-analysis-results ...

// 4. Upload the results as a new project (source_folder must be inside your own workspace)
{
  "tool": "upload_project",
  "arguments": {
    "source_folder": "/data/users/<you>/workspace/hg38-analysis-results",
    "project_id": "hg38-analysis-results"
  }
}
// → { "operation_id": "op_01HABC..." }

// 5. Poll until uploaded
{
  "tool": "get_operation",
  "arguments": { "operation_id": "op_01HABC..." }
}
// → { "status": "completed", "dataset": "datasets/hg38-analysis-results@v0.1.0" }

4. Scopes

The scope parameter controls where a downloaded project lands. It defaults to "user".

user scope

• Destination: the requesting user’s personal workspace directory
• Access: any project the user can read in ExoHub
• Use case: exploratory analysis, personal pipelines

global scope

• Destination: a shared path accessible to all processes in the deployment
• Access: public projects only — read_access == "public" in ExoHub
• Restrictions: externally-stored projects (IAM-only access) are never eligible for global scope, even if they are marked public
• Use case: shared reference data, pre-staged inputs for batch jobs

Fail-closed: if the project does not meet the global eligibility criteria, the operation is rejected with an access error. No partial downloads occur.

On-disk layout

Within the scope’s destination root, each project gets a container directory, and each version gets its own git worktree backed by a single shared object store:

{destination}/{project_id}/
├── .git-store/        shared git + annex objects for every version
├── v0.1.0/            worktree — what you read
└── v0.2.0/            worktree — a second version, sharing the store above

result.repo_dir points at the per-version worktree, which is a fully-valid ExoHub clone you can read directly. Because versions share one object store, downloading a second version of the same project only transfers what is genuinely new. Version names are URL-encoded, so a ref like release/2.9.0 becomes the directory release%2F2.9.0.

5. Async Operations

Downloads, uploads and prunes are long-running operations backed by Temporal workflows inside ExoWorkers; they may take seconds to minutes depending on project size. describe is the exception: it answers synchronously and starts no operation.

Operation Lifecycle

POST /api/projects/download   (or /upload, /prune)
        │
        ▼
  202 Accepted — operation_id + operation_url returned immediately
        │
        ▼
  Temporal workflow executes in background
        │
        ├── status: "running"    (actively transferring / committing)
        ├── status: "completed"  (done; `result` populated)
        ├── status: "failed"     (`error` populated)
        ├── status: "canceled"   (cancelled before finishing)
        └── status: "terminated" (forcibly stopped by an operator)

Operation identity

Operation IDs are deterministic, derived from what the operation acts on and for whom:

Re-issuing a request that is already in flight does not start a second workflow — it returns the existing operation_id. This makes retries safe, lets several agents converge on one shared global download, and means a repository-shaped request and a project-shaped request for the same project return the same operation. Different versions of the same project are distinct operations; whether they may run at the same time is decided by the project lock below.

Project lock

Every download, upload and prune holds its project — the directory directly inside your workspace or the shared path — for as long as it runs:

An operation that would conflict is still accepted (202) but fails at once with reason: "project_busy", and message names the operation in its way — wait for it, then send the request again. describe shows who holds a project in running_operations. A holder that ended without releasing (a crash, an operator terminating it) never blocks the project: the next operation finds it is no longer running and takes over.

Polling

Poll get_operation (MCP) or GET /api/operations/{id} (REST) until status is completed, failed, canceled, or terminated. There is no push or webhook mechanism — polling is the intended pattern.

A few seconds between polls is plenty; every poll is a Temporal query, and progress counters only advance as files land. Back off to 15–30 seconds for large projects, and treat the activities entries as the place to look when an operation seems stuck — a climbing attempt with a last_failure means it is retrying, and stderr_tail usually says why.

6. Auth & Access Control

JWT Validation

The Data Service validates the Authorization: Bearer <jwt> token independently on every request. Tokens are issued by ExoHub’s OIDC OAuth server. The service never caches user identity across requests.

Capability Tokens

Every 202 response includes an operation_url — the operation’s status URL with an unguessable capability token appended as ?t=<token>:

https://eds.example.com/api/operations/ds-download-dev-mnist-1ad7be7e5b6d17ac?t=Ux3n_S9pQ2t7bK1v

Anyone holding that URL can poll the operation without logging in, which makes it safe to paste into a browser, a chat message, or a CI log line. The operation ID itself is deterministic and therefore guessable, so it is not treated as a secret — the token is. A request carrying a wrong token returns 404 rather than 403, and never falls back to identity auth, so a bad link cannot be used to probe which operations exist.

Credential Vending

For data held in object storage, the Data Service does not hold standing data-access credentials. Instead it exchanges the requesting user’s identity token via OIDC (the STS exchange) for that user’s own short-lived cloud credentials, and uses those to read the data directly. Access is therefore bounded by the user’s own identity and entitlements — the service cannot read anything the user could not read themselves. These per-user credentials are cached briefly server-side (until shortly before they expire), so a multi-file transfer reuses a single token exchange rather than re-exchanging per object.

Per-Project Access Checks

Before starting any download, the Data Service checks that the requesting user (identified by the JWT) has read access to the requested project. This check is performed against ExoHub’s access control system, not the service account’s permissions. The outcome is recorded in the operation’s result.permission so you can see why access was granted.

Deployment-Level Authorization

Each deployment restricts which user groups may use it via an allowed_groups configuration key. Requests from users outside the allowed groups are rejected with 403 before any project check occurs. The check is fail-closed: a deployment with no allowed groups configured accepts nobody, and the wildcard "*" is required to allow all users.

7. Provenance

The Data Service records precise provenance for every upload operation, ensuring projects are always traceable back to their origin. This is git-level provenance (who committed what, and how data was fetched) — a different, lower-level concept from the exo provenance W3C PROV records covered in the user guide , which track the lineage of individual files and folders within a repository.

Download Provenance

Downloads do not use exo download. Instead, the service performs a full git workflow:

1. git clone — clones the ExoHub project repository
2. exo init — initializes the local ExoHub workspace
3. exo get — materializes the requested file content from storage backends

exo get is a uni-directional pull (it only fetches annexed content into the local repo) — deliberately not exo sync, which is bidirectional and would push local state back, violating the read-only contract of a download. This ensures the local workspace is a fully-valid ExoHub clone, not a bare file copy.

Upload Provenance

Every upload_project operation commits as the requesting user, not a service account:

There is no Co-authored-by trailer and no separate “Data Service account as committer” step: the upload sets both GIT_AUTHOR_* and GIT_COMMITTER_* to the uploading user’s identity, so the requesting user is recorded as both the author and the committer of the resulting commit.

8. Configuration

Each deployment is configured independently. Below are the key configuration parameters an operator sets per deployment.

A deployment’s effective configuration can be read at runtime from its discovery endpoint:

curl -s "$EDS/" | jq .

One setting is an environment variable of the API process rather than a deployment key, and is not shown by the discovery endpoint:

Integration with ExoHub Components

The Data Service is deployed alongside ExoWorkers — Temporal workflows execute the actual git and transfer operations. The service itself is a thin HTTP frontend that dispatches work to ExoWorkers and forwards status back to the caller. The one exception is describe, which reads the project on the API itself; for that the API runs as the same user as the workers.