ExoHub Logo
ExoHub

ExoCLI: Opinionated Git-Annex Interface

Table of Contents

ExoCLI Logo

1. Defining ExoCLI and User Abstraction

ExoCLI serves as the primary command-line interface within the ExoHub ecosystem, designed to abstract the inherent complexity of git-annex and provide an accessible entry point for developers, data scientists, and automated systems.

The source code is available at https://github.com/genentech/exo-cli .

Instead of requiring users to master the extensive git-annex command set, ExoCLI provides intuitive, high-level commands that handle sophisticated data management workflows.

The interface introduces declarative YAML manifests, enabling complex, multi-step data operations to be defined, version-controlled, and executed repeatably. This approach transforms data management from imperative command sequences into declarative configurations, making workflows reproducible and shareable across teams.

ExoCLI bridges the gap between git-annex’s powerful capabilities and user productivity, ensuring that the benefits of distributed data versioning remain accessible without requiring deep technical expertise in the underlying tools.

ExoCLI Interface

2. Role in the ExoHub Architecture

ExoCLI functions as the user-facing layer of the ExoHub data management stack, translating high-level user intentions into precise git-annex operations while coordinating with other ExoHub components for comprehensive data lifecycle management.

Command Abstraction and Simplification

ExoCLI provides a comprehensive command set that maps complex git-annex operations to intuitive actions:

Core Data Management:

Metadata and Communication:

Repository Management:

Maintenance and Diagnostics:

Each command handles error management, progress reporting, and logging automatically, reducing the cognitive overhead for users.

YAML Manifest-Driven Operations

ExoCLI introduces a declarative approach to data operations through YAML manifests:

# pull-manifest.yaml
repository:
  url: "git@github.com:genentech/datasets/genomics-pipeline.git"
  version: "v2.1.0"
paths:
  - "raw-data/samples/*.fastq"
  - "processed/alignments"
remotes:
  primary: "s3-production"
  cache: "local-hpc"

This manifest-driven approach ensures that data operations are:

YAML Manifests

3. Integration with ExoHub Components

ExoCLI serves as the orchestration interface that coordinates interactions between ExoHub’s distributed components.

ExoGit Integration

ExoCLI seamlessly interfaces with ExoGit repositories, handling:

ExoWorkers Coordination

For large-scale operations, ExoCLI can delegate long-running tasks to ExoWorkers:

4. User Experience and Workflow Patterns

ExoCLI prioritizes developer experience through consistent interfaces and predictable behavior patterns.

Interactive and Scripted Workflows

Developer-Driven Data Management: For interactive exploration and modification of datasets, ExoCLI provides immediate feedback and intuitive command structures. A typical workflow involves:

  1. Repository Setup: Clone the dataset repository using standard git clone and run exo init to initialize git-annex
  2. Content Synchronization: exo sync downloads actual data files from configured remotes
  3. Local Processing: Work with data using existing tools and workflows
  4. Content Addition: exo add processed-results/ adds new files to git-annex
  5. Version Creation: git commit -m "feat: add processed analysis results" commits the git-annex changes
  6. Metadata Distribution: exo broadcast shares metadata updates across remotes

Automated Integration: ExoCLI commands integrate seamlessly into CI/CD pipelines, data processing workflows, and scheduled automation:

# CI/CD pipeline example
git clone $DATASET_REPO dataset
cd dataset
git checkout $VERSION_TAG
exo manifest .exo/production-manifest.yaml
exo sync
python process_data.py
exo add processed-results/
git commit -m "feat: add automated processing results"
exo export processed-results/ --to s3-production
exo submit .exo/production-manifest.yaml

Error Handling and Diagnostics

ExoCLI provides comprehensive error reporting and diagnostic capabilities:

ExoCLI Workflow

5. Configuration and Extensibility

ExoCLI supports flexible configuration and extensibility to adapt to diverse organizational needs.

Context-Based Configuration

ExoCLI uses a simple context system to manage git host and organization settings:

Extensibility

ExoCLI integrates with the broader ExoHub ecosystem through:

Environment Integration

ExoCLI is designed as a command-line tool with specific platform requirements:

6. Getting Started with ExoCLI

Installation and Setup

ExoCLI can be installed with a single command that automatically handles all dependencies:

# Install ExoCLI and all dependencies
curl -fsSL https://github.com/Genentech/exohub/releases/latest/download/install.sh | bash

This installer automatically downloads and configures:

# Verify installation
exo version

# Check available commands
exo help

# View current context status
exo context show

First Steps

Basic ExoCLI workflow for new users:

# Create and select a git host context
exo context create production --host https://github.com --org genentech
exo context select production

# Clone a dataset repository using standard git
git clone git@github.com:genentech/datasets/example-data.git
cd example-data

# Initialize git-annex in the repository
exo init

# Synchronize content from remotes
exo sync

# View repository information
exo info

# Check system health
exo heartbeat

Learning Resources

ExoCLI transforms complex distributed data management into an accessible, productive experience while maintaining the full power and flexibility of the underlying ExoHub architecture.