ExoHub Logo
ExoHub

Consuming a Dataset

Table of Contents

This tutorial walks you through consuming an existing dataset as an end-user. Rather than building a dataset from scratch (covered in the MNIST tutorial ), you will clone the published MNIST dataset, download only what you need, verify file provenance, and load the data for analysis.

Prerequisites

Run the installer (Linux/macOS):

curl -fsSL https://github.com/Genentech/exohub/releases/latest/download/install.sh | bash

Add ~/.local/bin to your PATH if needed:

export PATH="$HOME/.local/bin:$PATH"

Verify the installation:

exo version

Authenticate with ExoHub:

exo login

1. Clone the Dataset

The MNIST dataset is published at:

Clone it with exo clone:

exo clone https://github.com/genentech/exohub-data-tutorials/mnist.git

When a repository defines presets, exo clone automatically detects them and offers a selection menu:

Available presets:
  1. full-clone   - Complete repository with everything
  2. data-only    - Just the processed data for quick experimentation

Select a preset [1-2]:

Using a Specific Preset

If you already know which preset you want, pass it directly with --preset to skip the menu:

exo clone --preset data-only https://github.com/genentech/exohub-data-tutorials/mnist.git

The data-only preset uses sparse checkout to fetch only the data/ directory — ideal when you do not need the raw files or preprocessing scripts.

cd mnist
ls
# data/  README.md

Tip: Use --dry-run to see what a clone would do before committing:

exo clone --preset data-only --dry-run https://github.com/genentech/exohub-data-tutorials/mnist.git

2. Initialize the Repository

After cloning, initialize the remotes so ExoHub knows where to fetch the data from:

cd mnist
exo init

exo init reads the remote configuration from .exohub/remotes and sets up git-annex with the appropriate S3 backends. See Working with Repositories for details on what initialization does.

Confirm everything is in order:

exo info

Expected output (abbreviated):

Data Repo: https://github.com/genentech/exohub-data-tutorials/mnist.git

Clone Preset
  Name: data-only

Configured remotes
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ name        ┃ uuid                                 ┃
┣━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃   s3-annex  ┃ 0d80d418-7b86-421a-9690-e01b0910b317 ┃
┃ ♥ s3-export ┃ bba7ca80-6235-4d2c-9bb0-18cc25d25c62 ┃
┗━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

3. Download the Data

Option A — Sync Everything

To download all content in the current sparse checkout, run:

exo sync

This fetches every file pointer available in your checkout from the configured remotes. For a data-only clone, that means the processed Parquet files:

ls data/processed/
# test.parquet   train.parquet

Option B — Cherry-pick Specific Files with exo get

When you only need a few files, exo get is more efficient than a full sync — it downloads exactly what you specify without touching anything else:

# Get a single file
exo get data/processed/train.parquet

# Get an entire directory
exo get data/processed/

# Get from a specific remote
exo get --from s3-annex data/processed/train.parquet

Use exo locate to see what is already present locally and what still needs to be downloaded:

exo locate data/

annex      data/processed/train.parquet    *here, 📦 s3-annex
annex      data/processed/test.parquet     📦 s3-annex

Files marked *here are already on disk; others can be fetched with exo get.

See Syncing Data for the full exo get reference.

4. Verify Provenance

ExoHub tracks which remotes hold each file. Use exo locate to inspect where a file lives:

exo locate data/processed/train.parquet

annex      data/processed/train.parquet    *here, 📦 s3-annex

The output tells you the file is present locally (*here) and also stored on the s3-annex remote — meaning it is safely backed up and can be re-downloaded at any time.

To check all files at once and spot anything that hasn’t been downloaded yet:

exo locate data/

annex      data/processed/train.parquet    *here, 📦 s3-annex
annex      data/processed/test.parquet     *here, 📦 s3-annex

Files without *here are not yet on disk — fetch them with exo get (see the previous section).

For a machine-readable view (useful in scripts):

exo locate --json data/

See Working with Repositories for the full exo locate reference.

5. Work with the Data

The processed Parquet files are ready to use with any DataFrame library.

Python (pandas / pyarrow)

import pandas as pd

train = pd.read_parquet("data/processed/train.parquet")
test  = pd.read_parquet("data/processed/test.parquet")

print(train.shape)   # (60000, 785)
print(test.shape)    # (10000, 785)

# First column is the label, remaining 784 are pixel values (28x28)
X_train = train.iloc[:, 1:].values / 255.0
y_train = train.iloc[:, 0].values

Quick sanity check

import numpy as np

print("Classes:", np.unique(y_train))
# Classes: [0 1 2 3 4 5 6 7 8 9]

print("Pixel range:", X_train.min(), "-", X_train.max())
# Pixel range: 0.0 - 1.0

What’s Next