Storage

CEM-002

🤖 Relational Data

CEM-002-000

Specification

If the project uses relational data: - use Postgres - create a Dockerfile and docker compose entry for the service - create src/<project-name>/db directory for database-related code (e.g., migrations, session management, requirements, dockerfile)

• create src/<project-name>/db/session.py for database session management

• use alembic to run the migrations at src/<project-name>/alembic

• use service-model-repository pattern for database interactions:

  • create src/<project-name>/repositories directory for repository classes that handle database operations

  • create src/<project-name>/services directory for service classes that contain business logic and interact with repositories

  • create src/<project-name>/models directory for the ORM models implemented with sqlalchemy

    • each model has its own file with its class definition

  • create src/<project-name>/schemas directory for Pydantic schemas used for data

    • each schema has its own file with its class definition

Motivation

• Modern SQL databases have equivalent performance for JSON data types.

• Postgres offers robust JSON support across relational databases.

🤖 Similar Data (Vector Space)

CEM-002-001

Specification

• If you have to consume a large volume of vector data, use Milvus.

• If you have to consume a small volume of vector data, use Milvus Lite.

Motivation

Vector Database Comparison

Feature	Milvus Lite	Milvus (Full)	pgvector	ChromaDB	Qdrant	Weaviate
Scaling Potential	Small (Local)	Massive (Cluster)	High (SQL)	Limited	High (Native)	Very High (Sharding)
Complexity	Very Low	High	Medium	Very Low	Low to Medium	Medium (Docker/Cloud)
Key Features	Python-first	GPU Indexing	ACID, SQL Logic	Ease of use	Rust-speed, Payloads	GraphQL, Multi-modal
Best For	Rapid Prototypes	Global Enterprise	Existing DBs	Simple PoCs	High-speed filtering	Complex Data Objects
Updates (CRUD)	Local Upserts	Stream-based	Standard SQL	Basic Collection	Efficient Payload	Object-level Hybrid

🤖 Object

CEM-002-002

Specification

• For local deployments, use Minio.

• For remote deployments, if AWS is available, use AWS S3

• otherwise, use Azure Blob Storage.

Motivation

The storage layer for a RAG system is a contextual anchor. You need to store the original source documents such that the source is readily available.

AI Storage Layers

Feature	MinIO	AWS S3 (with S3 Vectors)	Cloudflare R2	Azure Blob
Primary Role	High-speed, On-prem/Hybrid	Massive Scale + Native Search	Zero-Egress, Edge Storage	Enterprise/Microsoft Stack
RAG Integration	S3-API (Works with all DBs)	Native Vector Indexing (2026)	Fast edge-retrieval	Integrated with Azure AI
Performance	Ultra-low latency (NVMe-tuned)	High (Express One Zone)	Medium (Global Edge)	High (Premium Tier)
Cost Model	Free (OSS) or Licensed	Pay-as-you-go + Request fees	Fixed price per GB, $0 Egress	Tiered (Hot/Cool/Cold)
Best For	Private clouds & Speed	Full AWS-ecosystem RAG	Global apps avoiding egress	Corporate/Internal QA

🤖 Related Data (Graph)

CEM-002-003

Specification

• If it is important to find relationships between data points, rather than distance, use GraphDB.

Motivation

Compared to relational (Postgres):

Relational excels at tabular records (Q&A items, sessions, attempts), transactions, and simple joins; we use it for the core data model per CEM-002-000 and it’s sufficient for CRUD and reporting. Graph excels at deep, variable-length relationship queries (e.g., “all concepts reachable within 2–3 hops that unlock ‘Eigenvectors’”), avoiding complex recursive SQL and performing better for path-finding.

Compared to vector DB (Qdrant/Weaviate):

Vector DB finds semantically similar content (“questions like this”). Graph DB captures structured pedagogical relationships and constraints (“what you must learn before this”). They complement each other: retrieve semantically relevant items, then filter/sequence via graph constraints.