Architecture Overview & Unified Data Flow

Theory of Operation

1. Fingerprint content. The WRL Chunker fragments every parsed document and live payload and fingerprints each fragment. See Characterization and the WRL Chunker.
2. Link the copies. Fingerprints are indexed in the Roaring Bitmap Index (.rbix). Fragments with the same fingerprint form one equivalence class across sources.
3. Collect context from every copy. Identity and metadata connectors supply owners, ACLs, groups, and timestamps. NER, driven by the configured labels, enabled ontologies, and enabled compliance frameworks, attaches entities. The graph holds (:Entity {id,text,label}) nodes linked by TAINTS to the Edges they occur in.
4. Resolve versions (beta). Two Stage IDF Stale Source Clustering builds version DAGs and assigns freshness_rank. See Freshness and Stale Clustering.
5. Decide at inference time. Compiled Rego runs in process with the fragment context, the Human Principal (obo.user), and the declared purpose. Metapolicies resolve multisource conflicts. See Metapolicies and Governance.
6. Record the decision. Each adjudication emits an Audit V2 receipt with the affected regions, determining policy, caller identity, and purpose.

CONSOLE FUNCTIONAL PROOF/admin/contextGraphNeo4j Context Graph & Ontological Provenance Explorer

Assertion Verified: Proves that identical sentence bytes in two different documents carry distinct meaning and governance verdicts depending on their ontological neighborhood (:Region NER rollup, LNKD-{xor}-{count} equivalence class, and CORI provenance).

Content is Finite; Context is InfiniteMorgan Company Ambiguity DisambiguatedLNKD-{xor}-{count} Equivalence ClassesDecision E7 :Region NER Rollup

Caber Console Context Graph Explorer (/admin/contextGraph) proving how ontology and provenance disambiguate identical sentence content across Morgan's company divisions.

4. End to End Data Flow Walkthrough

Plane 2: Synchronous SDK & Gateway Interception Flow

1. Zero Code Instrumentation (pycaber.auto): Calling import pycaber.auto activates the pycaber automatic instrumentation hooks (native engine pycaber._native, bundled in the pycaber wheel) across 15 HTTP/WebSocket/gRPC frameworks (FastAPI, Flask, Django, Starlette, AIOHTTP, Tornado, Quart, Falcon, Bottle, Pyramid, Sanic, Litestar, Grpcio, Websockets, Requests/HTTPX) and 5 AI/agent frameworks (LangChain/LangGraph, LlamaIndex, CrewAI, AutoGen, OpenAI/Anthropic/MCP clients).
2. Streaming WRL Characterization: Request and response payloads stream through streaming_core::WrlProcessor at 76.6 MiB/s per core, extracting 64 bit ChunkId fingerprints and exact [start_byte, end_byte) coordinates.
3. OBO Identity & Protocol Detection: caber_identity and caber_auth extract On Behalf Of (obo.user) JWT/HMAC delegation chains across A2A, MCP (tools/call, resources/read), and LLM API boundaries, binding every tool invocation and completion to the initiating human principal.

5. Architectural Pillar & Package Mapping

Architectural Pillar	Rust Crates (PyCaber/crates/)	Active Python Packages (PyCaber/python/)	Storage Backends & Queues	Runtime Binaries & Listeners
Transport, Discovery & Coordination	caber_transport, coordinator, caber_hub, ingest_stage, caber_mcp	pycaber_connector, pycaber_types	SQLite (Fold), PostgreSQL + AWS SQS (Split), shared-storage	caber-hub (/v1/dependencies), caber-core (QueueIngestor on csi-s3event-q)
Fingerprinting, WRL & RBIX Indexing	streaming_core, roaring_index, rocksdb_storage, base_utils	pycaber (PyO3 bindings)	RocksDB (roaring_index.db 13 CFs), .rbixb + .json sidecars	caber-core (in memory + RocksDB reader), caber-cli
Document Parsing & Zero Shot NER	ingest_core, document_parser_client, content_typer, sys_config	pycaber_parser (markdown_runner.py, gliner2_runner.py, docling_runner.py)	shared-storage (ParserCache/, redirect-{hash}/)	scripts/parser_service.py (:8765 GLiNER2, :8766 Docling), document_parser_container
Lineage, Context Graph & Governance	graph_builder, caber_governance, caber_auth, caber_identity, caber-core	pycaber (native engine pycaber._native; pycaber_graph, pycaber_graph_lib, pycaber_ingest decommissioned into caber-core)	Neo4j (bolt://neo4j:7687), OpenSearch (csi_doc_graph, csi_audit_trail)	caber-core (HTTP :8080, gRPC :50051, Redis Pub/Sub, QueueIngestor)