The Brain Behind
Every AI Factory

TonoFabric™ is the enterprise orchestration platform that transforms distributed TonoForge™ hardware into a unified, intelligent AI cloud. Ten microservices, six global regions, one seamless API.

10Microservices 6Global Regions 200Cluster Nodes gRPC+HTTPDual Transport OTelFull Observability Platform Architecture

Four Layers, One API

TonoFabric™ is structured as a layered microservices platform. Every request traverses the same deterministic path - from authentication through intelligent routing to distributed inference - with full telemetry at every hop.

Request Lifecycle

From Prompt to Response in 7 Hops

Every API call follows a deterministic path through the platform. Authentication, quota enforcement, model selection, cluster allocation, and inference happen in a single round-trip.

01AuthenticateJWT token validated via Auth Service 02Rate CheckToken-bucket limiter per user at Gateway 03Quota GateUsage Service enforces daily quota & plan 04Open SessionSession Service tracks conversation state 05RouteModel recommended → cluster allocated 06InfergRPC/HTTP proxy to GPU cluster 07RecordUsage metered, session updated, response sent POST /v1/query // Single-call API - TonoFabric™ handles everything
{
  "prompt": "Analyse this turbine sensor data for anomalies",
  "purpose": "general",
  "region_pref": "eu-west",
  "session_id": null // auto-created if omitted
}

// Response
{
  "session_id": "a3f8…",
  "model_id": "m-70b-general",
  "answer": "Based on the sensor readings, I detect 3 anomalies…",
  "latency_ms": 142
} Microservices

10 Services, Zero Single Points of Failure

API Gateway:8000 · FastAPI Unified entry point for all client requests. Validates JWT tokens, enforces per-user token-bucket rate limiting, checks quotas, orchestrates the full request lifecycle, and records usage after each call. Supports both gRPC and HTTP inference backends. JWT AuthRate LimitinggRPC ClientOTel Tracing Auth Service:8001 · SQLAlchemy + Alembic Handles user signup, login, and JWT issuance. Password hashing with SHA-256 salting. Token validation endpoint consumed by the Gateway on every request. Database-backed user store with Alembic migrations for schema evolution. JWT HS256SQLite/PostgresAlembic Migrations Model Registry:8002 · In-Memory Catalogue Central catalogue of all available AI models - from 10B general-purpose to 1T research-class. Intelligent recommendation engine matches prompt length, purpose (general/coder/research), latency SLA, and hardware affinity (NVIDIA/AMD/any). 10B-1T ModelsAuto-RecommendHW AffinityContext Sizing Cluster Manager:8003 · 200 Clusters Manages a fleet of 200 GPU clusters across 6 regions and dual data centres (dc-a / dc-b). Three-tier allocation with failover: exact region+DC match → same region fallback → global fallback. Health-aware - only routes to healthy racks. 6 RegionsDual DCMulti-Tier FailoverNVIDIA/AMD Router Service:8004 · Smart Orchestration The brain of TonoFabric™. Receives a prompt + purpose, calls Model Registry for the optimal model, then Cluster Manager for the best-fit GPU cluster. Returns a complete routing decision: model_id, cluster_id, rack_id, and endpoint URL. Model SelectionCluster AllocationRegion Preference Session Service:8005 · SQLAlchemy + APScheduler Manages conversational state with full event sourcing. Opens sessions, appends user/assistant messages with timestamps, enforces TTL (default 4 hours), and ships expired or closed sessions to central Storage for long-term retention. Event SourcingTTL ExpiryAuto-ArchivalAPScheduler Storage Service:8006 · File System / S3 Persistent JSON store for completed sessions. Receives session + events payload and writes to mounted volume or cloud object store. Each session is stored as an individual JSON file, enabling simple backup and compliance export. JSON PersistenceVolume MountSession Archive Inference Proxy:8007 HTTP · :9000 gRPC Dual-transport inference endpoint. Exposes both a REST /infer endpoint and a gRPC InferenceServicer (compiled from proto/inference.proto). The Gateway auto-detects the transport scheme (grpc:// vs http://) and routes accordingly. gRPC + HTTPProto CompileDual TransportGPU Backend Telemetry Service:8008 · Prometheus-compatible Collects metrics from all services and exposes them in Prometheus exposition format. Ingests structured telemetry items (name, value, labels, timestamp) and feeds the Prometheus → Grafana observability pipeline. PrometheusOTel CollectorGrafanaMetrics API Usage Service:8009 · Billing Engine Tracks per-user daily quotas, per-model pricing (price_per_request, price_per_1k_tokens), and billing summaries. Enforces quota gates before inference and increments counters after each request. Supports tiered plan-based rate limits. Daily QuotaPer-Model PricingBilling SummaryPlan Tiers Global Infrastructure

Six Regions, Dual Data Centres

The Cluster Manager seeds 200 GPU clusters across six regions, each with dual data centres for redundancy. The three-tier allocation algorithm guarantees a healthy endpoint for every request - with graceful failover across DCs and regions.

EU-WESTEurope West~34 clusters · dc-a / dc-b EU-CENTRALEurope Central~33 clusters · dc-a / dc-b US-EASTNorth America East~33 clusters · dc-a / dc-b US-WESTNorth America West~33 clusters · dc-a / dc-b AP-SOUTHAsia Pacific South~33 clusters · dc-a / dc-b AP-NORTHEASTAsia Pacific Northeast~34 clusters · dc-a / dc-b

Three-Tier Failover

TIER 1Exact MatchRequested region + preferred DC + vendor match TIER 2Region FallbackSame region, alternate DC, vendor match TIER 3Global FallbackAny healthy cluster in any region Model Catalogue

Six Model Classes, One API

Model ID	Parameters	Family	Hardware	Max Context	Throughput Hint
m-10b-general	10 B	General	Any (NVIDIA / AMD)	8,192 tokens	2,000 rps
m-34b-general	34 B	General	Any (NVIDIA / AMD)	16,384 tokens	1,200 rps
m-70b-general	70 B	General	NVIDIA preferred	32,768 tokens	600 rps
m-120b-coder	120 B	Coder	NVIDIA preferred	32,768 tokens	400 rps
m-250b-general	250 B	General	NVIDIA preferred	65,536 tokens	200 rps
m-1000b-research	1 T	Research	NVIDIA required	131,072 tokens	50 rps

The recommendation engine matches prompt token count, purpose, and latency SLA to the optimal model. Hardware affinity ensures large models run on NVIDIA clusters with sufficient HBM3e capacity.

Observability

Full-Stack Visibility

Every service is instrumented with OpenTelemetry. Traces flow through the OTel Collector to Prometheus for metrics and Grafana for visualisation. Every hop is observable - from API Gateway to GPU rack.

◈OpenTelemetry SDKDistributed tracing across all 10 services with BatchSpanProcessor and OTLP HTTP export ◇OTel CollectorCentral telemetry hub receiving traces on :4317 (gRPC) and :4318 (HTTP) ◆PrometheusTime-series metrics store scraping /metrics from Telemetry Service on :9090 ◈GrafanaReal-time dashboards on :3000 with pre-provisioned Prometheus datasource .env # Enable distributed tracing
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

# Grafana access
GRAFANA_ADMIN_USER=admin
GRAFANA_ADMIN_PASS=[secure-password] Security

Defence in Depth

JWT Authentication Every API call requires a Bearer JWT token (HS256). The Auth Service signs tokens on login with a configurable secret and 1-hour expiry. The Gateway validates every token before processing. Internal Token Guard Service-to-service calls are protected by HMAC-compared internal tokens. Every inter-service request carries an x-internal-token header, verified with constant-time comparison to prevent timing attacks. Secret Management Production secrets are managed via Sealed Secrets (kubeseal for in-cluster encryption) or External Secrets Operator for syncing from AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault. Rate Limiting & Quotas Token-bucket rate limiting (configurable RPS) at the Gateway level. Daily quotas enforced by the Usage Service. Plan-based tiers dynamically adjust rate limits per user. Excess requests receive HTTP 429 or 402. Deployment

From Docker Compose to Kubernetes

Development Stack

Single-command local development with Docker Compose. All 10 services, plus the observability stack (OTel Collector, Prometheus, Grafana), spin up with docker compose up.

docker-compose.yml services:
  api_gateway:    # :8000
  auth_service:   # :8001
  model_registry: # :8002
  cluster_manager:# :8003
  router_service: # :8004
  session_service:# :8005
  storage_service:# :8006
  inference_proxy:# :8007 + gRPC :9000
  telemetry_service:# :8008
  usage_service: # :8009
  otel-collector: # :4317/:4318
  prometheus:     # :9090
  grafana:        # :3000

Production Stack

Kubernetes deployment with Kustomize overlays or Helm umbrella chart. Horizontal Pod Autoscalers for central and edge clusters. CI/CD via GitHub Actions - build, push to GHCR, deploy.

Kubernetes Features HPAs - Central + edge autoscaling based on CPU/memory
Sealed Secrets - Encrypt secrets with kubeseal, commit safely
External Secrets - Sync from AWS/GCP/Azure secret stores
Kustomize - Environment overlays (dev, staging, prod)
Helm Chart - Umbrella chart for full platform deployment
GitHub Actions - Build → push GHCR → deploy overlay/Helm Tech Stack

Built for Scale

Layer	Technology	Purpose
API Framework	FastAPI (Python)	Async HTTP services with Pydantic validation
RPC Transport	gRPC + protobuf	High-performance inference calls (:9000)
HTTP Client	httpx (async)	Inter-service communication
Auth	PyJWT + HS256	Token signing & verification
Database	SQLAlchemy (async) + Alembic	User store, session store, schema migrations
Scheduling	APScheduler	Session TTL expiry & archival
Tracing	OpenTelemetry SDK	Distributed tracing across all services
Metrics	Prometheus + Grafana	Time-series metrics & dashboards
Orchestration	Docker Compose / Kubernetes	Dev & production deployment
CI/CD	GitHub Actions	Build, push GHCR, deploy Kustomize/Helm
Secrets	Sealed Secrets / External Secrets	Encrypted in-cluster or cloud-synced secrets