Ade
Daramola

GitOps Sentinel is an autonomous remediation platform for Kubernetes clusters. When Alertmanager fires a webhook, Sentinel doesn't just page on-call — it ingests the signal through an HMAC-validated API Gateway, deduplicates via DynamoDB, bundles Prometheus metrics and k8s events into S3, and routes the incident through a Step Functions multi-agent pipeline: Classifier → Root Cause → Action Planner → Confidence Scorer. Every remediation is a Git commit. Argo CD detects the merged PR and syncs the cluster. The cluster only changes through the same reviewed write path a human engineer would use.

The design comes from a real operational insight: not all incidents are equal, and the action shouldn't be either. Sentinel gates every remediation on a deterministic confidence score with three explicit routes — ≥80 auto-applies the PR, 40–79 opens a PR for engineer review, and <40 escalates to on-call with no automated change. Routine incidents (OOMKilled pods, replica drift, known bad image tags) resolve in seconds. Novel or high-risk scenarios escalate before anything touches the cluster. Five minutes after every remediation, an Outcome Validator queries Prometheus to confirm the fix held — and if it didn't, it opens an automatic revert PR. The cluster is never worse off than it was before Sentinel ran.

This is the kind of system that pays for itself the first time it silently fixes something at 4am that would have been a two-hour incident.

The Multi-LLM Platform is a production-grade AI gateway that routes requests across Anthropic Claude, OpenAI, and AWS Bedrock with a cost-aware complexity scoring engine. Every request is scored on token volume, code detection, and reasoning keywords — simple queries go to low-tier models (Nova Micro, Haiku), complex ones to mid or high-tier (Sonnet, GPT-4o, Opus). A client can pin a specific model via model_preference; the router falls back gracefully if that provider is unhealthy. The result is a platform that routes 60%+ of traffic to cheap models automatically, with a cost target of $0.004–$0.008 per average request.

The caching layer is two-tier: Redis exact-match (sub-millisecond) sits in front of a pgvector semantic cache (cosine threshold 0.92). Cache hits bypass the LLM entirely — a semantic hit on a prior equivalent question is served from the cache and the prompt embedding gets promoted to Redis. At 35%+ hit rates (the conservative target), the cache pays for itself against LLM API costs within weeks. Auth is DynamoDB-backed per-key with sliding-window rate limiting. A health-checker Lambda runs every five minutes on EventBridge and marks unhealthy providers, so the router never wastes a request on a down endpoint.

Observability is first-class. Every request emits CloudWatch EMF metrics via Lambda stdout — RequestCount, InputTokens, OutputTokens, LatencyMs, CacheHit, EstimatedCostUSD — dimensioned by provider, model, and tier. X-Ray traces cover auth, cache lookup, routing, provider call, and cache write. Terraform manages all infrastructure: networking, Aurora Serverless v2, ElastiCache Serverless, Lambda, API Gateway v2. OIDC-federated GitHub Actions deploys with no long-lived credentials.

Stratum is a domain-specific RAG engine designed around the assumption that retrieval quality and answer attribution are the actual problems — not the LLM. Queries flow through a hybrid retriever that runs BM25 sparse search and dense ANN in parallel, fuses results with Reciprocal Rank Fusion (RRF, K=60), expands matched child chunks back to their parent context, and reranks with a cross-encoder (ms-marco-MiniLM) before any chunk reaches the LLM. The fusion is parameter-free by design — robust to miscalibrated retrievers in a way that learned weights aren't, until you have enough labeled data to justify training them.

Generation is where most RAG systems quietly fail. Stratum's generator builds a context block with explicit [src N] markers, calls Claude with a citation-enforcing prompt, and parses and validates every citation marker before returning. If the model produces an answer without grounding it in the retrieved sources, the response is rejected at the generator layer — not surfaced to the user as a "best effort."

The infrastructure is dual-backend by design. Local development runs against in-process Chroma with zero Docker. Production runs against Weaviate 1.27 with HNSW tuning and gRPC, deployed on AWS via Terraform: ALB fronting EC2 instances for the FastAPI service and Streamlit UI, a separate node for Weaviate on a 20GB EBS volume, and S3 for raw document storage. Evaluation runs weekly via DeepEval against a golden dataset using a local Ollama judge — zero API cost on the eval gate.

The LLM Specialization Platform is an end-to-end pipeline for fine-tuning, evaluating, and exporting specialized language models — built adversarially against the failure modes that sink production ML. The initial task is structured JSON extraction from unstructured text using Qwen2.5-7B-Instruct. The pipeline runs Phase 0 (tokenizer audit + baseline) → Phase 1 (QLoRA SFT) → Phase 2 (DPO from best SFT checkpoint) → Phase 3 (evaluation: raw + constrained decoding + metrics.json) → Phase 4 (adapter → merged BF16 → GGUF Q8_0 + Q4_K_M → re-verify). Swapping tasks requires a new dataset, schema, and config — no code changes.

The results are real and measured on a frozen 375-example test set after training on 2,998 examples. DPO improved extraction recall 15% over SFT (0.650 vs 0.498). Null accuracy — the model's ability to correctly abstain when there's nothing to extract — is perfect across all artifacts including both GGUF quantizations. Constrained decoding recovers schema validity to 71.8% on the DPO model; production deployment uses outlines. The CI gate emits a versioned metrics.json with pass/fail flags designed to be consumed by downstream GitOps pipelines — including Sentinel.

The infrastructure discipline matters as much as the training decisions. Every run is reproducible: manifests capture git commit, lockfile hash, dataset hash, and full hardware fingerprint. Validation gates raise ValueError — they don't warn silently and let bad datasets reach training. Field-level F1 is computed on positive examples only — including null cases inflates it for any over-abstaining model, masking real extraction quality, and this pipeline doesn't do that.