Ade
Daramola

GitOps Sentinel is an autonomous remediation platform for Kubernetes clusters. When Alertmanager fires a webhook, Sentinel doesn't just page on-call — it ingests the signal through an HMAC-validated API Gateway, deduplicates via DynamoDB, bundles Prometheus metrics and k8s events into S3, and routes the incident through a Step Functions multi-agent pipeline: Classifier → Root Cause → Action Planner → Confidence Scorer. Every remediation is a Git commit. Argo CD detects the merged PR and syncs the cluster. The cluster only changes through the same reviewed write path a human engineer would use.

The design comes from a real operational insight: not all incidents are equal, and the action shouldn't be either. Sentinel gates every remediation on a deterministic confidence score with three explicit routes — ≥80 auto-applies the PR, 40–79 opens a PR for engineer review, and <40 escalates to on-call with no automated change. Routine incidents (OOMKilled pods, replica drift, known bad image tags) resolve in seconds. Novel or high-risk scenarios escalate before anything touches the cluster. Five minutes after every remediation, an Outcome Validator queries Prometheus to confirm the fix held — and if it didn't, it opens an automatic revert PR. The cluster is never worse off than it was before Sentinel ran.

This is the kind of system that pays for itself the first time it silently fixes something at 4am that would have been a two-hour incident.

Stratum is a domain-specific RAG engine designed around the assumption that retrieval quality and answer attribution are the actual problems — not the LLM. Queries flow through a hybrid retriever that runs BM25 sparse search and dense ANN in parallel, fuses results with Reciprocal Rank Fusion (RRF, K=60), expands matched child chunks back to their parent context, and reranks with a cross-encoder (ms-marco-MiniLM) before any chunk reaches the LLM. The fusion is parameter-free by design — robust to miscalibrated retrievers in a way that learned weights aren't, until you have enough labeled data to justify training them.

Generation is where most RAG systems quietly fail. Stratum's generator builds a context block with explicit [src N] markers, calls Claude with a citation-enforcing prompt, and parses and validates every citation marker before returning. If the model produces an answer without grounding it in the retrieved sources, the response is rejected at the generator layer — not surfaced to the user as a "best effort."

The infrastructure is dual-backend by design. Local development runs against in-process Chroma with zero Docker. Production runs against Weaviate 1.27 with HNSW tuning and gRPC, deployed on AWS via Terraform: ALB fronting EC2 instances for the FastAPI service and Streamlit UI, a separate node for Weaviate on a 20GB EBS volume, and S3 for raw document storage. Evaluation runs weekly via DeepEval against a golden dataset using a local Ollama judge — zero API cost on the eval gate.

The LLM Specialization Platform is the training and evaluation backbone for turning general-purpose foundation models into domain-specific specialists. The pipeline is built around the HuggingFace stack — transformers, PEFT for parameter-efficient fine-tuning (LoRA / QLoRA), TRL for preference optimization (DPO / RLHF), and Accelerate for multi-GPU orchestration — with bitsandbytes quantization so a 13B-parameter base model can be specialized on a single A100 instead of a fleet.

The infrastructure decisions matter more than the choice of optimizer. Every training run is configuration-driven via Pydantic and OmegaConf — no notebooks, no hardcoded paths, no "it worked on my machine." Datasets are versioned and pulled from S3 via boto3 and s3fs. Experiments are tracked end-to-end through Weights & Biases and MLflow, so six weeks later you can answer the question that always gets asked at deployment time: which dataset version, which hyperparameters, and which base model produced this adapter?

Output quality is enforced at inference time, not just at eval. The platform integrates Outlines and Instructor for constrained decoding, so specialized models produce schema-valid JSON, function calls, or structured reports by construction — not by hoping the model behaves. Evaluation runs through scikit-learn, ROUGE, and JSON-schema validators against held-out golden sets, with results logged to the same experiment tracker as training. The whole thing is containerized, reproducible, and pinned to exact dependency versions — the same discipline I apply to production cloud infrastructure, applied to model training.