Ade
Daramola

How the four repositories work together: ops-platform owns the user-facing application and AI workflow. Its CPU-based Agent orchestrates the AI request without direct database credentials. When ticket context needs organizational knowledge, the Agent calls rag-platform through an authenticated retrieval API; that service performs hybrid retrieval and reranking and returns bounded evidence with citation metadata rather than owning final generation. The Agent then sends the assembled context to multi-llm-platform, which provides the model-inference boundary: authentication, cost-aware routing, provider fallback across Amazon Bedrock, Anthropic, and OpenAI, usage accounting, and privacy-aware cache controls. eks-observability-platform supplies the AWS runtime beneath the workflow—networking, EKS, RDS PostgreSQL, ECR, SQS/DLQ, IAM, ingress, encryption, monitoring, and cross-system observability. OpsDesk owns the workflow; RAG supplies evidence; Multi-LLM performs generation; EKS Observability runs and monitors the platform.

OpsDesk is a server-rendered support-ticket and knowledge-management application built with Python, FastAPI, PostgreSQL, Docker, Kubernetes, and AWS EKS. The application implements role-based workflows, audit history, search, optimistic concurrency, secure sessions, CSRF protection, and object-level authorization. AI work is dispatched through a transactional outbox → Amazon SQS → CPU-based Agent path, so database state and work publication remain durable and independently retryable.

The Agent has no direct database credentials. It receives a minimized workflow payload, optionally retrieves bounded evidence from the authenticated RAG service, and sends the final generation request through the separate Multi-LLM gateway. Citation identifiers are validated before persistence, and every generated draft remains pending until a reviewer explicitly approves and applies it.

The RAG path combines dense retrieval and BM25, Reciprocal Rank Fusion, parent-child context expansion, and CPU cross-encoder reranking. In the deployed benchmark it achieved 100% source-level Precision@1 and Recall@5 across 55 answerable queries. Exact and semantic caching produced a 90.91% cache-hit rate and reduced warm median retrieval latency 36.8× — from about 9.0 seconds to 245 ms.

The serverless LLM gateway routes across Amazon Bedrock, Anthropic, and OpenAI with cost-aware selection, provider fallback, API-key controls, per-client usage accounting, privacy-aware cache policies, and SSE streaming. A validated live workflow used Bedrock Nova Micro successfully on the first attempt at an estimated cost of approximately $0.000018, while the generated response still remained behind human review.

The AWS platform is Terraform-managed across a two-AZ VPC with private EKS workloads, isolated RDS PostgreSQL, ECR, encrypted SQS/DLQ, ALB, Route 53, ACM, KMS, Secrets Manager, EKS Pod Identity, CloudWatch, and X-Ray. OpenTelemetry, Prometheus metrics, structured JSON logs, and W3C trace context propagate workflow correlation across service boundaries without making telemetry a readiness dependency.

The platform specializes Qwen2.5-7B-Instruct for structured extraction through QLoRA supervised fine-tuning, DPO preference alignment, frozen-test evaluation, BF16 model merging, and GGUF quantization. Versioned manifests capture Git revision, dependency locks, dataset hashes, and hardware fingerprints so every run can be traced back to the exact inputs that produced it.

On the frozen 377-example test set, the base model scored 0.000 Field F1. SFT reached 0.884, DPO reached 0.885, and the exported Q8 GGUF artifact reached 0.892 Field F1, with 100% schema validity and 100% null accuracy across shipped artifacts. MMLU and HellaSwag regression checks also passed.

The most important result came from the data pipeline: v1 labels were only 54% grounded in the source text and capped Field F1 around 0.47. Relabeling the same input texts under a strict grounding contract raised performance to 0.885 with no model or hyperparameter changes. The pipeline now fails label-audit checks before training when grounding falls below the configured threshold.

The system uses seven Python Lambda functions, Step Functions, Bedrock/OpenAI, EventBridge, DynamoDB, S3, and EKS to ingest alerts, classify incidents, analyze likely root cause, propose bounded actions, and calculate deterministic confidence and risk before choosing a route.

Git and Argo CD remain the workload mutation path. HMAC-authenticated alert intake, deduplication, audit records, Prometheus outcome validation, and automatic revert PRs provide controls around the AI-generated plan. The infrastructure is organized across 20 Terraform modules and validated with 123 automated tests.