ADR-0004 — AI Gateway Abstraction Layer

Status: Accepted
Date: 2026-09-07
Deciders: ASF Architecture Team

Context

ASF requires LLM capabilities for:

Code analysis
Requirement decomposition
Test generation
Context-aware chat ("Ask ASF")
Incident analysis

Target inference backends include on-prem DGX Spark, OpenAI-compatible local APIs, commercial providers, and customer-hosted endpoints.

Tenants may have different AI policies (allowed models, budgets, geographic restrictions).

Decision

Introduce an ASF AI Gateway service (services/asf-ai/) with a stable Python interface.

Interface

Business code requests capabilities, never models:

class AIGateway(Protocol):
    async def analyze_code(self, request: CodeAnalysisRequest) -> AnalysisResult: ...
    async def generate_tests(self, request: TestGenerationRequest) -> GeneratedTests: ...
    async def chat(self, request: ChatRequest, ctx: AIContext) -> ChatResponse: ...
    async def decompose_requirement(self, req: RequirementDecomposeRequest) -> Decomposition: ...

Provider adapters implement the gateway. Model selection, routing, priority queuing, usage metering, cost tracking and tenant AI policy live inside the gateway — not in callers.

Bootstrap

The first implementation ships with a MockProvider that returns deterministic canned responses with realistic latencies. Real providers (OpenAI-compatible, NVIDIA, etc.) are added as adapters later without touching domain/business code.

Consequences

Positive

No LLM provider lock-in — swap / add adapters freely.
Usage and cost tracking centralized.
Tenant-level policy enforceable at one boundary.
Tests for business logic run fast against the mock — no external API calls, no flakiness.

Negative

One more abstraction — the initial API surface must be designed carefully so future providers truly fit.

Mitigation

Provider adapters are written early for at least two real providers, to validate the abstraction before domain code depends heavily on it.

2.0 KiB Raw Blame History