Composable AI Micro‑services Architecture for Scalable Security Questionnaire Automation

Enterprises are drowning in an ever‑growing tide of security questionnaires, vendor assessments, and compliance audits. Traditional monolithic tools struggle to keep up, especially when they must integrate with disparate product ecosystems, support multilingual requests, and provide real‑time audit trails.

A composable micro‑services architecture, built around large language models (LLMs) and retrieval‑augmented generation (RAG), offers a way to scale automation while preserving the flexibility and governance that regulated industries demand. In this guide we’ll:

• Outline the core design principles that keep the system secure, auditable, and extensible.
• Walk through a reference implementation diagrammed with Mermaid.
• Show how each service can be deployed independently on Kubernetes, serverless FaaS, or edge runtimes.
• Provide concrete best‑practice recommendations for data governance, observability, and continuous improvement.

TL;DR: Break the questionnaire automation platform into small, well‑defined services, let LLMs sit behind a stateless inference layer, and use event‑driven pipelines to maintain a single source of truth for evidence and version control.

1. Why Compose Rather Than Build a Giant Monolith?

The Composable model embraces the “you build what you need, and you replace what you don’t” philosophy. It matches the dynamic nature of security questionnaires, where new control frameworks (e.g., ISO 27001 Rev 2) appear regularly and teams must quickly adapt.

2. Core Architectural Pillars

1. Stateless API Gateway – entry point for UI, SaaS connectors, and external tools. Handles authentication, request validation, and throttling.
2. Domain‑Specific Micro‑services – each encapsulates a bounded context:
   • Questionnaire Service – stores questionnaire metadata, versioning, and task assignments.
   • Evidence Service – manages artifacts (policies, screenshots, audit logs) in an immutable object store.
   • AI Orchestration Service – composes prompts, runs RAG pipelines, and returns answer drafts.
   • Change‑Detection Service – watches evidence updates, triggers re‑evaluation of affected answers.
   • Notification Service – pushes Slack, Teams, or email events to stakeholders.
3. Event Bus (Kafka / Pulsar) – guarantees at‑least‑once delivery of domain events (e.g., EvidenceUploaded, AnswerDrafted).
4. Observability Stack – OpenTelemetry traces across services, Prometheus metrics, and Loki logs.
5. Policy‑as‑Code Engine – evaluates compliance rules (written in Rego or OPA) before an answer is marked “final”.

All services communicate via gRPC (for low latency) or REST (for external integrations). The design encourages dumb pipes, smart endpoints—the business logic lives where it belongs, while the bus merely transports messages.

3. Data Flow – From Question to Auditable Answer

Below is a Mermaid diagram that visualises a typical request lifecycle.

  flowchart TD
    subgraph UI["User Interface"]
        UI1["\"Web UI\""] -->|Submit questionnaire| AG["\"API Gateway\""]
    end

    AG -->|Auth & Validate| QMS["\"Questionnaire Service\""]
    QMS -->|Fetch template| AIOS["\"AI Orchestration Service\""]
    AIOS -->|Retrieve relevant evidence| ES["\"Evidence Service\""]
    ES -->|Evidence objects| AIOS
    AIOS -->|Generate draft answer| RAG["\"RAG Pipeline\""]
    RAG -->|LLM output| AIOS
    AIOS -->|Store draft| QMS
    QMS -->|Emit AnswerDrafted| EB["\"Event Bus\""]
    EB -->|Trigger| CDS["\"Change‑Detection Service\""]
    CDS -->|Re‑run if evidence changed| AIOS
    CDS -->|Emit AnswerUpdated| EB
    EB -->|Notify| NS["\"Notification Service\""]
    NS -->|Push to Slack/Email| UI

    style UI fill:#f9f,stroke:#333,stroke-width:2px
    style AG fill:#bbf,stroke:#333,stroke-width:1px
    style QMS fill:#bfb,stroke:#333,stroke-width:1px
    style AIOS fill:#ffb,stroke:#333,stroke-width:1px
    style ES fill:#fbb,stroke:#333,stroke-width:1px
    style RAG fill:#fdd,stroke:#333,stroke-width:1px
    style CDS fill:#ddf,stroke:#333,stroke-width:1px
    style NS fill:#cfc,stroke:#333,stroke-width:1px

Key moments in the flow:

1. User submits a new questionnaire or selects an existing one.
2. API Gateway validates JWT, checks rate limits, forwards to the Questionnaire Service.
3. The Questionnaire Service pulls the questionnaire template and posts an event to the AI Orchestration Service.
4. AI Orchestration performs a retrieval step—it queries the Evidence Service for all artifacts relevant to the current control (using vector similarity or keyword match).
5. The retrieved contexts, together with the prompt template, feed a RAG pipeline (e.g., openAI/gpt‑4o‑preview).
6. The draft answer is stored back in the Questionnaire Service, marked “pending review.”
7. The Change‑Detection Service watches for new evidence uploads. If a policy is updated, it re‑triggers the RAG pipeline for impacted answers.
8. Final reviewers accept or edit the draft; upon acceptance, the Policy‑as‑Code Engine validates that the answer satisfies all rule constraints before committing it to an immutable audit log.

4. Implementation Details

4.1. API Gateway (Envoy + OIDC)

• Routing – POST /questionnaires/:id/answers → questionnaire-service.
• Security – Enforce scopes (questionnaire:write).
• Rate limiting – 100 requests/min per tenant to protect downstream LLM costs.

4.2. Questionnaire Service (Go)

type Questionnaire struct {
    ID          string            `json:"id"`
    Version     int               `json:"version"`
    Controls    []Control        `json:"controls"`
    Drafts      map[string]Answer `json:"drafts"` // key = control ID
    AssignedTo  map[string]string `json:"assigned_to"` // userID
}

• Uses PostgreSQL for relational data, EventStoreDB for domain events.
• Exposes gRPC methods GetTemplate, SaveDraft, FinalizeAnswer.

4.3. Evidence Service (Python + FastAPI)

• Stores files in MinIO or AWS S3 with bucket‑level encryption.
• Indexes content in Qdrant (vector DB) for similarity search.
• Provides an endpoint POST /search that accepts a query and returns top‑k artifact IDs.

4.4. AI Orchestration Service (Python)

def generate_answer(question: str, evidence_ids: List[str]) -> str:
    evidence = fetch_evidence(evidence_ids)
    context = "\n".join(evidence)
    prompt = f"""You are a compliance specialist.
    Using the following evidence, answer the question concisely:\n{context}\n\nQuestion: {question}"""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role":"system","content":prompt}]
    )
    return response.choices[0].message.content

• RAG – Combine vector search with a system prompt that instructs the model to cite evidence IDs.
• Caching – Store generated responses for 24 h to avoid duplicate LLM calls.

4.5. Change‑Detection Service (Rust)

• Subscribes to EvidenceUploaded events.
• Computes a hash of the new artifact and runs a diff against existing evidence linked to each control.
• If the diff exceeds a configurable threshold, it publishes AnswerRequiresRegen.

4.6. Notification Service (Node.js)

• Listens to AnswerDrafted, AnswerFinalized, AnswerRequiresRegen.
• Formats Slack blocks, Teams Adaptive Cards, or email templates.
• Supports deduplication – only notifies once per change per questionnaire.

5. Security & Governance

6. Scaling Strategies

1. Horizontal Pod Autoscaling (HPA) – Scale the AI Orchestration pods based on GPU utilization (nvidia.com/gpu).
2. Burst‑able Queues – Use Kafka partitioning to isolate high‑traffic tenants.
3. Cold‑Start Reduction – Keep a warm pool of containers for the LLM inference server (e.g., using KEDA with a custom scaler).
4. Cost Controls – Apply token‑based budgeting per tenant; throttle or charge over‑usage automatically.

7. Observability & Continuous Improvement

• Distributed Tracing – OpenTelemetry spans from UI request → API Gateway → AI Orchestration → RAG → Evidence Service.
• Metrics – answer_draft_latency_seconds, evidence_upload_bytes, llm_token_usage.
• Log Aggregation – Structured JSON logs with request_id propagated across services.
• Feedback Loop – After answer finalization, capture reviewer comments (review_score). Feed these into a reinforcement learning model that adjusts prompt temperature or selects alternative evidence sources.

8. Step‑by‑Step Migration Path for Existing Teams

By incrementally adopting the composable architecture, teams avoid the “big‑bang” risk and can demonstrate early ROI (often a 30‑50 % reduction in questionnaire turnaround).

9. Future‑Proofing the Stack

• Federated Learning – Train lightweight adapters on each tenant’s data without moving the raw evidence offsite, enhancing answer relevance while respecting data sovereignty.
• Zero‑Trust Service Mesh – Use Istio or Linkerd with mutual TLS to secure intra‑service traffic.
• Semantic Governance – Extend the Policy‑as‑Code engine to validate not only answer content but also the semantic similarity between evidence and control language.
• Generative Traceability – Store the exact LLM temperature, top‑p, and system prompt alongside each answer for forensic inspection.

10. Conclusion

A composable micro‑services architecture transforms security questionnaire automation from a painful manual chore into a scalable, auditable, and continuously improving engine. By decoupling responsibilities, leveraging LLMs via a stateless RAG layer, and wiring everything together with an event‑driven backbone, organizations can:

• Respond to vendor assessments in minutes instead of days.
• Keep compliance evidence always up‑to‑date with automated change detection.
• Provide regulators with a clear, immutable audit trail.

Start small, iterate fast, and let the micro‑services philosophy guide you toward a future where compliance is a feature, not a bottleneck.

See Also

• OPA Policy-as-Code for Micro‑services