Vault Secrets Broker

KV Engine vs Dynamic Secrets Engine

KV Secrets Engine (storage only)

✅ Stores the credential securely

✅ Audits every read/write

✅ Enforces TTL on Vault's copy

❌ Does NOT call the platform's API to rotate anything

❌ Old credential stays valid on the SaaS side

❌ No revocation on lease expiry

❌ Leaked credential remains usable indefinitely

Dynamic Secrets Engine — the gap-filler ✓

✅ Calls the platform's API to create a new credential

✅ Returns short-lived dynamic credential with TTL

✅ On lease expiry: calls API to revoke the credential

✅ Credential lifetime fully managed end-to-end

✅ Leaked credential is automatically expired and revoked

✅ No long-lived static secrets

✅ Transit-encrypted in transit between components

Architecture

flowchart TB
    subgraph consumers["Applications / Consumers"]
        app1["App A\nvault read generic/creds/my-role"]
        app2["App B\nvault read auth0/creds/my-role"]
    end

    subgraph spire["SPIFFE / SPIRE Identity Plane"]
        spire_server["SPIRE Server\nOIDC provider · JWT-SVID CA"]
        spire_agent["SPIRE Agent\nworkload attestation API"]
    end

    subgraph vault["HashiCorp Vault 1.18+ · TLS :8200"]
        engine1["vault-rest-engine\n(Generic REST Plugin)"]
        engine3["vault-auth0-engine\n(Auth0 Native Plugin)"]
        transit["Transit Engine\ncred-rotation-key\naes256-gcm96"]
        pki["PKI Engine\nInternal CA"]
        kv["KV v2\nTransit-encrypted\nadapter config"]
        jwt_auth["JWT Auth Method\nSPIFFE trust domain"]
    end

    subgraph broker["Credential Rotation API · mTLS :8443"]
        direction LR
        a0["Auth0\nAdapter"]
        sp["Splunk\nAdapter"]
        sq["SonarQube\nAdapter"]
        gh["GitHub\nAdapter"]
        dd["Datadog\nAdapter"]
        pd["PagerDuty\nAdapter"]
    end

    subgraph saas["SaaS Providers · TLS 1.3"]
        auth0["Auth0\nMgmt API"]
        splunk["Splunk\nREST API"]
        sonar["SonarQube\nWeb API"]
        github["GitHub\nTokens API"]
        datadog["Datadog\nAPI Keys API"]
        pagerduty["PagerDuty\nREST API v2"]
    end

    app1 -- "TLS 1.3" --> engine1
    app2 -- "TLS 1.3" --> engine3
    engine1 -- "mTLS\nPKI client cert" --> a0
    engine3 -- "TLS 1.3" --> auth0
    a0 -- "TLS 1.3" --> auth0
    sp -- "TLS 1.3" --> splunk
    sq -- "TLS 1.3" --> sonar
    gh -- "TLS 1.3" --> github
    dd -- "TLS 1.3" --> datadog
    pd -- "TLS 1.3" --> pagerduty
    engine1 & broker -- "Transit encrypt/decrypt\nenvelope encryption" --> transit
    broker -- "Transit-decrypt\nadapter config at boot" --> kv
    pki -- "issues client cert\nfor vault-rest-engine" --> engine1
    spire_agent -- "workload API\ngRPC unix socket" --> broker
    spire_server -- "OIDC JWKS\nJWT-SVID validation" --> jwt_auth
    broker -- "POST /v1/auth/jwt/login\nJWT-SVID credential" --> jwt_auth

vault-rest-engine dispatches to Credential Rotation API over mTLS. Adding a provider requires only a new adapter — zero plugin changes. SPIFFE/SPIRE provides workload identity so cred-rotation-api authenticates to Vault with a short-lived JWT-SVID, not a static token.

The AuthZEN Pattern Parallel

Just as AuthZEN standardises the authorization decision interface so any PDP can implement it, the Credential Rotation API standardises the credential lifecycle interface so any SaaS adapter can implement it.

AuthZEN — authorization decisions

POST /access/v1/evaluation { subject, resource, action, context } → { decision: bool }

Any PDP implements this contract. Callers don't know if it's OPA, OpenFGA, or an in-memory PDP.

CredentialBroker — credential lifecycle

POST /v1/credentials/rotate { provider, resource_id, lease_id } → { encrypted_value, expires_at }

Any adapter implements this contract. Vault doesn't know — or care — which platform is behind it.

Security Model — Defence in Depth

Layer 1 — TLS 1.3

All Vault API traffic (consumer → Vault). Enforced by Vault; cannot be disabled in production mode.

Layer 2 — Mutual TLS (mTLS)

Plugin → Credential Rotation API. Both sides present certificates. API rejects any client without a cert signed by the internal PKI CA. The cert IS the plugin's identity — no bearer tokens.

Layer 3 — Transit Envelope Encryption (credential values in motion)

New credentials are Transit-encrypted by the API before being sent to the plugin over mTLS. Each rotate call generates a unique DEK (AES-256-GCM) — the same credential rotated twice produces two unrelated ciphertexts. Correlation and replay attacks blocked at the envelope level.

Layer 4 — Transit Envelope Encryption (adapter config at rest)

Each adapter's admin credentials (API keys, management tokens) stored in KV encrypted via Transit. The KV backend at rest cannot be read without the Transit master key. Plaintext credentials never touch Vault storage.

Layer 5 — Vault Storage Encryption

Vault's backend storage (Raft) is always encrypted with Vault's own seal key. This baseline applies even before Transit adds its layer.

Layer 6 — SPIFFE / SPIRE Workload Identity

cred-rotation-api authenticates to Vault using a SPIFFE JWT-SVID — a short-lived (5 min), cryptographically-attested workload identity. No static Vault token, no AppRole secret stored on disk. SPIRE Agent attests the workload at startup via platform-level proofs (OS process, Kubernetes pod metadata, etc.) and issues SVIDs only to verified workloads.

🔒

Credential never plaintext at rest.

The plaintext credential exists only: (1) in the SaaS API's TLS response at the adapter, and (2) in the Vault plugin's process memory immediately before returning to the consumer. It is never logged, never stored in KV, never written to disk.

Credential Rotation Flow

sequenceDiagram
    participant App as Consumer App
    participant Vault as HashiCorp Vault
    participant Plugin as vault-rest-engine
    participant Transit as Transit Engine
    participant API as cred-rotation-api
    participant Provider as SaaS Platform API

    App->>Vault: vault read generic/creds/my-role (TLS)
    Vault->>Plugin: dispatch to plugin process (gRPC + mTLS)
    Plugin->>API: POST /v1/credentials/rotate (mTLS + client cert)
    API->>Provider: rotate credential via platform API (TLS 1.3)
    Provider-->>API: new_credential (plaintext, within TLS channel)
    API->>Transit: POST /v1/transit/encrypt/cred-rotation-key
    Note over API,Transit: Envelope encryption — unique DEK per call
    Transit-->>API: vault:v1:AbC123... (ciphertext)
    API-->>Plugin: { encrypted_value: "vault:v1:AbC123..." } (mTLS)
    Note over API,Plugin: plaintext never leaves API unencrypted
    Plugin->>Transit: POST /v1/transit/decrypt/cred-rotation-key
    Transit-->>Plugin: plaintext credential (in plugin memory only)
    Plugin-->>Vault: { secret_value, lease_id, ttl: 86400 }
    Vault-->>App: secret (TLS)

On Vault lease expiry, the plugin calls POST /v1/credentials/revoke — the platform immediately invalidates the old credential. No long-lived credential window exists.

Credential Rotation API Contract

POST

/v1/credentials/rotate

Generate a new credential on the target system. Returns the value Transit-encrypted. Body: { provider, resource_id, lease_id, context? }

POST

/v1/credentials/revoke

Invalidate a previously issued credential on the target system. Called by Vault on lease expiry. Body: { provider, resource_id, lease_id, credential_id }

GET

/v1/credentials/status/{lease_id}

Check if a credential is still valid on the target system. Does not expose the credential value. Used for debugging rotation failures.

GET

/v1/health

Liveness + readiness. Returns per-adapter connectivity status. Vault plugin can refuse to serve credentials if the API reports unhealthy.

Implementation Phases

✓

Phase 1 — Vault Foundation

Vault dev server, PKI engine (internal CA), Transit engine (cred-rotation-key, aes256-gcm96), KV v2 with Transit-encrypted adapter config, AppRole auth, least-privilege policies.

vault server -dev PKI · Transit · AppRole · KV v2 scripts/phase1-setup.sh

✓

Phase 2 — Credential Rotation API

Go mTLS server with client certificate validation, Adapter interface (Rotate / Revoke / Status), first reference adapter for a platform without a native Vault secrets engine, Transit-encrypted config loading at startup.

cred-rotation-api mTLS server reference adapter Transit decrypt on boot

✓

Phase 3 — Generic REST Secrets Engine

Custom Vault plugin (vault-rest-engine) using Vault Plugin SDK. config / roles / creds paths, mTLS call to Credential Rotation API, lease management with revocation callback, SHA256-pinned binary registration.

vault-rest-engine Vault Plugin SDK lease management

✓

Phase 4 — Custom Native Secrets Engine

vault-auth0-engine: a provider-specific plugin calling the platform API directly — no intermediate rotation service. Built to objectively compare the trade-offs against the generic REST engine. See the comparison →

vault-auth0-engine native plugin vs generic REST engine

✓

Phase 5 — Multi-Provider Adapter Suite

Six adapters for platforms with no native Vault secrets engine — proving the abstraction layer: each new platform requires only a new adapter, zero plugin changes. Examples include SIEM platforms, identity providers, observability tools, and incident management.

Auth0 Splunk SonarQube (Bearer) GitHub Datadog PagerDuty

▶

Phase 6 — SPIFFE / SPIRE + Hardening

SPIRE integration test harness (Docker Compose, distroless Chainguard runtime, Vault OIDC JWT auth). cred-rotation-api authenticates to Vault via JWT-SVID — no static Vault tokens. Circuit breaker, rotation failure handling, audit log review, and this interactive demo.

SPIFFE · SPIRE 1.11 JWT-SVID auth Chainguard distroless SPIRE integration tests demo pages

Quick Start

⚠️

Prerequisites:

Go 1.27+, Vault 1.18+ (brew install hashicorp/tap/vault), Docker (for SPIRE integration tests), and credentials for the target platform (admin API key or management token).

# Clone and configure
git clone https://github.com/jralmaraz/vault-secrets-broker.git
cd vault-secrets-broker
cp .env.example .env
# edit .env with your platform admin credentials

# Run Phase 1 — starts Vault and configures all engines
make setup

# Load the Vault environment
source .vault-env

# Verify the platform admin credential is Transit-encrypted (not plaintext)
vault kv get secret/cred-rotation-api/adapters/auth0
# admin_token_encrypted = vault:v1:...  ← ciphertext ✓

# Verify Transit decrypt round-trip
CIPHER=$(vault kv get -field=admin_token_encrypted \
  secret/cred-rotation-api/adapters/auth0)
vault write -field=plaintext transit/decrypt/cred-rotation-key \
  ciphertext="$CIPHER" | base64 -d
# → your original client_secret ✓

Supply Chain Security

go mod verify

Every CI run cryptographically verifies all dependencies against go.sum. Tampered modules fail immediately.

govulncheck v1.6.0

Scans the actual call graph for reachable vulnerabilities from the Go vuln database. Runs weekly even without new commits.

gosec + CodeQL

Static security analysis: hardcoded credentials, TLS misconfig, weak crypto, unsafe random. SARIF results in GitHub Security → Code Scanning.

SBOM (CycloneDX)

Software Bill of Materials generated per module on every main push. 90-day retention. Used for license compliance and CVE impact assessment.

Dependency Review

PRs that introduce dependencies with CVSS ≥ 4.0 or non-OSI licenses are blocked before merge.

Dependabot

Weekly automatic PRs for all 3 Go modules + GitHub Actions. Each Dependabot PR triggers full CI + security scan.

Genuine Credential Rotation Not Just Storage

KV Engine vs Dynamic Secrets Engine

Architecture

The AuthZEN Pattern Parallel

Security Model — Defence in Depth

Credential Rotation Flow

Credential Rotation API Contract

Implementation Phases

Quick Start

Supply Chain Security

Genuine Credential Rotation
Not Just Storage