Vault Secrets Broker

KV Engine vs Dynamic Secrets Engine

KV Secrets Engine (storage only)

✅ Stores the credential securely

✅ Audits every read/write

✅ Enforces TTL on Vault's copy

❌ Does NOT call Auth0/Splunk/any API

❌ Old credential stays valid on the SaaS side

❌ No revocation on lease expiry

❌ Leaked credential remains usable indefinitely

Dynamic Secrets Engine (this PoC) ✓

✅ Calls the target API to create a new credential

✅ Returns short-lived dynamic credential with TTL

✅ On lease expiry: calls API to revoke the credential

✅ Credential lifetime fully managed end-to-end

✅ Leaked credential is automatically expired and revoked

✅ No long-lived static secrets

✅ Transit-encrypted in transit between components

Architecture

flowchart TB
    subgraph consumers["Applications / Consumers"]
        app1["App A\nvault read generic/creds/my-role"]
        app2["App B\nvault read auth0/creds/my-role"]
    end

    subgraph vault["HashiCorp Vault 2.1+ · TLS :8200"]
        engine1["vault-rest-engine\n(Generic REST Plugin)"]
        engine3["vault-auth0-engine\n(Auth0 Native Plugin)"]
        transit["Transit Engine\ncred-rotation-key\naes256-gcm96"]
        pki["PKI Engine\nInternal CA"]
        kv["KV v2\nTransit-encrypted\nadapter config"]
    end

    subgraph broker["Credential Rotation API · mTLS :8443"]
        direction LR
        a0["Auth0\nAdapter"]
        sp["Splunk\nAdapter"]
        sq["SonarQube\nAdapter"]
    end

    subgraph saas["SaaS Providers · TLS"]
        auth0["Auth0\nMgmt API"]
        splunk["Splunk\nREST API"]
        sonar["SonarQube\nWeb API"]
    end

    app1 -- "TLS" --> engine1
    app2 -- "TLS" --> engine3
    engine1 -- "mTLS\nclient cert from PKI" --> a0
    engine3 -- "TLS" --> auth0
    a0 -- "TLS" --> auth0
    sp -- "TLS" --> splunk
    sq -- "TLS" --> sonar
    engine1 & broker -- "Transit encrypt/decrypt\nenvelope encryption" --> transit
    broker -- "Transit-decrypt\nadapter config" --> kv
    pki -- "issues client cert\nfor vault-rest-engine" --> engine1

Component 1 (vault-rest-engine) calls Component 2 (Credential Rotation API) over mTLS. Adding a new SaaS provider requires only a new adapter in Component 2 — zero plugin changes.

The AuthZEN Pattern Parallel

Just as AuthZEN standardises the authorization decision interface so any PDP can implement it, the Credential Rotation API standardises the credential lifecycle interface so any SaaS adapter can implement it.

AuthZEN — authorization decisions

POST /access/v1/evaluation { subject, resource, action, context } → { decision: bool }

Any PDP implements this contract. Callers don't know if it's OPA, OpenFGA, or an in-memory PDP.

CredentialBroker — credential lifecycle

POST /v1/credentials/rotate { provider, resource_id, lease_id } → { encrypted_value, expires_at }

Any SaaS adapter implements this contract. Vault doesn't know if it's Auth0, Splunk, or SonarQube.

Security Model — Defence in Depth

Layer 1 — TLS 1.3

All Vault API traffic (consumer → Vault). Enforced by Vault; cannot be disabled in production mode.

Layer 2 — Mutual TLS (mTLS)

Plugin → Credential Rotation API. Both sides present certificates. API rejects any client without a cert signed by the internal PKI CA. The cert IS the plugin's identity — no bearer tokens.

Layer 3 — Transit Envelope Encryption (credential values in motion)

New credentials are Transit-encrypted by the API before being sent to the plugin over mTLS. Each rotate call generates a unique DEK (AES-256-GCM) — the same credential rotated twice produces two unrelated ciphertexts. Correlation and replay attacks blocked at the envelope level.

Layer 4 — Transit Envelope Encryption (adapter config at rest)

SaaS adapter credentials (Auth0 mgmt secret, Splunk token) stored in KV encrypted via Transit. The KV backend at rest cannot be read without the Transit master key. Plaintext credentials never touch Vault storage.

Layer 5 — Vault Storage Encryption

Vault's backend storage (Raft) is always encrypted with Vault's own seal key. This baseline applies even before Transit adds its layer.

🔒

Credential never plaintext at rest.

The plaintext credential exists only: (1) in the SaaS API's TLS response at the adapter, and (2) in the Vault plugin's process memory immediately before returning to the consumer. It is never logged, never stored in KV, never written to disk.

Credential Rotation Flow

sequenceDiagram
    participant App as Consumer App
    participant Vault as HashiCorp Vault
    participant Plugin as vault-rest-engine
    participant Transit as Transit Engine
    participant API as cred-rotation-api
    participant Auth0 as Auth0 Mgmt API

    App->>Vault: vault read generic/creds/my-role (TLS)
    Vault->>Plugin: dispatch to plugin process (gRPC + mTLS)
    Plugin->>API: POST /v1/credentials/rotate (mTLS + client cert)
    API->>Auth0: POST /api/v2/clients/{id}/rotate-secret (TLS)
    Auth0-->>API: new_client_secret (plaintext, in TLS channel)
    API->>Transit: POST /v1/transit/encrypt/cred-rotation-key
    Note over API,Transit: Envelope encryption — unique DEK per call
    Transit-->>API: vault:v1:AbC123... (ciphertext)
    API-->>Plugin: { encrypted_value: "vault:v1:AbC123..." } (mTLS)
    Note over API,Plugin: plaintext never leaves API unencrypted
    Plugin->>Transit: POST /v1/transit/decrypt/cred-rotation-key
    Transit-->>Plugin: plaintext new_secret (in plugin memory only)
    Plugin-->>Vault: { secret_value, lease_id, ttl: 86400 }
    Vault-->>App: secret (TLS)

On Vault lease expiry, the plugin calls POST /v1/credentials/revoke — Auth0 immediately invalidates the old credential.

Credential Rotation API Contract

POST

/v1/credentials/rotate

Generate a new credential on the target system. Returns the value Transit-encrypted. Body: { provider, resource_id, lease_id, context? }

POST

/v1/credentials/revoke

Invalidate a previously issued credential on the target system. Called by Vault on lease expiry. Body: { provider, resource_id, lease_id, credential_id }

GET

/v1/credentials/status/{lease_id}

Check if a credential is still valid on the target system. Does not expose the credential value. Used for debugging rotation failures.

GET

/v1/health

Liveness + readiness. Returns per-adapter connectivity status. Vault plugin can refuse to serve credentials if the API reports unhealthy.

Implementation Phases

✓

Phase 1 — Vault Foundation

Vault dev server, PKI engine (internal CA), Transit engine (cred-rotation-key, aes256-gcm96), KV v2 with Transit-encrypted Auth0 adapter config, AppRole auth, least-privilege policies.

vault server -dev PKI · Transit · AppRole · KV v2 scripts/phase1-setup.sh

Phase 2 — Credential Rotation API

Go mTLS server with client certificate validation, Adapter interface, Auth0 adapter (POST /api/v2/clients/{id}/rotate-secret), Transit-encrypted config loading at startup.

cred-rotation-api mTLS server Auth0 adapter Transit decrypt on boot

Phase 3 — Generic REST Secrets Engine

Custom Vault plugin (vault-rest-engine) using Vault Plugin SDK. config / roles / creds paths, mTLS call to Credential Rotation API, lease management with revocation callback, SHA256-pinned binary registration.

vault-rest-engine Vault Plugin SDK lease management

Phase 4 — Auth0-native Secrets Engine

vault-auth0-engine plugin: same SDK structure but calls Auth0 Management API directly (no intermediate API hop). Comparison with Phase 2+3 approach shows the trade-off between flexibility and efficiency.

vault-auth0-engine Auth0 Mgmt API direct comparison study

Phase 5 — Splunk + SonarQube Adapters

Add two adapters to cred-rotation-api. Zero changes to Vault plugins. This is the key demonstration of the abstraction layer value: new SaaS providers with no plugin work.

Splunk adapter SonarQube adapter zero plugin changes

Phase 6 — Hardening + Demo Narrative

Circuit breaker, rotation failure handling (mid-flight revoke + new credential failure), audit log review, demo script showing the KV misconception in action and the full rotation lifecycle.

circuit breaker audit log demo script

Quick Start

⚠️

Prerequisites:

Go 1.26+, Vault 2.1+ (brew install hashicorp/tap/vault), an Auth0 dev tenant with a Management API M2M application.

# Clone and configure
git clone https://github.com/jralmaraz/vault-secrets-broker.git
cd vault-secrets-broker
cp .env.example .env
# edit .env with your Auth0 tenant credentials

# Run Phase 1 — starts Vault and configures all engines
make setup

# Load the Vault environment
source .vault-env

# Verify the Auth0 client_secret is Transit-encrypted (not plaintext)
vault kv get secret/cred-rotation-api/adapters/auth0
# mgmt_client_secret_encrypted = vault:v1:...  ← ciphertext ✓

# Verify Transit decrypt round-trip
CIPHER=$(vault kv get -field=mgmt_client_secret_encrypted \
  secret/cred-rotation-api/adapters/auth0)
vault write -field=plaintext transit/decrypt/cred-rotation-key \
  ciphertext="$CIPHER" | base64 -d
# → your original client_secret ✓

Supply Chain Security

go mod verify

Every CI run cryptographically verifies all dependencies against go.sum. Tampered modules fail immediately.

govulncheck v1.6.0

Scans the actual call graph for reachable vulnerabilities from the Go vuln database. Runs weekly even without new commits.

gosec + CodeQL

Static security analysis: hardcoded credentials, TLS misconfig, weak crypto, unsafe random. SARIF results in GitHub Security → Code Scanning.

SBOM (CycloneDX)

Software Bill of Materials generated per module on every main push. 90-day retention. Used for license compliance and CVE impact assessment.

Dependency Review

PRs that introduce dependencies with CVSS ≥ 4.0 or non-OSI licenses are blocked before merge.

Dependabot

Weekly automatic PRs for all 3 Go modules + GitHub Actions. Each Dependabot PR triggers full CI + security scan.

Genuine Credential Rotation Not Just Storage

KV Engine vs Dynamic Secrets Engine

Architecture

The AuthZEN Pattern Parallel

Security Model — Defence in Depth

Credential Rotation Flow

Credential Rotation API Contract

Implementation Phases

Quick Start

Supply Chain Security

Genuine Credential Rotation
Not Just Storage