CI/CD Integration

Integrate Lamdis into your CI/CD pipeline to automatically test your AI assistants on every pull request or deployment.

Integration Options

Approach	Best For	Requirements
Hosted Lamdis API	Teams using Lamdis dashboard	API key from dashboard
Self-hosted lamdis-runs	Local/air-gapped environments	lamdis-runs container or npm

Option 1: Hosted Lamdis API

Use the Lamdis API to trigger test suites configured in the dashboard.

1. Create an API Key

In the Lamdis dashboard, open Developers → API Keys
Create a key with runs:trigger and runs:read scopes
Store the secret as a CI secret (e.g., LAMDIS_API_KEY)

2. Trigger a Run


curl -s -X POST "$LAMDIS_API_URL/v1/integrations/ci/runs" \
  -H "Authorization: Bearer $LAMDIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "suiteSelector": {"slugs": ["default"]},
    "target": {"type":"http","baseUrl":"http://localhost:3000"},
    "source": {
      "provider": "github",
      "repository": "my-org/my-repo",
      "commit": "abc123"
    },
    "wait": false
  }'

3. Poll for Results


curl -s -H "Authorization: Bearer $LAMDIS_API_KEY" \
  "$LAMDIS_API_URL/v1/integrations/ci/runs/$RUN_ID"

Response:


{
  "runId": "run_abc123",
  "status": "completed",
  "summary": {
    "total": 10,
    "passed": 9,
    "failed": 1
  },
  "dashboardUrl": "https://app.lamdis.ai/runs/run_abc123"
}

4. Or Use Webhooks

Pass a callback.url to receive a POST when the run completes:


{
  "suiteSelector": {"slugs": ["default"]},
  "target": {"type":"http","baseUrl":"http://localhost:3000"},
  "callback": {
    "url": "https://ci.example.com/lamdis-webhook"
  }
}

Option 2: Self-Hosted lamdis-runs

Run tests directly in your CI job using lamdis-runs CLI.

Local CLI in CI Job


name: lamdis-local
on: [push]
 
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
      
      - name: Start lamdis-runs
        run: |
          cd lamdis-runs
          npm install
          npm run dev &
          sleep 5
        env:
          LAMDIS_API_TOKEN: ${{ secrets.LAMDIS_API_TOKEN }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      
      - name: Run tests
        run: |
          cd lamdis-runs
          npm run run-file -- suites/my-suite.json
        env:
          LAMDIS_API_TOKEN: ${{ secrets.LAMDIS_API_TOKEN }}
          LAMDIS_RUNS_URL: http://127.0.0.1:3101

The CLI exits with non-zero status on failures, failing the CI job.

Hosted Runner with Webhook

For a shared runner that multiple services can trigger:


curl -sS -X POST "$LAMDIS_RUNS_URL/internal/runs/start" \
  -H "content-type: application/json" \
  -H "x-api-token: $LAMDIS_API_TOKEN" \
  -d '{
    "mode": "json",
    "suites": ["legal-tests", "regression-tests"],
    "webhookUrl": "https://ci.mycompany.com/lamdis-webhook",
    "gitContext": {
      "repo": "my-org/my-service",
      "sha": "abc123",
      "runId": "github-run-456"
    }
  }'

Webhook payload on completion:


{
  "mode": "json",
  "status": "passed",
  "passRate": 1,
  "totals": { "passed": 10, "failed": 0, "skipped": 0 },
  "suites": [
    { "id": "legal-tests", "status": "passed", "totals": { "passed": 5, "failed": 0 } },
    { "id": "regression-tests", "status": "passed", "totals": { "passed": 5, "failed": 0 } }
  ],
  "gitContext": {
    "repo": "my-org/my-service",
    "sha": "abc123",
    "runId": "github-run-456"
  }
}

GitHub Actions Examples

Complete Hosted API Example


name: Lamdis CI
on:
  pull_request:
    types: [opened, synchronize, reopened]
 
jobs:
  lamdis:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Trigger Lamdis run
        id: trigger
        env:
          LAMDIS_API_URL: ${{ vars.LAMDIS_API_URL }}
          LAMDIS_API_KEY: ${{ secrets.LAMDIS_API_KEY }}
        run: |
          resp=$(curl -s -X POST "$LAMDIS_API_URL/v1/integrations/ci/runs" \
            -H "Authorization: Bearer $LAMDIS_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{
              "suiteSelector": {"slugs": ["default"]},
              "target": {"type":"http","baseUrl":"http://localhost:3000"},
              "source": {
                "provider": "github",
                "repository": "${{ github.repository }}",
                "commit": "${{ github.sha }}",
                "pullRequest": ${{ github.event.pull_request.number }}
              }
            }')
          echo "Response: $resp"
          echo "runId=$(echo "$resp" | jq -r '.runId')" >> $GITHUB_OUTPUT
      
      - name: Wait for result
        id: result
        env:
          LAMDIS_API_URL: ${{ vars.LAMDIS_API_URL }}
          LAMDIS_API_KEY: ${{ secrets.LAMDIS_API_KEY }}
          RUN_ID: ${{ steps.trigger.outputs.runId }}
        run: |
          for i in {1..60}; do
            s=$(curl -s -H "Authorization: Bearer $LAMDIS_API_KEY" \
              "$LAMDIS_API_URL/v1/integrations/ci/runs/$RUN_ID")
            status=$(echo "$s" | jq -r '.status')
            echo "Status: $status"
            if [ "$status" = "completed" ] || [ "$status" = "failed" ]; then
              echo "$s" > lamdis_result.json
              echo "status=$status" >> $GITHUB_OUTPUT
              break
            fi
            sleep 10
          done
      
      - name: Post PR comment
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const data = JSON.parse(fs.readFileSync('lamdis_result.json', 'utf8'));
            const summary = data.summary || {};
            const url = data.dashboardUrl || process.env.LAMDIS_API_URL;
            const emoji = data.status === 'completed' && summary.failed === 0 ? '✅' : '❌';
            const body = `${emoji} **Lamdis Test Results**\n\n` +
              `| Metric | Count |\n|--------|-------|\n` +
              `| Total | ${summary.total || 0} |\n` +
              `| Passed | ${summary.passed || 0} |\n` +
              `| Failed | ${summary.failed || 0} |\n\n` +
              `[View details in Lamdis](${url})`;
            github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body,
            });
      
      - name: Fail if tests failed
        if: steps.result.outputs.status == 'failed'
        run: exit 1

Self-Hosted with Docker


name: Lamdis Self-Hosted
on: [push]
 
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      lamdis-runs:
        image: ghcr.io/lamdis-ai/lamdis-runs:latest
        ports:
          - 3101:3101
        env:
          LAMDIS_API_TOKEN: ${{ secrets.LAMDIS_API_TOKEN }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Wait for lamdis-runs
        run: |
          for i in {1..30}; do
            if curl -s http://localhost:3101/health; then
              echo "lamdis-runs is ready"
              break
            fi
            sleep 2
          done
      
      - name: Run test suite
        run: |
          curl -sS -X POST "http://localhost:3101/internal/run-file" \
            -H "x-api-token: $LAMDIS_API_TOKEN" \
            -H "Content-Type: application/json" \
            -d '{"file": "suites/regression.json"}'
        env:
          LAMDIS_API_TOKEN: ${{ secrets.LAMDIS_API_TOKEN }}

GitLab CI Example


stages:
  - test
 
lamdis:
  stage: test
  image: node:20
  variables:
    LAMDIS_API_TOKEN: $LAMDIS_API_TOKEN
    OPENAI_API_KEY: $OPENAI_API_KEY
  script:
    - cd lamdis-runs
    - npm install
    - npm run dev &
    - sleep 5
    - export LAMDIS_RUNS_URL="http://127.0.0.1:3101"
    - npm run run-file -- suites/my-suite.json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Azure DevOps Example


trigger:
  - main
 
pool:
  vmImage: 'ubuntu-latest'
 
steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '20.x'
  
  - script: |
      cd lamdis-runs
      npm install
      npm run dev &
      sleep 5
    displayName: 'Start lamdis-runs'
    env:
      LAMDIS_API_TOKEN: $(LAMDIS_API_TOKEN)
      OPENAI_API_KEY: $(OPENAI_API_KEY)
  
  - script: |
      cd lamdis-runs
      npm run run-file -- suites/regression.json
    displayName: 'Run Lamdis tests'
    env:
      LAMDIS_API_TOKEN: $(LAMDIS_API_TOKEN)
      LAMDIS_RUNS_URL: http://127.0.0.1:3101

Environment Variables for CI

Variable	Required	Description
`LAMDIS_API_TOKEN`	✅	Token to authenticate with lamdis-runs
`LAMDIS_API_KEY`	For hosted	API key from Lamdis dashboard
`OPENAI_API_KEY`	For OpenAI judge	OpenAI API key
`AWS_ACCESS_KEY_ID`	For Bedrock	AWS credentials
`AWS_SECRET_ACCESS_KEY`	For Bedrock	AWS credentials
`AWS_REGION`	For Bedrock	AWS region (default: `us-east-1`)

Best Practices

1. Use Dedicated Test Suites

Create separate suites for CI vs comprehensive testing:


{
  "id": "ci-quick",
  "description": "Fast CI checks (P0 tests only)",
  "tests": {
    "includeIds": ["critical-safety-check", "basic-functionality"]
  }
}

2. Set Appropriate Timeouts

Configure reasonable timeouts to avoid hanging CI jobs:


- name: Run tests
  timeout-minutes: 15
  run: npm run run-file -- suites/ci-quick.json

3. Cache Dependencies

Speed up CI runs by caching npm dependencies:


- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}

4. Use Matrix Testing

Test against multiple assistant versions:


strategy:
  matrix:
    assistant: [v1, v2, canary]
steps:
  - run: npm run run-file -- suites/${{ matrix.assistant }}.json

5. Fail Fast on Critical Tests

Structure your workflow to run critical tests first:


jobs:
  critical:
    runs-on: ubuntu-latest
    steps:
      - run: npm run run-file -- suites/critical.json
  
  comprehensive:
    needs: critical
    runs-on: ubuntu-latest
    steps:
      - run: npm run run-file -- suites/full-regression.json

Security Considerations

Rotate keys regularly — Revoke and recreate API keys periodically
Use minimal scopes — Only grant runs:trigger and runs:read for CI
Protect secrets — Never log or expose API keys in CI output
Network isolation — For self-hosted, consider running in isolated networks

Troubleshooting

Tests timing out

Increase maxTurns in test definitions
Check if the assistant endpoint is reachable from CI
Verify network/firewall rules allow outbound connections

Authentication failures

Verify LAMDIS_API_TOKEN or LAMDIS_API_KEY is set correctly
Check token hasn’t expired
Ensure correct scopes are granted

Judge failures

Verify OPENAI_API_KEY is set and valid
For Bedrock, check AWS credentials and region
Review judge model configuration

Next Steps

Test Runner — Understand the execution engine
Test Steps — Configure test steps
Requests & Auth — Set up API authentication