CI setup

Running flaky-tests in CI gives you a shared failure history across every pull request and push, which makes pattern detection much more reliable than local-only data.

Prerequisites

You need a shared store — SQLite is local-only and won’t work for CI. Use Turso, Supabase, or Postgres.

Basic setup

GitHub Actions

.github/workflows/ci.yml

name: CI
on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: oven-sh/setup-bun@v2
        with:
          bun-version: latest

      - run: bun install

      # Capture failures during test run
      - run: bun test
        env:
          FLAKY_TESTS_STORE: turso
          FLAKY_TESTS_CONNECTION_STRING: ${{ secrets.TURSO_URL }}
          FLAKY_TESTS_AUTH_TOKEN: ${{ secrets.TURSO_AUTH_TOKEN }}

      # On main: check for new patterns and open issues if found
      - if: github.ref == 'refs/heads/main'
        uses: brewpirate/flaky-tests@v1
        with:
          store: turso
          connection-string: ${{ secrets.TURSO_URL }}
          auth-token: ${{ secrets.TURSO_AUTH_TOKEN }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          create-issues: 'true'

Vitest

.github/workflows/ci.yml

name: CI
on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24

      - run: npm ci

      - run: npx vitest run
        env:
          FLAKY_TESTS_STORE: turso
          FLAKY_TESTS_CONNECTION_STRING: ${{ secrets.TURSO_URL }}
          FLAKY_TESTS_AUTH_TOKEN: ${{ secrets.TURSO_AUTH_TOKEN }}

      - if: github.ref == 'refs/heads/main'
        uses: brewpirate/flaky-tests@v1
        with:
          store: turso
          connection-string: ${{ secrets.TURSO_URL }}
          auth-token: ${{ secrets.TURSO_AUTH_TOKEN }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          create-issues: 'true'

Setting up secrets

1. Create a Turso database (or Supabase/Postgres project)
2. Go to your GitHub repo → Settings → Secrets and variables → Actions
3. Add the required secrets:
   • TURSO_URL — your Turso database URL (libsql://...)
   • TURSO_AUTH_TOKEN — your Turso auth token

What runs when

Event	Captures failures	Checks for patterns	Opens issues
Pull request	Yes	No	No
Push to main	Yes	Yes (if create-issues: true)	Yes (if new patterns)
Scheduled (weekly)	No	Yes	Yes (if new patterns)

Tip

The recommended pattern is: capture on every run (PR + main), detect on main pushes and scheduled runs. This maximizes the data you collect while keeping noise low.

Reliability notes

• Writes are drained at process exit. The Bun preload tracks every fire-and-forget write and awaits outstanding ones before the test process shuts down, so transient remote-store latency at the end of a run doesn’t drop failures.
• Remote stores retry transient failures. Turso, Supabase, and Postgres adapters retry network errors with exponential backoff out of the box — flaky CI networking rarely needs special handling on your side.
• Migrations run automatically. Plugins and the CLI both call migrate() before the first query, so a fresh database works on the first run with no setup step.

Capture-only mode

If you want to collect data without running detection yet:

- run: bun test
  env:
    FLAKY_TESTS_STORE: turso
    TURSO_URL: ${{ secrets.TURSO_URL }}
    TURSO_AUTH_TOKEN: ${{ secrets.TURSO_AUTH_TOKEN }}
# No detection step — just accumulate data

Run in capture-only mode for a week or two to build up baseline data before enabling detection.