eval_752
  • English

      • Testing Guide

      Updated: 2026-03-13 · Owner: QA Guild

      This page summarizes the testing matrix for the Python + React stack.

      :::tip Related appendices

      • docs/testing/docker-integration-tests.md
      • docs/testing/lighteval-interoperability-fixtures.md
      • docs/testing/docs-smoke-checklist.md
      • docs/testing/core-alpha-signoff.md
      • docs/testing/browser-harness-signoff.md
      • docs/testing/active-runs-hardening-signoff.md :::

      Coverage & Gatekeeping

      LayerToolingRequired CoverageCI Gate
      Backenduv run pytest --cov100% branch-aware coverageci.yml backend job
      Frontendpnpm test --coverage>=80% lines on core journeysci.yml frontend job
      Integration (Docker)scripts/tests/run_docker_integration.sh --full-run --with-beatregression-focusednightly + local
      Playwright E2Epnpm test:e2e + pnpm test:e2e-smokescenario-basedfrontend + smoke jobs

      Quick Commands

      cd backend
uv run pytest --cov --cov-report=term-missing
uv run pytest -m lighteval
uv run pytest -m integration
uv run mutmut run

cd frontend
pnpm lint
pnpm typecheck
pnpm test
pnpm test:e2e
pnpm test:e2e-smoke

      Backend Testing Details

      1. Unit tests live under backend/tests.
      2. pytest -m lighteval validates LightEval config generation plus CLI/Python interoperability.
      3. pytest -m integration exercises the Docker-backed stack.
      4. The backend coverage gate remains branch-aware and enforced from backend/pyproject.toml.

      Test doubles

      • Local unit tests may still use mocks and fakes.
      • Docker integration and LightEval contract suites use a local OpenAI-compatible test gateway.
      • Production code always follows the standard LiteLLM client path.

      Docker-backed Integration Flow

      Run from the repository root:

      scripts/tests/run_docker_integration.sh --full-run --fresh [--with-beat]

      What it does:

      1. Starts postgres, redis, the local OpenAI-compatible gateway, celery-worker, and optional celery-beat
      2. Applies migrations and seeds deterministic data with eval_752.scripts.seed_integration
      3. Runs uv run pytest -m integration inside the backend container
      4. Collects coverage and logs under .artifacts/docker-integration/

      Frontend Testing Details

      • Vitest + React Testing Library cover unit and component behavior
      • Playwright covers browser journeys and Docker smoke
      • Active runs UI evidence is collected by frontend/tests/manual/active-runs-ui-audit.mjs

      Documentation Smoke

      Before shipping workflow changes, run the documentation smoke checklist in docs/testing/docs-smoke-checklist.md. This is the guardrail against getting-started, provider, results, i18n, and active-runs docs drifting away from the product.