eval_752
  • English

      • Managing Providers

      The Providers page is where you tell eval_752 which LLM endpoints to call.

      The workflow is simple: add a provider, run a smoke test, and only then move on to runs. If the smoke test doesn't pass, nothing else will work either.

      What a provider stores

      Each provider record contains:

      FieldWhat it does
      NameA label you'll see throughout the UI (keep it short and recognizable)
      Provider typeOpenAI, Anthropic, Google, or Custom
      Base URLThe LiteLLM-compatible endpoint root
      CapabilitiesFlags: system prompt, JSON mode, reasoning, image support
      Rate limitsOptional hints: RPM, RPS, concurrency
      API keysEncrypted at rest with AES-GCM

      Adding a provider

      1. Open Providers
      2. Fill in the name, type, and base URL
      3. Add at least one API key
      4. Optionally adjust capabilities and rate limits
      5. Click Connect provider

      For a first setup, keep it simple — one provider, one model, one smoke test.

      Local model servers in Docker

      If eval_752 runs in Docker and your model server runs on the host (LM Studio, Ollama, vLLM), use host.docker.internal instead of localhost in the base URL. Example: http://host.docker.internal:1234/v1

      Saving frequently used models

      Each provider card has a saved-model manager where you can store:

      • The upstream model name (the exact string sent to the API)
      • An optional human-friendly alias
      • Optional metadata for routing notes

      Saved models appear as suggestions in the Runs launch form and on the Schedules page — useful when model names are long or easy to mistype.

      Editing and rotating credentials

      Click Edit on any provider card to:

      • Rename the provider
      • Change the base URL or type
      • Update capabilities
      • Rotate API keys

      Smoke testing

      The smoke test sends a small request to verify that the base URL and API key work.

      1. Select a provider
      2. Enter the exact model name
      3. Click Run smoke test

      The result shows: success/failure, latency, response preview, and usage telemetry (when available).

      This is the fastest way to validate a connection before committing to a full evaluation run.

      Tips

      • Name providers clearly. You'll see these names in Runs and Comparison — "OpenAI Prod" is better than "Provider 1."
      • Smoke test after every change. Changed the base URL or rotated a key? Smoke test again.
      • Remove trailing slashes from base URLs. The backend normalizes them, but cleaner input avoids surprises.
      • Start with conservative rate limits, then raise them once the provider proves stable.
      • If smoke test breaks after changing ENCRYPTION_KEY: Your stored secrets are encrypted with the old key. Reset Docker volumes (docker compose down -v) and re-add the provider. See Troubleshooting.