fastevals

semenovdv/fastevals

★ 0 stars Python AI/LLM Updated today

Give your AI agents an evaluation tool - provider-agnostic LLM eval runner with an MCP server for Claude, structured output validation, datasets, evaluators and honest cost metrics

View on GitHub → 🔍 Audit Wallet Slippage →

Quick Install

Copy the config for your editor. Some servers may need additional setup — check the README.

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "fastevals": {
      "command": "uvx",
      "args": [
        "fastevals"
      ]
    }
  }
}

Or install with pip: pip install fastevals

README Excerpt

**Evaluation tooling your AI agents can drive.** fastevals is a small, provider-agnostic evaluation runner for LLM applications. Run one prompt — or a whole dataset — across a matrix of models, reasoning efforts and providers, save every response, and get a readable standalone HTML comparison report with cost, latency and token

Tools (3)

get_runlist_modelsrun_evaluation

Topics

benchmarkevaluationlitellmllmmcpmcp-serveropenaipython

fastevals

Quick Install

README Excerpt

Tools (3)

Topics

Related AI/LLM Servers