Guide · Model and provider selection

How to compare LLM providers for one workload: a checklist

Benchmarks and price sheets compare models in general. A team choosing a provider needs to compare them on its own workload, with its own definition of success, and keep the evidence that justified the choice. This checklist is the method AIYVO’s offline analyzer follows.

1. Fix the workload and the success check first

Write down what one unit of work is and how you decide it succeeded, before looking at any results. A comparison where the success rule is chosen afterwards can be bent to favour any model.

2. Count every attempt

Include failed requests, retries and fallbacks in cost. A provider that is cheap per request but needs three attempts per result is not cheap. See why the cheapest LLM per request can cost more per result.

3. Decide the unit: request or task

If a job involves several model calls, compare cost per completed task, not per request. Your usage export decides what is possible: one row per request supports request-level metrics; task identifiers are needed for task-level ones.

4. Require a minimum sample and report what is missing

Set a minimum number of records per model before ranking it; AIYVO’s analyzer uses 30. Report records with missing cost or outcome separately instead of dropping them silently, and never treat a missing cost as zero.

5. Keep price and quality apart

Compare cost per successful result, success rate, latency and any quality score as separate columns. A single blended score hides which trade-off you are making.

6. Record the price table and the date

List prices change. Keep the price table used for each comparison, with its date, so a result can be reproduced later and compared with what the provider actually billed.

7. Keep evidence you can check later

A decision record should let someone else reproduce it: the input file, the price table, the rules applied and the resulting report, each with a content fingerprint. AIYVO’s analyzer prints SHA-256 fingerprints for its inputs and outputs for this reason.

What AIYVO does today, and what is planned

Today, AIYVO is an offline analyzer: it reads a usage export you already have, on your machine, and reports cost per successful request or per completed task by model, with the evidence behind each figure. It does not call providers and does not verify provider billing. Routing work between providers, spend governance and authorization are planned, not available; the planned-platform simulation shows how they are intended to work, on synthetic data.