01CSTM-SW
Evaluation sets
Real examples with expected answers, versioned like code.
CSTM-SW · STDcstmAI™ · Software
AI model evaluation and fine-tuning on your own hardware.
cstmAI™ Studio is the workbench for deciding which model to trust with a task, and for making it better at that task: evaluation sets, side-by-side comparisons, and fine-tuning runs that never leave your systems.
Fig. 01 · Welding a transporter bracket, NASA Kennedy
Every claim about a model should come with a score on your own work. Studio holds the evaluation sets (real inputs with expected outputs, written with your experts) and runs candidate models, prompts and fine-tunes against them, so a change ships only when the numbers support it.
When a better prompt isn't enough, Studio fine-tunes open-weight models on your GPUs, often overnight on the same system that serves by day, and keeps the resulting weights in your storage.
| Evaluation | Task test sets with automatic and human-graded scoring |
|---|---|
| Comparisons | Models, prompts and versions side by side |
| Fine-tuning | Adapter methods; full fine-tuning where hardware allows |
| Data | Stays on your systems throughout |
| Outputs | Versioned weights and score reports that you own |
Where it sits
Its place in the stack.
Every module runs on the same platform and hardware, under the same governance.
L4 · Modules
L3 · Control
In the workbench
Numbers first, then changes.
02CSTM-SW
Bake-offs
Candidate models scored on the same set, with cost and speed beside quality.
03CSTM-SW
Adapters
Lightweight fine-tunes that teach a model your formats and vocabulary.
04CSTM-SW
Regression checks
Every new version re-scored before it replaces the old one.
Reel 02 · Lathe area of a machine shop, Paterson, NJ1994
How it's delivered
Four steps, each one signed off.
01
Collect examples
Real inputs and the outputs your experts would accept.
02
Baseline
Score current models and prompts to see the gap.
03
Tune
Prompt changes first, fine-tuning if the gap remains.
04
Promote
Ship the new version only if it wins on the set.
FAQ
Questions we hear first.
Do we need to fine-tune at all?
Often not. Retrieval and good prompts handle most tasks. We fine-tune when the evaluation set shows a gap prompts can't close: a strict house format, specialized vocabulary, or a small model that must match a large one.
Who writes the evaluation sets?
Your domain experts and our engineers, together, from real examples. It is the most valuable artifact of the project, and it belongs to you.
Can it train on the same hardware we serve from?
Yes, scheduled around serving load. Large full fine-tunes may call for a Cluster.
Get a quote
Spec your system.
Tell us the models you want to run, how many people will use them and where the hardware should live. An engineer replies with a first configuration and the questions that decide the quote.