GP-SOP-010 | Workflow Quality
Compare Local and Cloud AI Models for a Task
A task-specific model choice based on quality, privacy, speed, cost, and reliability.
When to use this SOP
Use this before standardizing a workflow around a model or promising that a local option will match a stronger cloud model.
What you need
- One stable prompt
- Five to ten representative test cases
- A scoring rubric
- Candidate local and cloud models
The procedure
Follow these steps
- 01
Define the task and the minimum acceptable result before selecting models.
- 02
Create representative easy, typical, difficult, and failure-prone test cases.
- 03
Use the same prompt, context, settings, and output format for every candidate.
- 04
Run each test more than once when consistency matters.
- 05
Score accuracy, instruction following, useful detail, style, latency, and correction effort.
- 06
Record privacy constraints, hardware load, API cost, rate limits, and operational dependencies.
- 07
Review failures, not just averages. One dangerous failure can outweigh several polished outputs.
- 08
Choose the smallest model that reliably meets the real requirement and document the fallback.
Human checkpoint
Stop and review before continuing
The best model is the one that meets the task requirement within your constraints, not the model with the most impressive general benchmark.
Definition of done
- The task and rubric were defined first
- Every model received the same test
- Quality and correction effort were scored
- Cost, privacy, and latency were recorded
- Failures and fallback behavior are documented
When the process gets stuck
If results vary too much to compare, tighten the prompt, expand the test set, and separate subjective style from factual performance.
Where automation fits
A test harness can send the same cases to several models and collect results. Final scoring should include human review.