PRODUCT CASE
Getsit AI
A model-fit workbench for cases where the same phrase becomes different work across models.
Inspect now
When “Organize This” Became Different Work
I gave the same lecture transcript to ChatGPT and Gemini with the same request:
Organize this lecture clearly without compressing the content.
The outputs differed in direction, not just wording. One model treated filtering information down to its essentials as part of “organize”; the other preserved more of the source's scope and sequence. The same term had become two operational definitions.
When the outputs were anonymized and the two models were asked which better followed the request, each preferred the style closer to its own. That led to a product hypothesis: models may assign different execution priorities and quality criteria to the same everyday language.
Change the Question From Ranking to Fit
Instead of asking:
Which model is best?
Getsit asks:
Which model best executes what this user cares about in this task?
It does not add another general benchmark. It turns an ambiguous request into user-confirmed criteria, runs the same input across models, and connects observed behavior to model choice and the next prompt.
The User Flow
- Enter the real request and source material.
- Clarify required outcomes, constraints, and unwanted behavior.
- Run the same input across selected models.
- Compare what each output preserved, compressed, restructured, or added.
- Review quality, latency, cost, evidence, and limitations together.
- Generate a model-specific prompt candidate for the next run.
Responsibility Split
| Owner | Responsibility |
|---|---|
| User | Confirm the desired output and priorities |
| Execution layer | Record equal inputs, model state, usage, latency, and cost |
| Evaluator model | Produce criterion-level differences, evidence, and a recommendation candidate |
| Product rules | Preserve scores with supporting excerpts, confidence, and limitations |
| Person | Judge the actual outputs and trade-offs rather than accepting the score as truth |
Problems Solved While Building
Intent Rubric: Fix the User's Goal Before Producing Scores
- Observed problem: An evaluation of “organize this well” can be precise while measuring the wrong thing.
- Reasoning: The user needed to confirm success criteria, required behavior, forbidden behavior, and format before comparison.
- Implementation: Questions turned the request into five criteria: sequence preservation, detail preservation, selective cleanup, navigable structure, and fidelity to the source.
- What changed: Results could be discussed against the intended task rather than model preference.
Behavior Over Self-Description: Separate What Models Say From What They Do
- Observed problem: A model may say it preserves examples while omitting them in the actual output.
- Reasoning: Model explanations are useful clues, but execution on the same input is the stronger signal.
- Implementation: Getsit records model explanations for key phrases alongside the actual output segments that preserve, omit, or expand material.
- What changed: The product can explain which language cue led to which output behavior.
Evaluator Humility: Do Not Turn the Judge Into Ground Truth
- Observed problem: LLM-as-judge scores inherit the evaluator's own assumptions and bias.
- Reasoning: A useful report needs evidence, confidence, excerpts, and trade-offs, not one authoritative ranking.
- Implementation: Criterion scores, supporting text, recommendation, confidence, latency, cost, and limitations stay together.
- What changed: A user can choose a lower-scoring model when speed, cost, or a concise overview matters more.
One Actual Product Run
- Source: Korean technical lecture transcript, approximately 21 minutes and 10,277 characters
- Request: organize the lecture clearly without compressing it
- Models: Claude Sonnet 4.6, GPT-5.5, Gemini 3.5 Flash
- Status: all three provider calls completed
| Model | Observed fit for this intent | Evaluator score | Latency · estimated cost |
|---|---|---|---|
| GPT-5.5 | Preserved the most detail, with some expansion beyond the source | 92 | 174.5 sec · ~$0.108 |
| Gemini 3.5 Flash | Clear high-level structure, but compressed substantial detail and context | 52 | 21.0 sec · ~$0.021 |
| Claude Sonnet 4.6 | Readable overview, but closer to a summary than replacement notes | 50 | 64.6 sec · ~$0.091 |
These are Getsit's evaluator scores for one input and one user-confirmed intent. They are not overall model rankings. A fast overview or lower cost could make another model the better choice.