Three Sample Lenses on One Bench: The Trial Protocol That Holds

Three sample lenses on one bench can produce a clean ranking and still leave you unable to defend the purchase order six months later. The gap is not statistics, it is scope: a sample trial measures three individual units under one set of conditions, while the decision it is asked to support is about a production stream. Agree the protocol before the first box is opened, and make it answer two separate questions. Which lens performs best on this station today, and what evidence will make the winner reproducible in a batch. The first question needs blinding, rotation and fixed conditions. The second needs records from the maker, a second unit drawn without your influence, and acceptance lines that survive into the purchase order. Get both, and the trial stops being an argument about whose day was better.

If you are comparing suppliers for this kind of optic, the Lensmind machine vision lens catalog is the hub for our standard models and custom options.

What a Sample Trial Proves, and What It Only Appears to Prove

Most sample trials are honest experiments with an over-generous conclusion attached. The measurements are real; the sentence written after them usually is not. The table below is the one to put in front of the team before the trial starts, because it names the claim each piece of evidence can carry and the claim it cannot.

Claim the trial seems to make	What the evidence actually covers	What the purchase order needs instead
This lens is good	One unit, built and handled by the maker, measured once on your station	A batch that repeats the result across the run, with per-batch test documentation
This lens beats the other two	A ranking produced under today’s conditions and today’s parts	Acceptance limits you can test at goods-in without repeating the whole trial
This lens fits our recipe	One operator, one shift, one fixture, one mounting sequence	Repeatability across operators and days, stated as a tolerance rather than a result
This maker can deliver it	The sample team, which is rarely the production team	Batch records, inspection coverage, and a named change-notification path
We decided on data	Notes in a lab book and a folder of images	A signed record with frozen conditions, so the decision can be re-examined later

Two questions the trial must answer separately

The first question is preference: of these three units, which suits the station. The second is capability: can the maker repeat that performance in parts you will actually receive. A trial that mixes them produces the familiar failure mode, where the winning sample is beautiful and the first production batch is merely acceptable, and nobody can prove which step changed. Keep two columns in the trial report, one for ranking and one for capability evidence, and refuse to let a high ranking fill the capability column. Measurement system discipline is the relevant reference here rather than lens folklore: the NIST/SEMATECH handbook treatment of gauge repeatability and reproducibility sets out why a single operator on a single day cannot separate the instrument from the part.

Why the losing lenses need to be documented too

A trial report that only describes the winner is a sales document. The two rejected units carry the reason the decision holds: which requirement each one missed, by how much, and whether the miss was a unit defect or a design limit. That distinction decides whether you re-order the loser for a second look or close the door. Record it while the units are still on the bench, because eighteen months later the only person who remembers why lens C was dropped will have changed departments, and the answer will be a rumour.

What to Agree Before Anything Is Unpacked

The sequence below is worth more than any single measurement, because it removes the two things that make sample comparisons indefensible: unblinded identity and unrecorded conditions.

1. Code the units off site from the scoring. Someone outside the test assigns neutral codes and keeps a sealed key. The operator who mounts and measures the lens should not be able to infer the maker from a serial number, a barrel engraving, or the order the boxes were opened.
2. Freeze the station in writing. Camera, sensor, lens mount, fixture, lighting, exposure, gain, aperture, working distance, part presentation, focus procedure, ambient temperature, and software version. Photograph the setup and file the images with the report; a condition list without a photograph invites later disagreement about what “same setup” meant.
3. Pick the parts before the units. Use a coded sample of known-good parts, known-bad parts covering each defect type you care about, and borderline parts near the limit. A trial run on convenient good parts ranks lenses on an easy problem and tells you nothing about the shift where yield matters.
4. Rotate the order and repeat it. Each unit should occupy the first, second and third slot roughly equally often, and the sequence should be repeated on a second day or shift. If time allows only one pass, at minimum randomise which unit goes first, so the fresh-equipment slot is not a permanent advantage.
5. Pre-register the decision rule. Mandatory gates first, then the scored criteria with weights, then the tie-break rule, all written before any image is examined. If two units are statistically indistinguishable, the pre-agreed tie-break decides, not the preference of whoever has to sign the report.
6. Define what invalidates a run. Equipment fault, lost trigger, damaged part, environmental excursion. Anything else that goes wrong stays in the data as part of the result, including a unit that needs refocusing mid-run.

The Sample Is Not the Part You Will Buy

This is where optics trials differ from most component comparisons. A sample may be a hand-built unit, a selected unit, or a unit from a pre-production run, and the maker may not say which. Ask directly, and write the answer into the report. A sample that cannot be traced to a build record is a demonstration, not evidence.

Ask for the sample’s own batch record

The cheapest truthfulness check available is to compare your measurement of the unit against the maker’s own report for the same unit. Ask for the test documentation that ships with the batch: the agreed parameters, batch number, test conditions, inspection coverage, and an explicit pass or fail judgement. For a machine vision lens family that typically means MTF, distortion, BFL and depth of field, measured at named image heights rather than described in adjectives. If the maker’s numbers and yours disagree beyond the stated uncertainty, you have found something more useful than a ranking, and you have found it before the order. The same discipline that makes quotes comparable, set out in what a lens quotation must contain, applies to the sample file.

Add a second unit drawn blind

Write one clause into the protocol: after the winner is chosen, the maker supplies one further unit from the same production lot, drawn without your selection, and it must meet the same gates on the same station. This single step converts a sample result into a first, weak statement about capability, and it costs one unit. If the second unit lands inside the gates, you have evidence of repeatability. If it does not, you have learned the difference between a sample and a product while the order is still small, which is the cheapest possible moment to learn it. Where a program later moves from prototype to volume, the discipline of freezing the drawing at first article covers the next stage of the same question.

Carrying the Trial Into the Purchase Order

The trial ends with a decision; the purchase order is where that decision either becomes enforceable or evaporates. Four pieces of paperwork do the work.

1. Turn the frozen conditions into an acceptance condition table. The parameter list, measurement method, image height, and pass limit that were used in the trial become the goods-in criteria. If a criterion cannot be tested at goods-in, name the test that will be run on a sample per lot and who runs it.
2. Name the parameters that decide, not every parameter. A twelve-line acceptance table nobody tests is worse than a four-line table that is enforced. Pick the few that predict station performance: resolution at the image heights you use, distortion where measurement accuracy matters, back focal length for mounting, and field coverage against the actual sensor.
3. Attach the tie-break and the invalidation rule. They look like internal documents, but they belong in the supply file, because they explain what the buyer will do when a lot sits at the edge of the limit rather than in the middle.
4. Keep the losing units’ data with the order file. When a future batch is questioned, the comparison set from the trial is the reference that shows what passing looked like on your own bench.

For a second-sourced part, this is the same package that makes two suppliers comparable, as described in the documents that make two lens quotes comparable. A trial protocol that produces this package has done its job twice: it ranked today, and it armed the specification for tomorrow.

Conclusion

Agree the protocol before unpacking, and keep it narrow enough to be followed. Code the units, freeze the station in writing, rotate and repeat, pre-register the gates and the tie-break, and log what invalidates a run. Then add the two steps most trials skip: request the sample’s own batch record so your measurement can be checked against the maker’s, and require one blind second unit to show that the winner is repeatable rather than lucky. Finish by converting the frozen conditions into an acceptance table that travels with the purchase order. Do that, and the argument at the end of the trial is about a number, not about whose test day felt better, which is the whole point of putting a protocol in front of three boxes.