Between October 15 and October 21, four shipments landed on the open-weight stack that deployers had been waiting on. StepFun’s Step-5-Preview-BF16 weights arrived October 15. NVIDIA shipped Nemotron 3.5 Super VL the same day, with open weights, open datasets, and training recipes. On October 20, DeepSeek opened its production inference engine, and llm-d 1.0 shipped the vendor-neutral serving layer the same day. Licenses moved, engines moved, serving layers moved. One layer did not: nothing shipped that would let anyone outside these companies check what the other four claim.
The claims moved at the seller’s speed. NVIDIA’s release page says the new model “delivers the fastest task completion and highest accuracy for specialized always-on agents,” and the company’s own comparison runs to its largest reasoning model at roughly one-fifth the parameter count. Both are vendor-reported. The DeepInfer repository publishes no numeric engine speedups, and no independent benchmarks of the engine exist yet; every performance figure attached to that release is vendor-reported. The serving layer reads the same way: the adoption figures in the llm-d announcement, from cache-hit gains at Google Cloud to IBM’s cost claims, are vendor-reported. Four shipments, one evidence class: the seller’s own word.
One document this week did tell the truth about risk, and it is the document written to survive a courtroom. StepFun’s Model License 1.0 provides the model “AS IS,” and its disclaimer names what that means: no warranties “REGARDING THE ACCURACY, RELIABILITY, OR QUALITY OF ANY OUTPUT, AND YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE MODEL OR ANY DERIVATIVE WORK.” The license disclaims exactly the qualities the release pages advertise, and it assigns the resulting risk to the deployer in writing. The marketing sells the capability. The contract gives it back.
The same license shows how far the terms reach past the download. A hardware maker embedding the model in phones, vehicles, smart glasses, wearables, robots, or smart-home devices needs a separate commercial license before any revenue question arises. A service operator needs one when combined group revenue crosses $20 million over any consecutive twelve months. Derivative works include models trained on the outputs of the original. The same class of gate sits in four labs’ terms on four different clocks, as the four-gate analysis read them in full: the gates are stops, not tolls, and the agreement you must sign stays private until you are in the room. For a deployer, what a license permits at their own scale cannot be answered from outside it.
The market has one independent reference point, and it is worth being precise about what it does. The Artificial Analysis Intelligence Index is a composite of ten evaluations, and the firm states that all of them are conducted independently by Artificial Analysis. Its benchmarking methodology covers models, inference endpoints, and systems, and the prices it lists are the prices providers list. The same page declines the open-source question in its own definition: open weights means “a model whose weights have been released publicly by the model’s creator,” and the firm avoids “open-source” because “many open LLMs have been released with licenses that do not meet the full definition of open-source software.” Its own guidance adds that individual benchmark results may be more relevant for specific use cases than the composite. One of the ten evaluations is described as a private evaluation. What none of this does is attest what a lab shipped, adjudicate what a license permits at your scale, or audit the protocol behind a vendor’s benchmark. That is not a failure. Nobody hired it to do that job.
Which leaves the deployer holding three checks and no instrument for any of them. Provenance first: what the weights are and where they came from. The record is uneven in the open. NVIDIA shipped datasets and training recipes beside the weights, more than the other acts disclosed, and the DeepInfer repository carries a lineage acknowledgment naming vLLM as its primary base. No independent attestation of any of the shipped artifacts appears on the release surfaces read for this piece, and where the evidence is thin the check shrinks to what survives. License reach second: the terms above are the evidence, and they bind at sizes the deployer cannot see in advance. Benchmark methodology third: every capability and cost figure attached to this week’s acts arrived vendor-reported, and the independent aggregate that exists measures a different thing than the claims being sold.
The week’s other coverage sets the boundary of this one. Model Moats mapped what shipped and where the value went, the shift from weights to watts. The index-compression analysis mapped what a composite score does and does not mean for a procurement decision. This piece is about what did not ship. Verification is expensive, it slows a release, and it exposes whoever funds it to findings they would rather not publish. Every incentive in the week points away from it, and the one license that speaks plainly names who absorbs the outcome: the deployer, in writing, whether or not anyone checked. The gap is not a delay. It is a choice with a designated payer.