Benchmark contrast and calibration
Researchers at Amazon recently detailed a method for optimizing Mixture of Experts (MoE) models, termed Higher-Order Pruning of Experts (HOPE). While the underlying research was detailed in an arXiv v3 paper dated Sept. 29, 2026, the team published a first-party Hugging Face article on Nov. 9, 2026, providing further context on the approach. In experiments using the Qwen3.5-35B-A3B model at a 50% pruning rate, the researchers observed a performance contrast based on the calibration data employed. The study utilized two distinct domains: Evol-CodeAlpaca-v1, which is a coding-focused calibration dataset, and SWE-Bench Verified, which serves as an agentic trajectory calibration dataset. When calibrated with Evol-CodeAlpaca, the model achieved a score of 42.33 ±1.23 on the SWE-Bench Pro benchmark. When calibrated using SWE-Bench Verified trajectories, the score was 40.78 ±0.86. For comparison, the unpruned baseline for this model was 41.82. These figures represent scores multiplied by 100, derived as means and standard deviations from three independent calibration trials. In this tested case, one calibration set produced a mean above the baseline, while the other produced a mean below it.
Mechanism: From first-order to HOPE
Calibration is the process of passing a specific dataset through a model to collect activation statistics, which are then used to determine which experts to remove. In ordinary first-order pruning, a method assigns each expert an individual score based on its contribution and removes those with the lowest values. HOPE, by contrast, records the co-activation of expert pairs, along with their gate weights and output norms, in an interaction matrix. Its quadratic-program selection step utilizes both the individual contributions found on the matrix’s diagonal and the pairwise contributions located off the diagonal. The premise is that an expert that appears expendable when evaluated in isolation may contribute to a cooperative group when experts work together. This makes the calibration set a fundamental component of the pruning decision, as it supplies the observed activations from which those individual and pairwise statistics are calculated.
Calibration stability and prune-set similarity
To understand the consistency of these pruning decisions, the authors analyzed the stability of the selected prune sets. The paper reports that same-calibration trial stability was greater than 0.95 Jaccard similarity on average across three random trials. When comparing pruning decisions across different calibration domains, the mean Jaccard similarity was 0.60 for HOPE and 0.61 for REAP, a first-order baseline. These metrics measure the similarity of the selected prune sets—specifically, which experts are removed—rather than model performance. The reported values show greater similarity between repeat trials using the same calibration domain than between prune sets produced from different domains. These Jaccard metrics describe the consistency of the selection process within the scope of the reported experiments and do not establish broader architectural behavior or performance outcomes.
Study scope and limits
Across the authors’ tested conditions, the study evaluates Qwen3.5-122B-A10B, Qwen3.5-35B-A3B, and GLM-4.5-Air, using two calibration sets and multiple benchmarks, including Tulu3 and SWE-Bench Pro. The paper describes 54 configurations across these three models, the two calibration sets, and the tested pruning rates for Tulu3 and SWE-Bench Pro. The authors state that HOPE’s advantage is most pronounced at 40–50% pruning, while first-order methods can be competitive at 10–20%. Furthermore, their evaluated method prunes each layer independently rather than accounting for cross-layer interactions, and it models pairwise interactions rather than higher-order combinations. These figures are the authors’ own experiments, and no independent replication has been established by these sources. The result highlights a question for researchers comparing pruning methods on agentic-coding benchmarks: what data was used to decide which experts to remove, and does the evaluation reflect a distinct task domain? This study does not establish which calibration set generalizes best to other models or workloads, nor does it predict how a pruned model will behave in a production agent. HOPE makes the choice of calibration visible in the experiment, but it does not remove the need to test the specific model and workload that a result is intended to describe.