How a model leaderboard puts mixed benchmarks on one scale
AIHOT documents how its model leaderboard turns heterogeneous benchmarks into one capability scale. Each pass-rate benchmark is fitted as a logistic ruler with a difficulty and a steepness estimated from the models that took it; a raw score is inverted into x=(logit(pass rate)+e)/A. Pass rates above 95% or below 5% are truncated to a floor or ceiling, same-family benchmarks count once, and any calibration more than two noise units from the rest has its pull on the total capped. Domain scores shrink toward the model's overall level, then combine at 38% coding, 19% research, 19% office work, 14% knowledge, 10% human blind preference, displayed as 50+15z and anchored to 123 frozen reference models. It also states the ranking thresholds, suspension rules and missing-data handling.
EVIDENCE BEFORE CONCLUSIONS Treat every benchmark as a ruler.
Different benchmarks use different units and have different difficulty levels. If you simply average them, a model that has only taken a few easy benchmarks comes out ahead. AIHOT first estimates the difficulty of each benchmark, then converts every result into the same capability score: hard and easy benchmarks now speak the same unit, and missing benchmarks neither add nor subtract points.
EVIDENCE BEFORE CONCLUSIONS把每项评测当成一把尺子。
不同评测的分数单位不同、难度不同,直接平均的话,只测了几项简单评测的模型反而占便宜。AIHOT 先给每项评测定出难度,再把每个成绩折算成同一种能力分:难的评测和容易的评测说同一种单位,没测的项不补分也不扣分。
50 is the average level of the reference models, and every 15 points is roughly one standard deviation. When two models differ by more than 3 points, their order almost never flips. The score is not answer accuracy. The overall leaderboard and the four category leaderboards show at most 50 models each.
50 分是参照模型的平均水平,每 15 分约是一个标准差;两个模型相差 3 分以上,先后基本不会变。评分不是答题正确率。综合榜与四个分类榜最多展示 50 个模型。
01 Only evaluator-run results count
Only scores produced by the evaluation organization itself count. It may run every model through the same pipeline, or run each model in the agent the evaluator assigns to it (usually the vendor's own). Vendor-reported scores and composite indices stitched together from other people's results do not count. For the same model, the representative configuration is chosen according to a pre-set reasoning tier; the highest score is not cherry-picked.
01只用评测方自己跑的成绩
评测机构自己给每个模型跑出来的成绩才计分:可以所有模型用同一套流程,也可以让每个模型在评测方为它指定的 Agent 里跑(通常是厂商自家的)。厂商自报、拿别人成绩拼成的综合指数都不计分。同一型号按预先定好的推理档位选代表配置,不挑最高分。
02 Each benchmark gets its own ruler
A benchmark has two numbers—difficulty and steepness—set jointly by the models that have taken it. If fewer than 20 models have taken a benchmark, the ruler is too uncertain, so it is not scored for now. Hard benchmarks carry weight in the high-score range, and benchmarks that are nearly saturated automatically become less important.
02给每项评测定一把尺
一项评测有难度和陡峭程度两个数,由测过它的模型共同定出;不到 20 个模型测过的评测,尺子定不准,先不计分。难的评测在高分段说话算数,刷满的评测自动变得不重要。
03 Convert results into the same score
Each result is back-solved along its own ruler and converted into the same score, so different benchmarks can be compared side by side. A pass rate above 95% only means “at least this strong”; below 5% only means “at most this strong.” Benchmarks of the same kind count as one vote. If one result is far out of line with the others, its influence on the total is capped (pull capping).
03把成绩折算成同一种分数
每个成绩顺着它的尺子倒推,折算成同一种分数,不同评测因此能放在一起比。通过率高于 95% 只算“至少这么强”,低于 5% 只算“至多这么强”。同类评测只算一票;和其他成绩差得太远的那项,对总分的影响设上限(拉力封顶)。
04 Score by domain, then weight
A domain score is the average of converted scores in that domain, pulled slightly toward the model's overall level. A domain that no one has tested yet is treated as the overall level—no points added or subtracted. The overall score adds up the domains using the weights below.
WEIGHTED BY DOMAIN Five domains, with programming weighted most.
The weights are a public choice, not a natural proportion of ability. Benchmarks for the same kind of task (two organizations rerunning the same suite, or two similar occupational-delivery benchmarks) together count as one vote. Vision, writing, and multilingual ability are not scored for now.
04分领域,再加权
每个领域的分数是这个领域折算分的平均,再向模型的整体水平收一点;还没人测的领域按整体水平看,不补分也不扣分。综合评分按下面的权重把各领域加起来。
WEIGHTED BY DOMAIN五个领域,编程最重。
权重是公开的选择,不是天然的能力比例。同一类任务的评测(两家机构复跑同一套题、两项同类的职业交付评测)合起来只算一票;视觉、写作、多语言目前不计分。
Programming 38% Terminal-Bench 4.0 (Artificial Analysis) · Terminal-Bench 4.0 (Vals AI) · Terminal-Bench 4.0 (Terminal-Bench) · LiveBench Coding (LiveBench) · SciCode (Artificial Analysis) · Code Migration v1.1 (Vals AI) · ProgramBench (Vals AI) · MirrorCode (Epoch AI) · Vibe Code Bench v1.1 (Vals AI) · Vibe Code Bench 1–100 (Vals AI, paused: only 0 of the 5 control models had been tested, below 60%) · CyberBench Patch v1.1 (Vals AI) · FrontierCode 2.0 (Artificial Analysis) · APEX-SWE (Mercor) · FrontierCode 2.0 (Cognition) · LiveBench Agentic Coding (LiveBench) · WeirdML v3 (Håvard Tveit Ihle) · FrontierSWE v2 (Proximal Labs) · TapTap Maker (TapTap Maker)
编程38%Terminal-Bench 4.0(Artificial Analysis) · Terminal-Bench 4.0(Vals AI) · Terminal-Bench 4.0(Terminal-Bench) · LiveBench Coding(LiveBench) · SciCode(Artificial Analysis) · Code Migration v1.1(Vals AI) · ProgramBench(Vals AI) · MirrorCode(Epoch AI) · Vibe Code Bench v1.1(Vals AI) · Vibe Code Bench 1–100(Vals AI,暂停:对照的 5 个型号只测了 0 个,不到 60%) · CyberBench Patch v1.1(Vals AI) · FrontierCode 2.0(Artificial Analysis) · APEX-SWE(Mercor) · FrontierCode 2.0(Cognition) · LiveBench Agentic Coding(LiveBench) · WeirdML v3(Håvard Tveit Ihle) · FrontierSWE v2(Proximal Labs) · TapTap Maker(TapTap Maker)
Research 19% Humanity's Last Exam (Artificial Analysis) · FrontierMath T1–3 (Epoch AI) · LiveBench Reasoning (LiveBench) · ZhiZhi · Logic Monthly (llm2014) · Sinan · Reasoning (OpenCompass) · ARC-AGI-2 (ARC Prize) · Sinan · Abstraction (OpenCompass) · CritPt (Artificial Analysis) · CritPt v2 Hard (Artificial Analysis) · Chess Puzzles (Epoch AI) · Mystery Game Puzzles (Epoch AI) · MysteryMechanism (Vals AI) · Terminal-Bench Science (Vals AI) · Terminal-Bench Science (Artificial Analysis) · BioMysteryBench (Vals AI)
科研19%Humanity's Last Exam(Artificial Analysis) · FrontierMath T1–3(Epoch AI) · LiveBench Reasoning(LiveBench) · 致知 · 逻辑月榜(llm2014) · 司南 · 推理(OpenCompass) · ARC-AGI-2(ARC Prize) · 司南 · 抽象(OpenCompass) · CritPt(Artificial Analysis) · CritPt v2 Hard(Artificial Analysis) · Chess Puzzles(Epoch AI) · Mystery Game Puzzles(Epoch AI) · MysteryMechanism(Vals AI) · Terminal-Bench Science(Vals AI) · Terminal-Bench Science(Artificial Analysis) · BioMysteryBench(Vals AI)
Professional work 19% GDPval-AA (Artificial Analysis) · AA-Briefcase (Artificial Analysis) · AutomationBench-AA (Artificial Analysis) · Finance Agent v2.1 (Vals AI) · APEX-Agents (Mercor) · Harvey Legal Agent Benchmark v1.1 (Vals AI) · Harvey LAB-AA v1.1 (Artificial Analysis) · τ³-Banking (Mercor) · LiveBench Data Analysis (LiveBench) · MedCode (Vals AI, paused: no new models tested for 30 days) · AA-Analyst Agent (Artificial Analysis) · Legal Research Bench (Vals AI) · TaxAgentBench (Vals AI) · Excel Modeling v1.1 (Vals AI)
专业办公19%GDPval-AA(Artificial Analysis) · AA-Briefcase(Artificial Analysis) · AutomationBench-AA(Artificial Analysis) · Finance Agent v2.1(Vals AI) · APEX-Agents(Mercor) · Harvey 法律 Agent 基准 v1.1(Vals AI) · Harvey LAB-AA v1.1(Artificial Analysis) · τ³-Banking(Mercor) · LiveBench Data Analysis(LiveBench) · MedCode(Vals AI,暂停:已经 30 天没有测新模型) · AA-Analyst Agent(Artificial Analysis) · Legal Research Bench(Vals AI) · TaxAgentBench(Vals AI) · Excel Modeling v1.1(Vals AI)
Knowledge Q&A 14% AA-Omniscience v1.1 (Artificial Analysis) · SimpleQA Verified (Epoch AI) · Sinan · Knowledge (OpenCompass) · AA-LCR (Artificial Analysis) · MLCR-AA (Artificial Analysis) · LiveBench Language (LiveBench) · NYT Connections Extended Edition (Lech Mazur) · LiveBench IF (LiveBench) · BullshitBench v2 (BullshitBench)
知识问答14%AA-Omniscience v1.1(Artificial Analysis) · SimpleQA Verified(Epoch AI) · 司南 · 知识(OpenCompass) · AA-LCR(Artificial Analysis) · MLCR-AA(Artificial Analysis) · LiveBench Language(LiveBench) · NYT Connections 扩展版(Lech Mazur) · LiveBench IF(LiveBench) · BullshitBench v2(BullshitBench)
Human blind preference 10% Arena Text (LMArena) · Arena WebDev (LMArena)
真人盲选10%Arena Text(LMArena) · Arena WebDev(LMArena)
To receive a rank, a model needs scores from at least two evaluation organizations; at least 3 benchmark categories in programming; and at least 2 categories each in research, professional work, and knowledge Q&A. The leaderboard must also have the model's official API price or public weights on record. Other models that have scores are listed as “not yet ranked.” Models released more than 18 months ago leave the current leaderboard, while their existing detail pages remain.
给出名次需要至少两家评测机构的成绩,编程至少 3 类评测,科研、专业办公、知识问答各至少 2 类,并且榜单已收录该型号的官方 API 价格或公开权重。其余有分数的模型列在“暂不排名”。发布超过 18 个月的型号退出当前榜单,已有详情页保留。
How many models must a new benchmark cover? Since v19, the 34 benchmarks that are new or have a new version (noted on each benchmark's source page) must cover a control group. From the models on the previous overall leaderboard that already had an official API price or public weights on record, take the highest-ranked model from each of the top 10 vendors. Using the same version, the evaluator's main pipeline, and the representative configuration, the benchmark is scored only if at least 60% of the control models have results. Below that, it is paused until the gap is filled. A model released less than a month ago and not yet tested does not count as missing. Missing results stay blank; a true zero still counts as tested. Domain weights do not change when benchmarks are added.
新增评测需要覆盖多少模型?自 v19 起新增或换版的 34 项评测(各评测来源页已注明)要覆盖对照组。从上一轮综合榜已收录官方 API 价格或公开权重的型号里,取前 10 家厂商各自名次最高的型号。按同版本、评测方的主流程和代表配置,对照型号至少 60% 有成绩才计分,低于就暂停,补齐再恢复;发布不满一个月的型号没测不算缺。缺测留空,真实零分仍算测过。各领域权重不因新增评测改变。
How should the AIHOT score be read? 50 is the average level of the reference models, and every 15 points is roughly one standard deviation. It is neither accuracy nor win rate; a higher score means the model is stronger overall on these third-party benchmarks. Category scores and the overall score sit on the same scale.
How large a gap is real? When the benchmark mix is randomly changed and the calculation is rerun, two models more than 3 points apart almost never switch order; those less than 1 point apart are close to a coin flip. When you look at rank, look at the score too.
AIHOT 评分怎么看?50 分是参照模型的平均水平,每 15 分约是一个标准差。它不是正确率,也不是胜率;分数越高,说明在这些第三方评测里整体越强。分类评分和综合评分在同一条刻度上。
差多少分才算真的有差距?随机换一套评测组合重算,相差 3 分以上的两个模型,先后几乎不会变;相差不到 1 分的,接近抛硬币。看名次时请连同分数一起看。
Why can a model score modestly on a hard benchmark yet still get a high converted score? The converted score asks, “How much ability would it take to get this result on this benchmark?” 30% on a hard problem set may say more than 90% on an easy one; once a benchmark is nearly saturated, even a very high score cannot convert into much more.
为什么难的评测考得一般,折算分反而高?折算分看的是“在这项评测上考成这样,需要多强的能力”。难题上的 30% 可能比简单题上的 90% 更能说明问题;快被刷满的评测,再高的分也折算不出更多。
Why do some models have a score but no rank? Ranking requires scores from at least two evaluation organizations; at least 3 benchmark categories in programming; and at least 2 categories each in research, professional work, and knowledge Q&A. The leaderboard must also include the model's official API price or public weights; an official free API is counted as zero price. Models that already have a score but do not yet meet these conditions are listed as “not yet ranked.”
为什么有的模型只有分数、没有名次?排名需要至少两家评测机构的成绩,编程至少 3 类评测,科研、专业办公、知识问答各至少 2 类。榜单也要收录该型号的官方 API 价格或公开权重,官方免费 API 按零价计算。已有评分但暂未满足这些条件的模型,列在“暂不排名”。
Why not use scores published by the vendors themselves? Scores that vendors run and report themselves cannot be checked for runtime conditions, and they may be tuned specifically for the leaderboard. This leaderboard uses only results run by the evaluation organizations.
What is “fallback”? During evaluation, a vendor's API sometimes routes part of the requests to an older model from the same vendor. The substitute is always weaker, so such a result can only be lower, not higher. It is therefore counted as usual and marked on the model page. If the evaluator reports that more than half of the affected questions were hit, the result is not counted.
厂商自己公布的成绩为什么不用?厂商自己跑、自己报的成绩没法核对运行条件,也可能专门为榜单优化。本榜只用评测机构自己跑的成绩。
“回退”是怎么回事?评测时,厂商的接口有时把一部分请求转给了自家更早的型号。替补都更弱,这样的成绩只会偏低、不会偏高,所以照常计入,并在模型页标注;评测方公布受影响题目超过一半的,不计。
When do new models appear? Evaluation scores are checked four times a day. A new model enters the calculation as soon as its scores are collected; it shows a score once it meets the scoring conditions and gets a rank once it meets the ranking conditions.
What happens if an evaluator is temporarily unavailable or cannot keep up with new models? If the site is temporarily down, we keep using the most recent successfully collected scores. If a model disappears from the latest data, we use the same-version scores verified within the last seven days for at most seven days. If an entire calculation run fails, the previous valid leaderboard and its original timestamp are kept. If a benchmark still has more than half of its models untested one month after the month's major model releases, or if no new models have been added for 30 consecutive days, it is paused and resumes after the gap is filled.
新模型什么时候会出现?评测成绩每天检查四次。新型号的成绩一收录就参与计算,满足评分条件后显示分数,满足排名条件后给出名次。
某家评测暂时打不开,或者跟不上新模型,会怎样?暂时打不开时,继续使用它最近一次成功采集的成绩;某个型号从最新数据里消失时,最多沿用七天内核验过的同版本成绩。整轮计算失败时保留上一轮有效榜及原时间。一项评测在当月主要模型发布一个月后还有一半以上没测,或者连续 30 天没加新模型,就暂停计分,补上再恢复。
Does price or speed affect ranking? Ranking is based on score only. Having the official API price or public weights on record is one of the conditions for being ranked. Prices on the page are shown per million tokens, with the vendor's official source, so you can estimate API cost.
价格或速度会影响名次吗?名次只按评分排。收录官方 API 价格或公开权重是给出名次的条件之一。页面上的价格按每百万 Token 展示,附厂商官方来源,供你估算 API 成本。
See calculation details and the current methodology version: 2026.11-capability-scale-v24. The approach belongs to the item response theory (IRT) family. Each pass-rate benchmark is a logistic ruler: for a model with ability z, pass rate = σ(A·z − e), where A is steepness and e is difficulty. A rated-classification benchmark (Elo) is a straight ruler: rating = c + s·z. Its scale is aligned to the same batch of models' levels in other domains; it cannot set its own scale.
查看计算细节与当前版本方法版本:2026.11-capability-scale-v24。做法属于项目反应理论(IRT)一类:每项通过率评测是一把 logistic 尺,能力为 z 的模型通过率 = σ(A·z − e),A 是陡峭程度、e 是难度;等级分类评测(Elo)是一把直尺,等级分 = c + s·z,它的刻度按同一批模型在其他领域的水平对齐,不能自定尺度。
A result is back-solved into a converted score x = (logit(pass rate) + e) / A. Pass rates above 95% or below 5% are treated as censored (scores near perfect or near zero cannot separate models): they only say “at least” or “at most,” and the farther a model's level goes beyond that line, the less effect it has. The noise of one converted score is taken as a standard deviation of 0.33. If a converted score deviates by more than twice that noise, its pull is capped (there is a limit to how hard it can pull the total), so one bad result or one inflated benchmark cannot drag the total off course.
成绩倒推成折算分 x = (logit(通过率) + e) / A。高于 95% 或低于 5% 的通过率按截尾处理(接近满分或零分的成绩拉不开差距):只说明“至少”或“至多”,模型的水平越过这条线越远,它的作用越小。一个折算分的噪声按标准差 0.33 计;偏离超过两倍噪声的折算分,拉力封顶(它拉动总分的力气有上限),一个坏成绩或被刷高的评测带偏不了总分。
A model's ability in a domain = overall level g + domain deviation δ. The prior standard deviation of δ is 0.25 (selected by “hide part of the scores and predict them”). For an untested domain, δ = 0. Rulers and abilities are fitted alternately until convergence. Models used to calibrate the rulers must have scores in at least 4 benchmark categories, and a benchmark must have at least 20 such models. Overall score = domain abilities added by weight; displayed score = 50 + 15·z.
模型在某个领域的能力 = 整体水平 g + 该领域的偏离 δ,δ 的先验标准差为 0.25(按“遮住一部分成绩再预测”选出);没测的领域 δ = 0。尺子和能力交替拟合直到收敛,参加定尺的模型至少要有 4 类评测的成绩,一项评测至少要有 20 个这样的模型。综合 = 各领域能力按权重相加,显示分 = 50 + 15·z。
The scale is anchored to reference models: 123 models that, when the methodology was finalized, were from the past 18 months, had been tested by at least two organizations, and had enough evidence to be listed. Their levels were frozen. In every later round, the reference models keep the same mean and spread as when frozen, so adding models, adding benchmarks, or retiring benchmarks does not make scores drift as a whole. Scores remain comparable across rounds.
刻度锚定在参照模型上:方法定稿时近 18 个月、两家以上机构测过、证据够列出的 123 个模型,它们的水平冻结保存。以后每一轮都让参照模型保持冻结时的平均值和离散程度,所以新增模型、新增或停用评测,都不会让分数整体漂移,评分可以跨轮比较。
Before launch, the method was tested on real data. When part of the scores were hidden and predicted, the error was clearly smaller than “convert each benchmark to a standard score and average.” New generations in the same family were less likely to rank below the previous generation. Recalculating after removing all scores from one organization still matched that organization's own composite index closely. On synthetic data with known ground truth, models with few scores and models with many scores showed no systematic bias.
这套方法上线前用真实数据检验过:遮住部分成绩再预测,误差明显小于“各项换成标准分再平均”;同系列新一代排在上一代后面的情况更少;去掉某一家机构的全部成绩重算,和这家机构自己的综合指数仍高度一致;在已知真值的合成数据上,成绩少的模型和成绩多的模型没有系统偏差。
View all benchmark sources →
查看全部评测来源 →