We estimate that the collecting Global HR & Payroll knowledge gets 15,000-20,000 updates on a yearly basis. To build a global payroll infrastructure we had to automate tracking / implementing these changes and we've replaced much of our operational scaffolding with AI systems built in-house

To help us navigate the accuracy of what we can achieve with AI, we built two benchmarks HRmark and Paymark - one benchmark for general HR questions and payroll-related questions.

HRmark

HRMark measures how accurately language models answer HR/Employment questions that depend on local law and practice. It contains 2,500 questions from 50 countries, each paired with a reference answer, and every answer is graded against that reference by a model grader.

We evaluate 12 systems from 8 providers at two reasoning-effort settings, low and high.

Key results

• 13 points separate the strongest and weakest models, significantly more than the effect of reasoning effort.
• Highest score of all was Fable 5.1 in High effort with 69%
• At Low effort: Claude Fable 5.1 scores highest(68.4%); Claude Opus 5.5, GPT-6 Astra, Gemini 3.8 Flash and GPT-6.1 Sol are within two points.
• At High effort: changes the median score only by +0.6 points; 2 of 12 systems improve significantly (Claude Sonnet 5.5, Qwen3.8-Max) but all else has very little improvement.
• High effort costs 2.7× as much and produces 2.9× the output tokens.
• The hardest domain is Immigration & Right to Work, where systems average 51%.

Effect of reasoning effort

Raising effort from low to high produces a significant improvement for Claude Sonnet 5.5 and Qwen3.8-Max, while GPT-6.1 Sol and Muse Spark 1.3 score slightly lower. Most systems move by about a point. Claude Opus 5.5 gains the most rank, from 4th to 2nd, and GPT-6.1 Sol falls from 2nd to 5th. At high effort the leading group is Claude Fable 5.1, Claude Opus 5.5, GPT-6 Astra and Gemini 3.8 Flash; GPT-6.1 Sol dropped out of it.

Longer reasoning does not appear to supply missing jurisdiction-specific knowledge: systems that score poorly at low effort also score poorly at high effort.

Cost and efficiency

There's a 2.7× increase for a median change of +0.6 points on going from low to high effort. Gemini 3.8 Flash is the least expensive system within two points of the top low-effort score, at $0.80 per 1,000 answers. The most expensive configuration is GPT-6 Astra at high effort ($60.37 per 1,000).

The cost increase comes mostly from reasoning tokens. At high effort, Qwen3.8-Max produces 12× as many output tokens per answer, and its mean latency rises from 12 to 153 seconds. GPT-6 Astra, GPT-6.1 Sol and GPT-6 Sol use at least four times as many tokens without a significant gain. Claude Sonnet 5.5 has the best return on additional tokens: +3.0 points for 2.3× the output.

Figure 3 · Accuracy against cost

Figure 4 · Output tokens, and token increase against accuracy change

Results by HR domain

Accuracy varies more by HR domain than by effort setting. Among domains with at least 100 questions, systems score highest on Working Time (68.5% averaged over systems at high effort) and lowest on Immigration & Right to Work (51.2%) and Employment Contract & Terms (55.6%). High effort helps most on Parental & Family Leave (+2.1 points) and slightly lowers the score on Employment Contract & Terms and Occupational Health & Safety. Claude Fable 5.1 has the top score in 5 of 12 domains at high effort.

Full results

Paymark

PayMark measures how accurately language models calculate payroll. It contains 2,500 real payslips from 50 countries. For each one, a model receives the earnings, gross pay and any amounts the employer has already decided, and must calculate the statutory employee deductions and employer contributions for that country and pay period.

Scoring is arithmetic: a payslip counts as right only when the employee deductions total, the employer contributions total and net pay all match the completed payroll run. We evaluate 12 systems from 8 providers at two reasoning-effort settings.

Key results

• GPT-6 Astra, Claude Opus 5.5, GPT-6.1 Sol and Claude Fable 5.1 lead at both effort levels, at most 2.5 points apart.
• The best system gets 26.0% of payslips fully right; employee deductions cause most errors.
• High effort changes the median score by +1.6 points and improves 9 of 12 systems significantly.
• High effort costs 3.0× as much

What the model is given and what it calculates

A payroll's deductions mix lines set by law (income tax, employee and employer social security, other statutory charges) with lines set by the employer's plan or a chosen provider (a company pension rate, a private insurer's premium, a recharge).

The model receives the country, pay period, pay frequency, currency, contract, pay inputs, earnings lines, gross pay and the items already decided, with their amounts. It must calculate every statutory employee deduction and employer contribution, decide for itself which ones apply, and return them as JSON with both totals. It gets no list of charges, no rates and no tools.

tolerance = the largest of 1% of the expected amount, 0.25% of gross pay, or 0.01 a number is right when |model − expected| ≤ tolerance

A payslip is fully right only when the employee total, the employer total and net pay are all right; in practice a right employee total gives a right net pay, so it comes down to both totals. A missing answer, or one with no usable figure, counts as wrong.

Results

GPT-6 Astra scores highest at both effort levels: 24.3% at low and 26.0% at high. The same group leads at both settings: GPT-6 Astra, Claude Opus 5.5, GPT-6.1 Sol and Claude Fable 5.1, at most 2.5 points apart, which is close to the margin of error. Claude Haiku 5.5, Qwen3.8-Max and DeepSeek V4.1-Flash score lowest at high effort. 16 points separate the strongest and weakest systems at low effort.

Effect of reasoning effort

High effort improves 9 of 12 systems by more than chance would explain; Grok 4.7, DeepSeek V4.1-Flash and Claude Haiku 5.5 show no clear change. The gains are modest: the median is +1.6 points, and the largest is Gemini 3.8 Flash's +4.8. DeepSeek V4.1-Flash scores slightly lower, within chance. The ranking barely moves: systems that calculate poorly at low effort still calculate poorly at high effort.

Cost and efficiency

Low vs High effort costs 3.0× as much for a median gain of +1.6 points. GPT-6.1 Sol is the least expensive system in the leading group, at $7.11 per 1,000 payslips. Claude Sonnet 5.5 gains the most per extra dollar. Grok 4.7 costs at least twice as much at high effort without a clear gain. The most expensive configuration is Qwen3.8-Max at high effort ($208.15 per 1,000).

The extra cost is mostly extra output. At high effort, GPT-6 Astra writes 5.7× as many tokens per payslip for +1.7 points, while Claude Sonnet 5.5 gains +3.4 with 1.8× the output. More output does not reliably buy more accuracy.

Figure 3 · Accuracy against cost

Figure 4 · Output tokens, and token increase against accuracy change

Results by region

Accuracy differs by region. At high effort, systems average 21.2% in the Middle East and Africa and 10.8% in the Americas. Claude Opus 5.5 has the top score in 2 of 4 regions.

Accuracy by region

Full results