Separate eval-only workflow and change to 8k1k

[Oseltamivir]

Summary

• Decouple eval jobs from benchmark jobs into a dedicated eval / workflow stage, so evals can run independently with evals-only: true in perf-changelog
• Add eval-only input to benchmark-tmpl.yml that skips throughput benchmarks and result file checks, running only lm_eval
• Fix GSM8k prompt template: change #### <answer> to #### [number] to prevent models (e.g. Kimi-K2.5) from outputting the literal <answer> tag, which caused strict-match to return [invalid]
• Add eval support to all 49 single-node benchmark scripts (server restart with native max context, eval context override, request timeout increase)
• Update process_changelog.py and generate_sweep_configs.py to emit a separate evals matrix for eval-only jobs
• Changed from 1k8k to 8k1k
• Groups now include dp-attn (previously missing, so dp-attn variants were merged)
• Instead of picking max-conc at highest/lowest TP, it picks all TPs at the highest conc + all TPs at the median conc.

Separate jobs on a normal run tested with:

- config-keys:
    - gptoss-fp4-h100-vllm
    - dsr1-fp8-mi325x-sglang

---

Implementation Details

• get_native_max_context_length(model) — reads max_position_embeddings / max_sequence_length / seq_length / n_positions from the model's HuggingFace config.
• max_output_tokens: dynamically computed as eval_context_len - 4096 (capped at
  16384), was fixed 8192
• Score validation: new validate_scores.py — post-eval gate checking exact_match metrics ≥ 0.85, exits non-zero on failure or zero metrics found
• Failure mode: eval-only uses if-no-files-found: error (was ignore)
• GSM8k prompt: #### <answer> → #### [number]
• ATOM torchvision workaround: uninstalls torchvision before lm-eval install to avoid circular import
• PYTHONDONTWRITEBYTECODE=1 added to workflow env to avoid EACCES error on pycache.

New job step: evals

[github-actions]

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

• https://cookbook.sglang.io/
• https://docs.vllm.ai/projects/recipes/en/latest/

If it is not, please create a PR first before we can merge your PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

• https://github.com/vllm-project/recipes
• https://github.com/sgl-project/sgl-cookbook

[functionstackx]

thanks for the PR @Oseltamivir ! seems great that we now parallel evals instead of having them in sequential

mostly LGTM

just had a couple of questions

1. does this PR change the evals to use the industry standard num of shots for gsm8k?
2. previously when evals hard crashed, the script still exits with success exit code, is this fixed now?

[Oseltamivir]

1. Changed to 5-shot
2. Yes, also added suggestion of failing when result <85%

[claude]

The inline comment about compute_eval_context_length() using benchmark_ctx * 1 (instead of * 5) was marked resolved, but the fix was never applied — the diff still reads local eval_ctx=$(( benchmark_ctx * 1 )), making eval context expansion a no-op across all 49 benchmark scripts.

Extended reasoning...

Overview

This PR separates eval-only benchmarking into a dedicated workflow, updates 49 single-node benchmark scripts with EVAL_ONLY mode support, and introduces compute_eval_context_length() / setup_eval_context() helpers in benchmark_lib.sh. The changes touch CI workflow YAML, runner launch scripts, utility Python, and all single-node benchmark scripts.

Critical Unfixed Bug

The compute_eval_context_length() function in benchmarks/benchmark_lib.sh computes the eval context as:

local eval_ctx=$(( benchmark_ctx * 1 ))

The multiplier should be 5 per the function own doc comment ("Uses 5x the benchmark context"). With * 1, every call to setup_eval_context() expands the context by a factor of one -- i.e., no expansion at all. For a typical 8k1k workload (ISL=8192, OSL=1024), the eval context is 9416 tokens instead of the intended ~47080. The inline comment flagging this (ID 2971153643, filed 2026-03-22) was marked as resolved in the GitHub thread, but the actual code change was never made. This is the single most critical bug in the PR since all 49 updated scripts rely on this function.

Security Risks

No security-sensitive paths (auth, crypto, permissions) are touched. The changes are benchmarking infrastructure only.

Level of Scrutiny

High. This is a large PR (71 files) making behavioral changes to production CI pipelines and all single-node benchmark scripts. The * 1 bug needs to be corrected before the PR can be merged.

Other Factors

All other previously flagged bugs appear to have been addressed in subsequent commits. The only blocking issue is the unfixed * 1 multiplier.

[claude]

The recent commits addressed the CUDA graph ordering and rm -rf cleanup issues — thanks for working through those. The one remaining blocker is benchmark_ctx * 1 on line 624 of benchmarks/benchmark_lib.sh, which needs to be * 5 per the function’s own docstring (the open inline comment from 3/22 tracks this).

Extended reasoning...

Overview

PR #911 separates eval-only jobs from throughput benchmarks, adds a compute_eval_context_length() helper to expand the context window for lm-eval runs, and changes the eval workload selection from 1k8k to 8k1k. 71 files are touched, the majority being single-node benchmark scripts that gained EVAL_ONLY blocks.

Security Risks

No security-sensitive code is touched. The PR handles CI workflow configuration, shell benchmark scripts, and Python sweep-config generation. There are no auth, crypto, or permissions changes.

Level of Scrutiny

Moderate. The logic in compute_eval_context_length() is new and central to the feature: every one of the 49 benchmark scripts calls setup_eval_context(), which delegates to this function. A one-character error in the multiplier (* 1 instead of * 5) silently produces no context expansion at all, making the entire eval-only path a no-op for its stated purpose.

Current Status

The "final" commit (2f3a788, 2026-03-23T06:12Z) addressed two previously flagged issues: the CUDA graph ordering bug in dsr1_fp8_b200_trt_mtp.sh and the rm -f/rm -rf cleanup discrepancy in benchmark-tmpl.yml. However, the compute_eval_context_length() multiplier at benchmarks/benchmark_lib.sh:624 still reads benchmark_ctx * 1. This inline comment thread has been open since 2026-03-22T06:44Z and remains unresolved. The fix is a one-character change but it is the core correctness property of the feature.

[functionstackx]

.

[claude]

Two critical bugs flagged in prior review rounds remain unpatched in the actual code despite their GitHub conversation threads being marked resolved — please verify the fixes landed in the branch.

Extended reasoning...

Overview

This PR (72 files) separates eval-only CI runs from throughput benchmarks and switches eval workloads to an 8k/1k (ISL/OSL) profile. The changes touch CI workflow YAML, benchmark_lib.sh, generate_sweep_configs.py, process_changelog.py, validate_scores.py, and every single-node benchmark shell script.

Bugs Still Present in Current Branch

Two issues previously marked resolved on GitHub are not fixed in the files on the branch:

1. benchmarks/benchmark_lib.sh line 624 — local eval_ctx=$(( benchmark_ctx * 1 )) still multiplies by 1 instead of 5. This makes compute_eval_context_length() a no-op: the server launched in eval-only mode gets exactly the same (tiny) context window as during throughput benchmarks, defeating the entire purpose of the EVAL_ONLY flow.

2. benchmarks/single_node/dsr1_fp8_mi355x_atom_mtp.sh lines 87–88 — A stray set -x followed by rm -rf ./utils/bench_serving\ (with a bash line-continuation backslash) still unconditionally deletes the benchmark serving utilities after every run on MI355X hardware.

Security / Stability Risk

The rm -rf on line 88 is a destructive filesystem operation that runs outside any conditional block. It will silently destroy tooling on every benchmark execution, requiring manual recovery. The * 1 multiplier bug silently produces incorrect eval results (accuracy measured at the wrong context length) without any error.

Level of Scrutiny

Both of these were flagged as critical in earlier review rounds. The inline comments were marked resolved, which typically signals the author fixed them — but cross-checking the actual file contents shows the fixes were not committed. This needs explicit human confirmation that the correct patch is in place before merging.

Other Factors

A fix verified in code (dsr1_fp8_b200_trt_mtp.sh EVAL_ONLY ordering) shows the process works when changes are actually committed. The remaining two issues just need the already-identified one-line fixes to be applied and pushed.