Context collection · active
Research and tools for measuring language-model quality, retrieval quality and coding-agent behavior.
Collection updated 2026-11-08T22:15:34.175128Z
Stanford CRFM introduced HELM to evaluate language models across varied scenarios using multiple metrics and standardized adaptation methods. Its research explains why accuracy alone gives an incomplete picture and why evaluation reports should identify gaps in coverage. This entry points to the original 2022 methodology.
Stanford CRFM research article · Updated 2026-11-08T22:15:34.336322Z
External source ↗The lm-evaluation-harness project provides a common framework for running language models against a range of evaluation tasks. Shared task definitions and configurable model backends help teams compare experiments. Recorded model, task and framework versions remain important context for interpreting a result.
EleutherAI project repository · Updated 2026-11-08T22:15:34.483863Z
External source ↗Hugging Face describes model cards, community leaderboards and evaluation libraries as complementary sources of evaluation information. Its guide links to Evaluate metrics and the more actively maintained LightEval library. Metric documentation includes usage and limitations that help explain what a reported score measures.
Hugging Face documentation · Updated 2026-11-08T22:15:34.649007Z
External source ↗Ragas provides evaluation metrics, experiment workflows and test-set generation for language-model applications. Its documentation covers retrieval-augmented generation, agent evaluation and multi-turn conversations. These tools support inspecting application behavior across examples instead of relying only on a single successful interaction.
Ragas project documentation · Updated 2026-11-08T22:15:34.806686Z
External source ↗SWE-bench evaluates systems on software issues collected from GitHub: a system receives a repository and an issue, then produces a candidate patch. The project supplies datasets and a Docker-based evaluation harness. This makes repository-level task completion a distinct evaluation target from isolated code-generation examples.
SWE-bench project documentation · Updated 2026-11-08T22:15:34.951965Z
External source ↗