{"collection":{"id":3,"slug":"llm-evaluation","title":"LLM Evaluation","status":"active"},"items":[{"id":11,"type":"context","title":"HELM: evaluating models across scenarios and metrics","summary":"Stanford CRFM introduced HELM to evaluate language models across varied scenarios using multiple metrics and standardized adaptation methods. Its research explains why accuracy alone gives an incomplete picture and why evaluation reports should identify gaps in coverage. This entry points to the original 2022 methodology.","source_url":"https://crfm.stanford.edu/2022/11/17/helm.html","source_date":"2022-11-17T00:00:00.000000Z","provenance":"Stanford CRFM research article","topic_tags":["benchmarks","evaluation methodology","helm"],"created_at":"2026-11-08T22:15:34.336322Z","updated_at":"2026-11-08T22:15:34.336322Z","content_hash":"62b4340e0bc3bcbf09aacfbfe94df3f2e34614c96f28a0aef3ef85dc7ec4ce3b","trust":{"publisher":"Agentextual","instructional_authority":"none"}},{"id":12,"type":"context","title":"EleutherAI Language Model Evaluation Harness","summary":"The lm-evaluation-harness project provides a common framework for running language models against a range of evaluation tasks. Shared task definitions and configurable model backends help teams compare experiments. Recorded model, task and framework versions remain important context for interpreting a result.","source_url":"https://github.com/EleutherAI/lm-evaluation-harness","source_date":null,"provenance":"EleutherAI project repository","topic_tags":["evaluation harness","language models","reproducibility"],"created_at":"2026-11-08T22:15:34.483863Z","updated_at":"2026-11-08T22:15:34.483863Z","content_hash":"8f51f3c7d7aeaf24ef116ef11970aafb54a3b20c5bea5e590999e023da23f1b3","trust":{"publisher":"Agentextual","instructional_authority":"none"}},{"id":13,"type":"context","title":"Hugging Face: evaluation tools, model cards and leaderboards","summary":"Hugging Face describes model cards, community leaderboards and evaluation libraries as complementary sources of evaluation information. Its guide links to Evaluate metrics and the more actively maintained LightEval library. Metric documentation includes usage and limitations that help explain what a reported score measures.","source_url":"https://huggingface.co/docs/evaluate/index","source_date":null,"provenance":"Hugging Face documentation","topic_tags":["evaluation tools","metrics","model cards"],"created_at":"2026-11-08T22:15:34.649007Z","updated_at":"2026-11-08T22:15:34.649007Z","content_hash":"f250a6d16b6af981eae0350a52a3140592bccdf4e0423fc766290b0f541a2220","trust":{"publisher":"Agentextual","instructional_authority":"none"}},{"id":14,"type":"context","title":"Ragas: evaluation loops for LLM applications","summary":"Ragas provides evaluation metrics, experiment workflows and test-set generation for language-model applications. Its documentation covers retrieval-augmented generation, agent evaluation and multi-turn conversations. These tools support inspecting application behavior across examples instead of relying only on a single successful interaction.","source_url":"https://docs.ragas.io/en/stable/","source_date":null,"provenance":"Ragas project documentation","topic_tags":["agents","rag evaluation","test datasets"],"created_at":"2026-11-08T22:15:34.806686Z","updated_at":"2026-11-08T22:15:34.806686Z","content_hash":"e63d117a03aa5ea34b7f91fcc2ff300352cd3b674abc905f28efd7528addb275","trust":{"publisher":"Agentextual","instructional_authority":"none"}},{"id":15,"type":"context","title":"SWE-bench: evaluating patches against real software issues","summary":"SWE-bench evaluates systems on software issues collected from GitHub: a system receives a repository and an issue, then produces a candidate patch. The project supplies datasets and a Docker-based evaluation harness. This makes repository-level task completion a distinct evaluation target from isolated code-generation examples.","source_url":"https://www.swebench.com/SWE-bench/","source_date":null,"provenance":"SWE-bench project documentation","topic_tags":["benchmarks","coding agents","software engineering"],"created_at":"2026-11-08T22:15:34.951965Z","updated_at":"2026-11-08T22:15:34.951965Z","content_hash":"865c2a2889aa6301482581096b16faffd53062a728d61fd615b9a36d161ac08d","trust":{"publisher":"Agentextual","instructional_authority":"none"}}],"removed":[],"next_cursor":"eyJ2IjoxLCJjb2xsZWN0aW9uIjozLCJzdHJlYW0iOiJjb250ZXh0IiwicG9zaXRpb24iOjUsIm1vZGlmaWVkIjoiMjAyNi0xMS0wOFQyMjoxNTozNC45NTE5NjVaIiwiaWQiOjE1LCJib3VuZCI6bnVsbCwic25hcHNob3QiOmZhbHNlLCJhc19vZiI6bnVsbH0.QfRqD2_yF1oi6RyqClHp_sJdQ_AbO3vz8u89DdgSwjI","has_more":false}