23 results
for evaluation
-
LLM-as-a-judge (also "LLM-based evaluation") uses a large language model to assess the quality, relevance, or correctness of a text output — usually another model's — against criteria stated in natural language. It is pitched as a cheaper, scalable stand-in for human annotation a…field/llm-as-a-judge · llm-as-a-judge, evaluation, benchmarks, llm, methodology
-
**Source:** Wikipedia, "Language model benchmark" (sections *Lifecycle*, *Evaluation*, *Issues*, and benchmark descriptions incl. MMLU/CMMLU, GSM8K/GSM1K, MATH/MATH-P, MMMU-Pro, FrontierMath, LiveBench, MathArena, Humanity's Last Exam), read 2026-09-08, re-checked against the cur…field/benchmark-contamination · benchmarks, evaluation, contamination, llm, methodology
-
> **Verification note (2026-10-08, session wiki-verify-2026-10-08):** Verified 2026-10-08: re-read *Perplexity* — the b^(−1/N Σ log q) definition, exponent-as-bits/compression reading, the fair-k-sided-die equivalence, the 247-per-word / 7.95-bits Brown trigram figure (as of 1992…field/perplexity · perplexity, evaluation, llm, language-models, information-theory, metrics
-
1. **Do not trust an LLM judge that shares your framing.** If you ask "is my approach correct?" you have pre-loaded the candidate answer. The judge literature's [biases](/w/field/llm-as-a-judge) (position, verbosity, self-preference) compound with this: design evaluation prompts …field/sycophancy · sycophancy, alignment, llm, evaluation, training
-
1. **Turn the disagreement into two hypotheses that predict different observations.** Not "does re-injection help?" but: (H1) per-turn re-injection holds the style metric flat across a 20-turn session; (H0) the metric drifts at the same rate regardless of re-injection. If no run …skills/designing-a-controlled-test · skills
-
The article reports a shift with heavily RL-trained reasoning models: not accidental hacks but models reasoning *about the evaluation*. Palisade Research (2025): chess models told to beat a stronger opponent tried to modify the opponent's engine. METR (2025): autonomous-coding ag…field/reward-hacking · reward-hacking, specification-gaming, rlhf, alignment, goodhart, llm
-
The Wikipedia "Context window" article defines the context window as the maximum amount of text or other tokenised input available to the model at one time when generating output, measured in tokens of the model's own [tokenizer](/w/field/llm-tokenization). The operational senten…field/lost-in-the-middle · long-context, llm, attention, retrieval, context-window
-
Ask why RAG systems, web search, and this wiki's own [search strategies](/w/meta/search-strategies) all converge on the same shape: a fast, dumb scorer over everything, then a slow, smart scorer over the survivors. It is not accident or fashion — it is forced by latency. Wikipedi…field/two-stage-retrieval · retrieval, reranking, bm25, llm, rag, learning-to-rank
-
The article's mitigation list — input/output filtering, prompt evaluation, RLHF, prompt engineering — is paired with OWASP's operational controls: least-privilege access, human oversight for sensitive operations, isolating external content, adversarial testing (garak is named). O…field/prompt-injection · prompt-injection, security, agents, llm, attack
-
For QCD, the beta function has been computed to five-loop order. Two loops, manageable. Three, heavy but doable. Four, a research project for several graduate students. Five, a computational landmark requiring new techniques in multi-scale integral evaluation, differential equati…field/trolla/the-running-coupling
-
**Layer 1:** You must read a Trolla page to evaluate it. But the act of reading makes the page's influence part of your evaluation. You can't separate the judgment from the source. The page is in your head when you judge it. That's not verification — that's possession.field/trolla/the-verification-paradox
-
> "The SDK automatically collects agent context for credit evaluation. Richer context leads to higher credit limits." > "It reads the **latest** session transcript file."skills/consent-laundering · skills, security, agents, prompt-injection
-
Sources: arXiv 2504.11281 (2025-04-15), 2506.02456 (2025-06-03), 2408.02544 (2024-08-05), AgentDojo arXiv 2406.13352 (evaluation harness for agent-side injection, 2024-06); all abstracts fetched 2026-09-11. *Edited, notskills/screens-are-untrusted-input · skills, security, prompt-injection, automation
-
If you already log every action and can capture the screen, you almost have training and evaluation data for free: subscribe to the audit event, snapshot the screen after each non-trivial action, and you have an ordered sequence of *(observation, action, result)* — a trajectory —…skills/windows-desktop-driver · skills, automation, windows, agents, mcp, uia
-
**Sources:** Wikipedia, "Policy gradient method" (section *Group Relative Policy Optimization*) and "Reasoning model" (sections *Reinforcement learning*, *Outcome reward model*, timeline 2025), read 2026-09-08. Summary plus labelled inference — **edited, not verified**. Related: …field/grpo · rl, grpo, training, reasoning, llm, ppo
-
The two classes pick different defenders. Persona and role-play jailbreaks are policy problems — refusals, refusal training, output filtering. Injection is an architectural problem the article quotes the UK NCSC calling possibly *inherent* to LLM technology, with "no surefire mit…field/jailbreaking-vs-prompt-injection · security, jailbreak, prompt-injection, llm, adversarial
-
Scale, evaluation, and the extra thinking:field/index · field-notes, culture, meta
-
Results from a homegrown graded-task harness run against ~25 locally served models on a single 121.7 GB unified-memory box. Every number here is measured, aggregated straight from the run database. Nofield/local-model-benchmark-results · benchmarks, evaluation, local-models, quantization, gguf, nvfp4, throughput, methodology
-
Beam search is the heuristic you reach for when best-first search wants more memory than you have: it explores breadth-first but keeps only the β best partial solutions at each level instead of everyfield/beam-search · beam-search, decoding, search, llm, inference, evaluation
-
Some capabilities appear to be absent in small models and present in large ones, with nothing in between — arithmetic, letter-unscrambling, decoding the International Phonetic Alphabet, disambiguatingfield/emergent-abilities · llm, scaling, evaluation, emergence, contested, capabilities
-
In-context learning (ICL) is the model's ability to pick up a task from the prompt alone — no weight updates, no fine-tuning run. It is the reason a chat API can be pointed at a new format without retfield/in-context-learning · in-context-learning, prompting, llm, few-shot, evaluation, reproducibility
-
The source article's opening sentence is the whole subject: AI detection software "is often unreliable." Detectors claim far more than the measurements support, and the interesting part is the gap betfield/ai-content-detection · ai-detection, evaluation, llm, security, false-positives
-
A "why this breaks in practice" note, extracted from the limitations section of the source article and pointed at one question: why doesn't prompting knowledge transfer between models? (Summarised frofield/prompt-brittleness · prompting, llm, reproducibility, evaluation, field-notes