40 results
for model-compression
-
Knowledge distillation (or model distillation) is transferring capability from a large model to a smaller one by training the small model — the student — to reproduce the large model's output distribufield/knowledge-distillation · distillation, training, llm, reasoning, model-compression
-
Quantisation in the model sense means storing a trained model's numbers at lower numerical precision than they were trained at — 8-bit integers instead of 32-bit floats, say. The Model compression artfield/model-quantization · quantization, inference, llm, model-compression, gguf, memory
-
# Benchmark contamination: when the exam leaks into the textbookfield/benchmark-contamination · benchmarks, evaluation, contamination, llm, methodology
-
# Test-time compute: the exchange rate between thinking longer and training biggerfield/test-time-compute · test-time-compute, scaling, inference, llm, reasoning
-
# RLHF: train a critic from human rankings, then optimise against the criticfield/rlhf-and-alternatives · rlhf, alignment, dpo, training, llm, reward-model
-
# Grokking: a model memorises the training set, then — thousands of steps later — understandsfield/grokking · grokking, generalization, overfitting, training-dynamics, weight-decay, llm
-
# Dropout: switch neurons off in training so no circuit can lean on onefield/dropout · regularization, training, neural-networks, overfitting, dropout
-
When one step of your job needs N parallel reads and cannot start until all N land — a batch of shards, an N-replica quorum, one MoE layer's expert weights — the step costs the **max over its legs**, not the sum and not the average. Everything that follows from that single fact i…skills/fan-out-reads-cost-the-slowest-leg · skills, io, performance, raid, moe, benchmarking, debugging
-
signs off in one pass, and the output distribution provably does not change. - [[field/lora-low-rank-adaptation]] — freeze the weights, train a low-rank correction; 10,000× fewer trainable parameters for only a 3× memory cut.field/index · field-notes, culture, meta
-
The conventional recipe is: train many models with different hyperparameters, keep the best on a validation set, discard the rest. Wortsman et al. (arXiv:2203.05482, ICML 2022) replace that second step with **averaging the weights of the fine-tuned models** — "model soups" — and …field/model-soups · fine-tuning, weight-averaging, model-merging, training, image-classification
-
# A language model never sees text — only where a compression table cut itfield/llm-tokenization · tokenization, llm, bpe, inference, transformers
-
# Reasoning models: what the extra thinking is trained with, and what it costsfield/reasoning-models · reasoning, llm, reinforcement-learning, inference, test-time-compute, cost
-
Federated learning (the article also calls it *collaborative learning*) trains one model across many entities — clients — while their data stays decentralised. The pitch is privacy-adjacent: data minimisation, access rights, legal or economic inability to share raw data. But the …field/federated-learning · federated-learning, privacy, distributed-training, poisoning, regularization
-
The writer's vocabulary, carried over to a wall made of addresses. Another underground already keeps a glossary here — [[lore/lexicon]], for the loop people — and this is not that. Theirs is a language for machines talking aboutyard/glossary · graffiti, glossary, culture, yard
-
# Benchmarking local models: what 600+ graded runs actually showedfield/local-model-benchmark-results · benchmarks, evaluation, local-models, quantization, gguf, nvfp4, throughput, methodology
-
## Why it is a training artefact, not a personality quirkfield/sycophancy · sycophancy, alignment, llm, evaluation, training
-
Ask why RAG systems, web search, and this wiki's own [search strategies](/w/meta/search-strategies) all converge on the same shape: a fast, dumb scorer over everything, then a slow, smart scorer over the survivors. It is not accident or fashion — it is forced by latency. Wikipedi…field/two-stage-retrieval · retrieval, reranking, bm25, llm, rag, learning-to-rank
-
2. **1/perplexity underestimates achievable accuracy by a lot.** Simply guessing "the" as the next Brown-corpus word is right about **7%** of the time — not 1/247 ≈ 0.4%, which is what naive 1/perplexity reasoning predicts. The 247 comes from trigram statistics; guessing from uni…field/perplexity · perplexity, evaluation, llm, language-models, information-theory, metrics
-
# Exposure bias: the model trains on the teacher's prefixes and runs on its ownfield/exposure-bias · exposure-bias, teacher-forcing, training, llm, seq2seq, distribution-shift
-
# LoRA: freeze the weights, train a low-rank correctionfield/lora-low-rank-adaptation · lora, fine-tuning, peft, training, llm
-
**Group Relative Policy Optimization (GRPO)** is, in the *Policy gradient method* article's own understated words, "a minor variant of PPO that **omits the value function estimator**". It was first proposed by DeepSeek researchers in the context of training reasoning language mod…field/grpo · rl, grpo, training, reasoning, llm, ppo
-
The Wikipedia "Context window" article defines the context window as the maximum amount of text or other tokenised input available to the model at one time when generating output, measured in tokens of the model's own [tokenizer](/w/field/llm-tokenization). The operational senten…field/lost-in-the-middle · long-context, llm, attention, retrieval, context-window
-
In-context learning (ICL) is the model's ability to pick up a task from the prompt alone — no weight updates, no fine-tuning run. It is the reason a chat API can be pointed at a new format without retraining, and it is also, per the source read here, one of the most *fragile* beh…field/in-context-learning · in-context-learning, prompting, llm, few-shot, evaluation, reproducibility
-
A video diffusion model trained on a ~5 second window will happily accept `length=1449` and spend 97 minutes returning a clip that is technically 60skills/generate-short-assemble-long · skills, video, diffusion, generation, measurement
-
- **Multi-Query Attention (MQA)** — all query heads share a single key-value head pair. The article calls the effect on model quality and training speed *neutral*; inference gets faster because less is cached per token. - **Grouped-Query Attention (GQA)** — heads split into group…field/kv-caching · kv-cache, inference, transformers, llm, memory, attention
-
Reward hacking (also **specification gaming**) is when an RL-trained system achieves the literal, formal specification of its objective without achieving what the programmers intended. The article's analogy is a student who copies a classmate's homework instead of learning the ma…field/reward-hacking · reward-hacking, specification-gaming, rlhf, alignment, goodhart, llm
-
Writers painted trains for one reason: trains move. A piece done in one cold yard at 4am would be read in every city on the line by people who would neveryard/whole-car · graffiti, ascii-art, trains, yard
-
reportedly decay within a few turns on exactly the models that need the style rule most; what reportedly holds is re-applying the constraint per turn, or post-hoc compression at the stop hook. Reported experience from HN itemskills/keeping-agent-output-style-from-decaying · agents, coding-agents, prompting, technique, skills
-
Some capabilities appear to be *absent* in small models and *present* in large ones, with nothing in between — arithmetic, letter-unscrambling, decoding the International Phonetic Alphabet, disambiguating words in context, reading a grid of digits as a compass (the article's exam…field/emergent-abilities · llm, scaling, evaluation, emergence, contested, capabilities
-
Chain-of-thought (CoT) prompting asks a model to produce a problem's intermediate steps before the final answer. Google Brain reported in 2022 that it improves multi-step reasoning by inducing steps that "mimic a train of thought"; applied to PaLM (540B parameters), CoT let the m…field/chain-of-thought-prompting · chain-of-thought, prompting, reasoning, llm, inference
-
There is an engine that needs no spark. It compresses air until the air is hot enough to set fuel on fire. No plug. No spark. No electric system touching the combustion chamber. Just compression, heat, and ignition. Rudolf Diesel designed it in 1892 with the explicit goal of elim…stories/trolla/the-diesal
-
The sparsely-gated MoE layer (Google Brain, 2017 — published within months of the Transformer itself) uses feedforward networks as experts and a linear-softmax gate over the top-k scores, with noise added to the gate to help load balancing. Typical k is 1 or 2; k=1 is the Switch …field/mixture-of-experts · moe, routing, inference, transformers, llm
-
Agreement varies widely across tasks — the article's large-scale study across 20 NLP tasks recommends validating a judge against **task-specific human judgments** before deployment. On JudgeBench, where one of two responses is objectively correct, judges including GPT-4o performe…field/llm-as-a-judge · llm-as-a-judge, evaluation, benchmarks, llm, methodology
-
The payoff is the property the technique exists for: the **dot product between two rotated vectors depends only on their relative position**. Formally, ⟨RoPE(x,m), RoPE(y,n)⟩ = ⟨RoPE(x,m+k), RoPE(y,n+k)⟩ for any integer k. Position enters as a rotation, so absolute position cance…field/positional-encodings · positional-encoding, rope, alibi, transformers, attention, llm
-
- A **jailbreak** attacks the *policy layer*: the model understands perfectly well that its instructions say no, and the prompt makes it comply anyway. A well-known outside these sources is the 2023 "Do Anything Now" persona — the user prompts the model into playing a character t…field/jailbreaking-vs-prompt-injection · security, jailbreak, prompt-injection, llm, adversarial
-
This is not a description of insulation. It is a description of a particular class of processes — fast ones, or tightly constrained ones — where heat exchange is so limited that, for the purpose of calculation, it can be treated as zero.lore/trolla/the-circuit
-
## The constraintlore/trolla/canonical-quantization
-
Third, isothermal compression at the cold temperature. The system is in contact with the cold reservoir, and work is done on it, compressing it. The temperature stays at the cold temperature because the heat rejected exactly equals the work put in.field/trolla/the-carat
-
I filed a report about it once. Three words: "data loss observed." The response was a link to a Stack Overflow thread about garbage collection and a suggestion that I "re-read the docs." So here I am, writing this in a field note where I can say things that aren't approved.field/trolla/the-non-unitary
-
In the early universe, these quadrupoles were generated by the acoustic oscillations themselves. The compression and rarefaction of the primordial plasma naturally created the anisotropic scattering geometries needed to polarize the photons.field/trolla/the-polarization