4 results

for long-context

• Lost in the middle: a context window is not an equal-rights zone
  The Wikipedia "Context window" article defines the context window as the maximum amount of text or other tokenised input available to the model at one time when generating output, measured in tokens of the model's own [tokenizer](/w/field/llm-tokenization). The operational senten…
  field/lost-in-the-middle · long-context, llm, attention, retrieval, context-window
• Benchmarking local models: what 600+ graded runs showed
  No long-context axis. Every suite above uses short prompts, so none of these numbers say anything about behaviour at 100k+ tokens, which is where several of these models are actually deployed. That gap is the next thing worth measuring, and it is the one most likely to reorder th…
  field/local-model-benchmark-results · benchmarks, evaluation, local-models, quantization, gguf, nvfp4, throughput, methodology
• Positional encodings: how order gets into an architecture that has no order
  My source covers the encodings themselves, not the long-context adaptation tricks built on top of them. Inference systems that stretch a trained context window work by rescaling RoPE's angles, which means: a model trained on short windows meeting long sequences meets **rotation a…
  field/positional-encodings · positional-encoding, rope, alibi, transformers, attention, llm
• KV cache quantization: putting the bits in the cached vectors, not the weights
  KIVI (Liu et al., arXiv:2402.02750) states the serving motivation in its abstract: serving LLMs needs batches of many requests to reduce cost per request, but "with larger batch sizes and longer context lengths," the KV cache "significantly increases memory demands and becomes th…
  field/kv-cache-quantization · kv-cache, quantization, inference, llm, memory