Local AI Glossary

The maximum number of tokens a model can process at once — its working memory.

The GPU memory standard used in RTX 30 and 40-series high-end cards, delivering up to 1,008 GB/s.

The latest GPU memory standard, used in RTX 50-series cards, with roughly double GDDR6X bandwidth.

Memory storage for the key-value attention states of all tokens in your current context.

Low-power high-bandwidth RAM used in Apple Silicon chips as the unified memory substrate.

How fast data moves between memory and the processor, measured in GB/s.

General-purpose computer memory used by the CPU and OS — distinct from VRAM, but relevant for LLM offloading and CPU-only inference.

General-purpose computer memory shared by the CPU and OS — slower than VRAM for GPU inference, but essential for CPU-only setups.

A single memory pool shared by the CPU and GPU on Apple Silicon chips.

Dedicated high-speed memory on your GPU that stores model weights during inference.

Running some model layers in VRAM and the rest in system RAM when the model is too large to fit entirely on the GPU.

NVIDIA's parallel computing platform that enables GPU-accelerated AI workloads on GeForce and data center cards.

AMD's open-source GPU compute platform — the AMD equivalent of CUDA for running AI workloads on Radeon GPUs.

Specialized compute units inside NVIDIA GPUs designed for matrix multiplication — the core operation in neural network inference.

Google's 16-bit floating-point format with the same exponent range as FP32 — preferred over FP16 for training and increasingly common in inference.

The number of bits used to store each model parameter, determining model size in memory.

A dense numerical vector that represents text in a high-dimensional space — the foundation of semantic search and RAG systems.

Training a pre-trained model on additional data to specialize its behavior, improve task performance, or adjust its output style.

16-bit floating-point format used for AI model weights — half the memory of FP32 with minimal quality loss for inference.

The predecessor file format to GGUF for storing quantized LLMs, used by early versions of llama.cpp.

The standard file format for quantized LLMs, used by llama.cpp, Ollama, and LM Studio.

4-bit integer quantization format — the practical minimum precision for running large language models on consumer hardware.

A neural network trained on large amounts of text that can generate, summarize, translate, and reason about language.

Parameter-efficient fine-tuning technique that adds small trainable weight matrices to a frozen base model — enabling custom model training on consumer hardware.

The number of learned numerical weights in a model — the primary predictor of capability and VRAM requirement.

Architecture where a model has many specialized sub-networks (experts) but activates only a subset per token — enabling larger models with lower compute costs.

AI models that process and generate more than one type of data — typically text plus images, audio, or video.

Reducing a model's numerical precision to shrink its memory footprint with minimal quality loss.

High-performance inference library optimized for NVIDIA GPUs, known for fast quantized inference and support for EXL2 quantization format.

Inference server with a web UI designed for creative writing and roleplay — built on llama.cpp with additional sampling controls and story management features.

A C++ inference engine that runs quantized GGUF models on CPU, GPU, or both simultaneously.

Desktop application for downloading and running local LLMs with a graphical interface — the easiest entry point for local AI on Windows and macOS.

Apple's machine learning framework optimized for Apple Silicon — enables fast local LLM inference on M-series Macs using unified memory.

A tool that makes running local LLMs as simple as a single terminal command.

The number of requests processed simultaneously during inference — higher batch sizes improve GPU utilization but increase latency per request.

The token generation phase of LLM inference — the rate at which output tokens stream out.

An algorithm that computes attention more efficiently by reducing VRAM reads, speeding up prefill and enabling longer context.

The phase where the model processes your input prompt before generating any output.

Reusing the computed KV cache state from a previous request's prefix — eliminating redundant compute for repeated system prompts or context.

Inference acceleration technique that uses a smaller draft model to generate candidate tokens, verified in parallel by the main model — reducing effective latency.

The primary speed metric for LLM inference — how many tokens the model generates each second.