Local AI Glossary

The maximum number of tokens a model can process at once — its working memory.

AMD's memory overclocking profile standard for DDR5, equivalent to Intel's XMP — lets RAM run at advertised speeds instead of slower JEDEC defaults.

The GPU memory standard used in RTX 30 and 40-series high-end cards, delivering up to 1,008 GB/s.

The latest GPU memory standard, used in RTX 50-series cards, with roughly double GDDR6X bandwidth.

Memory storage for the key-value attention states of all tokens in your current context.

Low-power high-bandwidth RAM used in Apple Silicon chips as the unified memory substrate.

How fast data moves between memory and the processor, measured in GB/s.

General-purpose computer memory used by the CPU and OS — distinct from VRAM, but relevant for LLM offloading and CPU-only inference.

General-purpose computer memory shared by the CPU and OS — slower than VRAM for GPU inference, but essential for CPU-only setups.

A Google Research KV cache compression technique that extends usable context length 4-5x on consumer GPUs without retraining the model.

A single memory pool shared by the CPU and GPU on Apple Silicon chips.

Dedicated high-speed memory on your GPU that stores model weights during inference.

Running some model layers in VRAM and the rest in system RAM when the model is too large to fit entirely on the GPU.

Intel's Extreme Memory Profile, a preset stored on DDR memory modules that lets the BIOS run RAM at its advertised speed and timings instead of conservative JEDEC defaults.

Apple's ARM-based system-on-chip family (M-series) that integrates CPU, GPU, and unified memory on a single die, used in Macs for local LLM inference.

NVIDIA's GPU architecture powering the RTX 50-series consumer cards and RTX Pro 6000 workstation GPUs, designed with GDDR7 memory and updated Tensor Cores for AI workloads.

The general-purpose processor that handles model layers and KV-cache when they don't fit in GPU VRAM. In local AI it's the slow fallback path, not the workhorse.

NVIDIA's parallel computing platform that enables GPU-accelerated AI workloads on GeForce and data center cards.

A graphics processing unit — the parallel-compute chip that runs the matrix math behind local LLM inference. For local AI, the GPU's onboard memory and bandwidth matter more than its gaming performance.

NVIDIA's data-center GPU architecture launched in 2022, powering the H100 and H200 accelerators that dominate large-scale AI training and inference.

A neural processing unit is a dedicated chip block built to accelerate small AI workloads at low power, separate from the CPU and GPU.

A high-bandwidth NVIDIA interconnect that lets two GPUs share data directly, bypassing the slower PCIe bus.

The high-speed slot standard that connects GPUs, NVMe drives, and other expansion cards to your motherboard. Each generation roughly doubles the bandwidth of the last.

AMD's next-generation GPU architecture, expected to launch in mid-2027 as the successor to RDNA 4.

AMD's open-source GPU compute platform — the AMD equivalent of CUDA for running AI workloads on Radeon GPUs.

Specialized compute units inside NVIDIA GPUs designed for matrix multiplication — the core operation in neural network inference.

NVIDIA's next-generation hyperscaler GPU platform announced at GTC 2026, succeeding Hopper and integrating Groq's LPU inference technology.

Activation-aware Weight Quantization — a 4-bit model compression format that protects the most important weights to preserve accuracy while shrinking VRAM footprint.

Google's 16-bit floating-point format with the same exponent range as FP32 — preferred over FP16 for training and increasingly common in inference.

The number of bits used to store each model parameter, determining model size in memory.

A model architecture where every parameter activates on every token, as opposed to mixture-of-experts designs that only fire a subset.

A dense numerical vector that represents text in a high-dimensional space — the foundation of semantic search and RAG systems.

A GPU-only quantization format from the ExLlamaV2 project that supports fractional, mixed bits-per-weight for fast local LLM inference on NVIDIA cards.

Training a pre-trained model on additional data to specialize its behavior, improve task performance, or adjust its output style.

16-bit floating-point format used for AI model weights — half the memory of FP32 with minimal quality loss for inference.

The predecessor file format to GGUF for storing quantized LLMs, used by early versions of llama.cpp.

The standard file format for quantized LLMs, used by llama.cpp, Ollama, and LM Studio.

A post-training quantization method that compresses LLM weights to 4-bit (or lower) precision for GPU inference, reducing VRAM use with minimal accuracy loss.

4-bit integer quantization format — the practical minimum precision for running large language models on consumer hardware.

A neural network trained on large amounts of text that can generate, summarize, translate, and reason about language.

Parameter-efficient fine-tuning technique that adds small trainable weight matrices to a frozen base model — enabling custom model training on consumer hardware.

The number of learned numerical weights in a model — the primary predictor of capability and VRAM requirement.

Architecture where a model has many specialized sub-networks (experts) but activates only a subset per token — enabling larger models with lower compute costs.

AI models that process and generate more than one type of data — typically text plus images, audio, or video.

A 4-bit quantization format for GGUF models that shrinks weights to roughly a quarter of their FP16 size while preserving most of the model's quality.

Reducing a model's numerical precision to shrink its memory footprint with minimal quality loss.

Pre-packaged retrieval-augmented generation tools that hide chunking, embedding, and retrieval logic behind a simple UI, often at the cost of correctness and privacy.

High-performance inference library optimized for NVIDIA GPUs, known for fast quantized inference and support for EXL2 quantization format.

A complete retrieval-augmented generation pipeline that chunks documents, embeds them into a vector database, and retrieves relevant context at query time to ground LLM responses.

AMD's GPU-accelerated BLAS library that lets llama.cpp and similar runtimes offload matrix math to Radeon and Instinct cards through the ROCm stack.

AMD's HIP C++ compiler driver, used to build GPU-accelerated code that runs on Radeon and Instinct cards via the ROCm stack.

Inference server with a web UI designed for creative writing and roleplay — built on llama.cpp with additional sampling controls and story management features.

A C++ inference engine that runs quantized GGUF models on CPU, GPU, or both simultaneously.

Desktop application for downloading and running local LLMs with a graphical interface — the easiest entry point for local AI on Windows and macOS.

A large language model that runs entirely on your own hardware — no cloud API, no per-token billing, no data leaving the machine.

Apple's machine learning framework optimized for Apple Silicon — enables fast local LLM inference on M-series Macs using unified memory.

A Linux command-line tool for binding processes to specific NUMA nodes, controlling which CPU cores and memory banks a workload uses on multi-socket systems.

NVIDIA's CUDA compiler driver — the toolchain component that compiles CUDA C++ source into GPU-executable code.

Command-line utility bundled with NVIDIA drivers that reports GPU utilization, VRAM usage, temperature, and power draw in real time.

A tool that makes running local LLMs as simple as a single terminal command.

Intel's open, cross-architecture programming toolkit for running compute workloads on Intel CPUs, GPUs, and accelerators. It's the Intel-side analog to CUDA or ROCm for local LLM inference on Arc cards.

Retrieval-Augmented Generation: a technique where an LLM pulls relevant text from an external document store at query time and uses it as context for its answer.

An open-source deep learning framework from George Hotz's tiny corp, designed as a minimal, hardware-agnostic alternative to PyTorch and CUDA for running and training neural networks.

Tenstorrent's open-source compiler and runtime stack for running ML models on its RISC-V-based Tensix accelerators, including the QuietBox 2 workstation.

A cross-vendor graphics and compute API used by llama.cpp as a portable GPU backend when CUDA or ROCm aren't available.

Windows Display Driver Model — the GPU driver framework Windows uses to share and manage graphics memory, which on WSL2 reserves a slice of your VRAM before any LLM ever loads.

The original Windows Subsystem for Linux that translates Linux syscalls to Windows kernel calls, running without a virtual machine layer.

Windows Subsystem for Linux 2 — a lightweight VM that runs a real Linux kernel inside Windows, letting you use Linux-native AI tooling without dual-booting.

The number of requests processed simultaneously during inference — higher batch sizes improve GPU utilization but increase latency per request.

The token generation phase of LLM inference — the rate at which output tokens stream out.

An algorithm that computes attention more efficiently by reducing VRAM reads, speeding up prefill and enabling longer context.

Running a trained model to generate output — the part of the LLM lifecycle that produces tokens in response to a prompt.

The phase where the model processes your input prompt before generating any output.

Reusing the computed KV cache state from a previous request's prefix — eliminating redundant compute for repeated system prompts or context.

Inference acceleration technique that uses a smaller draft model to generate candidate tokens, verified in parallel by the main model — reducing effective latency.

Splitting individual model layers across multiple GPUs so each card holds a slice of every weight matrix and the layers compute in parallel.

The primary speed metric for LLM inference — how many tokens the model generates each second.

Describes data, workloads, or environments that handle protected health information (PHI) and must comply with HIPAA's privacy and security rules.