LLaMA 2 Local Inference (Historical): Memory and Runtime Planning
A historical, reproducible planning guide for Llama 2 inference, with explicit license, memory, context, and runtime caveats.
By Hostnot GPU ·
Historical model note: Llama 2 is Meta's 2023 text-model family, with 7B, 13B, and 70B base and chat variants. This guide is for reproducing an older workload, not a recommendation over current models. Use the exact Llama 2 Community License and acceptable-use terms; open weights do not mean an unrestricted license.
Choose an artifact and a reproducible environment
The example targets the official Hugging Face-format meta-llama/Llama-2-7b-chat-hf repository on one CUDA GPU. Obtain authorized repository access first; an HTTP 401/403 is an access problem, not a CUDA problem. Use the Hub's interactive login or an approved secret mechanism with minimum necessary read access. Never embed a token in the example or upload it in a support ticket.
Prepare an isolated environment with a verified CUDA PyTorch build, a compatible pinned Transformers 4.x release providing the tokenizer's chat-template API, and its documented dependencies. Record exact resolved versions and the immutable model commit. Use the PyTorch smoke test before downloading weights. There is no automatic CPU offload in the example; enough host RAM, disk, and GPU headroom are prerequisites.
Budget memory beyond weights
As approximate planning arithmetic, 7 billion parameters at two bytes per parameter occupy 14 GB decimal (about 13 GiB) for weights alone. The exact checkpoint's count, buffers, temporary loading allocations, runtime state, and KV cache add to that. This is not a claim that a 14 GB GPU can run the model. Quantized derivatives need their own supported loader, metadata overhead, and quality test.
The original family was trained with a 4,096-token context. Input, chat-control tokens, and generated output share the context budget. Do not borrow longer-context claims from Llama 3 or an extended-context derivative. Llama 2 70B uses grouped-query attention; do not assume its KV-cache layout is identical to 7B/13B.
Generate one short answer
Set MODEL_REVISION to the approved immutable model commit in your shell. Save this as llama2_smoke.py and run it in the isolated environment. The native architecture is used with remote code disabled. This performs a model download if it is not cached.
import os
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
assert torch.cuda.is_available(), "Verify the CUDA environment first"
model_id = "meta-llama/Llama-2-7b-chat-hf"
revision = os.environ["MODEL_REVISION"]
tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
model_id, revision=revision, torch_dtype=torch.float16,
use_safetensors=True, trust_remote_code=False,
).to("cuda").eval()
messages = [{"role": "user", "content": "Explain what a GPU does in two sentences."}]
ids = tokenizer.apply_chat_template(messages, tokenize=True,
add_generation_prompt=True, return_tensors="pt").to("cuda")
assert ids.shape[-1] + 64 <= 4096, "Prompt plus output exceeds this test's context budget"
with torch.inference_mode():
output = model.generate(ids, attention_mask=torch.ones_like(ids),
max_new_tokens=64, do_sample=False, pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(output[0, ids.shape[-1]:], skip_special_tokens=True))
print("peak allocated GiB:", torch.cuda.max_memory_allocated() / 2**30)Success means a plausible answer, a valid stopping behavior, and no device errors. Peak allocated memory is PyTorch allocator telemetry, not the complete device reservation. Check it alongside nvidia-smi. If the tokenizer revision lacks a chat template, use the exact publisher-reviewed Llama 2 chat format; do not borrow a Mistral or later Llama template.
Troubleshoot and evaluate
OOM during load indicates insufficient weight/loading headroom; OOM only on longer inputs points toward context/cache demand. Repetition or echoed instructions suggests template/tokenizer mismatch before a hardware issue. Start with a fresh process after a device-side failure. Do not claim deterministic cross-hardware text merely because greedy decoding is selected.
Evaluate factuality, language coverage, and task-specific safety on your own fixed prompts. Its historical knowledge and behavior are limitations, and its outputs are not authoritative. For cost planning, include gated-access preparation, downloads, cold load, and repeated trials. Cache only where license and storage policy permit, save results, and terminate unused rentals. Next: quantization tradeoffs or a separately secured serving runtime.