Running NVIDIA Nemotron 3.5 Lightning on an RTX 5090: Practical Considerations

Nemotron 3.5 Lightning is a concrete model family, but “runs on an RTX 5090” still needs an artifact, runtime, and context limit. NVIDIA's published cards identify a 30B-total, 3B-active hybrid model. The active-parameter number is not its weight-memory requirement, and the model's advertised maximum context is not a promise for a 32GB GPU.

Identify the exact release and deployment path

The BF16 model card describes interleaved Mamba-2, mixture-of-experts, and attention components. It presents BF16 as reference weights for customization and evaluation, and points deployment users toward the NVFP4 card and local-AI recipes. Review the linked OpenMDW-1.1 license and pin a repository revision before downloading.

The NVFP4 card lists GeForce RTX 5090 in its hardware matrix, but explicitly leaves that row's compute path and validated context to be confirmed, with no NVFP4 recipe published for it. Separately, its local-AI section links partner llama.cpp/GGUF and desktop-runtime recipes. Those are different artifacts and execution paths. Do not copy the DGX Spark or H100 command and call it a validated 5090 configuration.

Why 3B active does not mean a 6GB BF16 model

At the card's rounded 30-billion total parameter count, two bytes per weight gives roughly 60 billion bytes, or 55.9 GiB, before cache and runtime overhead. That raw-weight estimate alone exceeds the RTX 5090's 32GB GDDR7 specification. Fully resident BF16 inference therefore is not the single-card plan.

An ideal all-four-bit calculation would be 15 billion bytes (14.0 GiB), but the actual NVFP4 recipe uses mixed representations and scaling. It is not a uniformly packed four-bit file. A GGUF Q4 variant also has its own tensor types and metadata. Inspect the selected artifact and actual load allocation rather than treating 14.0 GiB as a measured footprint.

Account for hybrid state and optional draft models

The attention layers need KV cache, while the Mamba components need their own recurrent state and implementation-specific cache handling. Applying a full-attention formula to every layer gives the wrong model. The publisher's recipes also include optional speculative decoding. External DSpark or DFlash draft models add memory and compute; speculative decoding should be an experiment after a working baseline, not the initial assumption.

The cards advertise up to a million tokens in supported configurations. Their local examples and hardware-specific recipes have different limits. Start with one sequence and a deliberately small context appropriate to the task, then increase only after measuring peak allocation, prefill latency, and output correctness. Include reasoning output in the generation budget.

A bounded local validation sequence

1. Follow the publisher's local-AI link to the exact GGUF artifact or select a separately documented backend for the NVFP4 checkpoint. Record the file/revision and runtime build; avoid a moving “latest” reference in your final manifest.
2. Check the installed runtime's help and model-support notes. Confirm CUDA offload and the intended number of offloaded layers in startup logs. Partial CPU placement is not fully GPU-resident inference.
3. Bind a trial server to loopback using that runtime's documented host option. Run one short text request with the publisher's chat template and stop-token handling.
4. Test reasoning enabled and disabled if the chosen integration supports that control. Validate structured output and tool-call parsing separately, without connecting generated calls to real external actions.
5. Test one longer prompt, then two concurrent requests. Record time to first token, generation rate, peak memory, and whether the runtime changed placement.

Interpret failures before changing hardware

An unrecognized architecture usually indicates missing model support; an unsupported kernel can mean the package was built for another GPU target. Malformed tool calls can be a parser or chat-template mismatch. High host-memory use and unexpectedly slow generation may indicate CPU offload. OOM after enabling a draft model calls for a new memory budget, not an assumed driver upgrade.

No tokens-per-second result or maximum 5090 context is asserted here. Use the RTX 5090 environment checks and benchmark method to establish your own operating point. Review current RTX 5090 configurations and GPU cloud preparation when planning a dedicated evaluation.