How to Connect to a Cloud GPU with SSH

A security-first connection workflow using an SSH key, allocated port, and host fingerprint verification.

By Hostnot GPU · 9/6/2026

Use this walkthrough after a dedicated instance is ready and its public key has been registered. SSH opens a shell on the rented machine; it does not itself install CUDA, launch a training job, or stop billing when you disconnect. Complete the access check before downloading large models.

Collect the current connection details

You need an OpenSSH client, the matching local private key, and the assigned host, port, username, and host fingerprint from the authenticated workspace. The assigned SSH port can differ from 22 and can change after replacement. Copy the username too rather than guessing root or ubuntu. The examples below use a Linux/macOS/WSL shell; the variables must be replaced before use.

GPU_HOST='replace-with-assigned-host'
GPU_PORT='replace-with-assigned-port'
GPU_USER='replace-with-assigned-user'
KEY="$HOME/.ssh/hostnotgpu_ed25519"
ssh -o IdentitiesOnly=yes -o StrictHostKeyChecking=ask -o ConnectTimeout=15 -i "$KEY" -p "$GPU_PORT" "$GPU_USER@$GPU_HOST"

Compare the presented host-key algorithm and SHA256 fingerprint with the workspace's trusted value. Stop if it differs or the expected value is unavailable. Do not substitute StrictHostKeyChecking=no. A passphrase prompt unlocks your local private key; it is not the remote account password.

Check where you landed

Run these read-only commands inside the remote shell. Match the device count and memory to the actual allocation, allowing for documented partitions or container visibility. A shell can work while GPU provisioning is incomplete.

whoami
hostname
uname -r
nvidia-smi -L
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv
df -h .

nvidia-smi should identify the assigned devices. A missing command can mean the selected image lacks the utility; a driver-communication error needs a host check. Neither is a reason to install a driver over a managed image. Also identify which directory is persistent before writing valuable checkpoints.

Copy a small file first

From a second local terminal with the same variables set, test a small, non-sensitive file. Unlike ssh -p, scp takes an uppercase -P for its port.

scp -P "$GPU_PORT" -i "$KEY" -o IdentitiesOnly=yes ./smoke-input.txt "$GPU_USER@$GPU_HOST:smoke-input.txt"
sftp -P "$GPU_PORT" -i "$KEY" "$GPU_USER@$GPU_HOST"

The destination is relative to the remote account's home. For large artifacts, use a resumable transfer method allowed by the environment and verify file hashes at each end. Do not put private SSH keys into the remote project directory.

Forward a local-only development service

If a service is already listening on remote 127.0.0.1:8000, this local command creates a tunnel. It requires SSH forwarding to be permitted by the server.

ssh -N -o ExitOnForwardFailure=yes -o IdentitiesOnly=yes -i "$KEY" -p "$GPU_PORT" -L 127.0.0.1:8000:127.0.0.1:8000 "$GPU_USER@$GPU_HOST"

Visit http://127.0.0.1:8000 locally. If that local port is occupied, change only the first 8000. A tunnel does not start the service, and successful tunnel creation does not prove the service is healthy. Loopback avoids a public listener but still permits access by other users on the local or remote machine; use application authentication when required.

Finish deliberately

Use exit to close a shell and Ctrl+C to close a foreground tunnel. Interactive jobs can end when the session disconnects; use an approved job supervisor or terminal multiplexer for long work, with checkpoints on verified storage. Conversely, a detached job can continue after logout. Save outputs, perform the documented instance termination when finished, and verify resulting status and usage. For timeout, refused connection, or key warnings, follow the layered SSH diagnosis.