GPU hosting for AI inference: VRAM and serving requirements
Estimate model weights, context cache and concurrency before choosing a 24, 48 or 80 GB GPU. Plan authentication and a representative inference test.
Content revised · English
- Architecture to evaluate
- Model weights / KV cache / runtime
- Prepared starting points
- 5 plans
- Responsibilities
- Application, rights and recovery
A fit shortlist, not installed software or live inventory.
Keep these in view.
- Model weights / KV cache / runtime.
- Quantisation changes the envelope.
- Keep serving private enough for its inputs.
Model weights / KV cache / runtime
Start with model memory, context length and concurrency. FP16 weights use roughly two bytes per parameter: 13 billion parameters require about 26 GB for weights alone, and 70 billion require about 140 GB. A 24 GB device cannot hold the former in FP16 and an 80 GB device cannot hold the latter in FP16 without additional techniques.
Quantisation changes the envelope
A four-bit weight estimate is roughly 0.5 bytes per parameter before metadata, runtime and KV cache. It can make a model smaller but changes its deployment and quality tradeoffs. Benchmark the actual model, context, batch size and engine rather than equating the card’s VRAM with usable model capacity.
Keep serving private enough for its inputs
Ollama or vLLM can be evaluated for a compatible stack. Put authentication, TLS and quotas around any public API, limit prompt logging and define retention. Check model and dataset licences. A GPU listing does not grant access to restricted models or an export-law exemption.
Make the inference test reproducible
Record the model version and licence, serving-engine version, precision or quantisation, prompt and output length, concurrent requests and the exact device. Measure loaded device memory, time to first token, sustained output rate, errors and queue behaviour. Compare the same inputs on every candidate; the prepared hardware listing supplies no measured inference result.
| Input to hold constant | Result to measure |
|---|---|
| Model, precision and engine version | Loaded GPU memory and startup success |
| Prompt length and requested output length | Time to first token and output rate |
| Concurrency and request arrival pattern | Queue depth, memory growth and error rate |
| User route and application gateway | End-to-end response time |
| Parameter count | FP16 at ~2 bytes / parameter | 4-bit at ~0.5 bytes / parameter |
|---|---|---|
| 7B | ~14 GB | ~3.5 GB |
| 13B | ~26 GB | ~6.5 GB |
| 70B | ~140 GB | ~35 GB |
Capacity to consider.
Match these starting points to the representative application. Software installation and live inventory are not connected; the exact specification and period total come from the catalogue.
Spectre 4090
Pair one NVIDIA RTX 4090 with 24 GB of GPU memory, 64 GB of host memory and 1 TB of NVMe storage for inference, rendering and experiments. CUDA image options and a 10 Gbps uplink support your software stack; model fit depends on precision, context length and runtime overhead.
- Compute
- 8 vCPU (Threadripper)
- Memory
- 64 GB DDR5
- Storage
- 1 TB NVMe SSD
- Transfer
- 20 TB / month
billed monthly
Spectre A6000
Use one NVIDIA RTX A6000 with 48 GB of GPU memory alongside 128 GB of ECC host memory and 2 TB of NVMe storage. The larger GPU memory allocation and 10 Gbps uplink suit more demanding inference or selected training workflows, with model and optimiser memory assessed before deployment.
- Compute
- 16 vCPU (EPYC)
- Memory
- 128 GB DDR5
- Storage
- 2 TB NVMe SSD
- Transfer
- 30 TB / month
billed monthly
Spectre H100
Build intensive accelerator workloads around one NVIDIA H100 with 80 GB of HBM3, 256 GB of host memory and 4 TB of NVMe storage. A 25 Gbps uplink and 50 TB monthly transfer allowance support large datasets; high-parameter models still require careful precision, context and memory planning.
- Compute
- 32 vCPU (EPYC)
- Memory
- 256 GB DDR5
- Storage
- 4 TB NVMe SSD
- Transfer
- 50 TB / month
billed monthly
Eclipse EPYC
Consolidate larger workloads on a physical EPYC 7443P server with 24 cores, 48 threads and 128 GB of ECC memory. Two 2 TB NVMe drives in RAID-1 pair with a 10 Gbps uplink, 100 TB of monthly transfer and a 200 Gbps network mitigation tier.
- Compute
- AMD EPYC 7443P (24c / 48t @ 2.85 GHz)
- Memory
- 128 GB DDR4 ECC
- Storage
- 2× 2 TB NVMe (RAID-1)
- Transfer
- 100 TB / month
billed monthly
Eclipse Dual
Reserve two Xeon Gold 6338 processors for a total of 64 physical cores and 128 threads, supported by 256 GB of ECC memory. Four 4 TB NVMe drives form RAID-10, with a 10 Gbps port, unmetered transfer and a 400 Gbps network mitigation tier for large parallel workloads.
- Compute
- 2× Intel Xeon Gold 6338 (32c / 64t each)
- Memory
- 256 GB DDR4 ECC
- Storage
- 4× 4 TB NVMe (RAID-10)
- Transfer
- Unmetered
billed monthly
Before this workload goes live.
- Estimate weights, context cache, runtime overhead and concurrency together.
- Verify the exact plan's device memory, compatible image and region.
- Benchmark the representative model and precision on an actual deployment.
- Protect the serving endpoint with authentication, TLS and resource limits.
- Define prompt retention, model rights and the recovery path.
Questions, answered.
Which model fits an 80 GB GPU?
Use the actual precision, weights, context, cache and batch size. A 70B FP16 weights-only estimate is about 140 GB, so it does not fit on an 80 GB device without additional techniques.
Can I expose an OpenAI-style application API?
A compatible serving engine can be evaluated, with authentication, TLS, quotas and request-retention controls. No serving engine or public endpoint is created by the checkout.
Is the GPU bare metal or virtualised?
Read the exact plan’s virtualisation and access specification. Device allocation, driver support and isolation need a verified launch platform; do not infer them from VRAM alone.
Will response latency beat an external API?
No measured comparison is made. Benchmark the representative model, prompt length, concurrency and user route on the actual deployment.
What is kept in my account workspace?
Your configuration, selected plan, image, location and billing term are kept with the order and payment request references. Quotes, payment-state reads and support conversations remain linked to your account. Read fulfilment details separately from payment status.