Start with memory, not speed
If a model does not fit in GPU memory, it either fails to load or runs far slower while it swaps. So the first question is how much memory the model needs, and only then how fast the card is.
- 16-bit (FP16 or BF16): about 2 GB per billion parameters, plus room for context.
- 8-bit: about 1 GB per billion parameters.
- 4-bit quantised: about 0.5 to 0.7 GB per billion parameters.
Leave 20 to 30% extra for the context window and batch size. Long prompts and many users at once use more memory.
Which card fits which model
| Model size | 4-bit memory | 16-bit memory | Good card |
|---|---|---|---|
| 7B–8B | about 5 GB | about 16 GB | NVIDIA L4, 24 GB |
| 13B–14B | about 9 GB | about 28 GB | L4 at 4-bit, or RTX PRO 4000 |
| 30B–34B | about 20 GB | about 68 GB | RTX PRO 6000, 96 GB |
| 70B | about 40 GB | about 140 GB | RTX PRO 6000 at 4-bit |
Figures are approximate and vary with the model and the serving software.
Matching the job
- Chatbots and AI agents: an L4 serves a quantised 7B–13B model to many users at low power.
- Image and video generation: L4 or RTX PRO 4000.
- Fine-tuning: LoRA fine-tuning of small models works on 24 GB; larger models need the 96 GB card.
- Rendering: RTX PRO cards come with professional drivers for 3D software.
Hourly or monthly billing
Hourly GPU clouds suit short experiments. If the GPU runs most of the month, a fixed monthly price is cheaper and easier to plan, and the GPU cannot be taken back mid-job. Dataklo bills GPU servers monthly in rupees, from ₹33,999 for an L4.