GPU Memory Calculator 🧮

Posted on May 21, 2024
tl;dr: Quickly see how much memory your model needs

GPU Memory needs scale up based on the size of the model. To quickly calculate the memory required for a model you can use the calculators below.

For inference memory required is typically 2 x the number of parameters, this is because each parameter is typically two bytes (FP16). So a 7B parameter model takes 14GB.

For training in mixed precision (FP16) it’s typically 18x the number of parameters plus activations, this is typically 22-32x the number of parameters. So for a 7 B parameter model, you’ll need at least 224GB of GPU memory.

These instance types have the following GPU memory:

Instance GPU GPU Memory
g5 A10G 24 GB
p4d A100 40 GB
p4de A100 80 GB
p5 H100 80 GB

The memory requirements will quickly exceed the memory available on a single instance type, hence the need to use parallelism technique which we’ll cover in a later post.


Number of parameters in (B):


Number of parameters in (B):

comments powered by Disqus