GPU Memory Calculator 🧮

GPU Memory needs scale up based on the size of the model. To quickly calculate the memory required for a model you can use the calculators below.

For inference memory required is typically 2 x the number of parameters, this is because each parameter is typically two bytes (FP16). So a 7B parameter model takes 14GB.

For training in mixed precision (FP16) it’s typically 18x the number of parameters plus activations, this is typically 22-32x the number of parameters. So for a 7 B parameter model, you’ll need at least 224GB of GPU memory.

These instance types have the following GPU memory:

The memory requirements will quickly exceed the memory available on a single instance type, hence the need to use parallelism technique which we’ll cover in a later post.

Inference

Training