How to Clear CUDA Memory in PyTorch

To clear cuda memory in PyTorch, you can “use the torch.cuda.empty_cache() method.” This method only releases the memory cache that PyTorch isn’t currently using, so it won’t be freed if a tensor is still referenced somewhere in your code.

Here are six ways to clear cuda memory in PyTorch.

  1. Empty cache using torch.cuda.empty_cache()
  2. Deleting variables
  3. Avoid Keeping References
  4. Use with torch.no_grad() for Inference
  5. Set Variables to None
  6. Restart the Kernel/Script

Method 1: Empty Cache using torch.cuda.empty_cache()

PyTorch provides a built-in torch.cuda.empty_cache() method that releases all the GPU memory that can be freed.

Here’s how to use empty_cache():

import torch

torch.cuda.empty_cache()

As I already told you, If a tensor is still referenced somewhere in your code, it won’t be freed.

Method 2: Deleting Variables

You can clear GPU memory by deleting the no longer needed variables. When a variable is deleted, its memory is freed and can be used by other variables.

import torch

# Define a tensor
x = torch.randn(100, 100).cuda()

# Use the tensor
y = x * 10

# Delete the tensor
del x

# Use the GPU memory for other variables
z = y * 2

print(z)

Method 3: Avoid Keeping References

Ensure there are no remaining references to the tensors you want to delete. This includes references in lists, dictionaries, or any other data structures.

Method 4: Use with torch.no_grad() for Inference

If you are just doing inference (and not training), you can wrap your inference code inside a with torch.no_grad(): block to prevent PyTorch from using memory to store intermediate computations for backpropagation.

Method 5: Set Variables to None

You can set variables to None that releases their memory. However, unlike deleting variables, assigning variables to None doesn’t immediately free their memory. Instead, the memory is freed by Python’s garbage collector, which runs periodically.

import torch

# Define a tensor
x = torch.randn(100, 100).cuda()

# Use the tensor
y = x * 10

# Set the tensor to None
x = None

# Use the GPU memory for other variables
z = y * 2

print(z)

Method 6: Restart the Kernel/Script

If everything fails and you work in an interactive environment like Jupyter Notebook, you can restart the kernel to free up all memory. If you’re running a script, you can simply terminate and rerun it.

Always remember that simply calling torch.cuda.empty_cache() without deleting the tensors or ensuring they’re out of scope won’t free up the memory.

That’s all!

Leave a Comment