To clear cuda memory in PyTorch, you can “use the torch.cuda.empty_cache() method.” This method only releases the memory cache that PyTorch isn’t currently using, so it won’t be freed if a tensor is still referenced somewhere in your code.
Here are six ways to clear cuda memory in PyTorch.
- Empty cache using torch.cuda.empty_cache()
- Deleting variables
- Avoid Keeping References
- Use with torch.no_grad() for Inference
- Set Variables to None
- Restart the Kernel/Script
Method 1: Empty Cache using torch.cuda.empty_cache()
PyTorch provides a built-in torch.cuda.empty_cache() method that releases all the GPU memory that can be freed.
Here’s how to use empty_cache():
import torch torch.cuda.empty_cache()
As I already told you, If a tensor is still referenced somewhere in your code, it won’t be freed.
Method 2: Deleting Variables
You can clear GPU memory by deleting the no longer needed variables. When a variable is deleted, its memory is freed and can be used by other variables.
import torch # Define a tensor x = torch.randn(100, 100).cuda() # Use the tensor y = x * 10 # Delete the tensor del x # Use the GPU memory for other variables z = y * 2 print(z)
Method 3: Avoid Keeping References
Ensure there are no remaining references to the tensors you want to delete. This includes references in lists, dictionaries, or any other data structures.
Method 4: Use with torch.no_grad() for Inference
If you are just doing inference (and not training), you can wrap your inference code inside a with torch.no_grad(): block to prevent PyTorch from using memory to store intermediate computations for backpropagation.
Method 5: Set Variables to None
You can set variables to None that releases their memory. However, unlike deleting variables, assigning variables to None doesn’t immediately free their memory. Instead, the memory is freed by Python’s garbage collector, which runs periodically.
import torch # Define a tensor x = torch.randn(100, 100).cuda() # Use the tensor y = x * 10 # Set the tensor to None x = None # Use the GPU memory for other variables z = y * 2 print(z)
Method 6: Restart the Kernel/Script
If everything fails and you work in an interactive environment like Jupyter Notebook, you can restart the kernel to free up all memory. If you’re running a script, you can simply terminate and rerun it.
Always remember that simply calling torch.cuda.empty_cache() without deleting the tensors or ensuring they’re out of scope won’t free up the memory.