PyTorch torch.no_grad()

by

PyTorch torch.no_grad() is a context manager “used to disable gradient calculation within a specific block of code.” This is helpful for inference or evaluation phases when you are sure you will not call backward(). Disabling gradient calculation can reduce memory consumption and speed up computations.

In this mode, the result of every computation will have requires_grad=False, even when the inputs have requires_grad=True.

Syntax

with torch.no_grad():
  # Your code here

The torch.no_grad() context manager is typically used with a with statement to specify the block of code where gradients should not be computed:

Example 1: How to Use torch.no_grad()

Here’s a simple example showing how the torch.no_grad() works:

import torch

x = torch.tensor([1.0, 2.0], requires_grad=True)
y = x * 2

with torch.no_grad():
  z = y + 2

print("Does z require gradients? : ", z.requires_grad)

Output

Does z require gradients? : False

In this code example, the tensor z will not require gradients, even though it is computed from y, which does require gradients.

Model Evaluation

When evaluating a model, wrapping the evaluation code in a torch is a good practice.no_grad() block. This can help save memory and make the model run faster.

import torch
import torch.nn as nn


class SimpleModel(nn.Module):
  def __init__(self):
    super(SimpleModel, self).__init__()
    self.fc = nn.Linear(10, 10)

  def forward(self, x):
    return self.fc(x)


# Initialize the model and set it to evaluation mode
model = SimpleModel()
model.eval()

# Create a sample input tensor
input_tensor = torch.randn(1, 10)

# Evaluate the model without gradient computation
with torch.no_grad():
  output = model(input_tensor)

print("Does output require gradients? : ", output.requires_grad)

Output

Does output require gradients? : False

In this example, the output tensor does not require gradients, which is what you would typically want during evaluation.

Related posts

torch.optim.Adam()

torch.unsqueeze()

Post Views: 3

Leave a Comment