If the forward pass in an evaluation loop already runs inside with torch.no_grad():, is model.eval() still needed? The two calls often sit next to each other in test-time code, and both have something to do with not training, so one of them can look like it covers the other.
A Dropout layer is a good place to check, since its output depends on the mode. The layer starts out in its default training mode:
import torch
from torch import nn
torch.manual_seed(0)
drop = nn.Dropout(p=0.5)
x = torch.ones(8)
with torch.no_grad():
print(drop(x))
print(drop(x))
drop.eval()
with torch.no_grad():
print(drop(x))
tensor([0., 0., 2., 0., 0., 0., 2., 2.])
tensor([0., 2., 2., 2., 2., 0., 2., 2.])
tensor([1., 1., 1., 1., 1., 1., 1., 1.])
(Python 3.14.3 and PyTorch 2.12.0, on CPU. The seed keeps the zeros in the same places from run to run on this setup, though another PyTorch version or machine may put them elsewhere.)
The first two lines come from the same input, inside the same no_grad block, but they don’t match. That is because Dropout picks which values to zero each time it is called: the nn.Dropout docs say that during training it “randomly zeroes some of the elements of the input tensor with probability” p, and that “the zeroed elements are chosen independently for each forward call”. The values it keeps come out as 2 rather than 1, because during training the outputs are scaled by 1/(1-p), and with p at 0.5 that means doubling.
The third line, after eval(), is all ones, because “during evaluation the module simply computes an identity function.” The forward method of nn.Dropout in the PyTorch source passes the mode along to F.dropout:
return F.dropout(input, self.p, self.training, self.inplace)
self.training is the flag that eval() sets to False and train() sets back to True, and BatchNorm is another common layer that reads it. None of those arguments is about gradients, so what does no_grad change? Here a Linear layer goes in front of the Dropout to give autograd a weight to track:
import torch
from torch import nn
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(8, 8), nn.Dropout(p=0.5))
model.eval()
x = torch.ones(1, 8)
out = model(x)
print(out.requires_grad, out.grad_fn)
with torch.no_grad():
out = model(x)
print(out.requires_grad, out.grad_fn)
True <AddmmBackward0 object at 0x000001877221FD30>
False None
In eval mode alone, the output still has requires_grad set and a grad_fn from the Linear layer, which means autograd recorded the forward pass so that backward() could be called on it later. Under no_grad the output has neither, even though the model is in the same eval mode. The autograd notes call eval() “completely orthogonal to no-grad mode and inference mode”. Because the recording stops, torch.no_grad also “will reduce memory consumption for computations that would otherwise have requires_grad=True.”
If a model’s test predictions come out different on every call, a missing model.eval() is a good first thing to check. Running out of GPU memory during evaluation, when nothing calls backward(), is a different problem: it is worth checking whether the forward passes are being recorded, and torch.no_grad() turns that recording off. An evaluation loop normally uses both. eval() also stays in effect until something changes it, so the training loop needs model.train() again before the next epoch.
PyTorch also has torch.inference_mode(), which the autograd notes call “the extreme version of no-grad mode”. They warn that tensors created in it “will not be able to be used in computations to be recorded by autograd after exiting inference mode”, so whether it can replace no_grad depends on what happens to those tensors after the block ends. The notes suggest trying it for model evaluation and going back to no_grad if the errors can’t be avoided, which is probably the easiest way to find out for a given loop.
This post comes from running the two snippets above on CPU and reading the PyTorch docs and source, so take it as a small idea to check against your own loop. If you have run into similar cases, feel free to share them.
