Python 优化器的运行顺序影响PyTorch中的预测

我正在为许多优化器运行线性回归.我注意到，如果首先激活了SGD，其他的都有很好的准确性.否则，Adam和RMSsprop提出了可怕的调整.

# Generate synthetic data
X_numpy, y_numpy = datasets.make_regression(n_samples=100, n_features=1, noise=20, random_state=15)

X = torch.from_numpy(X_numpy.astype(np.float32))
y = torch.from_numpy(y_numpy.astype(np.float32))
y = y.view(y.shape[0], 1)

# Define the model
n_samples, n_features = X.shape
input_size = n_features
output_size = 1
model = nn.Linear(input_size, output_size)

# Define learning rate
learning_rate = 0.01

# Define criteria
criterion = nn.MSELoss()

# Define different optimizers
optimizers = {    
    "Adam": torch.optim.Adam(model.parameters(), lr=learning_rate),
    "RMSprop": torch.optim.RMSprop(model.parameters(), lr=learning_rate),
    "SGD": torch.optim.SGD(model.parameters(), lr=learning_rate),
}

# Training loop for each optimizer
num_epochs = 100
predictions = {}
for optimizer_name, optimizer in optimizers.items():
    print(f"Optimizer: {optimizer_name}")
    predictions[optimizer_name] = []
    for epoch in range(num_epochs):        
        y_predicted = model(X)
        loss = criterion(y_predicted, y)
        loss.backward()
        optimizer.step() #update wights
        optimizer.zero_grad() #zero the gradients
    predictions[optimizer_name] = model(X).detach().numpy()

# Plotting predictions with different colors
plt.figure(figsize=(10, 6))
plt.plot(X_numpy, y_numpy, 'ro', label='Original Data')
for optimizer_name, prediction in predictions.items():
    plt.plot(X_numpy, prediction, label=optimizer_name)
plt.legend()
plt.show()

上面的代码生成预测:

如果我先运行SGD，会发生以下情况:

optimizers = {    
    "Adam": torch.optim.Adam(model.parameters(), lr=learning_rate),
    "RMSprop": torch.optim.RMSprop(model.parameters(), lr=learning_rate),
    "SGD": torch.optim.SGD(model.parameters(), lr=learning_rate),
}

为什么会这样？