[alejandroschuler]

Found while assessing #168 (see #433).

This PR first renamed the attribute in one breaking step. The maintainer decided to warn for one release first, because the code search below found downstream projects that read dist.var as an array. The PR now does that.

Problem

Normal, NormalFixedVar, NormalFixedMean, T, TFixedDf, TFixedDfFixedVar, Cauchy and CauchyFixedVar set self.var to the squared scale in __init__. These classes give unknown names to the scipy.stats object in self.dist through __getattr__. Python calls __getattr__ only when normal lookup fails, so the array hides the var() method of scipy:

>>> Normal(params).var()
TypeError: 'numpy.ndarray' object is not callable

mean(), std() and the other scipy methods work on these classes, and var() works on LogNormal and the other distributions. For the T and Cauchy classes the attribute was also not the variance. It was scale ** 2, and the variance of the t distribution is scale ** 2 * df / (df - 2).

Change

• The scores now read a private array _var, with the same values as the old var. The Cauchy classes reuse the T classes and scores, so cauchy.py does not change.

• var is a property that returns _var and gives a FutureWarning. deprecated_var() in distn.py makes the property, and Normal, T, TFixedDf and TFixedDfFixedVar each have one line var = deprecated_var(...). The other classes inherit it. With stacklevel=2, the warning names the line of the user that reads var. The messages follow the LogNormal.loc and LogNormal.scale warnings in Add mu and sigma to LogNormal and raise AttributeError for unknown names #433:

  • Normal.var is deprecated. In a future release, var will be the var() method of scipy.stats, as for the other distributions. Use dist.scale ** 2 for the current values.
  • T.var is deprecated. It is the squared scale, not the variance. In a future release, var will be the var() method of scipy.stats, as for the other distributions. Use dist.scale ** 2 for the current values.

  The class name in the message is the class of the object, for example NormalFixedVar.var or Cauchy.var.

• Nothing in ngboost reads var, so fitting, scoring, pred_dist and pickling give no warning.

• examples/experiments/regression_exp.py used forecast.var * rmse as the scale of a NormalFixedVar forecast. forecast.var was all ones, the same as forecast.scale, so the script now uses forecast.scale.

• notebooks/sympy_normal_demo.ipynb now uses hw.scale**2 in place of hw.var. This works in this release and after the removal.

• I searched ngboost/, tests/, examples/, notebooks/, figures/ and docs/, including the code cells of the notebooks. There are no other readers of the attribute.

dist.var() still raises TypeError in this release, because the property hides the scipy method. Until the removal, dist.dist.var() gives the variance.

Why warn first

gh search code found this third-party code that reads .var as an array on an ngboost distribution. All of it uses Normal.

Repository	Use
emdgroup/baybe (baybe/surrogates/ngboost.py)	torch.from_numpy(dists.var), with ngboost>=0.3.12,<1
34j/boost-loss (src/boost_loss/sklearn.py)	predict_var returns dist.var
the-matter-lab/dionysus	np.sqrt(y_dists.var)
pfilo8/NodeFlow, pfilo8/TreeFlow, MikeOMa/MV_Prediction	dist.var.reshape(-1, 1)
CreaperLost/BayesianOptimization	mean, var = dist.loc, dist.var
fif911/probabilistic_conflict_modelling, mojipao/fantasy-draft-assistant, Snowmmmm/RL_Agent	np.sqrt(dist.var)

BayBE is the largest of these, with about 520 stars, and its pin allows the next ngboost release. With an immediate rename, its NGBoost surrogate would raise TypeError on that release. With this PR, all of this code keeps working and shows the warning, which tells the authors to use dist.scale ** 2. That works on old and new versions. A comment in Snowmmmm/RL_Agent says that var on an ngboost distribution is an attribute and not a method, which is the confusion this change removes. No hit called .var() on an ngboost distribution. ucgmsim/ml_vs30_model defines its own distribution with its own var, and this change does not affect it. The search found no subclass of these classes that assigns self.var. Such code would now raise AttributeError, because the property has no setter.

The search gives a lower bound. The code search API indexes default branches only and ignores punctuation, and some of my queries hit its rate limit.

Alternatives

• Rename at once. The first version of this PR did this and marked it as breaking. The code above would fail on the next release without a warning first.
• An array that is also callable. A property cannot tell dist.var from dist.var(), so the only way to make dist.var() work in this release and keep dist.var as an array is an ndarray subclass with a __call__. I built a prototype and rejected it. df.assign(var=dist.var) calls the object with the DataFrame and raises TypeError. v.reshape(-1, 1), v[0], np.asarray(v) and np.concatenate give no warning. For the T and Cauchy classes, the array would hold scale ** 2 and the call would return the variance.

The removal release

• Delete deprecated_var from distn.py, T_VAR_NOTE from t.py, and the four var = deprecated_var(...) lines. _var stays as the array that the scores read. dist.var() then reaches scipy through __getattr__, as mean() does.
• In tests/test_distn_var.py, delete test_var_warns_and_returns_squared_scale and remove the xfail mark from test_future_var_is_scipy_var. The mark is strict, so the suite fails until someone removes it. I checked this in a scratch copy: without the property lines, the 8 cases pass, so they show as strict XPASS failures.
• Mark the release note as breaking. dist.var becomes a method. For the T and Cauchy classes, dist.var() gives the variance, not the squared scale: inf for df in (1, 2] and nan for Cauchy.

Compatibility

• Scores: for all 11 pairs of an affected class and a score, score, d_score, grad and, where it exists, metric on 50 random parameter vectors are bitwise identical to master, with FutureWarning as an error.
• Pickled models: a model stores the distribution class and the parameters, not distribution objects. I pickled Normal (LogScore and CRPScore), NormalFixedMean, T, TFixedDf and Cauchy models on master and loaded them on this branch. They load and predict with no FutureWarning, loc and scale are identical, and pred.var gives one FutureWarning and the same array as the master attribute. I did not add a pickle made on master to the test suite, because it would depend on the scikit-learn version. The suite tests a pickle round trip with FutureWarning as an error.
• Add mu and sigma to LogNormal and raise AttributeError for unknown names #433 removes the __getattr__ of each class and adds one to Distn. The var property is on the class, so normal lookup finds it before either __getattr__. After the removal, Distn.__getattr__ forwards var to scipy, because var is in dir(self.dist). I applied Add mu and sigma to LogNormal and raise AttributeError for unknown names #433 on top of this branch. With --slow and FutureWarning from ngboost as an error, test_distn_var.py, test_score.py, test_distn_attributes.py and test_pickling.py give 103 passed, 1 skipped (skops is not installed) and 8 xfailed. deprecated_var is at the end of distn.py, away from the __getattr__ that Add mu and sigma to LogNormal and raise AttributeError for unknown names #433 adds.
• Other open work edits NormalFixedVar.__init__ and NormalFixedMean.__init__ in normal.py, and params in t.py. This PR changes one line in each of those two __init__ methods, so a merge can need a one-line fix there.

Example

import numpy as np
from ngboost import NGBRegressor

X = np.random.normal(size=(100, 3))
Y = X[:, 0] + np.random.normal(size=100)
pred = NGBRegressor(n_estimators=5, verbose=False).fit(X, Y).pred_dist(X[:3])

	pred.var	pred.var()
master	array of scale ** 2	TypeError
this PR	array of scale ** 2, with a FutureWarning	TypeError
removal release	bound method of the scipy distribution	the variance from scipy

pred.scale ** 2 gives the same array in all three, with no warning.

Tests

• tests/test_distn_var.py (new):
  • test_var_warns_and_returns_squared_scale, for each of the 8 classes: reading var gives one FutureWarning with the file and line of the test, the message has the class name, the scipy note and dist.scale ** 2, the T and Cauchy messages also have the squared-scale note, and the value is scale ** 2 as before. These 8 fail on master, which gives no warning.
  • test_ngboost_gives_no_var_warning, for each class with LogScore and the Normal classes with CRPScore (11 cases): with every FutureWarning as an error, it evaluates total_score and grad of the scores, fits an NGBRegressor, calls pred_dist, and pickles and loads the model. These pass on master too. I checked that they catch an internal read: when I changed one line of NormalLogScore to read self.var, the test failed.
  • test_future_var_is_scipy_var, for each class: var() equals dist.dist.var() and std() ** 2. These 8 record the behavior after the removal. They are marked xfail(raises=TypeError, strict=True) with the reason "dist.var() works once the deprecated var attribute is removed", and a comment above them says to remove the mark with the attribute.
• tests/test_score.py: no test checked the LogScore of NormalFixedVar and NormalFixedMean, and this PR changes its lines. Both are now in TEST_METRIC, so test_dists_grad checks their gradients and the slow test_dists_metric checks their metrics. These 4 tests pass on master and on this branch.
• Full suite: 155 passed, 61 skipped, 8 xfailed (134 passed, 59 skipped on master). The 21 new passes are the 19 new tests that are not xfail and the 2 new test_dists_grad cases. The 2 new skips are the slow metric cases, and the 8 xfails are the future tests. test_score.py and test_distn_var.py with --slow give 73 passed and 8 xfailed.
• With FutureWarning from ngboost as an error: 155 passed, 61 skipped, 8 xfailed. The command was pytest tests -W error::FutureWarning:ngboost -o "filterwarnings=error::FutureWarning:ngboost". The -o part is necessary. pytest matches the module of a command-line -W filter exactly, so -W error::FutureWarning:ngboost alone catches only warnings from the module ngboost itself and not from ngboost.distns.normal. I checked this with the same planted self.var read: the -W flag alone let the test pass, and the -o filter made it fail.
• black, isort, flake8 and pylint are clean on the changed files, with the options in .pre-commit-config.yaml.
• I ran notebooks/sympy_normal_demo.ipynb with nbconvert --execute. It gives no FutureWarning, and 13 of 14 code cells print the same text as the stored outputs. The other cell shows a figure size of 1500x400 in place of 1080x288, because of the figure DPI of the environment. That cell does not use var, and the run of the previous commit gave the same difference.

🤖 Generated with Claude Code