One-Hot encoded + SNARIMAX

[ColdTeapot273K]

Been tinkering with pipelines on multiple shops demand forecasting dataset

river version: from github, this week

What grinds my gears is:

1. The fact that i cannot examine weights for eXogenous regressors of the SNARIMAX model (it just doesn't expose such atteibutes). Necessary for debugging & explainability purposes. The cool "debug_one" method not available.
2. The fact that its regressor needs to include some scaler. Its not like it doesn't make sense, but i can't examine (because of 1.) wether it scales, for instance, my One-Hot encoded features or not (because it shouldn't). Imagine you OHE different shops, then predict for "shop 3" and see non-zero effects from "shop 4" and "shop 5" because they got scaled into continuous. Not good.

[MaxHalford]

The fact that i cannot examine weights for eXogenous regressors of the SNARIMAX model (it just doesn't expose such atteibutes)

You can access the regressor attribute.

The cool "debug_one" method not available.

Agreed, it would be nice to have. Sadly we currently don't have any time resources to allocate.

The fact that its regressor needs to include some scaler. Its not like it doesn't make sense, but i can't examine (because of 1.) wether it scales, for instance, my One-Hot encoded features or not (because it shouldn't). Imagine you OHE different shops, then predict for "shop 3" and see non-zero effects from "shop 4" and "shop 5" because they got scaled into continuous. Not good.

Good point! You could pass a custom pipeline to the regressor attribute and to this yourself. For instance (not tested):

snarimax = SNARIMAX(
    regressor=(
        (
            (
                compose.Select('shop') |
                preprocessing.OneHotEncoder()
            ) +
            (
                compose.Select('numeric_feature_1', 'numeric_feature_2') |
                preprocessing.StandardScaler()
            )
        ) |
        linear_model.LinearRegression()
    )
)

[ColdTeapot273K]

Thanks for quick reply Max

You can access the regressor attribute.

Figured, my problem was that I 'piped' both feature engineering & model into one pipeline and looks like there's no way to access regressor afterwards (i went all the way to model._get_params()["SNARIMAX"]["regressor"][1]['LinearRegression'] with no result)

Workaround for folks who find this - quick dirty way is to define regressor separately to have direct reference to it, like this:

feature_pipeline = encoders + scalers
ts_regressor = time_series.SNARIMAX(
regressor=(preprocessing.StandardScaler() | linear_model.LinearRegression())
)
model = feature_pipeline  | regressor

Then you can inspects weights via:

ts_regressor.regressor.debug_one(feature_pipeline.transform_one(xi))

UPD: But the One-Hot encoded variables will be scaled ^this way, which is undesirable

Good point! You could pass a custom pipeline to the regressor attribute and to this yourself. For instance (not tested):

snarimax = SNARIMAX(
    regressor=(
        (
            (
                compose.Select('shop') |
                preprocessing.OneHotEncoder()
            ) +
            (
                compose.Select('numeric_feature_1', 'numeric_feature_2') |
                preprocessing.StandardScaler()
            )
        ) |
        linear_model.LinearRegression()
    )
)

Stuff like this leads to TypeError: unsupported operand type(s) for +: 'dict' and 'int' inside river\time_series\snarimax.py. I tried different permutations of the same idea.

This example leads to exploding gradients:

feature_pipeline = encoders + scalers
ts_regressor = time_series.SNARIMAX(regressor=(linear_model.LinearRegression()))
model = feature_pipeline | regressor 

---

Weirdly enough, what worked was:[see below]

feature_pipeline = encoders + scalers
ts_regressor = time_series.SNARIMAX(
    regressor=(
    (compose.Select(*encoded_cols)) + (compose.Select(*scaled_cols))
    | linear_model.LinearRegression())
)
model = feature_pipeline | regressor 

Afterwards, this both gives weights for eXogenous & doesn't distort OHE features:

ts_regressor.regressor.debug_one(feature_pipeline.transform_one(xi))

Just need a safe way to select all the columns from feature_pipeline output🤔

---

UPD: Tried selecting all the columns via SelectType:

feature_pipeline = encoders + scalers
ts_regressor = time_series.SNARIMAX(
    regressor=(
    compose.SelectType(object)
    | linear_model.LinearRegression())
)
model = feature_pipeline | regressor 

and the gradients explode, so the solution above just turns the thing into usual regression instead, with no pdqmSDQ terms

I suppose the scaler inside SNARIMAX is vital for internal variables (pdqmSDQ) in this implementation. And, it appears, currently there's no working way to bypass it (hence, you'll always have to apply standard scaling to all eXogenous regressors, including OHE variables, in SNARIMAX model of river)