The purpose of adjust_anomaly_scores

[severous]

Thanks for your briliiant work. I would like to know the purpose of adjust_anomaly_scores. Thanks for your time.

 # Remove errors for time steps when transition to new channel (as this will be impossible for model to predict)
    if dataset.upper() not in ['SMAP', 'MSL']:
        return scores

    adjusted_scores = scores.copy()
    if is_train:
        md = pd.read_csv(f'./datasets/data/{dataset.lower()}_train_md.csv')
    else:
        md = pd.read_csv('./datasets/data/labeled_anomalies.csv')
        md = md[md['spacecraft'] == dataset.upper()]

    md = md[md['chan_id'] != 'P-2']

    # Sort values by channel
    md = md.sort_values(by=['chan_id'])

    # Getting the cumulative start index for each channel
    sep_cuma = np.cumsum(md['num_values'].values) - lookback
......................

[axeloh]

Hi, thanks for your question.

Recall that SMAP and MSL actually consist of multiple individual time-series (A-1, C-2, etc.), each of which has 24/54 (SMAP/MSL) one-hot encoded features and 1 continuous feature. Each of these time-series is very short (typically 1-3k timesteps), so most implementations (including ours) concatenate all these time-series in the time direction, creating one large time-series.

So, the dataset will "jump" up and down whenever it transitions to a new channel. This has some effects on the forecastings and reconstructions of the model:

1. It will be "impossible" for model to predict correct values when the data is transitioning to a new channel, so the error at these timestamps will be high. Because the errors are used to fit the threshold, we set the errors at these timestamps to zero, so that they do not affect the thresholding.
2. Different channels will have different ranges (min-max values), and will therefore typically yield errors with different ranges. Therefore, we want to normalize the errors for each channel before they are used to fit the threshold.

These two steps are performed in adjust_anomaly_scores and are only applied when dataset is MSL or SMAP.
labeled_anomalies.csv is used to get the length of each channel of the test set, while smap/msl_train_md.csv is used for the same purpose but for the train set.