.motifs no longer returning more than one motif

[waynedam]

I was finding this problem in my own work. Only one motif was being returned when there were at least two apparent in the time series.

I tried reproducing the example discussed in #438 where the motifs function properly returns multiple motifs but I find now it no longer does. I used the data provided in issue #438.

The csv file:
RoomA_CO2.csv

import csv
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams["figure.figsize"] = [20, 6]  # width, height
plt.rcParams['xtick.direction'] = 'out'

co2 = []
with open('./RoomA_CO2.csv') as csvfile:
    reader = csv.DictReader(csvfile, delimiter=';')
    for row in reader:
        if len(row['CO2-Concentration (in ppm)'].strip()) > 0:
            co2.append(float(row['CO2-Concentration (in ppm)']))

m = 24
c = np.array(co2)
p = stumpy.stump(c, m)

dists, inde = stumpy.motifs(c, p[:, 0], max_motifs=2)

fig, axs = plt.subplots(2, sharex=True, gridspec_kw={'hspace': 0})
plt.suptitle('Motif (Pattern) Discovery', fontsize='30')
axs[0].plot(co2)
axs[0].set_ylabel('CO2 ppm', fontsize='20')

cols = ['red' , 'green', 'blue' ]

for z in range(0, inde.shape[0]):
    col = cols[z]
    start = inde[z, 0]
    stop = inde[z, 0] + m
    matches = stumpy.match(c[start:stop],c, max_distance=2.0)
    for mt in range(matches.shape[0]):
        s = matches[mt, 1]
        st = s + m
        axs[0].plot(np.arange(s, st), c[s : st], c=col)

axs[1].plot(p[:, 0])
axs[1].set_ylabel('Matrix profile', fontsize='20')
plt.show()

I understand the stumpy.match is not necessary but the stumpy.motifs only returns a single motif row.

inde.shape
(1,4)
inde
[39,179,77,223]

The fifth match in the plot around 125 is from the stumpy.matches.

I tried changing distance parameters, etc., but nothing seems to make a difference.

Has something changed in the version 1.11.1 stumpy.motifs that is not reflected in the documentation?
I noticed someone having an issue with stumpy.config.STUMPY_EXCL_ZONE_DENOM config in #780.

[seanlaw]

@waynedam Thank you for your question and welcome to the STUMPY community.

@waynedam Thank you for your question and welcome to the STUMPY community

From what I can tell, nothing has changed.

Maybe I'm not understanding what the problem is but, in your case, one possible reason that inde is [39,179,77,223] and seems to be missing 125 is because you've chosen to specify a max_distance = 2.0 in matches = stumpy.match(c[start:stop],c, max_distance=2.0). However, in stumpy.motifs(c, p[:, 0], max_motifs=2) you've omitted a max_distance but provided a max_motifs instead. If you do stumpy.motifs(c, p[:, 0], max_motifs=2, max_distance=2.0 then you should see the 125 being a match to the first motif.

Does that help?

[waynedam]

Sorry for the confusion. The issue is not the missing 125 in the extracted motif.
It's that in his reply here: #438 (reply in thread)

He show shows that he extracted two motifs which show as red and green in the graph.
They are plotted in the loop:

for z in range(0, inde.shape[0]):

However, when I run the code inde has inde.shape[0] equal to 1. Only a single motif is extracted in my case whereas he shows two. I am running the identical code to his.

And in my own case I can see two distinct motifs but do not seem able to extract them both. I only have one returned.
So I am using the code from the old issue as a proven test case and still seem to have the same problem.

Is there an example available using stumpy.motifs that extracts more than one motif that I might use for testing? I could not find an example in the docs that had more than one motif.

[seanlaw]

@waynedam Thanks for the clarification. I think the secret lies in the cutoff parameter (see docs). As per the docs, when no cutoff is provided, this default is to use 2 standard deviations smaller than the average matrix profile distance (np.nanmax([np.nanmean(P) - 2.0 * np.nanstd(P), np.nanmin(P)])). In this case, the default cutoff for p[:, 0] is 0.14825539679421665. So, any matrix profile distance greater than this will not be captured as a "valid" motif given the default constraint. This is precisely the case in this example as the next largest matrix profile value is 0.42612511, which does not meet the default criteria.

However, you can ignore/omit this setting by doing cutoff = np.inf (i.e., setting the cutoff to infinity). Now, when I run the code below:

import csv
import numpy as np
import matplotlib.pyplot as plt
import stumpy

plt.rcParams["figure.figsize"] = [20, 6]  # width, height
plt.rcParams['xtick.direction'] = 'out'

co2 = []
with open('./RoomA_CO2.csv') as csvfile:
    reader = csv.DictReader(csvfile, delimiter=';')
    for row in reader:
        if len(row['CO2-Concentration (in ppm)'].strip()) > 0:
            co2.append(float(row['CO2-Concentration (in ppm)']))

m = 24
c = np.array(co2)
p = stumpy.stump(c, m)

# dists, inde = stumpy.motifs(c, p[:, 0], max_motifs=2)
dists, inde = stumpy.motifs(c, p[:, 0], cutoff=np.inf, max_motifs=2)

fig, axs = plt.subplots(2, sharex=True, gridspec_kw={'hspace': 0})
plt.suptitle('Motif (Pattern) Discovery', fontsize='30')
axs[0].plot(co2)
axs[0].set_ylabel('CO2 ppm', fontsize='20')

cols = ['red' , 'green', 'blue' ]

for z in range(0, inde.shape[0]):
    col = cols[z]
    start = inde[z, 0]
    stop = inde[z, 0] + m
    # matches = stumpy.match(c[start:stop],c, max_distance=2.0)
    matches = stumpy.match(c[start:stop],c, max_distance=2.0)
    print(matches)
    for mt in range(matches.shape[0]):
        s = matches[mt, 1]
        st = s + m
        axs[0].plot(np.arange(s, st), c[s : st], c=col)

axs[1].plot(p[:, 0])
axs[1].set_ylabel('Matrix profile', fontsize='20')
plt.show()

I get:

Can you please confirm whether this solves your problem and then report back?

[waynedam]

Yes, this solves the issue.
Thank you very much for the insight. It's very helpful understanding how to best apply the library.