How To Interpret the 1-D Matrix Profile Column Output

[Samraw003]

Hi Sean,

Good Morning.

Thanks a lot for STUMPY and all the support and clarifications you are providing to the community.

I am new to Python and STUMPY as well. So I have taken the enormous time and effort to get it working.

I am trying to identify patterns in: "Personal Consumption Expenditure" (PCE) as released by BEA. BEA releases around 100+ series to understand different facets of Consumer expenditure. I have also tried the Multidimensional version of STUMPY as well.
The data is monthly and is available from 1959 onwards.

However, I am not able to understand the 4 attributes/columns of mp (for single series)?

Additionally, point me if I am incorrect in understanding that the index of the array (mp) is the location of each segment (m)?

Following is my mp:

array([[1.626257115121311, 202, -1, 202],
[1.7138456780667977, 65, -1, 65],
[1.880293454724256, 66, -1, 66],
[1.796922109741226, 67, 0, 67],
[1.4943082939628236, 11, 1, 11],
[1.4504278114808016, 12, 2, 12],
[1.6294354134867932, 19, 0, 19],
[1.5349365731102185, 229, 0, 229],
[1.3930265554289831, 186, 1, 186],
[1.5265881687159586, 187, 2, 187],
[1.8022253384245739, 33, 3, 33],
[1.4943082939628236, 4, 4, 118],
[1.4504278114808016, 5, 5, 137],
[1.680920620705546, 201, 6, 201],
[1.5625058007723722, 237, 8, 237],
[1.2860008417613522, 66, 9, 66],

I understand this is a very naive question :)

Thanks and regards

Sammeer

[seanlaw]

Thanks a lot for STUMPY and all the support and clarifications you are providing to the community.

@Samraw003 Thank you for the question and kind words and welcome to the STUMPY community

I am new to Python and STUMPY as well. So I have taken the enormous time and effort to get it working.

I don't mean to get too hung up on this point but can you elaborate on what you mean by "enormous time and effort to get it working". From my personal experience (which may not be representative), it shouldn't take too much time/error to get STUMPY installed and up and running. If that is the case, then we've done something wrong! Perhaps, you can share more?

However, I am not able to understand the 4 attributes/columns of mp (for single series)?

Admittedly, we could probably do more to clarify in the STUMPY Basics Tutorial exactly what the 4 columns are referring to so I will take a note and add some more details. However, the API documentation provides some limited information.

So, starting with a time series, T, and a window size, m, you've correctly identified that the 1-D matrix profile, mp, computed from stumpy.stump(T, m) contains 4 explicit columns, which we'll clarify in a moment. Implicitly, the ith row of the mp array corresponds to the set of (4) nearest neighbor values computed for the specific subsequence T[i : i + m] (note that this is a subsequence that starts at position i on the left but the right edge is exclusive and actually stops one short at i + m - 1).

The first column of the mp contains the matrix profile (nearest neighbor distance) value, P (note that due to zero-based indexing, the "first column" has a column index value of zero).
The second column contains the (zero-based) index location, I, of where the (above) nearest neighbor is located along T (note that any negative index values are "bad" values and indicates that a nearest neighbor could not be found).

So, for the ith subsequence T[i : i + m], it's nearest neighbor (somewhere along T) has a starting index location of I = mp[i, 1] and, assuming that I >= 0, this corresponds to the subsequence found at T[I : I + m]. And the matrix profile value for the ith subsequence, P = [i, 0], is the exact (z-normalized Euclidean) distance between T[i : i + m] and T[I : I + m]. Note that the nearest neighbor index location, I, can be positioned ANYWHERE. That is, dependent upon ith subsequence, its nearest neighbor, I, can be located before/to-the-"left" of i (i.e., I <= i) or come after/to-the-"right" of i (i.e., I >= i). In other words, there is no constraint on where a nearest neighbor is located. However, there are times when you might like to only know about a nearest neighbor that either comes this is where columns 3 and 4 come into play.

The third column contains the (zero-based) index location, IL, of where the "Left" nearest neighbor is located along T. So, here, there is a constraint that IL < i or that IL must come before/to-the-left of i. Thus, the "left nearest neighbor" for the ith subsequence would be located at IL = mp[i, 2] and corresponds to T[IL : IL + m].
The fourth column contains the (zero-based) index location, IR, of where the "Right" nearest neighbor is located along T. So, here, there is a constraint that IR > i or that IR must come after/to-the-right of i. Thus, the "right nearest neighbor" for the ith subsequence would be located at IR = mp[i, 3] and corresponds to T[IR : IR + m].

Again, note that any negative index values are "bad" values and indicates that a nearest neighbor could not be found.

To reinforce this more concretely, let's use your array as an example:

array([[1.626257115121311, 202, -1, 202],
[1.7138456780667977, 65, -1, 65],
[1.880293454724256, 66, -1, 66],
[1.796922109741226, 67, 0, 67],
[1.4943082939628236, 11, 1, 11],
[1.4504278114808016, 12, 2, 12],
[1.6294354134867932, 19, 0, 19],
[1.5349365731102185, 229, 0, 229],
[1.3930265554289831, 186, 1, 186],
[1.5265881687159586, 187, 2, 187],
[1.8022253384245739, 33, 3, 33],
[1.4943082939628236, 4, 4, 118],
[1.4504278114808016, 5, 5, 137],
[1.680920620705546, 201, 6, 201],
[1.5625058007723722, 237, 8, 237],
[1.2860008417613522, 66, 9, 66],

So, here, the subsequence at i = 0 would correspond to the T[0 : 0 + m] subsequence and the nearest neighbor for this subsequence is located at I = 202 (i.e., mp[0, 1]) and corresponds to the T[202 : 202 + m] subsequence. The z-normalized Euclidean distance between T[0 : 0 + m] and T[202 : 202 + m] is actually P = 1.626257115121311 (i.e., mp[0, 0]). Next, notice that the location of the left nearest neighbor is IL = -1 (i.e., mp[0, 2]) and, since negative indices are "bad", this tells us that the left nearest neighbor could not be found. Hopefully, this makes sense since T[0 : 0 + m] is the first subsequence in T and there are no other subsequences that can possibly exist to the left of T[0 : 0 + m]! Conversely, the location of the right nearest neighbor is IR = 202 (i.e., mp[0, 3]) and corresponds to the T[202 : 202 + m] subsequence.

Additionally, the subsequence at i = 5 would correspond to the T[5 : 5 + m] subsequence and the nearest neighbor for this subsequence is located at I = 12 (i.e., mp[5, 1]) and corresponds to the T[12 : 12 + m] subsequence. The z-normalized Euclidean distance between T[5 : 5 + m] and T[12 : 12 + m] is actually P = 1.4504278114808016 (i.e., mp[5, 0]). Next, the location of the left nearest neighbor is IL = 2 (i.e., mp[5, 2]) and corresponds to T[2 : 2 + m]. Conversely, the location of the right nearest neighbor is IR = 12 (i.e., mp[5, 3]) and corresponds to the T[12 : 12 + m] subsequence.

Additionally, point me if I am incorrect in understanding that the index of the array (mp) is the location of each segment (m)?

Hopefully, the above explanations helps to clarify things but please do not hesitate to ask follow up questions if you have any! I am here to help!

[Samraw003]

Hi Sean,

Thanks for the super-fast and detailed response.

My bad to have mentioned "enormous time and effort to get it working". Actually, it is very jiffy and effortless to run the first code, as the examples are very well explained. That can be done in minutes!

Being a new bee to Python it took time in setting up Anaconda, configuring/fingering out db connection, converting the sql data into the correct format in the dataframe. Matplot, pandas and all the other required stuff were Latin to me till now :)!!

Also, you did provide a peep into mp in response to #353!

I previewed all of your videos, blogs, articles, git discussions and documentation a couple of times to understand mp!

I do have some queries on multidimensional patterns that will ask in a bit!

Have a great day!

Sammeer