Bug/WP-380: Remove id prefix filter on workspace search

[chandra-tacc]

Overview

Workspace search has id field with system id, project id and actual id used in construction of id. There is a id prefix post-filter on search, which is not working on indexed content in frontera.

There is filtering on results to ensure the results are within the list of projects available to the user.

Are we going to get results from other portals ? No. The project index searched is specific to a portal. So, the extra id based prefix filter does not add value, and the prefix filter on the id field is not working on Frontera (see Testing Results)

Related

• WP-380

Changes

Remove the id based prefix filter.

Testing

1. Deployed the branch on dev.cep and ensured no regression in behavior.
2. Frontera testing
   Used django shell to validate.
   In my frontera, below is list of projects:
   

With prefix filter: There are no results, even though the indexed doc has right prefix.

In [4]: query_string = "OpenSees"
In [5]: ngram_query = Q("query_string", query=query_string,fields=["title", "id"],minimum_should_match='100%',default_operator='or')
In [6]: wildcard_query = Q("wildcard", title=f'*{query_string}*') | Q("wildcard", id=f'*{query_string}*')
In [7]: search = IndexedProject.search()
In [8]: search = search.query(ngram_query | wildcard_query)
In [9]: search = search.extra(from_=0, size = 100)
In [10]: search = search.filter('prefix', **{'id': f'{settings.PORTAL_PROJECTS_SYSTEM_PREFIX}'})
In [11]: res = search.execute()
In [12]: print(res)
<Response: {}>

Without prefix filter: This gets the right results.

In [13]: search = IndexedProject.search()
In [14]: search = search.query(ngram_query | wildcard_query)
In [15]: search = search.extra(from_=0, size = 100)
In [16]: res = search.execute()
In [17]: print(res)
<Response: [IndexedProject(index='frontera-pprd-projects', id='dcaa80274f67853c83764ed054af140235eabf0e41b2306788dc7b68a1425d19')]>
In [18]: print([hit.to_dict() for hit in res])
[{'id': 'frontera-portal.project.FRONTERA-PORTAL-105', 'path': '/corral-repl/tacc/aci/Frontera/projects/FRONTERA-PORTAL-105', 'name': 'FRONTERA-PORTAL-105', 'host': 'cloud.data.tacc.utexas.edu', 'updated': datetime.datetime(2023, 11, 9, 14, 48, 16, 920528), 'owner': {'username': 'cyemparala', 'first_name': 'Chandra', 'last_name': 'Yemparala', 'email': 'cyemparala@tacc.utexas.edu'}, 'title': 'OpenSees', 'description': ''}]
In [19]: print(settings.PORTAL_PROJECTS_SYSTEM_PREFIX)
frontera-portal.project

UI

Notes

[codecov]

Codecov Report

Merging #898 (a555d65) into main (fb996d5) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #898   +/-   ##
=======================================
  Coverage   63.41%   63.41%           
=======================================
  Files         432      432           
  Lines       12389    12388    -1     
  Branches     2579     2579           
=======================================
  Hits         7856     7856           
+ Misses       4323     4322    -1     
  Partials      210      210           

Flag	Coverage Δ
javascript	69.76% <ø> (ø)
unittests	56.94% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
server/portal/apps/projects/views.py	34.23% <ø> (+0.30%)	⬆️

[rstijerina]

LGTM

Question: @chandra-tacc why do you think the prefix filter fails for frontera specifically? Perhaps due to the hyphen in frontera-portal?

[shayanaijaz]

LGTM

[chandra-tacc]

LGTM

Question: @chandra-tacc why do you think the prefix filter fails for frontera specifically? Perhaps due to the hyphen in frontera-portal?

@rstijerina - that is a great catch. Yes, hyphen is throwing it off. searching for frontera works. id and id._exact are not analyzed, it is a keyword field.

If I used query:
Q("match_phrase_prefix", **{'id':'frontera-portal.project'})

the search works. Phrase prefix worked when I hit the index directly and in frontera. But, it threw an error in my local as unsupported.

I'll explore more on hyphen