.Net MEVD: Filter-only search API (without vector similarity) #11112

adamsitnik · 2025-03-21T15:34:37Z

GetFilteredRecordOptions<FilterRecord> options = new();

options.OrderBy
    .Ascending(r => r.Int)
    .Descending(r => r.String);

return await collection.GetAsync(record => record.Int > 2, top: 100, options).ToListAsync();

Implemented with tests for:

fixes #10295

…o queryAsync # Conflicts: # dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresVectorStoreDbClient.cs # dotnet/src/Connectors/Connectors.Memory.Postgres/SqlBuilder.cs # dotnet/src/Connectors/Connectors.Postgres.UnitTests/PostgresVectorStoreCollectionSqlBuilderTests.cs # dotnet/src/VectorDataIntegrationTests/AzureAISearchIntegrationTests/Support/AzureAISearchTestStore.cs

…o queryAsync # Conflicts: # dotnet/src/Connectors/VectorData/VectorStorage/LoggingVectorStoreRecordCollection.cs

...src/Connectors/Connectors.Memory.Weaviate/WeaviateVectorStoreRecordCollectionQueryBuilder.cs

Co-authored-by: westey <164392973+westey-m@users.noreply.github.com>

…o queryAsync # Conflicts: # dotnet/samples/Concepts/Memory/VectorStoreEmbeddingGeneration/TextEmbeddingVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.AzureAISearch/AzureAISearchVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.AzureCosmosDBNoSQL/AzureCosmosDBNoSQLVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.MongoDB/MongoDBVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.Qdrant/QdrantVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.Weaviate/WeaviateVectorStoreRecordCollection.cs

dotnet/src/Connectors/Connectors.Memory.Pinecone/PineconeVectorStoreRecordCollection.cs

roji

Awesome to see this, am really happy we'll have this capability!

...s/Concepts/Memory/VectorStoreEmbeddingGeneration/TextEmbeddingVectorStoreRecordCollection.cs

roji · 2025-04-04T13:38:40Z

...onnectors/Connectors.Memory.AzureAISearch/AzureAISearchVectorStoreCollectionCreateMapping.cs

+        {
+            IsFilterable = dataProperty.IsFilterable,
+            // Sometimes the users ask to also OrderBy given filterable property, so we make it sortable.
+            IsSortable = dataProperty.IsFilterable && !fieldType.IsCollection


Yeah, I think this is one of the reasons we thought about renaming IsFilterable to IsIndexable indeed... In the future, we can have an Azure AI-specific IsSortable flag too.

...t/src/Connectors/Connectors.Memory.AzureAISearch/AzureAISearchVectorStoreRecordCollection.cs

roji · 2025-04-04T13:45:17Z

dotnet/src/Connectors/VectorData.Abstractions/ConnectorSupport/VectorStoreRecordModel.cs

+    public VectorStoreRecordPropertyModel GetDataOrKeyProperty<TRecord>(Expression<Func<TRecord, object?>> expression)
+        => this.GetMatchingProperty<TRecord, VectorStoreRecordPropertyModel>(expression, data: true);
+
+    private TProperty GetMatchingProperty<TRecord, TProperty>(Expression<Func<TRecord, object?>> expression, bool data)


Just noting that this parameter is mainly there because of VectorStoreGenericDataModel, which has different dictionaries for data/vectors; that's going away with the transition to Dictionary<string, object?> tracked by #10802 (PR should be out this weekend).

In other words, I think this is super temporary and will be gone soon (so should be OK to merge as-is for now). We'd also rename GetDataOrKeyProperty to something like BindProperty (I'm already thinking about this also in relation to the filter translators...).

dotnet/src/Connectors/VectorData.Abstractions/VectorStorage/IVectorStoreRecordCollection.cs

dotnet/src/Connectors/Connectors.Memory.Qdrant/QdrantVectorStoreRecordCollection.cs

dotnet/src/Connectors/Connectors.Memory.Postgres/SqlBuilder.cs

roji · 2025-04-04T20:49:40Z

dotnet/src/Connectors/Connectors.Memory.Postgres/SqlBuilder.cs

+        foreach (var property in model.Properties)
+        {
+            if (options.IncludeVectors || property is not VectorStoreRecordVectorPropertyModel)
+            {
+                query.AppendFormat("\"{0}\",", property.StorageName);
+            }
+        }


Yeah, we should definitely do some passes to consolidate/refactor.

dotnet/src/Connectors/Connectors.Memory.MongoDB/MongoDBVectorStoreRecordCollection.cs

.../Connectors.Memory.AzureCosmosDBNoSQL/AzureCosmosDBNoSQLVectorStoreCollectionQueryBuilder.cs

dotnet/src/Connectors/VectorData.Abstractions/RecordOptions/FilterOptions.cs

…o queryAsync # Conflicts: # dotnet/src/Connectors/Connectors.Memory.AzureCosmosDBMongoDB/AzureCosmosDBMongoDBVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.InMemory/InMemoryVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.Pinecone/PineconeVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresVectorStoreDbClient.cs # dotnet/src/Connectors/Connectors.Memory.Postgres/PostgresVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.Redis/RedisHashSetVectorStoreRecordCollection.cs # dotnet/src/Connectors/Connectors.Memory.Redis/RedisJsonVectorStoreRecordCollection.cs # dotnet/src/Connectors/VectorData.Abstractions/CompatibilitySuppressions.xml

roji

OK, there seem to be a few more unresolved comments, but nothing that looks blocking - LGTM. Thanks @adamsitnik!

dotnet/src/Connectors/VectorData.Abstractions/RecordOptions/FilterOptions.cs

…o queryAsync # Conflicts: # dotnet/src/Connectors/Connectors.Memory.AzureAISearch/AzureAISearchVectorStoreRecordCollection.cs

- rename FilterOptions to GetFilteredRecordOptions - rename Sort to OrderBy - remove the limitation to select Key as the Redis ordeby field

QueryAsync: initial implementation for Pinecone, Qdrant and Weaviate

0096e21

adamsitnik added .NET Issue or Pull requests regarding .NET code msft.ext.vectordata Related to Microsoft.Extensions.VectorData labels Mar 21, 2025

markwallace-microsoft added kernel Issues or pull requests impacting the core kernel memory labels Mar 21, 2025

github-actions bot changed the title ~~[DRAFT] .Net MEVD: QueryAsync: initial implementation for Pinecone, Qdrant and Weaviate~~ .Net: [DRAFT] .Net MEVD: QueryAsync: initial implementation for Pinecone, Qdrant and Weaviate Mar 21, 2025

adamsitnik had a problem deploying to integration March 21, 2025 15:35 — with GitHub Actions Failure

don't perform any kind of client-side ordering

c82106b

adamsitnik had a problem deploying to integration March 21, 2025 16:14 — with GitHub Actions Failure

implement QueryAsync for InMemory, Azure AI Search, MongoDB and CosmosDB

ff98cd8

adamsitnik had a problem deploying to integration March 24, 2025 17:44 — with GitHub Actions Failure

adamsitnik changed the title ~~.Net: [DRAFT] .Net MEVD: QueryAsync: initial implementation for Pinecone, Qdrant and Weaviate~~ .Net: [DRAFT] MEVD: QueryAsync Mar 24, 2025

implement QueryAsync for Redis, SQL Server

c93dacd

adamsitnik had a problem deploying to integration March 25, 2025 17:53 — with GitHub Actions Failure

implement QueryAsync for PostgreSql and Sqlite

ddfc5e5

adamsitnik had a problem deploying to integration March 26, 2025 14:11 — with GitHub Actions Failure

fix the build

c4bf2b5

adamsitnik temporarily deployed to integration March 27, 2025 16:14 — with GitHub Actions Inactive

adamsitnik changed the title ~~.Net: [DRAFT] MEVD: QueryAsync~~ .Net MEVD: QueryAsync Mar 27, 2025

adamsitnik marked this pull request as ready for review March 27, 2025 16:38

adamsitnik requested a review from a team as a code owner March 27, 2025 16:38

This comment was marked as outdated.

Sign in to view

This comment was marked as resolved.

Sign in to view

adamsitnik added 4 commits April 1, 2025 16:47

change the API design to what we have agreed on part 1/2

72319f8

change the API design to what we have agreed on part 2/2

ef18c35

Merge remote-tracking branch 'upstream/feature-vector-data-preb2' int…

0f92d08

…o queryAsync # Conflicts: # dotnet/src/Connectors/VectorData/VectorStorage/LoggingVectorStoreRecordCollection.cs

adamsitnik had a problem deploying to integration April 2, 2025 11:27 — with GitHub Actions Failure

fix the build

9072402

adamsitnik temporarily deployed to integration April 2, 2025 11:43 — with GitHub Actions Inactive

westey-m reviewed Apr 4, 2025

View reviewed changes

...src/Connectors/Connectors.Memory.Weaviate/WeaviateVectorStoreRecordCollectionQueryBuilder.cs Outdated Show resolved Hide resolved

Apply suggestions from code review

1b34a5f

Co-authored-by: westey <164392973+westey-m@users.noreply.github.com>

adamsitnik had a problem deploying to integration April 4, 2025 11:51 — with GitHub Actions Failure

adamsitnik added 2 commits April 4, 2025 16:28

address code review feedback

69d44fc

adamsitnik had a problem deploying to integration April 4, 2025 14:34 — with GitHub Actions Error

adamsitnik commented Apr 4, 2025

View reviewed changes

dotnet/src/Connectors/Connectors.Memory.Pinecone/PineconeVectorStoreRecordCollection.cs Outdated Show resolved Hide resolved

Apply suggestions from code review

9b226c5

adamsitnik had a problem deploying to integration April 4, 2025 14:36 — with GitHub Actions Failure

fix the build

281a9e0

adamsitnik temporarily deployed to integration April 4, 2025 15:01 — with GitHub Actions Inactive

adamsitnik requested a review from westey-m April 4, 2025 15:33

roji requested changes Apr 4, 2025

View reviewed changes

address code review feedback

287b4f6

adamsitnik temporarily deployed to integration April 7, 2025 10:27 — with GitHub Actions Inactive

adamsitnik commented Apr 7, 2025

View reviewed changes

dotnet/src/Connectors/VectorData.Abstractions/RecordOptions/FilterOptions.cs Outdated Show resolved Hide resolved

adamsitnik requested a review from roji April 7, 2025 10:36

westey-m approved these changes Apr 7, 2025

View reviewed changes

adamsitnik had a problem deploying to integration April 7, 2025 15:01 — with GitHub Actions Failure

try to fix a build error that I can't reproduce locally ...

247749f

adamsitnik had a problem deploying to integration April 7, 2025 15:44 — with GitHub Actions Failure

why on earth is it still failing and only in CI?

304068e

adamsitnik had a problem deploying to integration April 7, 2025 15:56 — with GitHub Actions Failure

roji approved these changes Apr 8, 2025

View reviewed changes

dotnet/src/Connectors/VectorData.Abstractions/RecordOptions/FilterOptions.cs Outdated Show resolved Hide resolved

dotnet/src/Connectors/VectorData.Abstractions/RecordOptions/FilterOptions.cs Outdated Show resolved Hide resolved

adamsitnik added 2 commits April 9, 2025 11:01

Merge remote-tracking branch 'upstream/feature-vector-data-preb2' int…

0cc1bf5

…o queryAsync # Conflicts: # dotnet/src/Connectors/Connectors.Memory.AzureAISearch/AzureAISearchVectorStoreRecordCollection.cs

address code review feedback:

de72aec

- rename FilterOptions to GetFilteredRecordOptions - rename Sort to OrderBy - remove the limitation to select Key as the Redis ordeby field

adamsitnik temporarily deployed to integration April 9, 2025 09:17 — with GitHub Actions Inactive

adamsitnik merged commit 747531c into microsoft:feature-vector-data-preb2 Apr 9, 2025
11 checks passed

adamsitnik mentioned this pull request Apr 9, 2025

[MEVD] Filter-only search API (without vector similarity) #10295

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.Net MEVD: Filter-only search API (without vector similarity) #11112

.Net MEVD: Filter-only search API (without vector similarity) #11112

adamsitnik commented Mar 21, 2025 •

edited

Loading

This comment was marked as outdated.

This comment was marked as resolved.

roji left a comment

roji Apr 4, 2025

roji Apr 4, 2025

roji Apr 4, 2025

roji left a comment

.Net MEVD: Filter-only search API (without vector similarity) #11112

.Net MEVD: Filter-only search API (without vector similarity) #11112

Conversation

adamsitnik commented Mar 21, 2025 • edited Loading

This comment was marked as outdated.

This comment was marked as resolved.

roji left a comment

Choose a reason for hiding this comment

roji Apr 4, 2025

Choose a reason for hiding this comment

roji Apr 4, 2025

Choose a reason for hiding this comment

roji Apr 4, 2025

Choose a reason for hiding this comment

roji left a comment

Choose a reason for hiding this comment

adamsitnik commented Mar 21, 2025 •

edited

Loading