Non Uniform Performance with GPU - Indexing running over days even with GPU. #39110

rohitreddy1698 · 2025-01-09T06:38:47Z

rohitreddy1698
Jan 9, 2025

Hi,

I was trying to run some tests on GPU vs CPU Index performance.
I am trying to use the Zilliz Vector DB Bench Tool to run the benchmarking and capture performance metrics.

I have GPU Cluster setup on GKE :

Index Nodes : 3 n1-highmem-8 , nodes with 4 GPUs per node as index nodes :

4 NVIDIA T4 GPUs per node - 16* 4 GiB Memory = 64 GiB

index_node_replicas: 3
index_node_cpu_limit: 6
index_node_gpu_limit: 4
index_node_memory_limit: 48Gi
index_node_cpu_request: 3
index_node_gpu_request: 4
index_node_memory_request: 24Gi

1 ) When I run GPU IVF PQ for 100M Vectors 768 D with parameters :
nlist=10000, m=64 , n=16 - Is running over days. And ended in timeout.
On Index status is being build :

>>> 
>>> utility.load_state(COLLECTION_NAME)
<LoadState: NotLoad>
>>> utility.index_building_progress(COLLECTION_NAME)
{'total_rows': 100000000, 'indexed_rows': 0, 'pending_index_rows': 100000000, 'state': 'InProgress'}

>>> utility.wait_for_index_building_complete(COLLECTION_NAME) - BLOCKING , showing that index build is progress.

But on Grafana GPU Utilisation is show as 0. Attaching Image for reference.

Also is there a way to increase the Recall : with nlist=10000, m=16 , n=8 - recall is 19.13

For GPU CAGRA for same 100M Vectors 768 D dataset
Screenshot for parameters used.
The Indexing time is 5x more as compared to HNSW

Load Duration Output from Zilliz Vector DB Bench Tool :
GPU CAGRA : 195,700 s
HNSW : 37,292 s

The Same observed even for 10 M dataset :
GPU CAGRA : 32220 s
HNSW : 17430 s

But as per blogs from Milvus and in open source blogs I think CAGRA should be performing better.

Can you please help us on this.

Regards,
Rohit Mothe

yhmo · 2025-01-09T07:15:57Z

yhmo
Jan 9, 2025
Collaborator

{'total_rows': 100000000, 'indexed_rows': 0, 'pending_index_rows': 100000000, 'state': 'InProgress'} ------------------------ I believe there must be errors in log of the index nodes, all the index tasks failed so the indexed_rows is zero.
nlist=10000, m=32 , n=16 - recall is 31.42 --------------------- I guess the ground-truth is wrong or something else. Wait @alwayslove2013 comment
Wait @Presburger comment

3 replies

rohitreddy1698 Jan 10, 2025
Author

@yhmo Thank you for your speedy reply.

I do see the error logs in the Index Nodes, Attaching the logs in the thread.
Also Can you help me with how the Indexing process differs when you use a GPU Index ?
CPU Index - When index build triggered --> Segments are loaded to Index Node Memory ( CPU ) --> Index is built --> moved back to object storage
GPU Index - When index build triggered --> Segments are loaded to Index Node Memory ( GPU ) --> Index is built --> moved back to object storage
Are these how it works ? Please correct me If I am wrong.
Recall - Waiting for @alwayslove2013 - Ground truths are proper from my knowledge because using official tool : https://github.com/zilliztech/VectorDBBench#quick-start. Same dataset working fine ~98% recall with IVF_FLAT , GPU_IVF_FLAT and HNSW , GPU_CAGRA. Only very less with IVF_PQ and GPU_IVF_PQ
CAGRA - Will try out as @Presburger suggested.

Thank you !

rohitreddy1698 Jan 10, 2025
Author

Index build taking time. I see the following errors in the logs :

I20250110 06:35:32.517393    42 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus] start to precheck chunk manager with configuration: [address=storage.googleapis.com:443, bucket_name=isds-genai-milvus-on-gpu-gcp-dev-a735359, root_path=files, storage_type=remote, cloud_provider=gcp, iam_endpoint=, log_level=warn, region=, useSSL=true, sslCACert=19, useIAM=true, useVirtualHost=false, requestTimeoutMs=10000]
I20250110 06:35:32.577046    42 ChunkManager.cpp:150] [SERVER][GcpChunkManager][milvus] init GcpChunkManager with parameter[endpoint=storage.googleapis.com:443][bucket_name=isds-genai-milvus-on-gpu-gcp-dev-a735359][root_path=files][use_secure=true]
I20250110 06:35:32.577109    42 index_factory.cc:54] [KNOWHERE][Create][milvus] use key GPU_IVF_PQ_fp32 to create knowhere index GPU_IVF_PQ with version 5
[2025/01/10 06:35:32.613 +00:00] [INFO] [indexnode/indexnode_service.go:213] ["Get Index Job Stats"] [traceID=92f5fd3f102fa7b7dd9686028581a94e] [unissued=0] [active=1] [slot=0]
**E20250110 06:35:33.146478    42 config.h:380] [KNOWHERE][Load][milvus] Out of range in json: param [nbits] should be in [4, 8].
 => failed to build index, out of range in json at /workspace/source/internal/core/src/index/VectorMemIndex.cpp:439**

[2025/01/10 06:35:33.162 +00:00] [WARN] [indexcgowrapper/helper.go:71] ["failed to create index, C Runtime Exception:  => failed to build index, out of range in json at /workspace/source/internal/core/src/index/VectorMemIndex.cpp:439\n\n"]
[2025/01/10 06:35:33.162 +00:00] [WARN] [indexnode/task_index.go:486] ["failed to build index"] [clusterID=isds-milvus] [buildID=454583754739451372] [collection=454583754739031389] [segmentID=454583754739451273] [currentIndexVersion=5] [error="failed to create index, C Runtime Exception:  => failed to build index, out of range in json at /workspace/source/internal/core/src/index/VectorMemIndex.cpp:439\n\n: segcore error[segcoreCode=2004]"] [errorVerbose="failed to create index, C Runtime Exception:  => failed to build index, out of range in json at /workspace/source/internal/core/src/index/VectorMemIndex.cpp:439: segcore error[segcoreCode=2004]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrSegcore\n  | \t/workspace/source/pkg/util/merr/utils.go:993\n  | github.com/milvus-io/milvus/internal/util/indexcgowrapper.HandleCStatus\n  | \t/workspace/source/internal/util/indexcgowrapper/helper.go:78\n  | github.com/milvus-io/milvus/internal/util/indexcgowrapper.CreateIndex\n  | \t/workspace/source/internal/util/indexcgowrapper/index.go:109\n  | github.com/milvus-io/milvus/internal/indexnode.(*indexBuildTask).Execute\n  | \t/workspace/source/internal/indexnode/task_index.go:481\n  | github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).processTask.func1\n  | \t/workspace/source/internal/indexnode/task_scheduler.go:222\n  | github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).processTask\n  | \t/workspace/source/internal/indexnode/task_scheduler.go:235\n  | github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).indexBuildLoop.func1\n  | \t/workspace/source/internal/indexnode/task_scheduler.go:262\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) failed to create index, C Runtime Exception:  => failed to build index, out of range in json at /workspace/source/internal/core/src/index/VectorMemIndex.cpp:439\nWraps: (3) segcore error[segcoreCode=2004]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/01/10 06:35:33.162 +00:00] [WARN] [indexnode/task_scheduler.go:236] ["process task failed"] [error="failed to create index, C Runtime Exception:  => failed to build index, out of range in json at /workspace/source/internal/core/src/index/VectorMemIndex.cpp:439\n\n: segcore error[segcoreCode=2004]"] [errorVerbose="failed to create index, C Runtime Exception:  => failed to build index, out of range in json at /workspace/source/internal/core/src/index/VectorMemIndex.cpp:439: segcore error[segcoreCode=2004]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/pkg/util/merr.WrapErrSegcore\n  | \t/workspace/source/pkg/util/merr/utils.go:993\n  | github.com/milvus-io/milvus/internal/util/indexcgowrapper.HandleCStatus\n  | \t/workspace/source/internal/util/indexcgowrapper/helper.go:78\n  | github.com/milvus-io/milvus/internal/util/indexcgowrapper.CreateIndex\n  | \t/workspace/source/internal/util/indexcgowrapper/index.go:109\n  | github.com/milvus-io/milvus/internal/indexnode.(*indexBuildTask).Execute\n  | \t/workspace/source/internal/indexnode/task_index.go:481\n  | github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).processTask.func1\n  | \t/workspace/source/internal/indexnode/task_scheduler.go:222\n  | github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).processTask\n  | \t/workspace/source/internal/indexnode/task_scheduler.go:235\n  | github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).indexBuildLoop.func1\n  | \t/workspace/source/internal/indexnode/task_scheduler.go:262\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) failed to create index, C Runtime Exception:  => failed to build index, out of range in json at /workspace/source/internal/core/src/index/VectorMemIndex.cpp:439\nWraps: (3) segcore error[segcoreCode=2004]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2025/01/10 06:35:33.265 +00:00] [INFO] [indexnode/indexnode_service.go:213] ["Get Index Job Stats"] [traceID=e6602d11e36a344ffc52480c0fc2b8f5] [unissued=0] [active=0] [slot=1]
[2025/01/10 06:35:33.365 +00:00] [INFO] [indexnode/indexnode_service.go:213] ["Get Index Job Stats"] [traceID=3ae1e622289fc6217d1b85c932ff0fb5] [unissued=0] [active=0] [slot=1]
[2025/01/10 06:35:33.368 +00:00] [INFO] [indexnode/indexnode_service.go:291] ["IndexNode receive CreateJob request..."] [traceID=eb8e438046ecff0d8953640703219614] [clusterID=isds-milvus] [taskID=454583754739451232] [jobType=JobTypeIndexJob]
[2025/01/10 06:35:33.368 +00:00] [INFO] [indexnode/indexnode_service.go:296] ["IndexNode building index ..."] [traceID=eb8e438046ecff0d8953640703219614] [clusterID=isds-milvus] [taskID=454583754739451232] [jobType=JobTypeIndexJob] [indexID=0] [indexName=] [indexFilePrefix=files/index_files] [indexVersion=82394] [dataPaths="[]"] [typeParams="[{\"key\":\"dim\",\"value\":\"768\"}]"] [indexParams="[{\"key\":\"m\",\"value\":\"64\"},{\"key\":\"nbits\",\"value\":\"16\"},{\"key\":\"cache_dataset_on_device\",\"value\":\"false\"},{\"key\":\"metric_type\",\"value\":\"L2\"},{\"key\":\"index_type\",\"value\":\"GPU_IVF_PQ\"},{\"key\":\"nlist\",\"value\":\"10000\"}]"] [numRows=43976] [current_index_version=5] [storePath=] [storeVersion=0] [indexStorePath=] [dim=768]
[2025/01/10 06:35:33.459 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=gcp] [bucketname=isds-genai-milvus-on-gpu-gcp-dev-a735359] [root=files]
[2025/01/10 06:35:33.459 +00:00] [INFO] [indexnode/task_index.go:309] ["IndexNode IndexBuilderTask Enqueue"] [buildID=454583754739451232] [segmentID=454583754739451129]
[2025/01/10 06:35:33.459 +00:00] [INFO] [indexnode/indexnode_service.go:345] ["IndexNode index job enqueued successfully"] [traceID=eb8e438046ecff0d8953640703219614] [clusterID=isds-milvus] [taskID=454583754739451232] [jobType=JobTypeIndexJob] [indexName=]
[2025/01/10 06:35:33.459 +00:00] [INFO] [indexnode/task_index.go:316] ["Begin to prepare indexBuildTask"] [buildID=454583754739451232] [Collection=454583754739031389] [SegmentID=454583754739451129]
[2025/01/10 06:35:33.459 +00:00] [INFO] [indexnode/task_index.go:381] ["Successfully prepare indexBuildTask"] [buildID=454583754739451232] [collectionID=454583754739031389] [segmentID=454583754739451129]
[2025/01/10 06:35:33.459 +00:00] [INFO] [indexnode/task_index.go:479] ["debug create index"] [clusterID=isds-milvus] [buildID=454583754739451232] [collection=454583754739031389] [segmentID=454583754739451129] [currentIndexVersion=5] [buildIndexParams="clusterID:\"isds-milvus\" buildID:454583754739451232 collectionID:454583754739031389 partitionID:454583754739031390 segmentID:454583754739451129 index_version:82394 current_index_version:5 num_rows:43976 dim:768 index_file_prefix:\"files/index_files\" insert_files:\"files/insert_log/454583754739031389/454583754739031390/454583754739451129/102/454583754739051093\" insert_files:\"files/insert_log/454583754739031389/454583754739031390/454583754739451129/102/454583754739051105\" insert_files:\"files/insert_log/454583754739031389/454583754739031390/454583754739451129/102/454583754739051112\" insert_files:\"files/insert_log/454583754739031389/454583754739031390/454583754739451129/102/454583754739051117\" insert_files:\"files/insert_log/454583754739031389/454583754739031390/454583754739451129/102/454583754739051123\" insert_files:\"files/insert_log/454583754739031389/454583754739031390/454583754739451129/102/454583754739051131\" insert_files:\"files/insert_log/454583754739031389/454583754739031390/454583754739451129/102/454583754739051133\" insert_files:\"files/insert_log/454583754739031389/454583754739031390/454583754739451129/102/454583754739051149\" field_schema:<fieldID:102 name:\"vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"768\" > > storage_config:<address:\"storage.googleapis.com:443\" useSSL:true bucket_name:\"isds-genai-milvus-on-gpu-gcp-dev-a735359\" root_path:\"files\" useIAM:true storage_type:\"remote\" cloud_provider:\"gcp\" request_timeout_ms:10000 sslCACert:\"/path/to/public.crt\" > index_params:<key:\"m\" value:\"64\" > index_params:<key:\"nbits\" value:\"16\" > index_params:<key:\"cache_dataset_on_device\" value:\"false\" > index_params:<key:\"metric_type\" value:\"L2\" > index_params:<key:\"index_type\" value:\"GPU_IVF_PQ\" > index_params:<key:\"nlist\" value:\"10000\" > index_params:<key:\"dim\" value:\"768\" > type_params:<key:\"dim\" value:\"768\" > "]
I20250110 06:35:33.460109   277 MinioChunkManager.cpp:161] [SERVER][InitSDKAPIDefault][milvus] init aws with log level:warn
[2025/01/10 06:35:33.464 +00:00] [INFO] [indexnode/indexnode_service.go:213] ["Get Index Job Stats"] [traceID=2be7663ee2be8da02be7db64f578afde] [unissued=0] [active=1] [slot=0]

E20250110 06:35:33.146478 42 config.h:380] [KNOWHERE][Load][milvus] Out of range in json: param [nbits] should be in [4, 8].
=> failed to build index, out of range in json at /workspace/source/internal/core/src/index/VectorMemIndex.cpp:439 - Specifically this.
Is it necessary or is there a restriction on the nbits while using IVF_PQ or GPU_IVF_PQ.

yhmo Jan 10, 2025
Collaborator

Which version of your milvus? @rohitreddy1698 Could you attach the full log of an index node?

Out of range in json: param [nbits] should be in [4, 8]. -------------------------------------- It is weird here.
In milvus document, the range of nbits is [1, 16], but this error log shows the range is [4, 8]
https://milvus.io/docs/gpu_index.md#GPUIVFPQ

In Knowhere source code, the nbits range is [1, 24]: https://github.com/zilliztech/knowhere/blob/52c03030028ca0e89c9f31538b5391e5b4a0e5dd/src/index/ivf/ivf_config.h#L75

Wait @Presburger comment.

Presburger · 2025-01-09T07:29:37Z

Presburger
Jan 9, 2025

@rohitreddy1698 Hi, please replace the IVF_PQ in the build algorithm with NN_DESCENT. IVF_PQ performs a brute-force search-like method to construct the adjacency list, which can be very time-consuming on cards like the T4.

3 replies

rohitreddy1698 Jan 10, 2025
Author

@Presburger .
Sure got it. For GPU_CAGRA - Let me try replacing the IVF_PQ with NN_DESCENT to see if this reduces the build time !
Thank you for the information.

Problem -
Can you please help me on the GPU_IVF_PQ Indexing taking forever to contruct as well. The process has been running for 2-3 days and after that timedout.

It is timing out after 50h for 100M records. Which is too much time.

2025-01-07 17:25:20,438 | INFO: (SpawnProcess-1:1) Finish loading all dataset into VectorDB, dur=27251.372383206617 (serial_runner.py:61) (252)
2025-01-07 17:25:22,925 | INFO: 9999999 - Task Runner Finished Loading 101 (task_runner.py:165) (84)
2025-01-07 17:25:22,929 | INFO: 9999999 - Task Runner Optimise 103 (task_runner.py:247) (84)
2025-01-07 17:25:23,067 | INFO: 9999999 - Task Runner Optimise 104 (task_runner.py:251) (84)
2025-01-07 17:25:33,345 | INFO: 9999999 - Starting Post Insert 11 (milvus.py:92) (200465)
2025-01-07 17:25:33,345 | INFO: Milvus post insert before optimize (milvus.py:103) (200465)
2025-01-07 17:25:33,346 | INFO: 9999999 - Starting Flush 12 (milvus.py:105) (200465)
2025-01-07 17:25:33,346 | INFO: 
index_param: {'metric_type': 'L2', 'index_type': 'GPU_IVF_PQ', 'params': {'nlist': 10000, 'm': 64, 'nbits': 16, 'cache_dataset_on_device': 'false'}} (milvus.py:106) (200465)
2025-01-07 17:25:33,346 | INFO: 
_vector_field: vector (milvus.py:107) (200465)
2025-01-07 17:25:33,346 | INFO: 
_index_name: vector_idx (milvus.py:108) (200465)
2025-01-07 17:25:35,914 | INFO: 9999999 - Finished Flush 13 (milvus.py:111) (200465)
2025-01-09 19:25:23,071 | WARNING: VectorDB optimize timeout in 180000 (task_runner.py:254) (84)
2025-01-09 19:25:23,242 | WARNING: Failed to run performance case, reason = Performance case optimize timeout (task_runner.py:194) (84)
Traceback (most recent call last):
  File "/opt/code/vectordb_bench/backend/task_runner.py", line 252, in _optimize
    return future.result(timeout=self.ca.optimize_timeout)[1]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 458, in result
    raise TimeoutError()
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/code/vectordb_bench/backend/task_runner.py", line 166, in _run_perf_case
    build_dur = self._optimize()
                ^^^^^^^^^^^^^^^^
  File "/opt/code/vectordb_bench/backend/task_runner.py", line 257, in _optimize
    raise PerformanceTimeoutError("Performance case optimize timeout") from e
vectordb_bench.models.PerformanceTimeoutError: Performance case optimize timeout
2025-01-09 19:25:23,297 | WARNING: [1/1] case {'label': <CaseLabel.Performance: 2>, 'dataset': {'data': {'name': 'LAION', 'size': 100000000, 'dim': 768, 'metric_type': <MetricType.L2: 'L2'>}}, 'db': 'Milvus'} failed to run, reason=Performance case optimize timeout (interface.py:179) (84)

Presburger Jan 10, 2025

Could you provide some logs from Milvus?

rohitreddy1698 Jan 13, 2025
Author

Hi @Presburger , I have provided some logs in thread above with @yhmo , let me add additional logs.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Non Uniform Performance with GPU - Indexing running over days even with GPU. #39110

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 6 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Non Uniform Performance with GPU - Indexing running over days even with GPU. #39110

rohitreddy1698 Jan 9, 2025

Replies: 2 comments · 6 replies

yhmo Jan 9, 2025 Collaborator

rohitreddy1698 Jan 10, 2025 Author

rohitreddy1698 Jan 10, 2025 Author

yhmo Jan 10, 2025 Collaborator

Presburger Jan 9, 2025

rohitreddy1698 Jan 10, 2025 Author

Presburger Jan 10, 2025

rohitreddy1698 Jan 13, 2025 Author

rohitreddy1698
Jan 9, 2025

Replies: 2 comments 6 replies

yhmo
Jan 9, 2025
Collaborator

rohitreddy1698 Jan 10, 2025
Author

rohitreddy1698 Jan 10, 2025
Author

yhmo Jan 10, 2025
Collaborator

Presburger
Jan 9, 2025

rohitreddy1698 Jan 10, 2025
Author

rohitreddy1698 Jan 13, 2025
Author