Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[intel-npu] Support new internal cached_model_buffer config for memory mapped cached blobs #27822

Open
wants to merge 28 commits into
base: master
Choose a base branch
from
Open

[intel-npu] Support new internal cached_model_buffer config for memory mapped cached blobs #27822

wants to merge 28 commits into from
+156 −56

Conversation

MirceaDan99
Copy link
Contributor

@MirceaDan99 MirceaDan99 commented Nov 29, 2024 •
edited
Loading

Details:

  • Based on new import_model API from PR New caching api #27644 new plugins property to pass mmap buffer from PR New plugins property to pass mmap buffer #27981
  • Added BlobContainer class for IGraph objects that may derive with BlobContainerAlignedBuffer for the new import_model API and BlobContainerVector for the old one
  • Refactored getGraphHandle function to allow passing const uint8_t and size_t params instead of `std::vector<uint8_t>

Tickets:

  • 157192

@github-actions github-actions bot added category: inference OpenVINO Runtime library - Inference category: Core OpenVINO Core (aka ngraph) category: CPU OpenVINO CPU plugin category: NPU OpenVINO NPU plugin labels Nov 29, 2024
@MirceaDan99 MirceaDan99 force-pushed the POC/add_caching_with_mmap_property branch from ed220a4 to 94e33c4 Compare November 29, 2024 14:36
@MirceaDan99 MirceaDan99 force-pushed the POC/add_caching_with_mmap_property branch 3 times, most recently from 539c459 to 8cfb954 Compare December 11, 2024 09:34
@github-actions github-actions bot removed category: inference OpenVINO Runtime library - Inference category: Core OpenVINO Core (aka ngraph) category: CPU OpenVINO CPU plugin labels Dec 11, 2024
@MirceaDan99 MirceaDan99 force-pushed the POC/add_caching_with_mmap_property branch 2 times, most recently from 1f163fa to 5a29e3c Compare December 11, 2024 10:11
@MirceaDan99 MirceaDan99 changed the title [intel-npu] Support new internal import_model API for memory mapped cached blobs [intel-npu] Support new internal cached_model_buffer config for memory mapped cached blobs Dec 11, 2024
@MirceaDan99 MirceaDan99 marked this pull request as ready for review December 11, 2024 10:30
@MirceaDan99 MirceaDan99 requested review from a team as code owners December 11, 2024 10:30
@MirceaDan99 MirceaDan99 force-pushed the POC/add_caching_with_mmap_property branch from 5a29e3c to 60a36c4 Compare December 11, 2024 12:12
@MirceaDan99 MirceaDan99 force-pushed the POC/add_caching_with_mmap_property branch 6 times, most recently from 9859b15 to 99b823b Compare December 12, 2024 15:47
@alexandruenache1111 alexandruenache1111 mentioned this pull request Dec 18, 2024
Merged
@MirceaDan99 MirceaDan99 force-pushed the POC/add_caching_with_mmap_property branch from 99b823b to 122788e Compare January 13, 2025 13:24
@MirceaDan99 MirceaDan99 requested review from a team as code owners January 13, 2025 13:24
@github-actions github-actions bot added the category: CPU OpenVINO CPU plugin label Jan 13, 2025
alexandruenache1111 and others added 24 commits January 15, 2025 15:41
@alexandruenache1111 @MirceaDan99
Updated writeAndReadInvalidMetadataVersion test body
ebb78a4 
Signed-off-by: Alexandru Enache <alexandru.enache@intel.com>
@alexandruenache1111 @MirceaDan99
Move version functions as static methods inside MetadataBase
45e3fef 
Signed-off-by: Alexandru Enache <alexandru.enache@intel.com>
@alexandruenache1111 @MirceaDan99
Nitpicks
37c53b9 
Signed-off-by: Alexandru Enache <alexandru.enache@intel.com>
@alexandruenache1111 @MirceaDan99
Move version field to MetadataBase
a1b89e6 
Signed-off-by: Alexandru Enache <alexandru.enache@intel.com>
@MirceaDan99
Add ov::internal::caching_with_mmap property logic
8aff5a6
@MirceaDan99
Refactor compiler type selection
0544093
@MirceaDan99
Fix OV cache header not being removed from blob for memory mapped cac…
f09b447 
…he file
@MirceaDan99
Keep shared_ptr of blob in IGraph to fix export_model for import …
e8b6b72 
…scenario
@MirceaDan99
Refactor changes for CIP & Drop parse function from `ICompilerAdapt…
1ffb961 
…er`, so only `std::shared_ptr<ov::AlignedBuffer>` parameters will be used
@olpipi @MirceaDan99
Update plugin API to import model with mmap buffer
2793039
@MirceaDan99
Use new import_model with model_buffer API
27f8ec7
@MirceaDan99
New fix for adding offset to model_buffer relative to end position …
e53c3ea 
…of OV cache header
@MirceaDan99
Fix std::vector being moved after accesing its .data() and `.size…
42ca42e 
…()` functions
@MirceaDan99
Refactor getGraphHandle to drop dependency to ov::AlignedBuffer
4932d30
@MirceaDan99
Refactor import_model new API to accept only either std::istream …
aa44d2e 
…or `ov::AlignedBuffer`
@MirceaDan99
Re-add DriverGraph::release_blob method and adapt to `ov::AlignedBu…
cf006c2 
…ffer` (no release for mmap shared object)
@MirceaDan99
Code clean-up
67cd64c
@MirceaDan99
Revert changes in new import_model API, so NPU plugin will have `…
2f404c1 
…unusedStream` when `model_buffer` is given
@MirceaDan99
Add BlobContainer class and derivates for each `std::vector<uint8_t…
c8f4abf 
…>` and `std::shared_ptr<ov::AlignedBuffer>` blob types
@MirceaDan99
Fix clang formats
6badc9a
@MirceaDan99
Use alternative from PR openvinotoolkit#27981 instead for memory ma…
1077884 
…pped buffers
@MirceaDan99
Add suggested changes
0eea082
@MirceaDan99
Prepare BlobContainerAlignedBuffer for OV versioning metadata
0f8caef
@MirceaDan99
Fix broken stream processed by NPUW
2d290ef
@MirceaDan99 MirceaDan99 force-pushed the POC/add_caching_with_mmap_property branch from ab9eff2 to fcd4a45 Compare January 15, 2025 13:55
@github-actions github-actions bot removed the category: build OpenVINO cmake script / infra label Jan 15, 2025
@MirceaDan99 MirceaDan99 force-pushed the POC/add_caching_with_mmap_property branch from fcd4a45 to 49e880b Compare January 15, 2025 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pereanub pereanub pereanub left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@pereanub pereanub

@razvanapetroaie razvanapetroaie

Labels
category: NPU OpenVINO NPU plugin
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@MirceaDan99 @pereanub @razvanapetroaie @alexandruenache1111 @olpipi