Merge branch 'master' into uk/changing-sub-byte-i4-element-order

Author: ujjayant-kadian

.github/workflows/coverage.yml

@@ -25,7 +25,7 @@ jobs:
 
 
       - name: Setup ccache
-        uses: hendrikmuhs/ccache-action@c92f40bee50034e84c763e33b317c77adaa81c92 # v1.2.13
+        uses: hendrikmuhs/ccache-action@ed74d11c0b343532753ecead8a951bb09bb34bc9 # v1.2.14
         with:
           max-size: 50G
 

.github/workflows/mac.yml

@@ -131,7 +131,7 @@ jobs:
       #
 
       - name: Setup ccache
-        uses: hendrikmuhs/ccache-action@c92f40bee50034e84c763e33b317c77adaa81c92 # v1.2.13
+        uses: hendrikmuhs/ccache-action@ed74d11c0b343532753ecead8a951bb09bb34bc9 # v1.2.14
         with:
           max-size: "2000M"
           # Should save cache only if run in the master branch of the base repo

.github/workflows/mac_arm64.yml

@@ -131,7 +131,7 @@ jobs:
       #
 
       - name: Setup ccache
-        uses: hendrikmuhs/ccache-action@c92f40bee50034e84c763e33b317c77adaa81c92 # v1.2.13
+        uses: hendrikmuhs/ccache-action@ed74d11c0b343532753ecead8a951bb09bb34bc9 # v1.2.14
         with:
           max-size: "2000M"
           # Should save cache only if run in the master branch of the base repo

docs/articles_en/get-started/configurations.rst

@@ -1,6 +1,4 @@
-.. {#openvino_docs_install_guides_configurations_header}
-
-Additional Configurations For Hardware
+Additional Configurations
 ======================================
 
 
@@ -16,10 +14,10 @@ Additional Configurations For Hardware
 
    For GPU <configurations/configurations-intel-gpu>
    For NPU <configurations/configurations-intel-npu>
+   GenAI Dependencies <configurations/genai-dependencies>
 
-For certain use cases, you may need to install additional software, to use the full
-potential of OpenVINO™. Check the following list for components for elements used in
-your workflow:
+For certain use cases, you may need to install additional software, to benefit from the full
+potential of OpenVINO™. Check the following list for components used in your workflow:
 
 | **GPU drivers**
 |   If you want to run inference on a GPU, make sure your GPU's drivers are properly installed.
@@ -33,6 +31,11 @@ your workflow:
     See the :doc:`guide on NPU configuration <configurations/configurations-intel-npu>`
     for details.
 
+| **OpenVINO GenAI Dependencies**
+|   OpenVINO GenAI is a flavor of OpenVINO, aiming to simplify running generative
+    AI models. For information on the dependencies required to use OpenVINO GenAI, see the
+    :doc:`guide on OpenVINO GenAI Dependencies <configurations/genai-dependencies>`.
+
 | **Open Computer Vision Library**
 |   OpenCV is used to extend the capabilities of some models, for example enhance some of
     OpenVINO samples, when used as a dependency in compilation. To install OpenCV for OpenVINO, see the

docs/articles_en/get-started/configurations/genai-dependencies.rst

@@ -0,0 +1,31 @@
+OpenVINO™ GenAI Dependencies
+=================================
+
+OpenVINO™ GenAI depends on both `OpenVINO <https://github.com/openvinotoolkit/openvino>`__ and
+`OpenVINO Tokenizers <https://github.com/openvinotoolkit/openvino_tokenizers>`__. During OpenVINO™
+GenAI installation from PyPi, the same versions of OpenVINO and OpenVINO Tokenizers
+are used (e.g. ``openvino==2024.3.0`` and ``openvino-tokenizers==2024.3.0.0`` are installed for
+``openvino-genai==2024.3.0``).
+
+Trying to update any of the dependency packages might result in a version incompatiblibty
+due to different Application Binary Interfaces (ABIs), which will result in errors while running
+OpenVINO GenAI. Having package version in the ``<MAJOR>.<MINOR>.<PATCH>.<REVISION>`` format, allows
+changing the ``<REVISION>`` portion of the full version to ensure ABI compatibility. Changing
+``<MAJOR>``, ``<MINOR>`` or ``<PATCH>`` part of the version may break ABI.
+
+GenAI, Tokenizers, and OpenVINO wheels for Linux on PyPI are compiled with ``_GLIBCXX_USE_CXX11_ABI=0``
+to cover a wider range of platforms. In the C++ archive distributions for Ubuntu, ``_GLIBCXX_USE_CXX11_ABI=1``
+is used instead. Mixing different ABIs is not possible as doing so will result in a link error.
+
+To try OpenVINO GenAI with different dependencies versions (which are **not** prebuilt packages
+as archives or python wheels), build OpenVINO GenAI library from
+`Source <https://github.com/openvinotoolkit/openvino.genai/blob/releases/2024/3/src/docs/BUILD.md#build-openvino-openvino-tokenizers-and-openvino-genai-from-source>`__.
+
+Additional Resources
+#######################
+
+* :doc:`OpenVINO GenAI Installation Guide <../install-openvino/install-openvino-genai>`
+* `OpenVINO GenAI repository <https://github.com/openvinotoolkit/openvino.genai>`__
+* :doc:`OpenVINO Installation Guide <../install-openvino>`
+* :doc:`OpenVINO Tokenizers <../../learn-openvino/llm_inference_guide/ov-tokenizers>`
+

docs/articles_en/get-started/install-openvino/install-openvino-genai.rst

@@ -11,7 +11,9 @@ To see GenAI in action, check the Jupyter notebooks:
 `LLM-powered Chatbot <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-chatbot/README.md>`__ and
 `LLM Instruction-following pipeline <https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-question-answering/README.md>`__.
 
-The OpenVINO GenAI flavor is available for installation via PyPI and Archive distributions:
+The OpenVINO GenAI flavor is available for installation via PyPI and Archive distributions.
+A `detailed guide <https://github.com/openvinotoolkit/openvino.genai/blob/releases/2024/3/src/docs/BUILD.md>`__
+on how to build OpenVINO GenAI is available in the OpenVINO GenAI repository.
 
 PyPI Installation
 ###############################

docs/dev/build_linux.md

@@ -74,7 +74,7 @@ You can use the following additional build options:
      ```
   3. After the build process finishes, export the newly built Python libraries to the user environment variables:
      ```
-     export PYTHONPATH=<openvino_repo>/bin/intel64/Release/python:$PYTHONPATH
+     export PYTHONPATH=<openvino_repo>/bin/intel64/Release/python:<openvino_repo>/tools/ovc:$PYTHONPATH
      export LD_LIBRARY_PATH=<openvino_repo>/bin/intel64/Release:$LD_LIBRARY_PATH
      ```
      or install the wheel with pip:

docs/dev/build_windows.md

@@ -61,7 +61,7 @@ Supported configurations:
      ```
   3. After the build process finishes, export the newly built Python libraries to the user environment variables:
      ```
-     set PYTHONPATH=<openvino_repo>/bin/<arch>/Release/python;%PYTHONPATH%
+     set PYTHONPATH=<openvino_repo>/bin/<arch>/Release/python;<openvino_repo>/tools/ovc;%PYTHONPATH%
      set OPENVINO_LIB_PATHS=<openvino_repo>/bin/<arch>/Release;<openvino_repo>/temp/tbb/bin
      ```
      or install the wheel with pip:

src/common/transformations/include/transformations/common_optimizations/move_eltwise_up_data_movement.hpp

@@ -11,10 +11,56 @@
 namespace ov {
 namespace pass {
 
-class TRANSFORMATIONS_API MoveEltwiseUpThroughDataMov : public ov::pass::MatcherPass {
+/// This transformation tries to put element-wise operations (Unary or Binary with scalar second input) before a set of
+/// data movement ops in order to allow further element-wise op fusion to previous op and zero-copy optimizations for
+/// data movement op itself.
+///     ┌───────────┐                           ┌───────────┐
+///     │   AnyOp   │                           │   AnyOp   │
+///     └─────┬─────┘                           └─────┬─────┘
+///           │                                       │
+///           │                                       │
+///   ┌───────┴────────┐                      ┌───────┴────────┐
+///   | DataMovementOp |          =>          |  Element-Wise  |
+///   └───────┬────────┘                      └───────┬────────┘
+///           │                                       │
+///           │                                       │
+///   ┌───────┴────────┐                      ┌───────┴────────┐
+///   │  Element-Wise  |                      │ DataMovementOp |
+///   └────────────────┘                      └────────────────┘
+class TRANSFORMATIONS_API MoveEltwiseUpThroughDataMovScalar : public ov::pass::MatcherPass {
+public:
+    OPENVINO_RTTI("MoveEltwiseUpThroughDataMovScalar", "0");
+    MoveEltwiseUpThroughDataMovScalar(std::vector<DiscreteTypeInfo> allowed_data_movement_ops);
+};
+
+/// This transformation tries to put element-wise operations before Reshape/Squeeze/Unsqueeze ops
+/// when second input to eltwise is per-channel Constant op
+///     ┌───────────┐       ┌────────────────┐                    ┌───────────┐            ┌────────────────────┐
+///     │   AnyOp   │       │  TargetShape   │                    │   AnyOp   │            │  Per-Channel Const │
+///     └─────┬─────┘       └────────┬───────┘                    └─────┬─────┘            └─────────┬──────────┘
+///           │                      │                                  │                            │
+///           │                      |                                  │                            |
+///           │   ┌─────────┐        │                                  │   ┌──────────────┐         │
+///           └───┤ Reshape ├────────┘                    =>            └───┤ Element-Wise ├─────────┘
+///               └────┬────┘                                               └───────┬──────┘
+///                    │                                                            │
+///                    │                                                            │
+///            ┌───────┴────────┐    ┌────────────────────┐                   ┌─────┴─────┐    ┌─────────────┐
+///            │  Element-Wise  ├────┤  Per-Channel Const │                   │  Reshape  ├────┤ TargetShape │
+///            └────────────────┘    └────────────────────┘                   └───────────┘    └─────────────┘
+class TRANSFORMATIONS_API MoveEltwiseUpThroughDataMovPerChannel : public ov::pass::MatcherPass {
+public:
+    OPENVINO_RTTI("MoveEltwiseUpThroughDataMovPerChannel", "0");
+    MoveEltwiseUpThroughDataMovPerChannel();
+};
+
+class TRANSFORMATIONS_API MoveEltwiseUpThroughDataMov : public ov::pass::GraphRewrite {
 public:
     OPENVINO_RTTI("MoveEltwiseUpThroughDataMov", "0");
-    MoveEltwiseUpThroughDataMov(std::vector<DiscreteTypeInfo> allowed_data_movement_ops = get_default_allowed_ops());
+    MoveEltwiseUpThroughDataMov(std::vector<DiscreteTypeInfo> allowed_data_movement_ops = get_default_allowed_ops()) {
+        this->add_matcher<MoveEltwiseUpThroughDataMovScalar>(allowed_data_movement_ops);
+        this->add_matcher<MoveEltwiseUpThroughDataMovPerChannel>();
+    }
 
 private:
     static std::vector<DiscreteTypeInfo> get_default_allowed_ops();

src/common/transformations/src/transformations/common_optimizations/move_eltwise_up_data_movement.cpp

@@ -4,13 +4,17 @@
 
 #include "transformations/common_optimizations/move_eltwise_up_data_movement.hpp"
 
+#include <algorithm>
 #include <memory>
-#include <numeric>
 #include <openvino/opsets/opset8.hpp>
 
 #include "itt.hpp"
 #include "openvino/core/rt_info.hpp"
 #include "openvino/core/type.hpp"
+#include "openvino/op/constant.hpp"
+#include "openvino/op/reshape.hpp"
+#include "openvino/op/squeeze.hpp"
+#include "openvino/op/unsqueeze.hpp"
 #include "openvino/pass/pattern/op/wrap_type.hpp"
 #include "transformations/utils/utils.hpp"
 
@@ -50,9 +54,9 @@ std::vector<ov::DiscreteTypeInfo> ov::pass::MoveEltwiseUpThroughDataMov::get_def
     };
 }
 
-ov::pass::MoveEltwiseUpThroughDataMov::MoveEltwiseUpThroughDataMov(
+ov::pass::MoveEltwiseUpThroughDataMovScalar::MoveEltwiseUpThroughDataMovScalar(
     std::vector<DiscreteTypeInfo> allowed_data_movement_ops) {
-    MATCHER_SCOPE(MoveEltwiseUpThroughDataMov);
+    MATCHER_SCOPE(MoveEltwiseUpThroughDataMovScalar);
     auto eltwise_pattern = ov::pass::pattern::wrap_type<ov::op::util::UnaryElementwiseArithmetic,
                                                         ov::op::util::BinaryElementwiseArithmetic,
                                                         ov::op::v0::FakeQuantize>(ov::pass::pattern::has_static_rank());
@@ -126,3 +130,102 @@ ov::pass::MoveEltwiseUpThroughDataMov::MoveEltwiseUpThroughDataMov(
     auto m = std::make_shared<ov::pass::pattern::Matcher>(eltwise_pattern, matcher_name);
     register_matcher(m, callback);
 }
+
+ov::pass::MoveEltwiseUpThroughDataMovPerChannel::MoveEltwiseUpThroughDataMovPerChannel() {
+    MATCHER_SCOPE(MoveEltwiseUpThroughDataMovPerChannel);
+
+    auto const_predicate = [](const ov::Output<ov::Node>& output) {
+        auto constant_op = std::dynamic_pointer_cast<ov::opset8::Constant>(output.get_node_shared_ptr());
+        if (!constant_op)
+            return false;
+
+        if (output.get_target_inputs().size() != 1)
+            return false;
+
+        const auto& shape = constant_op->get_shape();
+        return std::count_if(shape.begin(), shape.end(), [](size_t v) {
+                   return v > 1;
+               }) == 1;
+    };
+
+    auto eltw_predicate = [](const ov::Output<ov::Node>& output) {
+        if (output.get_target_inputs().size() != 1)
+            return false;
+
+        auto node = output.get_node();
+
+        if (node->get_output_partial_shape(0).rank().is_dynamic())
+            return false;
+
+        const size_t const_idx = ov::is_type<ov::op::v0::Constant>(node->get_input_node_ptr(0)) ? 0 : 1;
+        const size_t data_flow_idx = (const_idx + 1) % 2;
+
+        if (node->get_input_partial_shape(data_flow_idx).size() < node->get_input_partial_shape(const_idx).size())
+            return false;
+
+        return true;
+    };
+
+    auto eltw_data_flow_in =
+        ov::pass::pattern::wrap_type<ov::op::v1::Reshape, ov::op::v0::Squeeze, ov::op::v0::Unsqueeze>();
+    auto eltw_const_in = ov::pass::pattern::wrap_type<ov::op::v0::Constant>(const_predicate);
+    auto eltwise_pattern =
+        ov::pass::pattern::wrap_type<ov::op::util::BinaryElementwiseArithmetic>({eltw_data_flow_in, eltw_const_in},
+                                                                                eltw_predicate);
+
+    ov::matcher_pass_callback callback = [OV_CAPTURE_CPY_AND_THIS](ov::pass::pattern::Matcher& m) {
+        const auto& pattern_map = m.get_pattern_value_map();
+
+        auto eltwise = pattern_map.at(eltwise_pattern).get_node_shared_ptr();
+        if (transformation_callback(eltwise)) {
+            return false;
+        }
+
+        const size_t const_idx = ov::is_type<ov::op::v0::Constant>(eltwise->get_input_node_ptr(0)) ? 0 : 1;
+        const size_t data_flow_idx = (const_idx + 1) % 2;
+
+        auto const_shape = eltwise->get_input_shape(const_idx);
+        size_t channel_idx = 0;
+        size_t channel_val = 0;
+        for (size_t i = 0; i < const_shape.size(); i++) {
+            if (const_shape[i] > 1) {
+                channel_idx = i;
+                channel_val = const_shape[i];
+            }
+        }
+
+        auto parent = eltwise->get_input_node_shared_ptr(data_flow_idx);
+        const auto& parent_in_pshape = parent->get_input_partial_shape(0);
+        auto parent_in_channel_dim =
+            parent_in_pshape.size() <= channel_idx ? ov::Dimension(1) : parent_in_pshape[channel_idx];
+        auto parent_out_channel_dim = parent->get_output_partial_shape(0)[channel_idx];
+        if (parent_in_channel_dim.is_dynamic() || parent_in_channel_dim != channel_val ||
+            parent_out_channel_dim.is_dynamic() || parent_out_channel_dim != channel_val)
+            return false;
+
+        auto new_shape = ov::Shape(parent->get_input_partial_shape(0).size(), 1);
+
+        new_shape[channel_idx] = const_shape[channel_idx];
+        auto old_const = std::dynamic_pointer_cast<ov::op::v0::Constant>(eltwise->get_input_node_shared_ptr(const_idx));
+        auto new_const = std::make_shared<ov::op::v0::Constant>(*old_const, new_shape);
+        ov::replace_node_update_name(old_const, new_const);
+        ov::replace_output_update_name(eltwise->output(0), eltwise->input_value(data_flow_idx));
+
+        ov::OutputVector eltwise_inputs = eltwise->input_values();
+        eltwise_inputs[data_flow_idx] = parent->input_value(0);
+        auto new_eltwise = eltwise->clone_with_new_inputs(eltwise_inputs);
+        ov::copy_runtime_info(eltwise, new_eltwise);
+
+        ov::OutputVector parent_inputs = parent->input_values();
+        parent_inputs[0] = new_eltwise;
+        auto new_parent = parent->clone_with_new_inputs(parent_inputs);
+        ov::copy_runtime_info(parent, new_parent);
+        new_parent->set_friendly_name(parent->get_friendly_name());
+
+        ov::replace_node(parent, new_parent);
+        return true;
+    };
+
+    auto m = std::make_shared<ov::pass::pattern::Matcher>(eltwise_pattern, matcher_name);
+    register_matcher(m, callback);
+}