Whisper: Fix decoder inputs for static pipeline

[eshiryae]

Tickets:

• CVS-159064

[ilya-lavrenov]

why do we need to store model? once model is compiled, we need to release ov::Model to free memory consumed by its weights

[TolyaTalamanov]

We can't compile this model until generate() called

[eshiryae]

It's a temporal solution as wee need to reshape and recompile decoder model for specific number of input tokens, but we know it only on generation stage.

[TolyaTalamanov]

Well, I'd assume we have something like this:

class DecoderCache {
public:
    ov::CompiledModel get_model(uint8_t input_id_size) {
        // Get from hash table, otherwise compile and store...
    }
private:
    // [input_ids_size -> CompiledModel]
    std::unordered_map<uint8_t, ov::CompiledModel> m_cache;
    std::shared_ptr<ov::Model> decoder_model; // <- this is dynamic w/o transformation applied    
}


// Whenever we need a model:
auto decoder = m_decoder_cache.get(input_ids_size);

[TolyaTalamanov]

What does this check do?

[eshiryae]

That we have correct shape for input_ids for decoder model.
In prepare_init_ids() infer request for decoder can be called for language detection (it runs with 1 token as input).

[TolyaTalamanov]

I assume it can't be incorrect, as we explicitly reshape model for input_ids size.

Besides, I'd rather check: get_tensor("input_ids").get_size()

[TolyaTalamanov]

Btw, do we still need this?

openvino.genai/src/cpp/src/whisper_pipeline_static.cpp

Lines 553 to 556 in 7e8bbfe

	// TODO: Support models produced by optimum-cli
	if (!check_decoder_model_compatibility(decoder_model)) {
	OPENVINO_THROW("StaticWhisperPipeline expects decoder model has \"attention_mask\" input!");
	}

[TolyaTalamanov]

1. What if language detection isn't needed? But we already compiled model for [1,1] shape here:

   openvino.genai/src/cpp/src/whisper_pipeline_static.cpp

   Lines 553 to 556 in 7e8bbfe

   	// TODO: Support models produced by optimum-cli
   	if (!check_decoder_model_compatibility(decoder_model)) {
   	OPENVINO_THROW("StaticWhisperPipeline expects decoder model has \"attention_mask\" input!");
   	}

2. What if user runs the same model, but on multiple audio? So, we will compile it every time here:

   openvino.genai/src/cpp/src/whisper_pipeline_static.cpp

   Line 663 in 7e8bbfe

   m_models.decoder = core.compile_model(m_decoder_model, "NPU").create_infer_request();

3. What if every of such runs from the 2) example, requires language detection?

In order to encapsulate this logic, you can introduce something like DecoderCache as I proposed previously.

At any point, when you need CompiledModel for particular shape, you can ask DecoderCache and it will check if such model already exists (compiled) or reshape and compile a new one.

[eshiryae]

1. check_decoder_model_compatibility() checks only if decoder_model has attention_mask as an input, shape will not affects this check. Btw i suppose we don't need this check and we can remove it.

Agree with other points, it's a great idea to store already compiled models in cache, will do that.

[TolyaTalamanov]

Here the model compiled upfront again. What if it's not even needed and language already known?

I believe you need it somewhere here:

openvino.genai/src/cpp/src/whisper_pipeline_static.cpp

Line 269 in 614e6d9

language_token_id = detect_language(encoder_hidden_state, decoder, config, raw_metrics);

[TolyaTalamanov]

Weird, I don't expect we need this check...

Let's discuss locally

[TolyaTalamanov]

Well, decoder model has two input layers:

1. encoder_hidden_states - does it change? I believe it depends on the encoder model and once we know it it's no longer change?
2. decoder_input_ids - This is what we track in hash map. Changes depends on the case

Can we set encoder_hidden_states in the ctor? If so, I believe we don't need to change if model dynamic or not in the get() method

[TolyaTalamanov]

I believe the size of encoder_hiddne_states depends only on feature_size and it's set in StaticWhisperPipeline and no longer changes since then:

openvino.genai/src/cpp/src/whisper_pipeline_static.cpp

Line 535 in 614e6d9

reshape_to_static_encoder(encoder_model, m_feature_extractor.feature_size);

. Can you confirm this?

If so, I believe we definitely can set shape for encoder_hidden_states once.

// Also, it should probably be `ov::Shape` rather than ov::PartialShape as we don't have dynamism (input is fixed)
DecoderCache(std::shared_ptr<ov::Model> model, ov::PartialShape encoder_hidden_shape) {
    model->reshape({{"encoder_hidden_states", encoder_hidden_shape}});
}
DecoderCache::get(ov::CompiledModel DecoderCache::get_model(uint8_t input_ids_size) {
    if (m_cache.counts(input_ids_size) == 0) {
        m_model->reshape({{"decoder_input_ids", ov::Shape({1, input_ids_size})}});
        m_cache.emplace(input_ids_size, core.compile_model(m_model));
    }
    return m_cache.at(input_ids_size);
}