graph: fix the intermediate data types in SDPA patterns #2894
Conversation
github-actions
bot
added
documentation
TaoLv
force-pushed
the
lvtao/main/fix-sdpa-intermediates
branch
2 times, most recently
from
257f3f9 to
4a506ff
Compare
TaoLv
force-pushed
the
lvtao/main/fix-sdpa-intermediates
branch
from
4a506ff to
2cdb502
Compare
TaoLv
force-pushed
the
lvtao/main/fix-sdpa-intermediates
branch
from
2cdb502 to
5b05972
Compare
TaoLv
force-pushed
the
lvtao/main/fix-sdpa-intermediates
branch
from
5b05972 to
76f2cf9
Compare
| @@ -52,7 +52,6 @@ Graph operations support bf16 and f16 data types. | |||
|
|
|||
| A TypeCast operation performing down conversion should be inserted clearly to | |||
| indicate the use of low numeric precision. oneDNN Graph implementation fully | |||
| honors the API-specified numeric precision and only performs the computation | |||
| using the API-specified or higher numeric precision. | |||
| honors the API-specified numeric precision. | |||
There was a problem hiding this comment.
Just to make sure we are aligned. This still allows to use f32 values to store f16/bf16 data, as long as we respect roundings to f16/bf16 accuracy, right?
There was a problem hiding this comment.
Yes, in my understanding, it's still allowed for backend implementations. From this perspective, it seems I need to keep the original statement. My intention here was to align the implementations. As the original statement sounds like different backends (eg. DNNL & GC, CPU & GPU) can have different numerical behaviors.
| .set_type_constraints( | ||
| "T2", {data_type::f32, data_type::bf16, data_type::f16}) | ||
| .set_type_constraints( | ||
| "T3", {data_type::f32, data_type::bf16, data_type::f16}) |
There was a problem hiding this comment.
This requires some documentation about type promotion as users might wonder what happens for example with f16 <- f16 + bf16.
There was a problem hiding this comment.
In fact, we don't allow f16 + bf16. It's mentioned in the "supported data types" section in the op document. When src0 and src1 have different data types, one of them should be f32 and the other one (f16 or bf16) will be promoted to f32 for calculation.
Address MFDNN-13091
The purpose is to align the intermediate data types in the pattern with those in ATen SDPA and GPU ukernel SDPA.
With these, now a typical f16 SDPA which can be dispatched to GPU ukernel SDPA looks like below: