graph: backend: dnnl: support permute for scale and zps inputs #2291
+26
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wzt1997
force-pushed
the
zhitao/support-scale-permute
branch
from
953b5a6 to
ce87611
Compare
TaoLv
reviewed
Dec 19, 2024
tests/benchdnn/inputs/graph/complex_fusion/mha/sdpa-compressed-k-int8-gs32.json
Outdated
Show resolved
Hide resolved
gyhintel
reviewed
Dec 19, 2024
wzt1997
force-pushed
the
zhitao/support-scale-permute
branch
2 times, most recently
from
cb5d6c4 to
3a713a8
Compare
|
make test |
wzt1997
force-pushed
the
zhitao/support-scale-permute
branch
from
3a713a8 to
1b80646
Compare
|
make test |
rongzha1
reviewed
Dec 24, 2024
xiang1guo
reviewed
Dec 25, 2024
xiang1guo
approved these changes
Dec 25, 2024
wzt1997
force-pushed
the
zhitao/support-scale-permute
branch
from
1b80646 to
b1eed4d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We recently supported the compressed SDPA patterns which incorporate scales and zero points as inputs. Also, we aim to support the problem with a Key shape of ( N, H, S, D ) with
transpose_b=true for the QK MatMul. For such case with transposed MatMul, the graph library addspermuteops prior to the pattern input, including the scale and zp tensors. To be more specific, for the following woq MatMul with tranpose_b=true:After the graph compilation, we will have the graph as follows. The permute ops will not touch the physical memories but only change the mds.
However, we faced a issue that DNNL backend always regards scale and zp tensors as
abxformat, leading to failing results, hence we need to set the tag explicitly asabxand execute extra reorders. After the change, the graph after compilation is like:TODO