ML-DSA fails the FIPS 204 ACVP signature verification test vectors

[strongX509]

We are currently adding ML-DSA support to the existing strongSwan botan plugin (see our ml-dsa branch on github). Unfortunately the FIPS 204 ACVP signature verification test cases (tcId: 1, 20, 31) fail. Thus we are wondering whether the botan-3.6.1 library correctly implements the final FIPS 204 standard.

The same ACVP test vectors work fine with the strongSwan wolfssl plugin using the wolfssl-5.7.4-stable library, so that we are quite sure that the generic strongSwan ML-DSA wrappers are correct. We are using the ffi.h C-language interface to integrate the botan code into our strongSwan code.

We are not sure what constant string to use for the hash_and_padding argument of the botan_pk_op_verify_create() function for the ML-DSA case without pre-hashing. We have a similiar question for the botan_pk_op_sign_create() function where we want to use Randomized signatures. For the existing Ed25519 case we used Pure both for signature generation and verification, so for the time being we use the same string for the ML-DSA case.

int botan_pk_op_verify_create(botan_pk_op_verify_t* op,
                              botan_pubkey_t key,
                              const char* hash_and_padding,
                              uint32_t flags);

int botan_pk_op_sign_create(botan_pk_op_sign_t* op,
                             botan_privkey_t key, 
                            const char* hash_and_padding,
                             uint32_t flags);

On the positive side the FIPS 204 ACVP key generation test cases (tcID: 1, 26, 51) are correct and our own roundtrip tests (private/public key generation / signature generation / signature verfication) using the botan-3.6.1 library work.

[strongX509]

The hash_and_padding settings have been resolved. Randomized and Pure work fine for signature generation and signature verification, respectively.

The failure of the FIPS 204 ACVP signature verification test cases is still persistent.