AES_set_encrypt_key zeroed the whole schedule and set rounds before it
checked the key length, so a refused call changed the AES_KEY. As in
LibreSSL 2.5.4's aes_core.c, it now returns -1 or -2 before writing
anything and writes only the 44, 52 or 60 words the key length needs.
The test fills the key with a sentinel and checks, for both setup
functions, that refused calls leave all 244 bytes unchanged, that
AES-128 and AES-192 keep the 16 and 8 unused words, and that every
written word matches the FIPS-197 appendix A schedules and the
decryption schedules computed independently and with OpenSSL 3.0.13.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
AES_set_encrypt_key, AES_set_decrypt_key, AES_encrypt, AES_decrypt,
AES_ecb_encrypt, AES_cbc_encrypt, AES_cfb128_encrypt and
AES_ctr128_encrypt threw. They now follow LibreSSL's aes.h and its CBC,
CFB-128 and CTR mode code on the caller's 244-byte AES_KEY: key setup
returns -1 for null arguments and -2 for a key length other than 128,
192 or 256 bits, CBC pads a partial final block with the IV and writes
the whole block, CFB and CTR resume from *num, and the CTR counter
carries across all 128 bits.
The test checks the FIPS-197 appendix A and C vectors and the SP 800-38A
CBC, CFB-128 and CTR vectors, split across calls, in place and with
partial blocks.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
boykopovar