Skip to content

Fix GCM tag state after one-shot decrypt; RSA decrypt2 exact length - #88

Closed
MarkAtwood wants to merge 2 commits into
wolfSSL:mainfrom
MarkAtwood:fix-gcm-tag-rsa-decrypt2
Closed

MarkAtwood wants to merge 2 commits into
wolfSSL:mainfrom
MarkAtwood:fix-gcm-tag-rsa-decrypt2

Conversation

@MarkAtwood

Copy link
Copy Markdown

Two fixes in the wrapper, each with a test that fails on main and passes here.

AES-GCM tag callback (cipher.c). wolfssl_cipher_aead_decrypt() set ctx->enc = 0, and wolfssl_cipher_tag() then treated its buffer as an input tag on that handle. GnuTLS calls the tag callback to get the tag it compares with the received one. So after a one-shot gnutls_aead_cipher_decrypt() on a handle, gnutls_aead_cipher_decryptv2() with empty ciphertext on the same handle accepted any tag over any AAD (an all-zero tag over attacker AAD verified) and rejected the correct one. encryptv/encryptv2 on empty plaintext returned the caller's buffer as the tag. This needs that specific API sequence on one handle; TLS does not use it. The tag callback is now output only: the tag of the data processed, or with no data the tag over the AAD. Returning the tag ends the message (tag_set reset). If the tag cannot be computed it writes random bytes (gnutls_rnd, then getrandom(), else abort()), which no received tag matches.

RSA decrypt2 (pk.c). gnutls_privkey_decrypt_data2() gives an exact plaintext size, but a valid shorter message returned success with the buffer partly written. TLS RSA key exchange (lib/auth/rsa.c, rsa_psk.c) prefills that buffer with a random premaster and ignores the result, so on failure the buffer must stay unchanged. Otherwise the handshake outcome tells the client whether the padding was valid. decrypt2 now decrypts into a scratch buffer and copies it into the caller's buffer with a constant-time masked select only when the length is exactly the requested size. Any other result, length or padding, returns GNUTLS_E_DECRYPTION_FAILED with the caller's buffer unchanged, as nettle's rsa_sec_decrypt does. The scratch buffer is cleared before return.

Tests:

  • tests/test_aesgcm_tag_state.c: one-shot decrypt, then decryptv2 with the correct tag (accepted) and a zero tag (rejected); encryptv2 tag equals the encrypt tag.
  • tests/test_rsa_decrypt2_length.c: 48-byte buffer prefilled with a pattern; the 48-byte message decrypts; 16, 47 and 64-byte messages and a corrupted or truncated ciphertext fail and leave the buffer unchanged.

Checked on wolfSSL master (non-FIPS): make test 27/27 with the provider and with GNUTLS_NO_PROVIDER=1; both new tests valgrind clean; both new tests fail against main.

GnuTLS calls the provider's tag callback to get the tag it then compares
with the received one. wolfssl_cipher_aead_decrypt() set ctx->enc to 0,
and on such a handle the callback copied the tag FROM the caller's buffer
instead of writing it. After a one-shot gnutls_aead_cipher_decrypt(), a
gnutls_aead_cipher_decryptv2() with no ciphertext therefore accepted any
tag over any AAD (an all-zero tag over attacker AAD verified) and
rejected the correct one, and encryptv/encryptv2 on empty plaintext
returned the caller's buffer as the tag.

The callback now always writes the tag: the tag of the data processed or,
with no data, the tag over the AAD alone. The tag ends the message, so a
handle reused without a new IV does not return it again. If it cannot
compute one it writes random bytes (gnutls_rnd, then getrandom), which no
received tag will match, and aborts if neither gives random bytes.
aead_decrypt no longer changes ctx->enc, and the "tag set externally"
state is removed.

New test test_aesgcm_tag_state, values from pyca/cryptography AESGCM.
gnutls_privkey_decrypt_data2() gives the exact plaintext size. A valid
PKCS#1 v1.5 or OAEP message shorter than that decrypted "successfully"
with the caller's buffer only partly written, and a failed decryption
could leave decrypted bytes in it.

TLS RSA key exchange prefills the premaster buffer with random bytes and
ignores decrypt2's result, so on any failure the buffer must be left
unchanged, or the handshake becomes a padding oracle. decrypt2 now always
decrypts into a zeroed scratch buffer and copies to the caller's buffer
with a constant-time masked select that writes the plaintext only when
its length is exactly the requested size; otherwise it returns
GNUTLS_E_DECRYPTION_FAILED, as GnuTLS's own implementation does. The
scratch buffer is wiped before return.

New test test_rsa_decrypt2_length: 48 bytes into a 48-byte buffer
decrypts; 16-, 47- and 64-byte messages and corrupted or truncated
ciphertexts fail and leave the prefilled buffer unchanged. The test
passes without the provider (GnuTLS's nettle path) as the oracle.
Copilot AI balanced review requested due to automatic review settings October 2, 2026 03:35

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The unconditional Linux-only random API breaks the supported macOS build, and several contract tests need strengthening.

Review effort: Balanced
Findings: 1 High severity · 2 Medium severity · 1 Low severity

Open (4)
What changed in this PR

Fixes AES-GCM tag state reuse and enforces exact-length RSA decrypt_data2 output.

Changes:

  • Makes GCM tag callbacks output-only and resets message state.
  • Uses constant-time RSA plaintext selection with scratch-buffer clearing.
  • Adds regression tests for both fixes.
File Description
src/​cipher.c Corrects GCM tag generation and state handling.
src/​pk.c Enforces exact RSA plaintext length.
tests/​test_aesgcm_tag_state.c Tests reused-handle GCM behavior.
tests/​test_rsa_decrypt2_length.c Tests RSA length and failure handling.
tests/​Makefile Registers the new tests.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

#include "gnutls_compat.h"
#include "logging.h"
#include <stdlib.h>
#include <sys/random.h>
Comment on lines +1140 to +1142
if (!alloc_plaintext) {
ret = rsa_decrypt2_select(plaintext, plain, ret);
gnutls_memset(plain, 0, scratch_size);
Comment on lines +61 to +64
if (ret >= 0) {
printf("FAILURE - reported as decrypted\n");
return 1;
}
Comment on lines +1133 to +1135
/* The tag is output only: gnutls compares it with the received tag after
* a decrypt, so it must always be written. When it cannot be computed it
* is filled with random bytes, which no received tag will match. */
@MarkAtwood

Copy link
Copy Markdown
Author

Closing in favour of #91, per Reda.

@MarkAtwood MarkAtwood closed this Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants