ECDHE_ECDSA ClientKeyExchange rejected as too short when DTLS_PSK_MAX_CLIENT_IDENTITY_LEN > 64 #298

Summary

When DTLS_PSK_MAX_CLIENT_IDENTITY_LEN is configured above 2 * DTLS_EC_KEY_SIZE (64), a TinyDTLS server rejects every legitimate ECDHE_ECDSA ClientKeyExchange with handshake_failure (40), because the minimum-length check uses a constant that is sized for the PSK identity.

Where

dtls.c:

#define DTLS_CKXEC_LENGTH (1 + 1 + max(DTLS_EC_KEY_SIZE + DTLS_EC_KEY_SIZE, DTLS_PSK_MAX_CLIENT_IDENTITY_LEN))

and in dtls_check_client_keyexchange() (ECDHE_ECDSA branch):

if (length < DTLS_HS_LENGTH + DTLS_CKXEC_LENGTH) {
  dtls_debug("The client key exchange is too short\n");
  return dtls_alert_fatal_create(DTLS_ALERT_HANDSHAKE_FAILURE);
}

Checked against 501d93c (current main). develop has the same form.

Why it fails

An ECDHE_ECDSA ClientKeyExchange carries a 1-octet length, a 1-octet point format (0x04) and an uncompressed P-256 point of 2 * DTLS_EC_KEY_SIZE = 64 octets: 66 octets in total. With the default DTLS_PSK_MAX_CLIENT_IDENTITY_LEN of 32 the macro evaluates to 1 + 1 + max(64, 32) = 66, which happens to be exact. With -DDTLS_PSK_MAX_CLIENT_IDENTITY_LEN=128 (RFC 4279 section 5.1 allows identities up to 128 octets) it evaluates to 130, so the 66-octet message is "too short" and the handshake fails. The PSK identity length has no bearing on the size of an ECDHE message.

Reproduction

Build with -DDTLS_PSK_MAX_CLIENT_IDENTITY_LEN=128 and both DTLS_PSK and DTLS_ECC enabled. Run a TinyDTLS server offering TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8 (raw public keys or certificates) and connect with any client that completes the ECDHE_ECDSA flight. The server logs error in check_client_keyexchange err: -552 and sends a fatal handshake_failure alert. With the default of 32 the same pair handshakes normally.

Suggested fix

Check the ECDHE message against its true size, independent of the PSK identity length:

if (length < DTLS_HS_LENGTH + 1 + 1 + 2 * DTLS_EC_KEY_SIZE) {

DTLS_CKXEC_LENGTH is still correct as the size of the send buffer in dtls_send_client_key_exchange(), since that buffer holds either the PSK message (2 + identity length) or the ECDHE message.

I have verified the fix locally: with only that comparison changed, the ECDHE_ECDSA handshake succeeds at an identity limit of 128. Happy to send a PR with a test if that is useful.