CVE-2023-6237

Issue summary: Checking excessively long invalid RSA public keys may takea long time.

Impact summary: Applications that use the function EVP_PKEY_public_check()to check RSA public keys may experience long delays. Where the key thatis being checked has been obtained from an untrusted source this may leadto a Denial of Service.

When function EVP_PKEY_public_check() is called on RSA public keys,a computation is done to confirm that the RSA modulus, n, is composite.For valid RSA keys, n is a product of two or more large primes and thiscomputation completes quickly. However, if n is an overly large prime,then this computation would take a long time.

An application that calls EVP_PKEY_public_check() and supplies an RSA keyobtained from an untrusted source could be vulnerable to a Denial of Serviceattack.

The function EVP_PKEY_public_check() is not called from other OpenSSLfunctions however it is called from the OpenSSL pkey command lineapplication. For that reason that application is also vulnerable if usedwith the '-pubin' and '-check' options on untrusted data.

The OpenSSL SSL/TLS implementation is not affected by this issue.

The OpenSSL 3.0 and 3.1 FIPS providers are affected by this issue.