Due to a bug in handling return code from openssl native calls, the SslHandler that uses OpenSslEngine can end up in an infinite loop and eat up all CPU cycles, which may lead to DoS of the system.

This can only happen if the server has renegotiation enabled (which is set as default). Only systems using SslProvider.OpenSsl are affected, that will be true if netty-tcnative is in the classpath and openssl is installed.

If netty-tcnative is linked against boringssl, the problem does not exist, since renegotiation is not supported in boringssl.

Versions affected: Netty 4.0.0.Final - 4.0.36.Final and 4.1.0.Final

Workaround:

Users can use -Djdk.tls.rejectClientInitiatedRenegotiation=true to disable renegotiation and avoid this issue.