To successfully upgrade to HDP 2.4.0, you must change the value of hadoop.rpc.protection to authentication in core-site.xml:

<property>
 <name>hadoop.rpc.protection</name>
 <value>authentication</value>
</property>