OpenShift IPI Install: Failed to get power state for node. Error: Redfish exception occurred.

What if you are trying to install OpenShift using IPI on bare metal (HPE) and you find that some of your bare metal hosts are failing to register or provision. If you run oc -n openshift-machine-api get bmh you may see the error: Failed to get power state for node 24cabdca-ac30-4990-a6b4-a24b7a6b1f74. Error: Redfish exception occurred. Error: Resource https://10.0.1.11/redfish/v1/registrystore/registries/en/smartstoragemessages.v2_0_1.smartstoragemessages not found

This issue is actually caused by the state of the HPE iLO 5 RESTful API being corrupted. This can happen if you perform a firmware update and it fails or if you make configuration changes that don’t properly execute. This can be something like a BIOS change that needs a reboot to take affect but during the reboot the server gets powered off. HPE doesn’t specify specific scenarios as to why this occurs but just that it sometimes does.[1][2]

OCP/Ironic tries to do a HTTP GET on https://10.0.1.11/redfish/v1/registrystore/registries/en/smartstoragemessages.v2_0_1.smartstoragemessages and gets a 404 Not Found. When the RESTful API state is not corrupted this returns a 200 OK with JSON output. You can verify this by doing a curl or using Postman.

In order to resolve this you need to clear the RESTful API state. If you are using the python-redfish-utility you can run the clearrestapistate command and then reboot ColdBoot. If you want to do this yourself you can issue the following HTTP POST. Make sure to add in your own Basic Auth credentials and change the IP address.

Clear HPE RESTful API State:curl -k --location --request POST 'https://10.0.1.11/redfish/v1/Managers/1/Actions/Oem/Hpe/HpeiLO.ClearRestApiState/' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ' \
--data-raw '{
   "Action" : "HpeiLO.ClearRestApiState"
}'

Perform a ColdBoot on an HPE Gen10 Server (server must be powered on):curl -k --location --request POST 'https://10.0.1.11/redfish/v1/Systems/1/Actions/Oem/Hpe/HpeComputerSystemExt.SystemReset' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ' \
--data-raw '{
  "ResetType" : "ColdBoot"
}'

Verify the URL no longer returns a 404 Not Found error.:curl -k 'https://10.0.1.11/redfish/v1/registrystore/registries/en/smartstoragemessages.v2_0_1.smartstoragemessages' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ' \
--compressed

[1] https://hewlettpackard.github.io/python-redfish-utility/#clearrestapistate-command
[2] https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00028474en_us