[4.21] OCPBUGS-115166: idrac10 retry conflict power-on - #501
Conversation
Below is an old output of the slowest tests. The redfish power tests are made faster by having a sleep mocking strategy for the BackOffLoopingCall, and the snmp tests are sped up by reducing the config timeout. The slowest tests now take 1 to 4 seconds which are likely all sleep related. ironic.tests.unit.drivers.modules.redfish.test_power.RedfishPowerTestCase.test_reboot_not_reached 43.002 ironic.tests.unit.drivers.modules.redfish.test_power.RedfishPowerTestCase.test_set_power_state 22.958 ironic.tests.unit.drivers.modules.redfish.test_power.RedfishPowerTestCase.test_set_power_state_not_reached 12.134 ironic.tests.unit.drivers.modules.test_snmp.SNMPDeviceDriverTestCase.test_power_reset_on_invalid_state 9.045 ironic.tests.unit.drivers.modules.test_snmp.SNMPDeviceDriverTestCase.test_power_off_invalid_state 9.044 ironic.tests.unit.drivers.modules.test_snmp.SNMPDeviceDriverTestCase.test_power_reset_off_invalid_state 9.043 ironic.tests.unit.drivers.modules.test_snmp.SNMPDeviceDriverTestCase.test_power_on_invalid_state 9.043 ironic.tests.unit.drivers.modules.test_snmp.SNMPDeviceDriverTestCase.test_power_reset_off_timeout 9.042 ironic.tests.unit.drivers.modules.test_snmp.SNMPDeviceDriverTestCase.test_power_on_timeout 9.042 ironic.tests.unit.drivers.modules.test_snmp.SNMPDeviceDriverTestCase.test_power_reset_on_timeout 9.042 Change-Id: I19bdfc6bced9b98f14a26d2b71024e84f410368a Signed-off-by: Steve Baker <sbaker@redhat.com> (cherry picked from commit b13db57)
Some BMCs (observed on Dell iDRAC10 / PowerEdge 17G) transiently reject the Redfish "On" reset type with HTTP 409 ActionParameterValueConflict for a short window (up to ~60 seconds) right after a power-off, until the BMC settles. During a deploy, boot_instance issues the power-on only a few seconds after tear_down_agent powers the node off, so the reset lands inside that window and the deploy fails with: Deploy step deploy.boot_instance failed: ... ComputerSystem.Reset returned code 409 ... ActionParameterValueConflict ... the parameter 'ResetType' with the requested value of 'On' does not meet the constraints of the implementation. The value itself is correct -- the same "On" succeeds once the BMC has settled, confirmed by issuing it manually -- and such BMCs may not implement "ForceOn", so switching the reset type is not an option. The existing 400/409 handler in RedfishPower.set_power_state only treats a conflict as success when the node is already in the target state; a transient "not settled yet" conflict is re-raised and fails the deploy. Retry the power-on in _set_power_state when the BMC returns HTTP 409 ActionParameterValueConflict and the node is not yet powered on, until the BMC accepts it or the attempts are exhausted. Two new [redfish] options control the behaviour: power_on_conflict_retry_attempts (default 12) and power_on_conflict_retry_interval (default 5 seconds); set attempts to 0 to disable. Closes-Bug: #2162995 Change-Id: Id8d7535409a560b66a5d771be1751acd1248f61a Signed-off-by: Jay Jahns <jayjahns@gmail.com> (cherry picked from commit 238c9e0)
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: iurygregory The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
@iurygregory: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
iurygregory
changed the title
openshift-ci-robot
added
jira/severity-important
|
@iurygregory: This pull request references Jira Issue OCPBUGS-115166, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
4 participants
Cherry-picked b13db57 to have a clean cherry-pick of 238c9e0.
Shouldn't cause problems since b13db57 speeds-up unit tests.