Skip to content

Fault manager cannot report that its storage is unavailable #694

Description

@bburda

Summary

When the fault manager cannot reach its storage, a client cannot tell that apart from "no faults".

ListFaults.srv has no error field, so the handler answers with an empty list. The services that do have an error field answer success=false, and the gateway turns any answered success=false into 404 Fault not found (ros2_medkit_gateway/src/http/handlers/fault_handlers.cpp:224), so a storage outage reads as a missing fault.

There is also no way to ask the fault manager how it is doing. With a PostgreSQL backend the node keeps running while the server is unreachable, so a node that is up says nothing about whether its faults are readable.

Proposed solution (optional)

Two parts.

  1. ListFaults.srv gets a result field, the same shape the other fault services use (success plus error_message). The gateway maps a storage outage to 503 Service unavailable, which the handler already supports (fault_handlers.cpp:228), and keeps 404 for a fault that really is not there.

  2. A status service on the fault manager: which storage backend is configured, whether it is connected, the reason of the last failed connection, and which startup work was skipped (the near-miss trim and the HEALED reclassification are skipped when the storage is unreachable at startup). The gateway reports that status in /health, so an operator sees a degraded fault manager without asking the node directly.

Additional context (optional)

The PostgreSQL backend (PR 651 and the draft PR 693) is what makes this visible: the node now starts and keeps serving while the server is unreachable, and answers Fault storage unavailable on every service that can carry an error. ListFaults and /health are the two places where that state is still invisible.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions