Skip to content

fix: recover rshell server after command timeout and bound memory - #45

Open
k-ziomek wants to merge 2 commits into
intel:mainfrom
k-ziomek:task_MFD_8903
Open

k-ziomek wants to merge 2 commits into
intel:mainfrom
k-ziomek:task_MFD_8903

Conversation

@k-ziomek

Copy link
Copy Markdown

No description provided.

Copilot AI lite review requested due to automatic review settings July 30, 2026 11:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Ready to approve

The changes are focused, are covered by updated unit tests, and align with the goal of preventing unbounded output retention.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR addresses potential unbounded memory growth in the rshell_server by ensuring stored command outputs don’t accumulate indefinitely and are removed after retrieval or after a TTL expires.

Changes:

  • Added timestamp tracking for stored command outputs and a TTL-based cleanup routine to evict stale entries.
  • Updated get_output() to remove outputs (and timestamps) once consumed to prevent indefinite retention.
  • Extended unit tests to cover timestamp bookkeeping, cleanup behavior, and queue removal on client disconnect.
File summaries
File Description
mfd_connect/rshell_server.py Adds output timestamp tracking + stale-output cleanup, and removes outputs after retrieval/disconnect.
tests/unit/test_mfd_connect/test_rshell_server.py Updates tests to validate output eviction, timestamp handling, and stale cleanup behavior.
Review details

Comments suppressed due to low confidence (1)

mfd_connect/rshell_server.py:65

  • For consistency with monotonic TTL comparisons, store timestamps using time.monotonic() (not time.time()).
def _store_output(command_id: str, output: str, rc: int) -> None:
    """Persist command output together with its insertion timestamp."""
    _cleanup_stale_outputs()
    output_queue[command_id] = output_object(output=output, rc=rc)
    output_queue_timestamps[command_id] = time.time()

  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Comment thread mfd_connect/rshell_server.py Resolved
@coveralls-official

coveralls-official Bot commented Jul 31, 2026 •

Copy link
Copy Markdown

Coverage Status

coverage: 80.839% (+0.2%) from 80.639% — k-ziomek:task_MFD_8903 into intel:main

Copilot AI review requested due to automatic review settings July 31, 2026 10:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The TTL cleanup uses a monotonic clock while stored timestamps use wall-clock time, which prevents stale entries from ever expiring in production.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Copilot AI review requested due to automatic review settings August 7, 2026 10:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (1)

mfd_connect/rshell_server/rshell_server.py:64

  • _cleanup_stale_outputs() measures current_time with time.monotonic(), but _store_output() records created_at with time.time(). Mixing monotonic time with wall-clock time means current_time - created_at will typically be a large negative number, so stale entries will never be collected in production (defeating the memory-growth fix). Use the same clock source for both (prefer time.monotonic() for TTL calculations).
def _store_output(command_id: str, output: str, rc: int) -> None:
    """Persist command output together with its insertion timestamp."""
    _cleanup_stale_outputs()
    output_queue[command_id] = output_object(output=output, rc=rc)
    output_queue_timestamps[command_id] = time.time()

Copilot AI review requested due to automatic review settings August 7, 2026 10:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@adrianlasota adrianlasota left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix the commits, there is merge commit

@k-ziomek k-ziomek changed the title fix: prevent rshell server unbounded server memory growth fix: recover rshell server after command timeout and bound memory Aug 19, 2026
@k-ziomek
k-ziomek force-pushed the task_MFD_8903 branch 2 times, most recently from 7ecfc0d to 8d482a3 Compare August 24, 2026 10:10
@k-ziomek
k-ziomek force-pushed the task_MFD_8903 branch 2 times, most recently from 88a99a5 to 52ee998 Compare September 1, 2026 12:04
@abaczek

abaczek commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Added a follow-up commit to this PR:

  • Track active commands and reject unsolicited, duplicate and late results, including after abandoned-ID eviction.
  • Release result payloads, queue entries and per-client state after completion or disconnect; wake disconnected callers.
  • Bound results to 16 MiB each / 64 MiB total, with 512 active commands and 256 clients. Reject overflow explicitly instead of silently evicting another caller's work.
  • Report transport errors separately from command output and retry client polling after HTTP errors.

Validation: 102 RShell tests and mfd-code-standard passed for this change.

The independent unread-output-pipe fix is in #53. Its branch includes both changes for combined testing; its staging base mirrors this PR. No merge is required to test the combined branch.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved command-lifecycle and diagnostic-header issues can block admission and break timeout reporting.

3 open findings

🧠 Review effort: Balanced

time.sleep = _sleep


def _close(connection): # noqa: ANN001, ANN202

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, add input params to docstring as for it's for other, already existing methods

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 76c5d98: added :param connection: to _close() using the existing docstring convention, including the supported None value.

@a-gorczew

Copy link
Copy Markdown
Contributor

@abaczek Please capitalize the commit title.

@abaczek

abaczek commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Addressed the commit-title comment: the updated commit is Fix: bound rshell result storage and clean up completed commands (76c5d98).

The four open review threads are addressed in the same commit, with focused regression tests. Validation: 119 RShell tests and mfd-code-standard passed. No changes were pushed to #53 or other PRs.

k-ziomek and others added 2 commits October 16, 2026 08:17
…ient

The EFI client executes commands with a blocking os.system() call, so a tool
that hangs on the DUT stops its polling loop for good and every later command
times out. Track when each client last polled and which command it picked up,
and return that diagnosis in the X-RShell-Timeout-Reason header.

Signed-off-by: Ziomek, Kamil <kamil.ziomek@intel.com>
Signed-off-by: Arkadiusz Baczek <86051266+abaczek@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants