Skip to content

Add timeout, retry and clear errors to get_maap_pgt_token - #21

Merged
sujen1412 merged 2 commits into
developfrom
fix/pgt-token-retry
Oct 16, 2026
Merged

sujen1412 merged 2 commits into
developfrom
fix/pgt-token-retry

Conversation

@sujen1412

@sujen1412 sujen1412 commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #8

Changes

  • get_maap_pgt_token.py: requests go through a session with HTTPAdapter(Retry(...)): (5, 30) s timeout, up to 5 retries (6 attempts), exponential backoff with jitter (backoff_factor=1, backoff_jitter=1, backoff_max=30), retrying connect errors, timeouts and 5xx. raise_on_status=False so the final response is turned into a clear PgtTokenError. 4xx (not retried), a missing session_key and non-JSON bodies also raise PgtTokenError.
  • On failure the message goes to stderr (stdout is captured into MAAP_PGT, so the old stdout print never reached _stderr.txt), _alt_error.txt is written so HySDS uses it as the job error, and the script exits with code 78.
  • dps_wrapper.sh: || exit $? so the failure code propagates out of the command substitution.
  • MAAP_API_HOST is now read at call time rather than import time.
  • urllib3>=2 added to the Dockerfile pip install and the CI test job: Retry(backoff_jitter=...) needs urllib3 2.x. Resolving the image's unpinned deps for Python 3.12 gave urllib3 2.8.0 already; the pin guards against a future downgrade. The local Python 3.9 .venv has urllib3 1.26 and cannot run these tests.

Notes

  • Exit code 78 was chosen per hysds-agent's reading of job_worker.py: any non-zero code yields "Got non-zero exit code: N", and _alt_error.txt in the job dir overrides error. Codes 126/127/130/137/143 and -15 are avoided. That guidance is from reading the code, not from a run on HySDS.
  • Assumes the wrapper's cwd is the job dir when _alt_error.txt is written; not verified in the container.

Tests

tests/test_get_maap_pgt_token.py: retry behaviour is tested against a local flaky HTTP server (5xx then success, exhaustion after 6 attempts, 4xx not retried, connection errors), plus error messages, _alt_error.txt/stderr, and exit code 78 with clean stdout. Full suite: 73 passed on Python 3.12 / urllib3 2.8.0.

- 5s connect / 30s read timeout; up to 6 attempts with exponential
  backoff and jitter on connect errors, timeouts and 5xx
- 4xx and malformed responses fail immediately with PgtTokenError
- on failure, message goes to stderr (stdout is captured into MAAP_PGT),
  _alt_error.txt is written for the HySDS job `error` field, and the
  script exits 78; dps_wrapper.sh propagates the exit code
- resolve MAAP_API_HOST at call time instead of import time

Closes #8

Claude-Session: https://claude.ai/code/session_0171HvL7yCdP2wCuxh7UngYP
Replace the hand-rolled retry loop with a requests session mounting an
HTTPAdapter(Retry(...)): exponential backoff with jitter (backoff_jitter
needs urllib3>=2), retries on connect errors, timeouts and 5xx. Pin
urllib3>=2 in the Dockerfile and CI. Retry behaviour is tested against a
local flaky HTTP server.

Claude-Session: https://claude.ai/code/session_0171HvL7yCdP2wCuxh7UngYP
@sujen1412
sujen1412 merged commit 8bbe493 into develop Oct 16, 2026
2 checks passed
@sujen1412
sujen1412 deleted the fix/pgt-token-retry branch October 16, 2026 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant