Skip to content

Feature/9348 improve download queue - #358

Merged
utas-raymondng merged 9 commits into
mainfrom
feature/9348-improve-download-queue
Oct 9, 2026
Merged

utas-raymondng merged 9 commits into
mainfrom
feature/9348-improve-download-queue

Conversation

@shaunahu

@shaunahu shaunahu commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Tag each generating-data-file-for-* download job with a fair-share class, small or large, taken from the existing size estimate. A download estimated under 50MB (aws.batch.job.share.small-max-size) is small. Anything else is large: 50MB or more, a failed estimate, or the size limit switched off.

  • The tag is always sent as the share_identifier job parameter. DAS can read it, and parquet downloads carry it into their prepare/collect child jobs.
  • It is also sent as the top-level Batch shareIdentifier, but only when aws.batch.job.share.enabled=true. That defaults to false, so no environment changes behaviour: AWS rejects shareIdentifier on a queue without a fair-share policy.

Follow-up, once infra has a queue with a fair-share scheduling policy whose share identifiers are small and large:

  • ogcapi: set share.enabled=true and point aws.batch.job.queue / child-queue at that queue.
  • DAS: set shareIdentifier on the prepare/collect child jobs it submits.
image

@shaunahu
shaunahu marked this pull request as ready for review October 9, 2026 01:18

@utas-raymondng utas-raymondng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@utas-raymondng
utas-raymondng merged commit 90af039 into main Oct 9, 2026
4 checks passed
@utas-raymondng
utas-raymondng deleted the feature/9348-improve-download-queue branch October 9, 2026 01:24

This branch was successfully deployed

1 active deployment
test — 25272c7d Deployed Oct 9, 2026 by shaunahu via build_test #873
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants