Skip to content

Do not hold whole STAC object in memory before pushing result to client - #360

Merged
utas-amberbian merged 3 commits into
mainfrom
bugfix/9165-avoid-too-big-query-oom
Oct 8, 2026
Merged

utas-amberbian merged 3 commits into
mainfrom
bugfix/9165-avoid-too-big-query-oom

Conversation

@utas-raymondng

@utas-raymondng utas-raymondng commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator
  1. The previous implementation require the whole STAC object create before return to client, this consume a lot of memory and repeat big query will fail it.
  2. Now we hold 1 page of Elastic Search value in memory at most
  3. Lower the page_size for 1 page to 800 items
  4. No matter how many query, the max is limited by the number of query to Elastic which is 10 by default, so other query need to wait. With 10 request the same time
image
  1. We do not have config for this setting and accept the elastic default, we need code change if we need to tune this value due to still experience OOM
  2. Update extension version due to maven in CI/CD now goes to 3.10 and cause build fail

@utas-raymondng utas-raymondng changed the title Do not hold a whole object in memory before pushing a result to a client Do not hold whole STAC object in memory before pushing a result to client Oct 8, 2026
@utas-raymondng utas-raymondng changed the title Do not hold whole STAC object in memory before pushing a result to client Do not hold whole STAC object in memory before pushing result to client Oct 8, 2026

@utas-amberbian utas-amberbian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@utas-amberbian
utas-amberbian merged commit 642b52e into main Oct 8, 2026
4 checks passed
@utas-amberbian
utas-amberbian deleted the bugfix/9165-avoid-too-big-query-oom branch October 8, 2026 04:31

This branch was successfully deployed

1 active deployment
test — cb65bc42 Deployed Oct 8, 2026 by utas-raymondng via build_test #866
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants