Skip to content

feat: make the queue volume optional - #198

Open
armru wants to merge 3 commits into
mainfrom
dev/135
Open

feat: make the queue volume optional#198
armru wants to merge 3 commits into
mainfrom
dev/135

Conversation

@armru

@armru armru commented Aug 31, 2026

Copy link
Copy Markdown
Member

A Server had to declare a queue PVC whenever tier1 was configured. The queue
only needs durable storage, not a volume of its own, so the queue section is
now optional: when omitted, the queue lives in the queue directory of the
tier1 data volume. A dedicated volume is still recommended in production, so
that a full data volume cannot also stall the queue that drives retention.

Adding or removing the section moves the queue during the resulting rolling
restart: the server copies it from the previous location, named by
QUEUE_MIGRATION_SOURCE, before opening it. On removal the StatefulSet stops
templating the queue PVC, which is then kept mounted by claim name until it is
deleted by hand.

Closes #135

@armru
armru force-pushed the dev/135 branch 3 times, most recently from af2a787 to 82b80a3 Compare September 2, 2026 08:08
@gabriele-wolfox
gabriele-wolfox force-pushed the dev/135 branch 3 times, most recently from e9eac37 to 6d4baea Compare September 3, 2026 13:38
armru and others added 3 commits September 3, 2026 16:57
The work queue can now live either on its own volume or inside the tier1
data volume, so the server has to relocate it when the Server resource
changes which one it is. QUEUE_MIGRATION_SOURCE names the previous
location: its content is copied into QUEUE_DIRECTORY before the queue is
opened, then removed.

The copy is staged next to the destination and fsynced before the rename,
so an interrupted migration never leaves a partial queue behind. A
non-empty destination always wins, as it holds the WAL files still
pending transfer to tier2.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
The queue only needs durable storage, not a volume of its own, so the
`queue` section is no longer required alongside tier1: when omitted, the
queue lives in the `queue` directory of the tier1 data volume. A dedicated
volume is still recommended in production, so that a full data volume
cannot also stall the queue that drives retention.

Adding or removing the section moves the queue during the resulting
rolling restart. On removal the StatefulSet stops templating the queue
PVC, which is then kept mounted by claim name until it is deleted by hand.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Adding a dedicated queue volume to an existing Server crash looped it:
copyQueue replaced destination with a rename, assuming it shared a
filesystem with the staging copy next to it, but destination is itself
a mount point in that case, so removing or renaming into it fails.

Stage the copy inside destination instead, then adopt each entry with
its own rename, which stays on destination's filesystem either way.

TestMigrateQueueDirectory/moves_the_queue_into_a_freshly_mounted_destination
reproduces this with a real bind mount and now passes; it failed with
"device or resource busy" before this change.

Assisted-by: Claude
Signed-off-by: Gabriele Quaresima <gabriele.quaresima@enterprisedb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make queue volume optional

2 participants