Allow precise-code-intel-worker /tmp scratch on a per-pod PVC - #956
Conversation
The worker writes large SCIP uploads to /tmp before processing them in multiple passes, and /tmp was always an unbounded emptyDir. That puts the write on the node boot disk, where a large upload competes with every other pod on the node and can fill the disk. A new storageType value selects the backing store. emptyDir stays the default, so rendered output is unchanged for existing installs. pvc switches /tmp to a generic ephemeral volume, giving each pod its own claim on storageClass.name that is discarded with the pod. storageSize is required for pvc and sets sizeLimit when left on emptyDir. The mount path stays /tmp, so the worker needs no TMPDIR redirect. A freshly provisioned volume is root-owned and the worker runs as a non-root user, so the pvc branch defaults the pod fsGroup to the container runAsGroup (with fsGroupChangePolicy OnRootMismatch) to keep /tmp writable. A user-set podSecurityContext.fsGroup still wins. An unknown storageType now fails rendering instead of silently falling back to emptyDir. Amp-Thread-ID: https://ampcode.com/threads/T-01a0f4b6-78bf-7109-82dc-4545a652ecdf Co-authored-by: Dinesh Kumar <dinesh.kumar@sourcegraph.com>
filiphaftek
left a comment
filiphaftek
left a comment
There was a problem hiding this comment.
Very nice!
I did some follow up in diff-your, and looks like using 100GB size we can increase 4x throuput + fully offload the boot disk: https://sourcegraph.sourcegraph.com/deepsearch/a9003fa4-3d60-408c-8931-57ddc3034cbc
1230ddf to
bc7adc8
Compare
Don't think it's last statement about 100GB is valid. We'll get headroom for boot-disk also we won't need 100GB per pod. the PCI workers usage's way less. On a different note pd-ssd throughput's already limited. Will monitor to see how this improves. https://sourcegraph.sourcegraph.com/deepsearch/8fa6fe56-3f4f-4538-ac58-6b22bea3f1ec |
3 participants
The precise-code-intel-worker writes large SCIP uploads to /tmp before processing them in multiple passes, and /tmp was always an unbounded emptyDir. That places the write on the node boot disk, where a large upload competes with other pods on the node and can fill the disk.
Changes
preciseCodeIntel.storageTypeto select the backing store for the/tmpscratch volume. One ofemptyDir(default) orpvcpvcmounts a generic ephemeral volume, so each pod gets its own claim onstorageClass.namethat is created and deleted with the podpreciseCodeIntel.storageSize, required forpvcand applied assizeLimitwhen left onemptyDirThe default renders
emptyDir: {}exactly as before, so existing installs are unaffected. The mount path stays/tmp, so the worker needs no TMPDIR redirect.ref https://app.incident.io/sourcegraph/response/incidents/531
ref EPD2-427
Test Plan
Will follow up with controller PR to enable this based on toggle.