Skip to content

Implement stack/heap collision protection for processes - #42

Merged
automas-dev merged 5 commits into
mainfrom
agents/stack-heap-protection-implementation
Sep 12, 2026
Merged

automas-dev merged 5 commits into
mainfrom
agents/stack-heap-protection-implementation

Conversation

@automas-dev

@automas-dev automas-dev commented Sep 5, 2026 •

Copy link
Copy Markdown
Owner

Implemented stack and heap collision protection so a process's heap and stack can never grow into each other, plus several related correctness fixes discovered and validated while testing this feature under headless QEMU boots.

Stack/heap collision guard (original feature)

  • addr.h: Added HEAP_STACK_GUARD_PAGES, the minimum unmapped gap required between a process's heap and stack.
  • process.c: process_add_pages now rejects heap growth that would come within the guard gap of the stack; process_grow_stack now rejects stack growth that would come within the guard gap of the heap. Both check before touching paging structures and use additive comparisons to avoid unsigned underflow.
  • process.h: Documented the new guard behavior.
  • test_process.cpp: Added process_add_pages_CollidesWithStack / process_grow_stack_CollidesWithHeap.

Bug fixes found via headless-boot testing

  • system_call_proc.c: SYS_CALL_PROC_PANIC handler had inverted null-checks that silently discarded the real panic message, and a %s/%u format mismatch on the line number that crashed the kernel while trying to log a panic. Both fixed - process panics now log correctly instead of taking down the kernel.
  • io_device.c: io_device_close had an inverted success/failure check, logging a spurious "error 0" on every successful close and never logging real failures. Fixed.
  • process.c / memory.c: process_add_pages and kernel_alloc_page (kmalloc's page source) both had an off-by-one when calling paging_add_pages, mapping one extra page per call. This silently leaked/orphaned a physical page on every heap growth, and - since HEAP_STACK_GUARD_PAGES == 1 - could fully erode the new stack/heap guard band in a single call.

Range-convention refactor

Rather than patch each call site with a -1/+1, converted paging_add_pages, paging_remove_pages, and paging_id_map_range from end-inclusive [start, end] to end-exclusive [start, end) ranges (matching common C/iterator conventions), and updated every call site accordingly. This removes the off-by-one bug at its root instead of just patching symptoms, and empty ranges (start == end) are now valid no-ops.

Verification

  • Full kernel unit test suite passing (13/13), including new tests for the guard checks and the exclusive-range boundary/empty-range behavior.
  • make lint clean.
  • OS builds successfully and boots cleanly under headless QEMU with no panics, page faults, or GPFs.

@automas-dev
automas-dev force-pushed the agents/stack-heap-protection-implementation branch 2 times, most recently from a75453b to e980edd Compare September 6, 2026 03:20
- Update I/O device handling in io_device.c
- Modify system call processing in system_call_proc.c
- Enhance process management in process.c
- Improve test coverage for process handling in test_process.cpp
Adjust memory allocation logic for improved stack and heap protection.
- Update paging.h to enhance memory management definitions.
- Modify memory.c to support new paging features.
- Improve paging.c with additional stack-heap protection logic.
- Adjust process.c to align with updated memory management.
- Expand test coverage in test_paging.cpp and test_process.cpp.
@automas-dev
automas-dev merged commit 4dd6020 into main Sep 12, 2026
4 checks passed
@automas-dev
automas-dev deleted the agents/stack-heap-protection-implementation branch September 12, 2026 03:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant