Clean up file handle on interrupt, timeout

[gaborcsardi]

Closes #28.

[eliotmcintire]

I am a heavy user of filelock, through reproducible::Cache(), on a cluster where many R processes share one cache. I hit this leak regularly. I only work in R, so I used Claude (an AI assistant) to find the cause and test a fix. This is a person and an AI working on it together.

Thanks for this PR. It fixes the leak that hurts us most. reproducible::Cache() waits on a lock with lock(path, timeout = 2500) in a loop. On current filelock, a process that waited about 1.5 hours had 1,230 descriptors open on one lock file, and parallel::mclapply() then failed with "file descriptor is too large for select()".

Two failure paths still leak with this PR:

1. With timeout = 0, when the lock is held, filelock_lock() returns R_NilValue without closing the file. r_call_on_early_exit() only runs on an error or interrupt, not on a normal return.
2. On Windows, filelock_lock() does not close the CreateFileW() handle when the lock is not acquired.

Descriptors left open on the lock file after 10 failed calls while another process holds the lock (Linux, R 4.6.1):

	timeout = 0	timeout = 100	interrupt
main	10	10	left open
this PR	10	0	closed
this PR + patch	0	0	closed

The patch is one line (close(args->filedes); before that return R_NilValue) plus CloseHandle(file) on the Windows failure paths, with a test that fails on this branch and passes with the patch: https://github.com/PredictiveEcology/filelock/tree/fix/pr56-remaining-leaks. Happy to open it as a PR against this branch, or you can take the commit.

Two things we noticed while testing this branch: the vendored cleancall calls Rf_findVar (src/cleancall.c, src/init.c), which R CMD check on R 4.6 reports as a non-API call, and gcc warns about its implicit declaration; and the cleanup handler still has a debug REprintf("Cleaning up: closing file descriptor ...").

In case it helps, we also tried a version without cleancall, on current main: it closes the file on the failed returns and uses R_UnwindProtect() for errors and interrupts. It gives 0 leaks in all of the cases above, but needs R >= 3.5: main...PredictiveEcology:filelock:fix/close-fd-on-failed-lock. The Windows changes in both branches are only CloseHandle() calls; we could not build or test on Windows or macOS.

🤖 Generated with Claude Code

[gaborcsardi]

Thanks! FWIW the UnwindProtect version fails the tests on Windows: https://github.com/r-lib/filelock/actions/runs/36536194887

[eliotmcintire]

Thank you.

The Windows failure was in the new test, not the fix: on Windows the wait loop closes the handle and raises "Locking interrupted for file …", so the condition is an error rather than an interrupt. The check that the file was closed already passed there. #58 now accepts that error on Windows and passes on all platforms: https://github.com/r-lib/filelock/actions/runs/38965602145

#58 covers what this PR does (closing the file on a timeout, an error or an interrupt while waiting) and also the timeout = 0 and Windows handle leaks, without the vendored cleancall. It does need R >= 3.5. If you agree, this PR could be closed in favour of #58.