[karthickmk2001]

Closes #55

What does this change, and why?

The in-process Python probe (secfoo.sensor). It records which watched vulnerable functions actually run in a live app and writes #53 events. Docs are in docs/REFERENCE.md → Python probe.

secfoo instrument --watchlist watchlist.json -o events.ndjson -- python app.py
SECFOO_SENSOR=1 SECFOO_WATCHLIST=watchlist.json SECFOO_EVENTS_OUT=events.ndjson python -m secfoo.sensor app.py

Approach (as the issue describes, in src/secfoo/sensor/probe.py):

• Python ≥ 3.12: sys.monitoring with its own tool id. PY_START is set locally on the watched functions' code objects, so unwatched code costs nothing.
  • App-code coverage uses global PY_START that returns DISABLE after each function's first event, for app_functions_executed.
  • Watched functions keep counting through a lock-free path: one dict lookup and two stores, with timestamps kept as floats until flush.
• 3.10/3.11: a sys.setprofile + threading.setprofile fallback filtered to watched code (forced with SECFOO_SENSOR_BACKEND=profile). CI exercises it on py3.10.
• Symbol resolution: watched symbols resolve when their module is imported, through a meta-path hook that never finds modules itself.
• Workers: forked workers get a new session via register_at_fork. Spawned or multiprocessing workers start their own probe through sitecustomize, and flush through multiprocessing.util.Finalize, because workers leave via os._exit and skip atexit. {pid} in SECFOO_EVENTS_OUT gives one file per process. Otherwise each flush is a single O_APPEND write, so processes can share a file.
• Output: a daemon thread flushes running totals every SECFOO_FLUSH_INTERVAL (default 10 s) and at exit. That's safe to resend because Runtime evidence format, storage and secfoo runtime ingest #53 replaces totals.
• Live-sensor additions:
  • Watchlist reload: checks the file's mtime at most every 30 s.
  • Inventory: load lines for imported distributions.
  • Start without code changes: secfoo instrument injects a temporary sitecustomize.
  • Transports: SECFOO_SENSOR_ADDR supports file://, unix://, tcp:// (loopback only), npipe:// and https:// direct mode (the Runtime ingest API: sensors use your Secfoo API key #66 API, with a 200 ms cap). Stream transports send the Probe ↔ sensor protocol and conformance kit (one contract for every runtime) #73 hello first.
• Route and auth tagging is left to Attribute runtime hits to live traffic (route, auth state) #69, so hits report route: unknown (shown as Executed).

Must not (all tested):

• It doesn't read f_locals, arguments or return values. A test asserts that the argument password=hunter2 never appears in the events.
• It doesn't raise into app code. An internal error disables the probe, with one stderr line, and the app keeps running.
• It opens no connection other than SECFOO_SENSOR_ADDR, and without SECFOO_SENSOR=1 it installs nothing.

Overhead benchmark (python benchmarks/sensor_overhead.py --requests 200000, CPython 3.14, Windows, a ~7 µs request-like handler; each mode in a fresh interpreter):

mode                                     req/s  overhead
no probe, unwatched                    140,405
probe (profile), unwatched              13,932    907.8%
probe (monitoring), unwatched          148,280     -5.3%
no probe, watched                      138,287
probe (profile), watched                13,576    918.6%
probe (monitoring), watched            115,571     19.7%

• sys.monitoring: ~0 cost for unwatched code. About 1.4 µs per call of a watched function, even when it's called on every request. That's well inside Tracking: Runtime reachability for SCA (in-app probe + live sensor) #51's 1 ms p99 budget.
• setprofile fallback: about 10× slower, as the issue anticipates, because it fires on every call. The docs say to prefer 3.12+ in production.

The #73 conformance kit (#73 says this probe is its reference) comes in the next PR, which builds on these transports.

Testing

• pytest tests passes. 1204 passed locally. The only failures are the 4 existing Windows-only POSIX permission and symlink tests that fail on main too; Linux CI is unaffected.
• ruff check . adds no new findings.
• Added tests/test_sensor/test_python_probe.py (21 tests, real subprocesses, both backends):
  • a sample app calling and not calling the watched function
  • the entry frame and stack sample
  • coverage counts
  • no argument values in the output
  • the output ingesting cleanly through Runtime evidence format, storage and secfoo runtime ingest #53
  • inert without SECFOO_SENSOR
  • fail-open: a broken watchlist, an error inside the probe, and the sensor dying mid-run
  • a multi-process run (a spawn pool: separate sessions, every call counted once), and {pid} files
  • TCP sending hello, and loopback being enforced
  • transport selection, and direct mode requiring an explicit key and project

🤖 Generated with Claude Code

[github-actions]

Leaderboard update

• Closed Python live sensor (sys.monitoring) #55 (difficulty: hard): +10 points

karthickmk2001 now has 51 points (rank #1, Silver). 49 more to reach Gold.

Full leaderboard