[Feature] Nested executor

[jan-janssen]

Allow using the executor inside a submitted function

Example

import executorlib

def echo(i):
    return i

def nested_echo(i, executorlib_executor):
    future_obj = executorlib_executor.submit(echo, i)
    return future_obj

with executorlib.SingleNodeExecutor() as exe:
    future_obj = exe.submit(nested_echo, 4)
    print(future_obj.result())

Current challenges

• The example code hangs, so something is wrong with the implementation.
• At the moment the user is not allowed to set the executor, but simply has to name the variable executorlib_executor and keep it empty, which is not very intuitive.
• It is unclear how to access the output of the nested functions, as the user does not have access to the corresponding future objects.
• Dependencies for nested functions does not work.
• The resolving of nested functions is not possible when disable_dependencies=False but no error message is raised.

Summary

It is primarily a concept and not something which is going to be available any time soon.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e65ce5ad-9338-4051-b535-d5ae7ebdc289

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch nested_executor

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 73.43750% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.67%. Comparing base (faf71ea) to head (193e92a).

Files with missing lines	Patch %	Lines
src/executorlib/backend/executor_nested.py	61.90%	8 Missing ⚠️
src/executorlib/backend/interactive_parallel.py	0.00%	4 Missing ⚠️
...ecutorlib/task_scheduler/interactive/dependency.py	62.50%	3 Missing ⚠️
...rlib/task_scheduler/interactive/blockallocation.py	75.00%	1 Missing ⚠️
...executorlib/task_scheduler/interactive/onetoone.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1030      +/-   ##
==========================================
- Coverage   94.22%   93.67%   -0.56%     
==========================================
  Files          39       40       +1     
  Lines        2113     2166      +53     
==========================================
+ Hits         1991     2029      +38     
- Misses        122      137      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.