feat: updates to the py_extension rule for building C extensionsPy extension#3875
Open
rsartor-cmd wants to merge 437 commits into
Open
feat: updates to the py_extension rule for building C extensionsPy extension#3875rsartor-cmd wants to merge 437 commits into
rsartor-cmd wants to merge 437 commits into
Conversation
…b#3626) Copies the changelog updates in 1.8.5 to main. Also fixes some links for the other 1.8 releases --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…b#3589) This is testing the new API to make use of remote caching mechanisms. Needs: bazelbuild/bazel#27634 --------- Co-authored-by: Richard Levasseur <rlevasseur@google.com>
…rsion (bazel-contrib#3635) With this change we are handling more of the edge cases for when the filenames are more complex. Initial code had bugs when the sdist name had `-` in the name part. This code is easier to read and a little bit more explicit how it handles things. We will use it later to only return the `whl` and `sdist` entries for the versions requested through the requirements lock file. This is to make it possible to write facts only for the versions that we use. Work towards bazel-contrib#2731
…ib#3637) Replace bare except clauses with except Exception.
Have it more explicitly reflect the reality that this is a volunteer run project. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…azel-contrib#3639) We want to keep a `dict` like interface and later we would like to use the same interface to also do more things. I expect the cache key to become different in the future (i.e. include requested versions in it) so that we can check if we have the right versions in the MODULE.bazel.lock file or if we should actually call to PyPI. Work towards bazel-contrib#2731
…azel-contrib#3593) This fixes the incompatibility between `packaging==24.0` and `setuptools==78.1.1` by just updating the requirements. This works as intended in my own repositories (where I had problems with requirement updates before), but I am not sure if there are still some issues like what caused the original revert of the `packaging` update. I checked and I don't see any absolute paths in the compiled lock files (as originally mentioned in bazel-contrib#908), but I don't know if there are any other known issues. Another solution is to just pick compatible versions of `packaging` and `setuptools` some other way. This closes bazel-contrib#3532. Note: I haven't updated the changelog, since I was not sure if this solution is even desirable? Or if you prefer to only change `packaging` and `setuptools` versions? --------- Co-authored-by: Richard Levasseur <richardlev@gmail.com> Co-authored-by: Ignas Anikevicius <240938+aignas@users.noreply.github.com>
bazel-contrib#3450) With this change, when user build with a native python flag, `flag_alias` will point to the starlark version of the flag. Please see design doc [here](https://docs.google.com/document/d/1yOvi4hVV7Ja32ocwVb4lsEUnijftk8nilXPncYm-BH8/edit?tab=t.0#heading=h.qn3unswby87l) for more details. This feature should work as a no-op in the latest bazel 8.x and 7.x releases. --------- Co-authored-by: Richard Levasseur <richardlev@gmail.com> Co-authored-by: Richard Levasseur <rlevasseur@google.com>
The flag_alias() function in MODULE.bazel has a bug where it doesn't properly parse labels when transitions are involved in certain ways (not entirely clear). A fix will com in a later Bazel release. Until them, remove the calls. Fixes bazel-contrib#3648
Remove most of the usage of os.environ.get and use getenv instead. The question is if we should just replace all usages blindly?
Upcoming RC builds may have regressions, so instead of blocking our CI on their failures, mark them as soft-fail. This way we can be aware of upcoming problems, but not block regular development.
…bazel-contrib#3653) The compression level isn't being correctly passed by the zipapp rule to the zipper program, resulting in an error if custom compression is specified. To fix, pass the arg correctly to the zipper. `venv_zipapp_test.py` is updated to also confirm zipapps are properly compressed when configured in the BUILD target. Fixes bazel-contrib#3646
…ontrib#3641) Currently, the toolchain only sets the make variables if it's an in-build toolchain. This breaks for platform toolchains. To fix, check for platform toolchains and set the make vars appropriately. This helps fix the BCR rules for `@glib`, which recently added a requirement that the Make variable `$(PYTHON3)` must always be defined by the current Python toolchain. Co-authored-by: Richard Levasseur <richardlev@gmail.com>
With this PR we move the processing of the `index_url` to the `whl_library` as a preparatory step for easier `facts` implementation. The motivation is many-fold: 1. Do not have too much duplication in the facts file by potentially naturally eliminating the `index_url` prefix from the `whls` if it appears like so on the index contents. 2. Avoid doing `envsubst` too early and have logic that has to deal with it. 3. Make the cache just return fact values from the lock file in the future instead of needing to change to an absolute URL and do envsubst on it. 4. We should have a better performance because we should be doing way fewer calls to make the URL absolute during parsing of the index. 5. With the `index_url` passed to the `whl_library`, we can help out the `purl` construction as what has been discussed in bazel-contrib#3531 about wheels from non-public indexes. Summary: - Attempt to put the `index_url` in the fewest structs possible. - Extract the `urllib` utilities file for manipulation of the URLs. - Simplify tests testing the `absolute_url` logic. Work towards bazel-contrib#2731
…azel-contrib#3650) This runs the default test jobs using the test matrix so that Bazel 9 is better covered by testing. The main thing needed is setting `--incompatible_strict_action_env=false`. This is enabled in Bazel 9, but breaks Windows. See bazel-contrib#3655 for details. It also changes to using `N.x` instead of `N.*` for specifying the Bazel version. The difference is `x` matches the latest released version, while `*` matches the latest released _or release candidate_. Since release candidates can have regressions, and we have a separate job for RCs, use the latest released version. Along the way, mark the last_rc job as soft-fail
Updates the project bazel version and bzlmod examples to use Bazel 9 Along the way, upgrade rules_go to 0.60.0; this is a dev-only dependency. This is needed because Bazel 9 removed CcInfo, and earlier rules_go versions refer to the builtin symbol. Also remove the test to check the bazel version. It doesn't do much useful.
…cs (bazel-contrib#3658) aignas suggested I add a link to this YouTube video somewhere in the docs: https://bazelbuild.slack.com/archives/CA306CEV6/p1773133824891709?thread_ts=1773089164.263869&cid=CA306CEV6
Supporting Windows without full symlink support is quite difficult because we rely on symlinks in many places. Since Windows support is mediocre already, just require symlinks to be enabled. Symlink support in Windows has been around for a long time now and is available via non-admin mechanisms such as DevMode.
This makes the logic in the parser a little bit more sophisticated, but we also start handling the yank reason. With this the yank reason will be present in the logs, which should be a usability improvement. The `yanked` attribute will be None if the package is not yanked and a string value if it is yanked. An empty string value should be treated as "yanked with no reason provided". This implementation assumes that we have HTML escaped sequences as tag values. It also un-escapes them when returning the strings. The possibilities that it gives us are, but out of scope for this PR: - Use the `data-requires-python` to potentially discard any Python packages that are unsupported in the `select_whl` function. - Also retrieve the provenance attributes if they are supported. Work towards bazel-contrib#260. Work towards bazel-contrib#2731.
This PR adds functionality needed to write data that we find useful on the SimpleAPI responses to the lock file. I.e. this will no longer connect to the network if it can find the necessary information in the lock file. Superseeds bazel-contrib#3559 Fixes bazel-contrib#2731 --------- Co-authored-by: Richard Levasseur <rlevasseur@google.com>
This PR deletes the `pip_repository_annotations` example. This is because it is a workspace-only example, and its removal is preparatory work for dropping workspace support from rules_python.
…ib#3628) This is to make it easier to check for missing version markers by running the script directly
…contrib#3661) The runtime repositories are being constantly invalidated due to pyc creation under Bazel 9 because, starting in Bazel 9, `glob()` functions implicitly register `repository_ctx.watch()` calls on the files and directories they match. Thus, the directories where `__pycache__` directories are created end up being considered changed (either directly because their mtimes change, or indirectly, because their directory listing changes), which then invalidates the repo, causing it to re-run. This glob-induced-watching seems to occur even if an `exclude` would have excluded the file. Note that this only seems to occur if `reproducible=False`, which generally wouldn't occur, but could occur if a user is registering their own runtime and doesn't care about the sha. Regardless, this still seems worthwhile because it allows pyc to be more safely be generated without causing repo invalidations, while allowing them to be persisted between repo-phase invocations. To fix, create `__pycache__` directories ahead of time and symlink them to a location that Bazel isn't watching, i.e. outside the repository's directory. I tried creating a separate top-level folder that wasn't matched by any globs and symlinking to it, but Bazel would read through the symlinks and watch the underlying locations. This also has a side-bonus that allows pyc files to be re-used in between repository-phase invocations. Fixes bazel-contrib#3643
Splitting sphinxdocs into its own module is desirable to reduce the dependency footprint of rules_python outside of development mode modules. Work towards bazel-contrib#2511 --------- Co-authored-by: Ignas Anikevicius <240938+aignas@users.noreply.github.com>
…3664) The term "module space" isn't very clear. The directory it refers to is the runfiles root directory. So refer to it as that instead of "module space"
…contrib#3665) Clarify that some variables are runfiles-root relative paths so that their values aren't confused to be "main repo" runfiles relative paths like "../foo"
…trib#3233) When specifying `data_files` in `py_wheel`, allow just the directory to be specified (with a trailing slash), in which case it will use the existing filename. This avoids duplicating (potentially platform-specific) names. Additionally, targets with multiple files can be installed as a group to a folder, with the same filename-preserving behavior. In general I think this is a better starting point, as I imagine most of the time users would want to preserve the names. Before, this would result in the file simply not being installed, so this only changes already-broken behavior. --------- Co-authored-by: Richard Levasseur <rlevasseur@google.com>
…es (bazel-contrib#3636) This fixes bazel-contrib#3563. Verified by running the repro in https://github.com/mering/reproduction_rules_python_1_7. The identified regression in bazel-contrib@b8e32c4 is problematic because prepending `ctx.workspace_name` to `short_path` results in paths containing `..` (e.g., `_main/../sub+/path/to/file.py`) when building from a root module that includes other modules. This causes the `_find_runfiles_root` [logic](https://github.com/faximan/rules_python/blob/eb6ac472eb86cc263acc336a2b73982043069aae/python/private/site_init_template.py#L73), which _counts slashes_, to incorrectly calculate the runfiles root. The fix is simply using the available `runfiles_root_path` function instead. In the example above, this makes the path simply `sub+/path/to/file.py`. --------- Co-authored-by: Ignas Anikevicius <240938+aignas@users.noreply.github.com>
…ib#3667) This uses snake_case function names in the system python bootstrap. This is just to modernize the code a bit. Along the way ... * Add some additional debug logging * Fix the `if is_windows:` conditional that was always executing (because it was referring to a function), but a no-op, on Linux.
…used (bazel-contrib#3666) Before this PR there would be confusing failures when downloader config is set to disallow certain values or when the authentication is not setup properly. This is a small fix towards a better goal state where we set `allow_fail = False` in cases where we know that we have to succeed to download metadata from that particular URL. The use-cases covered: - Only one index_url is passed to `pip.parse`. - `index_url_overrides` are passed which means that we should fail if there are insufficient overrides. The downside to this is that it is really hard to return custom error messages telling the user what to do, but on the flip side, the failures coming from bazel itself might be more descriptive in the case of outh-misconfiguration or bazel downloader configuration settings. Work towards bazel-contrib#2632 and bazel-contrib#3260.
…nfirm py_extension behaves similarly to cc_shared_library.
…ry and cc_shared_library targets.
…using the obsolete PLATFORMS global.
rsartor-cmd
force-pushed
the
py-extension
branch
from
96b6ba2 to
acf55b4
Compare
Author
|
Attempting a rebase to fix the CI failures. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per feedback from #3851, this PR re-works the platform detection logic to rely on platform constraints instead of the Python runtime/toolchain.
Description
This PR contains updates to the experimental
py_extensionimplementation. It re-works the platform tag detection to rely on modern platform constraints, and refactors the compilation and linking mechanismto delegate to Bazel's native
cc_shared_libraryandcc_libraryrules.This PR targets the main repository's
py-extensionbranch (notmain).Key Changes
1. Platform Tag & ABI Tag Derivation
abi_tagtoPyCcToolchainInfo: Added theabi_tagfield to thePyCcToolchainInfoprovider, populated by thepy_cc_toolchainrule. It defaults to deriving the tag frompython_version(e.g.cpython-311) for backward compatibility.cc_toolchainCPU names with direct lookup in rules_python's centralPLATFORMSregistry using platform constraints..abi3.so) suffixes are appended based on the toolchain configuration.2. Compilation & Linking Delegation (Refactor to
cc_shared_library)py_extensionrule with a macro of the same name. It now accepts C/C++ source/header files directly (srcs,hdrs,copts,defines),implicitly wrapping them in a private
cc_libraryunder the hood.deps(removing the redundantstatic_depsattribute) and aligned linker arguments withcc_shared_library'suser_link_flags._py_extension_wrapper): Added a lightweight, private rule that wraps thecc_shared_libraryoutput to:.cpython-311-x86_64-linux-gnu.soor.abi3.so).symlinkfrom the CSL output to the PEP 3149 name.PyInfo(for python rules) andCcSharedLibraryInfo(for dynamic C++ dependencies).cc_shared_libraryensures strict analysis-time validation against One Definition Rule (ODR) violations (e.g., duplicate static linkage in dynamic chains).3. Runfiles and Data Support
builders.RunfilesBuilderto support runtime assets (dataattribute) and dynamic library dependencypropagation.
Verification & Testing
dependency_graph_tests.bzlto verify dynamic dependency chains and static sharing behavior.py_extension_tests.bzlto verify data asset propagation.//tests/cc/py_extension/...are compiling and passing.──────
TAG=agy
CONV=23f8a5e8-2d99-401a-9903-1256b7d42b0e