Repository navigation
Warn when request dicts contain unrecognized keys - #138
Open
aryannpanwarr wants to merge 1 commit into
Open
aryannpanwarr wants to merge 1 commit into
aryannpanwarr wants to merge 1 commit into
Conversation
Request dictionaries are validated against pydantic models that ignore
unknown keys, so a typo such as "solve_captcha" or a wire-format name
such as "useStealth" was dropped silently and the default (False) was
sent instead. The session was created with the feature off and no error.
coerce_request now emits a UserWarning listing each unrecognized key,
including nested request models, with a suggestion ("did you mean
'solve_captchas'?" / "use 'use_stealth'"). Open mappings such as JSON
schemas, mounts and custom actions are not checked. Payloads are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
If you pass an option name the SDK doesn't recognize, it's silently dropped and the default (
False) is sent instead. No error, no warning.We hit this on Oct 11 while testing the SDK (1.10.0, still present in 1.11.0) against the live API on a free-plan key:
{"use_stealth": True, "solve_captchas": True}useStealth: True, solveCaptchas: True{"use_stealth": True, "solve_captcha": True}(typo)useStealth: True, solveCaptchas: False{"useStealth": True, "solveCaptchas": True}(API/Node names)useStealth: False, solveCaptchas: FalseSo a one-letter typo, or copying option names from the API docs or the Node SDK, turns the feature off and you still get (and pay for) a session. You only find out later when a site blocks you.
Fix
coerce_request(which every dict request goes through) now checks the keys against the request model and warns about the ones that would be dropped, with a suggestion:scrape_options.only_main→ did you meanonly_main_content?).mounts,custom_action, and anything typed as a dict orAny.Testing
tests/test_unknown_request_keys.py.-W error::UserWarningto make sure no existing valid request triggers a false warning.ruff check/ruff formatclean (0.3.7).🤖 Generated with Claude Code