Skip to content

docs: recommend several example URLs of one page type for prepare_domain() #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
zenrows-dev-pool wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

docs: recommend several example URLs of one page type for prepare_domain() #68

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,9 @@ The legacy `402` with `code: "AUTH010"` means the domain isn't enabled. `extract

#### Preparing a domain

`client.prepare_domain(domain, example_urls)` submits a domain for Extract (`POST /v1/extract/prepared-domains`), and `client.prepared_domain_status(domain)` reads where it is (`GET /v1/extract/prepared-domains/{domain}`). Both return the parsed status object. Pass the domain without a scheme and 1 to 5 example URLs, one per kind of page you want to extract. These calls cost no credits.
`client.prepare_domain(domain, example_urls)` submits a domain for Extract (`POST /v1/extract/prepared-domains`), and `client.prepared_domain_status(domain)` reads where it is (`GET /v1/extract/prepared-domains/{domain}`). Both return the parsed status object. Pass the domain without a scheme and 1 to 5 example URLs on it. These calls cost no credits.

One example URL works, but several related URLs work better. We recommend at least 3 pages of each page type you need extracted, up to 5 in total; several page types can share the 5, for example 3 product pages and 2 search-result pages. Extract compares the pages of one type against each other, so it can check the fields it found and fill in fields that one page is missing. More pages of a type give more complete, more reliable results for that type.

Preparation takes minutes, and the client does not wait for it. Poll the status until it is no longer `"preparing"`:

Expand All @@ -243,7 +245,12 @@ client = ZenRowsClient("YOUR-API-KEY")
try:
status = client.prepare_domain(
"www.scrapingcourse.com",
["https://www.scrapingcourse.com/ecommerce/"],
[
# Three pages of one page type: product pages on the same domain.
"https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/",
"https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/",
"https://www.scrapingcourse.com/ecommerce/product/aeon-capri/",
],
)
except PreparedDomainError as err:
# e.g. 409 in_progress, 429 out_of_allowance, 503 overloaded
Expand Down
8 changes: 7 additions & 1 deletion src/zenrows/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,13 @@ def prepare_domain(self, domain: str, example_urls: list[str]) -> dict:
`ready` or `failed`), `state`, `submitted_at`, `updated_at`.

`domain` is the domain without a scheme. `example_urls` are 1 to 5
URLs on that domain, one per kind of page you want to extract.
URLs on that domain. One URL works, but several related URLs work
better: pass at least 3 pages of the same page type when you can,
up to 5 (several page types can share the 5, for example 3 product
pages and 2 search-result pages). Extract compares the pages of one
type against each other, so it can check the fields it found and
fill in fields that one page is missing; more pages of a type give
more complete, more reliable results for that type.
Submitting a domain that is already prepared returns its current
status instead of starting a new run.

Expand Down
Loading