Skip to content

Search: move indexing from the legacy docsearch-scraper to the Algolia Crawler #34

Description

@vharseko

Follow-up of #25 (item 5, DocSearch migration).

#33 moves the site search front end to @docsearch/js 3.9.0 on the existing index doc_openidentityplatform. The index is still written by the legacy docsearch-scraper (algolia/docsearch-scraper:v1.12.0, no longer maintained), which the "Reindex docs" step of .github/workflows/publish.yml runs with docsearch/config.json after every deployment.

The replacement is the Algolia Crawler. #33 adds docsearch/crawler-config.js — the same crawl (pages, exclusions, selectors), writing a new index doc_openidentityplatform_v3 — but it has not been run: it needs Crawler access on the Algolia application X0ME9NKL6F, which an open-source documentation site gets through the DocSearch program.

Steps (also in docsearch/readme.md)

  1. Get Crawler access for the application X0ME9NKL6F (or apply to the DocSearch program).
  2. Create a crawler in the Crawler dashboard from docsearch/crawler-config.js; the dashboard sets the API key.
  3. Run it into doc_openidentityplatform_v3 and check the results (e.g. a local build with that indexName in supplemental-ui/partials/footer-scripts.hbs); schedule it, or trigger it after each deployment.
  4. In one pull request: switch indexName to doc_openidentityplatform_v3, remove the "Reindex docs" step from publish.yml and docsearch/config.json. The ALGOLIA_API_KEY secret (write key of the legacy scraper) can then be removed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciContinuous integration, build and publish workflowsdocumentationImprovements or additions to documentationenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions