Add recipe: BYO Vector Store — LlamaCloud Index V2 to Azure AI Search

[farzad528]

---

Description

Adds a runnable cookbook: BYO Vector Store — LlamaCloud Index V2 → Azure AI Search.

LlamaCloud parses your documents; Azure AI Search governs and grounds the result. This recipe runs the official index-v2-data-sinks Azure AI Search exporter end-to-end against your own service — keyless via Entra ID — proves server-side tenant isolation, and layers LlamaIndex Retrieval Harness primitives (List Files, File Grep, File Read, Hybrid Retrieve) on the exported index. The payoff for LlamaIndex customers: land parsed content in their tenant with zero indexing code, then graduate the same index into Foundry IQ agentic retrieval.

Included (5 files): the notebook, its architecture image, the registry.yaml entry, new tags (llamaindex, byo-vector-store, entra-id, multi-tenant, foundry-iq), and openai + beautifulsoup4 in requirements.txt.

Validated live before outputs were cleared: executed top-to-bottom, fully keyless, against a live Azure AI Search + Azure OpenAI text-embedding-3-large — Auth: Entra ID (keyless), 60 chunks exported via the official AzureSearchExporter, tenant isolation (forgebook-demo → 5 hits vs another-tenant → 0), all four harness primitives, idempotent re-export (60 → 60, no duplicates), and list_snapshots() returning one record per file. Outputs contain no secrets. Registry validator passes (18 notebooks). REST api-version pinned to 2026-04-01 (SDK 12.0.0 default); install pinned to a specific index-v2-data-sinks commit (Python 3.13+, GitHub, not PyPI).

Closes # — N/A (no linked issue)

Review focus

• Product accuracy & positioning — the LlamaIndex ↔ Azure AI Search partnership framing, the governance claims (keyless Entra ID, tenant isolation), and the RBAC guidance: Cognitive Services OpenAI User for the direct /openai/v1 embedding path, Foundry User only for Foundry project endpoints (verified against Microsoft Learn).
• The Foundry IQ graduation is prose + link, not runnable (kept to preserve ~15-min fast first value). Good candidate for a Part 2 cookbook that runs knowledge source → knowledge base → agentic retrieve.
• Dependency install pins index-v2-data-sinks from a GitHub commit because it has no PyPI release; requires Python 3.13+.

---

Checklist

Notebook contributions

• Recipe author is an approved Microsoft Forgebook author, or this contribution has maintainer approval
• Recipe was drafted or validated through the internal authoring workflow — author to confirm
• Foundry PM, engineering, or DX reviewer has checked product accuracy and positioning — requested via this PR
• Notebook added to notebooks/ directory
• Entry added to registry.yaml with all required fields (slug, path, title, authors)
• Approved author entry exists in authors.yaml (with at least name and title)
• Tags are from the allowed list in tags.yaml
• Registry validation passes: cd scripts && npx tsx validate-registry.ts
• Notebook runs cleanly end-to-end
• Cell outputs cleared before committing
• No site-specific metadata inside the notebook
• No secrets, tenant-specific values, or private endpoints are committed
• Images stored in notebooks/media/<slug>/ with descriptive alt text
• Data files stored in notebooks/data/<slug>/ when applicable — N/A (no data files)

Site changes

N/A — this PR only adds a notebook plus its registry.yaml / tags.yaml / requirements.txt metadata. The registry drives the site, so preview.yml CI will run npm run build + astro check on the PR.

• Site builds without errors: cd site && npm run build — runs in CI
• No TypeScript errors: cd site && npx astro check — runs in CI

---

[nicholasdbrady]

Hypothesis: This is an intermediate tutorial for developers who want LlamaCloud parsing while retaining Azure AI Search as the governed retrieval store.

Success criterion: A reader can export parsed content keylessly, verify retrieval behavior safely, and reuse the resulting index without mistaking a query filter for an authorization boundary.

Confidence: High from the full notebook, pinned exporter source, registry/site validation, and current Azure AI Search security guidance. I did not repeat the author's live Azure execution.

Verdict

Revise and resubmit — 72/100. The exporter walkthrough and Retrieval Harness are useful, but the current notebook overstates tenant isolation and can delete a pre-existing index with a predictable name. Those need correction before publication.

Required before publishing

• Critical — Cell 23/24 and related prose/tags: The notebook calls a caller-supplied tenant_id OData filter “server-side tenant isolation” and says one tenant can never see another tenant's chunks. Azure AI Search executes the filter server-side, but it does not authorize the filter value; a caller with index query access can omit it or request another value. Reframe this as server-side scoped retrieval, not a security boundary, and remove the multi-tenant tag unless the recipe implements enforced security trimming/ACLs or an authorization layer that derives tenant context and injects the filter. Azure's documented security-filter pattern ties stored principal identifiers to the authenticated caller: https://learn.microsoft.com/azure/search/search-security-trimming-for-azure-search

• Critical — Cell 18 and teardown: The notebook deletes INDEX_NAME before export, while INDEX_PREFIX is a predictable shared default. A reader can destroy an existing index from an earlier run or another user. Generate a unique demo index name for each run (or require explicit confirmation before replacing an existing index), track whether this run created it, and delete only that resource during teardown.

• Required — Cell 2: Remove the slug/tags/author/last-verified table from the notebook. Forgebook publishing metadata belongs in registry.yaml; notebooks remain source lessons without site-specific metadata.

• Required — Prerequisites/setup: The pinned package declares requires-python = ">=3.11,<3.14", not Python 3.13+. Correct the prerequisite and troubleshooting text. If the notebook intentionally requires 3.13 for another concrete dependency, name and verify that dependency explicitly.

• Required — Cell 6: Setup checks only whether index_v2_data_sinks imports. If another revision is already installed, the pinned commit and compatible Azure SDK versions are not applied, so later behavior can drift from the reviewed source. Install the exact pinned revision unconditionally or verify installed package/dependency versions and fail with a corrective command.

• Required — Cells 12, 21, 24, 29/32/35/38, and 41: Add assertions for the claims that make this a recipe rather than a demo: expected embedding dimension; exported document count; scoped query has results while the alternate scope has none; Retrieval Harness returns expected file/chunk shapes; re-export leaves the count exactly unchanged; snapshots contain the expected file IDs. The current idempotency loop exits after eight seconds and always prints “no duplicates,” even if the final count increased.

• Required — Opener, architecture image, and production-path section: Qualify “no keys.” The Azure demo defaults to Entra ID, but the production export_pipeline path requires LLAMA_CLOUD_API_KEY and embedding-provider authentication. Say “keyless for Azure services in this demo” so readers do not infer that the complete LlamaCloud workflow is credential-free.

Recommended improvements

• Optional — tags.yaml: Keep the genuinely new llamaindex integration tag, but prefer existing taxonomy where possible: iq over foundry-iq, identity over entra-id, and indexes/azure-ai-search over byo-vector-store. Add new global tags only when they will classify multiple recipes rather than restating this recipe's title.

• Optional — registry.yaml: Shorten the 241-character description to 200 characters or fewer so it does not overflow the card grid.

What is working

• Praise: The notebook has a clear thesis, one editable configuration cell, explicit runtime/cost guidance, and a useful progression from export to inspectable Retrieval Harness primitives.
• Praise: Pinning the unreleased exporter to an exact commit and grounding the walkthrough in the official sink implementation are strong reproducibility choices.
• Praise: Registry validation, notebook health checks, site preview, and CodeQL all pass.

Re-review gate

After these updates, please include a clean top-to-bottom run with assertion-backed evidence and sanitized outputs. I will re-review only the revised claims/cells plus the registry/tag changes.

[farzad528]

Thanks @nicholasdbrady — this was a genuinely useful review. I accepted all 8 required items and all 3 optional ones; nothing rejected. Pushed as a separate commit (c6f24ea) so you can diff just the fixes.

Re-verified with a clean keyless top-to-bottom run against a live Azure AI Search service: 0 errors, every assertion passing. Outputs cleared afterward per repo convention; the evidence is quoted below.

Critical

1. The tenant_id filter is not an authorization boundary. You're right, and the old framing ("tenant isolation", "zero leakage", "a tenant can never see another tenant's chunks") was the worst kind of wrong — it read as a security guarantee for something that is only a query scope. Step 4 is now "Scoped retrieval on the fields the sink stamps" and says so directly:

An OData filter is a query scope, not an authorization boundary. Nothing here verifies that the caller is entitled to the tenant_id they pass — any client holding a credential that can query this index can supply a different value, or omit the filter entirely, and read everything. What actually restricts access is Azure RBAC on the search service, which is why the keyless setup above matters.

Rather than just soften the prose, the cell now demonstrates the limitation — it runs the same query a third time with no filter at all:

scope 'forgebook-demo': 5 hits  |  scope 'another-tenant': 0 hits
no filter at all:     5 hits  <- the same caller, unscoped, sees everything

The remaining value is stated honestly (the service never ranks or bills for out-of-scope chunks, so they can't reach the model's context window), security trimming is linked as the path to enforceable per-user entitlement and added to Next steps, and the multi-tenant tag is dropped.

2. Destructive delete of a predictably named index. Also right — forgebook-index-v2-sink on a shared service is a landmine. Removed the pre-delete entirely and made it structurally impossible instead of just careful:

• INDEX_PREFIX carries a random per-run id (forgebook-index-v2-sink-7d614e66-...)
• setup asserts the name doesn't already exist
• the export records CREATED_INDEXES, and only on success
• teardown iterates that set, then asserts nothing it created survives

Created by this run (teardown will delete only this): ['forgebook-index-v2-sink-7d614e66-openai-text-embedding-3-large']
Deleted index 'forgebook-index-v2-sink-7d614e66-openai-text-embedding-3-large'.
Teardown complete — only indexes created by this run were deleted.

"Make it yours" notes that a stable name means dropping the teardown cell too.

Required

3. Metadata table. Removed — it duplicated registry.yaml. Replaced with the facts a reader actually needs before starting: difficulty, 15 min, and cost ($0 incremental on an existing service; ~$75/mo if they stand up a Basic tier just for this).

4. Python version. You were right and I'd propagated the error to three places. The pinned commit declares requires-python = ">=3.11,<3.14". Now stated as 3.11–3.13 in Prerequisites, the install cell, and Troubleshooting, plus a runtime assert.

5. Conditional install. Fixed — the try: import ... except ImportError guard meant a stale revision silently won. Now an unconditional --upgrade install of the pinned ref, and verification (you offered "or", I did both, since installing doesn't prove what got loaded). It reads the resolved commit from the distribution's direct_url.json and pins the Azure SDKs, failing with the exact corrective command:

Setup OK — sink @ 798cac7, Python 3.13, embeddings return 3072 dims.

I deliberately did not use --no-deps — the sink pulls real transitive dependencies (chonkie, llama-cloud, catsu, typer), so that would break a fresh environment.

6. Assertions. Added throughout, and this caught a real bug you'd flagged: the idempotency cell printed (no duplicates) unconditionally — it would have said that even if the count had doubled. It now asserts after == before and derives the message from the comparison.

Coverage: embedding dim vs. configured model · 1:1 embeddings-to-chunks per file · export total vs. expected · index exists after export · all seven provisioned fields present · vector field dimension matches the model · scoped/alternate/unscoped hit counts and tenant_id of every returned row · Hybrid Retrieve row shape and rank ordering · List Files set equality with ESSAYS and chunk sum vs. export total · File Grep containment plus a nonsense-term negative control · File Read count against the List Files number and index ordering · document count unchanged after re-export · one snapshot per file.

Evidence from the live run:

exported great-work: 34 chunks / superlinear: 13 / how-to-do-what-you-love: 13
Exported 60 chunks via the official AzureSearchExporter
Documents: 60
'exponential' -> 7 chunk(s) in superlinear; a nonsense term -> 0
Reassembled 13 chunks in order (indexes 0..12)
doc count before re-export: 60  |  after: 60  ->  no duplicates
snapshots (one per file): ['great-work', 'how-to-do-what-you-love', 'superlinear']

7. "No keys". Qualified everywhere (opener, Prerequisites, image alt text, "Make it yours"). The keyless claim is now explicitly scoped to the Azure services used here — AI Search and Azure OpenAI — and the production section states plainly that export_pipeline is not keyless: it needs LLAMA_CLOUD_API_KEY and embedding-provider credentials.

Optional — took all three

8. Tags. Fair point that I was minting near-duplicates of tags that already exist. Now using iq, identity, and indexes from the existing taxonomy; llamaindex is the only new tag, and tags.yaml is a one-line diff instead of five.

Final tags: retrieval, rag, azure-ai-search, indexes, llamaindex, identity, iq

9. Description. 241 → 170 chars.

10. Re-review gate. Clean run, assertions passing, sanitized outputs — covered above.

Validation

Gate	Result
Live keyless run, top to bottom	0 errors, all assertions passed
recipe-health-check.py	all gates pass
npx tsx validate-registry.ts	18 notebooks ✅
npm run build (astro check + build + pagefind)	18 pages ✅
Playwright a11y / SEO / callouts	10 passed

One note on local testing: site/public/notebook/media is a git symlink (mode 120000) that Windows checks out as a plain file, so the notebook-images spec fails locally for every recipe, not just this one. It resolves correctly on Linux, which is why the PR's build-and-preview check passes.

[nicholasdbrady]

Hypothesis: This is an intermediate tutorial for developers who want LlamaCloud parsing while retaining Azure AI Search as the governed retrieval store.

Success criterion: A reader can export parsed content keylessly for Azure services, verify scoped retrieval safely, and reuse the resulting index through inspectable retrieval primitives.

Confidence: High from the revised notebook, the exact pinned upstream sources, local Forgebook gates, and current Azure AI Search/OpenAI client behavior. I did not repeat the author's live Azure execution.

Verdict

Publish after fixes — 88/100. The earlier security-framing and destructive-lifecycle blockers are resolved. Three smaller issues still need correction before publication because they affect the reusable harness and product accuracy.

Axis	Score
Thesis	14/15
Opinionated defaults	9/10
When / What / How	8/10
Code cell discipline	9/10
Before/after evidence	13/15
Runnability	12/15
Scope	9/10
Dev-to-dev voice	5/5
Failure modes	5/5
Takeaway artifact	4/5

Required before publishing

• Required — Cell 26 (list_files / file_read) and scale guidance: Both helpers pass top=1000, which is a total-result ceiling, not complete-corpus pagination. Above 1,000 chunks, list_files silently returns incomplete counts; file_read can also truncate a large file, and its assertion can still pass because the expected count came from the same capped listing. This conflicts with “every file,” “whole file,” and the thousands-of-files guidance. Page until exhaustion (with a stable order/key), or detect the ceiling and fail explicitly; then add a test/assertion that crosses one page.

• Required — Prerequisites, Cell 3: The notebook says it calls the direct /openai/v1 endpoint, but the pinned AzureOpenAI(azure_endpoint=..., api_version=...) client routes embeddings through Azure's deployment-scoped /openai/deployments/{model}/embeddings?api-version=... path. Keep the RBAC distinction, which is useful, but describe this as the Azure OpenAI data-plane deployment endpoint rather than /openai/v1.

• Required — requirements.txt: The new comment still says the notebook requires “Python 3.13+,” while the revised notebook and pinned package correctly require >=3.11,<3.14. Change it to Python 3.11–3.13 so repository setup guidance does not contradict the recipe.

Recommended improvement

• Optional — Cells 7/14 and prerequisites: The clean run also depends on outbound access to three mutable paulgraham.com pages, but prerequisites name only Azure services. State that dependency and its failure mode, or commit a small sample under notebooks/data/<slug>/ so the teaching corpus remains reproducible if the pages change or reject automated requests.

What is working

• Praise: Step 4 now distinguishes server-side query scope from authorization and demonstrates the limitation with the same credential's unfiltered query. Keep that counterexample.
• Praise: The random per-run index name, create tracking, and teardown assertions remove the destructive shared-name behavior from the first revision.
• Praise: Exact source pinning, version verification, assertion-backed export checks, and the troubleshooting table make the lesson substantially more reproducible.

Mechanical checks

Check	Result	Evidence
Registry/authors/tags	Pass	npx --no-install tsx validate-registry.ts validated 18 notebooks
Media/data paths	Pass	Recipe health check; slug-scoped architecture image
Notebook structure	Pass	48 cells, 15 non-empty code cells, no leading H1 or site metadata
Site build	Pass	npm run build; 19 pages built
PR checks	Pass	Preview and CodeQL checks are green
Live execution	Not repeated	Author reports a clean assertion-backed Azure run; committed outputs are cleared

Fix order

1. Make corpus/file reads complete or explicitly fail on truncation.
2. Correct the Azure OpenAI route description and Python comment.
3. Document or stabilize the external sample corpus.

[nicholasdbrady]

Live Azure execution follow-up

I repeated the notebook against live Azure resources in westus3 using Entra ID only:

• Dedicated Basic Azure AI Search service
• text-embedding-3-small (DataZoneStandard) on the specified Foundry resource/project
• Python 3.12
• PR head c6f24ea
• Source notebook unchanged; only the documented EMBEDDING_MODEL configuration knob was set to openai:text-embedding-3-small

Result: top-to-bottom execution passed. All 15 code cells executed, there were 0 error outputs, and every assertion passed.

Evidence from the run:

Auth: Entra ID (keyless)
Setup OK — sink @ 798cac7, Python 3.12, embeddings return 1536 dims.
Exported 60 chunks via the official AzureSearchExporter.
Documents: 60
scope 'forgebook-demo': 5 hits | scope 'another-tenant': 0 hits
no filter at all: 5 hits
Reassembled 13 chunks in order (indexes 0..12)
doc count before re-export: 60 | after: 60 -> no duplicates
snapshots: ['great-work', 'how-to-do-what-you-love', 'superlinear']
Teardown complete — only indexes created by this run were deleted.

I also queried the Search service independently after execution: 0 indexes remain. This resolves my earlier static-only execution caveat and confirms the current notebook is runnable with the small embedding model. The three requested publication fixes in my re-review still stand because they concern pagination at scale and documentation accuracy rather than this 60-chunk happy-path run.