Skip to content

About

CloudDesk: RAG support assistant with source citations (LangChain, FAISS, FastAPI, Docker)

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

CloudDesk AI Customer Support Knowledge Assistant

A production-style Retrieval-Augmented Generation (RAG) system that answers customer-support questions for a fictional SaaS company ("CloudDesk") using only its own knowledge base — never the LLM's general knowledge. Built as a portfolio project demonstrating the full RAG engineering lifecycle: ingestion, chunking, embeddings, vector search, grounded generation, citations, hallucination guardrails, and measured evaluation.

Every metric in this README comes from a script in this repo, actually executed against this project's own data. See DEVELOPMENT_LOG.md for the full phase-by-phase build log, including two real bugs found and fixed along the way.


1. Project Overview

The assistant answers questions like "How many days do I have to request a refund?" by retrieving the relevant passages from CloudDesk's real (fictional) documentation, then asking an LLM to answer grounded strictly in that retrieved context, with citations pointing back to the actual source document and section used — not invented by the model.

2. Business Problem

Customer-support teams need consistent, policy-accurate answers. A generic LLM chatbot will confidently answer using general internet knowledge, which is exactly wrong for company-specific policies (refund windows, plan limits, escalation SLAs) that don't exist anywhere on the public internet. This project's core constraint, enforced end-to-end, is:

Groundedness > Creativity. If the knowledge base doesn't contain the answer, the assistant must say so — never guess.

3. Architecture

Architecture diagram

Knowledge Base (18 .md docs) --> Ingestion --> Chunking --> Embeddings --> FAISS Index
                                                                                |
User Question --> Query Embedding --> Retriever (Top-K) --> Guardrail Check ---+
                                                                    |
                                                     pass:   Prompt Construction --> LLM --> Grounded Answer + Citations
                                                     fail:   "Insufficient information" (no LLM call)

4. Tech Stack

Layer Choice Why
Language Python 3.11+ Project requirement
RAG framework LangChain 0.3 Standard tooling for FAISS/HF embeddings/prompt orchestration
Embeddings sentence-transformers/all-MiniLM-L6-v2 Evidence-based — see §7
Vector DB FAISS (faiss-cpu) Local, free, persistent, no infra
LLM Gemini (gemini-flash-latest), OpenAI (gpt-4o-mini) supported Configurable via LLM_PROVIDER
Frontend Streamlit Fast, interactive demo UI
Evaluation Custom LLM-as-judge (Recall@K/Precision@K + faithfulness/relevance) See §12 for why not RAGAS
Containerization Docker + docker-compose Reproducible deployment

5. Knowledge Base

18 markdown documents for "CloudDesk," a fictional SaaS helpdesk product, under data/raw/:

  • Policies (7): Refund (v1 deprecated + v2 current — a deliberate versioned conflict: the window shrank from 14 days to 7 days with a new 20%-usage cap), Cancellation, Payment, Account, Escalation, Privacy.
  • Product (4): Basic / Professional / Enterprise plans, Features & Usage Limits (comparison table).
  • FAQ (1): 18 Q&A pairs.
  • Troubleshooting (1): login, payment failure, account locked, connection issues, subscription issues.
  • Procedures (5, internal/agent-facing): refund, complaint escalation, technical escalation, customer verification, account recovery.

Each document has YAML frontmatter (title, document_type, category, version, status, last_updated) so citations and version-conflict handling are metadata-driven, not guessed. Topical overlap is deliberate (refund info appears in 4+ documents) to stress-test retrieval precision.

6. RAG Pipeline

src/ingestion/loader.py splits each markdown file's frontmatter from its body, then splits the body into one Document per ## section (not per file) — 98 section-level documents total — so citations can be as precise as "Refund Policy v2 > Refund Eligibility Window."

7. Chunking Strategy — Evidence, Not Assumption

Compared fixed-size vs. recursive splitting at 5 sizes each (150/300/500/800/1200 chars) against the 62-question evaluation set's expected_sources (script: scripts/run_chunking_experiment.py, full results: data/processed/chunking_experiments/results.csv):

config Recall@5 Precision@5 num_chunks
recursive_300_45 (chosen) 0.9490 0.2408 116
recursive_500_75 / 800_120 / 1200_180 0.9490 0.2408 102 / 98 / 98
fixed_300_45 / 500_75 / 800_120 / 1200_180 0.9490 0.2408 115 / 102 / 98 / 98
fixed_150_25 0.9150 0.2286 195
recursive_150_25 0.8946 0.2286 206

Finding: 300-1200 all tie because this KB's sections are naturally short (median 231 chars); 150 measurably fragments sections and hurts recall. Chose recursive, size=300, overlap=45 — ties for best recall with the smallest/most citation-precise chunks.

A real bug this surfaced: the title/section header was originally embedded directly into the chunkable text. For a section long enough to split, the splitter's greedy merge sometimes stranded that short header as its own near-empty chunk — which caused the LLM to (correctly, given the fragmented context it received) decline to answer an easily-answerable question. Fixed by removing the header from chunkable content entirely (it's redundant with metadata and already re-added at prompt-construction time). Recall dropped by exactly 1 point across every config after the fix (0.959→0.949) — an honest, worthwhile tradeoff for eliminating a real correctness defect. Full account in DEVELOPMENT_LOG.md.

8. Embedding Model — Evidence, Not Popularity

Compared 3 HuggingFace models on the winning chunk config (scripts/run_embedding_experiment.py, data/processed/embedding_experiments/results.csv):

model dim Recall@5 corpus embed time
all-MiniLM-L6-v2 (chosen) 384 0.9490 0.77s
BAAI/bge-small-en-v1.5 384 0.9218 1.40s
all-mpnet-base-v2 768 0.9116 4.78s

The smallest, fastest model won outright on this short-sentence, single-domain corpus — a genuine example of not choosing a model just because it's popular or larger. (Likely explanation for BGE's relative underperformance: it typically needs an instruction-style query prefix not applied here — noted as a limitation, not tuned away, to keep the comparison a fair default-usage test.)

9. Vector Store

src/retrieval/vector_store.py: build_index / save_index / load_index / load_or_build_index wrap FAISS's save_local/load_local. Verified for real: building the production index, restarting, and reloading without --rebuild reused the persisted index (116 vectors) without re-embedding.

10. Retrieval Strategy — Baseline vs. "Improved," Honestly

Top-K sweep (scripts/run_retrieval_experiment.py):

K Recall@K Precision@K
3 0.8912 0.3673
5 (chosen) 0.9490 0.2408
10 0.9898 0.1265

Full ablation of retrieval-improvement techniques, tested individually (not just bundled):

strategy Recall@5 Precision@5
baseline_topk (chosen) 0.9490 0.2408
topk_plus_version_pref 0.9388 0.2367
threshold_only 0.9286 0.2592
mmr_only 0.8571 0.2082
full_combo (mmr+threshold+version_pref) 0.8231 0.2265

Honest finding: none of the "improvements" beat plain baseline Top-K on Recall@5 here. MMR trades relevance for diversity these single-topic questions don't need. Naively dropping deprecated-version chunks at the retrieval layer actively hurt recall on a question that legitimately needs to reference the old policy to explain what changed. Decision: retrieval stays baseline Top-K@5; the similarity threshold is repurposed as a guardrail signal (not a retrieval filter — see §11), and version-conflict handling moved to the prompt layer instead (§11), where it doesn't cost any recall.

11. Guardrails & Document-Versioning Strategy

Implemented in src/generation/rag_pipeline.py::answer_question:

  1. Relevant question → retrieve → LLM → citations built from what was actually retrieved.
  2. No relevant context / out-of-domain → if the top retrieval score is below RETRIEVAL_SCORE_THRESHOLD (default 0.35), short-circuit to "I couldn't find sufficient information in the provided knowledge base to answer this question" without calling the LLM at all. A second defense layer: the system prompt instructs the LLM to self-decline via an INSUFFICIENT_CONTEXT sentinel if a borderline question slips past the score threshold.
  3. Ambiguous question → the system prompt instructs the LLM to ask a short clarifying question rather than guess.
  4. Conflicting/versioned documents → rather than dropping the deprecated version at retrieval time (shown in §10 to hurt recall), both versions are retrieved and passed to the LLM with their status: current/status: deprecated metadata visible; the system prompt instructs it to always answer from the current version and only reference the deprecated one if the question is specifically about what changed.

Citations (src/generation/citations.py) are built only from retrieved-chunk metadata — never parsed from the LLM's text — so a citation can never name a source that wasn't actually used.

12. Evaluation

Evaluation dataset (data/evaluation/eval_dataset.json): 62 hand-written questions across 8 categories (easy, semantic, multi-document, detail-specific, ambiguous, out-of-domain, no-answer-exists, conflicting/similar-document), each with an expected answer and expected source document(s).

Retrieval evaluation: Recall@K / Precision@K, see §10.

Generation evaluation: custom LLM-as-judge (src/evaluation/generation_metrics.py) scoring faithfulness, answer relevance, and context relevance per answer, using the same LLM as judge. Why not RAGAS: attempted per the project plan, but the project's Gemini-based setup and RAGAS's OpenAI-centric default judge wiring created integration friction not worth burning significant time on (explicitly permitted by this project's own rules); a custom judge with a clearly-specified rubric was faster to get correct and verify.

Status: partial, in progress (blocked on a real infrastructure limit, not fabricated). The Gemini free tier caps this project's key at roughly 20 generateContent calls/day per model, which was exhausted during setup/debugging (see DEVELOPMENT_LOG.md for the full account, including discovering gemini-2.5-flash had been deprecated mid-project). 1 of 62 questions has been fully generation-evaluated end-to-end:

id category faithfulness answer_relevance context_relevance
E001 easy 1.0 1.0 1.0

This single real result already demonstrates the pipeline and the LLM-judge scorer work correctly (an earlier version of this same question, before a chunking bug fix, correctly scored 0.0/0.0 when the assistant wrongly declined to answer — see DEVELOPMENT_LOG.md). The remaining 61 questions will be filled in with real, non-fabricated scores once the daily quota resets (the evaluation script caches completed rows, so resuming replays none of the finished work — see scripts/run_generation_eval.py).

13. Error Analysis

scripts/run_error_analysis.py classifies every generation-eval result into failure types (Retrieval Failure, Missing Context, Wrong Citation, Guardrail Failure/Over-Triggering, Possible Hallucination, Incomplete/Irrelevant Answer) with root cause and suggested fix, from src/evaluation/error_analysis.py.

Status: pipeline verified, full report pending §12's data. Run for real against the current 1-row partial dataset: 1 row evaluated, 0 issues flagged (correct — E001 scored perfectly). This confirms the classifier code path itself is correct; a meaningful failure-type breakdown needs the full 62-row batch from §12.

14. Testing

pytest tests/ -v → 37 passed, ~8s. The LLM is mocked everywhere (per project rule, to avoid paid API calls in the test suite); embeddings/FAISS are real. Covers ingestion metadata preservation, chunking/chunk-ID assignment, retrieval ranking and version-preference re-ranking, citation de-duplication, all RAG-pipeline guardrail branches, Recall/Precision edge cases, LLM response-content normalization (regression test for a real bug, see DEVELOPMENT_LOG.md), and a full end-to-end integration test against the real 18-document knowledge base.

15. Streamlit Application

app/streamlit_app.py — chat interface, sidebar controls (Top-K, relevance threshold, "show retrieved context" debug toggle), per-turn source citations, graceful degradation when no LLM key is configured (retrieval/citations still explorable). Verified live in-browser: index loads, chat works, and — before a key was configured — correctly showed a "not configured" message instead of crashing.

16. Local Installation

cd "RAG AI Knowledge Assistant"
python -m venv .venv
.venv/Scripts/activate        # Windows; use `source .venv/bin/activate` on Linux/Mac
pip install -r requirements.txt
cp .env.example .env          # then fill in GEMINI_API_KEY (or OPENAI_API_KEY)
python scripts/build_index.py
streamlit run app/streamlit_app.py

16b. Deploying to Streamlit Community Cloud

  1. Push this repo to GitHub (see §16c).
  2. On share.streamlit.io, create a new app pointing at this repo, branch main, main file path app/streamlit_app.py. Streamlit Cloud auto-detects requirements.txt at the repo root — no extra config needed.
  3. In the app's Settings > Secrets, paste the contents of .streamlit/secrets.toml.example with your real API key. (Streamlit Cloud secrets aren't exposed as OS environment variables the way a local .env is — src/utils/config.py handles this by falling back to st.secrets automatically.)
  4. First load will take ~20-30s while it downloads the embedding model and builds the FAISS index from data/raw/ (which is committed); subsequent loads within the same running container are fast. A redeploy rebuilds the index again since data/processed/ isn't committed (intentional — it's a build artifact of the source markdown, not source itself).

16c. Pushing to GitHub

git add -A
git commit -m "Initial commit: CloudDesk RAG knowledge assistant"
git remote add origin https://github.com/<your-username>/<your-repo>.git
git push -u origin main

(Nothing has been committed yet in this project — do this once you're ready, or ask me to do it for you.)

17. Docker Installation

docker compose up --build

Serves the app at http://localhost:8501. .env is read via env_file in docker-compose.yml; the FAISS index and the HuggingFace embedding-model cache persist across restarts via named volumes. (Build not yet verified in this environment — Docker Desktop's engine was not running; see DEVELOPMENT_LOG.md.)

18. Environment Variables

See .env.example — LLM_PROVIDER, GEMINI_API_KEY/OPENAI_API_KEY, EMBEDDING_MODEL, FAISS_INDEX_DIR, RETRIEVAL_TOP_K, RETRIEVAL_SCORE_THRESHOLD.

19. Project Structure

RAG AI Knowledge Assistant/
├── data/{raw,processed,evaluation}/
├── docs/architecture_diagram.svg
├── src/{ingestion,chunking,embeddings,retrieval,generation,evaluation,utils}/
├── app/streamlit_app.py
├── tests/
├── scripts/            # build_index, run_*_experiment, run_generation_eval, run_error_analysis
├── Dockerfile, docker-compose.yml, requirements*.txt, .env.example, .gitignore
└── DEVELOPMENT_LOG.md   # full phase-by-phase build log

20. Future Improvements

  • Hybrid dense + sparse (BM25) retrieval, since some questions hinge on exact policy terms.
  • A reranker stage (cross-encoder) now that the retrieval-improvement ablation shows plain Top-K's ceiling on this corpus.
  • Query rewriting for genuinely ambiguous questions before retrieval, not just at generation time.
  • RAGAS integration once the ecosystem's non-OpenAI judge support matures further.

21. Limitations

  • Fictional, small (18-document) knowledge base — real deployments need continuous ingestion pipelines for a much larger, changing corpus.
  • The custom LLM-as-judge evaluator is a defensible but not industry-standard-certified metric.
  • Free-tier LLM API quotas (see DEVELOPMENT_LOG.md) constrain how large a batch evaluation can run per day without a paid tier.

22. Lessons Learned

Three real, non-obvious bugs were found and fixed by testing rather than assuming: a LangChain FAISS relevance-score formula mismatch, an LLM response occasionally returning content as a list of duplicate parts (silently defeating naive sentinel string matching), and a chunking defect that stranded section headers as near-empty chunks. All three were caught because this project's own rule is to verify every claim against real execution — see DEVELOPMENT_LOG.md for the full account, including root-cause analysis for each.


Portfolio Materials

Resume Bullets

  • Designed and built an end-to-end RAG (Retrieval-Augmented Generation) customer-support assistant in Python/LangChain/FAISS, running controlled A/B experiments across 10 chunking configurations and 3 HuggingFace embedding models to select a production configuration by measured Recall@K/Precision@K rather than default assumptions.
  • Implemented hallucination guardrails and a metadata-driven citation system for a grounded-QA pipeline; found and root-caused 3 real bugs during development (a vector-store relevance-score formula mismatch, a multi-part LLM response format that silently broke sentinel-based safety checks, and a text-chunking defect that fragmented source content) via direct reproduction and unit-test regression coverage, not guesswork.
  • Built a 37-test automated suite (LLM mocked, embeddings/retrieval real) and a Streamlit demo app with observability into retrieval scores and citations, containerized with Docker.

GitHub Description

AI Customer Support Knowledge Assistant — a RAG system (LangChain + FAISS + Gemini/OpenAI + Streamlit) that answers questions grounded only in a company knowledge base, with measured chunking/embedding/retrieval experiments, hallucination guardrails, and citation tracing.

LinkedIn Project Description

I built an AI Customer Support Knowledge Assistant — a Retrieval-Augmented Generation system that answers customer questions using only a company's own documentation, never the LLM's general knowledge. Rather than guessing at chunk sizes or embedding models, I ran controlled experiments (10 chunking configs, 3 embedding models, a Top-K sweep, and a 5-way retrieval- strategy ablation) and picked every configuration by measured Recall/Precision. Along the way I found and fixed three real bugs — a vector-store scoring-formula mismatch, an LLM response format that silently defeated a safety check, and a chunking defect that fragmented source text — each caught by testing against real output rather than assuming the code was correct. The system includes hallucination guardrails, metadata-driven citations, a Streamlit demo, a 37-test automated suite, and Docker packaging.

60-Second Interview Explanation

"I built a RAG-based customer support assistant for a fictional SaaS company. It answers questions using only the company's own policy and product docs — refund windows, plan limits, escalation SLAs — and explicitly says so when it can't find an answer, instead of guessing from general knowledge. What I'm most proud of isn't the happy path, it's the engineering process: I ran real experiments to pick the chunk size, embedding model, and retrieval strategy instead of assuming defaults, and two of those experiments actually changed my decisions — one, a 'retrieval improvement' I expected to help (MMR plus a similarity threshold) actually hurt recall, so I kept the simpler baseline. I also found and fixed three real bugs by testing against actual output: a scoring formula mismatch in the vector store, an LLM response format that silently broke a safety check, and a chunking bug that fragmented a policy document enough that the assistant wrongly refused to answer an easy question. All three were caught by verification, not assumed away."

5-Minute Technical Deep Dive

Architecture: Markdown knowledge base → section-level ingestion (metadata from YAML frontmatter, not inferred) → recursive chunking (300 chars/45 overlap, chosen by experiment) → all-MiniLM-L6-v2 embeddings → FAISS (cosine, persisted to disk) → Top-K=5 similarity retrieval → a guardrail check on the top relevance score → prompt construction (numbered context passages tagged with status: current/deprecated) → Gemini (gemini-flash-latest, temperature=0) → citations built from the same metadata the LLM saw, never parsed from its text.

Why these choices: I compared 10 chunk configs and found this KB's sections are naturally short (median 231 chars), so 300-1200 char chunks all tied — but a 150-char config measurably hurt recall by fragmenting sections, which told me why the tie existed rather than just that it did. For embeddings, the smallest/fastest model beat two larger ones, including a retrieval-specialized model (BGE) — a concrete example of validating rather than assuming a 'better' model is actually better for a specific corpus. For retrieval, I ran a 5-way ablation (baseline, version-preference re-ranking, similarity threshold, MMR, and the full combo) and baseline Top-K won outright — MMR's diversity tradeoff doesn't help single-topic factual questions, and naively dropping a deprecated policy version at the retrieval layer actively hurt recall on a question that legitimately needed to reference the old version to explain what changed. That finding moved version-conflict handling from the retrieval layer to the prompt layer instead, where both versions reach the LLM labeled by status.

Guardrails: a relevance-score floor short-circuits before ever calling the LLM for low-relevance/out-of-domain questions (cheaper and more reliable than trusting the model to self-decline), backed by a second-layer sentinel the LLM emits if it should decline anyway.

Bugs I'd highlight if asked: the vector store's built-in "relevance score" conversion assumed the wrong distance metric for a cosine-configured FAISS index (verified by a direct 2-document test before trusting the fix); the LLM's SDK sometimes returned a list of duplicate response parts instead of a string, which silently defeated a .startswith() safety check until I noticed a generation-eval score of 0.0 and traced it back; and a chunking bug where a short section header got orphaned as its own near-empty chunk, which I only found by asking why the LLM had wrongly declined an easy question instead of accepting the guardrail's behavior at face value.

Potential Interview Questions & Answers

What is RAG, and why use it instead of fine-tuning? RAG retrieves relevant documents at query time and includes them in the prompt, rather than baking knowledge into model weights. It's cheaper to update (edit a document vs. retrain), more auditable (you can see exactly what the model was given), and reduces hallucination since the model has real context to ground on. Fine-tuning is better for teaching a style or format, not for injecting facts that change often — like a refund policy that changed from 14 to 7 days.

Why FAISS over other vector databases? Local, free, no infrastructure to manage, and fast for a corpus this size (hundreds of vectors). The architecture (a thin vector_store.py wrapper) is deliberately swappable — a managed vector DB would be a drop-in replacement for FAISS if this needed to scale to millions of vectors or multi-user concurrent writes.

How did you choose your chunk size? I didn't guess — I ran Recall@5/Precision@5 across 10 configs (fixed and recursive, 150-1200 chars) against a 62-question evaluation set with known expected sources. 300-1200 tied; 150 measurably hurt recall by fragmenting sections. I picked 300 for the best citation granularity among the tied options.

How did you reduce hallucination? Three layers: (1) a relevance-score floor that refuses to call the LLM at all when nothing retrieved is actually relevant, (2) a system prompt that instructs context-only answers and an exact sentinel string to emit when it can't answer, checked by application code before showing anything to the user, and (3) citations built exclusively from retrieved-chunk metadata, so the model can never claim a source it wasn't actually given.

What happens when no relevant document exists? The top retrieval score is compared against a threshold before the LLM is even called; below threshold, the app returns a fixed "insufficient information" message with no citations and no LLM cost.

How would you scale this system? Swap FAISS for a managed vector DB (Pinecone/Weaviate/pgvector) behind the same vector_store.py interface, move ingestion to an async pipeline triggered by document changes rather than a full rebuild, and add caching for repeated/similar queries. The RAG logic itself (rag_pipeline.py) wouldn't need to change.

RAG vs. fine-tuning, dense vs. sparse retrieval, hybrid search, reranking, MMR, query rewriting? — see §10/§20 above: this project measured MMR directly and found it hurt recall here (it trades relevance for diversity, which single-topic factual questions don't need) — a concrete, evidence-backed answer rather than a textbook one.

Final Definition-of-Done Checklist

Item Status
Knowledge base (18 docs, metadata, deliberate overlap + version conflict) ✅ Done
Ingestion, metadata preservation ✅ Done, tested
Chunking implemented + experiments (10 configs) ✅ Done, real numbers
Embedding model selected by evidence (3 models compared) ✅ Done, real numbers
FAISS vector store: build/save/load/rebuild ✅ Done, tested
Retrieval implemented + Top-K experiment + baseline-vs-improved ablation ✅ Done, real numbers
RAG generation + citations from metadata ✅ Done, tested
Hallucination guardrails (no-context, out-of-domain, ambiguous, conflicting) ✅ Done, tested
Evaluation dataset (62 questions, 8 categories) ✅ Done
Retrieval evaluation (Recall@K/Precision@K) ✅ Done, real numbers
Generation evaluation (LLM-as-judge: faithfulness/relevance) 🟡 Partial (1/62 — blocked on Gemini free-tier daily quota; pipeline verified correct)
Error analysis 🟡 Partial (pipeline verified on available data; full breakdown pending above)
Automated tests ✅ Done — 37/37 passing
Streamlit application ✅ Done, verified live in-browser
Docker configuration 🟡 Written and path-verified; docker build not yet executed (Docker Desktop not running in this environment)
README + architecture diagram ✅ Done (this document)
Resume/GitHub/LinkedIn/interview prep ✅ Done (above)

Two open items, both blocked on things outside this session's control, not on unfinished engineering work:

  1. Generation evaluation batch (61 remaining questions) — needs the Gemini daily quota to reset, billing enabled on the key, or an OpenAI key. Resuming re-uses the cache; no repeated work.
  2. docker build verification — needs Docker Desktop running on this machine.

About

CloudDesk: RAG support assistant with source citations (LangChain, FAISS, FastAPI, Docker)

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Contributors

Languages