GATE PYQ Practice Platform

This project is a modern Next.js application designed to process, structure, and render GATE Computer Science previous year questions (PYQs) and study notes from raw PDF materials.

The platform features an automated pipeline that extracts data from PDF documents, parses it into structured JSON, and serves it through a sleek, highly-responsive frontend built with Next.js and Tailwind CSS.

📊 Data Flow Architecture

The data pipeline transitions the content from unstructured PDFs to a rich, interactive web experience.

flowchart TD
    subgraph DC [Data Collection]
        A["Raw GATE PDFs<br>public/pdfs/"] --> B("DataLab Processing")
    end

    subgraph DP [Data Processing Pipeline]
        B --> C["DataLab JSON Exports<br>datalab_json/*.json"]
        A --> |PyMuPDF Extraction| D["scripts/parse_datalab.py"]
        C --> D
        D --> |Extracts Text, Math, Layout, Images| E["Formatted Unified JSON<br>data/formatted_all.json"]
    end

    subgraph FR [Frontend Rendering]
        E --> F["src/lib/data.ts<br>Enrichment & Formatting"]
        F --> G["page.tsx (Server Component)<br>Reads data"]
        G --> H["ServerMarkdown.tsx<br>Pre-renders Markdown & KaTeX"]
        H --> I["PracticeClient.tsx (Client Component)<br>Receives Static HTML"]
        I --> J["QuestionCard.tsx<br>Interactive UI"]
    end

    classDef default fill:#f8fafc,stroke:#94a3b8,stroke-width:2px,color:#0f172a;
    classDef script fill:#2563eb,stroke:#1e40af,stroke-width:2px,color:#ffffff,font-weight:bold;
    classDef data fill:#ea580c,stroke:#9a3412,stroke-width:2px,color:#ffffff,font-weight:bold;
    classDef frontend fill:#16a34a,stroke:#14532d,stroke-width:2px,color:#ffffff,font-weight:bold;
    
    class D script;
    class E,C,A data;
    class G,H,I,J,F frontend;

Loading

1. Data Collection

• The original source material consists of GATE CS textbooks in PDF format (Volumes 1, 2, and 3) stored in public/pdfs/. They are collected from the Go-Pdfs Gateoverflow repo releases.
• These PDFs are initially processed by DataLab, which performs OCR and structural analysis, converting the PDF pages into rich JSON documents containing bounding boxes, HTML fragments, math blocks (LaTeX), and base64-encoded images.
• These exported JSON files are placed in the datalab_json/ directory.

2. Data Processing

• The core processing engine is scripts/parse_datalab.py.
• This Python script merges the structural data from datalab_json/ and layout features (like embedded image coordinates and explanation URLs) directly from the raw PDFs using PyMuPDF.
• It intelligently pieces together questions, their options, and correct answers from tables. It separates study notes from practice questions and embeds necessary diagram images.
• The output is a single, highly-structured database file: data/formatted_all.json.

3. Frontend Rendering

• The Next.js frontend imports data/formatted_all.json through src/lib/data.ts.
• data.ts performs on-the-fly enrichment: it extracts clean subtopic headers from question text, structures plaintext study notes into proper Markdown hierarchies, and prepares the data for rendering.
• During build time (Static Site Generation), the Server Component page.tsx reads this data and passes it to ServerMarkdown.tsx.
• ServerMarkdown.tsx uses react-markdown, remark-math, and rehype-katex on the server side to compile all complex mathematical formulas and tables into pure, static HTML.
• The interactive UI components (PracticeClient.tsx and QuestionCard.tsx) simply receive and display this pre-rendered HTML. This offloads heavy computation from the client's browser, resulting in lightning-fast mobile performance and lightweight JavaScript bundles for offline PWA viewing.

---

🚀 How to Add New GATE PDFs (Step-by-Step)

If you have future releases of GATE PDFs or new volumes and want to integrate them into the platform, follow these steps:

Step 1: Add the Raw PDFs

Place the new PDF files inside the public/pdfs/ directory. Ensure they follow a consistent naming convention (e.g., filter1_volume4.pdf).

Step 2: Generate DataLab JSON

Upload the new PDFs to your DataLab workspace for processing. Once DataLab finishes OCR and layout analysis, export the results as JSON and place them in the datalab_json/ directory (e.g., datalab-output-filter1_volume4.pdf.json).

Step 3: Update the Python Script Configuration

Open scripts/parse_datalab.py and locate the volumes_config array in the __main__ block (around line 340). Add a new entry for your volume:

volumes_config = [
    # ... existing volumes ...
    {
        "id": "volume4", 
        "name": "Volume 4", 
        "json": "datalab-output-filter1_volume4.pdf.json", 
        "pdf": "public/pdfs/filter1_volume4.pdf"
    }
]

Step 4: Run the Parsing Script

Activate the Python virtual environment and run the parsing script to generate the updated formatted_all.json database.

# Activate virtual environment (Windows)
.\venv\Scripts\activate

# Or on macOS/Linux:
# source venv/bin/activate

# Install dependencies if you haven't already
pip install -r scripts/requirements.txt

# Run the parser
python scripts/parse_datalab.py

Note: The script will output "Total Data parsed successfully" along with the number of questions extracted.

Step 5: Run Data Sanitization (Fix KaTeX Errors)

During the Python parsing pipeline, certain LaTeX commands in the raw data (like \frac, \text, \forall) or HTML entities (like >, <) can be misinterpreted as control characters (e.g., \f becoming a form-feed character) or left unescaped. Furthermore, the parser sometimes accidentally splits options (A), (B) from inside active math blocks, or misinterprets literal $ signs, causing KaTeX Parse Errors in the frontend.

To fix this, always run the sanitization script after generating the JSON:

python scripts/fix_json.py

This script traverses the database, reconstructs improperly split string boundaries, escapes raw characters, and repairs all corrupted math symbols.

Step 6: Start the Frontend

Start the Next.js development server to verify the new content is rendering correctly.

npm run dev
# or
yarn dev
# or
pnpm dev

Open http://localhost:3000 to browse the new volumes and verify the rendering of mathematical equations and diagrams.