SystemsLtd Data Analytics

This repository holds two separate data projects built during the internship, both running on the same Databricks workspace but solving different problems.

Projects

agentic-dashboard-files

A natural language analytics dashboard for NYC taxi trip data. A user asks a question in plain English, an AI agent writes the SQL, runs it against a Databricks warehouse, and returns a chart with a short explanation. Full details, including the system design diagram, are in agentic-dashboard-files/agentic-dashboard.md.

flight-analytics-files

A flight tracking pipeline built on Databricks. It polls the OpenSky Network API on a schedule, ingests raw aircraft state data, and works it through a medallion architecture (bronze, silver, gold) into a set of aggregated tables covering aircraft activity, country level statistics, alerts, and emergency squawks. Full details are in flight-analytics-files/flight-analytics.md.

Both projects follow the same bronze, silver, gold pattern on the data side, but they differ in how data gets in. The taxi project loads a fixed historical dataset. The flight tracking project keeps pulling new data on a timer.

Repository layout

.
├── agentic-dashboard-files/    Next.js app, see agentic-dashboard.md for full detail
│   └── agentic-dashboard.md
└── flight-analytics.md

Setting up agentic-dashboard-files

1. cd agentic-dashboard-files
2. npm install
3. Copy .env.example to .env.local and fill in real values:
   • GOOGLE_GENERATIVE_AI_API_KEY from Google AI Studio
   • DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_WAREHOUSE_ID for your Databricks SQL warehouse
   • DATABRICKS_CATALOG and DATABRICKS_SCHEMA pointing at the taxi data (taxi_dw and gold by default)
4. npm run dev to run locally, or vercel --prod to deploy once the same environment variables are also set in the Vercel project settings

See agentic-dashboard-files/agentic-dashboard.md for the full architecture, guardrails, and known limitations.

Setting up flight-analytics-files

This project runs inside Databricks as a scheduled job rather than as a deployed web app.

1. Import the notebooks into your Databricks workspace
2. Create a job with tasks in order: fetch new data from OpenSky, write to bronze, clean into silver, aggregate into gold
3. Set the job trigger to run on an interval if you want it to keep collecting live data, or run it manually for a one off pull
4. Point each notebook at your own catalog and schema names if they differ from flight_tracking

This section is a starting outline. Fill in the exact notebook names, parameters, and any credentials this project needs, since those details live in the project itself.

Requirements

• A Databricks workspace with a running SQL warehouse
• Node.js 18 or later, for agentic-dashboard-files only
• A Google AI Studio API key, for agentic-dashboard-files only

License

See LICENSE for terms.