This application helps you learn Azure Data Engineering from basic to advanced levels. It covers many tools used by data engineers, such as SQL, PySpark, Kafka, Databricks, Snowflake, and Airflow. You will work on over 15 projects that use Azure services like Azure Data Factory, Synapse, and Event Hubs. This guide also introduces you to cloud platforms like Google Cloud Platform (GCP) and modern data formats like Delta Lake, Iceberg, and Hudi.
The projects help you understand concepts like real-time streaming, the Medallion architecture, and cloud data warehousing. The labs provide hands-on experience with practical data engineering tasks.
- Azure Data Engineering basics through to advanced skills
- Using SQL for data manipulation and queries
- Writing data processing scripts with PySpark
- Managing data streams with Apache Kafka
- Working with Databricks for big data analytics
- Storing and querying data in Snowflake
- Scheduling workflows using Airflow
- Using Azure Data Factory (ADF) to move and transform data
- Running jobs on Azure Synapse Analytics
- Managing data formats like Delta Lake, Iceberg, and Hudi
- Building scalable data pipelines on cloud platforms like Azure and GCP
Make sure your computer meets these minimum requirements before starting:
- Operating System: Windows 10 or later
- Processor: 2.0 GHz dual-core or better
- Memory: 8 GB RAM or higher recommended
- Storage: At least 5 GB free disk space
- Internet connection for downloading and updating files
You do not need coding experience to use this guide. The projects and instructions guide you through every step.
Follow these steps to download and set up the application on your Windows computer.
Click the button below to visit the project's GitHub page, where you will find all files needed to run the guide.
On the GitHub page, look for the green Code button near the top right. Click it and select Download ZIP. This downloads all the project files to your computer in a compressed folder.
Once the ZIP file finishes downloading:
- Open the file location in your File Explorer.
- Right-click the ZIP file and choose Extract All.
- Select a folder where you want to save the files and click Extract.
The files now appear in the chosen folder, ready to use.
Inside the extracted folder, find a file named README.html or index.html. Double-click to open it in your web browser.
This file contains the full guide with instructions and links to projects. It acts as your main manual.
Each section of the guide explains a concept or project with detailed steps. You will learn by doing, working through examples and labs that build on each other.
Each project focuses on a different topic or tool. You will find folders labeled by technology like SQL, PySpark, or Kafka.
To run code or notebooks included in the projects, you need to install some software:
- Python: Many projects use Python and PySpark. Download it from python.org.
- VS Code or another code editor helps you open and edit files.
- Azure Data Factory and Databricks accounts if you want to try cloud services (free trials are available).
The guide explains these tools and how to set them up step by step.
This repository teaches you to work with the following components:
- Azure Data Factory (ADF) for building ETL pipelines
- Azure Synapse Analytics for big data querying
- Apache Kafka for streaming data
- Apache Spark and PySpark for processing large datasets
- Databricks platform
- Snowflake cloud data warehouse
- Apache Airflow for automating workflows
- Modern open-source table formats: Delta Lake, Iceberg, Hudi
- Google Cloud Platform (GCP) BigQuery
- MongoDB and Cassandra databases
The guide introduces these technologies clearly and without assuming prior knowledge.
The downloaded folder contains:
- README.html or index.html - Main guide overview and instructions
- Projects/ - Individual project folders organized by topics
- Datasets/ - Sample data files to use in the projects
- Scripts/ - Python and Spark scripts for data processing
- Notebooks/ - Jupyter notebooks for interactive lessons
Each project folder includes a README explaining its goal and how to run it.
- If files do not open, ensure you extracted the ZIP correctly.
- Check you have the latest version of Python installed.
- Use the command prompt to run Python scripts if double-clicking files does not work.
- Restart your computer if you face unexpected errors.
- Verify your internet connection for cloud-related labs.
Return to the main download page anytime using this link:
Visit the page to download the ZIP file and follow the setup steps described above.
The guide points you to official documentation for each tool it covers. It also includes tips for learning more advanced topics after you complete the projects.
If you want to practice using cloud platforms, create free trials on Azure and GCP. The projects give a practical path to build your skills through data engineering tasks.