Data Pipeline Dashboard

Overview

This project implements a complete data pipeline for ingesting, processing, and visualizing temperature data collected from IoT sensors.

The pipeline reads raw data from a CSV file, performs data cleaning and transformation, and loads it into a PostgreSQL database running in a Docker container. Analytical SQL views are created to structure the data for efficient querying. Finally, a Streamlit dashboard provides interactive visualizations for exploring temperature patterns and detecting heat anomalies.

Architecture

The project follows a simple and scalable pipeline architecture:

CSV → Data Processing (Pandas) → PostgreSQL → SQL Views → Streamlit Dashboard

Technologies Used

• Python
• Pandas
• PostgreSQL
• SQLAlchemy
• Streamlit
• Plotly
• Docker

Prerequisites

• Docker
• Python 3.9+
• Dataset downloaded IOT Temperature Data

Setup Instructions

1. Clone the repository

git clone https://github.com/RobisonTorres/Data_Pipeline_Dashboard.git
cd Data_Pipeline_Dashboard

2. Start PostgreSQL with Docker

docker-compose up -d --build

3. Create and activate virtual environment

Windows

python -m venv venv
venv\Scripts\activate

4. Install dependencies

pip install -r requirements.txt

5. Prepare dataset

Place the CSV file inside:

/data/IOT-temp.csv

6. Run data pipeline (ETL + Views)

cd src
python process_data.py

7. Run the dashboard

streamlit run main.py

Database Views

1. Heat Peaks Throughout the Day

• Displays hourly average temperatures
• Highlights the hottest hour per environment (indoor/outdoor)

---

2. Evolution of Daily Average Temperature

• Shows temperature trends over time
• Displays maximum thermal amplitude per location

---

3. Heat Alerts

• Identifies temperature spikes above 32°C
• Highlights critical alerts above 36°C

---

License

This project is licensed under the MIT License.