End-to-End Retail Analytics Pipeline

Data Engineering | Data Quality Audit | Business Intelligence Dashboard

Project Overview

This project simulates a retail data ecosystem for the Dubai market. It covers the data lifecycle from synthetic data generation and data-quality auditing to SQL-based performance analysis and an interactive stakeholder dashboard.

The goal is to identify data discrepancies and generate actionable insights into stock and sales performance across major Dubai events such as Ramadan, Eid, and the Dubai Shopping Festival.

---

Tech Stack

• Language: Python 3.x (Pandas, NumPy, Faker)
• Database: MySQL (Data Storage & Analysis)
• Configuration: Python-dotenv (Environment Variables)
• Visualization: Streamlit & Plotly
• Deployment: GitHub & Streamlit Cloud

---

The 4-Phase Architecture

1: Synthetic Data Architecture

Generated a dataset of 5,000+ transactions containing fields such as Transaction_ID, Product_Category, Store_Location, and Sales_Qty.

• Data Quality Scenarios: Introduced intentional irregularities, including negative stock values, duplicate records, and missing event labels, to test the data-quality pipeline.

2: Data Quality Audit & Cleaning

Built a custom audit process to programmatically identify, correct, and log data-quality issues.

• Resolved: 100+ duplicate records.
• Corrected: 51 instances of negative stock.
• Standardized: 1,000+ missing event labels to support consistent reporting.

3: SQL Intelligence & Stock Performance

Loaded the cleaned data into a relational MySQL database for KPI analysis.

• Stock Performance: Analyzed product turnover during major events such as Black Friday and Ramadan.
• Location Analysis: Compared average basket value across key locations, including Mall of the Emirates and Dubai Marina.

4: Stakeholder Reporting (Streamlit)

Deployed an interactive dashboard that allows users to explore sales and stock performance by event, location, and other business dimensions.

• Live Dashboard: [https://retail-data-pipeline-forensics-na94ekakbwdrg5e4rrcqmq.streamlit.app/]

---

Key Business Insights

Based on the simulated data:

• Stock-out Risk: The simulated turnover rate for “Smart Watch Ultra” exceeds the defined safety threshold by 70% during Black Friday.
• Location Strategy: Mall of the Emirates shows a 15% higher average basket value for luxury goods compared with the other analyzed locations.
• Event Impact: The Dubai Shopping Festival (DSF) generates the highest simulated sales volume for “Limited Edition Sneaker.”