SmartDataCleaner 🚀

A powerful, user-friendly, and lightning-fast web application built with Streamlit for intelligent data cleaning and preprocessing. Upload your messy datasets and let SmartDataCleaner handle the heavy lifting!

✨ Features

• ⚡ High-Performance Pipeline Caching

  • Powered by Streamlit's @st.cache_data to cache heavy operations like file reading, data type conversion, and imputation.
  • Interactive cleaning settings (like removing duplicates or dropping features) apply instantly without re-running the entire pipeline.

• 📂 Robust File Reader

  • Supports CSV, Excel (.xlsx, .xls), JSON, and TXT formats.
  • Auto-Encoding Fallback: Automatically tries multiple encodings (utf-8, latin-1, cp1256, utf-16) to read files with special/foreign characters (e.g., Persian/Arabic) without crashing.
  • Auto-Separator Detection: Automatically detects delimiter characters (commas, tabs, semicolons) for .txt files using a smart python engine.

• 🎛️ Interactive Sidebar Configuration

  • Fine-tune all data cleaning settings dynamically from the sidebar.
  • Adjust imputation algorithms, outlier IQR multipliers, correlation thresholds, and feature removal parameters on the fly.

• 🧩 Flexible Missing Value Imputation

  • Offers multiple imputation strategies: KNN, Mean, Median, and Mode for numeric columns.
  • Standardized Scaling: Automatically scales numeric data using StandardScaler before running KNN imputation to ensure correct Euclidean distance calculations, then inverse-scales it back.
  • Automatically handles categorical columns with Mode Imputation and datetime columns with Forward Fill.
  • Drop columns and rows containing 90%+ missing values automatically.
  • Visual comparison of data before and after imputation with interactive histograms.

• 📌 Outlier Detection & Treatment

  • IQR Method for numeric columns with a configurable multiplier slider (1.0 to 3.0, default 1.5).
  • Frequency-Based Detection for categorical columns (flags rare categories <1%).
  • Outlier Treatment: Choose between None (just visualize), Drop rows containing outliers, or Cap values (clips values to the lower/upper IQR bounds).
  • Scatter plot visualization of detected outliers with Normal/Outlier classification.

• 🧬 Smart Data Type Standardization

  • Detects mixed/inconsistent data types and standardizes them.
  • Cleans boolean columns intelligently (standardizing true/false string representations and keeping invalid ones as nullable NA).
  • Auto-converts to dominant types with data loss reporting.
  • Consistent data indexing starting from 1.

• 📊 Correlation & Duplicate Removal

  • Pearson correlation heatmap visualization.
  • Filter and remove highly correlated feature pairs based on a configurable threshold.
  • Detect and remove duplicate rows with persistent states.

• 🧾 Dynamic Data Quality Scoreboard

  • Comprehensive quality score (0-100) based on missing data, duplicates, schema issues, outliers, and feature redundancy.
  • Dynamic Updating: Watch your quality score increase in real-time as you check boxes to remove duplicates, highly correlated features, or treat outliers.

---

🚀 Quick Start

Prerequisites

• Python 3.7 or higher
• pip (Python package manager) or uv

Installation

1. Clone the repository

   git clone https://github.com/mjmajlesi/SmartDataCleaner.git
   cd SmartDataCleaner

2. Install dependencies

   pip install -r requirements.txt

Usage

1. Start the application

   cd src
   streamlit run app.py

2. Open in your browser

   • The app will automatically open at http://localhost:8501

3. Upload and clean your data

   • Upload your dataset, adjust settings in the sidebar, and preview the changes instantly!
   • Download the final cleaned dataset as CSV, Excel, or JSON.

---

📁 Project Structure

SmartDataCleaner/
├── src/
│   ├── app.py                  # Main Streamlit application
│   ├── data_read.py            # Robust reading utilities (CSV, Excel, JSON, TXT)
│   ├── data_analyst.py         # Data analysis & cleaning functions (standardization, correlation)
│   ├── data_missing.py         # Missing value detection & multiple imputations (KNN, Mean, Median, Mode)
│   ├── data_duplicate.py       # Duplicate row detection and removal
│   ├── data_outlier.py         # Outlier detection & treatment (IQR, Cap/Clip, Drop)
│   ├── clean_dtype.py          # Data type standardization and conversion
│   └── output.py               # Data quality scoreboard and summary metrics
├── test/
│   └── test.py                 # Test suite
├── requirements.txt            # Python dependencies
├── README.md                   # This file
└── LICENSE                     # Project license

---

📦 Dependencies

• streamlit - Web framework for data apps
• pandas - Data manipulation and analysis
• numpy - Numerical computing
• matplotlib - Data visualization library
• seaborn - Statistical data visualization
• scikit-learn - Machine learning library (KNN Imputer, StandardScaler)
• missingno - Missing data visualization
• dateparser - Date parsing and conversion
• openpyxl - Engine for writing Excel files

---

🤝 Contributing

Contributions are welcome! Feel free to:

• Report bugs and issues
• Suggest new features
• Submit pull requests with improvements

---

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.