1. Abstract
This paper conducts a comprehensive survey on data quality (DQ) evaluation and improvement tools for machine learning (ML). It emphasizes the critical role of high-quality data in ML model performance, fairness, robustness, safety, and scalability. The paper introduces four DQ dimensions (intrinsic, contextual, representational, and accessibility) and twelve metrics specific to ML, providing definitions and examples.
The survey reviews seventeen open-source DQ tools released in the past five years, analyzing their strengths and limitations based on the DQ dimensions and metrics. It proposes a roadmap for developing new DQ tools, highlighting the importance of integrating automation, monitoring, and AI technologies. The paper also discusses emerging trends like large language models (LLMs) and generative AI in DQ evaluation and improvement, showcasing their potential applications and future directions.
2. Quick Read
a. Research Methodology
The paper adopts a survey-based approach, extensively reviewing existing literature on DQ dimensions, metrics, and tools. It analyzes the functionality, usability, and effectiveness of seventeen open-source DQ tools, identifying their strengths and limitations.
Innovation and Improvements
- Comprehensive DQ metrics: The paper compiles a comprehensive set of DQ metrics specifically tailored for ML tasks, addressing the gap in existing research.
- Roadmap for tool development: It proposes a roadmap for developing new DQ tools, emphasizing the integration of automation, monitoring, and AI technologies to meet the evolving needs of data-centric AI.
- Focus on emerging trends: The paper explores the potential applications of LLMs and generative AI in DQ evaluation and improvement, highlighting their transformative potential.
b. Experimental Process
The paper does not present specific experiments but focuses on analyzing existing tools and identifying research gaps. It evaluates the tools based on their functionality, usability, and effectiveness in addressing DQ issues in ML.
c. Advantages and Potential Impact - Practical insights: The paper provides valuable insights into the current landscape of DQ tools for ML, helping practitioners and researchers choose suitable tools and identify areas for improvement.
- Framework for tool development: The proposed roadmap offers a practical framework for developing new DQ tools, guiding developers in designing effective and user-friendly solutions.
- Implications for data-centric AI: The paper highlights the importance of DQ in data-centric AI and discusses how emerging technologies like LLMs and generative AI can revolutionize DQ evaluation and improvement, leading to more reliable and robust ML models.
3. Summary
a. Contributions
- Comprehensive overview of DQ dimensions and metrics in ML.
- Detailed analysis of existing open-source DQ tools.
- Roadmap for developing new DQ tools with a focus on automation, monitoring, and AI integration.
- Discussion of emerging trends and their potential impact on DQ in ML.
b. Innovation Points - Compilation of comprehensive DQ metrics for ML.
- Roadmap for developing AI-powered DQ tools.
- Exploration of LLMs and generative AI in DQ evaluation and improvement.
c. Future Research Directions - Development of AI-powered DQ tools that can automatically detect and fix DQ issues.
- Integration of DQ evaluation and improvement into ML pipelines.
- Exploration of LLMs and generative AI for generating high-quality synthetic data for ML training.
- Research on DQ metrics and dimensions specific to different ML tasks and domains.
View PDF：https://arxiv.org/pdf/2406.19614
