Structured Data Preparation Pipeline for Machine Learning-Applications in Production

Maik Frye, Robert Heinrich Schmitt

Abstract

The application of machine learning (ML) is becoming increasingly common in production. However, many ML-projects fail due to the existence of poor data quality. To increase its quality, data needs to be prepared. Through the consideration of versatile requirements, data preparation (DPP) is a challenging task, while accounting for 80 % of ML-projects duration [1]. Nowadays, DPP is still performed manually and individually making it essential to structure the preparation in order to achieve highquality data in a reasonable amount of time. Thus, we present a holistic concept for a structured and reusable DPP-pipeline for ML-applications in production. In a first step, requirements for DPP are determined based on project experiences and detailed research. Subsequently, individual steps and methods of DPP are identified and structured. The concept is successfully validated through two production use-cases by preparing data sets and implementing ML-algorithms.

Keywords

Artificial Intelligence, Machine Learning, Data Preparation, Data Quality

Download

IMEKO-TC10-2020-034.pdf

DOI

-

IMEKO TC

TC10 - Measurement for Diagnostics, Optimization and Control

Back to the proceedings