Industrial AI Data: What It Needs & How to Prep It
- Approx. CSV size (MB)
- MB
- Result
- Time window (hours)
- 24 h
- You entered
- Number of sensors
- 10
- You entered
Industrial AI needs clean, labeled, time-stamped data that reflects real operating conditions, and preparing it means fixing gaps, removing noise, and aligning it with the specific prediction or control task. You typically enter raw sensor logs, maintenance records, and process parameters, and you get back a structured dataset ready for training or analysis. This article explains the data types, preparation steps, common pitfalls, and costs involved.
Formula
The first output, Expected data rows, is the total number of sensor readings collected over a time window. It is calculated as:
rows = time (hours) × 3600 (seconds/hour) × sampling rate (Hz) × number of sensors
Where:
- t = time window in hours (e.g., 24 for one day)
- r = sampling rate in hertz (readings per second per sensor)
- n = number of sensors
The second output, Approx. CSV size (MB), estimates the storage needed for a CSV file containing those rows, assuming each value is stored as an 8-byte floating-point number:
size (MB) = (rows × 8 bytes) / (1024²)
This gives a rough idea of the data volume you will handle.
Worked Example
Using the default inputs: t = 24 hours, n = 10 sensors, r = 1 Hz.
- Calculate seconds in the window: 24 × 3600 = 86,400 seconds.
- Multiply by sampling rate: 86,400 × 1 = 86,400 readings per sensor.
- Multiply by number of sensors: 86,400 × 10 = 864,000 total rows.
- Estimate CSV size: (864,000 × 8) / (1,048,576) ≈ 6.59 MB.
The calculator shows 864,000 rows and about 6.59 MB for a single day of data from 10 sensors at 1 Hz. This is a small dataset; industrial systems often run at higher rates for longer periods, producing gigabytes or terabytes.
What Data Does Industrial AI Actually Need?
Industrial AI systems—predictive maintenance, quality control, process optimization—rely on several data types:
- Sensor data: temperature, pressure, vibration, flow, current, voltage, and other process variables logged over time.
- Event logs: alarms, warnings, start/stop events, and operator actions with timestamps.
- Maintenance records: repair history, part replacements, and inspection results (often in CMMS or ERP systems).
- Quality data: product measurements, defect counts, and lab test results.
- Operational context: shift schedules, production plans, raw material batches, and environmental conditions (e.g., ambient temperature).
For supervised learning (e.g., predicting machine failure), you also need labels—the outcome you want to predict, such as “failed within 7 days” or “defect detected.” Labels often come from maintenance logs or quality inspections, and they must be aligned with the sensor data in time.
How to Prepare Data for Industrial AI
Preparation transforms raw data into a clean, structured format. The typical workflow:
- Collect and centralize: Pull data from PLCs, historians, databases, and CSV exports into one storage system (e.g., a data lake or SQL database).
- Clean the data: Remove or correct duplicate timestamps, out-of-range values, and constant readings (sensor stuck). Use domain knowledge to set plausible min/max limits.
- Handle missing values: For time series, forward-fill short gaps (e.g., < 5 minutes) or interpolate; for longer gaps, mark them as missing or drop the segment if the model cannot handle it.
- Resample to a uniform time grid: Most models expect evenly spaced samples. Aggregate or interpolate to a fixed interval (e.g., 1 minute, 1 hour) that matches the prediction horizon.
- Align labels: For predictive maintenance, create a target variable by looking forward from each timestamp. For example, label each row as 1 if a failure occurs within the next 7 days, else 0.
- Engineer features: Compute rolling statistics (mean, standard deviation, min, max) over windows, frequency-domain features (FFT peaks), and event counts. This captures trends and patterns.
- Split data carefully: Use time-based splits (e.g., first 80% for training, last 20% for testing) to avoid data leakage from the future.
- Normalize/scale: Scale features to a similar range (e.g., 0-1 or z-score) for many algorithms, especially neural networks and SVMs.
Each step should be documented and versioned so that experiments are reproducible.
Common Problems and How to Avoid Them
Several issues trip up industrial AI projects:
- Data leakage: Using future information in training (e.g., normalizing with global statistics before splitting). Solution: fit scalers on training data only.
- Imbalanced labels: Failures are rare, so the model may just predict “no failure.” Use techniques like oversampling, undersampling, or weighted loss functions.
- Non-stationary processes: Machines degrade, seasons change, and operating conditions shift. Models trained on old data may not generalize. Solution: retrain periodically and monitor drift.
- Sensor drift and misalignment: Sensors can drift or be installed at different times, causing inconsistent baselines. Solution: calibrate and use relative changes when possible.
- Missing timestamps: Data historians often skip periods when the machine is off. Decide whether to treat those as missing or as a separate state.
A common misconception is that more data is always better. In practice, irrelevant or noisy data can hurt model performance. Focus on collecting data that is directly related to the target outcome.
How to Choose the Right Data and Tools
Choosing what data to collect and which tools to use depends on your goal:
- Define the problem: Are you predicting failure (classification), estimating remaining useful life (regression), or detecting anomalies (unsupervised)? This determines the labels and features.
- Start with available data: Use existing historians and maintenance logs before adding new sensors. Many insights can come from data you already have.
- Select tools based on team skills: Python (pandas, scikit-learn, TensorFlow) is common, but some teams use R, MATLAB, or commercial platforms like DataRobot or Siemens MindSphere.
- Consider edge vs. cloud: Real-time control may require edge deployment; batch analysis can run in the cloud. Data preparation often happens in the cloud due to compute needs.
There is no one-size-fits-all tool. Start simple with a Jupyter notebook or a spreadsheet for small datasets, then move to distributed systems (Spark, Dask) when data exceeds memory.
Costs and Infrastructure Considerations
Data preparation is often the most time-consuming and costly part of an AI project. Costs include:
- Storage: Cloud storage (e.g., S3, Azure Blob) costs per GB per month. The calculator above shows a day of data at 1 Hz from 10 sensors is ~6.6 MB; a year is ~2.4 GB. At 100 Hz, that becomes 240 GB per year.
- Compute: Cleaning and feature engineering require CPU/GPU time. Cloud instances cost per hour; a typical prep job might cost a few dollars to tens of dollars per run.
- Data engineering time: A data engineer or scientist spends days to weeks on data cleaning. This is often the largest hidden cost.
- Software licenses: Some commercial tools charge per user or per node.
To control costs, sample data during exploration, use efficient file formats (Parquet instead of CSV), and consider on-premises storage if data is highly sensitive.
Frequently Asked Questions
How much data do I need for industrial AI?
There is no fixed number. For simple anomaly detection, a few weeks of normal operation may suffice. For predictive maintenance with rare failures, you may need months or years of data to capture enough failure events. A rule of thumb: at least 100 examples of the rare event (e.g., failures) for classification, and more for complex models.
What if I don’t have labels (failure records)?
You can use unsupervised learning (anomaly detection) to find unusual patterns. Or you can create labels from indirect signals, such as maintenance work orders that mention “breakdown” or “repair.” Another option is to use survival analysis, which handles censored data (machines that haven’t failed yet).
How do I handle different sampling rates from different sensors?
Resample all sensors to the same time grid, usually the lowest rate if you don’t need high frequency, or the highest if you need fast dynamics. Use interpolation (e.g., linear or forward-fill) to fill gaps. Be careful not to create false precision by upsampling too aggressively.
What is data leakage in industrial AI?
Data leakage occurs when information from the future or from the test set is used during training, making results look better than they are. Examples: scaling using the entire dataset’s mean, or including a feature that is only known after the prediction time. To avoid it, split data chronologically and apply all transformations separately to train and test sets.
Should I use cloud or on-premises for industrial data?
It depends on data sensitivity, latency, and budget. Cloud offers scalability and managed services, but may have compliance issues for sensitive production data. On-premises gives control and low latency but requires hardware maintenance. Many companies use a hybrid: edge for real-time, cloud for batch analytics.
How often should I retrain my model?
Retrain when performance drops or when the process changes significantly. Monitor model accuracy and data drift (e.g., using statistical tests on feature distributions). For stable processes, monthly or quarterly retraining may be enough; for dynamic ones, weekly or even daily.
Related
Formula used
in t | Time window (hours) | h | 24 | 1 in n | Number of sensors | - | 10 | 1 in r | Sampling rate (Hz) | Hz | 1 | 0.1 out rows | Expected data rows | - | t * 3600 * r * n | 0 out size_mb | Approx. CSV size (MB) | MB | (t * 3600 * r * n * 8) / (1024^2) | 2
Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.