A predictive maintenance AI model is only as good as the labels it learns from — and in industrial settings, high-quality labeled failure data is the scarcest resource in the entire development pipeline. Run-to-failure experiments are expensive. Expert annotators are scarce. Historical maintenance records are incomplete, inconsistently logged, and frequently missing the precise failure onset timestamps that supervised learning requires. Building a labeling strategy that extracts maximum training signal from imperfect data sources is the foundational skill that separates PdM programs that deliver real accuracy from those that generate impressive-looking dashboards with poor predictive performance in production. Get iFactory Support to connect your labeled maintenance data to production AI models that actually predict failures.
Turn Your Maintenance Records Into High-Quality AI Training Labels
iFactory's labeling pipeline combines expert annotation tools, weak supervision frameworks, and semi-automated labeling to extract maximum training signal from the incomplete, inconsistent maintenance data every industrial organization actually has.
Why Data Labeling Is the Hardest Problem in Industrial AI
Industrial AI practitioners consistently identify labeled data scarcity as the primary bottleneck in PdM model development — not sensor hardware, not compute, and not algorithmic complexity. The core challenge is that supervised learning requires examples of failure to learn from, but industrial equipment is maintained specifically to avoid failures, creating an inherent scarcity of labeled failure events. The strategies below address this scarcity systematically, extracting usable training signal from the data that actually exists. Contact iFactory to build a labeling strategy appropriate for your specific maintenance data situation.
Strategy 1
Run-to-Failure Experiments
Deliberately operating selected non-critical assets to failure generates the highest-quality labeled data — precise failure timestamps, complete degradation trajectories, and ground truth fault signatures. The NASA C-MAPSS turbofan dataset and PRONOSTIA bearing dataset both used run-to-failure methodologies to create the benchmark datasets that underpin most PdM research. Practical implementation requires identifying expendable assets in non-production environments and instrumenting them densely from the start of the experiment.
Strategy 2
Expert Technician Annotation
Experienced maintenance technicians annotating historical sensor data with their knowledge of fault characteristics, degradation sequences, and equipment behavior provide labels that capture implicit knowledge unavailable in maintenance records alone. Structured annotation protocols using iFactory's labeling interface present sensor data time-series in context with maintenance logs, guiding technicians to mark fault onset, degradation stages, and healthy baseline windows with standardized label taxonomy.
Strategy 3
Weak Supervision with Labeling Functions
Weak supervision uses programmatic labeling functions — rules encoding domain knowledge — to generate noisy labels at scale without individual human review of each training example. A labeling function for bearing failure might mark any 30-day window before a bearing replacement work order as "degrading" and any window immediately after replacement as "healthy." Multiple labeling functions with known accuracy characteristics are combined using a label model to produce probabilistic labels suitable for downstream model training.
Strategy 4
Semi-Supervised Learning
Semi-supervised approaches leverage the large volume of unlabeled sensor data alongside the scarce labeled examples. Autoencoders trained on unlabeled healthy data establish anomaly detection baselines. Self-training iteratively labels high-confidence unlabeled examples and incorporates them into the supervised training set. Graph-based methods propagate labels from labeled to similar unlabeled examples using feature similarity metrics — expanding effective training set size without additional human annotation cost.
Strategy 5
Transfer Learning from Related Assets
Labeled data from similar assets, different sites, or related equipment types can be used to pre-train models that are then fine-tuned on sparse labeled data from the target asset. Domain adaptation techniques adjust for distribution shift between source and target domains — the difference in operating conditions, sensor configurations, and failure rate distributions between the source labeled dataset and the target deployment environment. Transfer learning from public datasets (C-MAPSS, FEMTO, PHM08) provides a starting point when no site-specific labels exist.
Strategy 6
Synthetic Data Augmentation
Physics-based simulation models and data augmentation techniques generate synthetic failure examples that supplement scarce real failure data. Physics-informed generative models simulate bearing spall propagation, gear tooth wear, and insulation degradation, producing synthetic sensor signatures that share the statistical characteristics of real failures. GANs and VAEs trained on available real failure data generate augmented examples that increase effective training set diversity without additional run-to-failure experiments.
Labeling Framework Comparison: Which Strategy Fits Your Data Situation
No single labeling strategy is universally optimal — the right choice depends on the volume of available unlabeled data, the frequency of historical failure events in the maintenance record, the availability of domain experts for annotation, and the budget for run-to-failure experiments. The matrix below guides strategy selection based on your actual data situation. Book a demo to see how iFactory implements the right labeling strategy for your maintenance data.
Scroll for more →
| Labeling Strategy | Best When | Label Quality | Scale Potential |
|---|---|---|---|
| Run-to-Failure | Non-critical test assets available, research-grade accuracy needed | Highest — precise timestamps, full degradation curves | Low — costly and time-intensive per experiment |
| Expert Annotation | Experienced technicians available, moderate label budget | High — captures tacit knowledge, good failure onset precision | Medium — limited by expert time and annotation throughput |
| Weak Supervision | Large historical maintenance record, consistent work order logging | Medium — noisy but scalable, quality improvable with label model | High — programmatic, scales to millions of examples |
| Semi-Supervised | Very few labeled examples, abundant unlabeled sensor data | Medium — depends on representativeness of labeled seed set | High — leverages full unlabeled dataset |
| Transfer Learning | No site-specific labels, related source domain available | Medium — accuracy limited by domain shift magnitude | High — leverages external labeled datasets |
| Synthetic Augmentation | Failure physics well-understood, augmenting rare failure classes | Variable — depends on simulation fidelity | High — unlimited synthetic generation capability |
Label Quality Metrics: Measuring What Matters for PdM Model Accuracy
Failure Onset Timestamp Precision
Target: ±24 Hour Precision
The precision of failure onset labeling directly determines the accuracy window within which a trained model can predict failure. Labels with ±1 week precision produce models that alert within ±1 week of failure — which may be acceptable for scheduled maintenance planning but insufficient for time-critical interventions. Expert annotation combined with high-resolution sensor data review consistently achieves ±24 hour precision for incipient failure identification.
Inter-Annotator Agreement Rate
Target: Cohen's κ ≥ 0.75
When multiple annotators independently label the same sensor data segments, agreement rate measured by Cohen's kappa statistic quantifies label consistency. Kappa below 0.6 indicates the labeling task has insufficient specification or the annotators lack adequate domain knowledge — producing noisy training labels that increase model variance. Achieving κ ≥ 0.75 requires clear labeling guidelines, annotator calibration sessions, and systematic ambiguity resolution protocols.
Class Imbalance Ratio
Target: Managed to ≤50:1
Industrial failure datasets are inherently imbalanced — healthy operating periods vastly outnumber failure periods, often by ratios of 100:1 to 10,000:1. Unmanaged class imbalance produces models that achieve high accuracy by predicting "healthy" on every window while missing all actual failures. Resampling (SMOTE, ADASYN), cost-sensitive learning, and appropriate evaluation metrics (F1, AUC-PR rather than accuracy) address imbalance — with target ratio kept ≤50:1 in the training set through oversampling or undersampling strategies.
Label Coverage Rate
Target: ≥85% of Failure Events Labeled
Label coverage measures what fraction of known historical failure events have labeled sensor windows in the training dataset. Low coverage — where most maintenance actions lack corresponding labeled sensor segments — limits model exposure to the full diversity of failure patterns. Coverage below 60% produces models that are well-calibrated for the labeled subset of failure types but blind to unlabeled failure modes that are equally common in production.
The Labeling Pipeline: From Raw Maintenance Records to Training-Ready Labels
Maintenance Record Parsing and Failure Event Extraction Starting Point
Every labeling pipeline begins with extracting structured failure event records from unstructured maintenance records. Work orders, repair logs, and inspection records contain implicit failure information — but buried in free text, inconsistently coded, and often recorded days after the actual failure event. NLP-based extraction classifies work orders by failure type, identifies the affected component, and estimates the failure timestamp from the combination of report date, description of failure mode, and technician observations about how long the fault had been present before the repair.
Sensor Data Alignment and Window Generation
Each extracted failure event is aligned with the corresponding sensor data time series from the historian. Time windows are generated around each failure event: the failure window (N hours before failure), the degradation window (the period where fault indicators are expected to be detectable), and healthy windows (periods confirmed free of fault conditions, typically separated from failure events by a safety margin). Window lengths are failure-mode-specific — bearing failure windows differ from gradual efficiency degradation windows in both duration and signal characteristics.
Ambiguous Period Handling
Not all sensor data windows can be confidently labeled as either healthy or failing. Periods where the failure onset is uncertain, where multiple concurrent maintenance issues exist, or where sensor data quality is poor are ambiguous and must be handled explicitly. Three options: exclude ambiguous windows from training (conservative, reduces dataset size), include with uncertainty-weighted loss functions (advanced, requires probabilistic training), or mark for expert review (adds annotation cost but improves label quality). iFactory recommends exclusion for initial model development and uncertainty weighting in mature programs.
Weak Supervision Label Model Training
Where expert annotation budget is limited, weak supervision labeling functions encode domain rules into programmatic labelers that generate noisy labels at scale. Each labeling function has a known accuracy and coverage rate — accuracy being the fraction of labels it assigns correctly and coverage being the fraction of examples it assigns a label to (rather than abstaining). A generative label model learns the accuracy and correlation structure of all labeling functions simultaneously and produces probabilistic labels that are more accurate than any individual labeling function.
Active Learning for Expert Annotation Prioritization
Active learning directs expert annotation time to the examples where labels would provide the most model improvement — typically the examples the current model is most uncertain about, or examples from regions of feature space underrepresented in the current training set. Rather than annotating chronologically or randomly, active learning queries select the highest-information examples for annotation, achieving equivalent model accuracy with 30–60% fewer total annotations than passive labeling approaches.
Label Validation and Quality Scoring
Before any labeled dataset enters model training, systematic quality validation catches labeling errors that would degrade model performance. Quality checks include: temporal consistency (failure labels preceded by healthy labels), feature distribution plausibility (failure windows showing expected signal characteristics), inter-annotator consistency on overlapping annotations, and coverage completeness against the full failure event list. iFactory generates a dataset quality score report that quantifies label quality dimensions before model training begins. Contact iFactory Support to validate your existing labeled dataset quality.
Labeling Infrastructure Requirements
Time-Series Annotation UI
Browser-based annotation interface showing multi-channel sensor data with maintenance record context — enabling technicians to mark fault onset, severity stages, and healthy periods on actual signal waveforms
Label Version Control
Git-style versioning of label sets — tracking who labeled what, when, and with what rationale — enabling audit trails, rollback to prior label versions, and comparison of model performance across label set iterations
Labeling Function Library
Pre-built weak supervision labeling functions for common failure modes — bearing degradation, pump wear, heat exchanger fouling — that encode standard domain knowledge in reusable, auditable rule functions
CMMS Integration for Auto-Parsing
Automated work order parsing that extracts failure events from CMMS records and pre-populates the annotation queue with candidate labeling windows — reducing manual annotation setup time by 60%
Labeling Strategy Implementation: 6-Phase Roadmap
01
Maintenance Record Audit
Before building a labeling strategy, audit 2–3 years of CMMS work orders for the target asset class. Quantify: how many failure events are recorded per year, what fraction have adequate description for failure mode classification, what is the typical recording lag between failure and work order creation, and what fraction of maintenance actions represent preventive versus corrective work.
02
Failure Taxonomy Development
Define the label taxonomy before annotation begins — the set of failure mode categories that labels will distinguish. A taxonomy that is too fine-grained (distinguishing bearing inner race from outer race failures early) produces too few examples per class. A taxonomy that is too coarse loses predictive specificity. Start with 3–5 failure mode categories and refine as label volume allows.
03
Weak Supervision Baseline
Implement weak supervision labeling functions first — they generate initial training labels within days and establish the baseline model performance that active learning and expert annotation will improve upon. Write labeling functions for each failure mode using the failure taxonomy and CMMS parsing rules. Train the label model and generate probabilistic labels for the full historical dataset.
04
Expert Annotation Campaign
Train domain expert annotators on the labeling protocol using a calibration set of pre-labeled examples with known ground truth. Conduct inter-annotator agreement measurement on the calibration set before production annotation begins. Annotators achieving κ < 0.7 on calibration receive additional training. Plan annotation in 2-hour focused sessions to minimize cognitive fatigue effects on label quality.
05
Active Learning Cycle
Train an initial model on the weak supervision labels and use active learning to identify the examples most deserving of expert annotation budget. Run 3–5 active learning cycles, training a model, selecting the most informative unlabeled examples, annotating them with experts, and retraining. Track model performance improvement per annotation hour to identify the point of diminishing returns where additional annotation provides minimal accuracy gain.
06
Continuous Label Maintenance
Labels require ongoing maintenance as new failure events occur, equipment is modified, and model performance drift is detected in production. Establish a monthly label review process that incorporates new CMMS events into the labeled dataset and a quarterly label quality audit that checks for distribution shift between training labels and recent production data. iFactory Support manages ongoing label maintenance for production deployments.
Frequently Asked Questions
How many labeled failure examples are needed to train a reliable PdM model?
The minimum labeled failure set depends heavily on feature dimensionality, model architecture, and failure mode diversity. As a practical starting point, 30–50 labeled failure examples per failure mode enables a reasonably well-calibrated model using classical ML methods (random forest, gradient boosting). Deep learning methods generally require 200+ examples per class. Transfer learning from pre-trained models can reduce these requirements by 60–70% for target domains similar to the source domain.
What is the difference between labeling at the event level versus the window level?
Event-level labeling assigns a label to an entire failure event (e.g., "bearing failure on 2024-03-15") without specifying which sensor windows before the event are fault-indicative. Window-level labeling assigns labels to specific time windows in the sensor data (e.g., "this 4-hour window from 2024-03-12 shows early bearing fault signatures"). Window-level labels produce more precise training signal but require significantly more annotation effort — expert annotators must review sensor waveforms rather than just maintenance records.
Can weak supervision labels produce models as accurate as expert-annotated labels?
Weak supervision labels typically produce models 5–15% less accurate than high-quality expert annotations in head-to-head comparisons on well-defined failure modes. However, weak supervision scales to 10–100× more labeled examples at negligible marginal cost — and more training data often more than compensates for lower label quality. The optimal strategy for most industrial deployments is weak supervision for volume plus targeted expert annotation on the most informative examples identified by active learning.
How should labels handle the transition period between healthy and failing states?
The transition period — where a fault is developing but not yet confirmed as a labeling-worthy degradation event — is the most labeling-challenging period in PdM. Three approaches are used: binary labeling (hard threshold separating healthy from failing, ignoring transition uncertainty), fuzzy labeling (continuous label values like remaining useful life estimates), and explicit transition class labeling (a third "degrading" class between healthy and failing). Binary labeling is simplest and most common; fuzzy labeling produces the most informative training signal for regression-based RUL models.
Build the Labeled Dataset Your PdM Model Actually Needs
iFactory's labeling pipeline combines weak supervision, active learning, and expert annotation tools to extract maximum training signal from the incomplete, inconsistent maintenance data every industrial organization actually has — not the idealized datasets academic research assumes.
