A Three-Stage Transient-Preserving Co-optimised Framework for High-Fidelity Behind-the-Meter Data Processing

As data-driven planning of modern power grids evolves, the integrity of behind-the-meter (BTM) telemetry has become a critical determinant of both grid stability and financial viability. Traditional data processing often falls into a ’peak-preservation impasse’ (PPI), where statistical filters erroneously smooth out high-magnitude transients by misidentifying them as outliers. Such data corruption masks true physical stress on the network, leading to undersized infrastructure and significant risks to grid reliability. This paper introduces a novel three-stage, holistically co-optimised framework designed to resolve this impasse. Employing a central Bayesian Optimisation (BO) engine, the framework simultaneously tunes statistical and machine learning (ML) parameters to distinguish between sensor noise and legitimate operational surges. Validated against twelve real-world datasets (1,036,800 data points) at one-second resolution, the framework achieved a 29.13% average Affinity Score and a minimum Wasserstein distance of 9.62, accurately preserving transients that baseline models underestimated by over 95%. Performance was assessed using Mean Absolute Error (MAE), R2, Wasserstein distance, and Kolmogorov-Smirnov (K-S) statistics. Crucially, this high-fidelity reconstruction resulted in a 42% improvement in downstream forecasting accuracy and a 47% reduction in computational overhead compared to the average performance of alternative methods. Non-parametric Friedman andWilcoxon signed-rank tests across the 12 datasets, with Holm-Bonferroni correction, confirm the Affinity advantage is statistically significant against baselines. The proposed framework prevents systematic underestimation of capacity requirements, thereby safeguarding the economic and operational stability of the utility ecosystem.