Riess 2023 — Temporal Loss Functions for Remaining Cycle-Time Prediction

Single-authored paper in Nordic Machine Intelligence, open-access. Corresponds to Paper I of Riess's PhD thesis.

Summary

The usefulness of remaining-cycle-time models for operational decision support (queue prioritisation, dynamic resource planning) depends not only on accuracy but on earliness — predictions should be as accurate as possible as early as possible in a case, when information is minimal and prediction is hardest. Two research questions drive the paper: (RQ1) how do temporally weighted loss functions influence earliness of LSTM-based remaining-time models? (RQ2) do such models satisfy the criterion of temporal consistency (monotonically decreasing predictions)?

Method: the paper adopts the now-standard LSTM-with-prefix-log approach of Tax et al. 2017 and Navarin et al. 2017, then swaps in three L1 losses with different temporal-decay functional forms:

MAEEtD — exponential decay: |y-ŷ| + |y-ŷ|/e^t.
MAEPtD — power decay: |y-ŷ| + |y-ŷ|·((T_i - t)/T_i).
MAEMtD — moderate decay: |y-ŷ| + |y-ŷ|/t.

All three upweight early-prefix residuals relative to late-prefix residuals, but by different amounts. The three losses are compared to unweighted MAE on four public event logs: Sepsis, Helpdesk, BPIC 2012 Traffic Fines, Hospital Billing. Prefix logs are truncated to length 20; per-log hyper-parameters (units, layers, optimiser, batch size, learning rate) are tuned separately and then held fixed across loss variants.

Results: MAEEtD gives statistically significant earliness improvements on Sepsis (F=3.94, p=0.026) and Traffic Fines (F=4.03, p=0.024); MAEPtD and MAEMtD do not reach significance. Average MAE improvement at early prefixes is up to ~4% on Traffic Fines within the first five events. The paper then introduces Temporal Consistency (TC) — operationalised as an average difference-in-direction metric over consecutive predictions — and finds that none of the four loss functions produce monotonically decreasing predictions; TC errors spike at specific prefixes, particularly for long traces with low support.

Conclusion: temporal decay is useful as a hyper-parameter, not a universal improvement, and comes with a potential trade-off against temporal consistency. The field should evaluate remaining-time models on all three dimensions: accuracy, earliness, and temporal consistency.

Connections

Paper I of sources/2023-riess-phd-thesis-ppm.
Feeds concepts/remaining-time-prediction as the canonical Riess citation on earliness/consistency.
Builds directly on sources/2017-tax-lstm-process-prediction, sources/2017-navarin-lstm-data-aware-remaining-time, sources/2019-verenich-survey-ppm (Verenich et al.'s cross-benchmark survey) and sources/2020-rama-maneiro-deep-learning-ppm-review (Rama-Maneiro et al.'s DL-PPM review).
Related to sources/2018-difrancescomarino-genetic-hpo-ppm (GA-based HPO with earliness term in fitness) and sources/2018-dumas-fundamentals-of-bpm (BPM fundamentals).
Concepts: concepts/remaining-time-prediction, concepts/lstm-ppm, concepts/predictive-process-monitoring, concepts/trace-encoding.
Author: entities/mike-riess.