Study sketch — Temporal consistency of LLM-agent runtime recommendations

Motivation and gap

Riess 2023 introduced Temporal Consistency (TC) as a third axis of PPM evaluation alongside accuracy and earliness: a model whose predictions oscillate across prefix lengths is operationally unusable regardless of mean accuracy. The APM Manifesto elevates runtime actionability as a core capability — agents are expected to recommend interventions during execution — and Kubrak 2022 catalogues "intervention policy" as the sixth (and least-developed) dimension of PrPM. Yet no existing work evaluates whether LLM-agent recommendations are temporally consistent. If an agent flips its recommendation mid-case, downstream queue prioritisation, staffing, and customer-communication decisions destabilise — the same operational pathology Riess named for LSTM predictions, now in a higher-stakes setting. PHI403 L18's Popperian framing sharpens this: recommendations that can't commit to stable falsifiable predictions aren't scientific outputs.

Research questions

RQ1 (measurement). How do LLM-agents compare to conventional PPM models (Tax LSTM, ProcessTransformer) on the three-axis evaluation (accuracy, earliness, TC) when making runtime recommendations on identical prefix sequences?
RQ2 (mechanism). Is recommendation-flipping driven primarily by epistemic uncertainty (longer reasoning traces, low prompt-signal) or aleatoric (process-variability-induced genuine ambiguity)?
RQ3 (intervention). Do specific prompt-engineering techniques (chain-of-thought, explicit prior-commitment instructions, confidence-calibration requests) improve TC without degrading accuracy or earliness?

Hypotheses

H1. LLM-agents exhibit significantly more recommendation-flips per prefix than calibrated LSTM/Transformer baselines on the same cases — the "3-axis generalisation" hypothesis.
H2. Recommendation-flipping is positively correlated with reasoning-trace length (a proxy for epistemic uncertainty per Fournier's observability framing) and weakly correlated with process entropy (aleatoric).
H3. Chain-of-thought prompting worsens TC despite improving accuracy (an explicit-cost trade-off); explicit "commit to prior unless strong evidence" instructions improve TC at modest accuracy cost.
H4. Across BPIC logs with varying regularity, TC degradation is larger on spaghetti logs than lasagna logs — generalising Riess 2023's log-dependency finding to agentic recommendations.

Method

Datasets. Direct reuse of Riess 2023's public logs (Sepsis, Helpdesk, BPIC Traffic Fines, Hospital Billing) to preserve comparability + one private BRAGE-adjacent Telenor customer-service log for industry validity.

Recommendation targets. Three canonical PPM outputs per the landscape synthesis: next activity, remaining time, binary outcome. Each target evaluated independently.

Models compared.

LLM-agents (3): Claude, GPT-4-class, Gemma2 — the BRAGE setup extended to per-prefix recommendation.
PPM baselines (3): unweighted-L1 LSTM (Tax), temporally-weighted-L1 LSTM (Riess 2023 exponential variant), ProcessTransformer.
Rule-based control: most-likely-next-activity from transition matrix.

TC instrument. Adapted from Riess 2023 — for remaining-time: monotonicity violations per prefix. For next-activity / outcome: recommendation-flip rate = count of recommendation changes across consecutive prefixes of the same case, normalised by prefix count.

Uncertainty instrumentation. Per recommendation, collect (a) LLM reasoning-trace length (epistemic proxy), (b) local transition-entropy at the current process state (aleatoric proxy), (c) self-reported confidence.

Experimental conditions. 3 prompt strategies × 3 LLMs × 3 PPM-baseline models × 5 logs × 3 recommendation targets. Per-case bootstrap for confidence intervals.

Analysis. Three-axis tables per Riess 2023 convention + regression of flip-rate on uncertainty proxies (H2) + ANOVA with prompt strategy × model interaction (H3).

Validity threats

LLM version drift during the study — pin model versions; publish exact APIs used.
Prompt-underfitting for non-CoT control: use standardised prompt templates published in an appendix.
Construct of "flipping": some flips are corrections on new evidence — mitigated by separating Bayes-coherent updates (flip toward higher-likelihood option given new events) from incoherent oscillation.

Deliverables and venues

Short paper. Target: Nordic Machine Intelligence (direct continuity with Riess 2023 that introduced TC) or ICPM 2027. A crisp extension paper, 10–12 pages.
Benchmark harness. Public code + evaluation scripts on GitHub. Consumers of Riess 2023 can immediately run TC on their own agents.

Connections

Extends the three-axis evaluation beyond remaining-time to full PrPM recommendation space. Feeds PrPM with a missing evaluation lens. Connects concepts/conversational-actionability (APM capability) to a measurable operational property. Empirically grounds the operational framing in Riess 2023's commitment #5.