Study sketch — SynBPS-APM: A controlled testbed for agentic BPM

Motivation and gap

The APM Manifesto defines four required capabilities — concepts/framed-autonomy, concepts/explainability-apm, concepts/conversational-actionability, concepts/self-modification — but supplies no evaluation methodology. Existing agent benchmarks are generic and non-BPM (TheAgentCompany), while BPM evaluation remains fixated on a small stable of public event logs (Sepsis, Helpdesk, BPIC) whose external validity is directly challenged by the Cartwright-Hardie argument in PHI403 L11 and the RCT-limitations catalogue in L19. SynBPS (Riess 2024) already solved this for single-model PPM research by introducing a parametric Markov-chain-based simulator where process characteristics are control variables. An extension to agent-driven processes is a natural next step and would make the framework the reference testbed for APM research.

Research questions

RQ1 (feasibility). Can SynBPS be extended with LLM-agent decision hooks at branching activities + a frame specification DSL, while preserving its parametric-factorial design properties?
RQ2 (capability mapping). How does LLM-agent performance on the four APM capabilities vary under controlled manipulation of process variability (memory order, transition entropy) and autonomy level (Elyasaf's 5-level roadmap)?
RQ3 (failure-mode discovery). Which process characteristics reveal APM agent failure modes that single-log benchmarks obscure — particularly under drift conditions drawn from Riess 2022's taxonomy?

Hypotheses

H1. Agent-driven process completion rate decreases with increasing transition entropy (holding frame constant), but the slope differs significantly across autonomy levels — operationalising the autonomy-predictability trade-off the APM manifesto asserts but does not measure.
H2. High-autonomy agents (Elyasaf levels 4–5) exhibit larger between-run variance on identical process configurations than low-autonomy agents (behavioural variability) — quantifying a phenomenon named in Fournier 2025.
H3. Drift-injection (sudden transition-probability shifts mid-run) disproportionately degrades agents without self-modification capability — extending Riess 2022's drift claim from predictive models to agentic execution.
H4. Framed autonomy (Calvanese 2025's normative frames) measurably constrains failure modes compared to unframed agents on the same process — supplying the first empirical check on a capability the manifesto treats as postulate.

Method

Architecture. SynBPS-APM adds three components to the existing SynBPS Markov backbone:

Agent-hook interface at branching activities — when case reaches a decision node, a registered LLM agent receives the current trajectory + frame and returns a next-activity choice.
Frame specification language — JSON/DSL expressing allowed transitions, goal conditions, and normative constraints (drawn from the Calvanese 2025 frame taxonomy).
Trajectory-scoped logger instrumented per Fournier's observability requirements — perceive / reason / act tuples captured alongside the standard event log.

Experimental design. Factorial: 4 transition-entropy levels × 3 autonomy levels (Elyasaf L2/L3/L4) × {stationary, drifting} × 3 LLM backbones (per BRAGE-style model selection) = 72 cells × N ≥ 30 replications.

Measurements. Per case: (a) goal attainment (binary), (b) frame-conformance (count of violations), (c) cycle time, (d) MAPE-K activations (Elyasaf), (e) run-to-run variance. Per batch: Kubrak's six PrPM dimensions scored.

Baselines. Oracle agent (knows transition probabilities); rule-based agent (deterministic policy); workflow-only execution (no agent). These triangulate the agent's marginal contribution.

Analysis. Mixed-effects ANOVA with random effect on replication seed; simple-slopes for H1; Brown-Forsythe on H2; change-point detection for H3 drift response; pre-registered per Kitchenham-style protocol to avoid post-hoc slicing.

Validity threats and responses

Ecological validity (per L19): Markov backbone cannot model concurrency or resources — acknowledged as in SynBPS 2024. Complement with one calibrated industry log for cross-check.
Construct validity of "autonomy level": operationalised via prompt-scaffolding rigour, not LLM capability alone. Pre-registered operational definitions.
External validity of LLM choice: 3 models is thin; mitigate by making the harness model-agnostic and releasing benchmark so others can add models.

Deliverables and venues

Software. SynBPS-APM as a PyPI package + GitHub extension of the existing mikeriess/SynBPS repo. Apache-2.0 or MIT. Benchmark suite + agent harness separately.
Paper. Target primary venue: BPM 2027 main track (method + evaluation infrastructure fits). Fallback: SIMULATION (SCS) for methodology emphasis (continuity with 2024 paper). Workshop first venue if time-pressured: PMAI'26 / AutoBiz — where the APM manifesto co-authors gather.

Connections

Extends the Riess research arc capability #2 (control-variable manipulation via simulation) to APM. Supplies the missing evaluation layer to syntheses/abpms-to-apm-evolution. Feeds concepts/agentic-bpm and concepts/agent-process-observability.