Published at ACM AIware 2026Honorable Mention Paper Award

Deterministic vs. LLM-Controlled Orchestration for COBOL-to-Python Modernization

Authors

  • Naing Oo Lwin1,2*
  • Rajesh Kumar2
  1. 1Astrio Labs
  2. 2Bucknell University

* Core contributor

Research
Deterministic vs. LLM-Controlled Orchestration for COBOL-to-Python Modernization research poster
Read the paper

Abstract

Modernizing legacy COBOL systems remains difficult due to scarce expertise, large and long-lived codebases, and strict correctness requirements. Recent large language model (LLM)-based modernization systems increasingly rely on agentic workflows in which the model controls multi-step tool execution. However, it remains unclear whether delegating execution control to the LLM improves correctness, robustness, or efficiency in structured software engineering workflows. We present a controlled empirical study of deterministic and LLM-controlled orchestration for COBOL-to-Python modernization. Using a unified experimental framework, we hold the language models, prompts, tools, configurations, and source programs constant while varying only the execution control strategy. This isolates orchestration as the sole experimental variable. We evaluate both approaches using functional correctness, robustness across repeated stochastic runs, and computational efficiency. Across multiple models, deterministic orchestration achieves comparable computational accuracy to LLM-controlled orchestration while improving worst-case robustness and reducing performance variability across runs. Deterministic execution also reduces token consumption by up to 3.5x, leading to substantially lower operational cost. These results suggest that, in structured modernization workflows with explicit validation stages, fixed execution policies provide more stable and cost-efficient behavior than fully agentic orchestration without reducing translation quality.

Please cite as

@inproceedings{10.1145/3805760.3814891,
  author = {Lwin, Naing Oo and Kumar, Rajesh},
  title = {Deterministic vs. LLM-Controlled Orchestration for COBOL-to-Python Modernization},
  year = {2026},
  isbn = {9798400726019},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3805760.3814891},
  doi = {10.1145/3805760.3814891},
  abstract = {Modernizing legacy COBOL systems remains difficult due to scarce expertise, large and long-lived codebases, and strict correctness requirements. Recent large language model (LLM)-based modernization systems increasingly rely on agentic workflows in which the model controls multi-step tool execution. However, it remains unclear whether delegating execution control to the LLM improves correctness, robustness, or efficiency in structured software engineering workflows. We present a controlled empirical study of deterministic and LLM-controlled orchestration for COBOL-to-Python modernization. Using a unified experimental framework, we hold the language models, prompts, tools, configurations, and source programs constant while varying only the execution control strategy. This isolates orchestration as the sole experimental variable. We evaluate both approaches using functional correctness, robustness across repeated stochastic runs, and computational efficiency. Across multiple models, deterministic orchestration achieves comparable computational accuracy to LLM-controlled orchestration while improving worst-case robustness and reducing performance variability across runs. Deterministic execution also reduces token consumption by up to 3.5x, leading to substantially lower operational cost. These results suggest that, in structured modernization workflows with explicit validation stages, fixed execution policies provide more stable and cost-efficient behavior than fully agentic orchestration without reducing translation quality.},
  booktitle = {Proceedings of the 3rd ACM International Conference on AI-Powered Software},
  pages = {43--50},
  numpages = {8},
  keywords = {Agentic AI, COBOL-to-Python Translation, LLM-based Software Engineering, Legacy Code Modernization},
  location = {Montreal, QC, Canada},
  series = {AIware '26}
}
Share

More research

VeriPy: Source-Preserving Verification and Compatibility Checking for Python Components

Keeping a Python program and its formal guarantees aligned is a continuing maintenance problem. Specifications must describe the implementation that actually runs, and updates must preserve the behavior on which existing callers depend. VeriPy brings these obligations into a common workflow for annotated Python components. Developers and agents express contracts, invariants, and proof hooks as Python comments, then refine checked auxiliary lemmas using source-located diagnostics. Direct encoders produce Dafny or Lean artifacts while retaining admitted executable bodies and explicit models of their dependencies. A relational product checks whether an updated component still admits old inputs and preserves returned values and modeled exceptions. The resulting workflow connects source-level proof development with functional verification and backward compatibility, recording the assumptions under which each guarantee applies.

Forall-Lean-Agent for Auditable Reasoning in Formal Mathematics and Software Verification

Coding agents increasingly automate Lean proof development, but successful compilation alone does not establish that a candidate proves the intended statement under acceptable assumptions. We present FORALL-LEAN-AGENT, a frontend-agnostic framework for auditable reasoning in formal mathematics and software verification. The framework combines isolated workspaces, Lean tools, and fresh review with statement comparison, axiom audits, and independent proof checking where supported. Verification evidence and reviewer decisions are bound to the same candidate artifact, making acceptance traceable. We evaluate the framework on PutnamBench, both problems in the Lean Eval software-verification track, and VeriSoftBench. The PutnamBench evaluation achieves 672/672 in the Lean answer-given configuration at an average actor and reviewer cost of $4.72 per accepted task across retained final runs. On the 100-task VeriSoftBench subset, integration with FORALL-LEAN-AGENT raises benchmark-rule success from 93 to 100 for GPT-5.6 Sol at low effort while reducing cost from $69 to $62. These results show that agent harness design can improve correctness and efficiency while providing evidence beyond aggregate solve counts.

Same Score, Different Evidence: Decodability, Surface Sufficiency, and Causal Relevance in Code Models

Interpretability experiments must distinguish model mechanisms from measurement artifacts. We present a case-study-backed reporting matrix for linear probes over five identifier properties and roles in code models. The study includes three models and collectively covers all seven XLCoST programming languages without forming a complete factorial design. The matrix binds each claim to an estimand, comparator, matching unit, uncertainty statement, outcome, and falsifier. On Python, boolean occurrence type is decodable at 0.981–0.988 macro-F1, while a masked source-line classifier reaches 0.983 without a language model. Paired problem-clustered differences are −0.003 to +0.002, with intervals excluding probe advantages above 0.014–0.017. A full-residual patch at a class or structure query-name site fails its specified bidirectional gate in Qwen2.5-1.5B, recovering 0.009 to 0.020 of the matched behavioral gap. A third case exposes a failed control. Role-conditioned iterator renaming yields 0.998–0.999 F1 at embedding layer zero, consistent with a lexical code introduced by the intervention. These findings require distinct claims about decodability, surface sufficiency, and causal relevance.