EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

Shiva Krishna Reddy Malay, Shravan Nayak, Sagar Davasam, Srinivas Sunkara, Sai Rajeswar
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:85969-86001, 2026.

Abstract

The rapid evolution of Large Language Models (LLMs) has shifted their role from passive information providers to active agents capable of executing complex workflows. However, the realization of a true "AI worker" is currently hindered by benchmarks that fail to capture the intricacy of professional environments, which demand long-horizon planning, complex tool usage, and adherence to strict access protocols. To bridge this gap, we introduce EnterpriseOps-Gym, a benchmark environment designed to evaluate agentic planning in realistic enterprise settings. EnterpriseOps-Gym provides: (i) 1,150 expert-curated tasks across eight interconnected domains (including HR, IT, Customer Service and productivity tools) that require managing persistent state and adhering to strict outcome-based verification logic; and (ii) a high-fidelity, containerized sandbox environment hosting 164 database tables and 512 functional tools. Our evaluation reveals critical limitations in state-of-the-art models: even the top-performing Claude Sonnet 4.5 achieves only 34.1% success, struggling significantly with planning consistency, error recovery, and policy constraints. Furthermore, we observe that agents frequently fail to refuse infeasible tasks, leading to unintended and potentially harmful side effects on the system. These findings indicate that current agents are not yet ready for enterprise deployment. By releasing EnterpriseOps-Gym, we provide a concrete testbed to advance the reliability of autonomous agents in professional workflows.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-malay26a, title = {{E}nterprise{O}ps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings}, author = {Malay, Shiva Krishna Reddy and Nayak, Shravan and Davasam, Sagar and Sunkara, Srinivas and Rajeswar, Sai}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {85969--86001}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/malay26a/malay26a.pdf}, url = {https://proceedings.mlr.press/v306/malay26a.html}, abstract = {The rapid evolution of Large Language Models (LLMs) has shifted their role from passive information providers to active agents capable of executing complex workflows. However, the realization of a true "AI worker" is currently hindered by benchmarks that fail to capture the intricacy of professional environments, which demand long-horizon planning, complex tool usage, and adherence to strict access protocols. To bridge this gap, we introduce EnterpriseOps-Gym, a benchmark environment designed to evaluate agentic planning in realistic enterprise settings. EnterpriseOps-Gym provides: (i) 1,150 expert-curated tasks across eight interconnected domains (including HR, IT, Customer Service and productivity tools) that require managing persistent state and adhering to strict outcome-based verification logic; and (ii) a high-fidelity, containerized sandbox environment hosting 164 database tables and 512 functional tools. Our evaluation reveals critical limitations in state-of-the-art models: even the top-performing Claude Sonnet 4.5 achieves only 34.1% success, struggling significantly with planning consistency, error recovery, and policy constraints. Furthermore, we observe that agents frequently fail to refuse infeasible tasks, leading to unintended and potentially harmful side effects on the system. These findings indicate that current agents are not yet ready for enterprise deployment. By releasing EnterpriseOps-Gym, we provide a concrete testbed to advance the reliability of autonomous agents in professional workflows.} }
Endnote
%0 Conference Paper %T EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings %A Shiva Krishna Reddy Malay %A Shravan Nayak %A Sagar Davasam %A Srinivas Sunkara %A Sai Rajeswar %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-malay26a %I PMLR %P 85969--86001 %U https://proceedings.mlr.press/v306/malay26a.html %V 306 %X The rapid evolution of Large Language Models (LLMs) has shifted their role from passive information providers to active agents capable of executing complex workflows. However, the realization of a true "AI worker" is currently hindered by benchmarks that fail to capture the intricacy of professional environments, which demand long-horizon planning, complex tool usage, and adherence to strict access protocols. To bridge this gap, we introduce EnterpriseOps-Gym, a benchmark environment designed to evaluate agentic planning in realistic enterprise settings. EnterpriseOps-Gym provides: (i) 1,150 expert-curated tasks across eight interconnected domains (including HR, IT, Customer Service and productivity tools) that require managing persistent state and adhering to strict outcome-based verification logic; and (ii) a high-fidelity, containerized sandbox environment hosting 164 database tables and 512 functional tools. Our evaluation reveals critical limitations in state-of-the-art models: even the top-performing Claude Sonnet 4.5 achieves only 34.1% success, struggling significantly with planning consistency, error recovery, and policy constraints. Furthermore, we observe that agents frequently fail to refuse infeasible tasks, leading to unintended and potentially harmful side effects on the system. These findings indicate that current agents are not yet ready for enterprise deployment. By releasing EnterpriseOps-Gym, we provide a concrete testbed to advance the reliability of autonomous agents in professional workflows.
APA
Malay, S.K.R., Nayak, S., Davasam, S., Sunkara, S. & Rajeswar, S.. (2026). EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:85969-86001 Available from https://proceedings.mlr.press/v306/malay26a.html.

Related Material