Abstract

This study addresses the novel risks and governance challenges introduced by agentic AI systems in enterprise software environments. Unlike traditional AI or deterministic software, agentic AI autonomously executes complex workflows, interacts with tools, and accesses sensitive data, creating security, reliability, and accountability risks that are inadequately managed by existing software engineering and AI governance approaches. To mitigate these issues, this study proposes a Risk-Aware Human-in-the-Loop (RA-HIL) Governance Framework, which integrates layered controls across governance policy, agent planning, tool authorization, runtime monitoring, and audit-feedback management. This framework enables risk-based, fine-grained autonomy decisions that balance automation and the necessary human oversight. A scenario-based evaluation using a defect triage workflow demonstrated significant improvements over baseline agentic AI operations in terms of security control, reliability, auditability, accountability, and operational efficiency. Key enhancements include restricted unauthorized tool use, evidence-based action recommendations, comprehensive audit trails, clear human approval for medium and high-risk actions, and preserved automation for low-risk tasks. The RA-HIL framework reconceptualizes agentic AI as a controlled software workflow rather than a standalone model, emphasizing action-level governance tailored to impact, sensitivity, and reversibility. The limitations include the qualitative nature of the evaluation and the need for domain-specific adaptations. Future work involves middleware implementation, quantitative validation in production environments, and exploration of adversarial resilience, providing a practical governance model for the secure, reliable, and accountable deployment of agentic AI in enterprise systems.

References

• D. Cahyono, H. E. Atmaja, and H. Zainarthur, “AI Agent Based Service Innovation to Enhance Efficiency and User Experience,” TMJ, vol. 10, no. 3, pp. 146–156, Jan. 2026, doi: 10.33050/tmj.v10i3.2585.
• I. Satpathy, A. Nayak, and V. Jain, “The Strategic Role of Artificial Intelligence (AI) in Service Delivery Systems,” Igi Global, 2024, pp. 291–310. doi: 10.4018/979-8-3693-7909-7.ch014.
• G. K. Sangam, “AI-Assisted CRM Agents for Sales, Service, and Support Operations,” IJAIDSML, vol. 7, pp. 245–249, Jan. 2026, doi: 10.63282/3050-9262.ijaidsml-v7i1p141.
• S. Ranjan, D. Chembachere, and L. Lobo, “Ethical and Security Considerations in Enterprise Agentic AI,” Apress, 2025, pp. 289–323. doi: 10.1007/979-8-8688-1542-3_7.
• S. K. Anuguthala, “Enterprise Agentic AI Lifecycle Governance: A Control-Driven Framework from Design to Decommissioning,” IJAIDSML, vol. 7, pp. 9–16, Jan. 2026, doi: 10.63282/3050-9262.ijaidsml-v7i2p103.
• B. Sateli, F. D. Castillo, and R. Moshtagi, “Towards determining the criticality of AI applications: A model risk management perspective,” Proceedings of the Canadian Conference on Artificial Intelligence, June 2023, doi: 10.21428/594757db.74665221.
• J. Sayles, “The Current State of AI Governance and Model Risk Management,” Apress, 2024, pp. 1–18. doi: 10.1007/979-8-8688-0983-5_1.
• A. Chhabra, S. Datta, S. Nahin, and P. Mohapatra, “Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges,” Oct. 27, 2025, European Society for Socially Embedded Technologies. doi: 10.48550/arxiv.2510.23883.
• A. Abuadbba, N. Sultan, S. Nepal, and S. Jha, “Human Society-Inspired Approaches to Agentic AI Security: The 4C Framework,” Feb. 02, 2026, European Society for Socially Embedded Technologies. doi: 10.48550/arxiv.2602.01942.
• A. K. Sood, Combating Threats and Attacks Targeting The AI Ecosystem. De Gruyter, 2024. doi: 10.1515/9781501520549.
• A. Srivastava and S. Panda, “A Formal Framework for Assessing and Mitigating Emergent Security Risks in Generative AI Models: Bridging Theory and Dynamic Risk Mitigation,” Oct. 15, 2024, European Society for Socially Embedded Technologies. doi: 10.48550/arxiv.2410.13897.
• R. Rajubhai Golani, “The Rise of AI Agents: Transforming Enterprise Automation,” IJAEM, vol. 7, no. 4, pp. 132–139, Apr. 2025, doi: 10.35629/5252-0704132139.
• S. Sarferaz, “Implementing Agentic AI Into ERP Software,” IEEE Access, vol. 13, pp. 178945–178960, Jan. 2025, doi: 10.1109/access.2025.3621887.
• R. Tong, M. Cortês, J. Defalco, M. Underwood, and J. Zalewski, “A First-Principles Based Risk Assessment Framework and the IEEE P3396 Standard,” Mar. 31, 2025, European Society for Socially Embedded Technologies. doi: 10.48550/arxiv.2504.00091.
• R. J. Tong, M. Cortês, J. A. Defalco, M. Underwood, and J. Zalewski, “A First-Principles Based Risk Assessment Framework and the IEEE P3396 Standard,” Institute Of Electrical Electronics Engineers, May 2025, pp. 1588–1595. doi: 10.1109/cai64502.2025.00237.
• M. Fasha, F. A. Rub, N. Matar, B. Sowan, M. Al Khaldy, and H. Barham, “Mitigating the OWASP Top 10 For Large Language Models Applications using Intelligent Agents,” Institute Of Electrical Electronics Engineers, Feb. 2024, pp. 1–9. doi: 10.1109/iccr61006.2024.10532874.
• S. Sadiq, M. Zur Muehlen, and M. Indulska, “Governance, risk and compliance: Applications in information systems,” Inf Syst Front, vol. 14, no. 2, pp. 123–124, Aug. 2011, doi: 10.1007/s10796-011-9320-2.
• R. Cirabisi and K. V. Handal, “Governance of Risk and Compliance,” Apress, 2010, pp. 73–86. doi: 10.1007/978-1-4302-1593-6_6.
• A. Schmitz, M. Akila, D. Hecker, M. Poretschkin, and S. Wrobel, “The why and how of trustworthy AI,” at - Automatisierungstechnik, vol. 70, no. 9, pp. 793–804, Sept. 2022, doi: 10.1515/auto-2022-0012.
• R. Khan, D. Joyce, and M. Habiba, “AGENTSAFE: A Unified Framework for Ethical Assurance and Governance in Agentic AI,” Dec. 02, 2025, American Chemical Society. doi: 10.48550/arxiv.2512.03180.
• K. Huang et al., “AAGATE: A NIST AI RMF-Aligned Governance Platform for Agentic AI,” Oct. 29, 2025, European Society for Socially Embedded Technologies. doi: 10.48550/arxiv.2510.25863.
• L. Chiris and A. Mishra, “AURA: An Agent Autonomy Risk Assessment Framework,” Oct. 17, 2025, American Chemical Society. doi: 10.48550/arxiv.2510.15739.
• A. Xu et al., “Agentic AI for Enterprise: Emerging Applications and Real-world Challenges,” Association for Computing Machinery, Aug. 2025, pp. 6300–6301. doi: 10.1145/3711896.3737871.
• K. Tallam, “A Five-Plane Reference Architecture for Runtime Governance of Production AI Agents,” June 10, 2026. doi: 10.48550/arxiv.2606.12320.
• N. Kshetri, “Governing Agentic AI: Security, Identity, and Oversight in the Age of Autonomous Intelligent Systems,” Computer, vol. 58, no. 8, pp. 123–129, Aug. 2025, doi: 10.1109/mc.2025.3572173.
• R. Khanna et al., “Finding AI’s Faults with AAR/AI: An Empirical Study,” ACM Trans. Interact. Intell. Syst., vol. 12, no. 1, pp. 1–33, Mar. 2022, doi: 10.1145/3487065.
• J. Mchugh, K. Šekrst, and J. Cefalu, “Prompt Injection 2.0: Hybrid AI Threats,” July 17, 2025, American Chemical Society. doi: 10.48550/arxiv.2507.13169.
• C. Xiang et al., “Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks,” Mar. 31, 2026. doi: 10.48550/arxiv.2603.30016.
• Y. Sung, H. Kim, and D. Zhang, “VeriLA: A Human-Centered Evaluation Framework for Interpretable Verification of LLM Agent Failures,” Mar. 16, 2025, European Society for Socially Embedded Technologies. doi: 10.48550/arxiv.2503.12651.
• V. Bamil and R. K. K. Lingamgunta, “A Unified Evaluation and Governance Framework for Trustworthy LLM Agents,” Jan. 09, 2026, Institute Of Electrical Electronics Engineers. doi: 10.36227/techrxiv.176799772.28164151/v1.
• J. Sayles, “Human Oversight of AI Systems,” Apress, 2024, pp. 267–276. doi: 10.1007/979-8-8688-0983-5_12.
• J. Kraprayoon, Z. Williams, and R. Fayyaz, “AI Agent Governance: A Field Guide,” May 27, 2025, American Chemical Society. doi: 10.48550/arxiv.2505.21808.
• M. Kumurdjieva, L. Doukovska, N. Ghouaiel, and A. Dhanani, “Governance of AI and Agentic Systems: Challenges and Methodologies,” Institute Of Electrical Electronics Engineers, Nov. 2025, pp. 1–5. doi: 10.1109/bdkcse67969.2025.11300520.
• D. Dickstein and R. Flast, “Role of Corporate Governance.” Wiley, pp. 245–271, Jan. 02, 2012. doi: 10.1002/9781119200987.ch13.
• J. Edwards and G. Weaver, “Governance, Risk Management, and Compliance.” Wiley, pp. 1–18, Mar. 29, 2024. doi: 10.1002/9781394250226.ch1.
• L. Chrpa, J. Gemrot, and M. Pilat, “Towards a Safer Planning and Execution Concept,” Institute Of Electrical Electronics Engineers, Nov. 2017, pp. 972–976. doi: 10.1109/ictai.2017.00149.
• J. H. Chae and N. Shiri, “Description Logic Framework for Access Control and Security in Object-Oriented Systems,” Springer Science Business Media, 2007, pp. 565–573. doi: 10.1007/978-3-540-72530-5_68.
• S. Chong, “Authorization Contracts,” Association for Computing Machinery, Oct. 2017, p. 75. doi: 10.1145/3139337.3139348.
• H. Errico, “Autonomous Action Runtime Management(AARM):A System Specification for Securing AI-Driven Actions at Runtime,” Feb. 10, 2026, European Society for Socially Embedded Technologies. doi: 10.48550/arxiv.2602.09433.
• A. Wasil, J. Clymer, D. Krueger, E. Dardaman, S. Campos, and E. Murphy, “Affirmative safety: An approach to risk management for high-risk AI,” Apr. 14, 2024, European Society for Socially Embedded Technologies. doi: 10.48550/arxiv.2406.15371.
• D. Bogdanov, P. Etti, L. Kamm, and F. Stomakhin, “Artificial Intelligence System Risk Management Methodology Based on Generalized Blueprints,” Institute Of Electrical Electronics Engineers, May 2024, pp. 123–140. doi: 10.23919/cycon62501.2024.10685644.
• P. K. Dutta, S. Das, and D. Ganguly, “Safeguarding Business in the Age of AI for Organizational Resilience and Risk Management,” Igi Global, 2024, pp. 78–101. doi: 10.4018/979-8-3693-1198-1.ch005.
• L. Sion, D. V. Landuyt, and W. Joosen, “An Overview of Runtime Data Protection Enforcement Approaches,” Institute Of Electrical Electronics Engineers, Sept. 2021, pp. 351–358. doi: 10.1109/eurospw54576.2021.00044.
• C. Bostock, “The Practising Manager’s Control Requirements,” Measurement and Control, vol. 3, no. 2, pp. T30–T32, Feb. 1970, doi: 10.1177/002029407000300205.
• S. Türpe, A. Poller, J. Trukenmüller, J. Repp, and C. Bornmann, “Supporting Security Testers in Discovering Injection Flaws,” Institute Of Electrical Electronics Engineers, Aug. 2008, pp. 64–68. doi: 10.1109/taic-part.2008.7.
• I. B. Utne, T. A. Johansen, and A. J. Sørensen, “Towards Risk-Based Rationality in Autonomous Systems and Operations,” Institute Of Electrical Electronics Engineers, Jan. 2025, pp. 1–6. doi: 10.1109/rams48127.2025.10935192.
• M. Luckcuck and M. Farrell, “Regulating Safety and Security in Autonomous Robotic Systems.,” July 2020, doi: 10.5281/zenodo.3937545.
• T. Opalana, “Integrating AI safety, security, and compliance controls to reduce systemic risk in enterprise AI deployments,” Int. J. Comput. Artif. Intell., vol. 4, no. 2, pp. 71–82, July 2023, doi: 10.33545/27076571.2023.v4.i2a.263.
• A. Pecchia, G. Carrozza, M. Cinque, and D. Cotroneo, “Industry practices and event logging: assessment of a critical software development process,” May 2015, pp. 169–178. doi: 10.5555/2819009.2819035.
• A. Pecchia, M. Cinque, G. Carrozza, and D. Cotroneo, “Industry Practices and Event Logging: Assessment of a Critical Software Development Process,” Institute Of Electrical Electronics Engineers, May 2015, pp. 169–178. doi: 10.1109/icse.2015.145.
• E. Bihari, “Frameworks: How to Build One?,” EDPACS, vol. 57, no. 4, pp. 7–14, Apr. 2018, doi: 10.1080/07366981.2018.1444009.
• D. Dodson, M. Souppaya, and K. Scarfone, “Mitigating the Risk of Software Vulnerabilities by Adopting a Secure Software Development Framework (SSDF),” National Institute Of Standards Technology, Apr. 2020. doi: 10.6028/nist.cswp.04232020.
• N. Z. Jhanjhi, I. A. Shah, and S. N. Brohi, “Evaluating Cybersecurity Frameworks Safeguarding the Software Industry Against Evolving Threats,” Igi Global, 2025, pp. 111–128. doi: 10.4018/979-8-3693-6250-1.ch006.
• K. Raja, “Responsible LLM Systems: Governance, Safety, and Evaluation Frameworks for Enterprise AI Agents,” cfs, pp. 659–669, Feb. 2026, doi: 10.52710/cfs.951.
• P. K. Natta, “Scalable Governance Frameworks for AI-Driven Enterprise Automation and Decision-Making,” JRPETM, vol. 8, no. 6, Nov. 2025, doi: 10.15662/ijrpetm.2025.0806022.