Before the deep dive
An AI assistant can identify a suspicious network event, recommend a firewall change, or execute actions through connected tools. Its ability to make a convincing recommendation does not establish that it should be allowed to carry it out.
The crucial distinction is between capability and authority. Machine learning finds useful patterns from training. Generative AI can interpret information and generate actions or explanations. Explicitly specified software can apply known rules. Each can be useful—and each can be wrong.
Three things architects should separate
An intelligent workflow needs more than a capable model. It must distinguish:
Even when each component behaves as designed, the complete system can fail. A model can cite genuine events yet draw the wrong conclusion. A policy engine can legitimately authorize an action based on inaccurate evidence. A reversible configuration change can still interrupt a business process that cannot be instantly restored.
That’s why consequential autonomy should be earned through demonstrated performance and constrained by the likely harm, detectability of errors, reversibility, blast radius, adversarial influence, and the cost of delay. Permissions should be independently enforced and reassessed as conditions change.
Choose one automated or AI-assisted production workflow. Identify the action it could take that would cause the greatest operational harm. Ask whether the evidence behind that action is independently verifiable, whether tool permissions are scoped, and whether the system can change its own limits.
Then decide one boundary to test: assign an accountable owner, define the failure you are protecting against, and verify that the system cannot exceed the authority you’ve granted.
The complete analysis examines AI and ML fundamentals, decision authority, prompt injection, model-evaluation evidence, the Epic Sepsis studies, the CrowdStrike outage, and a production security-response architecture.
GO DEEPER · COMPLETE ARTICLEFull technical analysisAI/ML foundations, authority and governance, case studies, diagrams, and 9 referencesOpenCollapse
As intelligent systems become more capable, the challenge is no longer simply what they can do, but what we should allow them to do.
Imagine an enterprise security platform identifies what it believes is a compromised endpoint. It correlates network activity, examines supporting telemetry, and recommends isolating the device. The proposed action falls within its approved permissions.
The system proceeds, and a critical business application goes offline.
The endpoint wasn’t compromised. The activity was legitimate, the analysis was wrong, and the system had sufficient authority to turn that mistake into an outage.
This is a hypothetical scenario, but the underlying problem is real. A technically permitted action can produce an unacceptable outcome when its supporting assumptions are wrong.
As artificial intelligence becomes more deeply integrated into enterprise technology, we’re moving beyond systems that simply recommend actions toward systems that can execute them. This raises an architectural question that deserves at least as much attention as the capabilities of the models themselves.
How do we determine the appropriate authority to grant a system whose behavior we cannot always fully anticipate?
Answering that question begins with understanding the technologies involved, how their behavior is developed, and how they differ from the software we’ve been engineering for decades.
AI and Machine Learning: Understanding the Relationship
Artificial intelligence is the broader field of developing systems capable of tasks associated with intelligent behavior, including learning, reasoning, planning, perception, and decision-making.
Machine learning is one approach within that field. Rather than explicitly programming every decision rule, developers train models to identify patterns, relationships, or behaviors from data and experience.
For example, a security platform might use predefined signatures to recognize known malicious traffic. A machine-learning model could instead identify suspicious patterns across connection frequencies, destinations, and historical endpoint behavior.
Both can contribute to an intelligent security architecture, but their underlying approaches differ.
Machine learning includes supervised learning, which learns from labeled examples; unsupervised learning, which identifies structure in data; and reinforcement learning, which develops action strategies through interaction and feedback. Self-supervised learning, which derives training signals from the data itself, is particularly important in modern foundation-model development.
Deep learning is a subset of machine learning that uses multilayer neural networks. It underpins many contemporary generative AI applications, including large language models.
Generative AI describes systems that produce content, such as language, code, images, and audio. Agentic AI describes systems capable of pursuing goals through planning, tool use, and sequences of actions. Intelligent agents aren’t a new AI concept, although language-model-based agents are expanding their practical applications.
These categories overlap. AI is broader than ML, deep learning is a subset of ML, and generative and agentic capabilities can be implemented through different combinations of techniques.
AI also includes approaches such as symbolic reasoning, where explicitly represented knowledge supports inference and planning. Conventional programming, while not necessarily AI, remains foundational to the systems in which AI operates.
The significance of these distinctions becomes clearer when we examine how systems arrive at their behavior.
Specified Behavior Versus Learned Behavior
Traditional software and machine learning are sometimes compared as deterministic versus probabilistic systems. That distinction can be misleading.
A trained machine-learning model can produce reproducible outputs when its inputs, parameters, and execution conditions are controlled. Traditional software can also exhibit nondeterministic behavior because of concurrency, external dependencies, and other operating conditions.
The more meaningful distinction is between behavior that is explicitly specified and behavior derived through learning.
Consider a network access policy requiring an authenticated user, a compliant device, and an approved application.
Those conditions can be encoded directly into policy logic. Engineers can inspect the requirements, test the implementation, and evaluate whether its behavior matches the intended policy.
Now consider detecting compromised endpoints based on thousands of behavioral signals. Attempting to explicitly define every suspicious combination would be difficult, perhaps impractical. Machine learning can identify relationships that would be challenging to represent manually.
Neither approach is universally more accurate.
Spam filtering, speech recognition, and image recognition illustrate where learning from data has enabled capabilities that would be difficult to reproduce with conventional rules.
Security anomaly detection is a more contested application. Unusual activity is common in real networks, malicious events are comparatively rare, and attackers adapt. Learned detectors can struggle with false positives, distribution changes, and operational relevance. Sommer and Paxson examined several fundamental challenges in applying machine learning to operational network intrusion detection in their influential 2010 research. [1]
Explicitly specified systems offer a different advantage: their intended behavior can often be traced to requirements, examined, and verified more directly.
That matters in safety-critical engineering, where requirements traceability, verification, and assurance play essential roles.
Verifiability, however, doesn’t guarantee correctness. A fully specified system can implement the wrong requirement or make decisions based on faulty inputs.
The appropriate approach depends on what must be learned, what can be specified, and how confidently the resulting behavior can be validated.
The distinction that matters
Rules can be inspected
Requirements and decision conditions are described explicitly.
Key strength: Traceability and requirements-based verification.
Watch for: Incorrect rules, blind spots and unforeseen interactions.
Patterns come from data
Models learn relationships that may be difficult to represent with hand-written logic.
Key strength: Pattern recognition in complex settings.
Watch for: Data quality, drift and deployment-specific performance.
Intelligence Is a Capability. Authority Is an Architectural Decision.
Modern AI models can classify information, reason through technical problems, generate executable code, and coordinate tools to complete complex tasks. Those capabilities create genuine opportunities for automation.
Imagine an AI assistant analyzing firewall configurations. It identifies an unnecessarily permissive rule, generates a replacement, and checks the proposal against available documentation.
The recommendation might be correct. The model might even demonstrate excellent performance across extensive testing.
Yet deploying the change introduces additional requirements. The system must account for application dependencies, incomplete telemetry, maintenance restrictions, and the possibility of disrupting legitimate traffic.
The risk comes not only from the correctness of the decision, but from the authority attached to it.
This principle isn’t new. Least privilege, defense in depth, separation of duties, and safety engineering have long addressed the relationship between system behavior and operational authority.
What modern AI changes is how flexibly software can interpret information and assemble actions.
Traditional automation often follows workflows and conditions defined in advance. An LLM-based agent may choose which tools to invoke, determine their parameters, and assemble a sequence of operations at runtime. Traditional software can also construct dynamic workflows, but language-model-based agents may introduce greater variability in the paths selected from natural-language goals and unstructured context.
The difficulty is establishing appropriate limits when not every possible action sequence has been individually reviewed.
A related challenge arises because language models process instructions and external information within a shared input context. Conventional software can enforce structural boundaries between executable instructions and data. Parameterized queries and memory protections illustrate different mechanisms for maintaining such boundaries, although their failure or circumvention can enable injection or memory-safety vulnerabilities. Language models lack an equivalently reliable built-in boundary between trusted instructions and attacker-controlled text presented in their input context. [2]
A malicious log entry, document, or message may attempt to redirect a model’s behavior.
This resembles the confused deputy problem: a component with legitimate privileges is induced to exercise those privileges on behalf of an unauthorized party.
Traditional systems can also suffer confused-deputy vulnerabilities. Language-model-based systems introduce new ways for untrusted information to influence an authorized workflow.
As models become more capable, independently enforced authority boundaries become increasingly important.
Seven Considerations for Granting Decision Authority
A system’s ability to perform a task is only one part of determining whether it should perform that task autonomously. A practical assessment should examine seven considerations.
How well does the system perform this particular task in the environment where it will operate? Evaluation should use metrics appropriate to the decision. In security detection, precision, recall, false-positive rates, and score calibration may be more meaningful than overall accuracy. Evidence should reflect the actual operating threshold, asset population, environment, and consequences of errors.
What happens when the system is wrong? A poor document recommendation may waste a few minutes. An incorrect security action might interrupt production services or create an availability incident. Consequences should be assessed against the complete business workflow, not merely the immediate technical action.
How quickly and reliably can an incorrect action be identified? A system whose failures are immediately visible presents different risks from one whose errors remain hidden. Where practical, detection of erroneous behavior should rely on independent monitoring rather than exclusively on the component responsible for the decision.
Can we reverse the action and recover from its consequences? A configuration rollback doesn’t necessarily restore interrupted transactions, lost availability, or downstream business processes. Operational recovery matters more than simply returning configuration state to its previous value.
What happens if the system waits? During a rapidly spreading cyberattack, delaying containment may increase damage. An automated, tightly scoped response could sometimes be safer than waiting for human authorization. That trade-off depends on the demonstrated reliability of the decision and the consequences of a false response.
How many systems, users, or processes can a single decision affect, and could multiple failures occur together? An automated isolation action affecting one workstation may be manageable. The same logic executing simultaneously across thousands of endpoints can become a business-wide outage. Staged execution, per-period action limits, and restrictions on critical asset classes help constrain correlated failure.
Can an attacker manipulate the system’s inputs or deliberately trigger its actions? A defensive workflow that automatically isolates endpoints may itself become an attack surface. An adversary might generate conditions designed to make the organization disrupt its own services. Systems operating in adversarial environments require evaluation against deliberate manipulation, not only normal operational errors.
These considerations are not an established numerical formula or a substitute for formal risk assessment. They are a practical architectural decision aid.
Two additional principles apply across all seven.
First, authority must be evaluated over complete sequences of actions. Individually permitted operations can combine into an unacceptable outcome. Permission to read sensitive files and permission to send external messages may each appear reasonable, while the combination creates a data-exfiltration path.
Second, the evidence supporting authority must remain valid over time.
A model update, changed operating conditions, deteriorating performance, or newly discovered attack technique may invalidate an earlier evaluation. Systems should be able to restrict or suspend autonomous behavior when the assumptions supporting it no longer hold.
Authority can also be dynamic. A system may receive narrowly scoped permissions based on current risk, asset criticality, or verified operating conditions, provided the process granting those permissions is itself constrained and auditable.
Autonomy is a continuum rather than a binary choice. A system may gather telemetry independently, recommend corrective actions, execute low-impact remediation, and escalate higher-consequence decisions. The appropriate boundary depends on both the system’s demonstrated performance and the consequences of granting it authority.
Who Governs the Authority to Act?
Dynamic authorization raises a further question: what constrains the mechanism granting permission?
An AI system may receive different permissions based on current risk, operating conditions, or the sensitivity of a resource. Such grants can be automated, but the granting mechanism must operate within boundaries established independently of the agent. Those boundaries include which permissions may be issued, what conditions justify them, how long they remain valid, and who may change the governing policies.
An agent could request temporary access to a remediation tool without being able to rewrite the policy deciding whether that request is approved. Policy changes should follow established review and change-control processes, with separation of duties, audit records, and a way to revoke grants. An automated policy decision point may issue permissions dynamically; the agent exercising them should not be able to unilaterally alter its governing rules.
Ultimately, identifiable people and organizational processes must own those boundaries. An auditable grant records what occurred, but accountability also requires someone authorized to approve the risks and correct the policy when it fails.
One emerging proposal addressing this problem is the Kinetic Trust Protocol (KTP), a draft specification that explores deriving an agent’s permitted autonomy from changing environmental conditions. Its governance specification proposes a recursive constraint: entities empowered to modify the governing rules must themselves operate within defined constraints. It also describes bounded governance roles, auditable decisions, and procedures for changing those rules. [9]
KTP is a proposal, not an established industry standard or evidence that those protections have been proven in production. It illustrates the underlying design question: authority over the policy must itself be governed, regardless of whether an agent or a human requests a change.
From Detection to Action: Applying the Principles
Let’s revisit our hypothetical endpoint-isolation scenario through a proposed security architecture.
Detection
Signature-based rules and machine-learning models can identify suspicious activity using different forms of evidence. A behavioral model may identify deviations from an endpoint’s normal activity, while explicitly defined rules detect known malicious indicators.
An anomaly is not proof of compromise. In an enterprise environment, legitimate events generally outnumber malicious ones by a substantial margin. Even modest false-positive rates can therefore produce large alert volumes.
Interpretation
A language-model-based assistant can correlate telemetry, summarize findings, and suggest investigative steps.
However, security telemetry contains attacker-influenceable data. Filenames, log entries, DNS names, command lines, and other fields must be treated as untrusted, even when retrieved through trusted tools.
Prompt-injection defenses therefore require controls enforced outside the language model.
A component interpreting untrusted material should not automatically possess unrestricted access to consequential tools. Constrained tool interfaces, independent policy enforcement, and separation of privileges can reduce the impact of manipulated model behavior.
Research such as Defeating Prompt Injections by Design, which introduces the CaMeL architecture, explores how data-flow tracking and privilege separation can constrain actions influenced by untrusted information. [2]
Those protections have important limitations. CaMeL’s authors explicitly exclude certain attacks that alter a model’s textual conclusions without changing tool or data flows. Their work does not establish that prompt injection has been universally solved.
An attacker may manipulate an assistant into concluding that malicious activity is benign, incorrectly identifying an endpoint, or overlooking important evidence. Constraining what a compromised model can execute does not guarantee the correctness of what it says.
Two checks with different jobs
Corroborate the specific events and values cited in an AI-generated explanation. Interpretive conclusions can still be wrong.
Enforce tool permissions and action scope independently of the model’s interpretation.
Verification
Before a consequential response, the workflow should independently validate relevant factual claims.
If the assistant reports that an endpoint contacted a suspicious destination, a verification component can confirm that cited events exist and contain the specific addresses, timestamps, or other values being reported.
Determining whether those events establish compromise is a separate judgment. That judgment may require independent telemetry, another analytical method, or human investigation.
An independent verifier can also make mistakes, particularly when it uses similar models or evidence. Verification should therefore be proportionate to the consequences of the action.
In security operations, even the alerts that initiate an investigation may be influenced by an attacker. The design must rely on corroboration, evaluated detection quality, and bounded consequences rather than assuming that an apparently trusted source is always correct.
Authorization
After evaluating the evidence, the system determines whether the proposed action is permitted.
Endpoint isolation might be allowed for ordinary workstations but restricted for critical servers. Authorization can account for asset classification, operator privileges, response type, and permitted scope.
The policy engine itself depends on accurate inputs. A stale asset inventory can cause correctly executed policy logic to produce an inappropriate authorization.
Authorization should account for combinations of actions, not simply individual API calls. Controls should limit opportunities for a system to circumvent restrictions through sequences of operations that appear permissible when evaluated independently.
Execution and Recovery
Approved actions should execute within defined limits, with monitoring, audit logging, rate controls, and recovery mechanisms.
For high-impact actions, failed verification should normally block execution or require an explicitly authorized exception. Situations where delay creates greater risk should have predetermined emergency procedures rather than relying on improvised exceptions.
Staged deployment and action limits can prevent one incorrect conclusion from becoming a widespread outage.
This is a proposed reference architecture, not a description of how every security operations center currently operates.
Returning to the opening scenario, several safeguards could have reduced the risk. Independent corroboration might have challenged the original diagnosis. Accurate asset classification could have identified the critical application. Scoped authorization could have required additional approval. Execution limits could have prevented simultaneous isolation of related systems.
Each safeguard addresses a different failure mode. Together, they make it less likely that a single analytical error becomes an enterprise outage.
Real-World Lessons: Measuring the Right Things
The need to evaluate systems against their intended purpose extends well beyond cybersecurity.
A 2021 study in JAMA Internal Medicine examined the Epic Sepsis Model, a predictive system intended to help identify hospitalized patients at risk of sepsis. [3]
Researchers at Michigan Medicine retrospectively evaluated 38,455 hospitalizations.
At the evaluated alert threshold, the model failed to identify 67% of hospitalizations involving sepsis while hypothetically producing alerts for 18% of hospitalizations.
Its positive predictive value was 12%, corresponding to approximately eight patients needing evaluation for every one who developed sepsis.
The study reported an area under the receiver operating characteristic curve of 0.63, below performance previously reported by the model developer.
A critical issue involved the timing of predictions. The researchers excluded scores calculated after sepsis had already occurred. When they included scores extending up to three hours after onset, measured discrimination improved to 0.80.
The authors identified differences in handling sepsis timing as a likely explanation for the discrepancy.
Why the measurement window matters · 2021 study
Reported AUROC when model scores after sepsis onset were excluded.
Measured discrimination increased when some post-onset scores were included.
This demonstrates why evaluation methodology matters. A system that detects a condition after clinicians have already recognized it may perform acceptably on some metrics without providing the intended benefit of earlier identification.
The study was retrospective, took place at one academic health system, and evaluated an earlier version of the model.
Epic subsequently introduced a second-generation model.
A February 2026 multicenter study evaluated version 2 across four US health systems, with the model locally fine-tuned at each site. It reported stronger discrimination than version 1 under the study’s evaluation methods, while identifying variation across institutions and continuing challenges involving positive predictive value and alert burden. [4]
The two studies used different methodologies and outcome definitions, so their performance figures should not be compared as though they represented a single controlled experiment.
The broader lesson is that evaluation must measure the outcome a system is intended to improve, under the conditions in which it will actually operate. For autonomous systems, that evidence becomes part of the justification for granting authority.
The Risk Isn’t Exclusive to AI
The July 2024 CrowdStrike outage offers a different example.
A faulty Rapid Response Content configuration update caused affected Windows systems running Falcon sensors to crash. [5] Microsoft estimated that approximately 8.5 million Windows devices were impacted. [6]
CrowdStrike’s published analysis identified problems involving content validation and processing. The faulty content related to behavioral heuristics and was distinct from the sensor’s on-device AI capabilities. The problematic update was deployed and reverted within roughly 78 minutes. [5]
Yet recovering affected machines and restoring dependent business services extended well beyond that window. [7]
CrowdStrike subsequently documented remediation measures addressing testing, staged deployment, validation, and monitoring. [5]
This wasn’t an AI model autonomously deciding to disable systems.
It illustrates how conventional software and automated deployment mechanisms can produce correlated failures when changes reach privileged components across large numbers of systems.
It also illustrates why technical rollback and operational recovery are different concepts.
Two different recovery clocks
The same architectural responsibilities apply whether the initiating decision comes from a human-written rule, an automated deployment process, or a learned model.
Engineering Systems We Can Trust
Intelligent architectures need more than individually high-performing components. They require evaluation of the complete workflow.
That begins with realistic testing, shadow-mode deployments, limited rollouts, and independent monitoring before consequential authority expands.
Production controls should enforce least privilege, constrain tool access, and provide fallback behavior when AI services are unavailable or outputs fail validation.
Organizations must also understand what information their models receive, how that information is processed, and whether sensitive operational data crosses organizational or security boundaries.
Performance requires continued evaluation. Data distributions change, model providers update systems, and adversaries adapt. Permission to act should be revisited when the evidence supporting it changes.
Feedback loops deserve particular attention. If a model influences which events analysts investigate, and those investigations later become training data, the system may reinforce its existing blind spots.
Human oversight must also be designed rather than assumed. Reviewers need relevant evidence, manageable alert volumes, and clearly defined responsibility. Approval and override rates can help reveal whether oversight remains meaningful.
These practices align with established AI risk-management guidance, including NIST’s AI Risk Management Framework and its emphasis on trustworthy system characteristics. [8]
A critical architectural principle is that permissions and safeguards must be enforced independently of the model they constrain.
AI can assist engineers in drafting and testing policies, but consequential policies should be reviewed, versioned, and enforced through mechanisms the model cannot unilaterally modify.
Accountability Cannot Be Delegated to a Model
Technical controls define what a system is permitted to do; they do not, on their own, establish who is answerable for the consequences. An organization should assign responsibility for policy, system operations, and accepted residual risk.
Defines permissible actions, escalation thresholds, and who can change the rules.
Maintains deployment, monitoring, operational controls, and incident recovery.
Accepts the remaining risk within the organization’s governance and approval structure.
These responsibilities can overlap, and their assignment depends on the organization. What matters is that they are explicit. A security organization might own response policy, an operations team might run the automation, and an accountable service or business owner might approve the risks to a critical application.
An AI system can execute actions within delegated authority, but organizational accountability for granting that authority, maintaining its limits, and responding to failures remains with the people and institutions governing it.
Perhaps most importantly, the complete architecture must be evaluated as one operational system.
A model may classify an event correctly while an orchestration component invokes the wrong tool. A policy engine may enforce authorization perfectly using stale information. A configuration change may execute successfully while creating an unacceptable business outcome.
Each component may function as intended while the overall system fails to achieve its purpose.
That is the level at which architecture must ultimately succeed.
The Decision That Matters
Advances in machine learning and generative AI will continue expanding what software can accomplish. Systems will become better at interpreting environments, generating policies, evaluating alternatives, and carrying out multistep operations.
Some tasks that currently require multiple components may eventually be consolidated into more capable models.
But improving model capability doesn’t automatically establish appropriate operational authority.
Architects still need evidence that a system works under the conditions where it will operate, boundaries that restrict the consequences of failure, and mechanisms that remain effective when assumptions change.
A system’s intelligence influences what it is capable of doing. Architecture establishes what it is permitted to do, under which conditions, and how an organization retains accountability when something goes wrong.
The more capable our systems become, the more consequential that decision will be.
References
Sommer, R., and Paxson, V. (2010). Outside the Closed World: On Using Machine Learning for Network Intrusion Detection. IEEE Symposium on Security and Privacy. https://doi.org/10.1109/SP.2010.25
Debenedetti, E., et al. (2025). Defeating Prompt Injections by Design. https://arxiv.org/abs/2503.18813
Wong, A., et al. (2021). External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Internal Medicine, 181(8), 1065–1070. https://doi.org/10.1001/jamainternmed.2021.2626
Wong, A., Currey, D., Schwinne, M., et al. (2026). Multicenter Prospective Validation of an Updated Proprietary Sepsis Prediction Model. JAMA Network Open, 9(2), e260181. https://doi.org/10.1001/jamanetworkopen.2026.0181
CrowdStrike (July 24, 2024). Preliminary Post Incident Review (PIR): Content Configuration Update Impacting the Falcon Sensor and the Windows Operating System (BSOD). https://www.crowdstrike.com/en-us/blog/falcon-content-update-preliminary-post-incident-report/
Microsoft (2024). Helping Our Customers Through the CrowdStrike Outage. https://blogs.microsoft.com/blog/2024/07/20/helping-our-customers-through-the-crowdstrike-outage/
CrowdStrike (2024). Channel File 291 Incident: Root Cause Analysis Is Available. https://www.crowdstrike.com/en-us/blog/channel-file-291-rca-available/
National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). https://www.nist.gov/itl/ai-risk-management-framework
New Mexico Cyber Intelligence & Threat Response Alliance. Kinetic Trust Protocol (KTP) — Governance Specification. Draft specification (repository accessed October 2026). https://github.com/nmcitra/ktp-rfc/blob/main/rfcs-md/ktp-governance.md
Leave a Reply