Guardrails should make the safe path obvious
A policy that says “use AI responsibly” leaves every important decision to the moment of pressure. Teams need specific boundaries they can apply during design, development, review, and operation.
Engineering organizations are adopting AI in two related but different ways. Developers use assistants to understand code, draft tests, and accelerate routine work. Product teams embed models into customer-facing workflows, internal operations, and automated decisions.
Both uses create value, but they do not have the same risk. Guardrails should therefore be proportional to the data involved, the authority given to the model, and the consequence of an incorrect output.
The following guidelines provide a practical baseline. They should be adapted to the organization’s legal obligations, customer commitments, and risk tolerance.
Classify the use case before selecting controls
Every AI use case should have a named owner and a simple risk classification. The classification should consider:
- Whether the model receives public, internal, confidential, regulated, or personal data.
- Whether output is advisory or directly changes production systems.
- Whether a person reviews the output before it has an effect.
- Whether errors are reversible and detectable.
- Whether the system affects employment, finance, health, safety, legal rights, security, or access to essential services.
- Whether third parties retain prompts, outputs, or uploaded files.
A low-risk use case might summarize public documentation for an engineer. A high-risk use case might recommend access decisions using customer data. The second case requires stronger approval, evaluation, monitoring, and human oversight even if both use the same model.
Define non-negotiable data boundaries
Teams should know what may and may not be sent to an AI provider. At minimum:
- Do not place credentials, private keys, access tokens, or production secrets in prompts.
- Do not submit customer data, personal data, source code, or internal documents unless the approved service and contract permit that data class.
- Use enterprise-managed accounts when organizational data is involved.
- Apply least privilege to connected repositories, tools, data sources, and retrieval indexes.
- Define retention, training-use, residency, deletion, and incident-notification expectations with providers.
- Redact or minimize data before it reaches the model whenever the full record is unnecessary.
System prompts are not access controls. If the model must not see a secret, the application should not place that secret in its context.
Keep model output untrusted by default
AI output is generated input from the application’s perspective. It should be validated before it reaches interpreters, databases, browsers, shells, APIs, or other privileged systems.
That means using schemas for structured output, allowlists for tools and operations, parameterized database access, output encoding, and ordinary input validation. Generated code must pass the same review, testing, dependency, and security controls as human-authored code.
Prompt injection deserves explicit treatment. Content retrieved from websites, documents, tickets, email, or source repositories can contain instructions that conflict with the application’s purpose. The application should separate trusted instructions from untrusted content and minimize the permissions available to the model even when the prompt appears well formed.
Put human approval at meaningful boundaries
“Human in the loop” is useful only when the person has enough context, time, and authority to reject the recommendation. A confirmation button after an opaque decision is not meaningful oversight.
Require explicit human approval before AI can:
- Merge or deploy code.
- Change production data or infrastructure.
- Send external communications on behalf of a person or organization.
- Approve financial, legal, employment, identity, or access decisions.
- Publish customer-visible claims that require factual accuracy.
- Execute destructive or difficult-to-reverse actions.
Lower-risk, reversible actions may be automated after evaluation demonstrates that the controls are effective. The escalation path should be designed before automation is expanded.
Evaluate the system, not just the model
A model benchmark does not describe how the complete application behaves with its prompts, retrieval data, tools, policies, and users. Teams need evaluations that reflect the actual workflow.
A useful evaluation set includes normal cases, ambiguous requests, prohibited requests, adversarial instructions, stale knowledge, malformed tool responses, and dependency failures. Measure task quality as well as policy compliance.
Record the model and prompt version associated with each evaluation. Run the suite before changing models, instructions, retrieval configuration, or tool permissions. For consequential workflows, include independent review by the domain experts who understand the harm an incorrect answer can cause.
Make runtime behavior observable
Production controls should answer more than whether the model endpoint is up. Monitor:
- Policy refusals and attempted prohibited actions.
- Tool calls, authorization failures, and approval outcomes.
- Retrieval sources and grounding coverage.
- Latency, cost, rate limits, and provider errors.
- User corrections, escalations, and reported harmful output.
- Changes in task success after model or prompt updates.
Logs should support investigation without becoming a second uncontrolled store of sensitive prompts and outputs. Apply retention, redaction, access control, and audit requirements to telemetry as well.
Maintain an AI system record
Each production AI capability should have a concise record covering its owner, purpose, approved users, data classes, provider and model, connected tools, evaluation evidence, known limitations, monitoring, fallback behavior, and incident contact.
This record turns governance into an engineering artifact. It also makes model changes reviewable instead of allowing a provider update or configuration edit to silently alter system behavior.
Use a clear incident and shutdown path
Teams need the ability to disable model calls, individual tools, or the entire feature without redeploying unrelated application code. Incident procedures should address data exposure, unsafe output, unexpected autonomous behavior, provider compromise, and evaluation regression.
The response should preserve evidence, notify the appropriate owners, assess affected users and data, and define the conditions for re-enabling the feature.
A practical review checklist
Before an AI capability moves into production, confirm that:
- The purpose, owner, users, and prohibited uses are documented.
- Data handling is approved for the selected provider and model.
- Tools use least privilege and consequential actions require approval.
- Outputs are validated and treated as untrusted.
- Evaluations cover quality, misuse, injection, and failure behavior.
- Monitoring and user feedback reach an accountable team.
- A fallback and shutdown mechanism has been tested.
- Model, prompt, data, and permission changes follow change control.
Closing view
Good guardrails do not remove judgment. They place judgment where it is most valuable: when selecting a use case, granting authority, approving data access, evaluating behavior, and responding to evidence.
Organizations can use the NIST AI Risk Management Framework and its Generative AI Profile to organize governance, mapping, measurement, and management work. The OWASP Top 10 for LLM and Generative AI Applications is a useful companion for application security risks such as prompt injection, sensitive information disclosure, and improper output handling.