Agentic AI promises to do more than analyze information or generate an answer. It can retrieve additional context, interact with connected systems, construct workflows, and take actions on a person’s behalf.
That ability creates real value. It can also create real consequences.
A system that summarizes a policy presents one level of risk. A system that changes financial information, approves an expense, deletes a record, or initiates a payment presents another.
Finance and IT leaders should not evaluate these systems based only on what they can do. They also need to understand what the AI is permitted to do, how its actions are controlled, and how people can intervene when something goes wrong.
Trusted agentic AI does not require blind trust. It gives people the evidence and controls they need to use it responsibly.
“”Trustworthy agentic AI needs more than a capable model. People need a way to verify what it is doing, and the guardrails must sit outside the agent so they still hold when the agent makes a mistake.
Seth Webster
Senior AI Engineer
Emburse
Use this checklist when evaluating an AI vendor, designing an agentic workflow, or deciding whether an existing use case is ready for greater autonomy.
Is the agent’s authority clearly defined?
Before an AI agent can act, the organization must define the area in which it is allowed to operate.
“Help with expense management” is not a sufficient boundary. The system needs a specific scope that distinguishes between reading information, recommending an action, preparing an action, and completing one.
For example, an expense agent might be permitted to:
- Read a receipt
- Extract and classify expense information
- Match a receipt to a card transaction
- Suggest a business purpose
- Identify a potential policy issue
- Prepare a report for employee review
That does not automatically mean it should be allowed to:
- Approve its own recommendation
- Change accounting information without review
- Delete supporting records
- Override company policy
- Initiate a reimbursement or payment
Questions to ask
- What actions can the agent take?
- Which actions can it recommend but not complete?
- Which actions always require human confirmation?
- Can its authority vary by role, transaction type, amount, or risk?
- Who can change these permissions?
Warning sign
The vendor describes the agent as fully autonomous but cannot provide a clear list of its permitted actions.
Ready when
Each capability has a defined level of authority, an accountable owner, and a documented approval requirement.
Status: ☐ Ready ☐ Needs work ☐ Not addressed
Does the agent inherit the user’s permissions?
An AI agent should not gain broader access simply because it can retrieve information more efficiently.
If an employee cannot view another business unit’s expenses, the agent acting for that employee should not be able to view them either. If a manager cannot approve a particular transaction, the agent should not be able to approve it on the manager’s behalf.
This restriction should be enforced through the underlying permissions and APIs. It should not depend on the AI remembering or interpreting a written instruction correctly.
Questions to ask
- Does the agent operate within the permissions of the authenticated user?
- Can it access data the user could not access directly?
- Are permissions enforced at the API and system level?
- What happens if the agent requests information outside the user’s scope?
- Are access attempts recorded for later review?
Warning sign
The AI has broad system access, and the vendor relies on prompts or model instructions to prevent improper retrieval.
Ready when
The agent cannot obtain, change, or disclose anything beyond the authenticated user’s established permissions.
Status: ☐ Ready ☐ Needs work ☐ Not addressed
Are guardrails enforced outside the AI model?
AI models can make mistakes. They may misunderstand an instruction, use incomplete information, or choose an inappropriate next step.
For that reason, the model should not be responsible for deciding whether its own action is allowed.
External controls should enforce spending limits, access rights, approval requirements, data restrictions, and prohibited actions. The AI can propose an action, but the surrounding system determines whether that action may proceed.
Think of the model as the reasoning layer, not the final authority.
Questions to ask
- Which controls exist outside the AI model?
- Can the agent override a system rule?
- What happens when the model’s recommendation conflicts with company policy?
- Are financial thresholds enforced independently of the AI?
- Can administrators block an action category entirely?
Warning sign
The vendor says the AI has been instructed not to perform restricted actions but cannot demonstrate a separate technical enforcement layer.
Ready when
Critical boundaries remain effective regardless of what the agent decides, generates, or attempts.
Status: ☐ Ready ☐ Needs work ☐ Not addressed
Do consequential actions require explicit verification?
Not every AI action carries the same risk.
Suggesting an expense category is low-impact and reversible. Deleting financial records, approving a high-value expense, changing payment information, or moving money creates greater consequences.
The level of verification should rise with the level of risk.
For significant actions, the system needs an unambiguous confirmation mechanism. A casual response buried in a conversation should not be enough to trigger a payment or destructive action.
The confirmation itself must also sit outside the agent’s discretion. When a person says no, the system must block the action regardless of what the AI believes the person intended.
Questions to ask
- Which actions require confirmation?
- Is confirmation separate from the agent’s interpretation of the conversation?
- Can the user inspect the proposed action before approving it?
- Does the confirmation show the amount, recipient, affected records, and other material details?
- Can an approval be withdrawn before execution?
Warning sign
The agent can infer approval from natural-language conversation without a separate confirmation step.
Ready when
High-impact actions require clear, enforceable, and recorded authorization from the appropriate person.
Status: ☐ Ready ☐ Needs work ☐ Not addressed
Can the agent explain what it did and why?
Convenience should not come at the cost of understanding.
An agentic system will sometimes hide complexity so people can work at a higher level. That is one of its main advantages. But the underlying details still need to be available when someone needs to inspect them.
If an expense is flagged, the system should identify the relevant concern. If a workflow cannot be completed, it should explain the constraint. If the agent changed a recommendation after gathering new information, the user should be able to understand what changed.
Good explanations help people distinguish among three very different situations:
- The agent made a mistake
- The user’s request was unclear or incomplete
- The requested action was impossible because of a policy, permission, or system constraint
Questions to ask
- Can users see which information influenced the decision?
- Does the agent explain why an item was flagged or routed?
- Can users inspect the steps taken across connected systems?
- Are explanations written for the business user, not only for technical teams?
- Can the system explain when and why it could not complete a request?
Warning sign
The system produces a decision or action without giving the user meaningful supporting context.
Ready when
Users can understand the outcome, inspect relevant details, and challenge or correct the result.
Status: ☐ Ready ☐ Needs work ☐ Not addressed
Is there a complete and accessible audit trail?
Agentic AI should not create a gap in accountability.
Every material action should leave a record showing what happened, when it happened, who initiated the request, which identity and permissions the agent used, and whether a person approved the final action.
The audit trail should include agent-prepared actions as well as completed ones. This helps finance, compliance, security, and internal audit teams reconstruct the full decision path rather than seeing only the end result.
Questions to ask
- Are the agent’s actions logged?
- Does the record distinguish between AI recommendations and human decisions?
- Can reviewers see who confirmed a consequential action?
- Are changes to permissions and guardrails recorded?
- How long are audit records retained?
- Can the organization export records for compliance or investigation?
Warning sign
The system records only the final transaction and not the agent’s role in preparing, recommending, or executing it.
Ready when
The organization can reconstruct the action from initial request through final outcome.
Status: ☐ Ready ☐ Needs work ☐ Not addressed
Can people intervene without supervising every step?
Human control does not mean placing a person in front of every routine action.
Doing so would preserve much of the manual work the agent is meant to remove.
The better model allows people to remain at a high level during normal activity, while making intervention easy when a decision, exception, or sign of risk requires attention.
Users should not be forced into every detail. They also should not be prevented from accessing those details.
A well-designed system helps people recognize when they need to examine the work more closely. It surfaces unusual outcomes, low-confidence decisions, policy conflicts, and high-risk actions instead of treating every transaction the same way.
Questions to ask
- When does the system bring a person into the workflow?
- Can thresholds for human review be configured?
- Are low-confidence or unusual results surfaced automatically?
- Can a person pause, reverse, or redirect the agent?
- Can users inspect details on demand without being forced through them every time?
- Does the workflow assign exceptions to the right employee, approver, auditor, or administrator?
Warning sign
The design offers only two modes: complete autonomy or manual approval of every step.
Ready when
Routine work can move forward efficiently, while people retain practical control over exceptions and consequential decisions.
Status: ☐ Ready ☐ Needs work ☐ Not addressed
Can trust increase gradually?
Organizations should not have to make a single permanent decision about whether an AI system is trustworthy.
Trust can be earned through observed performance.
An organization might begin by allowing the agent to prepare work while requiring a person to approve every action. After measuring accuracy across a defined period, it may allow low-risk, high-confidence activity to proceed automatically.
Higher-risk actions can continue to require approval. New workflows can return to a more supervised state until their performance has been validated.
This graduated model balances convenience and control. It also recognizes that trust can differ by workflow. A system may perform consistently when matching receipts but require closer review when interpreting complex policy exceptions.
Questions to ask
- Can autonomy be increased by workflow, action, or risk level?
- Does the platform report accuracy and exception rates?
- Can teams review where the agent required correction?
- Is there a controlled way to test new capabilities?
- Can automated actions return to manual review when performance changes?
- Are users able to submit feedback and corrections?
Warning sign
The system provides no practical way to measure performance or adjust the level of autonomy after deployment.
Ready when
The organization can start with limited authority, evaluate results, and expand automation only where the evidence supports it.
Status: ☐ Ready ☐ Needs work ☐ Not addressed
Readiness score
Give the system one point for every item marked Ready.
7–8 points: Ready for a controlled deployment
The system has strong foundations for bounded agentic action. Continue testing specific workflows, particularly those involving financial or destructive actions.
4–6 points: Promising, but important gaps remain
The system may be suitable for recommendations, data retrieval, or supervised workflows. Resolve the missing controls before increasing its authority.
0–3 points: Keep the agent in advisory mode
The system may still provide value by summarizing information or preparing recommendations, but it should not be trusted to complete consequential actions.
The final test
Before giving an AI agent more control, ask one straightforward question:
If the agent makes the wrong decision, can the system still protect the organization and keep a person in control?
If the answer depends on the AI recognizing and correcting its own mistake, the safeguards are not strong enough.
Trustworthy agentic AI should help finance teams operate at a higher level. It can connect information across systems, translate business questions into complex queries and workflows, and reduce the technical effort required to reach an answer.
But the force multiplier comes from combining action with accountability.
The agent should handle complexity without hiding control. It should reduce human effort without removing human authority. And it should earn greater autonomy through evidence, not promises.
Explore how Emburse approaches secure, governed AI for complex finance operations.