The same task, two ways: what changes when AI can act, not just answer

What does the same task look like, done two ways?

Take something an operations or support team runs constantly: a billing correction has to go out, the customer's account needs updating in two places, and the customer needs a reply. The example below is a plain illustration, not a real customer's account, built to be recognizable rather than measured. Walking through it step by step shows where the two versions diverge.

Done in a chat window: the rep opens the CRM, the system that stores customer accounts, copies the account details into the assistant, and asks it to work out the corrected numbers. The assistant drafts the math and a reply. Then the rep switches to the billing system and re-enters the correction by hand, switches to the CRM and updates the note by hand, copies the drafted reply back out of the chat, and sends it themselves. Four systems touched, one of them by the assistant.

Done through an agent connected to both systems: the rep asks the agent to apply the correction and notify the customer. The agent reads the account, computes the correction, writes it to the CRM and the billing system directly, and drafts the reply. The rep reviews what changed and approves it. Two steps. The one manual step left is a judgment call, not a retyping exercise.

Why does the chat version take four extra steps it looks like it shouldn't need?

The assistant in the chat window did the hard part: it worked out what the correct numbers were. What it couldn't do is write those numbers anywhere. It has no connection to the CRM or the billing system, so the rep becomes the wire between the model's answer and the two places that answer needs to land. Every extra step in the chat version is a manual repeat of work the model already finished once.

What changed in the second version?

Nothing about the model got smarter. What changed is that the agent had a governed connection to the CRM and the billing system, so the same output that used to end as a paragraph in a chat window now ends as a write to both systems, with the rep reviewing the result instead of retyping it. A governed connection is what turned a paragraph into a system write.

Is this just a made-up example, or does it show up in real usage?

It shows up, and at a larger scale than the illustration above. Perplexity's own production-data paper, published with Harvard Business School, compared its Search product against its newer Computer product on sessions where users attempted the same underlying task with both. Computer, which can act rather than only answer, performed 26 minutes of autonomous work per session versus 33 seconds for Search, and on matched tasks it cut completion time from 269 minutes to 36, an estimated 87% reduction in time and 94% in cost. Per-query dissatisfaction was 55% lower. The paper's sharper finding is about scope: a meaningful share of the work Computer did was work essentially absent from Search usage among the same users, composite tasks that cross occupational boundaries and bundle several dependent steps into one request. People took on work they'd never asked a chat window to do at all, since a chat window was never going to finish it anyway.

Does this same gap show up outside this one billing example?

It does, at the scale of a whole company rather than one workflow, and it's the subject of The AI productivity gap: most people who use AI report feeling more productive, and far fewer companies can point to a number that moved because of it. The mechanism is the one this example just walked through. Feeling more productive is what a chat window produces on its own, since drafting faster and getting a question answered are both real without anything else having to happen. A number moving requires the output to land somewhere that shows up in the business, a record updated, a ticket closed, a workflow completed end to end, which is what tool use adds and chatting alone doesn't.

Why does the agent version still keep a person reviewing the diff instead of running unsupervised?

Because reach without scope is a liability, not a shortcut. An agent connected to real systems can act with more reach than a chat assistant, but that reach is exactly what makes an unreviewed mistake more expensive: a wrong number confirmed in a chat window is a bad draft, the same wrong number written to a billing system is a real charge. The review step in the example above exists for that reason, not because the agent computed the correction wrong. Autonomy is worth having once a task has clear boundaries and a defined point where a person checks the result. It doesn't substitute for either.

Does this work for every kind of task?

No. EnterpriseBench, a 500-task benchmark published at EMNLP 2025 that simulates real enterprise conditions, fragmented data sources, access control hierarchies, cross-functional workflows, found that even the strongest models complete only around 42% of those tasks successfully. Enterprise work is harder for an agent than a clean benchmark suggests, precisely because it requires navigating the same access boundaries and system fragmentation a new human hire would need onboarding to handle. The fix is scoping what any given agent or skill can touch, and keeping a human in the loop on the review step, the same way the rep in the example above still approves the diff rather than the agent sending it unreviewed.

Doesn't a simpler approach, like a coding agent with direct API access, already solve this without a broader access layer?

For an engineer writing their own automation, often yes. A recent paper, "Terminal Agents Suffice for Enterprise Automation," found that a coding agent with direct API and filesystem access can match or beat agents routed through curated tool schemas, at lower cost, for engineering-shaped tasks. It's a finding about engineers who can already write to an API directly. The rep in the billing example above isn't an engineer, doesn't have API credentials, and shouldn't need them to fix a billing correction. This post is about what the other 80% of a company does when they can't wire up their own access, not whether a technical user already can.

So what has to be true for someone who isn't an engineer to get the second version of the task at all?

They need the agent they're already using to reach the CRM and the billing system without anyone provisioning API keys by hand, and they need the correction workflow itself to already exist as something repeatable rather than reinvented per person. Two separate things need to be in place: a governed connection between whatever AI client someone uses and the systems the company runs on (Magic MCP), and a shared, versioned way to hand an agent a specific task someone else already worked out how to do correctly (Skills), both provisioned by role through the identity system a company already uses (access control). Neither replaces judgment. Both replace the four extra steps in the example above.

What should change about how a team evaluates this?

Not "did AI help," which almost everyone can already answer yes to. Ask instead whether anything got written back to a system of record without a person manually carrying it there, because that's the difference between a chat window that helped and a workflow that finished. A chat window that drafts a good answer has real value. A workflow that finishes is a different thing entirely.

FAQ

Is the billing example above from a real customer?

No, it's an illustrative walkthrough built to be recognizable, not a measured case study. The data backing the argument, Perplexity's own production study, comes from a verified external source, cited below.

Does giving an agent tool access always produce a big productivity gain?

No. EnterpriseBench's own finding, that even strong models complete only around 42% of realistic enterprise tasks, is why. The gain depends on the task having clear boundaries and a defined path for human review, not on autonomy by itself.

Is this an argument against using chat assistants at all?

No. Chatting is useful for drafting, answering, and reasoning through a problem. The argument is narrower: chatting alone doesn't write anything back to a system of record, and that's specifically the part of the work that shows up in a company's numbers rather than in how productive an individual feels.

Doesn't a technical team already solve this by giving their own agents direct API access?

For engineers automating their own work, often yes, and a recent paper on terminal agents supports that directly. The gap this post addresses is what happens for the much larger group in most companies who aren't engineers and don't have API credentials to begin with.

What's the actual failure rate if a company just turns agents loose on everything?

Meaningful. EnterpriseBench found even top models complete only around 42% of realistic enterprise tasks. That's a reason to scope what an agent can touch and keep a human reviewing the result, not a reason to avoid tool access altogether.

Sources

1. Jeremy Yang, Kate Zyskowski, Noah Yonack, Jerry Ma, "How AI Agents Reshape Knowledge Work: Autonomy, Efficiency, and Scope," arXiv:2606.07489 (checked 2026-09-08)
2. "Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments" (EnterpriseBench), EMNLP 2025 (checked 2026-09-08)
3. "Terminal Agents Suffice for Enterprise Automation," arXiv:2604.00073 (checked 2026-09-08)