VI EN
Claude Code

Computer use in the Claude Code CLI: let the agent click, type and see your macOS screen

Binh 3 min read Đọc bài này bằng tiếng Việt

TL;DR

Computer use lets Claude open apps, click, type and screenshot to verify native apps and GUI-only tools right after writing code. It's the last resort after MCP, Bash and Chrome; approve as few apps as possible, Esc stops it.

Claude Code finishes writing a macOS app, the build succeeds, and then… you have to open the app yourself, click every button and send back screenshots of errors. With native apps, simulators or software without a CLI/API, the agent is "blind" at the verification step. Computer use in the CLI lets Claude open apps, click, type and see the screen like you do, right in the conversation where it wrote the code. It's currently a research preview on macOS for Pro and Max plans.

What can it do?

  • Build and validate native apps: write Swift, compile, launch the app and click through every control before you ever open it.
  • End-to-end UI testing: "test the onboarding flow" of an Electron app without configuring Playwright.
  • Debug visual bugs: shrink the window until the bug reproduces, screenshot it, fix the CSS, check again.
  • Drive GUI-only tools: the iOS Simulator, hardware control panels, design software.

Computer use is the last resort

It's the broadest but slowest option, so Claude prefers more precise tools first:

  1. There's an MCP server for the service → use MCP.
  2. The job can be done with a shell command → use Bash.
  3. It's browser work and Claude in Chrome is set up → use Chrome.
  4. None of the above applies → only then use computer use.

Enable it in 3 steps

  1. In an interactive session, type /mcp and find the computer-use server (off by default).
  2. Choose Enable; the setting is saved per project.
  3. On first use, grant two macOS permissions: Accessibility (click, type, scroll) and Screen Recording (see the screen). You may need to restart Claude Code.
Build the app target, launch it, and click through each tab to make
sure nothing crashes. Screenshot any error states you find.

Safety mechanisms

MechanismEffect
Per-app approval each sessionClaude only controls apps you've allowed in the current session
Warnings for broad-reach appsTerminals/IDEs ("equivalent to shell access"), Finder ("can read or write any file"), System Settings
Control level by app typeBrowsers and trading platforms view-only; terminals/IDEs click-only; everything else full control
Other apps hidden, terminal excluded from screenshotsClaude only sees approved apps, never its own output
Global Esc keyStops immediately from anywhere; the key press is consumed so malicious content can't exploit it
One-session lockOnly one session can control the machine at a time

Screenshots are downscaled automatically before being sent to the model (e.g. 3456×2234 down to about 1372×887). If text is too small, enlarge it in the app rather than changing your display resolution.

Risks to understand

  • Different from the Bash sandbox: computer use runs on your real desktop with the apps you approve. On-screen content (emails, web pages) can contain prompt injection.
  • Don't approve Terminal or Finder for convenience: they extend access to nearly the whole machine.
  • Your machine is busy while Claude works: other apps are hidden; avoid doing sensitive work in parallel.
  • Not available on Team/Enterprise, not usable with -p, and not on Linux/Windows in the CLI yet (on Windows, use the Desktop app).

How to apply it

  • iOS/macOS devs: Add "build, open the app, click through the screens, screenshot errors" to the end of every UI task so Claude verifies its own work.
  • Tech Lead: Use it for smoke-testing desktop apps that have no UI test suite yet; it doesn't replace long-term automated tests.
  • PM/QA: Describe a visual bug in words ("the modal clips on narrow windows") and let Claude reproduce it and capture evidence.

Sources & references

  1. Let Claude use your computer from the CLI — Claude Code Docs
More on Claude Code