Computer use

Let agents see and drive the desktop apps on your computer, with a confirmation before every sensitive action.

Computer Use lets an agent work in the apps on your computer, outside the browser. It can read a window, click, type and use keyboard shortcuts, the way you would.

Nothing to turn on

Computer Use is always available. Settings → Plugins → Computer Use shows Always on.

There is no switch because an agent in Full Access could already reach your desktop through the command line, without any safeguard. Going through Computer Use is what keeps some apps blocked, asks for your confirmation and logs every action.

Which apps an agent can use

Agents can use any app on your computer, except these, which are always blocked:

  • Neoo IDE itself,
  • ChatGPT and Codex,
  • terminals, such as Command Prompt, PowerShell, Windows Terminal, Terminal, iTerm2, Warp, Ghostty, Alacritty, kitty, GNOME Terminal, Konsole and xterm,
  • Keychain Access on macOS.

What an agent can do

  • List the apps that are running and the apps installed on your computer, and open an app even when it's closed (for example "open Epic Games"). On Windows, the installed apps come from the Start menu.
  • Read a window: its controls, the text it displays (a result, a status message) and a screenshot.
  • Click, drag, scroll, type text and press keys.
  • Restore a minimized window and bring it to the front.

Most actions are sent to the app in the background. The agent has its own cursor, so your mouse stays where it is. Some apps only react to real input: Neoo then brings the app to the front for that action.

Approve actions

Looking at the screen and scrolling don't need your approval. Opening an app, clicking, dragging, typing, pressing keys and bringing a window to the front do, even in Full Access mode. A card appears in the conversation:

ButtonEffect
Approve onceAllows this action only.
Allow everything in this conversationAllows all Computer Use in this conversation, including later turns.
DenyRefuses this action.
Cancel turnStops the agent's turn.

Allowing everything in one conversation never applies to your other conversations.

Watch and stop the agent

While an agent acts, a pill at the top of the screen says Neoo IDE is using your computer, with the app's name and a Stop button. The pill shows the icon of the app the agent is working in (a pointer when the app can't be identified). The agent can't see the pill in its screenshots. It appears on Windows and macOS.

On Windows, a blue pointer shows where the agent acts: it glides to each click, drag or control the agent uses, and stays at the last spot while the agent thinks. It's separate from your own mouse, you can click through it, and the agent can't see it.

The pill stays on screen while the agent thinks between two actions. It goes away when the turn ends, when you stop the agent, or after 60 seconds without any Computer Use action.

To stop the agent:

  • click Stop in the pill, or press Esc on your keyboard. The agent can't use Computer Use again for the rest of its turn. Your next message starts a new turn where it can use it again.
  • or stop the turn from the conversation.

Your mouse

When the agent acts on a control it found on screen, your mouse doesn't move. When it has to click, scroll or drag at a position (for example to draw in Paint), Windows moves your real mouse:

  • before acting, the agent waits for you to stop moving the mouse. If you keep using it, the conversation asks whether it should continue.
  • after the action, the mouse goes back to where you left it, unless you moved it yourself in the meantime. After a drawing, it goes back once, at the end.
  • when the agent hovers over something (to show a tooltip or a menu), the mouse stays there.

Check the activity log

Go to Settings → Plugins → Computer Use → Activity log. It lists each request with its result: Requested, Approved, Denied, Succeeded or Failed. Text the agent typed is never recorded.

Clear the log deletes this local history. Blocked apps and confirmations don't change.

Computer Use for Bots

Bots have their own switch, Shared computer control, in the Bot's Integrations and tools panel on Neoo Desktop. Approvals still apply.

  • You can't turn it on from the mobile app.
  • Imported Bots start with Computer Use off.
  • Bots running on Neoo Cloud use a virtual screen inside their machine instead of your computer: see Computer Use on Neoo Cloud.

Computer Use on Neoo Cloud

A bot on Neoo Cloud never touches your own screen. Its machine has a small virtual Linux screen (1280 x 720) with a window manager, a Chromium browser, a terminal and a file manager. The screen starts the first time the bot, or you, needs it and shuts down after about ten minutes without use, which closes the apps open on it. The browser's sign-ins are kept between sessions.

To use it:

  1. Open Settings → Integrations for the bot and turn on Shared computer control. The option is greyed out with Update pending until the Neoo Cloud sandbox image has been updated.
  2. Ask the bot for something that needs a real window. It uses the same Computer Use tools as on your computer (screenshot, click, type, key, scroll, drag) and opens an app with browser, terminal or files.

Watch it and take control

Open the bot's Overview panel, then its browser pane, and choose the Screen tab. You see the screen live. Click Take control to click, scroll and type yourself, for example to sign in or solve a captcha, and Hand back when you are done. While you are in control the bot's screen tools refuse to act, and control returns to the bot after ten minutes without input or when you leave the view.

Pictures of the screen travel end to end encrypted between your device and your machine, like the bot's session browser: Neoo's servers can't read them. Nothing is stored in the conversation: the bot sees only the latest capture, and it is not kept on disk.

When the bot needs you

The bot can ask for your help with request_takeover. A card appears in the conversation: The bot needs you on its screen, with what to do and an Open the screen button. The bot's turn is paused until you take the screen, act and hand it back. Dismiss the request in the screen view closes it without acting; it also expires after 15 minutes.

Limits

  • The screen has no accessibility tree: the bot works from screenshots and pixel positions, so it is slower and less precise than on your computer.
  • Each turn has a budget of 150 screen actions.
  • Only three apps open: browser, terminal and files. No command-line arguments, and the browser only opens http(s) addresses.
  • One person controls the screen at a time, and bots of the same machine share the one screen.
  • Passwords are never typed by the bot unless you gave them: use Take control to enter them yourself.
  • computer_fill_from_vault is not available there.

Known limits

  • In a minimized window, only actions on identified controls work. The agent can ask to restore the window.
  • Apps drawn on a canvas, such as some games or 3D views, expose no controls. The agent then has to click by position on the screenshot.
  • On Windows, some Store apps share one host window. If Neoo can't tell which one to act on, it refuses rather than guess.