Phones

A phone is an agent whose machine holds an Android phone: Android 13, a 720 by 1280 screen, and a browser. Like any agent it keeps everything, sleeps when idle and comes back exactly as it was, open apps and signed-in accounts included. It is ready about one second after you ask for it.

Phones come with paid plans. A phone uses two running slots, and always sleeps when idle.

Two ways to use one

  • An agent with a phone. Deploy with "phone": true. The harness inside the machine learns that it has a phone and how to use it, so you can just ask: "open this app and tell me what you see".
  • A phone for your own agent. Your agent runs wherever you want, and drives a Waken phone over MCP. One persistent phone per end user of your product.

An agent can have a desktop and a phone at once: set both "desktop" and "phone". The machine then needs 4 GB of memory, which is a paid add-on, and it is set for you when you do not give a size.

Driving a phone over HTTP

RouteWhat it does
GET /agents/{id}/phone/screenshotA PNG of the screen
POST /agents/{id}/phone/actionsUp to 50 actions in order, stops at the first failure
GET /agents/{id}/phone/appsThe apps that can be opened
POST /agents/{id}/phone/installInstalls an APK file from the agent's files
POST /agents/{id}/phone/viewerA link to watch and control the phone live in a browser, valid one hour
GET /agents/{id}/phone/streamThe screen as a stream of images, for an img tag
POST /agents/{id}/phone/actions
{ "actions": [
{ "type": "open_url", "url": "https://example.com" },
{ "type": "wait", "ms": 3000 },
{ "type": "tap", "x": 360, "y": 640 },
{ "type": "type", "text": "hello" },
{ "type": "key", "key": "enter" }
] }
ActionFields
tapx, y
long_pressx, y, and ms (800 by default)
swipex, y, to_x, to_y, and ms (300 by default). Swiping up scrolls the page down
typetext. Tap the field first
keykey: back, home, recents, enter, delete, tab, and the Android key names
open_urlurl, in the browser
open_apppackage, for example org.chromium.chrome
waitms

Coordinates are pixels on the 720 by 1280 screen. These routes need the computers scope.

Installing apps

Apps are installed from APK files. Upload the file to the agent's files, then install it:

Install an app
curl -X PUT "https://api.waken.sh/agents/$ID/fs/write?path=app.apk" \
-H "Authorization: Bearer $WAKEN_KEY" --data-binary @app.apk
curl -X POST https://api.waken.sh/agents/$ID/phone/install \
-H "Authorization: Bearer $WAKEN_KEY" -d '{"path": "app.apk"}'

Over MCP

The MCP server is the same as for desktops. Ask for a phone with os:

get_computer
{ "user_id": "user_42", "os": "android" }

Then computer and computer_batch take the phone actions above and return a screenshot. One end user can have a desktop and a phone.

Recording the screen

A phone can film its own screen as an MP4 video (H.264), at the size of the screen and 20 frames per second. One recording at a time, one hour at most. The agent stays awake while it records.

RouteWhat it does
GET /agents/{id}/phone/recordingsThe recording in progress and the finished ones
POST /agents/{id}/phone/recordingsStart recording. quality is low, medium or high
POST /agents/{id}/phone/recordings/stopStop, returns the video with its length and size
GET /agents/{id}/phone/recordings/file?id=The MP4 file
DELETE /agents/{id}/phone/recordings?id=Delete a video

Over MCP, start_recording and stop_recording work on a phone as they do on a desktop.

When a human is needed

Logins, CAPTCHAs and payments should be done by a person. Over MCP your agent calls request_takeover with a reason, and you send the returned link to your user. They see the phone live, tap and type from their own browser, and press "I'm done". Your agent polls takeover_status and continues. Because the phone persists, your user signs in once.

What to know

  • There is no app store on the phone. You bring the APK files.
  • Apps that check the device before they run, such as banking apps and some games, will refuse to start.
  • Typed text takes letters, digits, punctuation and Western European accents. Emoji and other scripts cannot be typed yet.
  • The live view shows around twenty images per second: smooth enough to follow the agent, sign in or scroll a page.
Next: container images