Phones
A phone is an agent whose machine holds an Android phone: Android 13, a 720 by 1280 screen, and a browser. Like any agent it keeps everything, sleeps when idle and comes back exactly as it was, open apps and signed-in accounts included. It is ready about one second after you ask for it.
Two ways to use one
- An agent with a phone. Deploy with
"phone": true. The harness inside the machine learns that it has a phone and how to use it, so you can just ask: "open this app and tell me what you see". - A phone for your own agent. Your agent runs wherever you want, and drives a Waken phone over MCP. One persistent phone per end user of your product.
An agent can have a desktop and a phone at once: set both "desktop" and "phone". The machine then needs 4 GB of memory, which is a paid add-on, and it is set for you when you do not give a size.
Driving a phone over HTTP
| Route | What it does |
|---|---|
| GET /agents/{id}/phone/screenshot | A PNG of the screen |
| POST /agents/{id}/phone/actions | Up to 50 actions in order, stops at the first failure |
| GET /agents/{id}/phone/apps | The apps that can be opened |
| POST /agents/{id}/phone/install | Installs an APK file from the agent's files |
| POST /agents/{id}/phone/viewer | A link to watch and control the phone live in a browser, valid one hour |
| GET /agents/{id}/phone/stream | The screen as a stream of images, for an img tag |
{ "actions": [{ "type": "open_url", "url": "https://example.com" },{ "type": "wait", "ms": 3000 },{ "type": "tap", "x": 360, "y": 640 },{ "type": "type", "text": "hello" },{ "type": "key", "key": "enter" }] }
| Action | Fields |
|---|---|
| tap | x, y |
| long_press | x, y, and ms (800 by default) |
| swipe | x, y, to_x, to_y, and ms (300 by default). Swiping up scrolls the page down |
| type | text. Tap the field first |
| key | key: back, home, recents, enter, delete, tab, and the Android key names |
| open_url | url, in the browser |
| open_app | package, for example org.chromium.chrome |
| wait | ms |
Coordinates are pixels on the 720 by 1280 screen. These routes need the computers scope.
Installing apps
Apps are installed from APK files. Upload the file to the agent's files, then install it:
curl -X PUT "https://api.waken.sh/agents/$ID/fs/write?path=app.apk" \-H "Authorization: Bearer $WAKEN_KEY" --data-binary @app.apkcurl -X POST https://api.waken.sh/agents/$ID/phone/install \-H "Authorization: Bearer $WAKEN_KEY" -d '{"path": "app.apk"}'
Over MCP
The MCP server is the same as for desktops. Ask for a phone with os:
{ "user_id": "user_42", "os": "android" }
Then computer and computer_batch take the phone actions above and return a screenshot. One end user can have a desktop and a phone.
Recording the screen
A phone can film its own screen as an MP4 video (H.264), at the size of the screen and 20 frames per second. One recording at a time, one hour at most. The agent stays awake while it records.
| Route | What it does |
|---|---|
| GET /agents/{id}/phone/recordings | The recording in progress and the finished ones |
| POST /agents/{id}/phone/recordings | Start recording. quality is low, medium or high |
| POST /agents/{id}/phone/recordings/stop | Stop, returns the video with its length and size |
| GET /agents/{id}/phone/recordings/file?id= | The MP4 file |
| DELETE /agents/{id}/phone/recordings?id= | Delete a video |
Over MCP, start_recording and stop_recording work on a phone as they do on a desktop.
When a human is needed
Logins, CAPTCHAs and payments should be done by a person. Over MCP your agent calls request_takeover with a reason, and you send the returned link to your user. They see the phone live, tap and type from their own browser, and press "I'm done". Your agent polls takeover_status and continues. Because the phone persists, your user signs in once.
What to know
- There is no app store on the phone. You bring the APK files.
- Apps that check the device before they run, such as banking apps and some games, will refuse to start.
- Typed text takes letters, digits, punctuation and Western European accents. Emoji and other scripts cannot be typed yet.
- The live view shows around twenty images per second: smooth enough to follow the agent, sign in or scroll a page.