Add an MCP server for driving and observing a client through rsprox

[CygnixDev]

This adds an MCP server to rsprox. A coding agent can use it to start a RuneLite client through rsprox, read the decoded packets of that session, and look at and drive the client. It can also read along with a client that a person launched by hand from the GUI. The goal is to let an agent check that a game server reacts correctly to what a real client sends, where it would otherwise need a hand-written protocol stub.

Nothing in this PR is a breaking change, and the GUI behaves as before until its new File > Serve MCP Endpoint item is ticked.

Never input on the official game

The tool is for testing a private server. On the official game, which is target 0 and any target whose jav_config comes from a Jagex domain, it only reads: the client's state, widgets, vars, entities, a screenshot, the camera's position, and packets. client_login, client_click, client_type, client_interact and a client_camera turn are refused there. Three layers enforce that:

• Each tool declares whether it reads or drives, and a session of the official game has read access only. The server refuses a drive tool before anything is sent.
• The link the server holds to the plugin carries the session's access and forwards no input op on a read-only one.
• The plugin builds its input ops, and the mouse they use, only when the server's welcome grants input and the client's home directory is not the stock .runelite, which is what rsprox launches the official game with. For any other connection those ops do not exist.

The plugin is listed in RuneLite as "rsprox MCP bridge" and can be switched off there. mcp.plugin=false in proxy.properties, or --no-plugin for the standalone server, stops the server from installing it; packets still work without it.

This guards what the shipped jar and server do. A rebuilt plugin can do anything, as with any RuneLite plugin.

What it adds

./gradlew :mcp:run starts rsprox without the GUI and serves MCP over HTTP on http://127.0.0.1:43580/mcp. It can run next to a GUI instance. It skips the GUI's port range, and it does not write proxy.properties or the saved filter sets.

The server launches clients with a small RuneLite plugin, rsprox-mcp-bridge, installed in the client's sideloaded-plugins directory. The plugin connects back to the server over loopback. In a client the server did not launch, the plugin does nothing.

There are fourteen tools:

Tools	What they do
session_start, session_stop, session_list	Launch, stop and list clients.
packets_read	Read decoded packets after a cursor, and optionally wait for one to arrive.
client_state, client_widgets, client_vars, client_entities, client_screenshot	Read the client.
client_login, client_interact, client_type, client_camera, client_click	Drive the client.

tools/list on the endpoint returns each tool with its description and argument schema. mcp/README.md walks through a full example.

Inside the GUI: attached sessions

The GUI serves the endpoint too, while File > Serve MCP Endpoint is ticked. It is off by default, and the item switches it at once, without a restart. A client that was already open when the endpoint is first turned on is not attached; clients launched from then on are. It listens on mcp.port (43580 by default). Each RuneLite or native client that is launched from the GUI shows up in session_list as a session of kind attached, numbered with the sessions that session_start launched. Its entry holds the target and the proxy port, and from the login on the world, the host, the local player index, the connect time, the recording's file name, the display name and the newest tick, so an agent can tell which client a person means.

An attached session is read-only. packets_read works on it. session_stop and every client_* tool refuse it, because the client belongs to the person at the GUI. Its packets are what the GUI shows: a packet that the GUI's filters hide is missing from the log.

The endpoint has no password and checks only the Origin header. Whatever can reach the loopback port can read the packets of every session, chat included. That is why the GUI serves it only while it is turned on, and the menu item's tooltip says so. Unticking the item closes the port at once, and a request that still runs can no longer launch a client, stop one or send one input. The sessions of the run stay in memory and are listed again when the endpoint is turned back on, because the proxy cannot detach a packet tap from an open client; they are gone when the GUI exits.

The GUI finds the endpoint through ServiceLoader. proxy gains two small interfaces, ProxyExtension and ClientListener, which mcp implements. The GUI's own change is one line, and its session panel is untouched. A monitor that fans out calls the GUI's monitor first and then the endpoint's, and what the endpoint's monitor throws never reaches the GUI.

If the port is taken, by a second rsprox or by the standalone server, the GUI logs one line and starts without the endpoint.

How interactions work

client_interact clicks a target by identity with the real mouse. The target is an NPC, an object, a ground item, a player, an interface widget, a dialog, or a tile to walk to. The plugin finds it in the client's own lists, moves the mouse onto its clickable shape, and reads the menu the client built for that point. It presses only when the client offers the intended action. An option that is not the default goes through the right-click menu.

The client decides what the click means and what to send, so the server receives what a player's click sends, including the mouse packets. Nothing calls the client's action dispatcher or writes a packet, which means a client update cannot leave the tool sending something the client no longer would.

The whole loop runs inside the plugin at the client's frame rate. It starts on a game tick and retries until a deadline, so a moving NPC gets many attempts without a round trip to the agent. A guard on MenuOptionClicked cancels a click that the client resolves to anything other than the intended action, before a packet is sent. When the target is out of view, or drawn under an interface such as the chatbox, the loop turns the camera toward it. The client offers only Cancel at a point an interface covers, so the tool never clicks through one.

After the click the server finds the client's mouse-click packet for the pressed point and returns the action packet that follows it, with the number of attempts and the tick it went out on. If the client sends no packet for a world target, the tool reports the action as dropped. An interface option that the client handles without a packet returns a note saying so.

Waits inside the loop are counted in the client's 20 ms game cycles, not in frames, so an unlocked frame rate cannot make it click twice. Ops that produce input run one at a time per client.

client_click is a raw click at a pixel with no guard. It is the fallback for things that have no menu action.

How the rest behaves

A session, such as s1, outlives its client. The packet log belongs to the session, so stopping and relaunching the client keeps the buffered packets and continues the same sequence numbers.

The packet log shows every packet in both directions, with the named values the GUI shows. It reuses the existing transcriber with filter and setting stores that hide nothing and live only in memory.

Each client_* tool returns the packet cursor it took before it acted. Reading from that cursor returns exactly the packets the action caused.

Screenshots are scaled to the size of the canvas, so a pixel in the image is a click coordinate.

The MCP endpoint is a small JSON-RPC handler on the JDK HttpServer. It adds no MCP SDK and no new dependencies.

Running without a display

RuneLite always opens a window, so a machine with no screen needs a virtual one. On Linux, run the server under Xvfb with --software-rendering:

xvfb-run -s "-screen 0 1280x800x24" ./gradlew :mcp:run --args="--software-rendering"

On a virtual display the GPU plugin renders through a software OpenGL driver, and the frames read back from it are black. --software-rendering stops the GPU plugin in each client the server launches, without changing the RuneLite profile. macOS and Windows have no virtual display, so there the client needs a real screen.

Changes to existing files

Outside the new mcp module and the plugin, the PR touches these:

• settings.gradle.kts includes the mcp module, and the root project puts it on the classpath of the proxy task.
• ProxyService lets a caller replace filterSetStore and settingsStore, and initializeHttpServer takes an optional target config that defaults to the selected target. It tells its ClientListeners of each RuneLite and native launch and of each client that closed.
• ProxyProperty gains mcp.enabled (default false), mcp.port and mcp.plugin.
• SessionMonitor gains onPacketDirection, with a default that does nothing, and SessionTracker calls it for each packet.
• ProxyToolGui starts the extensions after the proxy service, and the File menu gains the Serve MCP Endpoint item.
• New in proxy: ProxyExtension, ClientListener and FanOutSessionMonitor.

Existing callers compile unchanged.

Testing

99 automated tests in mcp, proxy and the GUI cover the packet log's cursor rules, the endpoint over real HTTP, the tool calls as an MCP client makes them, the session failure paths, the bridge handshake, the attached sessions, the fan-out monitor, for every tool on each kind of session and target whether it is forwarded or refused, and turning the GUI's endpoint on and off, including a request that is still running when it is turned off. The tool-call tests post to the real endpoint, with a fake plugin socket that answers canned JSON and models nothing about the game.

The tests do not run a game client. The branch history holds tests that ran the plugin against a faked game, and removes them again. That fake encoded guesses about how the client builds and lays out its menus, and every defect found in this work was found against a live client and none by the fake. In their place there is a checklist to run against a live client after a RuneLite update, a game revision change or a plugin change, with what proves each step. It is at the end of this description.

I ran the launched-session steps of that checklist by hand against a private revision 241 server on macOS, with the GUI running next to it:

• Session start, login, and logout through the game's own button.
• Interface buttons, typed input, and a conversation continued step by step.
• An NPC that walks around, a right-click option, an object, and walking to a tile.
• The camera, read, set and pointed at a target.
• The error when an interface blocks the game view.
• A walk to a tile drawn under the chatbox, where the loop turned the camera and then succeeded.
• A client killed by pid, and a session relaunched with its earlier packets still readable.

With two clients at once on two accounts, each listed the other as a player, and both acted at the same moment without interfering. Their packet logs stayed separate. The player options Follow and Trade with sent their packets, and stopping one session left the other working.

The read-only rule, the plugin switch and the GUI menu item were added after those runs and have only been run by the automated tests. Still to confirm against a live client: that a custom-target client is welcomed with drive access, that an official-target client is read-only, and that the menu item renders, switches the endpoint and reverts when the port is taken.

For the attached sessions, I started the GUI from an earlier commit of this branch, launched a client by hand, and read along from an agent: session_list listed the client with its login and display name, packets_read returned its packets in both directions as I played, and a second launch got its own session after the first client was closed.

Known limits

• An attached session offers packets only. The client_* tools do not reach a client that was launched by hand, not even the ones that read.

• While the GUI's endpoint is off, the sessions of clients launched in that run keep recording in memory.

• An attached session ends when its tab is closed or its processes exit. If the launcher hands the client over and exits early, only closing the tab ends it.

• RuneScape 3 and replay clients are not attached.

• The plugin has no automated tests. The compiler and the live checklist are its checks.

• Not exercised against a live server: ground items, a click cancelled by the guard, and the switch to the right-click menu after a cancelled left click.

• The run under Xvfb on Linux arm64 was done before the interaction loop was rewritten, so it has not been repeated for the current code.

• Two-step actions such as "use item on" and casting a spell on a target are untested. Dragging an item has no tool.

• Walking reaches only a tile in the loaded scene that the camera can bring into view. Secondary world views such as boats are not supported. Stretched mode is refused.

• A launcher that never completes its handshake is abandoned after 180 seconds, and the server then needs a restart.

• Sessions write .bin recordings into the target's usual folder.

• The rendezvous file is shared, so run one MCP server per user account.

Checklist against a live client

The automated tests cover the packet log, the endpoint's HTTP and JSON-RPC handling, the tool calls
as an MCP client makes them, the session failure paths, the attached sessions and the bridge handshake. They do not run a
game client, so what the client does is checked by hand. Run this after a RuneLite update, a game
revision change, or a change to the plugin. Each step names what proves it, so a step that fails
names what changed. A tool that reports a dropped action, or lists what the client offered instead,
is working as intended. Read its message before suspecting the tool.

Start the server and a session against a target you can log in to, with a new account:

1. session_start returns "state":"connected" with a pid, and that pid is the client process.
2. client_login returns "gameState":"LOGGED_IN", and packets_read {"origin":"proxy"} shows a
   LOGIN row. packets_read {"limit":20} shows packets of tick 0, with named values.
3. client_screenshot returns an image whose width and height equal canvas in client_state.
4. client_interact on an interface button returns a packet line and "proof":"click".
   packets_read from the returned cursor shows an EVENT_MOUSE_CLICK row before the button packet,
   so the click was a real one.
5. client_type into a prompt is followed by a packet that carries the typed text.
6. client_entities lists the NPCs and objects that the screenshot shows, with their options.
7. client_interact with the default option of an NPC that walks around returns its OPNPC packet.
   The attempts count stays low. Repeat it a few times.
8. client_interact with an option that is not the default, such as Examine, goes through the
   right-click menu: the last entry of tried starts with clicked row.
9. client_interact on an object and on a tile each return their packet, and after the walk
   client_state shows the player on that tile.
10. client_interact {"target":"dialog","option":"continue"} advances a conversation.
11. With an interface open that blocks the game view, client_interact on an NPC fails and says that
    the client offers only Cancel.
12. client_camera {"look_at":"npc","index":0} returns "onScreen":true, and the next screenshot
    shows the turned view.
13. Killing the client process makes session_list show stopped with the reason client exited,
    and session_start with the same session brings it back with the earlier packets still readable.

Then check the attached sessions. Stop the standalone server, start the GUI with ./gradlew proxy
and launch a client from it by hand:

1. session_list shows a session with "kind":"attached", the target and the proxy port of the
   client, and no login.
2. After you log in, session_list shows login with the world, the capture file and, a tick or two
   later, your display name and a tick that advances. packets_read on the session returns the
   packets that the GUI's own panel shows.
3. The GUI's panel shows the same packets as it does with mcp.enabled=false.
4. session_stop and client_state on the session fail with not available for an attached session, and the client keeps running.
5. Closing the client window makes session_list show "state":"ended". So does closing the tab of
   a second client. Both sessions stay listed with their packets.
6. With the standalone server running first, the GUI starts as usual and logs that the endpoint is
   not served.

🤖 Generated with Claude Code