diff --git a/ts/packages/agentServer/client/README.AUTOGEN.md b/ts/packages/agentServer/client/README.AUTOGEN.md index 63af87fce..37283cd42 100644 --- a/ts/packages/agentServer/client/README.AUTOGEN.md +++ b/ts/packages/agentServer/client/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # @typeagent/agent-server-client — AI-generated documentation @@ -12,58 +12,90 @@ ## Overview -The `@typeagent/agent-server-client` package is a TypeScript library designed to facilitate connections to a running agentServer. It is used by various components such as the Shell and CLI to manage conversations and ensure the server is running. +The `@typeagent/agent-server-client` package is a TypeScript library that provides tools for connecting to and interacting with a running `agentServer`. It is a core component of the TypeAgent ecosystem, enabling clients such as the Shell, CLI, and other integrations to manage conversations and ensure the availability of the agentServer. ## What it does -This package provides several key functionalities for interacting with an agentServer: +This package offers a comprehensive set of features for managing connections and interactions with an `agentServer`. Key functionalities include: -- **Connection Management**: Establishes WebSocket connections to the agentServer using `connectAgentServer`. -- **Conversation Management**: Supports creating, listing, renaming, and deleting conversations through methods on the `AgentServerConnection` object. -- **Server Management**: Ensures the agentServer is running, spawns it if necessary, and checks its status using functions like `ensureAgentServer` and `isServerRunning`. -- **Convenience Wrappers**: Provides simplified methods to ensure the server is running and connect to conversations in one call, such as `ensureAndConnectConversation`. +- **Connection Management**: The `connectAgentServer` function establishes WebSocket connections to an `agentServer` and provides an `AgentServerConnection` object for further interactions. +- **Conversation Management**: The `AgentServerConnection` object supports operations such as: + - Joining conversations (`joinConversation`) + - Leaving conversations (`leaveConversation`) + - Creating new conversations (`createConversation`) + - Listing existing conversations (`listConversations`) + - Renaming conversations (`renameConversation`) + - Deleting conversations (`deleteConversation`) +- **Server Management**: Functions like `ensureAgentServer` and `isServerRunning` help ensure the `agentServer` is running, spawn it if necessary, and check its status. +- **Convenience Wrappers**: Simplified methods like `ensureAndConnectConversation` combine multiple steps into a single call, allowing you to ensure the server is running, connect to it, and join a conversation in one operation. +- **Port Discovery**: The `discovery` module provides functionality for discovering the dynamically assigned port of an in-process agent. + +This package is used by various other components in the TypeAgent ecosystem, including the CLI, Shell, and browser-based clients. ## Setup -To use this package, you need to install it along with its dependencies. Ensure you have the following environment variables and prerequisites set up: +To use this package, you need to configure the following: -- **Environment Variables**: None required specifically for this package. -- **Dependencies**: Install the required dependencies using `pnpm install`. This package depends on other TypeAgent packages such as `@typeagent/agent-rpc`, `@typeagent/agent-server-protocol`, and `@typeagent/dispatcher-rpc`. +### Environment Variables -For detailed setup instructions, see the hand-written README. +- `TYPEAGENT_TUNNEL_TOKEN`: This token is required for secure communication with the agentServer. Refer to the hand-written README for details on how to obtain and configure this token. -## Key Files +### Installation + +Install the package and its dependencies using `pnpm`: -The package is structured into several key files: +```bash +pnpm install +``` -- **[index.ts](./src/index.ts)**: Exports the main functions and types used by external clients. -- **[agentServerClient.ts](./src/agentServerClient.ts)**: Contains the core implementation for connecting to the agentServer, managing conversations, and ensuring the server is running. -- **[discovery.ts](./src/discovery.ts)**: Provides functionality for discovering the dynamically-assigned port of an in-process agent. -- **[conversation/index.ts](./src/conversation/index.ts)**: Contains shared conversation-lifecycle helpers for clients of the agent server. -- **[conversation/lifecycle.ts](./src/conversation/lifecycle.ts)**: Implements connection-level lifecycle helpers shared by every client that joins an agent server. -- **[conversation/manage.ts](./src/conversation/manage.ts)**: Implements the dispatcher's `manage-conversation` client-action surface. -- **[conversation/naming.ts](./src/conversation/naming.ts)**: Provides conversation name primitives and utilities. +This package depends on the following workspace packages: -### Key Functions and Classes +- `@typeagent/agent-rpc` +- `@typeagent/agent-server-protocol` +- `@typeagent/dispatcher-rpc` +- `websocket-channel-server` -- **`connectAgentServer(url, onDisconnect?)`**: Opens a WebSocket connection to the agentServer and returns an `AgentServerConnection`. -- **`AgentServerConnection`**: Manages conversations with methods like `joinConversation`, `leaveConversation`, `createConversation`, `listConversations`, `renameConversation`, and `deleteConversation`. -- **`ensureAgentServer(port?, hidden?, idleTimeout?)`**: Ensures the agentServer is running, spawning it if needed. -- **`isServerRunning(url)`**: Checks if a server is already listening at the given WebSocket URL. -- **`stopAgentServer(port?)`**: Sends a shutdown RPC to the running server. -- **`ensureAndConnectConversation(clientIO, port?, options?, onDisconnect?, hidden?, idleTimeout?)`**: Ensures the server is running, connects, and joins a conversation in one call. +Additionally, it uses the external libraries `debug` and `isomorphic-ws`. + +## Key Files + +The package is organized into several key files, each responsible for specific functionality: + +- **[index.ts](./src/index.ts)**: The main entry point of the package, exporting core functions and types for external use. +- **[agentServerClient.ts](./src/agentServerClient.ts)**: Implements the primary logic for connecting to the `agentServer`, managing conversations, and ensuring the server is running. This file contains the implementation of key functions like `connectAgentServer`, `ensureAgentServer`, and `stopAgentServer`. +- **[discovery.ts](./src/discovery.ts)**: Provides a discovery client for locating the dynamically assigned port of an in-process agent. This is particularly useful for external clients like browser extensions or plugins. +- **[conversation/index.ts](./src/conversation/index.ts)**: Consolidates shared conversation-lifecycle helpers for clients of the `agentServer`. +- **[conversation/lifecycle.ts](./src/conversation/lifecycle.ts)**: Implements connection-level lifecycle helpers, including utilities for finding or creating conversations and handling race conditions. +- **[conversation/manage.ts](./src/conversation/manage.ts)**: Contains the implementation of the dispatcher's `manage-conversation` client-action surface, including subcommands for creating, listing, renaming, and deleting conversations. +- **[conversation/naming.ts](./src/conversation/naming.ts)**: Provides utilities for handling conversation names, such as normalization, uniqueness checks, and sorting. ## How to extend -To extend the functionality of this package, follow these steps: +To extend the `@typeagent/agent-server-client` package, follow these steps: + +1. **Identify the area to extend**: + + - For connection and conversation management, start with [agentServerClient.ts](./src/agentServerClient.ts). + - For port discovery, refer to [discovery.ts](./src/discovery.ts). + - For conversation lifecycle or management, explore the files in the [conversation](./src/conversation/) directory. -1. **Open the relevant file**: Depending on what you want to extend, start with either [agentServerClient.ts](./src/agentServerClient.ts) for connection and conversation management or [discovery.ts](./src/discovery.ts) for port discovery. -2. **Add new methods or modify existing ones**: Implement new features or enhance existing functionalities by adding or modifying methods in the appropriate file. -3. **Update exports**: Ensure that any new functions or types are exported in [index.ts](./src/index.ts) for external use. -4. **Write tests**: Add tests for your new functionality to ensure it works as expected. Follow the existing test patterns in the repository. -5. **Run tests**: Execute the test suite to verify your changes. Use `pnpm test` or the equivalent command configured in the project. +2. **Add or modify functionality**: -By following these steps, you can effectively extend the capabilities of the `@typeagent/agent-server-client` package. + - Implement new methods or enhance existing ones in the relevant file. + - For example, to add a new conversation management feature, you might start by modifying [conversation/manage.ts](./src/conversation/manage.ts). + +3. **Update exports**: + + - Ensure that any new functions or types are exported in [index.ts](./src/index.ts) so they are accessible to external consumers. + +4. **Write tests**: + + - Add tests for your new functionality. Follow the existing patterns in the repository to ensure consistency. + +5. **Run tests**: + - Use the project's test suite to verify your changes. Run `pnpm test` or the equivalent command configured in the project. + +By following these steps, you can contribute to the `@typeagent/agent-server-client` package and enhance its capabilities. ## Reference @@ -71,9 +103,9 @@ By following these steps, you can effectively extend the capabilities of the `@t ### Entry points -- default → [./dist/index.js](./dist/index.js) -- `./conversation` → [./dist/conversation/index.js](./dist/conversation/index.js) -- `./discovery` → [./dist/discovery.js](./dist/discovery.js) +- default → `./dist/index.js` _(not found on disk)_ +- `./conversation` → `./dist/conversation/index.js` _(not found on disk)_ +- `./discovery` → `./dist/discovery.js` _(not found on disk)_ ### Dependencies @@ -96,9 +128,9 @@ External: `debug`, `isomorphic-ws` - [browser-typeagent](../../../packages/agents/browser/README.md) - [coder-wrapper](../../../packages/coderWrapper/README.md) - [command-executor-mcp](../../../packages/commandExecutor/README.md) +- [remote-client-example](../../../examples/remoteClient/README.md) - [studio-service](../../../packages/studio-service/README.md) -- tools-scripts -- _…and 4 more workspace consumers._ +- _…and 5 more workspace consumers._ ### Files of interest @@ -111,8 +143,14 @@ External: `debug`, `isomorphic-ws` - [./src/discovery.ts](./src/discovery.ts) - [./src/tsconfig.json](./src/tsconfig.json) +### Environment variables + +_1 environment variable referenced from `./src/` (set in `ts/.env` or your shell). See the `## Setup` section above for guidance on obtaining each value._ + +- `TYPEAGENT_TUNNEL_TOKEN` + --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter @typeagent/agent-server-client docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter @typeagent/agent-server-client docs:verify-links` to spot-check._ diff --git a/ts/packages/agentServer/protocol/README.AUTOGEN.md b/ts/packages/agentServer/protocol/README.AUTOGEN.md index 1856adce2..45e00a854 100644 --- a/ts/packages/agentServer/protocol/README.AUTOGEN.md +++ b/ts/packages/agentServer/protocol/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # @typeagent/agent-server-protocol — AI-generated documentation @@ -12,82 +12,79 @@ ## Overview -The `@typeagent/agent-server-protocol` package defines the WebSocket RPC contract between agentServer clients and the server. It provides the necessary types, channel names, and methods for managing conversations and client connections within the Type Agent Server ecosystem. +The `@typeagent/agent-server-protocol` package defines the WebSocket RPC contract between the agentServer and its clients. It provides the foundational types, channel names, and methods required for managing conversations, client connections, and discovery mechanisms within the Type Agent Server ecosystem. This package is a TypeScript library and serves as a shared dependency for various components of the TypeAgent monorepo. ## What it does -This package primarily handles the protocol definitions for communication between the agentServer and its clients. It includes: +The primary purpose of this package is to standardize communication between the agentServer and its clients. It achieves this by defining: -- **Channel Names**: Fixed and session-namespaced channels for conversation lifecycle RPC. -- **Conversation Types**: Definitions for conversation metadata and results. -- **RPC Methods**: Methods exposed on the `agent-server` channel for managing conversations. -- **Client-type Registry**: Functions for registering and retrieving client types based on connection IDs. -- **Discovery Channel**: Methods for external clients to look up the live port of any in-process app-agent. +- **Channel Names**: Predefined and dynamically generated channel names for managing conversation lifecycles and client communication. +- **Conversation Types**: Type definitions for conversation metadata, connection results, and options for joining or creating conversations. +- **RPC Methods**: A set of methods exposed on the `agent-server` channel to manage conversations, including creating, joining, renaming, and deleting conversations, as well as server shutdown. +- **Client-type Registry**: A registry to associate client types with connection IDs, enabling client-specific behavior. +- **Discovery Channel**: A read-only channel for external clients to discover the live port of in-process app-agents. -The package exports several key types and functions, such as `AgentServerChannelName`, `getDispatcherChannelName`, `getClientIOChannelName`, `ConversationInfo`, `JoinConversationResult`, `DispatcherConnectOptions`, and various RPC methods like `joinConversation`, `leaveConversation`, `createConversation`, `listConversations`, `renameConversation`, `deleteConversation`, and `shutdown`. +This package is used by multiple components in the TypeAgent ecosystem, including `@typeagent/agent-server-client`, `agent-server`, and various agent implementations like `agent-coda` and `browser-typeagent`. ## Setup -No additional setup is required beyond installing the package. Simply run: +This package does not require any special setup beyond installation. To include it in your project, run: ```sh pnpm install ``` -## Key Files +For more details on usage, refer to the hand-written README. -The package is structured as follows: +## Key Files -- **[index.ts](./src/index.ts)**: This file exports all the necessary types and functions from `protocol.ts`. -- **[protocol.ts](./src/protocol.ts)**: Contains the definitions for types, channel names, and RPC methods. -- **[queue.ts](./src/queue.ts)**: Re-exports queue wire types and errors from `@typeagent/dispatcher-types`. -- **[tsconfig.json](./src/tsconfig.json)**: TypeScript configuration file for the package. +The package is organized into the following key files: -### Channel Names +### [index.ts](./src/index.ts) -- **AgentServerChannelName**: The fixed channel name for conversation lifecycle RPC. -- **DiscoveryChannelName**: The fixed channel name for read-only port discovery RPC. -- **getDispatcherChannelName(conversationId: string)**: Constructs session-namespaced channels for dispatcher communication. -- **getClientIOChannelName(conversationId: string)**: Constructs session-namespaced channels for client IO communication. +This is the main entry point of the package. It re-exports all the key types, constants, and functions defined in other files, making them available for external use. Notable exports include: -### Conversation Types +- **Channel Names**: `AgentServerChannelName`, `DiscoveryChannelName`, `getDispatcherChannelName`, `getClientIOChannelName`. +- **Conversation Types**: `ConversationInfo`, `JoinConversationResult`, `DispatcherConnectOptions`. +- **RPC Methods**: `AgentServerInvokeFunctions`, including methods like `joinConversation`, `leaveConversation`, `createConversation`, and more. +- **Discovery Handlers**: `createDiscoveryHandlers` for setting up the discovery channel. +- **Client-type Registry Functions**: `registerClientType`, `getClientType`, `unregisterClient`. -- **ConversationInfo**: Describes a conversation with fields like `conversationId`, `name`, `clientCount`, and `createdAt`. -- **JoinConversationResult**: Returned by `joinConversation`, includes `connectionId` and `conversationId`. -- **DispatcherConnectOptions**: Options passed to `joinConversation`, including `conversationId`, `clientType`, and `filter`. +### [protocol.ts](./src/protocol.ts) -### RPC Methods +This file contains the core protocol definitions, including: -- **AgentServerInvokeFunctions**: Methods exposed on the `agent-server` channel, including: - - `joinConversation(options?)` - - `leaveConversation(conversationId)` - - `createConversation(name)` - - `listConversations(name?)` - - `renameConversation(conversationId, newName)` - - `deleteConversation(conversationId)` - - `shutdown()` +- **Channel Names**: Fixed and dynamic channel name definitions for communication. +- **Conversation Types**: Detailed type definitions for conversations, including metadata, connection results, and options. +- **RPC Methods**: Definitions for the methods exposed on the `agent-server` channel, such as `joinConversation`, `createConversation`, and `shutdown`. -### Client-type Registry +### [queue.ts](./src/queue.ts) -- **registerClientType(connectionId: string, clientType: string)**: Registers a client type based on connection ID. -- **getClientType(connectionId: string)**: Retrieves the client type for a given connection ID. -- **unregisterClient(connectionId: string)**: Unregisters a client based on connection ID. +This file re-exports queue-related types and errors from the `@typeagent/dispatcher-types` package. This allows clients to access these types directly from `@typeagent/agent-server-protocol` without needing to depend on `@typeagent/dispatcher-types`. -### Discovery Channel +### [tsconfig.json](./src/tsconfig.json) -- **DiscoveryInvokeFunctions**: Methods exposed on the `discovery` channel, including: - - `lookupPort(param: { agentName: string; role?: string })`: Looks up the live port of any in-process app-agent. +The TypeScript configuration file for the package. It extends the base TypeScript configuration for the monorepo and specifies the input and output directories for the compiled files. ## How to extend To extend the functionality of this package, follow these steps: -1. **Open `protocol.ts`**: This file contains the core definitions and is the starting point for any modifications or additions. -2. **Add new types or methods**: Define new types or RPC methods as needed. Ensure they are well-documented and follow the existing structure. -3. **Export new additions**: Update `index.ts` to export any new types or methods added to `protocol.ts`. -4. **Test your changes**: Write tests to validate the new functionality. Ensure that all existing tests pass and cover the new additions. +1. **Start with `protocol.ts`**: This file contains the core protocol definitions. If you need to add new types, channel names, or RPC methods, this is the place to start. + + - For example, to add a new RPC method, define its type in `protocol.ts` and include it in the `AgentServerInvokeFunctions` type. + +2. **Update `index.ts`**: After adding new types or methods in `protocol.ts`, ensure they are exported in `index.ts` so they are available to other packages. + +3. **Modify the client-type registry if needed**: If your extension involves new client types or changes to how client types are managed, update the registry functions in `protocol.ts`. + +4. **Extend the discovery channel**: If your extension requires changes to the discovery mechanism, modify the `createDiscoveryHandlers` function in `protocol.ts`. Ensure that the new functionality is compatible with the existing `PortRegistrar` interface. + +5. **Write tests**: Add or update tests to cover your changes. Ensure that all existing tests pass and that your new functionality is thoroughly tested. + +6. **Document your changes**: Update the hand-written README and this documentation to reflect your additions. Clearly describe the new functionality and provide examples where applicable. -By following these steps, you can effectively extend the capabilities of the `@typeagent/agent-server-protocol` package. +By following these steps, you can ensure that your extensions are consistent with the existing structure and functionality of the `@typeagent/agent-server-protocol` package. ## Reference @@ -95,7 +92,7 @@ By following these steps, you can effectively extend the capabilities of the `@t ### Entry points -- default → [./dist/index.js](./dist/index.js) +- default → `./dist/index.js` _(not found on disk)_ ### Dependencies @@ -113,6 +110,7 @@ External: _None at runtime._ - [agent-server](../../../packages/agentServer/server/README.md) - [agent-shell](../../../packages/shell/README.md) - [browser-typeagent](../../../packages/agents/browser/README.md) +- [remote-client-example](../../../examples/remoteClient/README.md) - visualstudio-extension-webview - [vscode-chat](../../../packages/vscode-chat/README.md) - [vscode-shell](../../../packages/vscode-shell/README.md) @@ -123,6 +121,6 @@ External: _None at runtime._ --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter @typeagent/agent-server-protocol docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter @typeagent/agent-server-protocol docs:verify-links` to spot-check._ diff --git a/ts/packages/agentServer/server/README.AUTOGEN.md b/ts/packages/agentServer/server/README.AUTOGEN.md index 3c31c0c91..296efb854 100644 --- a/ts/packages/agentServer/server/README.AUTOGEN.md +++ b/ts/packages/agentServer/server/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # agent-server — AI-generated documentation @@ -12,51 +12,87 @@ ## Overview -The `agent-server` package is a TypeScript library that provides a long-running WebSocket server for hosting TypeAgent dispatchers with full conversation management capabilities. It listens for connections, manages conversations, and facilitates communication between clients and agents. +The `agent-server` package is a TypeScript library that provides a long-running WebSocket server for hosting TypeAgent dispatchers. It is responsible for managing conversations, facilitating communication between clients and agents, and handling server lifecycle events such as startup and shutdown. ## What it does -The `agent-server` package handles several actions related to conversation management and server control. These actions include `joinConversation`, `leaveConversation`, `createConversation`, `listConversations`, `renameConversation`, `deleteConversation`, and `shutdown`. The server listens on a WebSocket endpoint and processes these actions by interacting with the conversation manager and shared dispatcher components. It also supports graceful shutdown and idle timeout features. +The `agent-server` package is the backbone of the TypeAgent system, enabling real-time communication between clients and agents. It provides a WebSocket-based server that supports a range of actions for managing conversations and server operations. These actions include: + +- **Conversation Management**: Actions such as `joinConversation`, `leaveConversation`, `createConversation`, `listConversations`, `renameConversation`, and `deleteConversation` allow clients to interact with and manage conversations. +- **Server Control**: Actions like `shutdown` enable graceful termination of the server. +- **Idle Timeout**: The server can be configured to shut down automatically after a specified period of inactivity. +- **Integration with Agent Dispatchers**: The server manages a pool of `SharedDispatcher` instances, which handle multiple client connections within a single conversation context. +- **Persistence and Cleanup**: Conversation metadata and logs are stored persistently, and ephemeral conversations are cleaned up during server startup to ensure efficient resource usage. + +The server listens on a WebSocket endpoint (default: `ws://localhost:8999`) and can be started either manually or automatically when clients invoke `ensureAgentServer()`. ## Setup -To set up the `agent-server`, you need to configure the following environment variables: +To set up the `agent-server`, ensure the following environment variables are configured: -- `AGENT_SERVER_PORT`: The port on which the server will listen for WebSocket connections. -- `TYPEAGENT_USER_NAME`: The username for the TypeAgent user. +- `AGENT_SERVER_PORT`: Specifies the port on which the server listens for WebSocket connections. The default is `8999`. +- `TYPEAGENT_USER_NAME`: The username for the TypeAgent user. This can be set to override the default OS user name. +- `XDG_CONFIG_HOME`: Specifies the base directory for configuration files. If not set, the default is typically `~/.config`. -Ensure these variables are set in your environment or in a `.env` file. For detailed setup instructions, see the hand-written README. +You can set these variables in your environment or define them in a `.env` file. For additional details, refer to the hand-written README. ## Key Files -The `agent-server` package is structured around several key components: +The `agent-server` package is organized into several key files, each responsible for specific functionality: + +### [server.ts](./src/server.ts) — WebSocket Listener + +This file initializes the WebSocket server and manages client connections. It creates a `ConversationManager` at startup and uses `createWebSocketChannelServer` to handle WebSocket connections. The server exposes `AgentServerInvokeFunctions` over the `agent-server` RPC channel, enabling actions such as: + +- `joinConversation` and `leaveConversation` for managing client participation. +- `createConversation`, `listConversations`, `renameConversation`, and `deleteConversation` for conversation lifecycle management. +- `shutdown` for gracefully stopping the server. + +### [conversationManager.ts](./src/conversationManager.ts) — Conversation Pool + +This file manages the lifecycle of conversations and their associated `SharedDispatcher` instances. Key responsibilities include: + +- **Persistence**: Stores conversation metadata and logs in the user's configuration directory. +- **Lazy Initialization**: Creates `SharedDispatcher` instances only when a conversation is joined for the first time. +- **Default Conversations**: Automatically creates a default conversation if none exists and no `conversationId` is provided. +- **Ephemeral Cleanup**: Removes temporary conversations left over from previous sessions during server startup. +- **Idle Shutdown**: Monitors client activity and shuts down the server after a specified period of inactivity if the `--idle-timeout` flag is set. + +### [sharedDispatcher.ts](./src/sharedDispatcher.ts) — Routing Layer + +This file manages multiple client connections within a single conversation. It wraps a dispatcher context and routes client IO methods based on connection IDs. Key features include: + +- **Connection Management**: Assigns unique `connectionId` values to clients and maintains a routing table for client IO. +- **Routing Logic**: Ensures that client-specific actions, such as `setDisplay` and `askYesNo`, are routed to the appropriate client. +- **Broadcasting**: Supports broadcasting messages to all connected clients, with optional filtering for targeted delivery. -### `server.ts` — WebSocket listener +### [connectionHandler.ts](./src/connectionHandler.ts) — Connection Management -The `server.ts` file is responsible for setting up the WebSocket server and managing connections. It creates a `ConversationManager` at startup and calls `createWebSocketChannelServer` to accept connections. For each connection, it exposes `AgentServerInvokeFunctions` over the `agent-server` RPC channel, handling actions such as joining and leaving conversations, creating and listing conversations, and shutting down the server. +This file defines the logic for handling individual client connections. It integrates with the `ConversationManager` to manage conversations and uses the `agent-server` RPC channel to expose server functions to connected clients. -### `conversationManager.ts` — Conversation pool +### [copilot/displayLogSynthesis.ts](./src/copilot/displayLogSynthesis.ts) — Display Log Synthesis -The `conversationManager.ts` file maintains a pool of per-conversation `SharedDispatcher` instances. It handles persistence of conversation metadata, lazy initialization of dispatchers, automatic creation of default conversations, and cleanup of ephemeral conversations. It also supports idle shutdown based on the `--idle-timeout` flag. +This file provides utilities for generating `DisplayLogEntry` streams from Copilot session data. It ensures that imported sessions can be replayed in the conversation UI by synthesizing deterministic log entries. -### `sharedDispatcher.ts` — Routing layer +### [copilot/mirrorImporter.ts](./src/copilot/mirrorImporter.ts) — Copilot Session Importer -The `sharedDispatcher.ts` file manages multiple client connections within a single conversation. It wraps a dispatcher context and routes client IO methods based on connection IDs. This ensures that each client's display output is isolated while sharing the same dispatcher and conversation context. +This file handles the import of Copilot sessions into the `agent-server`. It reads session data from a SQLite database and converts it into a format compatible with the `agent-server`'s conversation system. -### `status.ts` and `stop.ts` — Server control +### [inProcessAgentServer.ts](./src/inProcessAgentServer.ts) — In-Process Server -The `status.ts` and `stop.ts` files provide utilities for checking the server status and stopping the server, respectively. They use the `@typeagent/agent-server-client` package to interact with the server. +This file provides an in-process implementation of the agent server, allowing it to run within the same process as the client. It is useful for scenarios where low-latency communication is required. ## How to extend -To extend the `agent-server` package, follow these steps: +To extend the `agent-server` package, follow these guidelines: -1. **Start with `server.ts`**: This file sets up the WebSocket server and manages connections. You can add new RPC functions or modify existing ones to handle additional actions. -2. **Modify `conversationManager.ts`**: If you need to change how conversations are managed, this is the file to edit. You can add new behaviors or modify existing ones related to conversation persistence, initialization, and cleanup. -3. **Update `sharedDispatcher.ts`**: To change how client connections are routed within a conversation, modify this file. You can add new routing logic or adjust the existing methods for handling client IO. -4. **Test your changes**: Ensure that your modifications are working correctly by running the server and testing the new or modified actions. You can use the provided `status.ts` and `stop.ts` utilities to control the server during testing. +1. **Understand the architecture**: Start by reviewing the key files mentioned above to understand the responsibilities of each component. +2. **Add new actions**: To introduce new server actions, modify [server.ts](./src/server.ts) to expose the new functionality over the `agent-server` RPC channel. Implement the corresponding logic in the appropriate file, such as [conversationManager.ts](./src/conversationManager.ts) or [sharedDispatcher.ts](./src/sharedDispatcher.ts). +3. **Enhance conversation management**: If your changes involve new conversation behaviors, update [conversationManager.ts](./src/conversationManager.ts). For example, you can add new persistence mechanisms or modify the logic for handling idle timeouts. +4. **Extend routing logic**: To customize how client IO is routed within a conversation, modify [sharedDispatcher.ts](./src/sharedDispatcher.ts). You can add new routing rules or adjust existing ones to meet your requirements. +5. **Test your changes**: Use the provided utilities in [status.ts](./src/status.ts) and [stop.ts](./src/stop.ts) to test your modifications. Ensure that all new functionality works as expected and does not introduce regressions. -By following these steps, you can extend the functionality of the `agent-server` package to meet your specific needs. +By following these steps, you can effectively extend the `agent-server` package to support additional features or integrate with other components in the TypeAgent ecosystem. ## Reference @@ -64,8 +100,8 @@ By following these steps, you can extend the functionality of the `agent-server` ### Entry points -- default → [./dist/server.js](./dist/server.js) -- `./in-process` → [./dist/inProcessAgentServer.js](./dist/inProcessAgentServer.js) +- default → `./dist/server.js` _(not found on disk)_ +- `./in-process` → `./dist/inProcessAgentServer.js` _(not found on disk)_ ### Dependencies @@ -83,7 +119,7 @@ Workspace: - [dispatcher-node-providers](../../../packages/dispatcher/nodeProviders/README.md) - [websocket-channel-server](../../../packages/utils/webSocketChannelServer/README.md) -External: `@azure/identity`, `debug`, `dotenv`, `ws` +External: `@azure/identity`, `better-sqlite3`, `debug`, `dotenv`, `ws` ### Used by @@ -91,17 +127,18 @@ External: `@azure/identity`, `debug`, `dotenv`, `ws` ### Files of interest -`./src/connectionHandler.ts`, `./src/conversationManager.ts`, `./src/inProcessAgentServer.ts`, …and 6 more under `./src/`. +`./src/connectionHandler.ts`, `./src/conversationManager.ts`, `./src/copilot/displayLogSynthesis.ts`, …and 10 more under `./src/`. ### Environment variables -_2 environment variables referenced from `./src/` (set in `ts/.env` or your shell). See the `## Setup` section above for guidance on obtaining each value._ +_3 environment variables referenced from `./src/` (set in `ts/.env` or your shell). See the `## Setup` section above for guidance on obtaining each value._ - `AGENT_SERVER_PORT` - `TYPEAGENT_USER_NAME` +- `XDG_CONFIG_HOME` --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter agent-server docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter agent-server docs:verify-links` to spot-check._ diff --git a/ts/packages/agents/agentUtils/graphUtils/README.AUTOGEN.md b/ts/packages/agents/agentUtils/graphUtils/README.AUTOGEN.md index 5f2cfaeb5..09c026b27 100644 --- a/ts/packages/agents/agentUtils/graphUtils/README.AUTOGEN.md +++ b/ts/packages/agents/agentUtils/graphUtils/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # graph-utils — AI-generated documentation @@ -12,47 +12,63 @@ ## Overview -The `graph-utils` package provides utility functions to access Microsoft Graph APIs, facilitating integration with Microsoft services such as calendar and email. It abstracts the complexities of interacting with these APIs, offering a simplified interface for common operations. +The `graph-utils` package provides utility functions for accessing Microsoft Graph APIs, enabling integration with Microsoft services such as calendars and email. It simplifies the process of interacting with these APIs by offering a unified interface for common operations, making it easier to work with multiple providers like Microsoft Graph and Google Calendar. ## What it does -The `graph-utils` package supports various actions related to calendar and email functionalities. It includes utilities for date range calculations, client creation for calendar and email services, and provider abstractions for multi-provider support (Microsoft Graph and Google Calendar). Key actions include `createCalendarEvent`, `deleteCalendarEvent`, `getCalendarEvents`, `updateCalendarEvent`, `createEmail`, `deleteEmail`, `getEmails`, and `updateEmail`. +The `graph-utils` package supports a range of actions related to calendar and email management. These include creating, updating, retrieving, and deleting calendar events and emails. The package also provides utilities for date range calculations, embedding management for calendar data, and factory functions for creating provider-specific clients. + +The package is designed to work with multiple providers, including Microsoft Graph and Google Calendar, and can detect the configured provider based on environment variables. Key actions include: + +- Calendar-related actions: `createCalendarEvent`, `deleteCalendarEvent`, `getCalendarEvents`, and `updateCalendarEvent`. +- Email-related actions: `createEmail`, `deleteEmail`, `getEmails`, and `updateEmail`. + +These actions are implemented using provider-agnostic interfaces, allowing the package to support multiple backends with minimal changes. ## Setup -To use the `graph-utils` package, several environment variables need to be set up. These variables are essential for authenticating and interacting with the Microsoft Graph and Google Calendar APIs. The required environment variables are: +To use the `graph-utils` package, you need to configure the following environment variables: -- `GOOGLE_CALENDAR_CLIENT_ID` -- `GOOGLE_CALENDAR_CLIENT_SECRET` -- `MSGRAPH_APP_CLIENTID` -- `MSGRAPH_APP_TENANTID` +- `GOOGLE_CALENDAR_CLIENT_ID`: The client ID for your Google Calendar API application. +- `GOOGLE_CALENDAR_CLIENT_SECRET`: The client secret for your Google Calendar API application. +- `MSGRAPH_APP_CLIENTID`: The client ID for your Microsoft Graph API application. +- `MSGRAPH_APP_TENANTID`: The tenant ID for your Microsoft Graph API application. + +These variables are required for authenticating with the respective APIs. To obtain the values: + +1. For `GOOGLE_CALENDAR_CLIENT_ID` and `GOOGLE_CALENDAR_CLIENT_SECRET`, set up a project in the Google Cloud Console, enable the Google Calendar API, and create OAuth 2.0 credentials. +2. For `MSGRAPH_APP_CLIENTID` and `MSGRAPH_APP_TENANTID`, register an application in the Azure portal and note the application (client) ID and directory (tenant) ID. -Ensure these variables are correctly configured in your environment. For detailed steps on how to obtain and set these values, refer to the hand-written README. +Refer to the hand-written README for additional details on setting up these environment variables. ## Key Files -The `graph-utils` package is structured to provide a clear separation of concerns, with dedicated modules for different functionalities: +The `graph-utils` package is organized into several key files, each responsible for specific functionalities: -- [index.ts](./src/index.ts): Exports utility functions and client creation methods for calendar and email services. -- [calendarClient.ts](./src/calendarClient.ts): Contains the `CalendarClient` class, which handles interactions with the calendar API, including login and data synchronization. -- [calendarDataIndex.ts](./src/calendarDataIndex.ts): Provides methods for managing and searching calendar event embeddings. -- [calendarProvider.ts](./src/calendarProvider.ts): Defines interfaces and types for calendar events, attendees, date range queries, and user information. -- [calendarProviderFactory.ts](./src/calendarProviderFactory.ts): Factory functions to create calendar providers based on configuration. -- [dateUtils.ts](./src/dateUtils.ts): Utility functions for date range calculations. -- [emailProvider.ts](./src/emailProvider.ts): Defines interfaces and types for email messages, addresses, search queries, and user information. -- [emailProviderFactory.ts](./src/emailProviderFactory.ts): Factory functions to create email providers based on configuration. +- [index.ts](./src/index.ts): The main entry point of the package, exporting utility functions and client creation methods for calendar and email services. +- [calendarClient.ts](./src/calendarClient.ts): Implements the `CalendarClient` class, which manages interactions with the calendar API, including login, synchronization, and data management. +- [calendarDataIndex.ts](./src/calendarDataIndex.ts): Provides methods for managing and searching calendar event embeddings, enabling efficient event lookups. +- [calendarProvider.ts](./src/calendarProvider.ts): Defines provider-agnostic interfaces and types for calendar events, attendees, date range queries, and user information. +- [calendarProviderFactory.ts](./src/calendarProviderFactory.ts): Contains factory functions to create calendar providers based on the configured environment variables. +- [dateUtils.ts](./src/dateUtils.ts): Offers utility functions for date range calculations, such as determining the current week or month. +- [emailProvider.ts](./src/emailProvider.ts): Defines provider-agnostic interfaces and types for email messages, addresses, search queries, and user information. +- [emailProviderFactory.ts](./src/emailProviderFactory.ts): Contains factory functions to create email providers based on the configured environment variables. +- [googleCalendarClient.ts](./src/googleCalendarClient.ts): Implements the `ICalendarProvider` interface for the Google Calendar API, including OAuth authentication and token management. ## How to extend To extend the `graph-utils` package, follow these steps: -1. **Identify the module to extend**: Determine whether your extension involves calendar functionalities, email functionalities, or general utilities. -2. **Open the relevant file**: For calendar-related extensions, start with [calendarClient.ts](./src/calendarClient.ts) or [calendarProvider.ts](./src/calendarProvider.ts). For email-related extensions, start with [emailProvider.ts](./src/emailProvider.ts). -3. **Follow existing patterns**: Review the existing code to understand the structure and patterns used. Implement your changes following these patterns to maintain consistency. -4. **Add new actions**: If you need to add new actions, ensure they are properly defined and exported in [index.ts](./src/index.ts). -5. **Test your changes**: Write tests to validate your extensions. Ensure that your changes do not break existing functionalities. +1. **Determine the area of extension**: Identify whether your changes pertain to calendar functionalities, email functionalities, or general utilities. +2. **Locate the relevant file**: + - For calendar-related extensions, start with [calendarClient.ts](./src/calendarClient.ts), [calendarProvider.ts](./src/calendarProvider.ts), or [calendarProviderFactory.ts](./src/calendarProviderFactory.ts). + - For email-related extensions, focus on [emailProvider.ts](./src/emailProvider.ts) or [emailProviderFactory.ts](./src/emailProviderFactory.ts). +3. **Follow existing patterns**: Review the existing code to understand the structure, naming conventions, and patterns. Implement your changes in a consistent manner. +4. **Add new actions**: If your extension involves new actions, define them in the appropriate files and ensure they are exported in [index.ts](./src/index.ts). +5. **Update tests**: Write or update tests to cover your changes. This ensures that your modifications work as intended and do not introduce regressions. +6. **Test thoroughly**: Run the test suite to verify that your changes integrate correctly with the existing functionality. -By following these steps, you can effectively extend the `graph-utils` package to meet your specific requirements. +By adhering to these guidelines, you can effectively contribute to and extend the `graph-utils` package. ## Reference @@ -60,7 +76,7 @@ By following these steps, you can effectively extend the `graph-utils` package t ### Entry points -- default → [./dist/index.js](./dist/index.js) +- default → `./dist/index.js` _(not found on disk)_ ### Dependencies @@ -93,6 +109,6 @@ _4 environment variables referenced from `./src/` (set in `ts/.env` or your shel --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter graph-utils docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter graph-utils docs:verify-links` to spot-check._ diff --git a/ts/packages/agents/browser/README.AUTOGEN.md b/ts/packages/agents/browser/README.AUTOGEN.md index ba1f62ee1..ef2ac80e0 100644 --- a/ts/packages/agents/browser/README.AUTOGEN.md +++ b/ts/packages/agents/browser/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # browser-typeagent — AI-generated documentation @@ -12,73 +12,90 @@ ## Overview -The `browser-typeagent` package is a TypeAgent application agent designed to control and automate browser actions. It enables the manipulation of browser windows, tabs, and web pages through a set of defined actions. This package is part of the TypeAgent monorepo and integrates with other TypeAgent components to provide a comprehensive browser automation solution. +The `browser-typeagent` package is a TypeAgent application agent designed for browser automation and control. It enables interaction with browser windows, tabs, and web pages through a set of predefined actions. This package integrates with other components in the TypeAgent ecosystem, such as the TypeAgent shell and CLI, to provide a cohesive browser automation experience. ## What it does -The `browser-typeagent` package provides a range of actions for browser control and automation. These actions include `openWebPage`, `indexPage`, `clickOn`, `captureScreenshot`, `getHtmlFragments`, and more. The package allows for the opening, closing, and navigation of browser tabs, scrolling, zooming, and interacting with web pages. It also supports enhanced website indexing with knowledge extraction capabilities. +The `browser-typeagent` package provides a comprehensive set of actions for controlling and automating browser behavior. These actions include: -The package interacts with other parts of the TypeAgent system, such as the TypeAgent shell and CLI, to receive commands and execute browser actions. It uses WebSocket connections to communicate with clients, including browser extensions and the Electron shell. +- **Navigation and Tab Management**: Actions like `openWebPage`, `closeWebPage`, `goBack`, `goForward`, `scrollDown`, `scrollUp`, and `changeTabs` allow for opening, closing, and navigating between browser tabs and web pages. +- **Content Interaction**: Actions such as `clickOn`, `followLinkByText`, `followLinkByPosition`, and `readPageContent` enable interaction with web page elements and content. +- **Content Extraction and Analysis**: Actions like `captureScreenshot`, `getHtmlFragments`, and `indexPage` allow for capturing and analyzing web page content, including AI-enhanced indexing and knowledge extraction. +- **Browser Control**: Actions such as `zoomIn`, `zoomOut`, `zoomReset`, and `reloadPage` provide control over the browser's display and functionality. + +The package uses a WebSocket server (`AgentWebSocketServer`) to facilitate communication between the browser agent and its clients. Supported clients include a Chrome extension and the Electron-based TypeAgent shell. The WebSocket server supports multiple concurrent sessions, with each session isolated by a unique `sessionId`. ## Setup -To set up the `browser-typeagent` package, you need to configure the following environment variables: +To set up the `browser-typeagent` package, follow these steps: -- `BROWSER_WEBSOCKET_PORT`: This variable allows you to pin the WebSocket server port for debugging. Set it to a specific port number before launching the host. -- `TYPEAGENT_BROWSER_FILES`: This variable should point to the directory containing the browser files required for the agent's operation. +1. **Configure Environment Variables**: -Additionally, you need to enable developer mode in your browser and load the unpackaged extension. The steps are as follows: + - `BROWSER_WEBSOCKET_PORT`: Set this variable to specify the port for the WebSocket server. If not set, the port will be assigned dynamically by the operating system. + - `TYPEAGENT_BROWSER_FILES`: Set this variable to the directory containing the browser files required for the agent's operation. -1. Enable developer mode in your browser (Chrome or Edge): +2. **Enable Developer Mode in Your Browser**: - - Launch the browser. - - Click on the extensions icon next to the address bar and select "Manage extensions." - - Enable the developer mode toggle on the extensions page. + - For Chrome or Edge: + - Launch the browser. + - Click on the extensions icon next to the address bar and select "Manage extensions." + - Enable the "Developer mode" toggle on the extensions page. -2. Build the extension by running `pnpm run build` in the package folder. +3. **Build the Extension**: -3. Load the unpackaged extension: - - Go to the "Manage extensions" page. - - Click on "Load unpackaged extension" and navigate to the `dist/extension` folder of the browser extension package. + - Run `pnpm run build` in the package folder to build the browser extension. -For detailed setup instructions, see the hand-written README. +4. **Load the Unpackaged Extension**: -## Key Files + - Navigate to the "Manage extensions" page in your browser. + - Click on "Load unpackaged extension" and select the `dist/extension` folder of the browser extension package. + +5. **Run the Extension**: + - Launch the browser where the extension is installed. + - Start the TypeAgent shell or CLI, which integrates with the extension to send commands. You can issue commands such as `openWebPage`, `scrollDown`, or `captureScreenshot` from the shell or CLI. -The `browser-typeagent` package is structured into several key components: +For additional details, refer to the hand-written README. -- **Agent WebSocket Server**: Exposes a WebSocket server (`AgentWebSocketServer`) on a dynamically assigned port, allowing clients to connect and communicate with the browser agent. Clients include the Chrome extension and the Electron shell. +## Key Files -- **Session Management**: Supports multiple concurrent sessions by registering handlers under unique `sessionId` keys. The session ID is stored in `BrowserActionContext.sessionId` and is set during context initialization. +The `browser-typeagent` package is organized into several key files, each responsible for specific functionality: -- **Client Type Detection**: Differentiates between `extension` and `electron` clients based on their `clientId`. Commands are routed to the active client based on the `preferredClientType`. +- **browserActionHandler.ts**: Implements the core browser actions, such as `openWebPage`, `clickOn`, and `captureScreenshot`. This file is central to defining and handling browser-related commands. +- **[agentWebSocketServer.ts](./src/agent/agentWebSocketServer.mts)**: Manages WebSocket connections and routes client requests to the appropriate session and handlers. It supports multiple concurrent sessions and client types. +- **[browserIndexingService.ts](./src/agent/indexing/browserIndexingService.ts)**: Provides AI-enhanced indexing capabilities, including content summarization and quality assessment for web pages. +- **[browserActionSchema.mts](./src/agent/browserActionSchema.mts)**: Defines the schema for browser actions, including their parameters and expected behavior. +- **[browserContentExtractor.mts](./src/agent/browserContentExtractor.mts)**: Handles content extraction from web pages, supporting both standard HTML extraction and browser-based downloading. +- **[crossContextHtmlReducer.ts](./src/common/crossContextHtmlReducer.ts)**: Optimizes HTML content by removing unnecessary elements and attributes, reducing its size for efficient processing. -- **Channel Multiplexing**: Uses `@typeagent/agent-rpc` to multiplex client connections into two logical channels: `agentService` for invoking browser agent actions and `browserControl` for controlling the browser. +## How to extend -- **Client Storage Model**: Stores connected clients in a nested `Map>`, allowing the same `clientId` to exist simultaneously in multiple sessions without collision. +To extend the functionality of the `browser-typeagent` package, follow these steps: -Key files and their responsibilities include: +1. **Identify the Relevant File**: -- browserActionHandler.ts: Handles browser actions such as opening web pages and capturing screenshots. -- agentWebSocketServer.ts: Manages WebSocket connections and client routing. -- [browserIndexingService.ts](./src/agent/indexing/browserIndexingService.ts): Provides AI-enhanced indexing with content summarization and quality assessment. -- [crossContextHtmlReducer.ts](./src/common/crossContextHtmlReducer.ts): Reduces HTML size by removing unnecessary elements and attributes. + - Determine which file corresponds to the functionality you want to modify or extend. For example: + - To add a new browser action, start with browserActionHandler.ts. + - To modify WebSocket behavior, refer to [agentWebSocketServer.ts](./src/agent/agentWebSocketServer.mts). -## How to extend +2. **Implement the New Feature**: -To extend the `browser-typeagent` package, follow these steps: + - Follow the existing patterns in the codebase to implement your changes. For example: + - Add a new action handler in browserActionHandler.ts. + - Update the action schema in [browserActionSchema.mts](./src/agent/browserActionSchema.mts) to define the parameters and behavior of the new action. -1. Open the relevant file based on the functionality you want to add or modify. For example, to add a new browser action, start with browserActionHandler.ts. +3. **Update Session and Client Management**: -2. Implement the new action or feature following the existing patterns. Ensure that the new functionality integrates with the WebSocket server and session management. + - If your changes involve session-specific behavior, ensure that the `sessionId` is properly handled in the `AgentWebSocketServer` and related files. + - If your changes involve new client types or roles, update the client type detection logic in [agentWebSocketServer.ts](./src/agent/agentWebSocketServer.mts). -3. Update the action schema if necessary. The schema files are located in the `src/agent` directory, such as [browserActionSchema.json](./src/agent/browserActionSchema.json). +4. **Test Your Changes**: -4. Test the new functionality by running the TypeAgent shell or CLI and issuing commands to verify the behavior. + - Use the TypeAgent shell or CLI to test the new functionality. Issue commands to verify that the new feature works as expected. -5. Document the new actions and features in the package's README and ensure that the environment variables and setup instructions are updated accordingly. +5. **Document Your Changes**: + - Update the package's documentation to include details about the new feature or action. Ensure that any new environment variables or setup steps are clearly described. -By following these steps, you can extend the capabilities of the `browser-typeagent` package and integrate new browser automation features. +By following these steps, you can effectively extend the `browser-typeagent` package to support additional browser automation capabilities. ## Reference @@ -87,12 +104,12 @@ By following these steps, you can extend the capabilities of the `browser-typeag ### Entry points - `./agent/manifest` → [./src/agent/manifest.json](./src/agent/manifest.json) -- `./agent/handlers` → [./dist/agent/browserActionHandler.mjs](./dist/agent/browserActionHandler.mjs) -- `./agent/types` → [./dist/common/browserControl.mjs](./dist/common/browserControl.mjs) -- `./agent/indexing` → [./dist/agent/indexing/browserIndexingService.js](./dist/agent/indexing/browserIndexingService.js) -- `./contentScriptRpc/types` → [./dist/common/contentScriptRpc/types.mjs](./dist/common/contentScriptRpc/types.mjs) -- `./contentScriptRpc/client` → [./dist/common/contentScriptRpc/client.mjs](./dist/common/contentScriptRpc/client.mjs) -- `./htmlReducer` → [./dist/common/crossContextHtmlReducer.js](./dist/common/crossContextHtmlReducer.js) +- `./agent/handlers` → `./dist/agent/browserActionHandler.mjs` _(not found on disk)_ +- `./agent/types` → `./dist/common/browserControl.mjs` _(not found on disk)_ +- `./agent/indexing` → `./dist/agent/indexing/browserIndexingService.js` _(not found on disk)_ +- `./contentScriptRpc/types` → `./dist/common/contentScriptRpc/types.mjs` _(not found on disk)_ +- `./contentScriptRpc/client` → `./dist/common/contentScriptRpc/client.mjs` _(not found on disk)_ +- `./htmlReducer` → `./dist/common/crossContextHtmlReducer.js` _(not found on disk)_ ### Dependencies @@ -145,7 +162,7 @@ _…and 17 more not shown._ - [./src/extension/contentScript/recording/index.ts](./src/extension/contentScript/recording/index.ts) - [./src/extension/serviceWorker/index.ts](./src/extension/serviceWorker/index.ts) - [./src/extension/webagent/crossword/crosswordSchema.agr](./src/extension/webagent/crossword/crosswordSchema.agr) -- _…and 286 more under `./src/`._ +- _…and 284 more under `./src/`._ ### Environment variables @@ -156,6 +173,6 @@ _2 environment variables referenced from `./src/` (set in `ts/.env` or your shel --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter browser-typeagent docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter browser-typeagent docs:verify-links` to spot-check._ diff --git a/ts/packages/agents/desktop/README.AUTOGEN.md b/ts/packages/agents/desktop/README.AUTOGEN.md index 79546c977..d2ec535ea 100644 --- a/ts/packages/agents/desktop/README.AUTOGEN.md +++ b/ts/packages/agents/desktop/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # desktop-automation — AI-generated documentation @@ -12,80 +12,134 @@ ## Overview -The `desktop-automation` package is a TypeAgent application agent designed to manage and control desktop environments on Windows systems. It integrates with Windows shell APIs to perform various actions such as launching programs, managing windows, adjusting display settings, and more. +The `desktop-automation` package is a TypeAgent application agent designed to automate and control desktop environments on Windows systems. It integrates with Windows shell APIs via a .NET component to perform a variety of desktop-related actions, such as managing windows, launching applications, and adjusting system settings. This package is particularly useful for automating repetitive tasks or creating custom workflows for desktop management. ## What it does -This package provides a wide range of actions to control and automate desktop tasks. Some of the key actions include: +The `desktop-automation` package enables a wide range of desktop automation tasks by providing a set of predefined actions. These actions are defined in the [actionsSchema.ts](./src/actionsSchema.ts) file and are processed by the [actionHandler.ts](./src/actionHandler.ts) file. The package communicates with a .NET component (`autoShell.exe`) to execute these actions on a Windows system. -- `LaunchProgram`: Launches a specified program. -- `CloseProgram`: Closes a specified program. -- `TileWindows`: Arranges windows in a specified layout. -- `MaximizeWindow`: Maximizes a specified window. -- `MinimizeWindow`: Minimizes a specified window. -- `SetVolume`: Adjusts the system volume. -- `SetWallpaper`: Changes the desktop wallpaper. -- `ConnectWifi`: Connects to a specified Wi-Fi network. -- `ToggleAirplaneMode`: Toggles the airplane mode on or off. +### Key Capabilities -These actions are defined in the [actionsSchema.ts](./src/actionsSchema.ts) file and are handled by the [actionHandler.ts](./src/actionHandler.ts) file. +The package supports the following types of actions: -## Setup +- **Program Management**: + + - `LaunchProgram`: Launch a specified program. + - `CloseProgram`: Close a specified program. + - `SwitchToWindow`: Switch focus to a specific window. + +- **Window Management**: + + - `TileWindows`: Arrange windows in a specific layout. + - `MaximizeWindow`: Maximize a window. + - `MinimizeWindow`: Minimize a window. + - `MoveWindowToDesktop`: Move a window to a specific virtual desktop. + +- **System Settings**: + + - `SetVolume`, `AdjustVolume`, `MuteVolume`: Manage system volume. + - `SetWallpaper`: Change the desktop wallpaper. + - `SetScreenResolution`: Adjust screen resolution. + - `SetTextSize`: Modify text size. + +- **Network and Connectivity**: + + - `ConnectWifi`, `DisconnectWifi`, `ListWifiNetworks`: Manage Wi-Fi connections. + - `ToggleAirplaneMode`: Enable or disable airplane mode. + +- **Desktop Management**: + + - `CreateDesktop`, `SwitchDesktop`, `NextDesktop`, `PreviousDesktop`: Manage virtual desktops. + - `PinWindowToAllDesktops`: Pin a window to all virtual desktops. -To set up the `desktop-automation` package, you need to configure the following environment variable: +- **Other System Features**: + - `ToggleNotifications`: Enable or disable system notifications. + - `BluetoothToggleAction`: Manage Bluetooth settings. + - `EnableWifiAction`: Enable Wi-Fi. + - `AdjustScreenBrightnessAction`: Adjust screen brightness. -- `AUTOSHELL_PATH`: Path to the `autoShell.exe` binary. This binary is built from the .NET code located in the `dotnet/autoShell` directory. +These actions allow users to interact with and control their desktop environment programmatically, making it easier to perform complex tasks or automate workflows. -### Steps to build and configure +## Setup + +To use the `desktop-automation` package, you need to complete the following setup steps: + +### Prerequisites -1. **Build the .NET component**: +1. **Build the .NET Component**: - Open Visual Studio. - - Navigate to the `dotnet/autoShell` directory. + - Navigate to the `dotnet/autoShell` directory in the repository. - Build the project to generate the `autoShell.exe` binary. -2. **Build the TypeAgent component**: +2. **Build the TypeAgent Component**: - Navigate to the `ts/packages/agents/desktop` directory. - - Run `pnpm run build` to build the TypeAgent application agent. + - Run the following command to build the TypeAgent application agent: + ```bash + pnpm run build + ``` + +3. **Set the Environment Variable**: + - Define the `AUTOSHELL_PATH` environment variable to point to the location of the `autoShell.exe` binary generated in step 1. For example: + ```bash + export AUTOSHELL_PATH=/path/to/autoShell.exe + ``` + +### Running the Automation -3. **Set the environment variable**: - - Ensure the `AUTOSHELL_PATH` environment variable points to the location of the `autoShell.exe` binary. +Once the setup is complete, you can use the `desktop-automation` agent with the [TypeAgent Shell](../../shell) or the [TypeAgent CLI](../../cli). Enable the agent using the following command: -For detailed setup instructions, see the hand-written README. +```bash +@config agent desktop +``` + +You can then issue commands such as: + +- "Launch calculator" +- "Maximize calculator" +- "Tile calculator to the left and Chrome to the right" + +Refer to the hand-written README for additional details on running the automation. ## Key Files -The `desktop-automation` package is structured as follows: +The `desktop-automation` package is organized into several key files, each serving a specific purpose: -- **Manifest**: The agent's manifest is defined in [manifest.json](./src/manifest.json), which includes descriptions and schema files for the agent and its sub-actions. -- **Schema**: The actions schema is defined in [actionsSchema.ts](./src/actionsSchema.ts), which lists all possible actions the agent can perform. -- **Grammar**: The grammar for parsing user commands is defined in [desktopSchema.agr](./src/desktopSchema.agr). -- **Handler**: The main action handler is implemented in [actionHandler.ts](./src/actionHandler.ts), which processes incoming actions and interacts with the .NET component. -- **Connector**: The [connector.ts](./src/connector.ts) file manages the connection between the TypeAgent and the .NET component, handling the execution of desktop actions. +- **[manifest.json](./src/manifest.json)**: Defines the agent's metadata, including its description, default state, and references to schema and grammar files. +- **[actionsSchema.ts](./src/actionsSchema.ts)**: Contains the TypeScript definitions for all supported actions, such as `LaunchProgram`, `CloseProgram`, and `TileWindows`. +- **[desktopSchema.agr](./src/desktopSchema.agr)**: Specifies the grammar for parsing user commands into structured actions. +- **[actionHandler.ts](./src/actionHandler.ts)**: Implements the logic for handling actions. This is where the agent processes incoming requests and interacts with the .NET component. +- **[connector.ts](./src/connector.ts)**: Manages the communication between the TypeAgent and the `autoShell.exe` binary. It handles the execution of desktop actions and ensures the .NET component is ready to process requests. +- **[readiness.ts](./src/readiness.ts)**: Contains logic to check the readiness of the `autoShell.exe` binary and provides utilities for setting up the environment. +- **[programNameIndex.ts](./src/programNameIndex.ts)**: Manages a searchable index of known program names, enabling the agent to match user commands to specific applications. ## How to extend -To extend the `desktop-automation` package, follow these steps: +To add new features or actions to the `desktop-automation` package, follow these steps: -1. **Add new actions**: +1. **Define New Actions**: - - Define new action types in [actionsSchema.ts](./src/actionsSchema.ts). + - Add new action types to [actionsSchema.ts](./src/actionsSchema.ts). - Update the grammar in [desktopSchema.agr](./src/desktopSchema.agr) to include the new actions. -2. **Implement action handlers**: +2. **Implement Action Handlers**: - - Add the logic for handling new actions in [actionHandler.ts](./src/actionHandler.ts). + - Add the logic for handling the new actions in [actionHandler.ts](./src/actionHandler.ts). Use the existing handlers as a reference for implementing new functionality. -3. **Update the manifest**: +3. **Update the Manifest**: - Modify [manifest.json](./src/manifest.json) to include descriptions and schema files for the new actions. -4. **Test your changes**: - - Write unit tests for the new actions and handlers. - - Run the tests to ensure everything works as expected. +4. **Test Your Changes**: + + - Write unit tests for the new actions and their handlers. + - Run the tests to ensure the new functionality works as expected. + +5. **Update Documentation**: + - Update the hand-written README and any other relevant documentation to reflect the new features. -By following these steps, you can extend the functionality of the `desktop-automation` package to support additional desktop automation tasks. +By following these steps, you can extend the `desktop-automation` package to support additional desktop automation tasks, making it more versatile and tailored to specific use cases. ## Reference @@ -94,7 +148,7 @@ By following these steps, you can extend the functionality of the `desktop-autom ### Entry points - `./agent/manifest` → [./src/manifest.json](./src/manifest.json) -- `./agent/handlers` → [./dist/actionHandler.js](./dist/actionHandler.js) +- `./agent/handlers` → `./dist/actionHandler.js` _(not found on disk)_ ### Dependencies @@ -137,6 +191,6 @@ _31 actions declared in the schema, none yet implemented in [`./src/actionHandle --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter desktop-automation docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter desktop-automation docs:verify-links` to spot-check._ diff --git a/ts/packages/agents/list/README.AUTOGEN.md b/ts/packages/agents/list/README.AUTOGEN.md index d4595a6c7..013229f49 100644 --- a/ts/packages/agents/list/README.AUTOGEN.md +++ b/ts/packages/agents/list/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # list-agent — AI-generated documentation @@ -12,68 +12,91 @@ ## Overview -The `list-agent` package is a TypeAgent application agent designed to manage lists. It provides actions to create, modify, and retrieve lists, making it useful for applications that require list management functionality, such as to-do lists, shopping lists, or any other type of item collection. +The `list-agent` package is a TypeAgent application agent designed to manage lists. It provides a set of actions for creating, modifying, and retrieving lists, making it suitable for use cases such as to-do lists, shopping lists, or other item collections. This agent is part of the TypeAgent monorepo and integrates with other components in the system to handle user requests related to list management. ## What it does -The `list-agent` package offers a set of actions that allow users to interact with lists. These actions include: +The `list-agent` package supports six actions, each designed to handle a specific aspect of list management: -- `addItems`: Adds one or more items to a specified list. If the list does not exist, it is created. -- `removeItems`: Removes one or more items from a specified list. -- `createList`: Creates a new list with the given name. -- `getList`: Retrieves the contents of a specified list, useful for queries like "What's on my grocery list?". -- `clearList`: Clears all items from a specified list. -- `startEditList`: Initiates the editing of a specified list. +- **`addItems`**: Adds one or more items to a specified list. If the list does not already exist, it will be created automatically. +- **`removeItems`**: Removes one or more items from a specified list. +- **`createList`**: Creates a new list with a given name. +- **`getList`**: Retrieves the contents of a specified list. This action is useful for answering user queries like "What's on my grocery list?" or "What are the contents of my to-do list?" +- **`clearList`**: Removes all items from a specified list. +- **`startEditList`**: Initiates the editing of a specified list, allowing for further modifications. -These actions are defined in the [listSchema.ts](./src/listSchema.ts) file and are handled by the [listActionHandler.ts](./src/listActionHandler.ts) file. +These actions are defined in the [listSchema.ts](./src/listSchema.ts) file and are implemented in the [listActionHandler.ts](./src/listActionHandler.ts) file. The agent uses a combination of schema definitions, grammar rules, and handler logic to process user requests and execute the appropriate actions. ## Setup -The `list-agent` package does not require any special setup beyond installing dependencies. Ensure you have the necessary packages installed by running: +The `list-agent` package does not require any special setup beyond installing its dependencies. To get started: -```sh -pnpm install -``` +1. Ensure you have `pnpm` installed on your system. +2. Run the following command in the root of the monorepo to install all dependencies: -For detailed setup instructions, refer to the hand-written README. + ```sh + pnpm install + ``` + +For additional setup details, refer to the hand-written README. ## Key Files -The `list-agent` package is structured around several key files: +The `list-agent` package is organized into several key files that define its functionality and behavior: -- **Manifest**: [listManifest.json](./src/listManifest.json) describes the agent, including its emoji representation and schema details. -- **Schema**: [listSchema.ts](./src/listSchema.ts) defines the types and actions supported by the agent. -- **Grammar**: [listSchema.agr](./src/listSchema.agr) contains patterns for user requests that map to actions. -- **Handler**: [listActionHandler.ts](./src/listActionHandler.ts) implements the logic for executing actions. +- **[listManifest.json](./src/listManifest.json)**: This file contains metadata about the agent, including its description, emoji representation, and references to the schema and grammar files. +- **[listSchema.ts](./src/listSchema.ts)**: This file defines the action types and their parameters. It is the primary source for understanding the actions the agent can perform. +- **[listSchema.agr](./src/listSchema.agr)**: This file contains grammar rules that map user utterances to specific actions. It defines how the agent interprets user input and translates it into actionable commands. +- **[listActionHandler.ts](./src/listActionHandler.ts)**: This file implements the logic for handling the actions defined in the schema. It is the core of the agent's functionality, processing user requests and executing the corresponding actions. -### Key Files and Their Responsibilities +### File Responsibilities -- [listManifest.json](./src/listManifest.json): Contains metadata about the agent, including its description and the schema file locations. -- [listSchema.ts](./src/listSchema.ts): Defines the action types and their parameters. This file is crucial for understanding what actions the agent can perform. -- [listSchema.agr](./src/listSchema.agr): Contains the grammar rules that map user utterances to actions. This file helps in parsing and understanding user requests. -- [listActionHandler.ts](./src/listActionHandler.ts): Implements the logic for handling the actions defined in the schema. This file is where the actual functionality of each action is coded. +- **Manifest**: The [listManifest.json](./src/listManifest.json) file provides a high-level description of the agent and its capabilities. It also specifies the locations of the schema and grammar files. +- **Schema**: The [listSchema.ts](./src/listSchema.ts) file is where the action types (`addItems`, `removeItems`, etc.) and their parameters are defined. This file is essential for understanding the agent's capabilities and how to extend them. +- **Grammar**: The [listSchema.agr](./src/listSchema.agr) file contains patterns for user requests that map to the defined actions. For example, it includes rules for interpreting phrases like "Add milk to my grocery list" or "Clear my to-do list." +- **Handler**: The [listActionHandler.ts](./src/listActionHandler.ts) file implements the logic for each action. It processes the parsed user input and performs the necessary operations, such as adding items to a list or retrieving a list's contents. ## How to extend -To extend the `list-agent` package, follow these steps: +To extend the `list-agent` package, you can add new actions, modify existing ones, or update the grammar to support additional user input patterns. Here are the steps to follow: + +### Adding a New Action + +1. **Define the Action**: + + - Add the new action type and its parameters to [listSchema.ts](./src/listSchema.ts). + - For example, if you want to add an action to rename a list, define a new type `RenameListAction` with the necessary parameters. + +2. **Update the Grammar**: + + - Add grammar rules for the new action in [listSchema.agr](./src/listSchema.agr). These rules should map user utterances to the new action and its parameters. + +3. **Implement the Handler**: + - Add the logic for the new action in [listActionHandler.ts](./src/listActionHandler.ts). Use the `executeListAction` function as the entry point for handling the action. + +### Modifying Existing Actions + +1. **Update the Schema**: + + - Modify the action type definitions in [listSchema.ts](./src/listSchema.ts) to reflect the changes. + +2. **Adjust the Grammar**: + + - Update the corresponding grammar rules in [listSchema.agr](./src/listSchema.agr) to account for the changes in the action's parameters or behavior. -1. **Add a new action**: +3. **Revise the Handler**: + - Update the logic in [listActionHandler.ts](./src/listActionHandler.ts) to handle the modified action correctly. - - Define the new action type in [listSchema.ts](./src/listSchema.ts). - - Add corresponding grammar patterns in [listSchema.agr](./src/listSchema.agr). - - Implement the action handling logic in [listActionHandler.ts](./src/listActionHandler.ts). +### Testing Your Changes -2. **Modify existing actions**: +1. **Write Tests**: - - Update the action type definitions in [listSchema.ts](./src/listSchema.ts). - - Adjust grammar patterns in [listSchema.agr](./src/listSchema.agr) as needed. - - Modify the handling logic in [listActionHandler.ts](./src/listActionHandler.ts). + - Add or update tests to cover the new or modified actions. Ensure that all possible scenarios are tested, including edge cases. -3. **Test your changes**: - - Ensure that your changes are covered by tests. Add or update tests in the appropriate test files. - - Run the tests to verify that your changes work as expected. +2. **Run Tests**: + - Use the testing framework specified in the monorepo to run the tests and verify that your changes work as expected. -By following these steps, you can extend the functionality of the `list-agent` package to meet your specific requirements. +By following these steps, you can extend the `list-agent` package to support additional functionality or adapt it to your specific use case. ## Reference @@ -82,7 +105,7 @@ By following these steps, you can extend the functionality of the `list-agent` p ### Entry points - `./agent/manifest` → [./src/listManifest.json](./src/listManifest.json) -- `./agent/handlers` → [./dist/listActionHandler.js](./dist/listActionHandler.js) +- `./agent/handlers` → `./dist/listActionHandler.js` _(not found on disk)_ ### Dependencies @@ -124,6 +147,6 @@ _6 actions implemented by this agent, parsed deterministically from `./src/listS --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter list-agent docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter list-agent docs:verify-links` to spot-check._ diff --git a/ts/packages/agents/montage/README.AUTOGEN.md b/ts/packages/agents/montage/README.AUTOGEN.md index 99a89f01f..cb1c2c55c 100644 --- a/ts/packages/agents/montage/README.AUTOGEN.md +++ b/ts/packages/agents/montage/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # montage-agent — AI-generated documentation @@ -12,43 +12,79 @@ ## Overview -The montage-agent package is a TypeAgent application agent designed to assist in creating photo montages. It facilitates collaborative work between the user and the agent to manage and organize images into montages. +The `montage-agent` package is a TypeAgent application agent designed to assist users in creating and managing photo montages. It serves as a collaborative tool, enabling users to interact with the agent to organize, edit, and display collections of images. The agent integrates with other system components for image processing, storage, and user interaction through a web interface. ## What it does -The montage-agent package provides a set of actions that allow users to create, modify, and manage photo montages. These actions include `createMontage`, `addPhotos`, `removePhotos`, `selectPhotos`, `setMontageViewMode`, and more. The agent interacts with various parts of the system, including image storage and processing, to perform these tasks. It also provides a web interface for users to interact with the montage creation process. +The `montage-agent` provides a range of actions to facilitate the creation and management of photo montages. These actions include: -## Setup +- **Montage Creation and Management**: Actions like `createMontage`, `deleteMontage`, and `listMontage` allow users to create new montages, delete existing ones, and view a list of available montages. +- **Photo Management**: Actions such as `addPhotos`, `removePhotos`, `selectPhotos`, and `clearSelection` enable users to add, remove, and manage photos within a montage. +- **Montage Customization**: Actions like `setMontageViewMode`, `changeTitle`, and `setSearchParameters` allow users to customize the appearance and behavior of their montages. +- **Interactive Features**: Actions such as `startSlideShow` and `showSearchParameters` provide interactive capabilities for users to engage with their montages. + +The agent also includes a web-based interface for users to interact with the montage creation process. This interface allows users to view, edit, and manage their montages in a user-friendly environment. -To set up the montage-agent package, you need to configure several environment variables and ensure that certain dependencies are installed. The required environment variables include: +## Setup -- `PORT`: The port number on which the server will run. -- `INDEX_CACHE_PATH`: The path to the image index cache. -- `ROOT_IMAGE_FOLDER`: The root folder where images are stored. +To set up the `montage-agent` package, follow these steps: -Additionally, you need to install the necessary dependencies using `pnpm install`. For detailed setup instructions, refer to the hand-written README. +1. **Install Dependencies**: Run `pnpm install` in the package directory to install all required dependencies. +2. **Configure Environment Variables**: Set the following environment variables: + - `PORT`: The port number on which the server will run. + - `INDEX_CACHE_PATH`: The file path to the image index cache. + - `ROOT_IMAGE_FOLDER`: The root directory where images are stored. +3. **Run the Server**: Start the server by providing the configured port as a command-line argument. For example: + ```bash + node dist/route/route.js + ``` +4. For additional setup details, refer to the hand-written README. ## Key Files -The montage-agent package is organized into several key components: +The `montage-agent` package is structured into several key files, each serving a specific purpose: -- **Agent Manifest**: The [montageManifest.json](./src/agent/montageManifest.json) file defines the agent's schema and description. -- **Action Handler**: The [montageActionHandler.ts](./src/agent/montageActionHandler.ts) file contains the logic for executing actions related to montage creation and management. -- **Action Schema**: The [montageActionSchema.ts](./src/agent/montageActionSchema.ts) file defines the types and structure of actions that the agent can perform. -- **Route Handling**: The [route.ts](./src/route/route.ts) file sets up the Express server and handles HTTP requests. -- **Web Interface**: The [index.html](./src/site/index.html), [index.ts](./src/site/index.ts), and [styles.css](./src/site/styles.css) files provide the user interface for interacting with the montage agent. +- **[montageManifest.json](./src/agent/montageManifest.json)**: Defines the agent's metadata, including its description, emoji representation, and schema details. +- **[montageActionHandler.ts](./src/agent/montageActionHandler.ts)**: Implements the logic for handling montage-related actions, such as creating montages, adding photos, and managing montage views. +- **[montageActionSchema.ts](./src/agent/montageActionSchema.ts)**: Specifies the schema for the actions, activities, and entities that the agent can process. This includes type definitions for actions like `addPhotos`, `removePhotos`, and `setMontageViewMode`. +- **[route.ts](./src/route/route.ts)**: Sets up the Express server, handles HTTP requests, and enforces security measures such as origin allowlists. +- **[originAllowlist.ts](./src/route/originAllowlist.ts)**: Implements an origin allowlist to restrict access to the server, ensuring only authorized requests are processed. +- **Web Interface Files**: + - [index.html](./src/site/index.html): The main HTML file for the web interface, providing the structure for user interaction. + - [index.ts](./src/site/index.ts): Contains the client-side logic for the web interface. + - [photo.ts](./src/site/photo.ts): Handles photo-related operations in the web interface. ## How to extend -To extend the montage-agent package, follow these steps: +To extend the functionality of the `montage-agent` package, follow these steps: + +1. **Define New Actions**: + + - Add new action types to the [montageActionSchema.ts](./src/agent/montageActionSchema.ts) file. Clearly define the action name, parameters, and expected behavior. + - For example, to add a new action for filtering photos, define a new type in the schema file with the required parameters. + +2. **Implement Action Logic**: + + - Implement the logic for the new actions in the [montageActionHandler.ts](./src/agent/montageActionHandler.ts) file. + - Follow the existing patterns for action handling, such as using helper functions like `createActionResult` or `createActionResultFromError`. + +3. **Update the Agent Manifest**: + + - Add the new actions to the [montageManifest.json](./src/agent/montageManifest.json) file. Include a description of the action and its purpose. + +4. **Modify the Web Interface (if needed)**: + + - Update the web interface files ([index.html](./src/site/index.html), [index.ts](./src/site/index.ts), etc.) to include UI elements or interactions for the new actions. + +5. **Test Your Changes**: + + - Write unit tests to validate the new actions. Ensure that the tests cover various scenarios, including edge cases. + - Run the server and test the new functionality through the web interface. -1. **Add New Actions**: Define new actions in the [montageActionSchema.ts](./src/agent/montageActionSchema.ts) file. Ensure that the action types and parameters are clearly specified. -2. **Implement Action Logic**: Implement the logic for the new actions in the [montageActionHandler.ts](./src/agent/montageActionHandler.ts) file. Use the existing patterns for handling actions and interacting with other system components. -3. **Update Manifest**: Update the [montageManifest.json](./src/agent/montageManifest.json) file to include the new actions and their descriptions. -4. **Test**: Write tests to verify the functionality of the new actions. Ensure that the tests cover various scenarios and edge cases. -5. **Run the Server**: Start the server using the configured port and test the new actions through the web interface. +6. **Documentation**: + - Update the hand-written README or other documentation to reflect the new features and provide usage instructions. -By following these steps, you can extend the capabilities of the montage-agent package to support additional features and functionalities. +By following these steps, you can enhance the `montage-agent` package to support additional use cases and improve its functionality. ## Reference @@ -56,9 +92,9 @@ By following these steps, you can extend the capabilities of the montage-agent p ### Entry points -- default → [./dist/route/route.js](./dist/route/route.js) +- default → `./dist/route/route.js` _(not found on disk)_ - `./agent/manifest` → [./src/agent/montageManifest.json](./src/agent/montageManifest.json) -- `./agent/handlers` → [./dist/agent/montageActionHandler.js](./dist/agent/montageActionHandler.js) +- `./agent/handlers` → `./dist/agent/montageActionHandler.js` _(not found on disk)_ ### Dependencies @@ -95,6 +131,6 @@ External: `body-parser`, `d3`, `d3-cloud`, `debug`, `express`, `express-rate-lim --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter montage-agent docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter montage-agent docs:verify-links` to spot-check._ diff --git a/ts/packages/agents/screencapture/README.AUTOGEN.md b/ts/packages/agents/screencapture/README.AUTOGEN.md index 5a701b906..18eb2ffda 100644 --- a/ts/packages/agents/screencapture/README.AUTOGEN.md +++ b/ts/packages/agents/screencapture/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # screencapture-agent — AI-generated documentation @@ -12,71 +12,103 @@ ## Overview -The `screencapture-agent` is a TypeAgent application agent designed for screen capture and recording. It supports taking screenshots and recording the screen on Windows and Linux (X11), including targeting specific program or window names. +The `screencapture-agent` is a TypeAgent application agent designed to handle screen capture and recording tasks. It supports taking screenshots and recording the screen on Windows and Linux (X11), with the ability to target specific application windows or capture the entire screen. The agent integrates with system-level tools like `ffmpeg`, `wmctrl`, and `xdotool` to perform its operations. ## What it does -This agent provides several actions for screen capture and recording: +The `screencapture-agent` provides the following actions: -- `takeScreenshot`: Takes a screenshot of the entire screen or a specified window. -- `startRecording`: Starts recording the screen or a specified window. -- `stopRecording`: Stops the currently active screen recording. -- `listWindows`: Lists all currently visible windows to help users target them by name. -- `recording`: Tracks the activity while a recording is in progress. +- **`takeScreenshot`**: Captures a screenshot of the entire screen or a specific window. If a `target` parameter is provided, the agent attempts to match it to a visible window by name (e.g., "Chrome" or "Visual Studio"). +- **`startRecording`**: Initiates a screen recording. Similar to `takeScreenshot`, the `target` parameter can specify a particular window or application to record. +- **`stopRecording`**: Stops the currently active screen recording. +- **`listWindows`**: Lists all currently visible windows, allowing users to identify and target specific windows by name. +- **`recording`**: Tracks the activity of an ongoing recording, including details such as the target, output file path, and start time. -Captured files are stored under the agent's session storage (`screenshots/` and `recordings/` siblings) and surfaced as entities in the action result. +Captured screenshots and recordings are saved in the agent's session storage under `screenshots/` and `recordings/` directories. These files are also surfaced as entities in the action results, making them accessible for further processing or sharing. + +### Platform Support + +- **Windows 10/11**: Supports full-screen and per-window capture using `gdigrab`. Window enumeration is performed via PowerShell's `Get-Process` command. +- **Linux (X11)**: Supports full-screen and per-window capture using `x11grab`. Window enumeration is achieved with `wmctrl -lp`, and per-window geometry is determined using `xdotool getwindowgeometry`. +- **Linux (Wayland)**: Not supported in this version. The agent detects `XDG_SESSION_TYPE=wayland` and provides a clear error message. Users are advised to switch to an X11 session. +- **macOS and other platforms**: Not supported. ## Setup -To use the `screencapture-agent`, ensure the following tools are installed on your system: +To use the `screencapture-agent`, you need to install specific system tools and configure environment variables. -### Windows +### Required Tools -- `ffmpeg`: Install via `winget install Gyan.FFmpeg` or download from `https://ffmpeg.org`. +The agent relies on external system binaries for its operations. These tools are not bundled with the agent and must be installed separately. -### Linux +#### Windows -- `ffmpeg`: Install via `sudo apt install ffmpeg`, `sudo dnf install ffmpeg`, or `sudo pacman -S ffmpeg`. -- `wmctrl`: Install via `sudo apt install wmctrl`. -- `xdotool`: Install via `sudo apt install xdotool`. +- **`ffmpeg`**: Install using `winget install Gyan.FFmpeg` or download it from `https://ffmpeg.org`. -Additionally, set the following environment variables: +#### Linux -- `DISPLAY` -- `XDG_SESSION_TYPE` +- **`ffmpeg`**: Install using your package manager: + - Debian/Ubuntu: `sudo apt install ffmpeg` + - Fedora: `sudo dnf install ffmpeg` + - Arch Linux: `sudo pacman -S ffmpeg` +- **`wmctrl`**: Install using `sudo apt install wmctrl` (Debian/Ubuntu). +- **`xdotool`**: Install using `sudo apt install xdotool` (Debian/Ubuntu). + +If any required tool is missing, the agent will provide an actionable installation hint when you attempt to use an action that depends on it. + +### Environment Variables + +The following environment variables must be set: + +- `DISPLAY`: Required for graphical display on Linux systems. +- `XDG_SESSION_TYPE`: Must be set to `x11` for Linux systems. Wayland is not supported. -For detailed setup instructions, see the hand-written README. +For more details on setting up these tools and environment variables, refer to the hand-written README. ## Key Files -The `screencapture-agent` is structured into several key files: +The `screencapture-agent` is organized into several key files that define its functionality and structure: -- [screencaptureActionHandler.ts](./src/screencaptureActionHandler.ts): Contains the logic for handling screen capture and recording actions. -- [screencaptureManifest.json](./src/screencaptureManifest.json): Defines the agent's manifest, including its description and schema. -- [screencaptureSchema.ts](./src/screencaptureSchema.ts): Defines the types and parameters for the actions supported by the agent. -- [screencaptureSchema.agr](./src/screencaptureSchema.agr): Contains the grammar rules for parsing user commands. -- [context.ts](./src/context.ts): Manages the context for active recordings and tool installations. -- [platform/](./src/platform/): Contains platform-specific logic for Windows and Linux, including tool detection and command execution. +- **[screencaptureActionHandler.ts](./src/screencaptureActionHandler.ts)**: Implements the logic for all supported actions, including taking screenshots, starting/stopping recordings, and listing windows. +- **[screencaptureManifest.json](./src/screencaptureManifest.json)**: Defines the agent's manifest, including its description, schema, and capabilities. +- **[screencaptureSchema.ts](./src/screencaptureSchema.ts)**: Specifies the types and parameters for the actions, ensuring proper validation and execution. +- **[screencaptureSchema.agr](./src/screencaptureSchema.agr)**: Contains grammar rules that map user commands to specific actions. +- **[context.ts](./src/context.ts)**: Manages the agent's runtime context, including active recordings and tool installation states. +- **[platform/](./src/platform/)**: A directory containing platform-specific logic for Windows and Linux, such as tool detection, command execution, and window enumeration. ### Key Components -- **Action Handler**: The [screencaptureActionHandler.ts](./src/screencaptureActionHandler.ts) file is responsible for implementing the logic for each action, such as taking screenshots, starting and stopping recordings, and listing windows. -- **Manifest**: The [screencaptureManifest.json](./src/screencaptureManifest.json) file describes the agent, including its capabilities and the schema it uses. -- **Schema**: The [screencaptureSchema.ts](./src/screencaptureSchema.ts) file defines the types and parameters for the actions, ensuring that the agent can correctly interpret and execute user commands. -- **Grammar**: The [screencaptureSchema.agr](./src/screencaptureSchema.agr) file contains the grammar rules that map user utterances to specific actions. -- **Context Management**: The [context.ts](./src/context.ts) file manages the context for active recordings and tool installations, ensuring that the agent can track ongoing activities and handle tool setup prompts. -- **Platform-Specific Logic**: The [platform/](./src/platform/) directory contains modules for Windows and Linux, handling tasks such as tool detection, command execution, and window enumeration. +1. **Action Handler**: The [screencaptureActionHandler.ts](./src/screencaptureActionHandler.ts) file is the core of the agent, implementing the logic for each action. It interacts with platform-specific modules to execute tasks like capturing screenshots and managing recordings. +2. **Manifest and Schema**: The [screencaptureManifest.json](./src/screencaptureManifest.json) and [screencaptureSchema.ts](./src/screencaptureSchema.ts) files define the agent's capabilities and the structure of its actions. +3. **Grammar**: The [screencaptureSchema.agr](./src/screencaptureSchema.agr) file contains the grammar rules that enable the agent to interpret user commands and map them to actions. +4. **Platform-Specific Logic**: The [platform/](./src/platform/) directory includes modules for handling platform-specific requirements, such as detecting and using system tools (`ffmpeg`, `wmctrl`, `xdotool`) and managing platform-specific capture methods. ## How to extend -To extend the `screencapture-agent`, follow these steps: +To add new features or modify existing functionality in the `screencapture-agent`, follow these steps: + +1. **Define New Actions**: + + - Add new action types and parameters in [screencaptureSchema.ts](./src/screencaptureSchema.ts). + - Update the schema to include the new action definitions. + +2. **Update Grammar**: + + - Add new grammar rules in [screencaptureSchema.agr](./src/screencaptureSchema.agr) to map user commands to the new actions. + +3. **Implement Action Logic**: + + - Extend the [screencaptureActionHandler.ts](./src/screencaptureActionHandler.ts) file to handle the new actions. This may involve adding new methods or modifying existing ones. + +4. **Add Platform-Specific Support**: + + - If the new action requires platform-specific logic, update or add modules in the [platform/](./src/platform/) directory. For example, you might need to add new commands for `ffmpeg` or other tools. -1. **Add new actions**: Define new action types in [screencaptureSchema.ts](./src/screencaptureSchema.ts). -2. **Update grammar**: Add corresponding grammar rules in [screencaptureSchema.agr](./src/screencaptureSchema.agr). -3. **Implement handlers**: Extend the logic in [screencaptureActionHandler.ts](./src/screencaptureActionHandler.ts) to handle the new actions. -4. **Test**: Run tests to ensure the new actions work correctly. You can use the TypeAgent Shell or CLI to invoke the actions and verify their behavior. +5. **Test Your Changes**: + - Use the TypeAgent Shell or CLI to test the new actions. Verify that they work as expected and produce the desired results. + - Add unit tests for the new functionality to ensure reliability and maintainability. -By following these steps, you can add new capabilities to the `screencapture-agent` and enhance its functionality. +By following these steps, you can extend the `screencapture-agent` to support additional screen capture and recording features or adapt it to new platforms and tools. ## Reference @@ -85,7 +117,7 @@ By following these steps, you can add new capabilities to the `screencapture-age ### Entry points - `./agent/manifest` → [./src/screencaptureManifest.json](./src/screencaptureManifest.json) -- `./agent/handlers` → [./dist/screencaptureActionHandler.js](./dist/screencaptureActionHandler.js) +- `./agent/handlers` → `./dist/screencaptureActionHandler.js` _(not found on disk)_ ### Dependencies @@ -135,6 +167,6 @@ _5 actions implemented by this agent, parsed deterministically from `./src/scree --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter screencapture-agent docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter screencapture-agent docs:verify-links` to spot-check._ diff --git a/ts/packages/aiclient/README.AUTOGEN.md b/ts/packages/aiclient/README.AUTOGEN.md index bdf069cb1..1b42a2dc4 100644 --- a/ts/packages/aiclient/README.AUTOGEN.md +++ b/ts/packages/aiclient/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # @typeagent/aiclient — AI-generated documentation @@ -12,72 +12,94 @@ ## Overview -The `@typeagent/aiclient` package is a TypeScript library designed to facilitate interactions with various AI APIs used by the Microsoft AI Systems team. It is primarily intended for sample agents and examples within the TypeAgent project. +The `@typeagent/aiclient` package is a TypeScript library designed to interact with various AI APIs utilized by the Microsoft AI Systems team. It is primarily intended for use in sample agents and examples within the TypeAgent project. ## What it does -The `aiclient` package supports calling AI endpoints and other REST services, specifically: +The `aiclient` package provides functionality for interacting with AI services, including: -- OpenAI model endpoints, both on Azure and OpenAI. -- Bing search APIs. +- **OpenAI model endpoints**: Supports both Azure-hosted and OpenAI-hosted models. +- **Bing search APIs**: Enables integration with Bing for search capabilities. -The library includes functionality for managing settings required to call these services, which are typically sourced from environment variables. It also features multi-region endpoint pools to handle API requests efficiently, rotating among endpoints to mitigate throttling and ensure high availability. +### Multi-region Endpoint Pools -### Multi-region endpoint pools +A key feature of this package is its ability to manage multi-region endpoint pools. These pools allow the client to distribute requests across multiple endpoints, each corresponding to a specific region or variant of a model. This approach helps mitigate issues such as throttling, timeouts, or regional outages by rotating requests among available endpoints. -The package includes a sophisticated mechanism for managing multi-region endpoint pools. Chat, embedding, and image factories resolve each model into an endpoint pool — a list of endpoints (one per region + variant) that the client rotates among on 429 / 5xx / timeout errors. This helps to survive single-region throttling without user-visible stalls and keeps a PTU reservation preferred when one is configured. +#### Endpoint Pool Discovery -### Selection algorithm +The library automatically discovers endpoints for a given model by scanning environment variables. For example, for a model named `GPT_4_O`, it looks for variables like: -Endpoints are grouped by priority tiers, with random selection within each tier. The lowest-priority tier that still has at least one healthy (non-cooling-down) member wins; within that tier, one member is picked uniformly at random. This ensures that multiple client processes spread across the regions in that tier instead of stampeding the same endpoint. +- `AZURE_OPENAI_ENDPOINT_GPT_4_O` +- `AZURE_OPENAI_ENDPOINT_GPT_4_O_` (e.g., `_EASTUS`, `_SWEDENCENTRAL`) +- `AZURE_OPENAI_ENDPOINT_GPT_4_O__PTU` (indicating a provisioned-throughput reservation) -On failure: +Matching API keys are also discovered using similar patterns, such as `AZURE_OPENAI_API_KEY_GPT_4_O_`. -- **429**: Parses `Retry-After`, marks the member as cooling for `max(Retry-After, base × 2^consecutive_429s)`, capped at 120 seconds, and rotates to the next healthy member. -- **5xx / timeout / network error**: Applies a floor cooldown of 5 seconds and rotates. -- **Non-transient 4xx** (e.g., 401): Returns immediately without rotating. +#### Endpoint Selection and Rotation + +The selection algorithm groups endpoints into priority tiers. The client selects an endpoint from the lowest-priority tier that has at least one healthy member, choosing randomly within the tier. This ensures that requests are distributed across available endpoints, reducing the risk of overloading a single endpoint. + +When an endpoint fails, the client applies cooldowns and rotates to the next healthy endpoint. The cooldown duration depends on the type of failure: + +- **429 (Too Many Requests)**: The client parses the `Retry-After` header and applies a cooldown based on the retry interval, with a maximum of 120 seconds. +- **5xx errors, timeouts, or network errors**: A minimum cooldown of 5 seconds is applied. +- **Non-transient 4xx errors (e.g., 401)**: The client does not rotate and returns the error immediately. + +The library also allows users to override the default priority and mode of endpoints by setting a JSON array in environment variables, such as `AZURE_OPENAI_POOL_`. + +### Debug Logging + +The package supports debug logging for endpoint pool operations, including selection, rotation, and cooldown events. To enable logging, set the `DEBUG` environment variable to include the `typeagent:pool` namespace: + +```bash +DEBUG=typeagent:pool,typeagent:rest:retry node your-app.js +``` ## Setup -To use the `aiclient` package, certain environment variables need to be set up. These include: +To use the `@typeagent/aiclient` package, you need to configure the following environment variables: + +- `TYPEAGENT_COPILOT_SDK_LOG_LEVEL`: Specifies the log level for the Copilot SDK. Valid values include `none`, `error`, `warning`, `info`, `debug`, and `all`. + +Additionally, the library relies on a set of environment variables for endpoint pool discovery and API key management. These include: - `AZURE_OPENAI_ENDPOINT__`: Specifies the endpoint for a given model and region. - `AZURE_OPENAI_API_KEY__`: Specifies the API key for a given model and region. - `BING_API_KEY`: Specifies the API key for Bing search APIs. -For detailed setup instructions, including how to provision more endpoints and configure multi-region deploy and secret-sync tooling, see the hand-written README. +For detailed instructions on setting up these environment variables, refer to the hand-written README. It also provides guidance on provisioning additional endpoints and using the multi-region deployment and secret-sync tooling. ## Key Files -The `aiclient` package is organized into several key files, each responsible for different aspects of the library: +The `aiclient` package is structured into several key files, each responsible for specific functionality: -- [index.ts](./src/index.ts): The main entry point, exporting various modules and functions. -- [auth.ts](./src/auth.ts): Handles authentication, including token management and Azure credentials. -- [azureSettings.ts](./src/azureSettings.ts): Manages settings for Azure OpenAI services, loading them from environment variables. -- [bing.ts](./src/bing.ts): Contains functions and types for interacting with Bing search APIs. -- [common.ts](./src/common.ts): Utility functions for retrieving and managing environment settings. -- [endpointPool.ts](./src/endpointPool.ts): Manages endpoint pools, including selection algorithms and cooldown mechanisms. -- [modelResource.ts](./src/modelResource.ts): Functions for managing model resources, including concurrency settings. -- [models.ts](./src/models.ts): Types and settings for various AI models, including completion settings and JSON schema definitions. +- [index.ts](./src/index.ts): The main entry point, exporting the library's primary modules and functions. +- [auth.ts](./src/auth.ts): Manages authentication, including token retrieval and Azure credentials. +- [azureSettings.ts](./src/azureSettings.ts): Handles configuration for Azure OpenAI services, including loading settings from environment variables. +- [bing.ts](./src/bing.ts): Provides functions and types for interacting with Bing search APIs. +- [common.ts](./src/common.ts): Contains utility functions for managing environment variables and settings. +- [endpointPool.ts](./src/endpointPool.ts): Implements the logic for managing endpoint pools, including selection and rotation algorithms. +- [modelResource.ts](./src/modelResource.ts): Manages model resources, including concurrency settings. +- [models.ts](./src/models.ts): Defines types and settings for various AI models, such as completion settings and JSON schema definitions. ## How to extend -To extend the `aiclient` package, follow these steps: +To extend the `@typeagent/aiclient` package, follow these steps: -1. **Identify the area to extend**: Determine which part of the library you need to modify or add functionality to. For example, if you need to add support for a new AI model, you might start with [models.ts](./src/models.ts). +1. **Identify the area to extend**: Determine the specific functionality you want to add or modify. For example, to support a new AI model, you might start by examining [models.ts](./src/models.ts). -2. **Modify or add code**: Open the relevant file and implement your changes. Follow the existing patterns and structures to ensure consistency. For example, if adding a new model, define its settings and types similarly to existing models. +2. **Modify or add code**: Implement your changes in the relevant file. For instance, if you're adding a new model, define its settings and types in a manner consistent with the existing codebase. -3. **Update environment variables**: If your changes require new environment variables, update the setup instructions and ensure they are correctly loaded in the relevant settings files. +3. **Update environment variables**: If your changes require new environment variables, update the setup instructions and ensure they are correctly loaded in the appropriate settings files. -4. **Test your changes**: Write tests to verify your modifications. You can find existing tests in the `./test` directory. Ensure your tests cover various scenarios and edge cases. +4. **Write tests**: Add tests to validate your changes. Use the existing tests in the `./test` directory as a reference for writing new test cases. -5. **Run tests**: Execute the tests to ensure everything works as expected. You can run the tests using the following command: +5. **Run tests**: Execute the tests to ensure your changes work as expected. Use the following command to run the tests: ```bash pnpm test ``` -By following these steps, you can effectively extend the functionality of the `aiclient` package while maintaining its integrity and consistency. +By following these steps, you can extend the `aiclient` package while maintaining its functionality and compatibility with the rest of the TypeAgent project. ## Reference @@ -94,7 +116,7 @@ Workspace: - [@typeagent/config](../../packages/config/README.md) -External: `@azure/identity`, `@github/copilot-sdk`, `async`, `debug`, `typechat` +External: `@azure/identity`, `@github/copilot-sdk`, `@huggingface/transformers`, `async`, `debug`, `typechat` ### Used by @@ -112,10 +134,16 @@ External: `@azure/identity`, `@github/copilot-sdk`, `async`, `debug`, `typechat` ### Files of interest -`./src/index.ts`, `./src/apiSettingsFromConfig.ts`, `./src/auth.ts`, …and 18 more under `./src/`. +`./src/index.ts`, `./src/apiSettingsFromConfig.ts`, `./src/auth.ts`, …and 20 more under `./src/`. + +### Environment variables + +_1 environment variable referenced from `./src/` (set in `ts/.env` or your shell). See the `## Setup` section above for guidance on obtaining each value._ + +- `TYPEAGENT_COPILOT_SDK_LOG_LEVEL` --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter @typeagent/aiclient docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter @typeagent/aiclient docs:verify-links` to spot-check._ diff --git a/ts/packages/chat-ui/README.AUTOGEN.md b/ts/packages/chat-ui/README.AUTOGEN.md index e5d72432c..20e8c9954 100644 --- a/ts/packages/chat-ui/README.AUTOGEN.md +++ b/ts/packages/chat-ui/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # chat-ui — AI-generated documentation @@ -12,63 +12,85 @@ ## Overview -The `chat-ui` package provides shared DOM-based chat rendering for TypeAgent surfaces, including the VS Code shell extension and the browser extension chat panel. It offers a framework-free chat UI that supports rendering user and agent bubbles, streaming display updates, dynamic status, history replay, command completions, and metrics tooltips. +The `chat-ui` package provides a shared, framework-free, DOM-based chat rendering solution for TypeAgent surfaces. It is used by the VS Code shell extension, the browser extension chat panel, and other TypeAgent hosts. The package includes components and utilities for rendering user and agent messages, handling streaming updates, replaying chat history, managing command completions, and displaying dynamic status and feedback. ## What it does -The `chat-ui` package includes several key components and functionalities: +The `chat-ui` package is designed to provide a consistent and customizable chat interface for various TypeAgent hosts. Its primary features include: -- **ChatPanel**: The main component for rendering the chat interface. It handles user input, agent messages, and display updates. -- **FeedbackWidget**: A component for collecting user feedback on interactions within the chat panel. -- **PartialCompletion**: Manages command completions using the `@typeagent/completion-ui` package. -- **PlatformAdapter**: Abstracts platform-specific behaviors for handling link clicks and settings. -- **Styles**: Shared CSS styles for consistent chat UI appearance across different hosts. +- **ChatPanel**: The main component for rendering the chat interface. It supports user input, agent messages, streaming updates, and history replay. Hosts can use actions like `addAgentMessage`, `setDisplayInfo`, and `replayHistory` to interact with the chat panel. +- **FeedbackWidget**: A component for collecting user feedback on interactions within the chat interface. It supports feedback submission and UI management. +- **PartialCompletion**: Integrates with the `@typeagent/completion-ui` package to provide command completions. It handles input updates, completion suggestions, and user selection. +- **PlatformAdapter**: Provides an abstraction layer for platform-specific behaviors, such as handling link clicks and settings, enabling the chat UI to work across different environments like Electron and Chrome extensions. +- **Connection Status Management**: Includes utilities for rendering and managing the agent-server connection status, including reconnect options and error handling. +- **Shared Styles**: The package includes a CSS file (`styles/chat.css`) to ensure a consistent visual appearance across all hosts. -The package supports actions such as `addAgentMessage`, `setDisplayInfo`, and `replayHistory`, which allow hosts to interact with the chat panel and update its content dynamically. +The package also includes built-in support for sanitizing HTML content using `DOMPurify`, ensuring safe rendering of dynamic content. ## Setup -To use the `chat-ui` package, you need to install it and its dependencies. Ensure you have the following workspace dependencies: +To use the `chat-ui` package, follow these steps: -- `@typeagent/agent-sdk` -- `@typeagent/completion-ui` -- `@typeagent/dispatcher-types` +1. **Install the package**: Add the `chat-ui` package to your project using your package manager. For example, with `pnpm`: -Additionally, the package relies on external dependencies such as `ansi_up`, `dompurify`, and `markdown-it`. + ```sh + pnpm install chat-ui + ``` -For detailed setup instructions, including environment variables and API keys, refer to the hand-written README. +2. **Install dependencies**: Ensure the following workspace dependencies are installed: -## Key Files + - `@typeagent/agent-sdk` + - `@typeagent/completion-ui` + - `@typeagent/dispatcher-types` -The `chat-ui` package is organized into several key files: + Additionally, the package relies on the following external dependencies: -- **[index.ts](./src/index.ts)**: Exports the main components and types used by the package. -- **[chatPanel.ts](./src/chatPanel.ts)**: Implements the `ChatPanel` component, handling user input, agent messages, and display updates. -- **[feedbackWidget.ts](./src/feedbackWidget.ts)**: Implements the `FeedbackWidget` component for collecting user feedback. -- **[partialCompletion.ts](./src/partialCompletion.ts)**: Manages command completions using the `@typeagent/completion-ui` package. -- **[platformAdapter.ts](./src/platformAdapter.ts)**: Defines the `PlatformAdapter` interface for handling platform-specific behaviors. -- **[setContent.ts](./src/setContent.ts)**: Contains functions for processing and sanitizing content before rendering. -- **[styles/chat.css](./styles/chat.css)**: Provides shared CSS styles for the chat UI. + - `ansi_up` + - `dompurify` + - `markdown-it` + +3. **Include styles**: Import the shared CSS styles in your project: + + ```ts + import "chat-ui/styles"; + ``` + +4. **Initialize the ChatPanel**: Use the `ChatPanel` component in your host application. Refer to the example in the hand-written README for initialization details. + +For further setup details, including any additional configuration, refer to the hand-written README. + +## Key Files -### Key Components +The `chat-ui` package is organized into several key files, each responsible for specific functionality: -- **ChatPanel**: The core component of the chat UI, responsible for rendering user and agent messages, handling user input, and updating the display dynamically. It uses DOMPurify to sanitize HTML content before insertion. -- **FeedbackWidget**: Collects user feedback on interactions within the chat panel. It includes methods for submitting feedback and managing the feedback UI. -- **PartialCompletion**: Integrates with the `@typeagent/completion-ui` package to manage command completions. It handles input updates, completion acceptance, and dismissal. -- **PlatformAdapter**: Abstracts platform-specific behaviors, such as handling link clicks and settings. This allows the chat UI to be adaptable to different environments like Electron and Chrome extensions. -- **Styles**: Shared CSS styles that ensure a consistent appearance of the chat UI across different hosts. +- **[index.ts](./src/index.ts)**: The main entry point of the package, exporting all public components, utilities, and types. +- **[chatPanel.ts](./src/chatPanel.ts)**: Implements the `ChatPanel` component, which is the core of the chat UI. It handles user input, agent messages, and dynamic updates. +- **[feedbackWidget.ts](./src/feedbackWidget.ts)**: Defines the `FeedbackWidget` component for collecting and managing user feedback. +- **[partialCompletion.ts](./src/partialCompletion.ts)**: Manages command completions using the `@typeagent/completion-ui` package. +- **[platformAdapter.ts](./src/platformAdapter.ts)**: Provides the `PlatformAdapter` interface for handling platform-specific behaviors. +- **[setContent.ts](./src/setContent.ts)**: Contains utility functions for processing and sanitizing content before rendering. +- **[connectionStatus.ts](./src/connectionStatus.ts)**: Manages the agent-server connection status and provides utilities for rendering connection-related UI elements. +- **[contextMenu.ts](./src/contextMenu.ts)**: Implements a lightweight context menu for copy/paste functionality in the chat interface. +- **[styles/chat.css](./styles/chat.css)**: Provides shared CSS styles for the chat UI, ensuring a consistent look and feel across different hosts. ## How to extend To extend the `chat-ui` package, follow these steps: -1. **Start with the main component**: Open the [chatPanel.ts](./src/chatPanel.ts) file to understand how the `ChatPanel` component is implemented. This is the core of the chat UI. -2. **Add new features**: Implement new functionalities or modify existing ones within the `ChatPanel` component. Ensure that any new HTML content is sanitized using DOMPurify. -3. **Update styles**: If you need to change the appearance of the chat UI, modify the [styles/chat.css](./styles/chat.css) file. -4. **Handle platform-specific behaviors**: If your extension needs to handle specific platform behaviors, update the [platformAdapter.ts](./src/platformAdapter.ts) file. -5. **Test your changes**: Run tests to ensure your changes work as expected. You can add new tests or modify existing ones to cover your new functionalities. +1. **Understand the core component**: Start by reviewing the [chatPanel.ts](./src/chatPanel.ts) file, which implements the `ChatPanel` component. This is the central part of the chat UI and is responsible for rendering messages, handling user input, and managing updates. + +2. **Add new features**: To introduce new functionality, you can extend the `ChatPanel` or other components. For example: + + - To add new message types, modify the `addAgentMessage` or `setDisplayInfo` methods in `chatPanel.ts`. + - To enhance feedback collection, extend the `FeedbackWidget` in [feedbackWidget.ts](./src/feedbackWidget.ts). + +3. **Customize styles**: If you need to adjust the appearance of the chat UI, edit the [styles/chat.css](./styles/chat.css) file. Ensure that your changes align with the overall design language of the TypeAgent ecosystem. + +4. **Handle platform-specific needs**: If your host application requires platform-specific behavior, extend the [platformAdapter.ts](./src/platformAdapter.ts) file. This file provides an abstraction layer for handling platform-specific features like link clicks and settings. + +5. **Test your changes**: After making modifications, test your changes thoroughly. Add or update tests to ensure that your new features work as intended and do not introduce regressions. -By following these steps, you can effectively extend the `chat-ui` package to meet your specific requirements. For detailed instructions and examples, refer to the hand-written README. +By following these steps, you can effectively customize and extend the `chat-ui` package to meet the needs of your specific TypeAgent host application. For additional guidance, consult the hand-written README. ## Reference @@ -76,7 +98,7 @@ By following these steps, you can effectively extend the `chat-ui` package to me ### Entry points -- default → [./dist/index.js](./dist/index.js) +- default → `./dist/index.js` _(not found on disk)_ - `./styles` → [./styles/chat.css](./styles/chat.css) ### Dependencies @@ -98,10 +120,10 @@ External: `ansi_up`, `dompurify`, `markdown-it` ### Files of interest -`./src/index.ts`, `./src/chatPanel.ts`, `./src/contextMenu.ts`, …and 9 more under `./src/`. +`./src/index.ts`, `./src/chatPanel.ts`, `./src/connectionStatus.ts`, …and 10 more under `./src/`. --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter chat-ui docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter chat-ui docs:verify-links` to spot-check._ diff --git a/ts/packages/config/README.AUTOGEN.md b/ts/packages/config/README.AUTOGEN.md index 03fba7cad..3a60c9e62 100644 --- a/ts/packages/config/README.AUTOGEN.md +++ b/ts/packages/config/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # @typeagent/config — AI-generated documentation @@ -12,61 +12,68 @@ ## Overview -The `@typeagent/config` package is a TypeScript library designed to provide a layered YAML configuration loader for the TypeAgent ecosystem. It enables the loading and merging of configuration settings from multiple sources, such as default YAML files, local overrides, environment variables, and Azure Key Vault. The package ensures that configuration settings are applied in a defined order of precedence and includes schema validation to maintain data integrity. +The `@typeagent/config` package is a TypeScript library that provides a layered YAML configuration loader for the TypeAgent ecosystem. It enables the loading, merging, and validation of configuration settings from multiple sources, ensuring a consistent and structured approach to managing application configurations. ## What it does -The `@typeagent/config` package provides the following functionality: +The `@typeagent/config` package offers the following key features: -1. **Layered Configuration Loading**: The package supports loading configuration from multiple sources, including: +1. **Layered Configuration Loading**: - - `ts/config.defaults.yaml`: A committed default configuration file. - - `ts/config.local.yaml`: A local, gitignored configuration file for overrides. - - `.env`: A legacy fallback for backward compatibility. - - Environment variables: These take the highest precedence and allow runtime overrides. - - Azure Key Vault: Future phases will include fetching configuration from Key Vault and caching it locally in an encrypted format. + - Loads configuration from multiple sources in a defined order of precedence: + 1. `.env` (legacy fallback, lowest precedence) + 2. `ts/config.defaults.yaml` (committed defaults) + 3. _Future_: Azure Key Vault YAML blob or encrypted cache + 4. `ts/config.local.yaml` (local overrides, gitignored) + 5. `process.env` (runtime overrides, highest precedence) -2. **Configuration Flattening**: Nested YAML configuration trees are flattened into flat key-value pairs that align with the `EnvVars` convention used by other TypeAgent packages. This ensures compatibility with existing code that relies on `process.env`. +2. **Configuration Flattening**: -3. **Schema Validation**: The package uses `zod` to validate configuration settings, ensuring they conform to expected schemas. + - Converts nested YAML configuration trees into flat key-value pairs that align with the `EnvVars` convention used by other TypeAgent packages. For example: + - `azure.openai.api_key` → `AZURE_OPENAI_API_KEY` + - `azure.openai.deployments[].endpoint` (with suffix `foo`) → `AZURE_OPENAI_ENDPOINT_FOO` + - This ensures compatibility with existing code that relies on `process.env`. -4. **CLI Commands**: A CLI is included for tasks such as importing `.env` files into YAML format and displaying the merged configuration. +3. **Schema Validation**: -5. **Merge Precedence**: Configuration values are merged in the following order (lowest to highest precedence): + - Uses `zod` to validate configuration settings, ensuring they conform to expected schemas and reducing the risk of misconfiguration. - - `.env` (legacy fallback) - - `ts/config.defaults.yaml` - - _Future_: Key Vault YAML blob or encrypted cache - - `ts/config.local.yaml` - - `process.env` (runtime overrides) +4. **CLI Utilities**: -6. **Redaction of Sensitive Data**: The package includes functionality to identify and redact sensitive values in configuration data, such as API keys and secrets. + - Includes commands for importing `.env` files into YAML format and displaying the merged configuration. These tools simplify the migration from legacy `.env` files to the new YAML-based configuration system. + +5. **Sensitive Data Handling**: + + - Identifies and redacts sensitive values (e.g., API keys, secrets) in configuration data to prevent accidental exposure. + +6. **Azure Key Vault Integration**: + - While not yet fully implemented, future phases will include fetching configuration from Azure Key Vault and caching it locally in an encrypted format. ## Setup -To use the `@typeagent/config` package, ensure the following environment variables are set: +To use the `@typeagent/config` package, you need to configure the following environment variables: -- `AZURE_OPENAI_`: Used for Azure OpenAI configuration. -- `JEST_WORKER_ID`: Used during Jest testing. -- `TYPEAGENT_ALLOW_KEYVAULT_IN_TESTS`: Enables Key Vault integration during tests. -- `TYPEAGENT_CONFIG_DEFAULTS`: Path to the default configuration YAML file. -- `TYPEAGENT_CONFIG_DIR`: Directory where configuration files are located. -- `TYPEAGENT_CONFIG_LOCAL`: Path to the local configuration YAML file. -- `TYPEAGENT_CONFIG_SECRET`: Secret name for the configuration in Azure Key Vault. -- `TYPEAGENT_CONFIG_VAULT`: Azure Key Vault name. -- `TYPEAGENT_DOTENV`: Path to the `.env` file. -- `TYPEAGENT_USER_DATA_DIR`: Directory for user data. +- `AZURE_OPENAI_`: Used for Azure OpenAI configuration. Refer to the hand-written README for details on how to set this. +- `JEST_WORKER_ID`: Utilized during Jest testing. +- `TYPEAGENT_ALLOW_KEYVAULT_IN_TESTS`: Enables the use of Key Vault during testing. +- `TYPEAGENT_CONFIG_DEFAULTS`: Specifies the path to the default configuration YAML file. +- `TYPEAGENT_CONFIG_DIR`: Defines the directory where configuration files are stored. +- `TYPEAGENT_CONFIG_LOCAL`: Specifies the path to the local configuration YAML file. +- `TYPEAGENT_CONFIG_SECRET`: The secret name for the configuration in Azure Key Vault. +- `TYPEAGENT_CONFIG_VAULT`: The name of the Azure Key Vault. +- `TYPEAGENT_DOTENV`: Path to the `.env` file for legacy fallback. +- `TYPEAGENT_USER_DATA_DIR`: Directory for user-specific data. -Refer to the hand-written README for detailed instructions on obtaining these values and configuring your environment. +For detailed instructions on obtaining and setting these values, refer to the hand-written README. ## Key Files -The package's source code is organized into the following key files: +The package is structured into several key files, each responsible for specific functionality: - [index.ts](./src/index.ts): Exports the main functions and types, such as `loadConfig`, `flatten`, and `fetchKeyVaultConfig`. -- [loader.ts](./src/loader.ts): Implements the core logic for loading and merging configuration settings from various sources. -- [flatten.ts](./src/flatten.ts): Handles the flattening of nested YAML configuration trees into flat key-value pairs. -- [keyVault.ts](./src/keyVault.ts): Provides functions for fetching configuration settings from Azure Key Vault. +- [loader.ts](./src/loader.ts): Implements the logic for loading and merging configuration settings from various sources. +- [flatten.ts](./src/flatten.ts): Handles the transformation of nested YAML configuration trees into flat key-value pairs. +- [keyVault.ts](./src/keyVault.ts): Provides methods for fetching configuration settings from Azure Key Vault. - [cli.ts](./src/cli.ts): Implements CLI commands for importing `.env` files and displaying the merged configuration. - [import.ts](./src/import.ts): Manages the conversion of `.env` files into YAML format and ensures compatibility with the existing configuration structure. - [redact.ts](./src/redact.ts): Contains logic for identifying and redacting sensitive values in configuration data. @@ -74,21 +81,32 @@ The package's source code is organized into the following key files: ## How to extend -To extend the functionality of the `@typeagent/config` package, consider the following approaches: +To extend the `@typeagent/config` package, follow these guidelines: + +1. **Add New Configuration Sources**: + + - Modify [loader.ts](./src/loader.ts) to include the new source and update the merging logic to incorporate it. + +2. **Customize Flattening Rules**: + + - Update [flatten.ts](./src/flatten.ts) to adjust how configuration settings are flattened. Ensure the changes align with the `EnvVars` convention used across the TypeAgent ecosystem. + +3. **Enhance Schema Validation**: -1. **Adding New Configuration Sources**: To include additional configuration sources, modify [loader.ts](./src/loader.ts) to incorporate the new source and update the merging logic. + - Modify [schema.ts](./src/schema.ts) to add or update validation rules. Use `zod` to define the new schema and ensure it aligns with the expected configuration structure. -2. **Customizing Flattening Rules**: If you need to adjust how configuration settings are flattened, update the logic in [flatten.ts](./src/flatten.ts). Ensure the changes align with the `EnvVars` convention. +4. **Expand CLI Functionality**: -3. **Enhancing Schema Validation**: To add or modify validation rules, update [schema.ts](./src/schema.ts). Use `zod` to define the new schema and validation logic. + - Add new commands to the CLI by extending [cli.ts](./src/cli.ts). Ensure the new commands integrate with the existing configuration logic. -4. **Extending CLI Functionality**: To add new CLI commands, modify [cli.ts](./src/cli.ts). Implement the new commands and ensure they integrate with the existing configuration logic. +5. **Improve Azure Key Vault Integration**: -5. **Improving Azure Key Vault Integration**: To enhance the integration with Azure Key Vault, update [keyVault.ts](./src/keyVault.ts). You can add new methods for fetching, caching, or managing secrets. + - Enhance the Azure Key Vault integration by updating [keyVault.ts](./src/keyVault.ts). You can add new methods for fetching, caching, or managing secrets. -6. **Redacting Additional Sensitive Data**: If new sensitive data types need to be redacted, update the logic in [redact.ts](./src/redact.ts). Ensure the new patterns are comprehensive and tested. +6. **Redact Additional Sensitive Data**: + - If new types of sensitive data need to be redacted, update the logic in [redact.ts](./src/redact.ts). Ensure the new patterns are comprehensive and tested. -When extending the package, ensure that new functionality adheres to the existing patterns and conventions to maintain consistency and compatibility across the TypeAgent ecosystem. +When making changes, adhere to the existing patterns and conventions in the codebase to maintain consistency and ensure compatibility with other TypeAgent packages. ## Reference @@ -140,6 +158,6 @@ _10 environment variables referenced from `./src/` (set in `ts/.env` or your she --- -_Auto-generated against commit `ff379b098decfab4eb45f78b6fa318358d7fbd75` on `2026-07-01T09:05:58.471Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter @typeagent/config docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter @typeagent/config docs:verify-links` to spot-check._ diff --git a/ts/packages/dispatcher/dispatcher/README.AUTOGEN.md b/ts/packages/dispatcher/dispatcher/README.AUTOGEN.md index 079bad8f2..c9e6d3dca 100644 --- a/ts/packages/dispatcher/dispatcher/README.AUTOGEN.md +++ b/ts/packages/dispatcher/dispatcher/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # agent-dispatcher — AI-generated documentation @@ -12,15 +12,15 @@ ## Overview -The TypeAgent Dispatcher is a core component of the TypeAgent repository that facilitates the creation of personal agents with natural language interfaces using structured prompting and large language models (LLMs). It can be integrated and hosted in various front ends, such as the TypeAgent Shell and TypeAgent CLI, and supports an extensible application agents architecture. +The TypeAgent Dispatcher is a core component of the TypeAgent repository, designed to facilitate the creation of personal agents with natural language interfaces. It leverages structured prompting and large language models (LLMs) to process user requests and translate them into structured actions. The Dispatcher can be integrated into various front ends, such as the TypeAgent Shell and TypeAgent CLI, and supports an extensible architecture for application agents. ## What it does -The Dispatcher processes user requests and translates them into actions based on schemas provided by application agents. It can automatically switch between different agents to provide a cohesive experience. The Dispatcher supports natural language requests and system commands, enabling users to interact with the system in a flexible manner. +The Dispatcher serves as the central hub for processing user requests, translating natural language inputs into structured actions based on schemas provided by application agents. It enables dynamic switching between agents to handle diverse tasks, making it a flexible and scalable solution for building personal agents. ### Natural Language Requests -Users can request actions provided by application agents using natural language. For example, in the CLI: +The Dispatcher allows users to interact with application agents using natural language. For example, in the CLI: ```bash [calendar]🤖> can you setup a meeting between 2-3PM @@ -29,50 +29,97 @@ Generating translation using GPT for 'can you setup a meeting between 2-3PM' Accept? (y/n) ``` -Other examples include: +Other examples of natural language requests include: - `play some music by Bach for me please` - `create a grocery list` - `add milk to the grocery list` -### Commands +### System Commands -Users can specify system commands with inputs starting with `@`. Examples include toggling dispatcher agents, configuring explainers, and managing conversations. +In addition to natural language, the Dispatcher supports system commands prefixed with `@`. These commands allow users to interact directly with the system. Examples include: + +- **Toggling Dispatcher Agents**: Enable or disable specific agents or groups of agents using commands like `@config agent ` or `@config agent --off `. +- **Explainer Configuration**: Change the explainer implementation with commands like `@config explainer name `. +- **Shortcut Commands**: Execute specific parts of the Dispatcher system, such as `@translate ` for translation or `@reasoning [--engine ] ` for reasoning. + +### Conversation Management + +The Dispatcher also supports managing conversations through natural language or system commands. Examples include: + +- "create a new conversation called research" +- "switch to my work conversation" +- "rename this conversation to project notes" +- "delete the old project conversation" + +These requests are translated into structured payloads and forwarded to the client for execution. ## Setup -The Dispatcher requires several environment variables to be set for proper operation: +To use the Dispatcher, you need to configure the following environment variables: -- `CLAUDE_CUSTOM_PROMPT_FILE` -- `CLAUDE_FORCE_REASONING` -- `COPILOT_REASONING_EFFORT` -- `COPILOT_REASONING_MODEL` -- `COSMOSDB_CONNECTION_STRING` -- `INSTANCE_NAME` -- `TYPEAGENT_REASONING_TIMEOUT_MS` -- `TYPEAGENT_REQUEST_ACTION_LOG_DIR` -- `TYPEAGENT_USER_DATA_DIR` +- `CLAUDE_CUSTOM_PROMPT_FILE`: Path to a custom prompt file for the Claude reasoning engine. +- `CLAUDE_FORCE_REASONING`: Boolean flag to force the use of Claude for reasoning. +- `COPILOT_REASONING_EFFORT`: Specifies the effort level for Copilot reasoning. +- `COPILOT_REASONING_MODEL`: Defines the model to use for Copilot reasoning. +- `COSMOSDB_CONNECTION_STRING`: Connection string for the CosmosDB instance. +- `INSTANCE_NAME`: Name of the instance for identification purposes. +- `TYPEAGENT_REASONING_TIMEOUT_MS`: Timeout (in milliseconds) for reasoning operations. +- `TYPEAGENT_REQUEST_ACTION_LOG_DIR`: Directory path for storing action request logs. +- `TYPEAGENT_USER_DATA_DIR`: Directory path for storing user data. Refer to the hand-written README for detailed instructions on obtaining and setting these values. ## Key Files -The Dispatcher is organized into several key components: +The Dispatcher is organized into several key components, each with specific responsibilities: + +### Handlers + +Handlers are responsible for processing specific commands. They are located in [./src/context/dispatcher/handlers/](./src/context/dispatcher/handlers/). Key files include: + +- [explainCommandHandler.ts](./src/context/dispatcher/handlers/explainCommandHandler.ts): Handles requests for explanations of actions. +- [matchCommandHandler.ts](./src/context/dispatcher/handlers/matchCommandHandler.ts): Processes matching-related commands. +- [reasonCommandHandler.ts](./src/context/dispatcher/handlers/reasonCommandHandler.ts): Handles reasoning commands. +- [requestCommandHandler.ts](./src/context/dispatcher/handlers/requestCommandHandler.ts): Manages general request commands. +- [translateCommandHandler.ts](./src/context/dispatcher/handlers/translateCommandHandler.ts): Handles translation of natural language into structured actions. + +### Schemas + +Schemas define the structure of actions and are located in [./src/context/dispatcher/schema/](./src/context/dispatcher/schema/). Key files include: -- **Handlers**: Located in [./src/context/dispatcher/handlers/](./src/context/dispatcher/handlers/), these files handle specific commands such as [explainCommandHandler.ts](./src/context/dispatcher/handlers/explainCommandHandler.ts), [matchCommandHandler.ts](./src/context/dispatcher/handlers/matchCommandHandler.ts), [reasonCommandHandler.ts](./src/context/dispatcher/handlers/reasonCommandHandler.ts), [requestCommandHandler.ts](./src/context/dispatcher/handlers/requestCommandHandler.ts), and [translateCommandHandler.ts](./src/context/dispatcher/handlers/translateCommandHandler.ts). -- **Schemas**: Located in [./src/context/dispatcher/schema/](./src/context/dispatcher/schema/), these files define the structure of actions, including [activityActionSchema.ts](./src/context/dispatcher/schema/activityActionSchema.ts), [clarifyActionSchema.ts](./src/context/dispatcher/schema/clarifyActionSchema.ts), [dispatcherActionSchema.ts](./src/context/dispatcher/schema/dispatcherActionSchema.ts), [lookupActionSchema.ts](./src/context/dispatcher/schema/lookupActionSchema.ts), and [reasoningActionSchema.ts](./src/context/dispatcher/schema/reasoningActionSchema.ts). -- **Helpers**: Various utility functions and classes are provided in [./src/helpers/](./src/helpers/), such as [console.ts](./src/helpers/console.ts), [userData.ts](./src/helpers/userData.ts), [userSettings.ts](./src/helpers/userSettings.ts), [config.ts](./src/helpers/config.ts), [status.ts](./src/helpers/status.ts), [command.ts](./src/helpers/command.ts), and [completion/index.ts](./src/helpers/completion/index.ts). +- [activityActionSchema.ts](./src/context/dispatcher/schema/activityActionSchema.ts): Defines schemas for activity-related actions. +- [clarifyActionSchema.ts](./src/context/dispatcher/schema/clarifyActionSchema.ts): Handles schemas for clarification actions. +- [dispatcherActionSchema.ts](./src/context/dispatcher/schema/dispatcherActionSchema.ts): Core schema for dispatcher actions. +- [lookupActionSchema.ts](./src/context/dispatcher/schema/lookupActionSchema.ts): Defines schemas for lookup actions. +- [reasoningActionSchema.ts](./src/context/dispatcher/schema/reasoningActionSchema.ts): Handles schemas for reasoning actions. + +### Helpers + +Utility functions and classes are provided in [./src/helpers/](./src/helpers/). Notable files include: + +- [console.ts](./src/helpers/console.ts): Console-related utilities. +- [userData.ts](./src/helpers/userData.ts): Manages user data. +- [userSettings.ts](./src/helpers/userSettings.ts): Handles user settings. +- [config.ts](./src/helpers/config.ts): Configuration utilities. +- [status.ts](./src/helpers/status.ts): Status management utilities. +- [command.ts](./src/helpers/command.ts): Command-related utilities. +- [completion/index.ts](./src/helpers/completion/index.ts): Handles command completion logic. ## How to extend -To extend the Dispatcher, follow these steps: +To extend the functionality of the Dispatcher, follow these steps: + +1. **Create a New Handler**: Add a new file in [./src/context/dispatcher/handlers/](./src/context/dispatcher/handlers/) and implement the logic for the new command. Use existing handlers as a reference for structure and patterns. +2. **Define a New Schema**: Add a new schema file in [./src/context/dispatcher/schema/](./src/context/dispatcher/schema/) to define the structure of the new action. Ensure the schema aligns with the expected input and output for the action. + +3. **Update the Dispatcher**: Modify the Dispatcher to recognize and process the new command and schema. This may involve updating the command registry or other relevant components. + +4. **Test Your Changes**: Write unit tests for the new functionality and run the existing test suite to ensure no regressions are introduced. Use the CLI or Shell to manually verify the new functionality. -1. **Add a new handler**: Create a new file in [./src/context/dispatcher/handlers/](./src/context/dispatcher/handlers/) and implement the necessary logic for the new command. -2. **Define a new schema**: Create a new file in [./src/context/dispatcher/schema/](./src/context/dispatcher/schema/) to define the structure of the new action. -3. **Update the Dispatcher**: Modify the Dispatcher to recognize and process the new command and schema. -4. **Test the changes**: Ensure that the new functionality works as expected by running tests and verifying the integration with existing components. +5. **Documentation**: Update the hand-written README or other relevant documentation to describe the new feature and how to use it. -For more detailed information on the Dispatcher architecture and design, refer to the dispatcher architecture documentation. +For more details on the architecture and design of the Dispatcher, consult the dispatcher architecture documentation. ## Reference @@ -80,16 +127,16 @@ For more detailed information on the Dispatcher architecture and design, refer t ### Entry points -- default → [./dist/index.js](./dist/index.js) -- `./helpers/console` → [./dist/helpers/console.js](./dist/helpers/console.js) -- `./helpers/data` → [./dist/helpers/userData.js](./dist/helpers/userData.js) -- `./helpers/userSettings` → [./dist/helpers/userSettings.js](./dist/helpers/userSettings.js) -- `./helpers/config` → [./dist/helpers/config.js](./dist/helpers/config.js) -- `./helpers/status` → [./dist/helpers/status.js](./dist/helpers/status.js) -- `./helpers/command` → [./dist/helpers/command.js](./dist/helpers/command.js) -- `./helpers/completion` → [./dist/helpers/completion/index.js](./dist/helpers/completion/index.js) -- `./internal` → [./dist/internal.js](./dist/internal.js) -- `./explorer` → [./dist/explorer.js](./dist/explorer.js) +- default → `./dist/index.js` _(not found on disk)_ +- `./helpers/console` → `./dist/helpers/console.js` _(not found on disk)_ +- `./helpers/data` → `./dist/helpers/userData.js` _(not found on disk)_ +- `./helpers/userSettings` → `./dist/helpers/userSettings.js` _(not found on disk)_ +- `./helpers/config` → `./dist/helpers/config.js` _(not found on disk)_ +- `./helpers/status` → `./dist/helpers/status.js` _(not found on disk)_ +- `./helpers/command` → `./dist/helpers/command.js` _(not found on disk)_ +- `./helpers/completion` → `./dist/helpers/completion/index.js` _(not found on disk)_ +- `./internal` → `./dist/internal.js` _(not found on disk)_ +- `./explorer` → `./dist/explorer.js` _(not found on disk)_ ### Dependencies @@ -144,7 +191,7 @@ External: `@anthropic-ai/claude-agent-sdk`, `@azure/core-client`, `@azure/core-r - [./src/context/dispatcher/schema/dispatcherActionSchema.ts](./src/context/dispatcher/schema/dispatcherActionSchema.ts) - [./src/context/dispatcher/schema/lookupActionSchema.ts](./src/context/dispatcher/schema/lookupActionSchema.ts) - [./src/context/dispatcher/schema/reasoningActionSchema.ts](./src/context/dispatcher/schema/reasoningActionSchema.ts) -- _…and 184 more under `./src/`._ +- _…and 185 more under `./src/`._ ### Environment variables @@ -162,6 +209,6 @@ _9 environment variables referenced from `./src/` (set in `ts/.env` or your shel --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter agent-dispatcher docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter agent-dispatcher docs:verify-links` to spot-check._ diff --git a/ts/packages/knowPro/README.AUTOGEN.md b/ts/packages/knowPro/README.AUTOGEN.md index aa82192f5..13c514ae3 100644 --- a/ts/packages/knowPro/README.AUTOGEN.md +++ b/ts/packages/knowPro/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # knowpro — AI-generated documentation @@ -12,45 +12,100 @@ ## Overview -The `knowpro` package is a TypeScript library designed to implement a basic structured Retrieval-Augmented Generation (RAG) system. It is part of the TypeAgent monorepo and is actively developed to explore various types of memory and structured information extraction from conversations and other forms of text. +The `knowpro` package is a TypeScript library that implements a basic structured Retrieval-Augmented Generation (RAG) system. It is part of the TypeAgent monorepo and serves as an experimental foundation for exploring structured information extraction and retrieval from various types of data, such as conversations, documents, emails, and images. The library is actively developed and integrates with other TypeAgent components, particularly those related to memory and knowledge processing. ## What it does -The `knowpro` package provides functionalities for structured prompting and leveraging large language models (LLMs) to handle natural language queries and generate answers. It supports the following capabilities: +The `knowpro` package provides tools and APIs to enable structured RAG workflows. Its primary capabilities include: -- **Natural language queries**: Translating user requests into structured query expressions. -- **Answer generation**: Using query results to generate natural language answers. -- **Search**: Converting user requests into query expressions and executing them to retrieve relevant entities, topics, and actions. -- **Indexing**: Storing and updating information in suitable indexes for efficient retrieval. +- **Natural Language Querying**: Converts user queries into structured query expressions using natural language processing and large language models (LLMs). This allows users to query unstructured and structured data using natural language. +- **Search and Retrieval**: Executes structured queries to retrieve relevant entities, topics, and actions from indexed data. The search process supports scope expressions (e.g., time ranges, topics) and tree-pattern expressions for precise matching. +- **Answer Generation**: Generates natural language answers by combining query results with relevant context, such as entities, topics, and messages. +- **Indexing**: Manages structured and unstructured data in indexes, enabling efficient search and retrieval. These indexes can be updated incrementally or in the background. -The package interacts with other parts of the system, such as the `conversation-memory` package for agent memory and various memory types including transcripts, documents, emails, and images. +The package is designed to work with conversations and other forms of text, extracting structured information such as entities, topics, relationships, and metadata. This information is stored in indexes that support efficient querying and retrieval. The results of these queries can then be used to generate answers to user questions. + +The `knowpro` package is primarily tested with GPT-4o, and its performance with other models may vary. ## Setup -To set up the `knowpro` package, ensure you have the necessary dependencies installed. The package relies on several internal and external libraries, including `aiclient`, `knowledge-processor`, `typeagent`, and others. +To use the `knowpro` package, follow these steps: + +1. **Install Dependencies**: Ensure that all required dependencies are installed. The package depends on other internal libraries in the TypeAgent monorepo, such as `@typeagent/aiclient`, `knowledge-processor`, and `typeagent`. External dependencies include `async`, `debug`, `fast-levenshtein`, and `typechat`. + +2. **Environment Configuration**: If the hand-written README specifies any environment variables or API keys, configure them accordingly. For example, you may need to set up access to an LLM like GPT-4o. + +3. **Integration**: The `knowpro` package is used by several other packages and examples in the monorepo, such as `agent-dispatcher`, `browser-typeagent`, and `knowpro-test`. Refer to their respective documentation for integration details. -For detailed setup instructions, including environment variables and API keys, refer to the hand-written README. +For additional setup details, consult the hand-written README. ## Key Files -The `knowpro` package is organized into several key components: +The `knowpro` package is organized into several key files, each responsible for specific functionality: + +- **Schemas**: + + - [answerContextSchema.ts](./src/answerContextSchema.ts): Defines the structure of the context used for generating answers, including entities, topics, and messages. + - [answerResponseSchema.ts](./src/answerResponseSchema.ts): Specifies the structure of the answer responses, including the type of answer and its content. + +- **Search**: + + - [search.ts](./src/search.ts): Handles the conversion of user queries into structured query expressions and executes them. + - [searchLang.ts](./src/searchLang.ts): Provides natural language processing capabilities for query generation. + +- **Answer Generation**: + + - [answerGenerator.ts](./src/answerGenerator.ts): Implements the logic for generating natural language answers based on query results and context. + +- **Indexes**: + + - [conversationIndex.ts](./src/conversationIndex.ts): Manages the primary index for storing and retrieving conversation data. + - [secondaryIndexes.ts](./src/secondaryIndexes.ts): Handles secondary indexes for efficient query execution, such as tree-pattern and scope expressions. + +- **Utilities**: -- **Schemas**: Define the structure of the data used in the package, such as [answerContextSchema.ts](./src/answerContextSchema.ts) and [answerResponseSchema.ts](./src/answerResponseSchema.ts). -- **Search**: Handles the conversion of natural language queries into structured query expressions and executes them. Key files include [search.ts](./src/search.ts) and [searchLang.ts](./src/searchLang.ts). -- **Answer Generation**: Generates answers based on the results of executed queries. This is managed by [answerGenerator.ts](./src/answerGenerator.ts). -- **Indexes**: Store and manage the information extracted from conversations and other text sources. Relevant files include [conversationIndex.ts](./src/conversationIndex.ts) and [secondaryIndexes.ts](./src/secondaryIndexes.ts). -- **Common Utilities**: Provide shared functionality across the package, such as [common.ts](./src/common.ts). + - [common.ts](./src/common.ts): Contains shared utility functions and types used across the package. + - [collections.ts](./src/collections.ts): Provides data structures and algorithms for managing query results and matches. + +- **Core Logic**: + - [compileLib.ts](./src/compileLib.ts): Implements query operators and processing logic for the library. + - [conversation.ts](./src/conversation.ts): Defines settings and utilities for managing conversations and their associated data. ## How to extend -To extend the `knowpro` package, follow these steps: +To extend the `knowpro` package, follow these guidelines: + +1. **Understand the Architecture**: + + - Familiarize yourself with the key files and their responsibilities as outlined above. + - Review the hand-written README and the [TypeAgent memory architecture](../../../docs/content/architecture/memory.md) document for a deeper understanding of Structured RAG and its implementation in `knowpro`. + +2. **Identify the Extension Point**: + + - Determine the area you want to extend, such as adding new schemas, enhancing search capabilities, or improving answer generation. + +3. **Modify or Add Files**: + + - For new schemas, start with [answerContextSchema.ts](./src/answerContextSchema.ts) or [answerResponseSchema.ts](./src/answerResponseSchema.ts). + - To enhance search functionality, consider modifying [search.ts](./src/search.ts) or [searchLang.ts](./src/searchLang.ts). + - To add new indexing capabilities, explore [secondaryIndexes.ts](./src/secondaryIndexes.ts). + +4. **Update Interfaces**: + + - Ensure that any new functionality is reflected in the base interfaces and types defined in [interfaces.ts](./src/interfaces.ts). + +5. **Test Your Changes**: + + - Use the `knowpro-test` package to validate your changes. This package provides a test suite for the `knowpro` API and its memory implementations. + +6. **Follow Existing Patterns**: + + - Adhere to the coding patterns and conventions used in the existing codebase. For example, use the `TypeChatJsonTranslator` for JSON translation tasks and the `typechat` library for schema validation. -1. **Identify the component to extend**: Determine whether you need to add new schemas, enhance search capabilities, or improve answer generation. -2. **Modify or add files**: Based on the identified component, modify existing files or add new ones. For example, to add a new type of index, you might start with [secondaryIndexes.ts](./src/secondaryIndexes.ts). -3. **Update interfaces**: Ensure that any new functionality is reflected in the base interfaces and types defined in [interfaces.ts](./src/interfaces.ts). -4. **Test your changes**: Write tests to validate your changes. The `knowpro-test` package can be used to test the `knowpro` API and memory implementations. +7. **Document Your Changes**: + - Update the hand-written README or other relevant documentation to reflect your changes. Include examples and usage instructions where applicable. -For detailed examples and further guidance, refer to the [KnowPro test app](../../examples/chat/README.md) and other related documentation. +By following these steps, you can effectively extend the `knowpro` package to meet your specific requirements. For further assistance, refer to the examples and related documentation in the TypeAgent monorepo. ## Reference @@ -58,7 +113,7 @@ For detailed examples and further guidance, refer to the [KnowPro test app](../. ### Entry points -- default → [./dist/index.js](./dist/index.js) +- default → `./dist/index.js` _(not found on disk)_ ### Dependencies @@ -91,6 +146,6 @@ External: `async`, `debug`, `fast-levenshtein`, `typechat` --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter knowpro docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter knowpro docs:verify-links` to spot-check._ diff --git a/ts/packages/knowledgeProcessor/README.AUTOGEN.md b/ts/packages/knowledgeProcessor/README.AUTOGEN.md index 0f327ea0a..dee4ceda4 100644 --- a/ts/packages/knowledgeProcessor/README.AUTOGEN.md +++ b/ts/packages/knowledgeProcessor/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # knowledge-processor — AI-generated documentation @@ -12,45 +12,90 @@ ## Overview -The `knowledge-processor` package is a TypeScript library designed to explore early ideas in Structured Retrieval-Augmented Generation (RAG). It is primarily used by the TypeAgent Dispatcher to implement Agent Memory for the TypeAgent Shell. The package focuses on extracting knowledge from various sources, indexing it for retrieval, generating queries, and producing answers based on the retrieved knowledge. +The `knowledge-processor` package is a TypeScript library that explores early concepts in Structured Retrieval-Augmented Generation (RAG). It is primarily used by the TypeAgent Dispatcher to implement Agent Memory for the TypeAgent Shell. The package focuses on extracting, indexing, and utilizing knowledge from various sources such as conversations, transcripts, images, and documents. + +This package is part of the TypeAgent monorepo and is considered sample code for experimenting with foundational ideas in knowledge processing. For the latest developments in Structured RAG, refer to the `knowPro` package. ## What it does -The `knowledge-processor` package provides functionalities to: +The `knowledge-processor` package provides a set of tools and actions to work with knowledge extraction, indexing, and retrieval. Its main capabilities include: -- **Extract knowledge**: from conversations, transcripts, images, and documents. -- **Index knowledge**: store and index the extracted knowledge and source text for efficient retrieval. -- **Generate queries**: translate natural language questions into structured queries to retrieve relevant knowledge. -- **Generate answers**: use the results of queries to generate natural language answers. +- **Knowledge Extraction**: Extracts structured knowledge from diverse sources, including conversations, documents, images, and transcripts. This is achieved through actions like `extractKnowledge`. +- **Knowledge Indexing**: Organizes and stores extracted knowledge for efficient retrieval. This includes text indexing, key-value indexing, and temporal indexing. +- **Query Generation**: Translates natural language questions into structured queries to retrieve relevant knowledge. Actions like `searchKnowledge` facilitate this process. +- **Answer Generation**: Uses retrieved knowledge to generate natural language answers to user queries. Actions such as `generateAnswer` are central to this functionality. +- **Conversation Management**: Manages conversation topics, tracks context, and generates responses. Actions like `createMessage` and `aggregateTopics` are part of this functionality. -The package includes several actions related to conversation management, knowledge extraction, and query processing. Key actions include `createMessage`, `extractKnowledge`, `generateAnswer`, and `searchKnowledge`. +These features make the package a critical component for enabling intelligent, context-aware interactions in the TypeAgent ecosystem. ## Setup -To set up the `knowledge-processor` package, ensure you have the necessary dependencies installed. The package relies on several external libraries such as `@azure-rest/maps-search`, `debug`, `exifreader`, `sharp`, and `typechat`. Additionally, it depends on other workspace packages like `aiclient`, `typeagent`, and `typechat-utils`. +To use the `knowledge-processor` package, ensure the following prerequisites are met: + +1. **Install Dependencies**: The package depends on several workspace and external libraries, including: + + - Workspace dependencies: `@typeagent/aiclient`, `@typeagent/config`, `typeagent`, and `typechat-utils`. + - External dependencies: `@azure-rest/maps-search`, `debug`, `exifreader`, `sharp`, and `typechat`. + + Use `pnpm install` to install all required dependencies. + +2. **Environment Variables**: If the package requires any specific environment variables or API keys, refer to the hand-written README for detailed instructions on how to configure them. -For detailed setup instructions, including environment variables and API keys, refer to the hand-written README. +3. **External Services**: Some features, such as knowledge extraction from images or external APIs, may require additional setup (e.g., API keys for `@azure-rest/maps-search`). Check the hand-written README for guidance on obtaining and configuring these keys. ## Key Files -The `knowledge-processor` package is organized into several modules, each responsible for different aspects of knowledge processing: +The `knowledge-processor` package is organized into several key modules, each responsible for a specific aspect of knowledge processing: + +### Conversation Management + +- [conversationManager.ts](./src/conversation/conversationManager.ts): Implements the `ConversationManager`, which is used by the TypeAgent Dispatcher to manage conversation topics and integrate with Agent Memory. +- [conversation.ts](./src/conversation/conversation.ts): Contains core logic for handling conversations, including topic management and knowledge extraction. +- [answerGenerator.ts](./src/conversation/answerGenerator.ts): Generates natural language answers based on retrieved knowledge and conversation context. + +### Knowledge Extraction + +- [knowledge.ts](./src/conversation/knowledge.ts): Core logic for extracting knowledge from various sources. +- [knowledgeSchema.ts](./src/conversation/knowledgeSchema.ts): Defines the schema for representing extracted knowledge. +- [knowledgeActions.ts](./src/conversation/knowledgeActions.ts): Implements actions related to knowledge extraction and processing. + +### Indexing -- **Conversation**: Handles conversation-related functionalities, including managing conversation topics, extracting knowledge from conversations, and generating answers. Key files include [conversationManager.ts](./src/conversation/conversationManager.ts), [conversation.ts](./src/conversation/conversation.ts), and [answerGenerator.ts](./src/conversation/answerGenerator.ts). -- **Knowledge Extraction**: Focuses on extracting knowledge from various sources. This includes schemas for knowledge representation and actions for knowledge extraction. Key files include [knowledgeSchema.ts](./src/conversation/knowledgeSchema.ts), [knowledgeActions.ts](./src/conversation/knowledgeActions.ts), and [knowledge.ts](./src/conversation/knowledge.ts). -- **Indexing**: Manages the indexing of extracted knowledge for efficient retrieval. This includes text indexing, key-value indexing, and temporal indexing. Key files include [textIndex.ts](./src/textIndex.ts), [keyValueIndex.ts](./src/keyValueIndex.ts), and [temporal.ts](./src/temporal.ts). -- **Storage**: Provides storage solutions for the indexed knowledge. Key files include [storageProvider.ts](./src/storageProvider.ts) and [modelCache.ts](./src/modelCache.ts). +- [textIndex.ts](./src/textIndex.ts): Handles text-based indexing for efficient knowledge retrieval. +- [keyValueIndex.ts](./src/keyValueIndex.ts): Manages key-value indexing for structured data. +- [temporal.ts](./src/temporal.ts): Provides functionality for temporal indexing and querying. + +### Storage + +- [storageProvider.ts](./src/storageProvider.ts): Defines storage interfaces and implementations for persisting indexed knowledge. +- [modelCache.ts](./src/modelCache.ts): Implements caching mechanisms for models and data. + +### Schemas + +- [aggregateTopicSchema.ts](./src/conversation/aggregateTopicSchema.ts): Defines schemas for aggregating and organizing conversation topics. +- [answerSchema.ts](./src/conversation/answerSchema.ts): Specifies the structure of generated answers, including relevance and content. +- [dateTimeSchema.ts](./src/conversation/dateTimeSchema.ts): Provides schemas for handling date and time information. ## How to extend To extend the `knowledge-processor` package, follow these steps: -1. **Identify the module to extend**: Determine whether you need to add functionalities related to conversation management, knowledge extraction, indexing, or storage. -2. **Open the relevant file**: Start by opening the file that corresponds to the module you want to extend. For example, if you want to add a new knowledge extraction method, open [knowledge.ts](./src/conversation/knowledge.ts). -3. **Follow existing patterns**: Review the existing code to understand the patterns and structures used. Implement your new functionality following these patterns. -4. **Add tests**: Ensure your new functionality is well-tested. Add unit tests in the corresponding test files to validate your changes. -5. **Run tests**: Execute the tests to verify that your changes work as expected and do not break existing functionalities. +1. **Understand the Existing Structure**: Familiarize yourself with the key files and their responsibilities. For example, if you want to add a new knowledge extraction feature, start by reviewing [knowledge.ts](./src/conversation/knowledge.ts) and [knowledgeActions.ts](./src/conversation/knowledgeActions.ts). + +2. **Add New Functionality**: + + - For new actions, define the action schema in the appropriate schema file (e.g., [knowledgeSchema.ts](./src/conversation/knowledgeSchema.ts)). + - Implement the action logic in the corresponding module. For example, if it's a knowledge-related action, add it to [knowledgeActions.ts](./src/conversation/knowledgeActions.ts). + +3. **Follow Existing Patterns**: Review similar implementations in the package to ensure consistency in coding style and structure. + +4. **Update Tests**: Add unit tests for your new functionality. Place these tests in the appropriate test files or create new ones if necessary. + +5. **Run Tests**: Execute the test suite to ensure your changes work as intended and do not introduce regressions. + +6. **Document Changes**: Update the hand-written README or other documentation to reflect your changes, if applicable. -By following these steps, you can effectively extend the `knowledge-processor` package to meet your specific requirements. +By following these steps, you can effectively contribute to and extend the `knowledge-processor` package. ## Reference @@ -58,7 +103,7 @@ By following these steps, you can effectively extend the `knowledge-processor` p ### Entry points -- default → [./dist/index.js](./dist/index.js) +- default → `./dist/index.js` _(not found on disk)_ ### Dependencies @@ -91,6 +136,6 @@ External: `@azure-rest/maps-search`, `debug`, `exifreader`, `sharp`, `typechat` --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter knowledge-processor docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter knowledge-processor docs:verify-links` to spot-check._ diff --git a/ts/packages/memory/conversation/README.AUTOGEN.md b/ts/packages/memory/conversation/README.AUTOGEN.md index f3108914f..d00ac14a7 100644 --- a/ts/packages/memory/conversation/README.AUTOGEN.md +++ b/ts/packages/memory/conversation/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # conversation-memory — AI-generated documentation @@ -12,55 +12,68 @@ ## Overview -The `conversation-memory` package is a TypeScript library designed to implement various types of conversational memory using Structured Retrieval-Augmented Generation (RAG). It leverages the `KnowPro` package to index and search through sequences of timestamped messages, documents, emails, and other forms of conversational data. +The `conversation-memory` package is a TypeScript library that implements various types of conversational memory using Structured Retrieval-Augmented Generation (RAG). It leverages the `KnowPro` package for indexing and searching through structured data, enabling efficient storage, retrieval, and analysis of conversational and document-based information. + +This package is designed to handle a wide range of conversational data, including chat messages, emails, documents, and transcripts. It supports incremental updates, natural language queries, and the generation of human-readable answers based on indexed knowledge. ## What it does -The `conversation-memory` package provides functionality to manage and query different types of conversational memories, including: +The `conversation-memory` package provides tools to manage and query different types of conversational memories. These include: + +- **Conversation Memory**: Tracks interactive chats, agent interaction history, and invocation/response memory. New messages can be added and indexed incrementally, with extracted knowledge such as entities, actions, and topics enabling precise search and retrieval. +- **Document Memory**: Manages collections of document parts, such as meeting transcripts, video captions, markdown files, and HTML documents. Documents can be imported, indexed, and queried for summaries, lists, or specific information. +- **Email Memory**: Handles collections of email messages. Emails can be imported from `.eml` files, indexed, and queried for specific information, such as conversations between specific participants or topics discussed. +- **Podcast Memory**: Manages podcast transcripts, enabling indexing and querying for specific segments or topics. + +The package supports two primary query methods: -- **Conversation Memory**: Interactive chats, agent interaction history, and invocation/response memory. -- **Document Memory**: Transcripts of meetings, videos, chats, markdown, and HTML documents. -- **Email Memory**: Collections of email messages. -- **Podcast Memory**: Transcripts of podcasts. +1. **Natural Language Queries**: Users can ask questions in plain language, and the package translates these into structured search expressions. +2. **Structured Search Expressions**: Advanced users can directly query the memory using `KnowPro` search expressions for precise results. -These memories are indexed incrementally, allowing new messages, emails, transcript chunks, or document parts to be added and indexed as they come in. The package extracts salient knowledge such as entities, actions, and topics from new messages, enabling precise search and retrieval with low latency. Users can query these memories using natural language or structured search expressions, and the package can generate answers, summaries, and analyses based on the indexed data. +The results of these queries can be used to generate answers, summaries, or analyses. Memories are mutable and can be persisted to disk or loaded on demand. ## Setup -To set up the `conversation-memory` package, follow these steps: +To use the `conversation-memory` package, follow these steps: -1. Install the necessary dependencies using `pnpm install`. -2. Set up environment variables as required by the `KnowPro` and `memory-storage` packages. -3. Ensure you have access to external services like OpenAI for embedding models. +1. **Install dependencies**: Run `pnpm install` to install the required packages. +2. **Environment variables**: Ensure that any necessary environment variables for the `KnowPro` and `memory-storage` packages are set. Refer to their respective documentation for details. +3. **External services**: If using embedding models, ensure access to the required external services, such as OpenAI. -For detailed setup instructions, see the hand-written README. +For additional setup details, refer to the hand-written README. ## Key Files -The `conversation-memory` package is organized into several key files and modules: +The `conversation-memory` package is organized into several key files, each responsible for specific functionality: -- **[index.ts](./src/index.ts)**: Exports the main functionalities of the package, including podcast, memory, conversation memory, email memory, and document memory modules. -- **[conversationMemory.ts](./src/conversationMemory.ts)**: Implements the `ConversationMemory` class, handling interactive chats and agent interaction history. -- **[docMemory.ts](./src/docMemory.ts)**: Implements the `DocMemory` class, managing collections of document parts. -- **[emailMemory.ts](./src/emailMemory.ts)**: Implements the `EmailMemory` class, managing collections of email messages. -- **[podcast.ts](./src/podcast.ts)**: Implements the `PodcastMemory` class, handling podcast transcripts. -- **[common.ts](./src/common.ts)**: Contains common utility functions used across different memory types. -- **[docImport.ts](./src/docImport.ts)**: Handles the import of text documents into `DocMemory`. -- **[emailImport.ts](./src/emailImport.ts)**: Handles the import of email messages into `EmailMemory`. +- **[index.ts](./src/index.ts)**: The main entry point, exporting the core functionalities of the package. +- **[conversationMemory.ts](./src/conversationMemory.ts)**: Implements the `ConversationMemory` class for managing interactive chats and agent interaction history. +- **[docMemory.ts](./src/docMemory.ts)**: Implements the `DocMemory` class for managing collections of document parts, such as transcripts and markdown files. +- **[emailMemory.ts](./src/emailMemory.ts)**: Implements the `EmailMemory` class for managing collections of email messages. +- **[podcast.ts](./src/podcast.ts)**: Implements the `PodcastMemory` class for managing podcast transcripts. +- **[common.ts](./src/common.ts)**: Provides utility functions for embedding models, indexing state, and error handling. +- **[docImport.ts](./src/docImport.ts)**: Handles the import of text documents into `DocMemory`, supporting formats like `.vtt`, `.md`, `.html`, and `.txt`. +- **[emailImport.ts](./src/emailImport.ts)**: Handles the import of email messages in MIME format into `EmailMemory`. +- **[docSearchQuerySchema.ts](./src/docSearchQuerySchema.ts)**: Defines the schema for document search queries, including entity and facet terms. ## How to extend To extend the `conversation-memory` package, follow these steps: -1. **Identify the memory type** you want to extend (e.g., conversation, document, email, podcast). -2. **Open the corresponding file** (e.g., [conversationMemory.ts](./src/conversationMemory.ts), [docMemory.ts](./src/docMemory.ts), [emailMemory.ts](./src/emailMemory.ts), [podcast.ts](./src/podcast.ts)). -3. **Add new functionalities** or modify existing ones. Ensure that new messages or data types are properly indexed and searchable. -4. **Update the schema** if necessary, such as modifying [docSearchQuerySchema.ts](./src/docSearchQuerySchema.ts) for document-related queries. -5. **Write tests** to validate your changes. Use the `test-lib` package for testing utilities. - -For example, to add a new type of conversational memory, you might start by creating a new class similar to `ConversationMemory` and implement methods for adding messages, indexing, and querying. Ensure that the new class integrates with the existing indexing and search mechanisms provided by `KnowPro`. - -By following these steps, you can extend the `conversation-memory` package to support additional types of conversational data or enhance its existing capabilities. +1. **Determine the memory type**: Identify whether you want to extend conversation, document, email, or podcast memory. +2. **Locate the relevant file**: Open the corresponding file for the memory type you want to extend: + - Conversation: [conversationMemory.ts](./src/conversationMemory.ts) + - Document: [docMemory.ts](./src/docMemory.ts) + - Email: [emailMemory.ts](./src/emailMemory.ts) + - Podcast: [podcast.ts](./src/podcast.ts) +3. **Add or modify functionality**: Implement new methods or enhance existing ones. For example: + - To add a new type of memory, create a new class similar to `ConversationMemory` or `DocMemory`. + - Implement methods for adding data, indexing, and querying. + - Ensure that new data types are properly integrated with the `KnowPro` indexing and search mechanisms. +4. **Update schemas**: If your changes involve new query types or data structures, update the relevant schema files, such as [docSearchQuerySchema.ts](./src/docSearchQuerySchema.ts). +5. **Test your changes**: Write unit tests to validate your modifications. Use the `test-lib` package for testing utilities. + +By following these steps, you can extend the `conversation-memory` package to support additional use cases or enhance its existing features. For example, you could add support for a new document format in [docImport.ts](./src/docImport.ts) or implement a new type of memory for a specific use case. ## Reference @@ -68,7 +81,7 @@ By following these steps, you can extend the `conversation-memory` package to su ### Entry points -- default → [./dist/index.js](./dist/index.js) +- default → `./dist/index.js` _(not found on disk)_ ### Dependencies @@ -101,6 +114,6 @@ External: `async`, `debug`, `mailparser`, `typechat`, `webvtt-parser` --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter conversation-memory docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter conversation-memory docs:verify-links` to spot-check._ diff --git a/ts/packages/memory/image/README.AUTOGEN.md b/ts/packages/memory/image/README.AUTOGEN.md index 178ef210d..99e33186f 100644 --- a/ts/packages/memory/image/README.AUTOGEN.md +++ b/ts/packages/memory/image/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # image-memory — AI-generated documentation @@ -12,64 +12,77 @@ ## Overview -The `image-memory` package is an experimental TypeScript library designed to implement image memory using structured Retrieval-Augmented Generation (RAG). It leverages the `KnowPro` library to create an image collection that can be queried based on both knowledge content and metadata. This package is part of the TypeAgent monorepo and integrates with several other packages to provide its functionality. +The `image-memory` package is an experimental TypeScript library designed to implement image memory using structured Retrieval-Augmented Generation (RAG). It leverages the `KnowPro` library to create an `ImageCollection` that can be queried using natural language. This package is part of the TypeAgent monorepo and integrates with other packages such as `@typeagent/aiclient`, `knowledge-processor`, and `memory-storage` to provide its functionality. -## What it does +The primary goal of this package is to enable the indexing and querying of images based on their metadata and knowledge content. This allows users to interact with image collections in a more intuitive and natural way, such as asking questions about the images and receiving meaningful answers. -The `image-memory` package provides the capability to index and query images using natural language. It supports actions such as `importImages` to index images from a specified path and `indexingService` to start an indexing service for images. The package allows images to be searched based on their content and metadata, making it possible to ask questions and get answers about the images in natural language. +## What it does -Key actions include: +The `image-memory` package provides tools to create, manage, and query an image collection. It supports the following key actions: -- `importImages`: Indexes images from a specified path. -- `indexingService`: Starts an indexing service for images. +- **`importImages`**: This action allows users to index images from a specified path. It processes image files, extracts metadata, and stores the information in a structured format for efficient querying. +- **`indexingService`**: This action starts an indexing service for images, enabling continuous monitoring and indexing of a specified folder. -These actions enable users to build and maintain a searchable image collection, leveraging natural language processing to enhance the querying capabilities. +The package uses the `KnowPro` library to extract knowledge from images and their metadata. This knowledge is then stored in a structured format, allowing users to query the image collection using natural language. For example, users can search for images based on specific metadata (e.g., date taken, location) or ask questions about the content of the images. ## Setup -To set up the `image-memory` package, ensure you have the necessary dependencies installed. The package relies on several internal and external dependencies, including `@azure-rest/maps-search`, `better-sqlite3`, `debug`, `get-folder-size`, and `typechat`. +To use the `image-memory` package, follow these setup steps: + +1. **Install dependencies**: Ensure that all required dependencies are installed. The package relies on both internal and external dependencies, including: + + - Internal: `@typeagent/aiclient`, `knowledge-processor`, `memory-storage`, and others. + - External: `@azure-rest/maps-search`, `better-sqlite3`, `debug`, `get-folder-size`, and `typechat`. -Environment variables: +2. **Set environment variables**: -- `DEBUG`: Set this variable to enable debug logging. + - `DEBUG`: Set this variable to enable debug logging. For example, you can set it to `typeagent:image-memory` to see debug logs specific to this package. -For detailed setup instructions, including how to obtain API keys and configure environment variables, refer to the hand-written README. +3. **Additional setup**: If you need to use external services (e.g., `@azure-rest/maps-search`), ensure you have the necessary API keys and configurations. Refer to the hand-written README for more details on obtaining and setting up these keys. + +Once the setup is complete, you can start using the package to index and query image collections. ## Key Files -The `image-memory` package is structured into several key files, each responsible for different aspects of the functionality: +The `image-memory` package is organized into several key files, each responsible for specific functionality: -- [index.ts](./src/index.ts): The entry point that exports various modules. -- [imageCollection.ts](./src/imageCollection.ts): Defines the `ImageCollection` class, which represents a collection of images and provides methods for indexing and querying. -- [imageMeta.ts](./src/imageMeta.ts): Defines the `Image` and `ImageMeta` classes, which represent individual images and their metadata. -- [importImages.ts](./src/importImages.ts): Provides the `importImages` function to index images from a specified path. -- [indexingService.ts](./src/indexingService.ts): Starts an indexing service for images. -- [tables.ts](./src/tables.ts): Defines database tables for storing image metadata. +- **[index.ts](./src/index.ts)**: The main entry point of the package, exporting all public modules and functions. +- **[imageCollection.ts](./src/imageCollection.ts)**: Defines the `ImageCollection` class, which represents a collection of images. This class provides methods for adding images, querying them, and managing their metadata. +- **[imageMeta.ts](./src/imageMeta.ts)**: Contains the `Image` and `ImageMeta` classes. These classes define the structure of individual images and their associated metadata, including methods for extracting knowledge from images. +- **[importImages.ts](./src/importImages.ts)**: Implements the `importImages` function, which indexes images from a specified path and returns an `ImageCollection`. This function supports recursive indexing of directories and can handle various image file types. +- **[indexingService.ts](./src/indexingService.ts)**: Provides functionality to start and manage an indexing service for images. This service can monitor a folder for changes and update the image collection accordingly. +- **[tables.ts](./src/tables.ts)**: Defines database tables for storing image metadata, such as geographic and exposure information, using SQLite. -### Detailed File Responsibilities +### File Responsibilities -- **[index.ts](./src/index.ts)**: Serves as the main entry point, exporting functions and classes from other modules. -- **[imageCollection.ts](./src/imageCollection.ts)**: Contains the `ImageCollection` class, which manages a collection of images, including methods for adding images and querying them. -- **[imageMeta.ts](./src/imageMeta.ts)**: Defines the structure for image metadata and individual image objects, including methods for extracting knowledge from images. -- **[importImages.ts](./src/importImages.ts)**: Implements the `importImages` function, which indexes images from a specified path and returns an `ImageCollection`. -- **[indexingService.ts](./src/indexingService.ts)**: Provides functionality to start and manage an indexing service for images, including monitoring changes in the indexed folder. -- **[tables.ts](./src/tables.ts)**: Defines the `GeoTable` and `ExposureTable` classes for storing geographic and exposure metadata of images in a SQLite database. +- **[index.ts](./src/index.ts)**: Serves as the central hub for exporting the package's functionality. +- **[imageCollection.ts](./src/imageCollection.ts)**: Manages the core logic for handling image collections, including indexing and querying. +- **[imageMeta.ts](./src/imageMeta.ts)**: Handles the representation and processing of individual images and their metadata. +- **[importImages.ts](./src/importImages.ts)**: Provides the main function for importing and indexing images from a file path or directory. +- **[indexingService.ts](./src/indexingService.ts)**: Implements a service for continuously indexing images in a specified directory. +- **[tables.ts](./src/tables.ts)**: Defines the database schema for storing image metadata, such as geographic coordinates and exposure settings. ## How to extend To extend the `image-memory` package, follow these steps: -1. **Open the relevant file**: Depending on the functionality you want to add or modify, open the appropriate file. For example, to add new indexing capabilities, start with [indexingService.ts](./src/indexingService.ts). +1. **Identify the area to extend**: Determine which part of the package you want to modify or enhance. For example, you might want to add new metadata fields, improve the indexing process, or introduce new querying capabilities. + +2. **Start with the relevant file**: + + - For changes related to image collections, begin with [imageCollection.ts](./src/imageCollection.ts). + - To modify or add metadata handling, work on [imageMeta.ts](./src/imageMeta.ts). + - For changes to the indexing process, focus on [importImages.ts](./src/importImages.ts) or [indexingService.ts](./src/indexingService.ts). -2. **Follow existing patterns**: Review the existing code to understand the patterns used for indexing and querying images. For example, the `ImageCollection` class in [imageCollection.ts](./src/imageCollection.ts) provides a good example of how to structure a collection of images. +3. **Follow existing patterns**: Review the existing code to understand the structure and design patterns used. For example, the `ImageCollection` class demonstrates how to manage a collection of images, while the `importImages` function shows how to process and index images. -3. **Add new functionality**: Implement the new functionality by extending the existing classes or adding new ones. Ensure that your code integrates well with the existing structure and follows the established patterns. +4. **Implement your changes**: Add new functionality or modify existing code as needed. Ensure that your changes are consistent with the overall design of the package. -4. **Test your changes**: Write tests to verify your changes. Ensure that your new functionality works as expected and does not break existing features. +5. **Write tests**: Create tests to verify the functionality of your changes. Use the existing test cases as a reference for writing new ones. -5. **Run tests**: Execute the tests to validate your changes. Make sure all tests pass before submitting your changes. +6. **Run tests**: Execute the tests to ensure that your changes work as expected and do not introduce any regressions. -By following these steps, you can effectively extend the `image-memory` package to add new features or improve existing ones. +By following these steps, you can effectively contribute to the `image-memory` package and enhance its capabilities. ## Reference @@ -77,7 +90,7 @@ By following these steps, you can effectively extend the `image-memory` package ### Entry points -- default → [./dist/index.js](./dist/index.js) +- default → `./dist/index.js` _(not found on disk)_ ### Dependencies @@ -106,6 +119,6 @@ External: `@azure-rest/maps-search`, `better-sqlite3`, `debug`, `get-folder-size --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter image-memory docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter image-memory docs:verify-links` to spot-check._ diff --git a/ts/packages/shell/README.AUTOGEN.md b/ts/packages/shell/README.AUTOGEN.md index ffaa861f6..91f143e61 100644 --- a/ts/packages/shell/README.AUTOGEN.md +++ b/ts/packages/shell/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # agent-shell — AI-generated documentation @@ -12,27 +12,58 @@ ## Overview -The `agent-shell` package is a TypeScript library that serves as the UI entry point for the TypeAgent sample code. It demonstrates architectures for building interactive agents with natural language interfaces using structured prompting and large language models (LLMs). The shell functions as a personal agent, capable of processing user requests, performing actions, answering questions, and engaging in conversations through an extensible set of agents. +The `agent-shell` package is a TypeScript library that serves as the graphical user interface (GUI) for the TypeAgent framework. It is designed to demonstrate how interactive agents with natural language interfaces can be built using structured prompting and large language models (LLMs). The shell acts as a personal agent, enabling users to issue requests, perform actions, ask questions, and engage in conversations. It integrates with other components of the TypeAgent ecosystem, such as the dispatcher and agent server, to provide a cohesive and interactive experience. ## What it does -The `agent-shell` package provides a graphical interface for interacting with TypeAgent. Its key features include: +The `agent-shell` package provides a rich set of features to facilitate interaction with the TypeAgent framework: -- **Conversation Management**: Users can create, switch, rename, and delete conversations. Commands such as `conversation list`, `conversation new [name]`, `conversation switch `, `conversation rename `, and `conversation delete ` allow for flexible conversation handling. Conversations persist across sessions, and the shell supports multi-conversation management when connected to an agent server. -- **Speech Input**: The shell supports voice input through Azure Speech Services or a Local Whisper Service, enabling users to interact with the system using speech in addition to text input. -- **Command Execution**: The shell processes various commands, including natural language inputs, to perform tasks or retrieve information. -- **Multi-client Notifications**: When multiple clients are connected to the same conversation, the shell displays status messages to inform users of client activity. -- **Local Mode**: In the absence of an agent server, the shell operates in local mode, hosting a WebSocket for in-process port discovery and enabling external clients to connect to in-process agents. +### Conversation Management -The shell integrates with other components of the TypeAgent ecosystem, such as the dispatcher and agent server, to provide a cohesive and interactive user experience. +- **Multi-Conversation Support**: Users can create, switch, rename, and delete conversations. Conversations persist across sessions, and the shell supports managing multiple conversations when connected to an agent server. +- **Default Conversation**: Upon connecting to the agent server, the shell automatically joins a default conversation named `"Shell"`. This conversation is persistent, and its history is replayed on reconnect. +- **Commands for Conversation Management**: Users can manage conversations using commands such as: + - `/conversation list` to list all conversations. + - `/conversation new [name]` to create a new conversation. + - `/conversation switch ` to switch to a specific conversation. + - `/conversation rename [id|name] ` to rename a conversation. + - `/conversation delete ` to delete a conversation. + +### Speech Input + +The shell supports voice input through: + +- **Azure Speech Services**: Allows users to interact with the shell using voice commands. Requires configuration of Azure Speech API credentials. +- **Local Whisper Service**: An alternative to Azure Speech Services for local speech-to-text processing. + +### Local Mode + +In the absence of an agent server, the shell operates in local mode: + +- A single default conversation is available. +- The shell hosts an in-process WebSocket on `ws://localhost:8999/` for port discovery, enabling external clients to connect to in-process agents. + +### Multi-Client Notifications + +The shell provides real-time notifications when multiple clients (e.g., another shell or CLI) connect to or disconnect from the same conversation. + +### Request Queue Management + +The shell manages a queue of user requests: + +- Requests are queued if another request is already in progress. +- Users can cancel queued or running requests directly from the UI. ## Setup -To set up the `agent-shell` package, follow these steps: +To set up and run the `agent-shell` package, follow these steps: + +1. **Install Dependencies**: -1. **Install Dependencies**: Ensure you have all required dependencies installed. The shell is built using Electron, so you may need to install additional libraries for your operating system. For Linux/WSL users, refer to the build instructions provided in the Electron documentation (`https://www.electronjs.org/docs/latest/development/build-instructions-linux`). + - The shell is built using Electron. Linux/WSL users should follow the build instructions provided in the Electron documentation (`https://www.electronjs.org/docs/latest/development/build-instructions-linux`) to install the necessary libraries. -2. **Configure Environment Variables**: Set the following environment variables in your `.env` file or system environment: +2. **Set Environment Variables**: + Configure the following environment variables in your `.env` file or system environment: - `ELECTRON_RENDERER_URL`: The URL for the Electron renderer. - `SPEECH_SDK_ENDPOINT`: The service URL or speech API resource ID for Azure Speech Services. @@ -40,40 +71,67 @@ To set up the `agent-shell` package, follow these steps: - `SPEECH_SDK_REGION`: The region of the Azure Speech Services (e.g., `westus2`). - `WEBSOCKET_HOST`: The host for WebSocket connections. -3. **Run the Shell**: Use the following command to start the shell: +3. **Run the Shell**: + Start the shell using the following command: ```shell pnpm run shell ``` -4. **Optional Configuration for Azure Speech Services**: To enable voice input via Azure Speech Services, additional setup is required: - - Set `SPEECH_SDK_KEY` to `identity` in your `.env` file or `config.local.yaml` for keyless API access. +4. **Optional: Configure Azure Speech Services**: + + - To enable voice input via Azure Speech Services, set `SPEECH_SDK_KEY` to `identity` in your `.env` file or `config.local.yaml` for keyless API access. - Replace the `SPEECH_SDK_ENDPOINT` value with the Azure resource ID of your cognitive service instance (e.g., `/subscriptions//resourceGroups/myResourceGroup/providers/Microsoft.CognitiveServices/accounts/speechapi`). - Configure your Azure Speech API to support identity-based authentication. -For more details on setup, including troubleshooting tips for Windows users, refer to the hand-written README. +5. **Windows-Specific Notes**: + - If you experience lag during startup, consider adding the source code folder to the exclusions list for Windows Defender. Refer to the Windows support documentation for instructions. + - If using MS Graph-based sample agents (e.g., Calendar or Email), you may need to clear the identity cache if the authentication token becomes corrupted. Run the following command to delete the cache: + ```shell + del %LOCALAPPDATA%\.IdentityService\typeagent-tokencache + ``` ## Key Files -The `agent-shell` package is organized into several key components: +The `agent-shell` package is organized into several key files, each responsible for specific functionality: -- **Main Entry Point**: The primary entry point is [index.ts](./src/main/index.ts), which initializes the shell and sets up configurations. -- **Command Handlers**: Command processing is implemented in files such as [localWhisperCommandHandler.ts](./src/main/localWhisperCommandHandler.ts) and [commands/pen.ts](./src/main/commands/pen.ts). These files define the logic for handling specific commands. -- **Speech Processing**: The [azureSpeech.ts](./src/main/azureSpeech.ts) file manages interactions with Azure Speech Services, including token handling and speech-to-text processing. -- **Browser IPC**: The [browserIpc.ts](./src/main/browserIpc.ts) file handles inter-process communication between the shell and the browser. -- **Chat Server**: The [chatServer.ts](./src/main/chatServer.ts) file implements the WebSocket server for managing chat interactions and serving the shell's HTML interface. +- **[index.ts](./src/main/index.ts)**: The main entry point for initializing the shell and setting up configurations. +- **[localWhisperCommandHandler.ts](./src/main/localWhisperCommandHandler.ts)**: Handles commands related to the Local Whisper Service for speech-to-text processing. +- **[azureSpeech.ts](./src/main/azureSpeech.ts)**: Manages interactions with Azure Speech Services, including token handling and speech-to-text processing. +- **[browserIpc.ts](./src/main/browserIpc.ts)**: Handles inter-process communication (IPC) between the shell and the browser. +- **[chatServer.ts](./src/main/chatServer.ts)**: Implements the WebSocket server for managing chat interactions and serving the shell's HTML interface. +- **[commands/pen.ts](./src/main/commands/pen.ts)**: Contains logic for handling pen-related commands and events, including integration with a local endpoint for pen events. ## How to extend -To extend the `agent-shell` package, follow these steps: +To extend the functionality of the `agent-shell` package, follow these steps: + +1. **Familiarize Yourself with the Codebase**: + + - Start by reviewing [index.ts](./src/main/index.ts) to understand the initialization process and overall architecture. + +2. **Add New Commands**: + + - Create new command handlers in the `commands` directory (e.g., [commands/pen.ts](./src/main/commands/pen.ts)). + - Use the `CommandHandler` and `CommandHandlerTable` utilities from `@typeagent/agent-sdk` to define and register new commands. + +3. **Enhance Speech Processing**: + + - To add new speech processing capabilities, extend the logic in [azureSpeech.ts](./src/main/azureSpeech.ts) or [localWhisperCommandHandler.ts](./src/main/localWhisperCommandHandler.ts). + +4. **Modify or Add IPC Functionality**: + + - If you need to enable new forms of inter-process communication, extend the [browserIpc.ts](./src/main/browserIpc.ts) file. + +5. **Test Your Changes**: + + - Use `pnpm run shell` to test your changes in the shell environment. + - Ensure that all new features are thoroughly tested and do not introduce regressions. -1. **Understand the Initialization Process**: Start by reviewing [index.ts](./src/main/index.ts) to understand how the shell is initialized and configured. -2. **Add New Command Handlers**: Implement new command handlers in files like [commands/pen.ts](./src/main/commands/pen.ts). Use the existing patterns for defining and registering handlers. -3. **Integrate Additional Services**: If you need to add new services (e.g., for speech processing or external APIs), modify or create files such as [azureSpeech.ts](./src/main/azureSpeech.ts) to handle the integration. -4. **Test Your Changes**: Run the shell using `pnpm run shell` and verify that your changes work as intended. Ensure that all new functionality is thoroughly tested. -5. **Follow Existing Patterns**: Adhere to the established coding conventions and patterns in the project to maintain consistency and readability. +6. **Follow Coding Standards**: + - Adhere to the existing patterns and conventions in the codebase to maintain consistency and readability. -By following these guidelines, you can effectively contribute to the development and enhancement of the `agent-shell` package. +By following these steps, you can effectively contribute to the development and enhancement of the `agent-shell` package. ## Reference @@ -126,6 +184,6 @@ _5 environment variables referenced from `./src/` (set in `ts/.env` or your shel --- -_Auto-generated against commit `ff379b098decfab4eb45f78b6fa318358d7fbd75` on `2026-07-01T09:05:58.471Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter agent-shell docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter agent-shell docs:verify-links` to spot-check._ diff --git a/ts/packages/typeagent/README.AUTOGEN.md b/ts/packages/typeagent/README.AUTOGEN.md index 9cd0457d9..7c2dc9803 100644 --- a/ts/packages/typeagent/README.AUTOGEN.md +++ b/ts/packages/typeagent/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # typeagent — AI-generated documentation @@ -12,62 +12,66 @@ ## Overview -The `typeagent` package is a sample library code used by and intended only for the example agents and apps in the TypeAgent project. It provides various utilities and components that facilitate async processing, vectors and embeddings, storage, text processing, and collections. +The `typeagent` package is a sample library designed to support the development of agents and applications within the TypeAgent project. It provides a collection of utilities and components for common tasks such as asynchronous processing, vector and embedding operations, storage management, text processing, and collection utilities. While it is primarily intended for internal use within the TypeAgent ecosystem, it serves as a foundational library for many other packages and examples in the project. ## What it does -The `typeagent` package offers a range of functionalities that are essential for building and running agents within the TypeAgent ecosystem. Key capabilities include: +The `typeagent` package provides a variety of utilities and tools that are essential for building and running agents. Its key functionalities include: -- **Async Processing**: Functions like `mapAsync` in [arrayAsync.ts](./src/arrayAsync.ts) and `callWithRetry` in [async.ts](./src/async.ts) enable concurrent and retryable async operations. -- **Vectors and Embeddings**: Utilities for handling vector operations and embeddings, such as `vectorIndex` and `semanticMap`, found in the `vector` directory. -- **Storage**: Components for managing storage, including `objectFolder`, `objectPage`, and `embeddingFS`, located in the `storage` directory. -- **Text Processing**: Tools for processing text, including `textClassifier` in [textClassifier.ts](./src/classifier/textClassifier.ts) and `createTypeChat` in [chat.ts](./src/chat.ts). -- **Collections**: Various collection utilities provided in [lib/index.ts](./src/lib/index.ts). +- **Async Processing**: Utilities like `mapAsync` (in [arrayAsync.ts](./src/arrayAsync.ts)) and `callWithRetry` (in [async.ts](./src/async.ts)) enable efficient handling of concurrent and retryable asynchronous operations. +- **Vectors and Embeddings**: A suite of tools for working with vectors and embeddings, such as `vectorIndex` and `semanticMap`, located in the `vector` directory. +- **Storage Management**: Components for managing data storage, including `objectFolder`, `objectPage`, and `embeddingFS`, which are implemented in the `storage` directory. +- **Text Processing**: Functions for text-related tasks, such as `createTypeChat` (in [chat.ts](./src/chat.ts)) for chat interactions and `createTextClassifier` (in [textClassifier.ts](./src/classifier/textClassifier.ts)) for text classification. +- **Collections**: A variety of collection utilities, such as `binarySearch` and `isUndefinedOrEmpty`, provided in [lib/array.ts](./src/lib/array.ts). -These components are used by multiple packages and examples within the TypeAgent project, making `typeagent` a foundational library for the ecosystem. +These features are widely used across the TypeAgent project, making `typeagent` a critical dependency for other packages and examples. ## Setup -The `typeagent` package does not require any special setup beyond installing its dependencies. To get started, simply run: +The `typeagent` package does not require any special setup beyond installing its dependencies. To get started, run the following command in the root of the TypeAgent monorepo: ```sh pnpm install ``` -For detailed setup instructions, see the hand-written README. +For additional details, refer to the hand-written README. ## Key Files -The `typeagent` package is organized into several key files and directories, each responsible for different aspects of the library: +The `typeagent` package is organized into several key files and directories, each serving a specific purpose: -- **[src/index.ts](./src/index.ts)**: The main entry point that exports various modules and components. -- **[src/arrayAsync.ts](./src/arrayAsync.ts)**: Contains functions for async array processing, such as `mapAsync`. -- **[src/async.ts](./src/async.ts)**: Provides utilities for retrying async operations with `callWithRetry`. -- **[src/chat.ts](./src/chat.ts)**: Implements the `createTypeChat` function for handling chat interactions. -- **[src/classifier/textClassifier.ts](./src/classifier/textClassifier.ts)**: Defines the `createTextClassifier` function for text classification. -- **[src/constraints.ts](./src/constraints.ts)**: Contains the `createConstraintsValidator` function for validating constraints. -- **[src/dateTime.ts](./src/dateTime.ts)**: Provides utilities for handling date and time operations. -- **[src/lib/array.ts](./src/lib/array.ts)**: Includes various array utilities, such as `binarySearch`. +- **[src/index.ts](./src/index.ts)**: The main entry point that re-exports the package's modules and components. +- **[src/arrayAsync.ts](./src/arrayAsync.ts)**: Contains utilities for asynchronous array processing, such as `mapAsync`, which supports concurrent operations. +- **[src/async.ts](./src/async.ts)**: Provides `callWithRetry`, a utility for retrying asynchronous operations with optional timeouts and error handling. +- **[src/chat.ts](./src/chat.ts)**: Implements `createTypeChat`, a function for managing chat interactions with context, history, and instructions. +- **[src/classifier/textClassifier.ts](./src/classifier/textClassifier.ts)**: Defines `createTextClassifier`, a utility for classifying text based on predefined schemas and classes. +- **[src/constraints.ts](./src/constraints.ts)**: Implements `createConstraintsValidator`, a utility for validating objects against custom constraints. +- **[src/dateTime.ts](./src/dateTime.ts)**: Provides functions for handling and formatting date and time values, such as `timestampString` and `parseTimestamped`. +- **[src/lib/array.ts](./src/lib/array.ts)**: Includes collection utilities like `binarySearch` and `isUndefinedOrEmpty` for array manipulation. +- **[src/vector/**](./src/vector/): A directory containing utilities for vector and embedding operations, such as `vectorIndex` and `semanticMap`. +- **[src/storage/**](./src/storage/): A directory with components for managing storage, including `objectFolder`, `objectPage`, and `embeddingFS`. ## How to extend To extend the `typeagent` package, follow these steps: -1. **Identify the area to extend**: Determine which component or utility you need to modify or add to. For example, if you need to add a new async processing function, start with [arrayAsync.ts](./src/arrayAsync.ts). +1. **Identify the area to extend**: Determine which functionality you want to add or modify. For example, if you need to add a new text processing utility, start by examining [textProcessing.ts](./src/textProcessing.ts). -2. **Create or modify files**: Add new functions or modify existing ones in the appropriate file. Ensure that your code follows the existing patterns and conventions. +2. **Modify or add new files**: Implement your changes in the appropriate file or create a new file in the relevant directory. For instance, if you're adding a new vector operation, place it in the `vector` directory. -3. **Export your additions**: Make sure to export your new functions or components in [index.ts](./src/index.ts) so they are accessible to other parts of the TypeAgent project. +3. **Update exports**: Ensure that your new functionality is exported in [index.ts](./src/index.ts) so it can be accessed by other parts of the TypeAgent project. -4. **Write tests**: Add tests for your new functionality to ensure it works as expected. Place your tests in the corresponding test files or create new ones if necessary. +4. **Write tests**: Create or update test files to cover your new functionality. Tests should be placed in the corresponding test files or in new test files if necessary. -5. **Run tests**: Execute the test suite to verify that your changes do not break existing functionality. Use the following command: +5. **Run tests**: Use the following command to run the test suite and verify that your changes work as expected: ```sh pnpm test ``` -By following these steps, you can effectively extend the `typeagent` package and contribute to the TypeAgent project. +6. **Document your changes**: Update the hand-written README or other relevant documentation to describe your new functionality and how to use it. + +By following these steps, you can contribute to the `typeagent` package and enhance its capabilities for the TypeAgent ecosystem. ## Reference @@ -75,7 +79,7 @@ By following these steps, you can effectively extend the `typeagent` package and ### Entry points -- default → [./dist/index.js](./dist/index.js) +- default → `./dist/index.js` _(not found on disk)_ ### Dependencies @@ -106,6 +110,6 @@ External: `async`, `cheerio`, `debug`, `typechat`, `typescript` --- -_Auto-generated against commit `127a36a95a15e918be533d6eaaf08adebe9070d9` on `2026-06-26T03:01:52.873Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter typeagent docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter typeagent docs:verify-links` to spot-check._ diff --git a/ts/packages/vscode-shell/README.AUTOGEN.md b/ts/packages/vscode-shell/README.AUTOGEN.md index e0451ea0b..d1400f413 100644 --- a/ts/packages/vscode-shell/README.AUTOGEN.md +++ b/ts/packages/vscode-shell/README.AUTOGEN.md @@ -3,7 +3,7 @@ - + # vscode-shell — AI-generated documentation @@ -16,7 +16,7 @@ The `vscode-shell` package integrates the TypeAgent shell chat into Visual Studi ## What it does -This package enables users to interact with TypeAgent conversations directly within the Visual Studio Code environment. It provides a rich set of features for managing and participating in conversations, including: +The `vscode-shell` package allows users to interact with TypeAgent conversations directly within the Visual Studio Code environment. It provides the following features: - **Activity Bar Integration**: A dedicated icon in the activity bar opens the persistent **Chat** side panel. - **Editor Tabs for Chats**: Users can open multiple chat tabs in the editor, each representing a separate conversation. @@ -150,6 +150,6 @@ External: `ansi_up`, `debug`, `dompurify`, `isomorphic-ws`, `markdown-it`, `ws` --- -_Auto-generated against commit `ff379b098decfab4eb45f78b6fa318358d7fbd75` on `2026-07-01T09:05:58.471Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter vscode-shell docs:verify-links` to spot-check._ +_Auto-generated against commit `15ef5aa0362e3296bd9d6bd2f001fab704375d27` on `2026-07-04T08:54:09.388Z` by `docs-generate.yml`. Links validated at that commit; the working tree may have drifted by up to 24h. Re-run `pnpm --filter vscode-shell docs:verify-links` to spot-check._