Details
> **You're getting a working system.**
> **You're also getting the method that built it.**
> **It will be complete when it solves your problem.**
>
> **You do not need to write code. You do not need to read the source. You only need to know what problem you are trying to solve, and to be willing to ask. The rest happens in plain language, the same way this system was built.**
---
# Free Agent — Chameleon
**A personal operating system that becomes what you need.**
---
## What It Is
Free Agent is purposely not a finished product. It is not a fixed tool. It is a system that was built to become whatever its user needs it to be — and it keeps becoming that, one problem at a time.
Think of a chameleon. It does not have one color. It has the capacity to become whatever color it needs to be. The color is not the point. The capacity is the point.
Free Agent is the same. It is not a memory system. It is the capacity to remember. It is not a web model driver. It is the capacity to drive whatever model exists. It is not a scheduler or a watchdog or a sandbox or a persona system. It is the capacity to grow any of those, and to grow whatever the next problem requires.
It runs on the user's own machine. It drives the user's own applications. It stores the user's own knowledge. It acts on the user's behalf.
The subsystems in this guide are colors the chameleon has worn. They are real, they work, and they solve the problems they were built to solve. But they are not the system. The system is the capacity that produced them — and that capacity is still running.
---
## How It Came To Be
Free Agent was not designed. It grew.
It began as an attempt to use AI without paying for API access. A local model was installed. It was slow, so a faster web model was wrapped in a small harness. Once that worked, the next question was asked: *what would a good memory system look like?* The models answered. Some of what they proposed was kept. Then the next question. Then the next.
There was no specification. There was no plan. There was no team. There was a person, an idea, and a set of capable models. Every subsystem in this guide exists because a problem was hit, a question was asked, and the system answered.
Seven months. Every day. Morning to night. Life happened, and the system waited. Not one line of code was written by hand. Every line was written by AI, on commission, by a user who does not write code.
The result is a working personal operating system that would have required a team of six to eight engineers to build conventionally. It is still growing. The questions have not stopped.
---
## The Loop
The method that built Free Agent is simple, and it is the most important thing in this guide:
1. Something doesn't work.
2. You ask: *what can we do?*
3. The AI proposes something.
4. You try it.
5. If it works, you keep it. If it doesn't, you ask again.
6. The thing you kept becomes a new capability.
7. That capability reveals the next failure.
8. Back to step 1.
This is how the system was built. It is also how the system continues to grow. There was no plan. There was a loop. The loop produced every subsystem in this guide.
Most software is designed first and built second. This system was built first and understood second. Every subsystem is the trace of a failure solved.
If you are reading this and thinking about building something similar, the loop is the most transferable thing here. Not the code. Not the tools. The loop. Ask what can be done. Try it. Keep what works. Ask again.
---
## How To Read This Guide
This guide is not a specification. There never was one. It is a record of problems solved, one after another, across seven months.
When you read a section, you are reading the answer to a question someone asked because something was broken, missing, slow, or unclear. Every subsystem solves at least one problem. Every name is a fossil of the moment it was made. Every file carries a date. If you want to know how the system came to exist, read the code in order of modification date — the history is legible in the file list.
Because every piece was built to solve a real problem, and problems do not respect names, every subsystem in this document is more than its name suggests. A "text editor" grew into a small IDE because writing prompts turned out to need all of that. A "web chat driver" grew into a three-thousand-line runtime with three recovery tiers because driving a web chat turned out to need all of that. A "memory system" grew into five memory types with dual embeddings and cross-database inheritance because remembering turned out to need all of that.
The name describes the primary function. The code implements everything the function needed to survive in the real world.
If you want to understand any subsystem in this guide, ask what problem it was solving. The answer will explain why it looks the way it does.
---
## What This Guide Offers
Free Agent is not a finished system. It was not designed to be one. It is a working system that was built by asking *"what can we do?"* every time something failed, and keeping what worked. The same loop is still running. It will keep running as long as the question keeps being asked.
What this guide offers is not a product. It is the current state of a solution in progress. If you use Free Agent, you are not inheriting a tool. You are inheriting a method. The system will keep growing — because it was built to grow — and how it grows is up to you.
The system that exists today is the answer to the questions that have been asked so far. The questions that come next will be yours. Nothing about the system is fixed. Nothing about it is final. Every subsystem in this document can be extended, replaced, or reshaped to fit a different set of problems.
If something in this guide does not do what you need, the answer is not "that feature is missing." The answer is the same question that built every subsystem in this document: *what can we do?* Ask it. Try something. Keep what works. Ask again.
The system does not decide what it is. You do.
---
# LAYER 1 — WHAT IS FREE AGENT
## 1. Introduction
Free Agent is a Windows desktop AI automation platform. It combines multiple AI models with a memory system, a scheduler, a watchdog, and deep UI automation, so that a person can command their computer in natural language and have complex work happen reliably in the background.
It was built for users with physical impairments, and the design reflects that priority: voice control, wake-word operation, phone integration, and click-free automation are first-class features, not afterthoughts.
### The core philosophy
Five principles shape everything:
1. **The AI makes decisions; scripts execute actions.** The AI reasons about what to do; a Python script or automation tool performs the mechanical steps. This is why Free Agent can automate unreliable UIs reliably.
2. **State persistence is non-negotiable.** Any task that takes more than one step saves its state to disk. Interruption does not mean loss.
3. **Explicit instructions, no ambiguity.** When the system needs the AI to do something, it says exactly what, in exactly the form required. No guesswork.
4. **The AI is a follower, not a decider.** The AI executes the script's instructions precisely. It does not "optimize" them.
5. **Test every component before integration.** Individual parts are proven before they are combined.
These five principles appear in every subsystem: the memory system, the persona system, the watchdog, and Universal Automate.
### What makes Free Agent unusual
- **No API keys required for web models.** Free Agent drives Gemini, Qwen, Comet, DeepSeek, and other web-based AI through the visible browser interface. No paid API, no token cost.
- **One interface for any web AI.** The Universal Automate engine drives any web chat by configuration alone. Adding a new model is adding a profile and some icons, not writing code.
- **Self-extending.** New automation tools are added by writing a script and registering it, without touching the core application.
- **Memory that works.** Semantic search, trigger keywords, and automatic linking mean context survives across sessions.
- **Recovery that is designed, not bolted on.** The Watchdog and Universal Automate are built to notice conditions they don't understand and hand control to the AI, rather than fail silently.
---
## 2. Getting Started
### 2.1 System requirements
- Windows 10 or 11
- Python 3.10 or higher (3.8+ may work; 3.10+ recommended)
- .NET Framework 4.8 or .NET Core 5.0+
- Ollama installed and running
- Internet connection (for web models and license validation)
- A microphone (for voice features)
- 16 GB RAM recommended, 50 GB free disk space for models
### 2.2 Required Ollama models
Ten models are recommended for full functionality. The most critical is the embedding model — without it, memory does not work:
| # | Model | Purpose | Size |
|---|-------|---------|------|
| 1 | nomic-embed-text:latest | Memory embeddings (critical) | ~274 MB |
| 2 | qwen2.5:1.5b | Coprocessor / light tasks | ~1 GB |
| 3 | qwen2.5:7b | Main chat / research | ~4.7 GB |
| 4 | qwen2.5_tool:latest | Tool calling | ~3 GB |
| 5 | qwen3.5:27b | Heavy reasoning | ~17 GB |
| 6 | llama3.2:3b | Fast fallback | ~2 GB |
| 7 | deepseek-r1:latest | Reasoning | ~5 GB |
| 8 | phi4-mini-reasoning:latest | Light reasoning | ~3 GB |
| 9 | minicpm-v:8b | Vision (page description, popup detection) | ~5.5 GB |
| 10 | moondream:v2 | Vision (secondary) | ~1.7 GB |
Pull each with: ollama pull <name>.
### 2.3 First launch
1. Start Ollama: ollama serve
2. Launch Free Agent: Free Agent.exe
3. Select a model from the dropdown
4. Start typing into the chat input and press Enter
The AI will use tools automatically when it needs to act on your behalf.
### 2.4 How to give commands
There are four ways to talk to Free Agent:
- **Type** into the chat input.
- **Speak** — say "Hey Agent" to wake the voice input, then speak your command.
-**Use the phone app to speak or type your command
- **Use a trigger phrase** — a stored SOP or memory can be activated by saying its trigger keywords.
If you want Free Agent to do something specific with a tool, you can write a flat JSON tool call directly. See Layer 3 for the full tool reference.
### 2.5 Your first useful commands
Try these early:
- *"Remember that I prefer concise answers."* — stores a preference.
- *"Show me my memory statistics."* — verifies the memory system is running.
- *"Create a daily reminder at 4:30 PM to check my messages."* — schedules a recurring task.
These give you a feel for each subsystem without needing any setup.
---
## 3. Where Everything Lives
Free Agent keeps its working files under C:/ai/. The important ones:
| Path | Purpose |
|------|---------|
| C:/ai/agent_memory.db | The memory database |
| C:/ai/conversation_history.json | Chat history |
| C:/ai/schedules/schedule.json | Scheduled events |
| C:/ai/masterwatch_triggers.json | Watchdog rules |
| C:/ai/memory_settings.json | Memory and trigger settings |
| C:/ai/app_settings.json | Application settings |
| C:/ai/aispace/automation_inventory.json | Registry of automation tools |
| C:/ai/prompt_scripts/ | Multi-step scripts |
| C:/ai/profiles/ | Web model profiles (Universal Automate) |
| C:/ai/icons/ | Icon PNGs used for web model recognition |
| C:/ai/scans/ | Output from UI scans |
| C:/ai/recordings/ | Recorded automation projects |
| C:/ai/logs/ | Log files |
| C:/ai/persona_memory/ | Home of the delivery-channel pattern (see Layer 4) |
| C:/ai/updated_docs/ | This and other documentation |
All required directories are created automatically if they don't exist.
---
# LAYER 2 — FEATURES
This layer describes each subsystem. Everything here is verified against source code or current documentation.
---
## 4. Multi-Model AI
Free Agent can talk to several kinds of AI model:
| Type | Examples | How it works |
|------|----------|--------------|
| **Web-based** | Gemini, Qwen, Comet, DeepSeek, HuggingChat, Perplexity, Grok | Browser automation via Universal Automate |
| **Local (Ollama)** | qwen2.5:7b, qwen3.5:27b, deepseek-r1, llama3.2:3b | Direct Ollama API |
| **Custom** | Any web AI you configure | Add a profile JSON and icons |
| **Openai compatable api service
### Switching models
Click the model selector in the toolbar. The system switches immediately. If you are mid-conversation, the conversation context carries over.
### Failover and rotation
When a web model becomes unavailable (rate limit, connection error, unresponsive UI), Free Agent's Watchdog can automatically switch to another model and hand off the task with a context-preserving prompt. See Section 9 for the full mechanism.
---
## 5. Memory System (Overview)
Free Agent's memory is a SQLite database with semantic search. It survives application restarts, model switches, and session breaks.
### What memory does
- **Stores facts, preferences, events, procedures.** Five types are defined: Fact, Preference, Episodic, Procedural, and System (protected).
- **Finds memories by meaning** — not just keywords — using embeddings.
- **Loads context automatically** when a trigger keyword appears in your input.
- **Links related memories** so retrieval surfaces nearby information.
- **Decays unused memories** so the system stays focused.
### Trigger keywords
A trigger keyword is a phrase that loads a specific memory when you say it. Example:
json
{"command":"remember","content":"How to run a duty check: ...","importance":"9","trigger_keywords":"duty check,run duty,check pampered"}
Once stored, saying "duty check" loads the memory into context automatically.
**Critical rule:** trigger keywords must be **comma-separated**. Space-separated keywords are treated as one phrase and will silently fail to trigger.
### Memory detail
Layer 3 contains the full memory reference — types, retrieval modes, settings, decay formula, and the dual-embedding design that makes trigger matching reliable.
---
## 6. The Pipeline
The pipeline is how every prompt gets processed, whether it comes from the user, a schedule, voice input, a watchdog trigger, an active plan, or a tool response that needs a follow-up. It is a singleton background worker with a request queue: one prompt is processed at a time, system-wide, and every other request waits in line.
### 6.1 Architecture
The pipeline is implemented in AgentPipeline.vb as a singleton. It holds a single concurrent queue of PipelineRequest objects and a single background worker that drains the queue. When a request is submitted, it is enqueued. If the worker is idle, it starts. If it is busy, the request waits. This design keeps the system serial and predictable: there is never more than one prompt in flight to the model.
Each PipelineRequest carries:
- **Prompt** - the text to send
- **Source** - User, Schedule, Voice, Watchdog, Plan, or Tool
- **UserPrompt** - the raw user input, when the source is a user or a tool
- **TurnMarker** - optional marker used to identify a response boundary
- **IsScriptValidation** - true when the request is a persona check or similar verification
- **OnComplete / OnError** - callbacks for the caller
### 6.2 Prompt assembly
Before the model sees anything, BuildPrompt assembles a structured context block. The order is fixed:
1. **System context (memory)** - retrieved by MemoryOrchestrator.RetrieveContextAsync using the user prompt as the query.
2. **Plan context** - included only if a plan is active.
3. **History context** - included only if the inject-history flag is set. The flag is one-shot: it clears after use.
4. **User prompt** - prefixed with === USER PROMPT ===, then the prompt itself.
If the request is a script validation (IsScriptValidation = true), the entire context assembly is bypassed. The prompt is prefixed with === NEW QUESTION - IGNORE ALL PREVIOUS CONTEXT === and sent alone. This is how persona checks and similar verifications stay isolated from accumulated context.
### 6.3 Response routing
When the model responds, the pipeline inspects the response and routes it. The order of checks is significant:
| Step | Check | Action |
|------|-------|--------|
| 0 | Always | Raw output displayed; the watchdog feed receives the line |
| 0b | Not flat JSON | Truncation check; if truncated, the active web model is stopped and the user is warned |
| 0c | JSON array present | Treated as a plan; passed to PlanManager.CreatePlan (display only) |
| 0d | Prompt script JSON | Auto-saved; the user is notified |
| 1 | Flat JSON (command: key present) | Dispatched to FlatJsonHandler; result is sent back to the model; the response is processed recursively |
| 2 | Nested JSON (name / parameters keys present) | Dispatched to ToolExecutor.ExecuteToolAsync; result is sent back to the model |
| 3 | Nothing above | Treated as the final answer; cleaned, displayed, and auto-stored to memory if important |
Before step 1, there is a force-JSON feedback loop. If the response contains no JSON at all and does not begin with done! or len=, the pipeline sends a corrective prompt back to the model asking it to either produce a tool call or declare completion. This keeps the model in tool-call mode during scripted work.
### 6.4 Cancellation
Each request is wrapped in a PipelineCancellationContext scope. Cancellation is scoped to the active request, not global: when a request is cancelled, its delivery unwinds cleanly before the queue continues. Queued requests are preserved. CancelPendingRequests clears the queue and signals the active request; the worker retains ownership until the active request finishes unwinding.
### 6.5 Connection to the Watchdog
The pipeline feeds every model response to the watchdog for pattern matching. If a watchdog trigger fires, it returns a recovery prompt that re-enters the pipeline as a new request. The watchdog editor and the automation editor are separate systems; the pipeline is the layer that connects the watchdog to model responses.
### 6.6 What the pipeline does not do
The pipeline does not decide what to do. It does not choose a model, choose a tool, or evaluate quality. It assembles context, sends the prompt, inspects the response, dispatches to the right handler, and returns the result. All decisions belong to the model, the watchdog, or the plan.
---
## 7. Voice and Speech
Free Agent supports two distinct kinds of voice interaction, and keeping them separate is the key to understanding the system:
- **Voice to AI** -- spoken words become a prompt that the AI sees and responds to.
- **Voice commands** -- specific phrases that control Free Agent directly and never reach the AI.
The two are handled at different layers. Saying "Hey Agent, sleep" does not send "sleep" to the model; it minimizes the window. Saying "Hey Agent, what is the capital of France" transcribes, shows the text for confirmation, and sends it only after you say "send."
### 7.1 Two listeners, one microphone
The system has three things that want the microphone at any given moment, and they must take turns:
1. **Wake word listener** -- always on in wake mode, watching for "Hey Agent," "Hello Agent," or "Wake Up."
2. **Recording session** -- captures the actual speech while you talk.
3. **Confirmation listener** -- after a transcription appears, listens for "send," "delete," or "edit."
The system arbitrates cleanly: when a transcription completes, the wake listener stops so the confirmation listener can take the mic. After confirmation is handled, the wake listener restarts. You never have to think about this -- it is handled by state, not by luck.
### 7.2 Voice to AI
Speaking a prompt goes through these steps:
1. **Start.** Click the voice button, or say the wake word in wake mode.
2. **Record.** Audio is captured at 16 kHz mono -- the native format Whisper expects.
3. **Transcribe.** whisper-cli.exe runs locally with the ggml-base model. No cloud, no API key.
4. **Confirm.** The transcribed text is placed in the input box, and the system listens for a voice command: **Send**, **Delete**, or **Edit**.
5. **Dispatch.** Send fires the send button. Delete clears the input. Edit focuses the field so you can correct it by hand.
This confirmation step is the safety net. In a noisy room or with an unfamiliar accent, you get to correct the transcription before anything is sent.
### 7.3 Wake mode and auto-stop
In wake mode, once the wake word fires, recording starts and there is a brief grace delay before silence detection begins. This is to avoid the tail of "Hey Agent" causing an immediate stop. After the grace delay, if roughly two seconds pass with no audio above the threshold, recording stops and transcription runs automatically.
In manual mode (click the button to start), no auto-stop occurs. You click again when you are done.
### 7.4 Voice commands (pre-pipeline)
These phrases are intercepted before the model ever sees them. They do not appear in chat history.
**Phone mode commands:**
| Phrase | Effect |
|--------|--------|
| full display / full mode | Phone shows all content, speaks nothing |
| speak summary / summary mode | Phone speaks summary only, displays nothing |
| hybrid / hybrid mode | Phone shows all content and speaks the summary |
| what mode / current mode / check mode | Reports the current phone mode |
**System commands:**
| Phrase | Effect |
|--------|--------|
| sleep / go to sleep | Minimize to system tray |
| wake up | Exit voice mode |
| help | Show available voice commands |
| send | Send the current prompt |
| edit | Open the prompt editor |
| type [text] | Type text into the prompt input |
| show history | Show recent conversation history |
**Tool launch commands:**
| Phrase | Effect |
|--------|--------|
| show me the mouse coordinates | Launch the mouse coordinate display |
| capture icon / launch icon capture / icon capture tool | Launch the Icon Capture tool |
| continue from where we left off / continue | Load conversation summaries |
### 7.5 Voice input in the Prompt Management System
The Prompt Management System (see Section 22) has its own voice button. When transcription completes there, it inserts at the cursor instead of routing through the confirmation loop. The same voice module powers both -- the difference is which surface requested the transcription. The module tracks the source, so the same microphone serves multiple windows without collision.
### 7.6 The read_this markers
When Free Agent responds on the phone, the spoken content is determined by two markers in the response:
read_this
...content that should be spoken...
end_read
Behavior by phone mode:
- **Full Display** -- shows all content, speaks nothing (markers ignored)
- **Speak Summary** -- speaks only the read_this content, displays nothing
- **Hybrid** -- shows all content and speaks the read_this content
If no markers are present, nothing is spoken. This lets the model choose which part of a long answer is worth hearing out loud.
### 7.7 The Phone Interface
The Phone Interface extends Free Agent to a phone on the same Wi-Fi network. It has two transports:
- **WebSocket on port 8765** -- the message channel. Carries typed text and responses.
- **UDP discovery on port 8766** -- the auto-find channel. The phone broadcasts a discovery message and Free Agent replies with its address.
Once connected, everything typed on the phone takes the **same pipeline path as typed input on the desktop.** A phone message is placed into the input box and the send button is fired programmatically, which means memory retrieval, triggers, the watchdog, and the full pipeline all work exactly as they do for typed prompts.
Three display modes control what the phone shows and speaks (see 7.4 and 7.6). The default is Hybrid.
The Phone Interface opens as its own window from the main application, shows the connection address, includes a Copy button to share the address, and tracks connected clients.
### 7.8 Voice troubleshooting
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| Voice button does nothing | Whisper not installed or wrong path | Verify C:/ai/whisper/Release/whisper-cli.exe exists |
| Transcription is empty | Model file missing | Verify C:/ai/models/ggml-base.bin exists |
| Wake word never fires | Microphone permission or threshold | Check Windows mic permissions; adjust threshold |
| Phone cannot connect | Firewall blocking ports | Allow ports 8765 (WebSocket) and 8766 (UDP) |
| Phone shows text but does not speak | Mode is Full Display | Say "speak summary" or "hybrid" |
| Phone speaks but shows nothing | Mode is Speak Summary | Say "full display" or "hybrid" |
| Voice button stuck on "No Voice" | Voice module not initialized | Restart Free Agent; check Whisper paths |
## 8. Scheduler
Schedule recurring or one-time tasks that run without your involvement.
### Repeat types
None, Minute, Minutes5, Minutes10, Minutes15, Minutes30, Hourly, Hours2, Hours4, Hours6, Hours8, Hours12, Daily, Weekly, Monthly.
### Action types
- **CustomPrompt** — sends a prompt through the AI pipeline
- **RunScript** — runs a Python, BAT, CMD, PS1, or EXE file
- **LaunchProgram** — launches any Windows program
### Eternal workflows
A schedule can update itself. A task that runs daily can adjust its own frequency based on what it finds, reschedule itself, and continue indefinitely. This is how Free Agent builds "set it and forget it" automations.
---
## 9. Watchdog
The Watchdog monitors Free Agent's chat output for configured patterns and performs a configured action when one matches.
### Five action types
| Action | What it does |
|--------|--------------|
| **Prompt** | Sends a recovery prompt back to the AI |
| **Execute** | Runs an external program or script (and sends back a prompt on completion) |
| **Reasoning** | Uses a small local model (qwen2.5:1.5b) to analyze the condition |
| **Alert** | Notifies you and takes no further action |
| **Switch Model** | Changes the active web AI model and hands off the task |
### Matching behavior
Matching is **literal** (case-insensitive, word-boundary). It is **not** regex. A trigger fires when the configured text appears inside the incoming line and is not part of a larger word.
### Variables
Two variables are expanded in most fields:
- {line} — the complete incoming line that matched the trigger
- {trigger} — the name of the trigger that fired
### Switch Model handoff
When Switch Model fires, Free Agent changes the active model and then sends a handoff prompt. The default handoff is: *"Continue the user's task using the current conversation context. Do not switch models."* The triggering error line is not automatically passed to the new model.
If the switch fails, no handoff is sent — the failure is logged.
### Full watchdog detail
Layer 3 contains the full field reference, test procedures, and recommended action table.
---
## 10. Universal Automate (Web Model Interface)
Universal Automate is the engine that drives every web-based AI model in Free Agent. It is what makes the system model-agnostic: any browser-hosted AI -- Gemini, DeepSeek, Qwen, Comet, Capriole, or anything else -- can be driven by adding a profile and a few icons, without writing code.
The design goal is simple: **interact with the browser the way a human would.** Find the window, focus it, find the input, paste the prompt, submit, wait, copy the response. No API keys, no reverse-engineered protocols, no per-model Python. Just visible UI interaction that works the same way the user does.
### 10.1 What it is
universal_automate.py is a single-shot runtime. It is invoked once per prompt with a target (process,title) and a prompt string. It finds the window, enters the prompt, submits it, waits for the model to finish, copies the response, prints it to stdout, and exits.
Every invocation is independent. The engine does not run as a background service. It is called when a prompt needs to go out, and it exits when the response comes back.
This is why the watchdog can be the decision layer. The engine handles normal operation and known conditions. The watchdog handles conditions the engine does not understand. The boundary between the two is defined by stdout markers -- a stable, simple protocol.
### 10.2 Why it matters
Without Universal Automate, Free Agent would be tied to APIs. APIs cost money, require keys, have rate limits, and are subject to the provider's terms. Universal Automate drives the free web UI instead, at human speed, using normal interaction.
Three consequences:
- **No API keys, no token costs.** Any model the user can reach in a browser is a model Free Agent can use.
- **One interface for every model.** A new model is a profile and some icons, not a Python file.
- **The engine inherits whatever the model provider ships.** When a model updates its UI, the user updates the icons. When it adds a capability, the user gets it.
The tradeoff is that this kind of automation is inherently less reliable than an API. Screens can change. Windows can lose focus. Models can throttle. Universal Automate is designed to expect all of this and to keep going or hand off cleanly.
### 10.3 The three-level recovery philosophy
Universal Automate recognizes three levels of condition during a response:
1. **Normal.** Find, focus, click, paste, submit, wait, copy. No decision needed. The engine completes and returns the response.
2. **LOOK CONTINUE.** A condition the engine knows how to handle (for example, a "Continue generating" button). The engine clicks it, resets the completion timer, and keeps waiting. The user does not need to be involved.
3. **LOOK ANYTHING.** A condition the engine does not know how to handle. The engine captures a screenshot, logs the exact filename and screen coordinates, prints a machine-readable marker on stdout, and stops. Control goes to the watchdog.
The design principle that holds it all together: **automate what the engine understands, document what it does not understand, and never pretend an unknown condition is understood.** That is what makes the same engine safe to use across many models.
### 10.4 How it is extended
The engine is configured by files, not code changes.
**Profiles.** Each model has a JSON profile at C:/ai/profiles/<ModelName>.json describing the target process, window title, safe click location, input box offset, submit method, completion detection method, and icon paths. If a profile does not exist, the engine creates a reasonable default and the user edits it.
**Icons.** Each profile references PNG icons for submit, searching, done, and copy. The user captures them from the model's actual UI and drops them in C:/ai/icons.
**Recovery icons.** Additional recovery conditions are added by creating numbered PNG files:
- <model>_look_continue_N.png -- the engine handles it automatically.
- <model>_look_anything_N.png -- the engine reports it and stops.
- <model>_wait_to_submit.png -- an optional pre-submit signal.
No code changes are required for a new model. The default experience for adding a model is: create a profile, capture four icons, test. Most models need nothing more.
### 10.5 What it does not do
Universal Automate does not diagnose unknown conditions, decide whether to retry after a timeout, choose a different model, or ask the user. Those decisions belong to the Watchdog (Section 9) and to the model itself. The engine classifies; the watchdog decides.
The full reference for Universal Automate -- the profile format, the search-region model, the stdout protocol, the recovery tiers, and the failure paths -- is Section 19.
## 11. API Emulator
The API Emulator turns Free Agent into an OpenAI-compatible HTTP endpoint. Any external application, script, or AI agent that can make HTTP requests can drive Free Agent as if it were OpenAI.
The purpose is integration. Free Agent already speaks to web models. The API Emulator lets other software speak to Free Agent.
### 11.1 What it exposes
When enabled, Free Agent listens on port 8080 and answers two endpoints:
- **POST /v1/chat/completions** -- OpenAI-compatible chat completion. Any tool that knows the OpenAI API can send a request and receive a response.
- **GET /v1/models** -- lists the single available model name, free_agent.
Requests flow through the same Universal Web LLM Interface that the GUI uses. That means they inherit browser automation, profile selection, and model failover exactly as if the user had typed the prompt. The emulator does not create a separate path; it is the same path with an HTTP front-end.
### 11.2 Why it matters
Three uses:
- **External orchestration.** A script, an agent, or another AI model can call Free Agent as a tool.
- **Interoperability.** Anything that already speaks the OpenAI API can talk to Free Agent without modification.
- **Local-only.** The emulator listens on localhost. It does not expose Free Agent to the network unless the user chooses to.
The emulator is what makes Free Agent callable by the rest of the user's toolchain.
### 11.3 Enabling it
The emulator is toggled from the Tools form with a single checkbox: "API Emulator (port 8080)". When checked, the status shows "Running on port 8080". When unchecked, the port is closed.
No keys, no configuration. The emulator is either on or off.
### 11.4 What a request looks like
A minimal chat completion request:
json
{
 "model": "free_agent",
 "messages": [
  {"role": "user", "content": "What is the capital of France?"}
 ]
}
Supported roles: system, user, assistant.
The response follows the OpenAI chat completion format: id, object, created, model, choices with an index, message, and finish_reason.
### 11.5 Interaction with the rest of the system
Because requests pass through the same interface the GUI uses, the emulator inherits everything:
- **Memory.** Retrieved context is prepended exactly as for a typed prompt.
- **Triggers.** Trigger memories fire the same way.
- **Watchdog.** Watchdog rules apply to the response.
- **Model failover.** If the current model throttles or fails, the failover router switches.
- **Browser automation.** The request is dispatched through Universal Automate to the active web model.
The emulator does not create a separate model instance. It uses the running one.
### 11.6 What it does not do
The emulator does not provide streaming responses, function calling, or the full OpenAI surface. It provides the chat completion endpoint and the model list. That is enough for most integrations and keeps the implementation small.
For the full endpoint reference, the request and response schemas, a PowerShell test example, and troubleshooting, see Section 20.
## 12. Prompt Scripts
A prompt script is a multi-step workflow. Each step has an instruction, optional branching, and optional completion criteria. Scripts can be run, paused, resumed, and started from any step.
### Script structure
Each step contains:
- Id — unique step identifier
- Instruction — what the AI should do
- CompletionCriteria — optional success condition
- BranchCondition — optional branching rule
- NextStepId — the next step to run
- BranchStepId — an alternative step to run if the branch condition is met
- IsExecuted — whether this step has run
- LastResponse — the AI's last response for this step
### Where scripts live
C:/ai/prompt_scripts/
### Commands
See Layer 3 for prompt_script_load, prompt_script_save, prompt_script_run, and prompt_script_pause.
---
## 13. Browser and External APIs
- **Embedded browser** — WebView2 panel for research and quick lookups
- **External APIs** — api_request supports weather, news, calendar, email, and SMS
- **Browser automation** — opening URLs, extracting page content, and building follow-up prompts are all available as tools
---
## 14. Safety and Transparency
- No harmful execution — the system refuses to perform malicious actions
- No sensitive data harvesting — it does not request or store credentials
- Full logging — every action is written to C:/ai/logs/
- Human-readable scripts — automations are plain JSON, reviewable before execution
- Emergency stop — F12 aborts any running automation without treating it as an error
- Sandbox — dangerous file paths and commands are blocked
- Cooldowns and anti-loop protection — prevent recursive trigger behavior
### 14.1 The Python sandbox
Free Agent runs Python through a sandbox that checks every action before it happens. The sandbox is a static rule gate: it consults a JSON configuration file, decides whether an operation is allowed or blocked, and logs the decision. It is not an OS-level jail -- it is a filter between the request and the execution.
**Three checks.** Before any of three action types is executed, the sandbox is consulted:
- **Path check** (IsPathSafe) -- called before file reads, writes, or deletes.
- **Command check** (IsCommandSafe) -- called before any shell command.
- **Code check** (IsPythonCodeSafe) -- called before running Python.
Each check returns allowed or blocked, and writes a reason to the log. If the sandbox is disabled, every check returns allowed.
**Where the rules live.** The rules file is C:/ai/sandbox_rules.json. It is plain JSON and can be edited by hand. If the file does not exist, the sandbox writes it on first use with sensible defaults.
**Rules file format.** The file has five fields:
 DangerousPaths   -- directory prefixes that cannot be touched
 DangerousCommands -- substrings that must not appear in a shell command
 DangerousPatterns -- regular expressions that must not match Python code
 AllowedRootPaths  -- informational list of roots the user considers safe
 IsEnabled     -- master switch (true or false)
Example (JSON, hand-editable):
 {
  "DangerousPaths": ["C:\\Windows", "C:\\Program Files", "C:\\System"],
  "DangerousCommands": ["format", "diskpart", "shutdown"],
  "DangerousPatterns": ["os\\.system", "subprocess\\.call", "exec\\("],
  "AllowedRootPaths": ["C:\\ai", "C:\\Users\\<yourname>", "C:\\Temp"],
  "IsEnabled": true
 }
**Default protections.** When the file is created with defaults, the following are blocked:
- Paths: C:/Windows, C:/System, C:/Program Files, C:/Program Files (x86), C:/Users/All Users, C:/$Recycle.Bin, C:/System Volume Information
- Any bare drive root (C:\, D:\, and so on)
- System files: boot.ini, pagefile.sys, hiberfil.sys, swapfile.sys, Recycle Bin, System Volume Information
- Commands: format, del /f /s, rmdir /s, rd /s, erase, diskpart, shutdown
- Python patterns: os.system, subprocess.call, shutil.rmtree, __import__, exec(, eval(
**Comment and string exemption.** When the code check finds a dangerous command substring inside Python code, it first checks whether the match appears inside a comment (starting with #) or inside a string literal. If so, the match is allowed. This prevents false positives -- for example, a print statement that mentions the word format is not blocked.
**Reloading rules.** Editing sandbox_rules.json and calling ReloadRules re-reads the file at runtime. The UI can also update rules through UpdateRules, which saves the file and clears the cache. No restart is required.
**What the sandbox is not.** The sandbox does not do network filtering, does not enforce resource limits (CPU, memory, time), and is not a substitute for careful review of unfamiliar code. It is a first-line filter, not a security boundary.
---
# LAYER 3 — REFERENCE
## 15. Tool Commands (Flat JSON)
All tools use flat JSON. One tool call per turn. Use forward slashes in paths. Escape quotes and backslashes correctly.
### 15.1 File Operations
**Read a file:**
json
{"command":"file_read","path":"C:/ai/file.txt"}
**Write a file (overwrites by default):**
json
{"command":"file_write C:/ai/file.txt","line1":"Line 1","line2":"Line 2"}
Append instead of overwrite:
json
{"command":"file_write C:/ai/file.txt","overwrite":"false","line1":"Appended line"}
**Append to a large file:**
json
{"command":"file_write_append C:/ai/file.txt","line1":"More content"}
**List a directory:**
json
{"command":"dir","path":"C:/ai","pattern":"*.txt"}
### 15.2 Python Execution
**Run code inline:**
json
{"command":"python_run","line1":"import sys","line2":"print('hello')","timeoutSeconds":"5"}
**Run a script file:**
json
{"command":"python_run_file","path":"C:/ai/script.py","timeoutSeconds":"30"}
### 15.3 Memory Tools
**Remember with optional trigger keywords:**
json
{"command":"remember","content":"User prefers Python","importance":"7","trigger_keywords":"python,code,programming"}
Trigger keywords **must be comma-separated**. Space-separated values become one phrase and will not trigger separately.
**Store with explicit type:**
json
{"command":"memorystore","content":"Backup routine","type":"Procedural","importance":"7"}
Types: Fact, Preference, Episodic, Procedural.
**Recall by semantic search:**
json
{"command":"memoryrecall","query":"python preferences","top_k":"5"}
**Statistics:**
json
{"command":"memorystats"}
**List System memories:**
json
{"command":"list_system_memories"}
**Suspend / resume / status:**
json
{"command":"memory_suspend"}
{"command":"memory_resume"}
{"command":"memory_status"}
### 15.4 Schedule Tools
**Add a scheduled event:**
json
{"command":"schedule_add","name":"Daily Backup","repeat_type":"Daily","action_type":"CustomPrompt","prompt":"Run backup","next_trigger":"2026-09-23T09:00:00","is_active":"true"}
Repeat types: None, Minute, Minutes5, Minutes10, Minutes15, Minutes30, Hourly, Hours2, Hours4, Hours6, Hours8, Hours12, Daily, Weekly, Monthly.
Action types: CustomPrompt, RunScript, LaunchProgram.
**List / update / delete / pause / resume:**
json
{"command":"schedule_list"}
{"command":"schedule_update","id":"event-id","name":"New Name"}
{"command":"schedule_delete","id":"event-id"}
{"command":"schedule_pause","id":"event-id"}
{"command":"schedule_resume","id":"event-id"}
### 15.5 Automation Tools
Format: a single pipe-separated string. First part is the tool name; subsequent parts are key=value pairs.
json
{"command":"automation_run","action":"tool_name | param1=value1 | param2=value2"}
**Available automation actions:**
| Action | Purpose |
|--------|---------|
| winlist | List visible windows |
| desktop_scanner_ultra | Deepest UI scan (native + OCR, ~387 items) |
| desktop_scanner_fast | Faster scan (~246 items) |
| desktop_scanner_vision | Vision-based scan using minicpm-v:8b — page description and popup detection |
| desk_executor_seq | Execute a sequence of UI actions from a JSON file |
| capture_window | Screenshot a window |
| web_model_chat | Send a prompt to a web AI model via Universal Automate |
Registered tools are read from C:/ai/aispace/automation_inventory.json.
The system prompt lists a short subset (winlist, desktop_scanner_ultra, desktop_scanner_fast, desktop_scanner_vision, desk_executor_seq, capture_window, web_model_chat).
The full registered set also includes: desktop_scanner, desktop_scanner_recursive, desktop_executor, detect_icon, run_sequence, ui_inspect, ui_action, plan_tool, gemini_upload, gmail_app_pass, ocr_menu_scan, text_to_audio, speechma_list_voice, speechma_set_voice, speechma_scroll_top.
**Examples:**
json
{"command":"automation_run","action":"winlist"}
{"command":"automation_run","action":"desktop_scanner_fast | prompt=msedge.exe,Facebook | file_path=C:/ai/scan.json"}
{"command":"automation_run","action":"desk_executor_seq | prompt=msedge.exe C:/ai/sequence.json C:/ai/scan.json"}
{"command":"automation_run","action":"web_model_chat | process=msedge.exe | title=Gemini | prompt=Hello"}
### 15.6 Prompt Script Tools
json
{"command":"prompt_script_load","path":"C:/ai/prompt_scripts/script.json"}
{"command":"prompt_script_save","path":"C:/ai/prompt_scripts/script.json"}
{"command":"prompt_script_run"}
{"command":"prompt_script_run","start_from_step":"step3"}
{"command":"prompt_script_pause"}
### 15.7 Browser and API
json
{"command":"browser_open","url":"https://example.com"}
{"command":"api_request","service":"weather","query":"London"}
Supported services: weather, news, calendar, email, sms.
### 15.8 Watchdog (via config file)
The Watchdog is not driven by a tool command. It is configured through the Watchdog Editor UI, saved to C:/ai/masterwatch_triggers.json, and runs continuously in the background. See Section 16 for the field reference.
### 15.9 System Utilities
json
{"command":"stop"}
{"command":"history","count":"10"}
stop halts the current model, pipeline, or script. history retrieves recent conversation events.
---
## 16. Watchdog Configuration Reference
Each trigger in C:/ai/masterwatch_triggers.json has the following fields:
| Field | Purpose |
|-------|---------|
| Name | Unique identifier for the trigger |
| Pattern | Literal text to match (case-insensitive, word-boundary) |
| Source | Enforced as Chat |
| ActionType | One of: Prompt, Execute, Reasoning, Alert, SwitchModel |
| PromptText / Execute Target | Prompt text, program path, or model target (Next or name) |
| ExecuteArgs | Args for Execute, or custom handoff prompt for SwitchModel |
| ReasoningInstructions | Reasoning prompt, or direct handoff prompt to main AI after Execute |
| Timeout | Max time for external program (default 300000 ms = 5 min) |
| CooldownMs | Minimum time before the same trigger can fire again |
| Severity | 1 (Low), 2 (Medium), 3 (High) |
| IsActive | Whether the trigger is enabled |
### Choosing an action
| Situation | Recommended Action |
|-----------|--------------------|
| AI can fix it itself | Prompt |
| Windows/program/script action is required | Execute |
| Recovery depends on circumstances | Reasoning |
| User should decide | Alert |
| Current model should be replaced | Switch Model |
### Recovery variables
- {line} — full triggering line
- {trigger} — trigger name
Both expand in program path, args, handoff prompt, Prompt text, and Alert message.
---
## 17. Memory System Reference
### 17.1 Memory types
| Type | Purpose |
|------|---------|
| Fact | General information |
| Preference | User preferences and settings |
| Episodic | Events, conversations, tool executions |
| Procedural | Step-by-step instructions, code, methods |
A fifth type, **System**, is a first-class enum member. System memories are protected from auto-cleanup, exempt from decay, and inherited by every new memory database the system creates. They are the recommended home for personas, SOPs, and any memory the user wants the system to treat as authoritative. They carry the system's identity forward as it grows and learns: personas, procedures, and critical knowledge survive database changes, session changes, and user changes, unless the user explicitly deletes them.
### 17.2 Retrieval modes
| Mode | Description |
|------|-------------|
| Fast | Embeddings only (~200 ms) |
| Smart | Embeddings plus coprocessor (~2 s) |
| Auto | Adaptive based on query |
| DeepResearch | Includes web search |
| CoprocessorOnly | Coprocessor drives retrieval |
### 17.3 Trigger match modes
| Mode | Behavior |
|------|----------|
| Exact | Input must equal the trigger phrase exactly (case-insensitive) |
| Phrase | Input contains any comma-separated phrase (default) |
| Word | Each phrase evaluated; threshold percent of words must match |
| Semantic | Embedding similarity above threshold |
Trigger keywords **must be comma-separated** for these modes to work as expected.
Every trigger match is then confirmed by a second stage: semantic similarity between the query embedding and the memory's trigger_embedding must be at least 0.65. If it falls below, the match is rejected even if the mode matched. This two-stage gate is why trigger false positives are low.
### 17.4 Settings file (C:/ai/memory_settings.json)
json
{
 "confidence_threshold": 0.65,
 "trigger_match_mode": "Phrase",
 "content_search_enabled": true,
 "max_results": 5,
 "min_importance": 5,
 "coprocessor_model": "qwen2.5:1.5b",
 "research_model": "qwen2.5:7b",
 "embedding_model": "nomic-embed-text:latest"
}
### 17.5 Dual embeddings
Each memory stores two vectors:
- **embedding** — computed from the memory content, used for semantic search.
- **trigger_embedding** — computed from the trigger keywords, used for trigger matching.
This is the design that makes triggers reliable: content can be verbose without diluting the trigger match, and trigger keywords can be short without losing content meaning.
### 17.6 Decay, linking, retrieval
- **Decay:** importance decays at 1% per day, floored at 50% retention. Accessing a memory resets its timer.
- **Linking:** memories with similarity > 0.75 are linked (related); > 0.90 are linked as very similar. Links are bidirectional. Max 3 linked memories per new memory.
- **Retrieval threshold:** ~0.7 similarity.
- **Retrieval chain:** direct trigger match → embedding search → keyword fallback. Trigger memories are excluded from embedding search to prevent duplicates.
### 17.7 Database schema (core tables)
- memories — id, content, type, importance, metadata, embedding, trigger_keywords, trigger_embedding, created_at, last_accessed, decay_rate
- memory_links — source_id, target_id, link_type, strength
- tool_examples — tool_name, tool_call, result, success
### 17.8 Debug markers
Watch the log for: [AUTO-LINK-DEBUG], [TRIGGER-STORE], [TRIGGER-MATCH], [TRIGGER-MATCHED], [EMBEDDING-RETRIEVE], [MEMORY-LINK].
### 17.9 Retrieval orchestration
The MemoryOrchestrator is the decision layer between the pipeline and the memory store. It decides which memories reach the model, in what order, and what to do when the smart path fails. The retrieval pipeline runs in a fixed order:
1. **Suspension check.** If memory is suspended, return empty immediately.
2. **Trigger match.** RetrieveMemoriesByTriggerAsync runs first. If any trigger matches, its memories are returned immediately and the rest of the pipeline is skipped. Trigger matches outrank embedding matches.
3. **Cache check.** Results are cached for five minutes, keyed by the exact user prompt. A cache hit returns without re-querying.
4. **Mode selection.** MainForm.DetermineRetrievalMode chooses the retrieval mode for this query: Fast, Smart, Auto, DeepResearch, or CoprocessorOnly.
5. **Embedding search.** RetrieveMemoriesByEmbeddingAsync returns up to ten candidate memories at a similarity threshold of 0.8 (0.6 for CoprocessorOnly). If none match, a fallback retrieval by importance runs.
6. **Junk filter.** FilterJunkMemories removes code fragments, log markers, tool output, short entries, and duplicates. Trigger memories are exempt from the size cap and are never filtered.
7. **Relevance filter.** ApplyRelevanceFilter keeps memories that score at least 0.6 on keyword overlap AND match one of seven topic clusters (name, birthday, color, email, phone, address, puzzle) OR have importance of 8 or higher. A recency boost is applied: max(0.5, 1.0 - days_since_creation / 30).
8. **Coprocessor reranking.** In Smart and CoprocessorOnly modes, the coprocessor is asked to rank the surviving memories by ID. The ranked IDs are parsed back and used to reorder the memories before context is built. This is a reranking step, not just analysis.
9. **Context assembly.** The top three memories are formatted into a context block and returned.
### 17.10 Fallback chain
Every layer degrades gracefully:
- Coprocessor failure -> falls back to Fast mode.
- Fast mode empty -> falls back to importance-based retrieval.
- Relevance filter trims to nothing -> returns empty context.
- Any uncaught error -> logged, empty returned, pipeline continues.
No error in the memory layer stops the pipeline.
### 17.11 Design decisions worth noting
- **Trigger beats embeddings.** A user-tagged memory is authoritative and outranks semantic similarity. This is how SOPs and identity triggers work reliably.
- **Trigger memories are never truncated.** When a trigger matches, the complete memory is injected, no matter its size. Trigger memories are also exempt from the junk filter.
- **The coprocessor is a reranker, not just a filter.** It returns a ranked order of memory IDs, and the orchestrator uses that order.
- **Five-minute cache.** Repeated identical prompts within five minutes are answered from cache. The cache clears when the memory system is swapped.
### 17.12 Storage and matching engine
LLMMemorySystem.vb is the storage and matching engine. It owns the SQLite connection and implements every retrieval and storage method the orchestrator calls.
**Database.** SQLite file at C:/ai/agent_memory.db. Three core tables: memories, memory_links, and tool_examples. Three indexes are created for speed: trigger_keywords, type, and importance.
**Embeddings.** Generated by Ollama using nomic-embed-text:latest at http://localhost:11434/api/embeddings. Each request retries with timeouts of 30, 45, then 60 seconds. Successful embeddings are cached for ten minutes.
**Two embeddings per memory.** The content has an embedding; the trigger keywords (if any) have a separate trigger_embedding. Both are generated at store time.
**Two-stage trigger gate.** Stage one is the mode-based match (Exact, Phrase, Word, or Semantic). Stage two is semantic confirmation: the query embedding must have at least 0.65 cosine similarity with the memory's trigger_embedding, or the match is rejected. Final score is 60 percent stage one, 40 percent semantic similarity.
**Trigger and embedding pools are separate.** Trigger-tagged memories are excluded from embedding search entirely (WHERE trigger_keywords IS NULL OR trigger_keywords = ''). The two retrieval paths never compete.
**Trigger-first inside the engine.** RetrieveMemoriesByEmbeddingAsync calls RetrieveMemoriesByTriggerAsync first and returns trigger matches immediately if any exist. Even a direct caller of the embedding path gets trigger priority.
**Decay is per-memory.** Each memory has a decay_rate column (default 0.01 = 1 percent per day). Effective importance = importance * max(0.5, 1 - decay_rate * days_since_last_access). Accessing a memory resets the timer.
**Auto-linking at store time.** Every new memory finds its top-three related memories above 0.8 similarity and creates bidirectional links. Similarity above 0.9 creates very_similar links; below 0.9 creates related links.
**Tool examples table.** The engine stores worked examples of tool calls, keyed by tool name. Only valid nested JSON tool calls are stored. RetrieveToolExamples returns the most recent examples for a given tool, which is how the system learns from its own past tool use.
**Inheritance.** When a new memory database is created, the System memories are inherited by it. The new database is born with them already in place. Personas, SOPs, and critical knowledge carry forward automatically; only the user can remove a System memory. This is what makes the system's learning cumulative rather than per-install.
**Migration.** MigrateExistingMemoriesAsync backfills embeddings for memories that predate the embedding feature, with a 100 ms delay between calls to avoid overloading Ollama.
### 17.13 System memory protection and inheritance
The MemorySystemManager is responsible for keeping System memories alive across the lifetime of the system. It has two jobs: protecting critical content as System, and carrying System memories forward when a new database is created.
**Pattern-based promotion.** ProtectCriticalMemories scans existing memories for content that matches critical-content markers (for example, SOP: and GUIDE: headers, and the phrase definitive guide). Anything matching is promoted to type = System. This is how important knowledge gets protected from cleanup, even if it was originally stored as a different type. The promotion pass is pattern-based and runs at the user initiative, not automatically.
**Inheritance flow.** When the user creates a new memory database, System memories are inherited by it in four steps:
1. Export. All System memories are read from the current database, preserving content, importance, metadata, trigger keywords, decay rate, and creation date.
2. Create. The new database is created with the same schema (memories, settings).
3. Import. All System memories are written into the new database in a single transaction, with type forced to System.
4. Re-embed. Embeddings and trigger embeddings are regenerated in the new database. The content is preserved; the vectors are rebuilt.
After import, the new database contains the full set of System memories from the previous one. A count check (GetSystemMemoryCount) confirms the number matches.
**What inheritance means in practice.** The knowledge the system has accumulated -- personas, SOPs, critical procedures -- survives the creation of a new database. It does not need to be re-created or re-taught. Only the user can remove a System memory; inheritance itself never drops one.
**Listing System memories.** ListSystemMemories returns a formatted view of every System memory, showing ID, importance, creation date, a truncated content preview (300 characters), and any associated trigger keywords. This is the read-side view of what the system considers authoritative.
---
## 18. Persona System Reference
### 18.1 What it is
The Persona System assigns a persistent role to a web model and keeps that role from fading during long conversations. It periodically checks that the model still remembers its role, and re-injects the full persona when drift is detected.
### 18.2 The three files per profile
| File | Purpose |
|------|---------|
| C:/ai/web_persona_<slug>.json | Config: persona name, turn interval, questions, expected answers, recovery file path, enabled flag |
| C:/ai/web_persona_<slug>_state.json | Runtime state: turn count, challenge count, failure count, state name |
| C:/ai/persona_recovery_<slug>.txt | The full persona text — what gets injected on recovery |
Optional companion: C:/ai/persona_activation_<slug>.txt — the three activation commands, to append to a personality file that should activate this persona.
### 18.3 Activation (three commands)
json
{"command":"remindme","action":"register","profile":"<slug>","persona":"<Display Name>","turns":"10","questions":["..."],"expected_answers":["..."],"recovery_file":"C:/ai/persona_recovery_<slug>.txt"}
{"command":"remindme","action":"select","profile":"<slug>"}
{"command":"remindme","action":"status","profile":"<slug>"}
After the third returns ActiveProfile=<slug> and State=Counting, the persona is live.
### 18.4 Validation
The system periodically prepends a [PERSONA CHECK] question. The model must respond with a marker in this form:
[[PERSONA:<check-id>:<answer>]]
The answer is compared **exactly** (case-sensitive, no extra spaces) against the registered expected answer.
Pass → counter resets, counting continues.
Fail → state becomes RecoveryPending; the full recovery file is injected on the next eligible send.
### 18.5 Answer authoring rules
Exact-match validation makes long answers fail. Follow these rules:
- Keep answers to 1–3 tokens
- All caps for workflow sequences (DIRECT, VERIFIED, UNDERSTAND INSPECT PLAN INSTALL)
- No punctuation except leading hyphen or numeric string
- No articles, no trailing periods, no brackets
- For multi-personality personas, **canonicalize the questions** so they force short answers (e.g., "Two modes Howard combines, two words:" → COACH CREATIVE). Long free-form questions invite sentences that will never match.
### 18.6 Commands
| Command | Purpose |
|---------|---------|
| remindme register | Register or overwrite a profile (resets counters) |
| remindme select | Change the active profile |
| remindme status | Show current state and counters |
| remindme reset_counter | Reset turn count, failures, and pending checks without unregistering |
| remindme disable | Disable monitoring for a profile without unregistering it |
There is no list or delete command. Listing profiles is done with dir C:/ai web_persona_*.json, and deletion is by removing the three files.
### 18.7 States
| State | Meaning |
|-------|---------|
| Disabled | Profile is not enabled |
| Counting | Monitoring; counter climbing |
| AwaitingCheck | A check was prepared, awaiting response |
| RecoveryPending | Recovery queued for next eligible send |
| AwaitingRecoveryVerification | Recovery delivered; awaiting verification |
| Suspended | Recovery file invalid; checks halted |
### 18.8 Troubleshooting
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| Validation always fails | Expected answers don't match what the model naturally produces | Re-register with shorter, canonical answers, and rewrite the questions to force the canonical form |
| FailedRecoveryChecks climbing | Same as above — recovery keeps getting triggered | Re-register with cleaner answers |
| State stuck in RecoveryPending | Repeated validation failure | Run remindme reset_counter, then re-register |
| State=Suspended | Recovery file missing or invalid | Verify the recovery file exists, is UTF-8, is under 256 KB, and contains no [[ or ]] |
### 18.9 Interval guidance
- 10–20 exchanges: active monitoring
- 50+: light monitoring
The default is 10.
### 18.10 The delivery-channel pattern
The Persona trigger fires reliably on every identity challenge. Anything attached to that trigger rides along — no code changes required. This makes the trigger a delivery channel for three kinds of payload:
| Payload | Where it lives | Lifetime | Who writes |
|---------|----------------|----------|------------|
| Inline reminder | A memory entry keyed to the identity trigger | Session or until changed | User or model |
| Baked-in file load | Inside the persona file itself | Permanent until edited | User |
| Dynamic memory file | A named file (e.g., C:/ai/persona_memory/edit.txt) read on wake and writable by either side | Until edited | User and model |
Full pattern documented at C:/ai/persona_memory/README.md and in C:/ai/updated_docs/persona reminder instructions.txt §22.
---
## 19. Universal Automate -- Full Reference
Universal Automate (universal_automate.py) is the engine that drives every web-based AI model. It is a single-shot runtime: invoked once per prompt with a target (process,title) and a prompt, it finds the window, focuses it, enters the prompt, submits, waits for completion, copies the response, prints it to stdout, and exits. It does not run as a daemon. Every invocation is independent.
At roughly 3,000 lines, it is by far the largest subsystem in Free Agent. Its design is organized around what can go wrong and what the engine does about it.
### 19.1 The interaction flow
START
 -> Parse target (process,title)
 -> Load or create profile
 -> Load prompt (text or file)
 -> Find target window (psutil + win32gui)
 -> Focus window (restore, maximize, SetForegroundWindow,
    thread-input attachment, alt-key fallback)
 -> Safe click (activate page)
 -> Find submit icon
 -> Calculate input location from submit offset
 -> Click input
 -> Clear existing text (End, Shift+Home, Backspace)
 -> Paste prompt via clipboard
 -> Submit (see 19.6)
 -> Safe click
 -> Wait for completion (see 19.7)
 -> Scroll to bottom
 -> Find copy icon (bottom-up, two confidence levels)
 -> Click copy until clipboard changes (up to 3 attempts)
 -> Print response to stdout
### 19.2 The profile
Profiles live at C:/ai/profiles/<ModelName>.json. If a profile does not exist, the engine creates a default one on first use.
The default profile uses:
- safe_click at (280, 887) -- activation click
- input_box_offset (-150, -60) -- input position relative to submit icon
- fallback_click at (685, 930) -- input position if submit icon not found
- submit_method: both
- completion_detection: searching_disappears
- search_region_size: 500, search_region_offset: 250
- searching_confidence: 0.85, searching_grayscale: false
The example icons in a fresh profile point at C:/ai/icons/model studio submit.png and similar. These are placeholders and should be replaced with real icons for the target model.
### 19.3 Window finding
The engine locates the target by process name and window title. It walks every running process with psutil and collects PIDs whose name matches the process argument. It then enumerates every visible window and picks the first whose PID is in the list and whose title contains the title argument (case-insensitive).
If no window is found, the engine takes a screenshot, logs it, prints a WATCHDOG TAKEOVER marker on stdout, and exits.
### 19.4 Window focusing
Focusing is more than SetForegroundWindow. The sequence:
1. If minimized, restore (SW_RESTORE).
2. Maximize (SW_MAXIMIZE).
3. Try SetForegroundWindow.
4. If it fails, attach thread input between the calling thread and the target thread, try again, detach.
5. If still not foreground, press Alt briefly (releases any modal keyboard grab), then SetForegroundWindow again.
6. Verify.
If verification still fails, the engine takes a screenshot, logs FOCUS_FAILED, prints the marker, and exits.
### 19.5 Prompt entry
Two paths.
**Standard models:**
1. Click the input location.
2. Press End, then Shift+Home, then Backspace (clears the current line).
3. Set the clipboard to the prompt.
4. Paste with Ctrl+V.
5. Log and proceed.
**Capriole (special):**
Capriole's input box is temperamental. It gets a three-attempt entry loop with verification:
1. Click the input.
2. Clear the current line.
3. Paste the prompt.
4. Read the input back via Ctrl+A / Ctrl+C and the clipboard.
5. If the input box contains the prompt, break.
6. If not, re-search for the submit icon to recalculate the input position, or fall back to descending Y offsets (-20 per attempt).
7. Retry, up to three attempts.
If all three attempts fail, the engine takes a screenshot, logs PROMPT_NOT_ENTERED, and prints the marker.
### 19.6 Submit
Submit behavior depends on the profile's submit_method (enter, submit, or both).
Before submit, the engine checks for a WAIT TO SUBMIT icon: <model>_wait_to_submit.png.
- **If the icon file exists:** poll the screen up to 10 seconds. Submit the moment it appears. If it never appears, submit anyway.
- **If the icon file does not exist:** wait 2 seconds (original settle), then submit.
This replaces the fixed 2-second settle with a model-specific, event-driven one when the user has provided the icon.
**DeepSeek special case:** DeepSeek has a dark submit icon at C:/ai/icons/deepseek_submit_dark.png. If that file exists, the engine polls for it for up to 8 seconds (confidence 0.7, grayscale) and clicks it if found. If not found, it falls back to the profile submit icon (three attempts). If that fails too, it falls back to Enter.
### 19.7 Completion detection
The engine supports two methods, selected by the profile:
- **done_icon** -- look for a done icon; completion when it appears.
- **searching_disappears** -- look for a searching icon; completion when it appears and then disappears. If the icon never appears, no completion is signaled.
The searching_disappears method uses a boolean searching_seen. Completion is only signaled after the searching icon has actually been seen. This prevents false "done" signals on fast responses where the icon never renders.
### 19.8 The three-zone search model
Every loop iteration does not scan the full screen. Instead:
1. **Dynamic region** -- 500x500 pixels around the submit icon, offset by search_region_offset. Configured per profile.
2. **One-time full-screen check** -- exactly one full-screen LOOK ANYTHING check at the start, before the loop begins.
3. **Scoped checks thereafter** -- LOOK ANYTHING and searching-icon detection happen only in the dynamic region.
Fallback: if the submit icon cannot be found, the engine uses the center half of the window as the search region.
### 19.9 The loop
Once the prompt is submitted, the engine enters a wait loop (timeout 300 seconds by default). Every iteration:
1. Check the window is foreground; refocus if not.
2. Safe click (activate the page).
3. Scroll to bottom.
4. Check for LOOK ANYTHING (in region).
5. Check for LOOK CONTINUE.
6. Check the completion condition.
7. Sleep 1 second.
The safe click and scroll keep the page responsive over long waits. Without them, the browser loses focus and the searching icon never renders.
### 19.10 The three recovery tiers
The engine recognizes three categories of recovery icon, all loaded from C:/ai/icons by a numbered filename convention.
**LOOK CONTINUE** -- modelname_look_continue_1.png, _2.png, _3.png...
A condition the engine knows how to handle. If found:
1. Click the icon.
2. Wait 1 second.
3. Reset the completion timer.
4. Continue waiting.
Continue can fire multiple times. Every occurrence resets the timer. The engine does not stop.
**LOOK ANYTHING** -- modelname_look_anything_1.png, _2.png, _3.png...
A condition the engine does not know how to handle. If found:
1. Capture the exact filename and number.
2. Capture the center X and Y coordinates.
3. Take a screenshot.
4. Print a [LOOK_ANYTHING_FOUND] marker to stdout.
5. Stop and hand control to the watchdog.
The engine does not guess what to do about an unknown condition.
**WAIT TO SUBMIT** -- modelname_wait_to_submit.png (single, not numbered)
An optional pre-submit signal. Poll for the icon up to 10 seconds and submit the moment it appears. If never found, submit anyway. If the file does not exist, use the standard 2-second settle.
### 19.11 Numbering rules
Recovery files are numbered sequentially starting at 1. The engine stops checking at the first missing number. If _1 and _2 exist but _3 does not, only _1 and _2 are checked. This keeps the recovery sequence deterministic.
### 19.12 Model name normalization
The window title is normalized into a filename-safe string before use:
- Lowercased.
- Alphanumeric characters and dash and underscore kept.
- Spaces, dots, slashes, backslashes become underscores.
- Repeated underscores collapsed.
- Leading and trailing underscores stripped.
Examples: DeepSeek becomes deepseek. Model Studio becomes model_studio.
This normalized name is the prefix for every recovery icon and every error screenshot.
### 19.13 Error screenshots
Screenshots are saved to C:/ai/logs/errors/ with stable filenames. The same error overwrites its previous screenshot instead of accumulating. The master log preserves the historical date/time record.
Filename format: <model>_<ERRORTYPE>.png.
Error types: WINDOW_NOT_FOUND, FOCUS_FAILED, PROMPT_NOT_ENTERED (Capriole only), TIMEOUT, NO_COPY_ICON_PATH, COPY_ICON_NOT_FOUND, COPY_FAILED, UNEXPECTED_ERROR.
LOOK ANYTHING events additionally produce screenshots named after the recovery file itself.
### 19.14 The master log
Universal Automate writes to a single append-only log: C:/ai/logs/universal_automate.log. Every entry includes date, time, PID, and message. The PID allows multiple concurrent instances to be distinguished.
The log is the historical record. Screenshots are the current evidence. Together they cover both.
### 19.15 The stdout protocol
The engine communicates with the watchdog entirely through stdout markers. The watchdog reads the last meaningful line.
| Marker | Meaning |
|--------|---------|
| (plain text) | Success -- this is the model's response |
| [WATCHDOG TAKEOVER] WINDOW_NOT_FOUND SCREENSHOT=... | Target window not found |
| [WATCHDOG TAKEOVER] FOCUS_FAILED SCREENSHOT=... | Window found but could not be focused |
| [WATCHDOG TAKEOVER] PROMPT_NOT_ENTERED SCREENSHOT=... | Capriole prompt entry failed after 3 attempts |
| [WATCHDOG TAKEOVER] TIMEOUT SCREENSHOT=... | 300-second timeout expired |
| [WATCHDOG TAKEOVER] NO_COPY_ICON_PATH SCREENSHOT=... | Profile has no copy icon path |
| [WATCHDOG TAKEOVER] COPY_ICON_NOT_FOUND SCREENSHOT=... | Copy icon could not be located |
| [WATCHDOG TAKEOVER] COPY_FAILED SCREENSHOT=... | Copy clicked but clipboard did not change |
| [WATCHDOG TAKEOVER] UNEXPECTED_ERROR:... SCREENSHOT=... | Unhandled exception |
| [LOOK_ANYTHING_FOUND] filename NUMBER=n X=x Y=y SCREENSHOT=... | Recovery condition found |
| {"name":"stop","parameters":{}} | User pressed F12 |
These markers are the entire API surface of the engine. Everything else is internal.
### 19.16 F12 emergency stop
A global keyboard listener via pynput watches for F12. On press:
1. Log the event.
2. Print the stop command on stdout.
3. Set the stop flag.
4. Exit.
F12 is treated as a user command, not an automation error. The watchdog sees the stop marker and understands.
### 19.17 Copy and verification
The engine does not assume copy succeeds. The copy step:
1. Save the current clipboard contents.
2. Locate the copy icon bottom-up (highest on screen first, since the newest response is at the bottom). Try confidence 0.6, then 0.5.
3. Click the copy icon.
4. Wait 0.8 seconds.
5. Re-read the clipboard.
6. If it changed, success.
7. If not, retry up to 3 times.
8. If still unchanged, log COPY_FAILED, screenshot, marker, exit.
The rule: click does not equal success. Only a changed clipboard means the copy worked.
### 19.18 Adding a new model
1. Create a profile at C:/ai/profiles/<ModelName>.json (or let the engine create the default and edit it).
2. Capture the four basic icons: submit, searching, done, copy.
3. Optionally capture recovery icons: <model>_look_continue_N.png, <model>_look_anything_N.png, <model>_wait_to_submit.png.
4. Test each stage.
Most models require configuration only. Code changes are not needed for new models.
### 19.19 What Universal Automate does not do
It does not diagnose unknown conditions.
It does not decide whether to retry after a timeout.
It does not choose a different model.
It does not ask the user.
It does not guess at what an unfamiliar screen means.
Those decisions belong to the watchdog. Universal Automate automates what it understands, documents what it does not, and hands control back.
## 20. API Emulator — Full Reference
Free Agent can act as an OpenAI-compatible HTTP endpoint. External applications and other AI agents can drive Free Agent through this interface.
### 20.1 Endpoints
**Chat completions:**
POST http://localhost:8080/v1/chat/completions
Request:
json
{
 "model": "free_agent",
 "messages": [
  {"role": "user", "content": "What is the capital of France?"}
 ]
}
Supported roles: system, user, assistant.
Response:
json
{
 "id": "chatcmpl-freeagent-xxxxxxxx",
 "object": "chat.completion",
 "created": 1234567890,
 "model": "free_agent",
 "choices": [
  {
   "index": 0,
   "message": {"role": "assistant", "content": "Paris."},
   "finish_reason": "stop"
  }
 ]
}
**List models:**
GET http://localhost:8080/v1/models
### 20.2 Enabling it
1. Open the Tools form
2. Check "API Emulator (port 8080)"
3. Status shows "Running on port 8080"
### 20.3 Architecture
External App
 -> POST /v1/chat/completions
 -> API Emulator (port 8080)
 -> Request Handler (parses messages)
 -> Universal Web LLM Interface
 -> Browser Automation (Gemini, Qwen, Comet, Ollama)
 -> AI Response
 -> API Response Formatter
 -> External App
The emulator does not create new model instances — it uses the existing interface, inheriting browser automation and model failover.
### 20.4 Quick test (PowerShell)
powershell
$body = @{
  model = "free_agent"
  messages = @(
    @{ role = "user"; content = "What is the capital of France?" }
  )
} | ConvertTo-Json -Depth 3
Invoke-RestMethod -Uri "http://localhost:8080/v1/chat/completions" -Method Post -Body $body -ContentType "application/json"
### 20.5 Troubleshooting
- **Connection refused** — Free Agent is not running, or the emulator is not toggled on, or port 8080 is in use
- **Empty response** — the browser may need focus; check the Raw Log tab; make sure the model is ready
- **Model not found** — the only recognized model name is free_agent
---
## 21. Voice and Phone Reference
This section is the reference detail for the Voice and Phone features introduced in Section 7. It documents the mechanisms: the audio pipeline, the shared-microphone arbitration, the wake and confirmation grammars, the phone protocol, and the marker parser.
### 21.1 Audio capture
Voice is captured with NAudio's WaveInEvent at 16 kHz, 16-bit, mono -- the native format Whisper expects. No resampling, no conversion. Each 500 ms block of samples is scanned for peak amplitude. If any sample exceeds the amplitude threshold (default 6000), the buffer is considered non-silent.
Audio is written to a timestamped WAV file in C:/ai/recordings/. Files are named recording_YYYYMMDD_HHMMSS.wav.
### 21.2 Transcription
Transcription runs whisper-cli.exe with:
- Model: C:/ai/models/ggml-base.bin
- Arguments: -m <model> -f <wav> --no-timestamps -l en
- Output parsed: lines containing -->, whisper, load, system_info, or processing are discarded. Only the actual transcription text is kept.
If the CLI cannot be found or the model file is missing, voice initialization fails and the voice button is disabled.
### 21.3 The three microphones
Three consumers want the microphone:
1. **Wake word recognizer** -- a SpeechRecognitionEngine with the grammar Hey Agent / Hello Agent / Wake Up.
2. **Recording session** -- the NAudio WaveInEvent capturing speech.
3. **Confirmation recognizer** -- a second SpeechRecognitionEngine with the grammar Send / Sending / Enter / Cancel that / Delete / Delete that / Edit.
Arbitration rules:
- Only one SpeechRecognitionEngine may run at a time. When the confirmation listener starts, the wake listener is stopped first.
- When confirmation is complete, the wake listener is restarted if wake mode is active.
- During recording, the wake listener is not stopped -- the wake word will not fire again because the recording session has the mic, but the listener is not explicitly torn down.
### 21.4 Wake mode with auto-stop
Wake mode adds two timers:
- **Grace delay**: 1500 ms after recording starts. Silence detection is disabled during this window so the tail of the wake phrase does not trigger an early stop.
- **Silence timer**: 500 ms tick. Each tick checks the running noAudioCount. When it reaches 4 (2 seconds of consecutive silent windows), recording auto-stops and transcription begins.
Manual mode does not auto-stop. The user clicks the button again.
### 21.5 Confirmation flow
After transcription completes for a source of "main", the text is placed in the main input box and the confirmation listener starts. Voice commands are matched against the confirmation grammar, case-insensitive:
- **send**, **sending**, **enter** -- sets input to the transcription and fires the send button.
- **cancel that**, **delete**, **delete that** -- clears input and logs "Cancelled."
- **edit** -- focuses input with the cursor at end, tells user to edit and click Send.
After any of the three, the confirmation listener stops, and if wake mode is active, the wake listener restarts.
Transcriptions from other sources (such as the Prompt Management System) do not go through confirmation. They are delivered via the TranscriptionCompleted event, filtered by source string.
### 21.6 Event contract
WhisperVoiceInput raises three events:
- TranscriptionCompleted(text, source) -- fires when a non-main transcription completes.
- RecordingStarted() -- fires when recording starts.
- RecordingStopped() -- fires when recording stops.
WakeWordDetected(word) is raised when the wake word is detected.
### 21.7 Supporting methods
- **PlaybackRecording** -- plays the most recent recording through System.Media.SoundPlayer.
- **TestMicrophone** -- records 3 seconds, then plays back. Used for setup diagnostics.
- **Cleanup** -- stops all three listeners, disposes all timers, releases the microphone.
### 21.8 Phone protocol
The Phone Interface (PhoneInterfaceForm.vb) runs two transports:
**UDP discovery on port 8766.** The phone broadcasts the literal string FREE_AGENT_DISCOVER. Free Agent responds with JSON containing the desktop's IP address, WebSocket port (8765), and version string.
**WebSocket on port 8765.** HTTP listener based. Accepts messages up to 64 KB.
### 21.9 Message types
Client-to-server:
| Type | Payload | Purpose |
|------|---------|---------|
| hello | capabilities list | Capability handshake |
| user_message | message, session_id | A prompt from the phone |
| ping | none | Keepalive |
| disconnect | none | Graceful close |
Server-to-client:
| Type | Payload | Purpose |
|------|---------|---------|
| connected | agent, version, session_id, capabilities | Handshake response |
| agent_message | message, speak flag | A response to display or speak |
| status | status, session_id | Processing / complete / error |
| pong | none | Ping response |
### 21.10 Message fragmentation
WebSocket frames can arrive fragmented. Each client has a StringBuilder buffer. Fragments are appended until EndOfMessage is true, then the complete message is processed. Buffers are cleared per client after each complete message.
### 21.11 The same-pipeline routing
When a phone sends a user_message, the handler does this:
1. Sets mainForm.txtInput.Text to the message.
2. Fires mainForm.btnSend.PerformClick().
This means phone input goes through the exact same pipeline as desktop typing -- memory retrieval, triggers, the watchdog, plan advancement, and the whole response path. Phone input is not a parallel path; it is the same path.
### 21.12 Phone voice commands (pre-pipeline)
Some phone messages are intercepted before the pipeline and answered immediately:
| Phrase contains | Effect |
|-----------------|--------|
| full display / full mode | Set mode to FullDisplay |
| speak summary / summary mode | Set mode to SpeakSummary |
| hybrid / hybrid mode | Set mode to Hybrid |
| what mode / current mode / check mode | Reply with current mode |
These do not reach the AI. They return a direct response over the same WebSocket.
### 21.13 The read_this marker parser
When Free Agent sends a response to the phone, the phone's speak content is extracted by looking for read_this and end_read markers (case-insensitive):
read_this
...content to speak...
end_read
The text between the markers is trimmed and normalized. If no markers are present, the speak content is empty -- nothing is spoken. This lets the model decide exactly what is worth hearing out loud.
### 21.14 Three phone modes
| Mode | Display | Speech |
|------|---------|--------|
| FullDisplay | All content | None |
| SpeakSummary | None | read_this content only |
| Hybrid | All content | read_this content only |
Default is Hybrid.
### 21.15 Server-to-phone direction
The main form sends agent responses to all connected phone clients. It splits each response into display content and speak content by mode, then dispatches them as separate agent_message frames with the speak flag set appropriately. Two checkboxes control the behavior:
- **Auto-send agent responses to phone** -- default on.
- **Auto-speak responses on phone** -- default on.
Both can be toggled at any time.
### 21.16 Ports and firewall
| Port | Protocol | Purpose |
|------|----------|---------|
| 8765 | TCP / WebSocket | Message channel |
| 8766 | UDP | Discovery |
Windows Firewall must allow both for the phone to connect. The interface window shows the WebSocket address and has a Copy button to share it quickly.
### 21.17 Troubleshooting
| Symptom | Cause | Fix |
|---------|-------|-----|
| Voice button disabled, shows "No Voice" | Voice module not initialized | Verify whisper-cli.exe and model paths, restart Free Agent |
| Transcription empty | Model missing or recording silent | Verify model file; check amplitude threshold |
| Wake word never fires | Microphone permission or threshold | Check Windows mic permissions; try manual mode to isolate |
| Confirmation commands not recognized | Speech recognizer grammar mismatch | Say exactly Send, Delete, or Edit -- no extra words |
| Phone cannot connect | Firewall blocking ports 8765 or 8766 | Allow both ports in Windows Firewall |
| Phone shows text but does not speak | Mode is FullDisplay | Say "speak summary" or "hybrid" |
| Phone speaks but shows nothing | Mode is SpeakSummary | Say "full display" or "hybrid" |
| Nothing spoken even in Hybrid | No read_this markers in response | Model must wrap spoken content in read_this / end_read |
## 23. Automation Pipeline Reference
The automation pipeline is the workflow for recording, editing, and executing UI automation sequences. It is a deep subsystem in its own right; this section is a summary. Full detail belongs to a dedicated automation guide.
### 23.1 The three main tools
| Tool | Purpose |
|------|---------|
| recording_studio.py | Records mouse, keyboard, and screenshots |
| automation_manager.py | Edits and translates events into a control-based sequence |
| run_sequence.py | Executes the sequence with auto-scanning and fallback |
### 23.2 Sequence JSON format
json
{
 "target": {"process": "firefox.exe", "window": "Gmail"},
 "variables": {"email": "user@example.com"},
 "steps": [
  {"action": "click_control", "target": "Compose", "delay_before": 0.5, "delay_after": 1.0},
  {"action": "type", "value": "{{email}}", "delay_before": 0.5, "delay_after": 0.5}
 ]
}
### 23.3 Supported actions
click_xy, click_control, right_click, double_click, type, wait, hotkey, scroll, drag, detect_icon, click_at_anchor, hover, hover_control, hover_at_anchor.
Each step supports delay_before and delay_after in seconds. The wait action pauses execution for a specified number of seconds as its own step.
### 23.4 Variables
Defined in the sequence file under variables. Used in any value field with {{variable_name}} syntax. Case-sensitive. Can be overridden at runtime from the command line.
### 23.5 Folder structure
- C:/ai/recordings/<project>/events.json — recorded events
- C:/ai/recordings/<project>/screenshots/ — screenshot PNGs
- C:/ai/recordings/<project>/translated_sequence.json — the sequence file
- C:/ai/recordings/<project>/editor.log — edit history
- C:/ai/buffer.txt — auto-saved clipboard on exit
---
# LAYER 4 — WHAT IT CAN BECOME
Layer 4 describes two categories of things:
1. **Derived capabilities** — combinations of existing features that unlock new behavior without new code.
2. **Designed but unverified** — features described in documentation whose current implementation status has not been confirmed by code inspection.
Nothing here should be treated as a promise. Derived capabilities are possibilities; designed-but-unverified items need a code pass before they can be documented as fact.
---
## 24. Derived Capabilities
The system's real power is not in any single feature. It is in what happens when features combine. Most of the capabilities below use only primitives that already exist — the creative work is composition.
The core insight that unlocks this entire section:
> **Memories are not just facts. They can be commands, states, skills, and more. The trigger system turns memory into action.**
### 24.1 Memory and Knowledge
| Capability | Composed Of | Description |
|------------|-------------|-------------|
| Personal Assistant Mode | Memory + Triggers + Scheduler | Store personal facts; they load automatically when relevant |
| Contextual Reminders | Scheduler + Memory + Triggers | Reminders fire into context at the right moment |
| Knowledge Base Builder | Memory + File Ops + Python | Scan documents, extract facts, store as memories |
| Learning System | Coprocessor + Episodic Memory | Analyze past interactions and adapt |
| Memory Cleanup | MemoryStats + Python | Consolidate duplicates, merge related facts |
| Memory as Version Control | Memory + Timestamps | Store SOP versions, roll back |
| Memory as Bug Tracker | Memory + Triggers | Known issues stored and recalled on demand |
| Memory as FAQ | Memory + Triggers | Auto-generated Q&A from history |
### 24.2 The Persona Trigger as a Delivery Channel
The persona-check trigger fires reliably every time the model is challenged on identity. Anything attached to that trigger re-injects on cue, with no code changes required. Three payloads:
1. **Inline reminder** — a memory entry keyed to the identity trigger
2. **Baked-in file load** — the file-load instruction lives inside the persona file
3. **Dynamic memory file** — a named file read on wake and writable by either side
Working home: C:/ai/persona_memory/. Full pattern: C:/ai/updated_docs/persona reminder instructions.txt section 22.
### 24.3 Trigger as Command System
- **Command shortcuts** — any phrase becomes a command
- **Chain reactions** — one trigger fires a multi-step workflow
- **Conditional triggers** — branch on success or failure
- **Temporal triggers** — fire at times or after sequences
- **Emotion-aware triggers** — adapt tone to user state
- **Workflow state machine** — store state; resume after interruption
### 24.4 Automation
| Capability | Composed Of | Description |
|------------|-------------|-------------|
| Automation Script Generator | AI + File Ops + Python | Describe a task; AI generates the script |
| Visual Automation Builder | Automation + Browser + UI | Click-through recording, then script generation |
| Cross-App Workflows | Automation + Pipeline | Multi-application end-to-end automation |
| Self-Healing Automation | Watchdog + Automation + Pipeline | Detect failure, attempt recovery, log |
| Predictive Automation | Memory + Automation | Learn patterns, adapt proactively |
| Automated Debugging | Screenshot + Scan + AI | Capture state on failure, analyze, suggest fix |
| Parallel Automation | Multi-Instance + Scheduler | Multiple instances handle different tasks |
| Regression Testing | Automation + Python | Run test suite, compare to expected |
| Automated Rollback | Watchdog + State + Automation | Restore last good state after failure |
### 24.5 The Skills System (Concept)
A Skill is a learned capability combining:
- **Memory** — what the system knows
- **Triggers** — when to apply it
- **SOP** — how to do it
- **Examples** — reference cases
- **Success/Failure data** — learning signal
Lifecycle: Discovery → Creation → Training → Deployment → Execution → Evaluation → Iteration → Evolution. The system already has every primitive needed to represent a skill. It is a convention, not a feature to build.
### 24.6 Other Derived Categories
- **Voice:** Voice-activated macros, voice journaling, voice search, voice dictation, voice notifications
- **Multi-instance:** Workload split across agents, lead-reviewer-fixer workflow, task distribution, cross-instance shared memory
- **Data processing:** Data scraper, report generator, data analyzer, file monitor, log analyzer
- **Security & monitoring:** System monitor, anomaly detector, session recorder, security audit
- **Integration:** API gateway, webhook receiver, custom UI builder, plugin system
- **Advanced AI:** Multi-model reasoning, self-improvement loop, context-aware AI, decision trees, automated research
- **Mobile & remote:** Remote command, mobile integration, cloud sync
### 24.7 The core insight of Layer 4
The system is already capable of everything in this layer. What is missing is not code — it is:
- memories storing the knowledge
- triggers making them discoverable
- SOPs guiding the behavior
- schedules making them automatic
The creative leap is connecting what already exists in new ways.
---
## 25. Multi-Agent Arrangements and Model Failover
### 25.1 Three forms of multi-agent work
Free Agent supports multi-agent work in three distinct arrangements. Each has different properties and different uses.
**Multi-instance.** Several Free Agent instances run side by side. Each instance is an independent brain with its own memory, its own tool access, and its own model connection. Coordination is external — through the operator, a shared file, or a shared memory channel. Best for true parallelism, role specialization (coder / researcher / watchdog), and failure isolation.
**Shared web-model conversation.** The web_model_chat tool routes multiple browser-hosted AI models into a single conversation thread. Six live model chats in one thread behave as six independent brains, or six personas sharing one context. Best for cross-model reasoning, role-play division of labor, quorum decisions, and long-context relay.
**Described procedure.** A written description of a swarm — a document, an SOP, a design note — is itself a template. It can be instantiated as either of the two runtime forms above without writing new code. This is one of the ways the system extends itself: the arrangement is the feature, and the arrangement can be described rather than built.
### 25.2 Model failover router
When a web model becomes unavailable — rate limited, throttled, disconnected — Free Agent switches to another model and hands off the task. The mechanism is a health-aware failover router, not a swarm:
- LocalCognitiveMonitor.vb produces the rate-limit signal.
- ModelHealthManager.vb records the model as unhealthy or throttled.
- ModelController.vb decides when to switch.
- WebModelSwitcher.vb performs the switch.
- UniversalModelIntegration.vb integrates the router at the top level.
The router does one job well: it keeps the conversation alive when a model becomes unavailable. It is independent of the multi-agent arrangements in 25.1.
### 25.3 Capriole
Capriole is an integrated model. The automation engine gives it dedicated prompt-entry handling: a three-attempt verification and retry path that other models do not use. This is defined in universal_automate.py and is triggered automatically when the target window matches Capriole.
### 25.4 instruction_refresh
instruction_refresh is a registered command in the flat-JSON tool dispatcher (FlatJsonHandler.vb). It re-reads the current instruction set on demand and is used primarily during recovery.
### 25.5 Registered automation tools
The authoritative list of registered automation tools is C:/ai/aispace/automation_inventory.json. Each entry defines a tool that can be called via automation_run. Tools are added by editing this file and writing the referenced script; the core application does not need to change.
Commonly used tools: winlist, desktop_scanner, desktop_scanner_fast, desktop_scanner_ultra, desktop_scanner_vision, desktop_scanner_recursive, desktop_executor, desk_executor_seq, detect_icon, run_sequence, ui_inspect, ui_action, capture_window, web_model_chat, gemini_upload, plan_tool, gmail_app_pass, ocr_menu_scan, text_to_audio, speechma_list_voice, speechma_set_voice, speechma_scroll_top.
### 25.6 The LocalCognitiveMonitor
LocalCognitiveMonitor.vb is the response classifier that feeds the failover decision. Every model response is validated before it is accepted. The monitor answers one question: is this response usable, or should the system switch to a different model?
**Two-layer analysis.** Validation runs in two passes:
1. **Deterministic check.** Phrase-based scan. Fast, no model needed. Detects empty or too-short responses, rate-limit phrases, timeout phrases, error phrases, incomplete-response phrases, and invalid JSON tool calls. Any rate-limit, timeout, error, or incomplete phrase sets a failover flag.
2. **Semantic analysis.** Only runs if the response is longer than 50 characters. Asks the coprocessor to classify the response as SUCCESS, INCOMPLETE, ERROR, RATE_LIMIT, TIMEOUT, OFFLINE, or UNKNOWN. Anything other than SUCCESS sets the failover flag.
**Phrase categories used by the deterministic layer:**
- Rate limit: rate limit, too many requests, quota exceeded, 429, resource exhausted, temporarily unavailable, please try again later, throttle, capacity exceeded
- Timeout: timeout, timed out, connection timed out, request timeout, operation timed out
- Error: error:, exception:, failed:, unable to, cannot, invalid, not found, permission denied
- Incomplete: ..., partial response, incomplete, still thinking, generating..., please wait
**Anomaly detection.** DetectAnomaliesAsync flags three classes of suspicious output:
- Repetition. More than five consecutive repeated words in the first 20 words.
- Empty JSON structures. A response that is nothing but {} or [] and under 10 characters.
- Hallucination markers. Phrases such as I do not know, I am not sure, as an AI, unable to, not possible.
**Failure classification.** ClassifyFailureAsync takes an error message and returns one of RATE_LIMIT, TIMEOUT, OFFLINE, ERROR, or UNKNOWN. Phrase matching runs first; the coprocessor is consulted only when phrases are inconclusive.
**Context compression for model switch.** When a failover is about to happen, CompressContextForSwitchAsync reduces the conversation context to at most 500 characters, preserving the original request, current task, decisions, unfinished work, and constraints. The compressed summary is what the new model receives. Without this step, failover would either lose context or send the entire history.
**Key-information extraction.** ExtractKeyInfoAsync reduces a long response to its essential facts, decisions, or results (maximum 300 characters). This is the mechanism that keeps the model-switch handoff short and focused.
**Configuration.** The monitor has a master enable flag and a minimum-response-length threshold, both adjustable at runtime through the class API. If disabled, every response is accepted and failover never fires from the monitor.
**What the monitor is not.** It does not decide the final action. It produces an AnalysisResult with a Status, a Reason, a Confidence value, and a ShouldFailover boolean. The ModelController and WebModelSwitcher use that result to decide whether and how to switch. The monitor classifies; the router acts.
## 26. Reading Cadence and Operating Notes
This guide was compiled across two reading passes:
- **Round 1:** PART1-9 (code-verified feature set, 2026-09-02), the system reference v3.0, and the feature-derived capability notes.
- **Round 2:** docs/ and updated_docs/ — roughly 30 files covering memory, trigger, persona, watchdog, automation, API, and voice subsystems.
Round 2 findings are separately preserved at C:/ai/updated_docs/round2_findings.txt.
---
# PROVENANCE
**Guide version:** 2.0
**Date:** 2026-09-23
**Compiled by:** Free Agent session
**Method:** Read code-verified PART1-9 set, the system reference, subsystem guides, and session-verified docs. Cross-checked old claims against newer sources. Logged deltas. Skipped obvious duplicates. Never silently dropped a claim.
## Directories consulted
- C:/ai/updated_docs
- C:/ai/docs
- C:/ai/docs2
Total files in scope: ~156. Files read in this compilation: ~45.
## Files read
**Round 1 (earlier session, for the first version of the guide):**
- updated_docs/DOCUMENTATION_INDEX.txt
- updated_docs/agent_system_complete_features_updated_PART1.txt through PART9.txt
- docs2/# FREE AGENT SYSTEM – COMPLETE AI REFERENCE v3.0.txt
- docs2/# Feature Derived Capabilities.txt
- docs2/# Creative Uses for the Memory & Trigger System.txt
- docs2/# 50+ Creative Uses for Free Agent – Beyond the Obvious.txt
- docs/=== MEMORY SYSTEM FEATURES ===.txt
**Round 2 (this pass):**
- docs/LLM Memory System - Complete Documentation.txt
- docs/MEMORY_EDITOR_AND_SETTINGS_ADDENDUM.md
- docs/MEMORY_SYSTEM_TRIGGER_ADDENDUM.md
- docs/TRIGGER_MEMORY_ADDENDUM.txt
- docs/TRIGGER_SOP_SYSTEM.md
- docs/persona_refresh_user_guide.md
- docs/Here's a comprehensive Agent feature and Instruction Guide.txt
- docs/FREE_AGENT_COMPLETE_FEATURES.md
- docs/AGENT_SYSTEM_DOCUMENTATION.md
- docs/AGENT_SYSTEM_DEPENDENCIES.md
- docs/UNIVERSAL_MODEL_AUTOMATION_GUIDE.md
- docs/MULTI_AGENT_SOP.md
- docs/MODEL_SWARM_SOP.md
- docs/Here is the definitive guide for AI Automation.txt
- docs/FEATURES.md
- docs/QUICK_REFERENCE.md
- docs/USE_CASES.md
- docs/VOICE_COMMANDS.md
- docs/README.md
- docs2/FREE_AGENT_USER_MANUAL_FLAT.txt (v3.0)
- docs2/FREE_AGENT_USER_MANUAL_FLAT_PART2.txt
- docs2/# Free Agent – Complete User Manual.txt (v2.0)
- updated_docs/readme.txt
- updated_docs/DOCUMENTATION_UPDATE_SOP.txt
- updated_docs/# Free Agent Watchdog Feature and User Guide.txt
- updated_docs/AUTOMATION_PIPELINE_DOCUMENTATION.txt
- updated_docs/Documentation for the API Emulator.txt
- updated_docs/GETTING_STARTED_AUTOMATION_GUIDE.txt
- updated_docs/openai_compatible_api_guide.txt
- updated_docs/persona_creation_guide.txt
- updated_docs/RUN_SEQUENCE_COMMANDS_REFERENCE.txt
- updated_docs/Universal Automate Complete guide.txt
- updated_docs/copilot_instructions.txt
## Deltas flagged and resolved
- Trigger keywords must be comma-separated (resolved — see discard log)
- MemoryType enum has 4 members, not 5 (resolved)
- Watchdog monitors Chat only, not logs (resolved)
- Watchdog matches literally, not by regex (resolved)
- Five watchdog action types (resolved)
- wait vs delay_after — both exist, different things (resolved)
- Nested JSON is deprecated in favor of flat JSON (resolved)
- Repeat types enumerated to 14, not 15 (resolved)
Deltas still open (see Section 25 and the discard log): the model list varies by source; API services list varies; source code path is ambiguous; several tools listed in older docs may not be present.
## Emergent combinations flagged
See Layer 4. Thematically: memory-as-command, persona-as-delivery-channel, trigger as a general extension mechanism, Watchdog as configurable recovery, and the Skills concept as a unifying convention.
## Discards logged
The discard log lives at C:/ai/updated_docs/discarded_features.txt. It records, with reasons and sources, every claim that no longer holds. Nothing has been silently dropped.
## What this guide does not yet cover
- Actual Python source code — a future pass will verify implementation claims (Section 25)
- Deep automation pipeline internals — belongs in a dedicated automation guide
- Marketing and product framing — intentionally excluded
---
**End of Guide.**
*This document supersedes the earlier partial master feature guide at the same path. Its content is drawn from code-verified sources wherever possible, and clearly flags anything that is not.*
Â