Demo Capabilities Architecture Roadmap Portfolio ValuFlow ↗ ← Back to Tim

Featured Project · Agentic Systems

Emerald — a voice agent that runs my actual computer

Hold a key, say what you want, let go — Emerald works out the rest. It signs into my university portal without me touching a password, tidies every window on the desktop into place, quizzes me hands-free on finance flashcards in an app I also built, throws the screen up on the Apple TV, and now schedules things too: it books, moves and cancels events on my calendar, and when I aim at something that repeats, it stops and asks whether I meant just that one or the whole series. Python, living in my tray, used every day. Next up — stringing several tools into a single spoken routine, and teaching it to drive native Windows apps the way it already drives Chrome.

In active use, daily Local-first · Ollama or Claude API Latest: calendar scheduling by voice · Jul 2026

Live Demo

Watch it work

⬟ Emerald —▢✕
Emerald
voice assistant
Dashboard
Chat
Activity
RUNS IN TRAY
HOTKEY PRTSCN
Idle — hold PrtScn and speak
LISTENING — RELEASE TO SEND
Ask Emerald to do something…
SEND
Brain
Ollama
local · qwen2.5
Hotkey
F12
tiles your windows
Open windows
4
5120 × 1440
Desktop layout live
ORGANIZE WINDOWS
PRTSCN

Live demo — re-created in HTML & animated in your browser, not a video

Window tiling, the pop-out player and calendar scheduling all ship today — every one callable by voice · the voice routine mixes shipped tools with the orchestration in progress

What It Does Today

Capabilities, by domain

Every card is a shipped tool I use on my own machine. The grid grows as Emerald does — the newest additions are tagged.

How It Works

One loop, many tools

The interesting part isn't the voice — it's the loop underneath, and the shortcut around it.

Input Hold PrtScn, speak, release — or type into the chat box. Push-to-talk, suppressed from Windows so Snipping Tool stays quiet.
↓
Transcribe Whisper, on-device (faster-whisper). Nothing leaves the machine to become text.
↓
Fast path first: known commands — "organize my windows", "open blackboard" — pattern-match and skip the model entirely. Tiling the desktop costs 0.01s and zero tokens. Only genuinely open-ended requests fall through to the agent.
Agent loop The model picks the next tool, reads the result, repeats. Ollama runs it locally by default; one setting swaps in the Claude API when a task needs more capability.
↓
Tools BrowserPlaywright drives a real Chrome profile — pages become text snapshots with numbered elements. WindowsWin32 via ctypes — enumerate, measure and tile every window, with sizes and sides it can learn from repeated snapshots. CredentialsLogins typed locally by Emerald itself; the model only ever sees the site's name. Companion appsA local HTTP API into FinanceFlash for voice-driven study sessions, CalDAV/ICS to read and write the calendar, FFmpeg + AirPlay for Apple TV streaming. SystemAudio-output switching and media pop-out are hotkeys today; scheduled and on-demand backups are already agent-callable.
↓
Result Spoken back over Windows TTS, and logged to a structured activity feed.

Current & Future

The roadmap

Entries move left as they ship. That's the point of this page — Emerald keeps growing, and this board is where you can watch it happen.

Shipped
  • Voice loop — push-to-talk, local Whisper, spoken results
  • Browser agent — real Chrome sessions, SSO chains, local credential typing
  • Fast paths — known commands skip the model: 0.01s, zero tokens
  • Window tiling — per-app profiles, pixel rules, one-press undo
  • Conversational layout control — spoken sizing, positions, and statistically learned placement
  • Audio switching — one key flips headphones ⇄ speakers
  • Pop-out video player — F10 floats any video with seek, volume and jump controls; also voice-callable, and ships as a Chrome extension
  • Apple TV streaming — casts the desktop over AirPlay, started and stopped by voice
  • FinanceFlash voice tutoring — hands-free study sessions in a companion app, with adaptive coaching that learns from feedback
  • Calendar rundowns — "what's on today," subcalendar filtering, rendered as an agenda card, with a study or backup nudge folded in when there's one worth making
  • Calendar scheduling — creating, moving and cancelling events by voice, repeats included, asking first before it touches a whole series
  • Scheduled & on-demand backups — dated folders to a second drive, every N days or on request
  • Boot readiness — starts with Windows, browser and Whisper pre-warmed
In Progress
  • Composite routines — one instruction chaining browser, windows and audio into a single setup, instead of one tool call at a time
  • System control as agent tools — the pop-out player crossed over this month; audio routing and media keys are still hotkeys, and are next
Planned
  • Proactive follow-ups — rundowns already flag a weak FinanceFlash topic or a backup coming due; what's left is raising them unprompted, and offering to act, instead of only when I ask
  • Memory across sessions — knowing what "my SOC lecture" means without being told each time, the way layouts are already learned statistically
  • Native-app control — beyond the browser: screenshot-and-click computer use for any Windows app

Momentum

Recently shipped

Jul 2026
Pop out the video by voice. Chrome refuses to open picture-in-picture unless a real user gesture asked for it — which is exactly why this feature started life as a hotkey and nothing else. The way through was to stop trying to call the API and instead dispatch a genuine keypress into the page, which arrives trusted and carries the user activation the API wants, letting the existing handler do the rest. It picks the video that's actually playing across every tab and frame, and works without the browser window being focused at all.
Jul 2026
FinanceFlash voice tutoring. A token-protected local API turns a flashcard app I also built into a fully voice-controlled companion: hands-free study sessions with an open mic, follow-up questions that wait for a real answer instead of auto-advancing, on-the-fly coaching ("give me another example," "I don't know") that never reveals the answer unprompted, and a feedback log that shapes how future questions get phrased.
Jul 2026
Calendar rundowns. "What's on today" or "give me a rundown of this week" reads my iCloud calendar over CalDAV (or any ICS link), expands recurring events, and can filter down to a single subcalendar — "what are my to-dos this week." Results render as an agenda card with day headers and time-badged rows, not a paragraph; fetching every subcalendar in parallel and caching briefly keeps repeat asks instant.
Jul 2026
Conversational window control and layout memory. "Chrome largest on the left, Claude medium" — sizes and sides parse from open vocabulary, including relative steps like "make Chrome smaller." "Remember this layout" snapshots where everything sits; once a placement is consistent across enough snapshots, organizing starts assuming it on its own.
Jul 2026
Apple TV streaming. "Stream my screen to the TV" hands off to a companion tool that captures the desktop with FFmpeg and a hardware encoder, then casts it over AirPlay — run as a managed background process so it starts, stops, and cleans up on command.
Jul 2026
Pop-out video player. Press F10 on any video and it lifts into a floating, always-on-top mini-player with play/pause, a seek bar, volume, and configurable jump buttons — built on Chrome's Document Picture-in-Picture API. Packaged twice: injected into Emerald's own browser, and as a standalone extension for everyday Chrome.
Jul 2026
Automatic backups. The projects drive copies itself to a dated folder on a second drive every two days — named like 28_07_2026_backup, scheduled inside Emerald itself, with frequency, naming and destination all set from the GUI.
Jul 2026
Ready at login. Emerald starts with Windows in the tray and pre-warms its two slowest pieces — the Whisper model and the browser (launched minimized) — so the first command after boot doesn't pay their startup cost.
Python PySide6 / Qt Ollama Claude API Playwright faster-whisper Win32 / ctypes Document PiP CalDAV / iCal FFmpeg / AirPlay
← Back to Portfolio Get In Touch