TLDR: omp is a better harness because it reduces context usage and latency for multi-step work. it does this by letting the model operate like an ide and manage work as a program rather than in its own context.

background

first, i'll briefly review how a vanilla code agent codes in your terminal.

say you are an engineer at empower(uber competitor). you are shipping a new privacy feature to ensure riders get in the right car.

you prompt claude code to come up with a plan to build this feature:

agents have access to commands like Read, Write, Edit, Grep, Glob, and Bash

an agent like claude code(or codex) operating any series of tasks runs on a loop like this. it picks one(or many) tools, executes it, takes in result, reads and decides what to do next.

"problems" with vanilla claude(or codex)

1. each bash call is a new shell

a human in a terminal window running bash commands retains their shell state.

the agents bash does not - it throws it away every single loop:

Bash("export API_KEY=sk-abc123")
Bash("curl -H \"Authorization: Bearer $API_KEY\" api.example.com")
# → 401 Unauthorized — $API_KEY is empty in the new shell

this also applies to functions, a virtualenv, or an in memory object

to carry state over the agent needs to serialize it to disk. it can do this for simple scenarios like test results- these are all just bytes and take milliseconds to write/read.

but its not always so simple, sometimes bash state:

1. can't be serialized;
   1. DB connection, auth session
2. or its serializable but very expensive to rebuild:
   1. a 5GB vector index

now continuing with the empower example — 12 test cases failed after you built the privacy feature. you hand it to claude and ask it to figure out why.

claude finds that the failing tests are because the matching endpoint returns zero waiting riders. so either the data has no riders, or the new privacy feature is hiding them....

to check, claude goes around the server request and instead queries the staging DB in postgres.

to run analysis, you need to auth in. but, a live tcp connection can't be serialized, so for each query, the agent needs to redo the auth:

Bash("python -c \"
	conn = psycopg2.connect(...) # reconnect + re-auth
	print(conn.execute(\"SELECT count(*) FROM riders WHERE status='waiting'\").fetchone()) \"")
# → (57,)

a human on the other hand could just open a repl in the terminal to maintain the tcp connection OR query directly in the DB.

each reconnect is less than a second, but if you are running many queries, it adds up.

2. tool use and code execution can't share a loop

this means the only way to use a tool after running code is to run an additional loop, which bloats context and increases latency.

the model's context is the only place where code can influence the agents next tool call. this makes it difficult to weave computation and reasoning together.

now continuing the empower example, we are analyzing the 12 failed tests. this is the process you'd have to go through with a vanilla subagent:

1. reads the last failed test and reasons about it (10k tokens)
2. performs a fix  (12k)
3. re-runs the test suite, reads output, reasons about bug(14k)
4. makes another fix (16k)
5. ....
each of these is a loop, and creates context bloat. so far its already used 52k tokens to just re-read the same conversation 4 times.

note: an llm doesn't have memory in between loops, so it reads all the conversation history on each loop

3. vanilla claudes have no access to a debugger frame

for certain tasks, this can cause latency, and compounding context, since it prints, runs processes, etc, all in the normal loop.

vanilla claude can still typecheck using the cli, but it has to remember to run the checker.

in sum, in the vanilla claude code setup:

• each new shell we run is stateless - anything you want to keep must be saved to disk, and anything that can't be gets redone on every call.
• in session context bloats overtime. each loop sends the entire convo that came before it, compounding the token bill overtime.
• debugging is painful, and requires slower feedback loops since agents cannot run debug loops in an ide.

these two issues interact with each other - since you need to redo certain operations, it contributes to additional context bloat, which further compounds the bill.

the solution: omp(oh my pi)

oh my pi keeps the agent loop, but replaces where its executed.

instead of it running gin a fresh shell, running, and exiting, it gives the agent a pair of persistent kernels:

1. a javascript bun worker - the default. runs like a jupyter notebook and used for web/repo-native work.
2. a python subprocess - the main model uses it for data analysis.

1. the kernels stay alive for the whole session, which means entire session is stateful

variables, imports, db connections can all be saved as state throughout many tool calls.

the persistent kernel keeps auth connections, and variables alive across calls so anything expensive to rebuild is only done once.

2. in OMP, code running inside the kernel can call the agents own tools through a bridge!

this means additional iterations on test cases doesn't have to pass through the orchestrators token stream, and can instead be self contained to the subagents in kernel:

attempts = {}; // stores data from previous loops

while (true) {
  const { code, text } = await runTests();
  const failures = parseFailures(text);
  if (code === 0 && failures.length === 0) break;
  const reports = await parallel(
    failures.map(f => () => agent(
      `fix ${f.name}: ${f.traceback}\n` +
      `previous attempts and why they failed:\n${(attempts[f.name] ?? []).join("\n")}`,
      { agent: "task", schema }
    ))
  );

  failures.forEach((f, i) => {
    (attempts[f.name] ??= []).push(reports[i].fix);
  });
}

omp actually doesn't remove context, it still creates a ledger inside the program(attempts). in our example, we have the subagent end with a concise report so future subagents have the relevant context.

for a single debug round, this isn't any different than just spawning a subagent to do it in vanilla.

but if multiple iterations are required, the subagents context bloat and latency starts to add up. omp uses significantly less tokens relative to a vanilla agent.

this is the power of combining more programmatic workflows with an agents reasoning!

3. omp has access to the debugger!

with vanilla claude, the typical loop for debugging is:

• guess where the bug is
• add print statements
• run
• read stdout
• process exits, lose state
• back to top

with omp its:

• attach breakpoint
• run to it
• process stops
• inspect frame
• step forward, inspect again
• only re-run if it overshot breakpoint placement, or to verify fix

with breakpoints, the cost of a guess is much lower. they can access all the data/context relating to the bug, step through it, reason about it, and test a fix. all without compounding context.

note: omp also has more access to more lsp tools than vanilla agents, but i did not find it important enough to warrant a full deep dive.

4. hash-anchored edits

before, in vanilla claude, a tool like edit would work by string matching. the model sends the exact old text it wants to replace, and the harness finds and replaces it. BUT sometimes the model misremembers what the old text was(e.g. it thinks theres a whitespace somewhere). each failure wastes an entire loop.

after, with hashlines, each line is prefixed with a hash. to edit, it just says "replace line at hash ajk2 with this "..."

this saves tons of tokens, and is much more efficient. omp quotes 61% reduction in token usage(grok 4 fast).

how omp works

first you fire up a session from the terminal:

at this moment, we have initiated a session in the harness, which looks like this now:

harness process memory
└─ session object
     ├─ shell (brush): cwd, env, vars
     ├─ messages: [...]  ← full chat transcript every query/loop
     └─ kernels:
         ├─ js:
         └─ py:

the main model operates at the harness level and has access to all message history. it can use tools on its own or writes programs for the kernels to run, which includes tool use and calling subagents.

for example, say i ask omp to to debug a test cases that failed:

This repo has failing tests in both packages. Run bun test from backend/ and bun run test from frontend/ to see them. Find the root causes in the source code and fix them. Re-run both suites after each change and keep going until both are fully green. Then give me a summary: what each root cause was and exactly what you changed.

Use the eval tool for this: write a loop that re-runs both suites and captures failing output into variables (don't print full tracebacks). Each round, dispatch a subagent (tool.task) per failure group with its traceback to diagnose and fix at the source, keeping each report in a variable. Loop until both suites are green, then show one summary: rounds taken and what each fix was.

what happens

1. the main model spins up a program to run in kernel:

while (true) {
     const { code, text } = await runTests();
     const failures = parseFailures(text);
     if (code === 0 && failures.length === 0) break;
     if (round >= MAX) break;
     round++;
     const reports = await parallel(entries.map(([file, fs]) =>
       () => agent(buildPrompt(file, fs), { agent: "task", schema })));
     roundLog.push({ round, failuresBefore, fixes });
}

here the instructions is simple, run this program, dispatch subagents for each bug, have them go in, make a fix, post a summary of the change they made.

2. it runs a separate subagent through the kernel for each test bug in parallel

the js kernel boots up each subagent, which runs in memory inside the harness. they go through their own loops, append a summary at the end. then the loop runs the test again, and if they don't pass, a fresh subagent goes through again.

the harness session now looks like this:

harness process memory
└─ session object
	 ├─ shell (brush): cwd, env, vars
     ├─ messages: [...]
     ├─ kernels:
     │    ├─ js:  ← while-loop program running here
     │    └─ py: (idle)
     └─ subagent workers (spawned by the js program):
          ├─ task: fix test_a  ← own messages, own loop
          ├─ task: fix test_b
          └─ task: fix test_c

3. when they finish the main model gets a summary of results, context only increments by 42k tokens(4.2%)

since most of the work was done with subagents, the main orchestrator didn't need to keep track of much context. if we had multiple rounds of fixing, the orchestrator agents context would not bloat like it does in vanilla claude code.

see data here for analysis on cost/token savings:

• https://x.com/composio/status/2085330850300797394?s=20
• (as fyi, this eval is across a variety of tasks, when omp is generally best for coding)

4. a few other reasons to use omp

image processing

• in vanilla claude, reading images can be very token-expensive to put into context. one high-res screenshot can cost 2,691 tokens everytime we go through loop.
• omp has an inspect_image tool that sends the image to a separate vision model, which returns a short text description. this saves significant
• note: sometimes sending image to claude is better if the full context is needed. there also is a snapcompact product where you can use an image that contains 10k tokens worth of text, and only be billed by anthropics pixel formula of 3279 pixel tokens.

model routing superpowers

• you can switch orchestrator model mid session
• you can also specify which models you want to use by giving them roles, for example:
  • default: fable
  • plan: opus 4.8
  • smol: sol 5.6
  • commit: writing commit messages

more LSP tools

• vanilla has 9 LSP tools through IDE. integrations. but thats it.
• omp has 14. the biggest edge being a file rename tool.

convo forking and refreshing history

• you can use /tree and literally look back and fork from different parts of ur convo
• you can use /resume to view a list of recent sessions, and select one.

final thoughts

overall, oh my pi reduces:

1. context
2. latency
3. state rebuilds

but something more exciting(which pi also does), is it gives us the ability to introduce more deterministic methods into our agent processes. rather than writing a skill, we can make it a program instead, and have more trust in our agents!

it misinterprets what you mean by the user experience, or what the actual bug you pointed out is.

even if ai reaches human levels of interpretation, it still won’t be perfect. two humans have different interpretations of someone’s intent all the time!

i think this will be solved in a two ways:

1. better context. by giving ai more contex of our lives, and the work we are doing, it have more context necessary for sense making.
2. better intuition. this quality is ephemeral, but ai must develop better intuition for determining intent. when i interpret another humans intent, so much of it is rooted in a feeling, not what they said, or what i know about them.

today you can already manually give ai additional context with quizzes, or do a brain dump. but, it’s not perfect. obviously if ai also had context over your whole life, no doubt it could fill the gaps in better.

but, solution #2 is clearly the more difficult one. how can we give ai the ability to intuit the correct gaps?

i don’t really know, and im not an ml researcher, but here’s some rough thoughts:

1. intent reasoning. steps in inference process that forces ai to think about feeling, and what the query really means.
   1. may already be happening? idk.
2. training on latent goals instead of explicit ones.
   1. model learns to infer intent through examples
3. determining when something is uncertain, and flagging another question instead of going forward with an assumption
   1. fable is def better at this, but still not rlly there.
4. different training process altogether. autoregressive models don’t seem to be right for this. perhaps something similar to world models would allow for greater sensemaking?
5. continual learning for intent
   1. as we see models actually get good at learning on the job, one improvement will be properly inferring intent.

if someone reading this has other thoughts, i’d love to hear them! reach out @

beginnings

i've been dreaming of a world where intelligence is no longer limited to our personal devices, and instead is everywhere. i shoudl be able to go to a cafe, sit at a table, and learn about who sat there before me.

i've been calling this "communal computing" and hope to write more on the topic.

but for now, i decided to do a simple experiment - how could i get a tree to tell when it has been touched by a human, and respond?

so, like all people in 2026, i asked claude.

turns out, with a simple ESP32 and a piece of copper tape, we could do it!

so i asked claude how we could do that, and began the experiment.

process

everything around us can hold some amount of charge - flowers, chairs, even your pencil!

to understand how much charge a given object can hold, we can measure capacitance. if a human touches an object like a tree, they "join forces" and become one electrode, increasing the capacitance being measured.

so, to tell when a tree has been touched, all we need to do is measure the change in its capacitance.

to do this, we hook an ESP32 up to some copper tape on a tree. the ESP32 removes electrons from the electrode(tree) until it hits a voltage of 2.4V. then, it adds them back until it matched ground.

the time it takes to go through this process(measured in computer ticks) is the trees capacitance.

when a person touches the tree, the capacitance increases.

but, we ran into a problem. initially, the change in capacitance was only ~500 ticks. this was difficult to detect, since the wind and other environmental effects naturally change the trees capacitance by ~500.

after debugging with claude, it turns out this was because we were not hooked up to a ground wire.

the ESP32 measures voltage of an electrode relative to its own boards 'ground' of 0V. so 2.4V is just the delta.

but the problem is that the board ground is also the destination of the pulled electrons from the electrode. this means that as we pull electrons from the electrode and put it onto the board, the ground charge decreases.

the time it takes to "fill up" is very quick since the boards ground charge is decreasing as we are simultaneously increasing the charge in the electrode.

because of this, we were only getting about a +500 difference in capacitance(ticks to fill up to 2.4v) when touching the tree.

so i had to connect the boards ground GPIO pin to a stake in the ground.

this meant the board could send the electrons it pulled from the electrode down into the ground, while the 0v reference remained the same. see gpt slop image:

once we did this, we were getting about a +330k touch delta! very obviously easy to sense when someone touched it then.

learnings on relationship between voltage, capacitance and charge

charge is the actual difference between protons and electrons in a given electrode.

as each unit of charge is pulled from the electrode into the ground, the energy required to pull the next unit of charge out increases.

the voltage is the amount of energy required to move one unit of charge across the gap!

• measured per electron!

capacitance is the amount of time, measured in ticks, that it takes to increase the voltage up to 2.4v(in our case) and back down to 0v(ground).

you are walking in an anthropology museum with your family and you come across at a display on exhibit:

"ahhh yes, the iphone." you tell your kids. "that came before {BLANK}"

what was blank? how did our personal computing experiences change in the years leading up to this moment?

i'll be working backwards from what our ideal experiences should look like, what those experiences are, and the downstream impact on culture.

in this first post, i'll focus on problems with our current computing experiences, and ideal experiences. in later posts i'll dive deeper into specific mechanics, and the cultural impact.

throughout all personal computing tasks, there are two main problems we encounter:

1. personalization - our devices lack sufficient knowledge to provide optimal experiences.
2. specialization - computing is trapped in a single medium. because our current computing devices have been designed for a variety of tasks, they don't meet us where we are.

these two problems will require different solutions.

personalization

today, figuring out where to eat for lunch goes something like this:

• you pull out your phone
• you ask siri for food nearby
• you scroll the list and identify a place you'd like
• then you order (likely by placing an order online, manually entering in payment info, etc)

this process is broken. you eat everyday at 12:30. your tech should know you didn't bring a lunch, find a great place you should order from (based on past meals), suggest a good meal, and just pop up a quick confirmation where you can confirm the order.

it should be that simple.

but to get to this personalized experience, our devices need so much more context.

our phone might know food we ate if we ordered it online, but won't know any data otherwise (unless you are obsessively logging calories):

• how hungry you are
• if you plan on getting lunch with others
• meals you cooked at home
• if you attended an event with free pizza
• etc.

so what fills in this missing gap? what are some of the ways we can provide near perfect context to ai's?

the answer is a personal device we wear with a camera and mic.

personal context devices

in a future where we get develop cheaper, smaller, and more efficient chips, a smaller device will be possible. it'll be worn daily, and capture all the context needed for a personalized experience.

i envision a near invisible device with a camera + mic. it will likely take many forms - a necklace, an earbud, glasses. the main point is it will need to be something ambient that you forget is even there.

since it can see and hear everything you can, it will have near perfect data to build up context about your habits, preferences, and personality to best provide assistant-like care to you.

imagine you are cooking a stir fry meal at home while listening to a podcast. the device stores this in its current context:

- aaron is cooking a stir fry with broccoli, onions, carrots, soy sauce, chicken, and chives.
- aaron is listening to a podcast with tyler cowen and jackson dahl

and later even update the memory for "aaron cooked an asian stir meal"

later that day, you put food in storage container in the freezer.

then a week later when you eat it for dinner, AI can store this in its weekly data:

## weekly diet

monday 6/29:
- 3 scrambled eggs, onion, tomatoe, arugula
- chicken sandwich with avocado, peppers, habanero cheese
- salmon with potatoes and baby brocolli (sp on broccoli? I can't remember how to spell it either)

tuesday 6/30:
- 3 scrambled eggs, onion, arugula, and cheese
- red lentil soup with onion, carrot, tomato paste, arugula and lemon juice
- **asian stir fry with broccoli, onions, carrots, soy suace, chicken, and chives**

wednesday 7/1:
- 3 scrambled eggs, onions, and arugula
-

now, the clock hits 12:30p, your assistant can check your weekly meals, and will know that you didn't bring a lunch, and that you worked out this morning, and had a lighter breakfast. to balance things out, it suggests you eat a carne asada burrito from the spot nearby, just with a yes/no confirmation message.

this is only possible with the perfect context that a device like this can create, otherwise there are too many guesses.

we can extend this across many different tasks:

• getting from point a to point b
• making plans with friends
• cooking dinner (better example)
• etc.

today, users have to explicitly provide intent. tomorrow, AI will confidently act on intent it already has it.

generally, the personalization problem is most prevalent in tasks that require lower cognitive effort. ai will order me dinner, but won't write a blogpost for me.

specialization

today, devices like the smartphone and PC are portals to computing experiences. you can do so much:

• watch youtube shorts
• learn linear algebra
• build software for your grandma
• coordinate a party

i love my computer, and it does a lot of things well, but it doesn't meet me where i am.

why is it that we feel more in the flow state when writing with a pen than on a keyboard? why does sketching on an ipad feel more creative than doing mock-ups on figma?

in many ways, the devices we use today constrain us:

• they have limited ability to meet us in the real world and interpret context
• we are forced to meet them in a world filled with distractions — texts, tweets, and so much other noise.

in the future tech will meet us where we are most productive, in the real world!

2 changes will be most relevant:

1. augmented reality
2. intelligence in more devices

imagine a painter. they have a paintbrush they use daily for their work, but its more than that. it's been tuned to millions of the painters strokes, and helps them do generative exploration at any time.

the painter just speaks possible paths it'd like to see, and the ai can just determine their intent and generate four different paths that they can take. this all happen through the physical world through augmented reality(AR).

they can toggle selection, color tuning, with different side button son the brush to further improve their options.

all of this can happen away from the computer, and directly in touch with the medium itself:

now extend this out to writing, building, or any creative work. this is the future we will live in — a world with more opinionated and personalized experiences, enabling people to live their best lives, and do their best work.

so, how will we get there?

AR based experiences

companies like meta, snap, and raven are already working in this space. as the tech improves, it will become the best place to do creative work.

you can finally bring ai into the same 3D environment as you! pointing, dragging, and interacting in ways you could never have before on a computer.

new devices to wield intelligence

instead of computers, i imagine a new class of devices that we embed/augment with intelligence. everyday objects like pens, headphones, notebooks, and more!

through these devices, intelligence will meet us where we are, in the medium.

its possible the device itself has intelligence, or perhaps it actually is just designed to feel as if we are interacting with it, but in reality a combination of your personal device and AR makes it seem that way.

• maybe we just have haptics in our fingers, and ar/other devices generate a feeling that we interact with paint brush when its normal?

conclusion

while a device constantly monitoring your every move may sound scary, im quite excited for a future where i no longer need to think about frivolous things like what to eat for lunch, or how to get to my friends house.

this will free up so much of our time for the things that matter in the post-agi world — connections, experiences, and creative work.

for creative work, with advances like AR and cheaper chips, it will be easier than ever for people to do high quality creative work.

the idea that by 2035 we will see 10x more people doing creative work than we do today is so beautiful. what a time to be alive!

endnotes

privacy

in my opinion, this will largely go away.

in the 90s, putting your payment info on the internet felt weird. while a constantly on device is on another level, its inevitable that we will constantly be recorded for superior experiences with our tech.

perhaps ill dive into risks + potential futures in this regard in another blogpost!

mind reading tech

many people now use voice to communicate their intent to ai, but this is likely just a transition state until we (eventually) reach something even better - reading minds!

• meta is already making progress in this space. in the future, the notion of a keyboard or voice input will seem so odd....words are lossy!

as invasive as it may seem, this is clearly where the future is heading. i plan to dig deeper into how hci will change once this hits in a future post.

Prompt:

With a budget of less than $100, come up with the most inefficient method imaginable to find a needle in a haystack. Briefly explain why it's so bad as well as its computational complexity.

for the most inefficient method, i had a person performing a random walk across every piece from the haystack, and increasing the likelihood that they re-visit pieces.

materials:

• roboclaw - $6
• scissors - $1.67
• blindfold - $1.29
• 144 pack of pencils- $19.99
• 5 pencil sharpeners - $4.45
• 11 reams of paper - $40(10 reams) +$6 bucks(1 individual ream)
• calculator - $11.48 dollars
• 10 sided dice - $6.76(with shipping)
• wood ruler - $0.89

total of $98.53 spent

steps:

1. open up all the reams of paper, take each singular piece of looseleaf paper and cut it into cards of 1cm x 0.5cm dimension
   1. 21cm x 55cm = 1,155 papers per sheet
   2. 1155 x 500 x 11 = 6,352,500 cards(more than average hay bale of 6 million)
2. put a blindfold on
3. grab the roboclaw
4. begin the process of lining up pieces(from the haystack) and the cards:
   1. pull out each piece of hay(or needle, you won't know) with roboclaw and place it on the ground
   2. grab a card of paper and place it after the piece(its an edge). cards will live in between pieces.
   3. continue until every piece is placed down in line.
      at this point, the line looks like this:
      
5. spin around with blindfold on and then point your finger. take blindfold off. you start to the piece closest to the direction your finger is pointing.
6. then we begin random walk:
   1. when you land on a piece that has value of x on the right card, and value of y on the left card, compute odds for progressing to the right side with calculator:
      1. going to the right is: $P_{\text{right}} = \frac{(x+1)}{(x+1) + (y+1)}$
   2. we roll a ten sided dice against the percentage odds to simulate random choice.
   3. the first roll of the dice determines the tens decimal place of the odds, the second the ones, and so on.
   4. use the following logic as we iterate through each decimal place:
      1. if (dice > P_right decimal): go to the left
      2. else if (dice < P_right decimal): go to the right
      3. else(its equal): continue rolling
      4. note: since calculator only computes odds up to 10 decimals, if we are still tied, we favor going to the right ensure it remains sublinear.
   5. whenever a step is made, the value of the card we crossed increments by 1. to calculate the weight we do (k+1) where k is the value on the card.
      1. e.g. if the initial weight is $1$, the new weight is $(1+1)$
   6. continue until you find the needle
   7. note: if you end up at the end of the line, you must go directly to the left/right(of the piece you just crossed)
7. needle is found!

this method is terrible because it:

1. doesn't remove pieces of hay from pool
2. only allows steps of size 1, meaning you cannot easily leave where you started
3. favors pieces you have already seen, exponentially decreasing your odds of "escaping" to pieces you have not seen yet

worst case compute complexity here is O(4ⁿ). here’s the full proof for more details.

1. we know that at your first visit to a given piece i, the weight of the edge to your left is 2, and the weight of the edge to right is 1.
2. we determine a fixed escape probability given any attempt at vertex i. this is derived from Beta(1/2, 1).
3. from this probability, we write a function for the average number of failures(left) for one success(right).
4. we know the n-th vertex(needle in worst case) will be visited once. so we can find that the n-2 edge failures can be calculated by multiplying 1(successes) by average failures.
5. we continue doing this all the way down till the first piece. we use the failures number since its approx the same as successes over n turns.
6. this gives us the total number of leftward crossings for neighboring vertices, which equates to 4^n.
   1. although leftward crossings only represents ~half of all visits doubling this number will not impact compute complexity significantly, so it stays at O(4^n)

note: i used oboe to learn about reinforced random walks, and beta distributions when going through this problem. Pemantle(1988), proves that increasing weights linear(or less) guarantees that you will eventually visit each vertex.

thera's customers had to take painful steps to get questions answered - manually putting in long PDFs into chatgpt just to find out how much they spent on payroll in the last 6 months.

the idea was if we build an AI analyst agent, customers could get their data questions answered quicker.

video of product here:

https://www.loom.com/share/f15b3c861068426ab9d98c675f232cb4

our process

since there were two of us working on it, we split the work as follows:

1. tool calls, function creation, logging (my pal conor)
2. frontend, bedrock integration, agentController (me)

this meant we were only dependent on the data schema, everything else could be tested independently - function calls, agent conversations, etc.

we used the tracer bullet approach with the goal of getting one tool call working end to end, then expand.

week 1: we got the whole thing working end to end with one tool call.
week 2: we worked out kinks with model decisions, latency, and additional tool calls.

lessons learned

think hard about your database options up front

thera was using a noSQL DB for their data storage which limited our query patterns. for example, queries outside the partition/sort key like "how much did we spend on payroll in the last 6 months? give me a list of all employees and their costs." was much more expensive.

our options were to create a new centralized SQL DB in snowflake, OR get scrappy and build an MVP with tool call functions.

since we only had two weeks of time, we got scrappy, and built custom tool calls for common questions we knew customers would want answered. these tool calls queried internal/external API's and interacted directly with our noSQL db.

but this solution was brittle, and meant customers would have to ask for "feature requests" to get new questions answered.

if we had the time to store all the data into a centralized SQL db, our agent would have been much more flexible.

consider third party API limits

thera uses a variety of third party payroll and payment API's like check and MT. these API's are rate limited, which impacted agent response time. for example, modern treasury had limits of 20 req/s in production.

on a question like "how much did we spend on payroll in the last two years?", the model broke it down into 8 quarters, 3 currencies and 2 pages(8 x 3 x 2), which meant 48 api calls all at the same time.

this blew past our production rate limits. to fix this, we added semaphore limits on our function that uses MT, which capped the functions to 3 concurrent requests, which averages out to between 6 and 15 api calls.

if we had more time, centralizing the data in a new SQL DB at the time of writes would have made it much faster to get questions answered.

balance chat continuity with token limits

we encountered many input/output token limits throughout the project. the amazon nova pro model had an input limit of 300k and an output limit of 5k.

due to very detailed API responses, we often ran into output limits when the agent was building a tables. to mitigate this we 1) stripped unimportant fields and 2) split tool calls into summary and detail.

this led to a reduction in ~2k tokens used when generating a response.

be mindful of where deterministic code is better then LLM

initially i had the agent writing out the .md syntax to build a table of data it received from an api.

later in the process, we realized it was not consistently rendering the table in the same way, and it would make more sense to make table generation programmatic.

replacing LLM reasoning with programmatic code helps reduce token output, and provide a more consistent user experience.

how the agent works

say a user sends a simple query like:

"how much did we spend on payroll in the last 12 months?"

below is the flow for how our system answers this question.

1. sending message to bedrock

agentService adds the message to history, and sends it over to bedrock.

• bedrock is an api service aws runs for accessing foundation models. it ensures data is not used for training, and is model-agnostic.

we used amazons nova pro model, since it appropriately balanced cost with quality.

the model reads from our tool_config.json on every request — a list of available tools and what they do:

{
  "tools": [
    {
      "toolSpec": {
        "name": "getSpendSummary",
        "description": "Returns aggregated spend totals or transaction counts over a date range, optionally grouped by month. Response includes: dateFrom, dateTo, aggregate......",
        "inputSchema": {
          "json": {
            "type": "object",
            "properties": {
              "dateFrom": {
                "type": "string",
                "format": "date",
                "description": "Inclusive start date in YYYY-MM-DD format."
              },
              "dateTo": {
                "type": "string",
                "format": "date",
                "description": "Inclusive end date in YYYY-MM-DD format."
              },
            },
            "required": ["dateFrom", "dateTo", "aggregate"],
            "additionalProperties": false
          }
        }
      }
    },
    //additional calls listed
  ]
}

the model then selects which tool to use based on description, and formulates the name of the tool, and any input parameters:

{ 
	"name": "getSpendSummary", 
	"input": {
      "dateFrom": "2026-01-01",
      "dateTo": "2026-03-31",
      "aggregate": "SUM"
  }
}

2. invoking the tool

this is sent back to agentService.

then, we execute methods from the tool list(in this case getSpendSummary) in toolCallService. the methods do a mix of:

1. calling external apis
2. calling internal apis
3. directly querying data in our databases

3. sending data back to the model

after getting a response, the toolOrchestrator passes the tool result through agentService back to the model via bedrock.

4. model determines visualization

the model then takes the response, and decides whether or not we should use a visualization and which type.

then it generates a vega-lite spec:

ConverseResponse(
      stopReason = "end_turn",
      message = ConversationMessage(
          role = "assistant",
          content = [
              ContentBlock.Text("""
                  Over the last 24 months, your company's total payroll was $523,951 USD.

                  <visualization>
                  {"$schema":"https://vega.github.io/schema/vega-lite/v5.json",
                   "mark":"bar",
                   "data":{"values":[
                     {"month":"Jan","amount":50000},
                     {"month":"Feb","amount":45000},
                     {"month":"Mar","amount":55000}
                     ....
                   ]},
                   "encoding":{
                     "x":{"field":"month","type":"nominal"},
                     "y":{"field":"amount","type":"quantitative"}
                   }}
                  </visualization>
              """)
          ]
      )
  )

vega-lite is a declarative library that renders charts from JSON specs.

5. frontend renders response + visualization

then, the frontend renders the text and the visualization using the vega-lite rendering library. boom - done.

a note on error handling

there were many possible errors that could occur through the flow, here's how we thought about it:

1. model outage (Bedrock down)
   1. bedrockClient.converse() throws an exception which bubbles up to the controller and sends an SSE event to the frontend which then shows to customer "there has been a model error, check back again later"
2. tool call fails (API/data error)
   1. the orchestrator catches the exception, returns {"error":"..."} with isError = true back to Bedrock as a tool result, and the model responds conversationally, e.g. "I wasn't able to retrieve that data."
3. Unknown tool
   1. Model hallucinates a tool name that doesn't exist in the handlers map — returns {"error":"Unknown tool: foo"} with isError = true.

by end of day 10 we had a working copilot: customers could ask natural language questions about their payroll, invoices, contracts, and payments, and get back answers and visuals(when relevant)!

a strong friend hits all 3 of these categories:

1. common thread
2. trust
3. comfort

common thread

maybe you both grew up in queens and were raised muslim, or you both read harry potter as a kid. shared context is ideal here

i'd argue you can find a common thread with anyone. but, this is the key to starting a relationship. if you don't have anything in common, its harder to converse and develop kinship.

trust

trust is the foundation of any relationship. you should not be second guessing if they care for you.

if you are reading this and aren't sure if someone you consider 'friend' fits this criteria - i got news for you: they aren't a friend.

comfort

you shouldn't feel obligated to conform to their preferences. if anything, you should feel emboldened to be yourself. to share your darkest secrets, with no fear of judgement.

its natural to behave differently depending on the friends. but if you dont feel comfortable sharing your dreams, faults, or problems with friends, then something is wrong.

vibes (i lied, theres a 4th)

the intangibles. how you feel when you are with them. things you cant cleanly categorize.

but, one problem i have is that the conversation history is not saved. theres no way to track my learning overtime.

so, i built diaHistory, a tool that automatically archives your diaConversation history using the macOS accessibility API.

How it works

see the full excalidraw here

the architecture is simple:

1. a while loop in chatWatcher polls to check if dia's is running(PID lookup)
   1. this changes every single time you launch the app
2. once we have dia's PID, we run a discovery poll every 15 seconds to scan the ax tree to discover any new chat panels with a message.
3. when a panel is discovered, we register an AXObserver to listen for new messages
4. whenever a message is sent/received - it goes through the whole pipeline:
   1. accessibility reader gets the conversation that changed.
   2. chatparser turns it into the message object
   3. we append it OR create new convo in conversation tracker
   4. THEN we write markdown file and save to documents

macOS accessibility tree

initially i hoped this tool could be a browser extension. but, dia's databases are (of course) encrypted, and dia's chat UI is native, meaning its not part of the DOM at all.

but, hope was not lost, the macOS accessibility tree ended up being what we needed. macOS has an accessibility API that lets you read the UI elements of any app.

its the same system that the voiceover feature uses for screen readers - every text field and label gets an accessibility role that describes what it is and what it contains. for example:

AXSystemWide
  ├─ AXApplication "Dia"
  │  └─ AXWindow "Claude — claude.ai"
  │     ├─ AXToolbar
  │     │  └─ AXComboBox "Address"
  │     │      .value = "https://claude.ai/chat/abc-123"
  │     ├─ AXWebArea (webpage DOM — skipped by diaHistory)
  │     │  └─ ...
  │     └─ AXScrollArea (Dia's native chat panel)
  │        └─ AXList
  │           └─ AXList
  │              ├─ AXGroup
  │              │  └─ AXStaticText "what is 2+2?"
  │              ├─ AXGroup
  │              │  └─ AXStaticText "4"
  │              └─ AXGroup
  │                 └─ AXStaticText "how many r's in strawberry?"
  │
  ├─ AXApplication "Messages"
  │  └─ AXWindow "Sarah"
  │     └─ AXSplitGroup
  │        ├─ AXScrollArea
  │        │  └─ AXTable
  │        │     ├─ AXRow "Sarah"
  │        │     └─ AXRow "Mom"
  │        └─ AXScrollArea
  │           └─ AXList
  │              ├─ AXGroup
  │              │  └─ AXStaticText "hey i had a great time last night!"
  │              └─ AXGroup
  │                 └─ AXStaticText "i think we should just be friends"

to make sure the daemon ran anytime the computer restarted, or dia changed, we made the tool a macOS launch agent.

Problems faced and how we solved them

no "new panel" AX notification exists
1. the AX Observer only had generic "AXValueChanged" events that fire on any change in the application:
1. tab switch, keystroke in address bar, etc.
2. even if we did decide to filter all these specifically for new chat events, sometimes these notifications can fail silently.
3. fix: set up a poller to check for new populated chat panels every 15 seconds, THEN we register AXObserver to listen for new messages.

the findChatGroups function was doing a recursive walk of the whole AX tree, causing a stack overflow on deep parts of the tree:

old:

func findChatGroups(in element: AXUIElement) -> [AXUIElement]? {
      for child in children(of: element) {
          if role(child) == "AXScrollArea", let groups = checkForChat(child) {
              return groups
          }
          if let groups = findChatGroups(in: child) {  // recurse into everything
              return groups
          }
      }
      return nil
  }

here the old version recurses into every single node, including the rendered DOM.

to fix, i added a depth limit and skipped the DOM:

func findChatGroups(in root: AXUIElement) -> [AXUIElement]? {
	var stack = [(element: root, depth: 0)]

	while let (element, depth) = stack.popLast() {
		guard depth < 64 else { continue }          // depth limit
		for child in children(of: element) {
			if role(child) == "AXScrollArea", let groups = checkForChat(child) {
			  return groups
		  }
		  guard role(child) != "AXWebArea" else { continue }  // skip web DOM
		  stack.append((child, depth + 1))
	  }
  }
  return nil
}

every AXObserver notification triggered a full rewrite to a new file.

1. this meant as a conversation continued, more and more files built up.
2. fix: fingerprint each conversation with a SHA256 of the domain + first user message and stores the file path it was written to.

memory bloat

1. a week after using the tool myself, i discovered that the daremon was eating 1.2GB of memory.
   1. since i had started the daemon, each conversation was being stored as state, with no way to way to evict.
2. fix: added a lastUpdatedAt timestamp to each record and scheduled a repeating timer that prunes stale convos every 24 hours.

ive been using this tool as part of my learning loop and its been marvelous. a future where our agents have context on everything we do is coming - and maybe the AX tree will be a part of it.

i hope this post encourages one person to build a more ambitious project using macOS AX tree!

you type a word, and you get a beautiful particle image generated.

• i.e. 'love' would show the image below.

the below follows what i did and lessons learned along the way. if you are also planning to fine tune an SDXL with LORA to do abstract art, this should save you 10 hours+ of iteration.

high level architecture

see the excalidraw here for more details

setup

training framework: diffuser's train_dreambooth_lora_sdxl_advanced.py — hugging face's advanced dreambooth + LoRA + TI training script for SDXL.

• TI: creates new token embeddings with no prior meaning
• LoRA: small trainable matrix we tack onto the SDXL transformer
• dreambooth: is a fine-tuning approach to bind a trigger token to the style you want. here we combine it with LORA + TI

compute: Modal — runs on an A10G GPU (24GB VRAM). $30 free credits(as of mar '26), which was more than enough for training + inference.

setup steps:

1. install uv (python package manager)
2. uv run modal setup — authenticates your modal account
3. clone the repo: git clone https://github.com/aaronw122/particleArt
   1. or create ur own, up to you :)

training data format:

images/curated/
  daily_life_001_v0.png        ← 1024x1024 training image
  daily_life_001_v0.txt        ← caption: "a figure lying on their side, curled in a sleeping position"
  gesture_and_action_005_v0.png
  gesture_and_action_005_v0.txt ← caption: "a figure bending down to pick something up"

each .png gets a matching .txt file with a scene description.
use an image generator (GPT, Midjourney, Stable Diffusion) to create training data. you may only find 10% of images are acceptable, so generate a LOT to choose from.

run training:

uv run modal run train_modal_adamw.py

download weights:

modal volume get lora-output-adamw-v4 /results/ ./lora_output/

run inference (generates images from multiple checkpoints):

uv run modal run generate_modal_sweep.py

principles

1. generate as much high quality training data as possible

for something abstract, 30 images isn't enough. you want at least 50, if not more.

see below for how to generate training data

2. use gpt OR stable diffusion to create an image prompt for generating training data

by doing this, you also prove to yourself if you even need to fine tune the model. ask yourself if the model consistently produce what you need with a prompt?

if the answer is yes, then you don't need to fine tune.

here's the prompt i used in chatGPT to generate the images:

• "Sparse black particle flecks on a pure white background forming {scene}. Minimal, lots of negative space. No gray tones, no shading — just scattered black dots/flecks suggesting the form. Abstract, not realistic."

but, only 10% of the images met my standards.

since the hit rate was so low, i tried to get GPT to hone in on the exact style i was looking for:

• "Sparse black particle flecks on a pure white background. Subject: an abstract human figure {scene}. Only small scattered dots — no lines, no strokes, no outlines. Approximately 200-300 small dots total. Keep 70% of the canvas empty white space. All elements — figures and objects — equally sparse and light. No element bolder or denser than any other."

this(and a couple iterations after) actually produced fewer acceptable images.

some prompts need to stay vague for now, as gpt doesn't yet have the lexicon to appropriately generate images from.

3. use TI(textual inversion)!

Textual inversion adds new embeddings to the lookup table, that start with zero prior meaning.

similar to LLMs, SDXL has a lookup table that maps tokens to vectors:

at the bottom, we add two(or more) fresh rows with random vectors that can be trained during fine tuning:

these two token weights get adjusted during fine tuning for a fraction of total steps (controlled by TI frac — in my case, 0.5, so the first half of training). then they freeze and the LoRA matrices continue to update.

• in training, these tokens are mapped to the trigger word TOK. whenever the fine tuned model sees TOK in a prompt, it activates the freshly trained style embeddings.

in sum:

• without TI: the text encoder has no way to represent 'particle figure art' as a concept.
  • then the LoRA net we trained has to fight against existing meanings of a word and generate image
• With TI: brand new embeddings are created and meaning is assigned to the word.

4. you will likely need to iterate on learning rate, rank, TI frac, and other params before you get the right outcome.

here is a table with the params i finalized through many iterations with codex and claude:

rank: determines the size of the LORA matrix we staple onto SDXL, the greater the rank, the more capacity to capture style nuance. BUT also more risk of overfitting.

this was the outcome of 5+ rounds of iteration on the training runs. each of them i ran into different issues and had to optimize.

5. modal is best rather than running locally OR using google colab

i frequently ran into rate limiting issues with google colab, and there was not enough VRAM on Colab's T4 GPU (15GB) for SDXL LoRA training. Modal's A10G was much better (24GB VRAM). see setup section above for how to get started.

6. develop consistent evaluation criteria before any model training

1. create a testing suite for tracking progress.
   1. i would recommend 1-2 novel prompts that were not part of the training data to ensure the style generalizes well. then 4-5 more to stress test range once your confident in the style.
2. add checkpoints in training and look at images at each one.
   1. every 100 steps is usually a good baseline.
   2. checkpoints also ensure you can resume training if something fails OR go back to older weights.
3. run for more steps than you expect — it is not as simple as text models where you can optimize for loss. you care about perceptual qualities that loss can't capture.
   • 
   • a loss curve like this is normal
4. evaluating image models is more subjective, i evaluated myself with criteria i built up overtime:
   1. does the overall style match what i want?
   2. is the background correct? (mine kept drifting off-white)
   3. are colors accurate? (got blue dots instead of black)
   4. is the rendering style right? (3D when i wanted 2D)

7. use negative prompts at inference

negative prompts steer the model away from unwanted qualities. here's what i used at inference:

prompt: "TOK, a figure [doing X], white background"
negative prompt: "photorealistic, detailed, shading, gradient, gray, color, dense, beige, tan, sepia, parchment,
  warm tones, blue, colored dots, 3D, lighting"

note: negative prompts are only used at inference, not during training. training captions are just scene descriptions we showed above.

8. claude.md for the project you can use.

https://github.com/aaronw122/particleArt/blob/main/CLAUDE.md

brief summary of my journey:

1. generated + curated sample image data.

• this was a <10% hit rate. had to generate >300 images to produce 30 samples

2. tried fineTuning using LoRA + textual inversion(TI), produced mid images

selection criteria was "does it match the aesthetic i am going for in the image above"

3. claude/codex said the model overfit at 500 steps, and we should stop using TI

1. i regretfully followed their advice, and did pure LoRA. this produced disastrous outcomes:
   

4. switched back to using TI, ran for 1000 steps with a rank of 32

some of the images generated had the wrong background. this was because 32% of my training data had a slightly off-white background.

5. train again on 2000 steps with the images fixed so they all had white backgrounds

model quality peaked at 1300 but wanted more out of it

1300 was identified as peak. see the white splothees in some areas.

6. curated 20 more images for a 4 hr training run, ran it at 2am on March 14

unfortunately i had a timeout set for 2 hours in my code, so it did not complete

7. re-ran with a checkpoint at halfway

this felt good.

8. better results, but the model was still pulling from its base styling:

see how the figure here is 3d instead of 2d

to resolve the issue, i worked with codex and increased the rank so it would capture more nuanced style differences.

9. increased rank was better, more 2d, BUT was using blue dots instead of black/grey

worked with codex/claude and identified two fixes:

1. added noise_offset (0.0357) to fix diffusion's luminance bias — without it, the model couldn't produce true white/black.
2. BF16 instead of FP16
   1. BF16 is more stable for extremes like pure white + black dots

10. images finally looking good at step 1800!

Appendix:

• github: https://github.com/aaronw122/particleArt

1. market
   1. who are you building for?
   2. should you pivot? 
   3. what is your products core value prop?
2. ideas/features
   1. evaluating good vs bad ideas/hypotheses
   2. is this feature worth building out? does it solve a painful problem? will it help enough of our customers? 
3. execution
   1. making tradeoffs during development
   2. asking yourself - is this experience intuitive, easy, good? 

endless time spent browsing amazon, but they just don't have the laundry basket you're looking for.

soon, the perfect anything will be at our fingertips.

instead of settling for something less than, you will find that perfect wicker laundry basket.

but shopping is just the tip of the iceberg. things you do in the physical world will be more curated as well.

imagine a world where it where novelty is effortless. you don't have to settle for the usual restaurant, but are nudged to try something new.

personally, a world where ai pushes us more out of our comfort zone couldn't come sooner. but, i do also wonder - will we miss the hunt? there's something cold about getting the best burrito of your life at a joint an ai recommended.

for example:

1. What are some of the downfalls of this strategy?
2. Can you walk me through the design decision you made?
3. What is the problem we are solving here?

this does 2 things:

1. Helps hone your own thinking on topics. a lot of times you may be wrong, and if you actually think through questions you have, your thinking will improve
2. forces the person to think through your questions and realize potential reasons a different approach/solution may better.

approach things from curiosity, not disagreement. otherwise, you'll be wrong much more often.

these 5 lessons were scrawled in an old college notebook:

1. don't work with close friends unless you trust them well.
   1. i would be working on the project, they'd be sleeping in. this made it difficult to hold folks accountable, and led to some resentment.
2. don't bring on more than one person to start
   1. i decided to bring on 4 friends. this made it difficult to make quick decisions, and slowed the flow of information.
3. agency > smarts
   1. i picked co-founders based on smarts, rather than their willingness to get shit done.
4. be decisive - pick a lane, test, iterate.
   1. i kept thinking if i read one more research paper on polarization, all the dots would connect and i'd have the perfect solution. this lead to indecisiveness, and a perfectionist mindset.
   2. it took us all summer to decide to pivot from an app to a newsletter. by then, we should have shipped 12 posts.
5. build something people want
   1. people don't want to read things they disagree with, but yet i was stuck on the news aggregation idea for far too long.
   2. it would have behooved me to ask 'is there a fun way we can help people be more aware of their cognitive biases?'