thera copilot

for two weeks in april i had the pleasure of helping thera build an AI analyst agent.

thera's customers had to take painful steps to get questions answered - manually putting in long PDFs into chatgpt just to find out how much they spent on payroll in the last 6 months.

the idea was if we build an AI analyst agent, customers could get their data questions answered quicker.

video of product here:

https://www.loom.com/share/f15b3c861068426ab9d98c675f232cb4

our process

since there were two of us working on it, we split the work as follows:

  1. tool calls, function creation, logging (my pal conor)
  2. frontend, bedrock integration, agentController (me)

this meant we were only dependent on the data schema, everything else could be tested independently - function calls, agent conversations, etc.

we used the tracer bullet approach with the goal of getting one tool call working end to end, then expand.

week 1: we got the whole thing working end to end with one tool call.
week 2: we worked out kinks with model decisions, latency, and additional tool calls.

lessons learned

think hard about your database options up front

thera was using a noSQL DB for their data storage which limited our query patterns. for example, queries outside the partition/sort key like "how much did we spend on payroll in the last 6 months? give me a list of all employees and their costs." was much more expensive.

our options were to create a new centralized SQL DB in snowflake, OR get scrappy and build an MVP with tool call functions.

since we only had two weeks of time, we got scrappy, and built custom tool calls for common questions we knew customers would want answered. these tool calls queried internal/external API's and interacted directly with our noSQL db.

but this solution was brittle, and meant customers would have to ask for "feature requests" to get new questions answered.

if we had the time to store all the data into a centralized SQL db, our agent would have been much more flexible.

consider third party API limits

thera uses a variety of third party payroll and payment API's like check and MT. these API's are rate limited, which impacted agent response time. for example, modern treasury had limits of 20 req/s in production.

on a question like "how much did we spend on payroll in the last two years?", the model broke it down into 8 quarters, 3 currencies and 2 pages(8 x 3 x 2), which meant 48 api calls all at the same time.

this blew past our production rate limits. to fix this, we added semaphore limits on our function that uses MT, which capped the functions to 3 concurrent requests, which averages out to between 6 and 15 api calls.

if we had more time, centralizing the data in a new SQL DB at the time of writes would have made it much faster to get questions answered.

balance chat continuity with token limits

we encountered many input/output token limits throughout the project. the amazon nova pro model had an input limit of 300k and an output limit of 5k.

due to very detailed API responses, we often ran into output limits when the agent was building a tables. to mitigate this we 1) stripped unimportant fields and 2) split tool calls into summary and detail.

this led to a reduction in ~2k tokens used when generating a response.

be mindful of where deterministic code is better then LLM

initially i had the agent writing out the .md syntax to build a table of data it received from an api.

later in the process, we realized it was not consistently rendering the table in the same way, and it would make more sense to make table generation programmatic.

replacing LLM reasoning with programmatic code helps reduce token output, and provide a more consistent user experience.

how the agent works

Thera_blog_v3.png
say a user sends a simple query like:

"how much did we spend on payroll in the last 12 months?"

below is the flow for how our system answers this question.

1. sending message to bedrock

agentService adds the message to history, and sends it over to bedrock.

  • bedrock is an api service aws runs for accessing foundation models. it ensures data is not used for training, and is model-agnostic.

we used amazons nova pro model, since it appropriately balanced cost with quality.

the model reads from our tool_config.json on every request — a list of available tools and what they do:

{
  "tools": [
    {
      "toolSpec": {
        "name": "getSpendSummary",
        "description": "Returns aggregated spend totals or transaction counts over a date range, optionally grouped by month. Response includes: dateFrom, dateTo, aggregate......",
        "inputSchema": {
          "json": {
            "type": "object",
            "properties": {
              "dateFrom": {
                "type": "string",
                "format": "date",
                "description": "Inclusive start date in YYYY-MM-DD format."
              },
              "dateTo": {
                "type": "string",
                "format": "date",
                "description": "Inclusive end date in YYYY-MM-DD format."
              },
            },
            "required": ["dateFrom", "dateTo", "aggregate"],
            "additionalProperties": false
          }
        }
      }
    },
    //additional calls listed
  ]
}

the model then selects which tool to use based on description, and formulates the name of the tool, and any input parameters:

{ 
	"name": "getSpendSummary", 
	"input": {
      "dateFrom": "2026-01-01",
      "dateTo": "2026-03-31",
      "aggregate": "SUM"
  }
}

2. invoking the tool

this is sent back to agentService.

then, we execute methods from the tool list(in this case getSpendSummary) in toolCallService. the methods do a mix of:

  1. calling external apis
  2. calling internal apis
  3. directly querying data in our databases

3. sending data back to the model

after getting a response, the toolOrchestrator passes the tool result through agentService back to the model via bedrock.

4. model determines visualization

the model then takes the response, and decides whether or not we should use a visualization and which type.

then it generates a vega-lite spec:

ConverseResponse(
      stopReason = "end_turn",
      message = ConversationMessage(
          role = "assistant",
          content = [
              ContentBlock.Text("""
                  Over the last 24 months, your company's total payroll was $523,951 USD.

                  <visualization>
                  {"$schema":"https://vega.github.io/schema/vega-lite/v5.json",
                   "mark":"bar",
                   "data":{"values":[
                     {"month":"Jan","amount":50000},
                     {"month":"Feb","amount":45000},
                     {"month":"Mar","amount":55000}
                     ....
                   ]},
                   "encoding":{
                     "x":{"field":"month","type":"nominal"},
                     "y":{"field":"amount","type":"quantitative"}
                   }}
                  </visualization>
              """)
          ]
      )
  )

vega-lite is a declarative library that renders charts from JSON specs.

5. frontend renders response + visualization

then, the frontend renders the text and the visualization using the vega-lite rendering library. boom - done.

a note on error handling

there were many possible errors that could occur through the flow, here's how we thought about it:

  1. model outage (Bedrock down)
    1. bedrockClient.converse() throws an exception which bubbles up to the controller and sends an SSE event to the frontend which then shows to customer "there has been a model error, check back again later"
  2. tool call fails (API/data error)
    1. the orchestrator catches the exception, returns {"error":"..."} with isError = true back to Bedrock as a tool result, and the model responds conversationally, e.g. "I wasn't able to retrieve that data."
  3. Unknown tool
    1. Model hallucinates a tool name that doesn't exist in the handlers map — returns {"error":"Unknown tool: foo"} with isError = true.

by end of day 10 we had a working copilot: customers could ask natural language questions about their payroll, invoices, contracts, and payments, and get back answers and visuals(when relevant)!