memory for agents is simple

a walkthrough of how memory is implemented for my personal AI agent.

when it came to memory for my agent, I had a few options, but I ended up rolling my own system. the deciding factor: I was accessing my agent through telegram and imessages (sendblue).

that setup meant the chat would be continuous and never-ending, so I handled it by capping a maximum message count and adding a clear_chat command for whenever I wanted to start fresh.

and since the interface was already uncommon, drop-in solutions like supermemory and mem0 just felt like they weren’t built for this.

---

my memory system consists of two tools that the agent can call whenever it wants:

• search_memories(query, k) called before responding whenever my message might relate to past preferences, facts, or context.
  
  the query describes what the agent is looking for (semantically) and a k for how many memories to pull. example: user says “write an email” → query=”email writing style preferences”, k=3.

• process_memory(fact) usually called after responding whenever I reveal anything worth remembering: a preference, a personal fact, a correction, or a stated habit.
  
  the agent is strictly told to NOT call this tool for transient requests like “what time is it” or things that don’t matter in future sessions.

---

searching memories

searching memories is pretty cheap. first, the query is embedded using OpenAI’s text-embedding-3-small. the embedding is then given to chroma along with the k.

chroma returns relevant results which are then neatly formatted for the LLM like this:

- Works as a software engineer.
- Prefers concise responses.
- Lives in Austin, TX.
- Has two kids.

processing memories

the search_memories tool is first called where the query is the fact. top 5 results and the new fact are given to a cheap model like gpt-4o-mini to perform any one of these 4 actions:

• ADD: new information not covered by existing memories

• UPDATE: existing memory needs to change (provide updated text)

• DELETE: existing memory is now wrong/contradicted

• NOOP: existing memory already covers this (skip)

the model then outputs a response like this:

{
  action: "ADD",
  id: "abc12345-6789-0abc-def1-23456789abcd",
  text: "User favorite color is red.",
}

based on the action, chroma is called to add/update/delete the respective memory document.

processing of memories is always done synchronously meaning that the agent never waits for a response. it just calls the tool whenever there’s something worth remembering and instantly moves on.

---

here’s an example of how this would work:

Rony: I'm planning on getting a new phone next month? Which one do you think I should get?

Agent: search("user phone preferences budget priorities past choices, 3)
Agent: If you’re already in Apple’s ecosystem, get the iPhone 16 or 16 Pro depending on budget; the Pro is worth it if you care about camera, display, and longevity. If you prefer Android, the Pixel 9 Pro is the best all-around choice.

Rony: Okay I'll get the Pixel 9 Pro.

Agent: process("user plans to get a Pixel 9 Pro next month")
Agent: Good choice - the Pixel 9 Pro has one of the best camera systems and cleanest Android experiences.

is my memory system perfect? nah it’s absolutely not. you should probably use something like supermemory or mem0. this memory system exists because I wanted something purpose-built for my agent and honestly, the system isn’t bad at all.

thanks for reading :)