Vector memory that remembers when
An agent rarely remembers the exact words. The user says “that team trip to Portugal”, and the memory says “met Alice at the Lisbon offsite”. Text recall, which tiramemsu already has, matches words: it finds “Lisbon” but not “Portugal”. Recall by meaning needs embeddings, vectors that put similar texts close together, and a search for the nearest ones.
Every vector database does that. The hard part for tiramemsu is that its memory travels in time. You can ask what the agent knew last Tuesday, what was true in March, or what one session believed. A vector search has to give the right answer to those questions too, and the usual way of building one does not. This article explains why, and how vector recall in tiramemsu (storage format 5, released in 0.5.0; writable from SPARQL and Cypher since 0.6.0) is built to avoid the problem. The engineering details are in docs/vector-architecture.md.
Why agents recall by meaning
An agent's memory fills up with notes, observations and facts written in many different words. When it needs something, it starts from the conversation, not from a graph pattern. Words get it part of the way: names, places, product codes. Meaning gets it the rest: paraphrases, synonyms, other languages, the same idea said another way. In practice an agent wants both, and wants to know which one found each result.
In tiramemsu the unit of recall is a statement. Each fact, note or layer has its own id, so a hit is not a loose chunk of text but a fact the agent can follow into the graph: who said it, how confident they were, what it was later corrected to. Vector recall keeps that. A vector belongs to one statement, and a hit is that statement with its evidence.
Why time travel breaks approximate indexes
Fast vector databases use an approximate index (HNSW, DiskANN, IVF). You ask for the 10 nearest vectors and get them in a few milliseconds. Then you apply your filters.
In tiramemsu the filter is the view. now() hides retracted and superseded statements. as_of(t) hides everything written after transaction t. valid_at(d) hides facts that were not true on date d. A graph scope hides everything outside the graph. Nothing is ever deleted, so the store holds every old version of every fact, and in a long-lived memory most rows are old versions.
Here is what happens when you search first and filter second. The agent asks “where does Alice work?” as of March 2024. The index returns the 10 nearest statements:
rank statement visible in valid_at(2024-03-15)?
1 alice worksAt Globex (2025-) no, not valid yet
2 alice worksAt Initech (2024-01..06) yes
3 alice worksAt Globex superseded no, old version of 1
4 alice worksAt Acme (2020-2023) no, ended
5 note: "Alice moved to Globex" no, written 2025
6 alice worksAt Initch superseded no, typo, corrected to 2
7 note: "Alice joined Initech" yes
8 alice worksAt Acme superseded no, old version of 4
9 note: "Alice leaving Acme" no, retracted
10 alice worksAt Globex superseded no, old version of 1
Eight of the ten are superseded, retracted or not valid at the asked time. Two are left. The agent asked for 10 and got 2, and the other correct answers were further down the index's list, never seen. Asking the index for 100 instead of 10 helps this time, but there is no safe number: the further back you go, and the more history the memory holds, the smaller the visible fraction gets. An answer that silently shrinks with the age of the question is the wrong kind of answer for a memory that promises time travel.
Exact first
So tiramemsu does it the other way round: filter first, then search. SQL selects the statements visible in the view, through the same function that every other read uses to turn “now”, “as of” or “valid at” into conditions on rows. Only their vectors are read. Rust computes the similarity of each one to the query and keeps the best. Then it reads the evidence (confidence, confirmations, authors, when it was added) for the hits only.
The search is exact. In the example above it returns 10 hits, all valid in March 2024, the 10 nearest among them. In as_of(t) it returns the 10 nearest among what the agent knew at t. The result says so: exact: true.
“What the agent knew” includes the vectors themselves. Each vector records the transaction that stored it, and as_of(t) sees only the vectors stored by t. If you embed an old note today, a question asked as of last Tuesday still does not find it by meaning, because last Tuesday the agent could not have. A past view therefore gives the same answer forever, and an answer saved on a past view stays valid. now() and the history view see every vector.
The cost is a scan. It is linear in the number of visible vectors, and there is no approximate index yet. For a memory of tens of thousands of statements this is not a problem; the measured numbers are below.
A side benefit is that it runs everywhere tiramemsu runs. It needs only SQL and Rust, no SQLite extension, so the same code works natively and in the browser on SQLite compiled to WebAssembly.
Where the vectors live
In the same SQLite file, in one table:
CREATE TABLE embedding (
model INTEGER NOT NULL, -- the model, an IRI
eid INTEGER NOT NULL, -- the statement this vector describes
t INTEGER NOT NULL, -- the transaction that stored it
bits BLOB NOT NULL, -- one sign bit per dimension
vec BLOB NOT NULL -- f32 or f16, unit length for cosine
) STRICT;
CREATE UNIQUE INDEX embedding_model_eid ON embedding(model, eid);
CREATE INDEX embedding_bits ON embedding(model, eid, bits); -- for the prefilter
One vector per model per statement. A model is an IRI, and its definition (dimensions, metric, encoding) is ordinary sys: triples on it, so the definition is versioned like any other fact. A vector is written in the same transaction as the statement it describes. Either both are there or neither is. There is no second system to keep in sync, and the agent's whole memory is still one file you can copy.
Who keeps the index
The engine does, the same way it keeps the text index. When you store a vector inside a write transaction, the engine checks that the statement is live, that the model is defined, that the length is right and every number is finite, normalises it for cosine, encodes it, computes the sign bits and inserts the row. You never update an index by hand, and a reader never sees a statement whose vector is half written. A failed transaction, a dry run or a speculation leaves no vector behind.
What the engine does not do is compute embeddings. Your application calls its model, of whatever kind, and hands the store the numbers. The store never calls a model and never sends your text anywhere.
In code
Rust. embed stands for your own call to an embedding model:
let model = v("model/minilm-768");
db.transact(TxOptions::default(), |tx| {
tx.define_embedding_model(model, &EmbeddingModel {
dimensions: 768, metric: Metric::Cosine, encoding: VectorEncoding::F32,
})?;
let text = "met Alice at the Lisbon offsite";
let e = tx.assert(v("alice"), v("note"), Value::str(text), Valid::ALWAYS)?.eid();
tx.set_embedding(e, model, &embed(text))?; // true: stored
Ok(())
})?;
let q = VectorQuery { limit: Some(10), ..VectorQuery::new(model, embed("team trip to Portugal")) };
let r = db.now().vector_search(&q)?;
for hit in &r.hits {
println!("{} {:.3} {:?}", hit.rank, hit.similarity, hit.evidence.confidence);
}
let then = db.as_of(TimeRef::Tx(150)).vector_search(&q)?; // what the agent knew at transaction 150
Storing the same vector again returns false and changes nothing, so replaying an import is safe. Storing a different vector for the same model and statement is an error, EmbeddingConflict. Vectors are history, and history is not overwritten. The same goes for the model: defining it again identically is a no-op, and defining it differently is refused (InvalidQuery), because the stored vectors depend on it.
Python:
with db.transact() as tx:
tx.define_embedding_model(model, dimensions=768, metric="cosine", encoding="f32")
e = tx.assert_(alice, note, "met Alice at the Lisbon offsite")
tx.set_embedding(e, model, embed("met Alice at the Lisbon offsite"))
for hit in db.now().vector_search(model, embed("team trip to Portugal"), limit=10):
print(hit.rank, hit.similarity, hit.evidence.confidence)
# statements that still need a vector, for example after a correction
for s in db.now().unembedded(model, limit=100):
...
In Python and Node a model can be a plain IRI string. Over MCP, an agent gets five tools for this: define_embedding_model and set_embedding to write, vector_search and recall to read, and unembedded to list the statements that still need a vector. In read-only mode the two that write are hidden, with every other writing tool. Node.js has its own methods: Tx.defineEmbeddingModel, Tx.setEmbedding, View.vectorSearch (which returns {hits, candidates, exact}), View.recall and View.unembedded.
SPARQL and Cypher reach the same search. In SPARQL, a group of tm:vector* patterns on one variable is one recall; tm:vectorMatch (the vector as a JSON array in a string) and tm:vectorModel are required:
SELECT ?e ?sim ?o WHERE {
?e tm:vectorMatch "[0.12, -0.4, …]" ; tm:vectorModel <urn:example:minilm> ;
tm:vectorSimilarity ?sim ; tm:vectorRank ?r ; tm:vectorLimit 10 .
?s ?p ?o ~ ?e }
GRAPH <g> restricts the recall and SERVICE <tm:asOf/…> times it. tm:vectorPredicate <p>, repeatable, keeps only statements with one of those predicates, before ranking. In Cypher it is a procedure; the model is an IRI or a short name resolved like a graph name, the options are limit, graphs and predicates, and the time clauses apply:
CALL tiramemsu.vector.search('minilm', $vector, {limit: 10, graphs: ['trip'], predicates: ['note']})
YIELD statement, subject, predicate, object, similarity, rank, confidence
Both compile to the same table function as every other surface, so they return the same hits in the same order.
Indexes and vectors from the query languages
Since 0.6.0 the query languages can do the writing as well. A vector index is the embedding model: creating one stores its definition, dropping one retracts it. Cypher has index commands and a write procedure:
CREATE VECTOR INDEX minilm IF NOT EXISTS OPTIONS {dimensions: 384, metric: 'cosine'}
MATCH ()-[r:visited]->(:City {name: 'Lisbon'})
CALL tiramemsu.vector.setEmbedding(r, 'minilm', $vector) YIELD stored
RETURN stored
SHOW VECTOR INDEXES
DROP INDEX minilm IF EXISTS
SPARQL Update writes the definition as the sys: statements it is stored as, and embeds a statement with tm:vectorEmbedding and tm:vectorModel, here asserting and embedding a note in one request:
PREFIX sys: <urn:tiramemsu:sys:>
INSERT DATA {
<urn:example:minilm> sys:dimensions 384 ; sys:metric "cosine" .
v:alice v:note "met Alice at the Lisbon offsite"
{| tm:vectorEmbedding "[0.12, -0.4, …]" ; tm:vectorModel <urn:example:minilm> |} }
INSERT { ?e tm:vectorEmbedding "[…]" ; tm:vectorModel <urn:example:minilm> }
WHERE { ?s v:note ?text ~ ?e }
DELETE DATA { <urn:example:minilm> sys:dimensions 384 } # drops the index
Dropping keeps the past: the vectors stay, a search as of a time before the drop still answers as it did, and creating the index again with the same definition brings them back. A dropped name cannot be reused with other settings. Every sample, in every language, is in the vector cookbook.
How hits are ranked
Similarity comes first. When two hits are equally similar, which mostly happens when several agents wrote the same thing, evidence breaks the tie: higher confidence first (no confidence last), then more confirmations, then more distinct authors, then the newer statement, then the id. This is the same tie-break text recall uses, and it has a name, tiramemsu-vector-rank/1, so that if it ever changes, the name changes with it.
Words and meaning together
Most of the time an agent should not have to choose. recall runs a text search and a vector search over the same view and merges them:
for hit in db.now().recall(text="lisbon offsite", model=model,
vector=embed("lisbon offsite"), limit=10, expand=2):
print(hit.rank, hit.text_rank, hit.vector_rank, hit.via)
The merge is reciprocal rank fusion: each statement scores 1/(60 + rank) in every list it appears in, and the scores add up. Only ranks are combined, because a bm25 score and a cosine similarity are not on the same scale. A statement that both searches find rises to the top. Each hit keeps its text_rank and vector_rank (absent when one search did not find it), so the agent can see why it is there.
Each search fetches more candidates than you asked for (by default max(4 × limit, 50)) before the merge, so a statement ranked low by one search can still rise when the other agrees. Equal scores are ordered by evidence, then by id. You can give only text or only a vector; a recall with neither is an error, and the text half needs the text index.
expand: 2 brings context along. For each hit it adds up to two statements about the hit itself (its confidence, its source) and up to two facts about the hit's subject. They come after all the hits with score 0, each carrying via, the hit it was reached from, so context is never mistaken for a match, and no statement appears twice. The order has a name too: tiramemsu-recall-rank/1.
The same recall works in SPARQL and Cypher, and gives the same hits in the same order:
SELECT ?e ?score ?via WHERE {
?e tm:recallText "lisbon offsite" ; tm:recallVector "[0.12, …]" ; tm:recallModel <urn:example:minilm> ;
tm:recallExpand 2 ; tm:recallScore ?score ; tm:recallRank ?r ; tm:recallVia ?via }
ORDER BY ?r
CALL tiramemsu.recall({text: 'lisbon offsite', model: 'minilm', vector: $v, expand: 2})
YIELD statement, subject, predicate, object, score, rank, via
Fusion, expansion and the evidence tie-breaks are explained in more depth in Words, meaning and evidence: hybrid recall.
Never forget, still
Embedding rows are append-only; triggers forbid changing or deleting them. That fits the rest of the store, and it has three consequences worth knowing.
- Retraction does not remove a vector. The view decides. A retracted note does not show up in
now(), and it shows up again inas_ofa time before it was retracted (and after its vector was stored), as it should. - A correction needs a new vector. Superseding a statement makes a new version with new content, and the old vector describes the old content. The store cannot recompute it, so it does not copy it.
unembeddedlists the statements that have no vector yet. - A new model means new rows. Switch to a better embedding model by defining a new model IRI and embedding again. The old vectors stay, and old questions asked against the old model still get their old answers.
A coarse first pass, if you want one
Each vector also stores one sign bit per dimension, 96 bytes for 768 dimensions. With prefilter: Some(n), the search first compares those bits (cheap XOR and bit counting), keeps the n closest, and computes exact similarity only for them. It can miss a true neighbour, so the result says exact: false. It is off by default.
The speed-up does not come from skipping big rows. A 768-number f32 row is about 3.2 KB, roughly one page, so the exact scan reads about a page per vector. The sign bits are also kept in a small covering index, about 100 bytes per vector, and the prefilter scans that index instead of the table, then reads only the n rows it kept. It still visits every visible vector, so it is a constant factor, not an index.
We measured what it costs in quality on real vectors: GloVe-100 word vectors from ann-benchmarks, 200 test queries, recall@10 against the exact answer (which matched a numpy brute force on 20 of 20 sampled queries).
n | Recall@10, 100k | Time, 100k | Recall@10, 200k | Time, 200k |
|---|---|---|---|---|
| exact | 1.000 | 44.5 ms | 1.000 | 89.9 ms |
| 200 | 0.544 | 28.5 ms | 0.515 | 58.1 ms |
| 1 000 | 0.785 | 30.8 ms | 0.754 | 60.6 ms |
| 5 000 | 0.945 | 46.0 ms | 0.913 | 72.5 ms |
| 10 000 | 0.979 | 92.2 ms | 0.961 | 96.3 ms |
The honest reading: scanning the bits alone costs about two thirds of an exact search, so the best speed-up is about 1.5×, and only when you accept losing three or more of the true ten. To keep 95 % of them you need about 5 % of the corpus as candidates, and then it is no faster than exact search. GloVe has only 100 dimensions, so only 100 sign bits; 384- and 768-dimension sentence models should do better, but we have not measured that. Exact search stays the default, and we recommend it.
How fast it is
Measured on an Apple laptop, in a now() view, top 10, with pseudo-random cosine vectors. These numbers measure the cost of the scan, not the quality of the answers.
| Statements | Dimensions, encoding | Exact | Prefilter 200 |
|---|---|---|---|
| 10 000 | 384 f32 | 7.4 ms | 2.4 ms |
| 10 000 | 768 f32 | 11.6 ms | 2.5 ms |
| 10 000 | 768 f16 | 11.0 ms | 2.5 ms |
| 100 000 | 384 f32 | 84.8 ms | 31.0 ms |
| 100 000 | 768 f32 | 134.4 ms | 31.3 ms |
| 100 000 | 768 f16 | 116.2 ms | 30.9 ms |
Exact search costs about 1.2 µs per 768-number vector, and the cost grows in a straight line with the number of vectors the view can see. A narrow view, such as one graph or an early as_of, scans fewer. The prefilter column is speed only; its quality is measured on real vectors in the section above.
Hybrid recall over 384-dimension notes with the text index on took 10.0 ms at 10 000 statements (15.2 ms with expand: 2) and 90.7 ms at 100 000 (112.4 ms). The vector half is nearly all of it: text alone took 0.49 ms and 7.6 ms.
How much room it takes
Measured on 10 000 vectors after VACUUM: a 768-number f32 vector is 3 072 bytes, but it adds 4 234 bytes to the file (4 105 in the table, 113 in the sign-bit index, 13 in the unique index). The row is about 3.2 KB, only one fits in a 4 KB page, and about a quarter of each page is left empty. A 768-number f16 vector or a 384-number f32 vector adds about 2 150 bytes, because two rows share a page. If file size matters, choose f16.
What comes later
Exact search is the base, and it stays the authority. Past roughly a million vectors, faster tiers will sit in front of it, each producing candidates that are checked against the view and re-ranked exactly:
- sqlite-vec, as an optional accelerator on hosts that can load extensions. Its tables would be derived from the embeddings and rebuildable, with a marker that says when they are behind, so a host without the extension can still write.
- HNSW snapshots, built from the table up to a known statement and kept as a cache outside the tiramemsu file. Newer vectors are scanned exactly. The snapshot only proposes candidates; the view and the exact re-rank decide, and when too few survive the view the search falls back to the exact scan.
- libSQL and its DiskANN index were considered and set aside; libSQL may come back later as an optional host, under the same rules.
- Prefilter quality on sentence models. Measured on 100-dimension GloVe only; 384- and 768-dimension models are next.
Honest limits
- No approximate index yet. Search cost grows with the number of visible vectors. 100,000 vectors of 768 f32 numbers add about 420 MB to the file (f16 about half), and an exact search over them took 134 ms on a laptop. Fine for an agent's memory, slow for millions of vectors.
- The prefilter is coarse. On real word vectors it is at best about 1.5× faster, and only while losing three or more of the true top ten. It is opt-in, and the result says
exact: falsewhenever it was used. - You supply the embeddings. The store checks the length, not the meaning. If the query vector comes from a different model than the stored ones, the similarities are meaningless and nothing will tell you.
- Corrections need re-embedding. Until you embed the new version of a superseded statement, vector search cannot find it. Text recall can.
- Late vectors do not reach the past. A vector stored today is invisible to
as_ofany earlier time, even if its statement existed then. That is the price of past answers that never change. - No overwrite. A changed vector for the same model and statement is refused. Improving your embeddings means a new model. Dropping an index hides it now but deletes nothing.
- Vectors are permanent. An embedding can reveal something about the text it came from, and like every row in tiramemsu it is kept.
Vector recall in tiramemsu is meant to be correct before it is fast: under every view, the hits are the nearest statements that the view can see. Speed comes in tiers on top, and none of them is allowed to change the answer without saying so.
← Recall, review, recheck · Vector cookbook → · Words, meaning and evidence: hybrid recall →