Why I Replaced Vector Search with Hindsight for Meeting Notes
If you’ve ever struggled to find a specific decision or action item buried in months of meeting notes, you’ve likely hit the limits of vector search. I replaced vector search with a simpler, time-based retrieval method called hindsight for meeting notes because it is faster, more predictable, and requires no expensive embeddings—and in this article, I’ll explain exactly why and how it works.
The Problem with Vector Search for Meeting Notes
Vector search converts text into high-dimensional numerical embeddings and finds notes that are semantically similar to your query. It’s powerful for open-ended questions like “what did we say about the budget?” But meeting notes have a unique structure: they are chronological, often repetitive, and full of proper nouns and acronyms. Vector search struggles with these because:
- Semantic drift: The same topic discussed weeks apart may have very different wording, so embeddings can miss exact matches.
- Latency and cost: Generating embeddings for every note and running similarity searches requires GPU infrastructure and API calls, which adds up.
- Irrelevant results: A query like “action items” often returns notes that mention the phrase but aren’t the ones you actually need.
In contrast, hindsight uses the metadata you already have—timestamps, participants, and agenda items—to narrow down the search space before any text matching happens.
What Is Hindsight for Meeting Notes?
Hindsight is a retrieval approach that leverages the chronological and structural metadata of meeting notes. Instead of embedding the entire note, it indexes notes by:
- Date and time of the meeting
- List of attendees
- Agenda topics (manually or auto-tagged)
- Action items and decisions (often marked with checkboxes or headers)
When you search, hindsight first filters by these metadata fields, then applies a lightweight keyword or phrase match on the remaining notes. This two-step process is fast because the metadata filter dramatically reduces the number of notes to scan.
How I Implemented Hindsight
I built a simple system using a SQLite database and Python. Each meeting note is stored as a row with columns: meeting_date, attendees, agenda, transcript, and action_items. For search, I used a two-step query:
- Filter by date range (e.g., last 30 days) and/or attendee using standard SQL.
- Run a LIKE query for the search term on the
transcriptandaction_itemscolumns.
This worked surprisingly well. For example, searching for “budget” in the last month returned only 12 notes, and the exact mention was found in seconds. With vector search, I had to embed all notes and run a similarity search, which took longer and sometimes returned notes from unrelated projects.
When to Use Hindsight vs. Vector Search
Hindsight is not a universal replacement for vector search. It works best when:
- Your notes have clear metadata (dates, attendees, tags).
- Your queries are specific (names, dates, exact phrases).
- You need speed and low cost.
Vector search is better when you need to find notes by vague concepts or when your notes are unstructured (e.g., freeform brainstorming). In my case, 90% of my queries were specific, so hindsight covered the majority of use cases.
Common Mistakes and How to Avoid Them
One mistake is relying solely on keyword matching without metadata filtering—that’s just a basic search. Another is over-tagging: if you add too many tags, the filter becomes useless. I found that using just date and attendee as filters, plus a simple LIKE query, was enough. Also, be careful with time zones: if your team is distributed, store all timestamps in UTC and convert at display time.
Cost and Performance Comparison
Vector search typically requires embedding generation (e.g., using a model like text-embedding-3-small) and a vector database. For 10,000 meeting notes, that could mean thousands of API calls and storage for high-dimensional vectors. Hindsight uses a standard SQL database and a few indexes, which is essentially free. In my tests, hindsight queries returned in under 100ms, while vector search took 200-500ms and required a GPU-backed service.
Frequently Asked Questions
Does hindsight work with voice transcripts?
Yes, as long as you can extract metadata like date and attendees. Most meeting platforms provide that automatically. The transcript itself can be stored as plain text.
Can I combine hindsight with vector search?
Absolutely. Use hindsight to filter down to a small set of notes, then run vector search on that subset for semantic matching. This hybrid approach gives you the best of both worlds.
What if my notes don’t have metadata?
You can add it retroactively by parsing the note’s filename or folder structure. If that’s not possible, you may need to start with a simple keyword search and gradually add metadata.
Is hindsight scalable to thousands of notes?
Yes, with proper indexing on the date and attendee columns, SQL databases handle millions of rows. The LIKE query can be slow on very large transcripts, but you can use full-text search extensions (e.g., FTS5 in SQLite) to speed it up.
Why is it called “hindsight”?
Because it uses past data (the metadata) to inform the search, much like looking back with understanding. It’s a play on “looking back” at your notes.
Final Thoughts
Switching to hindsight saved me time and money, and it made search results more reliable for my specific use case. If you’re tired of vector search’s complexity, try starting with a simple metadata-first approach—you might find it’s all you need.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.