I built an FHE (fully homomorphic encryption) project with a very ambitious initial plan: the most “cutting-edge” design, the most secure approach, and the most technically impressive tools.
Then reality hit with its ecosystem constraints, cryptography constraints, and the deadline.
This post is about the pivots I made, why I made them, and what I learned about choosing “doable” over “perfect.”
The original plan (and why it was tempting) #
On paper, the design looked amazing:
- Build my own database layer on top of RocksDB (a popular embedded key-value store).
- Implement fast vector lookup using HNSW (a common “approximate nearest neighbor” search method) in Rust.
- Add memory-block shuffling to reduce access-pattern leakage (so reads reveal less about what you searched).
Pivot 1: The “cool stack” vs the ecosystem #
The first red flag showed up when I ran into bugs in the Rust FHE library I chose.
It was a Rust binding over C++, with only a small number of maintainers. Debugging it was slow, and more importantly, it made the whole project fragile. I picked Rust partly because it felt “cool and edgy,” but for this project, the ecosystem and support just weren’t there.
So I pivoted to the same old Python, for its practicality: better tooling, more examples, easier iteration, and a bigger ecosystem.
Pivot 2: HNSW vs what FHE can actually do #
The next red flag was HNSW itself.
HNSW (and most ANN search approaches) depends on repeatedly computing distances and making decisions based on comparisons (e.g., “is distance A smaller than distance B?”). But the FHE scheme I chose (CKKS) is designed for approximate math on encrypted real numbers, and comparisons are not straightforward to do efficiently.
I could have switched schemes, but my data is inherently floating-point (vector/matrix embeddings), which is exactly where CKKS fits best. Schemes like BFV are better for exact integer arithmetic, but they don’t map cleanly to my data without additional encoding and complexity.
Instead of HNSW, I settled on LSH (locality-sensitive hashing) a hashing that groups “similar” vectors into the same bucket so you only search a smaller subset.
Pivot 3: Memory shuffling vs the deadline #
The final pivot was dropping memory-block shuffling.
I underestimated how much latency and implementation complexity this would add on top of an LSM-tree based storage engine. I probably could have built it with enough time, but projects don’t run on “eventually.” They run on deadlines.
To keep the system simple and meet the deadline, I replaced shuffling with random decoy bucket reads. The idea is to avoid reading only the “true” target bucket, and instead mix reads across random buckets so the access pattern is noisier and harder to correlate.
What I learned #
These changes made the system much easier to implement, but it also hit my ego. I had already invested time and identity into the “perfect” version.
The dangerous thought was: “I can make it work if I just spend more time debugging.” That’s how you slide past deadlines.
Pivoting early matters. Time spent going deep on the wrong path isn’t just lost time—if this were a startup or a corporate feature, it’s also lost money.
Next time #
Looking back, these changes made my system much easier to implement, but they were a big hit to my ego and my initial research effort. I had to accept the real-world limitations of the deadline and implementation complexity. I was so into the dreamy idea of the perfect system that it was starting to hinder my ability to finish the project.
Pivoting early is a must because time spent on the wrong path is time you’re throwing away. If this were a startup or a corporate feature, it would be lost money as well.
Obviously, there are times when you have plenty of resources and time, and researching or implementing newer, cutting-edge technology is worth it. However, most of the time, choosing boring tools over new and focusing on the core problem over ambition is the best choice.
Being able to look at the big picture and knowing when to change course is a skill in itself.