Marginalia Search

Marginalia started during the pandemic as a project to pass time — an indie search engine prioritizing text-heavy, non-commercial sites. Löfgren built his own crawler and index instead of wrapping Google or Bing.

Early designs targeted a Raspberry Pi cluster, forcing thrifty engineering. That constraint shaped code that still runs lean even after migrating to bigger iron.

When Hacker News discovered Marginalia, Löfgren’s “Against the Flood” post describes the moment: sustained load around two queries per second, half a million to a million visits — on a Ryzen 3900X with 128GB RAM humming in his living room next to a UPS.

No cookies except for his own admin tools, static HTML where possible, pages often under 10KB. A 2006-era router sat upstream. The ship floated because the design was thrifty, not because he bought cloud credits.

By late 2023 the index passed fifty million documents — up from ~28 million in February. Side-loading Stack Overflow alone accounted for much of the growth. Preconversion and transposition jobs that took twenty minutes needed tuning when mechanical disks fought for seek time.

Löfgren’s “Growing Pains” post is honest: MariaDB struggles at 218M URLs, ten-second page loads are a DoS vector, memory leaks need allocator swaps. He’d rather patch a home-built index than pay the “admittance fee” of hyperscaler architecture.