Experiences of The Many

What should I read to learn about the cold war? Is Bali a good place to visit in December? Where do I find good fanfiction? Has anyone else been automatically upgraded to this credit card? What are the tradeoffs in storing a browser session in the client versus on the server?

I used to enter questions like these into a normal search engine. Most of the top results would be blog entries or listicles, which led me to doubt the accuracy or motives of the authors. What I was really looking for were suggestions with some kind of “same for me” mechanism, indexed for search in a meaningful way.

Then I discovered site-wide search and the obstacles to answering hard questions greatly diminished. Site-wide search is a feature in many search engines that allows you to select only results on a specific website. Why use a website’s built-in search engine when search engines like Google, Bing, DuckDuckGo, etc. specialize in interpreting search terms into semantic meaning and searching their indexes for that meaning?

Initially, I only searched reddit.com. Reddit lets users share links in isolated, topical “subreddits”, but the strongest feature in Reddit is the commenting engine. It is so rich in features that many of the subreddits consist primarily of posts without links.

One Reddit commenting feature allows users to vote up or down specific comments. And we get to see the magnitude of the “upvotes” and “downvotes”; otherwise, it would not be a very useful feature. Site-wide searching in conjunction with this crowd ranking system is an effective way of sampling the experience of the many.

For a couple years thereafter, whenever I had a hard question, I consulted a site-wide search of Reddit. For example:

site:reddit.com cold war book

I added https://www.google.com/search?q=site%3Areddit.com+%s to my browser’s list of search engines as a shortcut. This got me 80% of the way there; however, some topics were still harder to answer, so I started searching more websites, like Hacker News (news.ycombinator.com) and Stack Exchange (stackexchange.com) for technical topics.

Eventually, I tried more websites that worked well for finance, travel, congitive science, law, etc., but it became unwieldy, until I found that site-wide search is built into the Google search engine: Custom Search (cse.google.com).

There is more work to be done here. It’d be nice to research how to correlate data across the different sites and potentially integrate review systems as well.

To hear more, subscribe to my newsletter or follow me on Twitter.