Elasticsearch, your website and you..

With all the projects, skills and other potpourri on this site there is no way people are going to get their questions answered.

Even I find this site hard to navigate because of the amount of information here, what do you do about that? Go overboard AGAIN of course and use Elasticsearch. SMH. (The first time I went overboard was setting up an AI chatbot to answer questions about this site)

The Elasticsearch implementation on this site provides a search feature that allows users to query content across various pages (such as blog, projects, skills, industries, etc.).

Here’s a terse explanation of what I did:

The first thing of course is installing Elasticsearch. It runs in Java and mercifully the install package from the web includes the bits of the JDK necessary to run it. Now, ideally you’d run it in a docker container, under Unix. That didn’t happen of course. Since everything here runs in Windows, why not install it on the same server where the back-end services that run this site do. It runs on the same VM as the services.

Step 1: Index Everything

Now that that its installed and running, its time to get the indexes built. Crawling this site with all of its dynamic content would be a big mistake. I tried this when getting RAG/AI working here and it failed miserably so I went with a curated approach and did it manually.

I created arrays of “facts”…
For skills: “Skill: C# Project: Petron”
Projects: “Project Petron: Skill C#”
Bullet Points: “Created something awesome in C#”

I sliced and diced everything out of the database along with links to the site. I then added them to the search engines index.

Step 2: Proxying

Theres no way I’m going to have the site directly access Elasticsearch, instead I wrote a proxy API that provided a way in, safely. (Safer might be a better word).. In this case I distilled everything down to two functions:

Search (string Term, int Limit)
Health()

Now, there are other functions on the backend API that load the index from the database etc.. Explaining that is scope creep. Nah, ask me and I’ll explain it.

Step 3: Implementation

The UI on this site is pushing 20 years old. SMH. In 20 years I haven’t changed it substantially. Its still Bootstrap beta 0.1 (joke) and utilitarian at best. No apologies for being a minimalist. I digress. That said I have not re-written it in Angular or React. Like many companies you make do with what you have, as long as its not completely trash. Yeah, it could be better.

At any rate so at 1 second intervals as you type it starts querying Elasticsearch and it returns answers and clicking on what it responds with will take you there. The search icon is fading in and out for a reason.. Try it.

Elasticsearch is used to power a robust, user-friendly search experience across the site, with both frontend and backend components, real-time updates, and (not so) advanced UI features. The implementation is well-documented and integrated into multiple pages for consistent user access.

Scroll up. Click the magnifying glass. See what happens.