Tracking websites that are not on the first page

If you are trying to rank a new website, you know how tricky it can be to monitor progress. I've spent way more time trying to find my website on page 6 or 7 on google than I would like to admit.
There are a few tools out there that will do this for you. I found them however slow and full of ads. Even worse, they hide the actual submit button and make other buttons look like the submit button, tricking you to click on unwanted ads.

As a result, I decided to create my own, lightweight, fast, and clutter free tool to check keyword rank.

Much better, this is my kind of interfaces!

Simply enter your search keywords, the domain you want to find and the tool will show you the top 100 results within a couple of seconds, highlighting the domain you are trying to find. Nice and simple, just how I wish the existing tools would be.

Discovering backlinks within the top 100 search results

While working on this tool and looking at the list of results for one of my websites (ranking on the third page at this point), I spotted a blog post about said website ranking on page 8. That sparked an idea: Wonder how many backlinks the very first result on google has from within the top 100 search results for this search term? Could I simply crawl the 100 search result pages and search for backlinks pointing to that domain?

Elixir to the rescue

The answer of course is yes, it's actually quite trivial to do in Elixir which I happened to use for this. Elixir can run thousands if not millions of concurrent processes, so 100 concurrent http requests and crawling the content is a piece of cake. Elixir also has a great library to parse html called floki, which is used by default in Phoenix test suites, so finding links and even figuring out wether they are follow or no follow is also very easy.

Next I just push the discovered backlinks down to Elm using web sockets, and Elm simply updates the loading spinner and displays the results.

Turns out the first search result had backlinks from 7 other pages in the top 100, with 6 of them being really easy to create for my own website.

So yeah, if you are wondering why a website, be it a competitor or just out of interest is ranking so well for particular keywords, give the keyword-rank checker tool a go! It can help you not only understand why they are ranking as they are, it can also be used to discover potential backlink opportunities.