Search engine optimization is something that website developers pretty much have to perform these days, in one form or another. Unless you can make sure your website ticks off all the boxes a search engine looks for, you’ll have to rely on much less reliable ways to improve your visibility, such as internet ads, word of mouth, and a social media presence. And if you can’t do any of those things, your site will languish in online obscurity for all time. But why exactly do search engines look for the items they do?
The First Search Engines Appear
Back when the internet was young and mainly consisted of a few college websites designed by and for student projects, the number of internet addresses was small and they were mostly passed around by the people who frequented the first message boards, the ancestor sites of places like Facebook and Twitter. But as the number of websites began to grow exponentially, it soon became clear that internet users would have to rely on a search program if they wanted to find what they were looking for.
Yahoo! was one of the first search engines to try and fully categorize the growing World Wide Web. Their big idea was to create an online directory, a sort of yellow pages for the internet that consisted of a bunch of nested folders that a user could search either with the automatic function or by checking each directory manually to find websites that matched what they were looking for. Also, much like the actual Yellow Pages, Yahoo! would demand that commercial sites pay them for the privilege of being on the list. These days, however, the company has mostly moved on to providing other services.
WebCrawler came out in 1994, and it was one of the first major search engines to use spider bots to look for and index websites instead of gathering them into directories. This meant that WebCrawler could search farther and scale more easily with the growing size of the internet, but it also meant it was easy to “game.” Since WebCrawler listed sites based on the key words you typed into the search bar, a malicious site could put a common search term on a page a dozen times just to trick online users into visiting the site.
The Growth Of Sophistication
As the number of websites using black hat SEO grew, the search engine programmers had to respond so that people would continue to use their site to find relevant web pages and not jump ship back to the curated directory sites. That’s why they began including certain conditions and adding extra requirements for a site to get a high rank on any given search engine.
One search engine that led the way in sophistication was AltaVista, which would later die out at around the time Google first showed up. AltaVista made searching easier for visitors by accepting natural queries like “What’s the best dog food?” instead of something like “’dog food’ PLUS best,” which is easier for a program to read but harder for a human to understand.
They were also the first search engine to include advanced search tools like “must have this term” and “don’t include that word.” These changes made it easier for guests to find what they were looking for, but it made it did nothing to stop malicious sites.
Ask Jeeves (which is now Ask.com) tried to split the difference between spider bot searches and directories by having human editors compile site lists for popular queries. However, the technology that powered their searches wasn’t much better than usual.
When Google finally showed up, one if its biggest contributions to search engine design was something the company called PageRank. By this point, search engines were looking for more than just key words to weigh their ranking systems, but PageRank added the number of outside links to that consideration. In other words, the more people linked to a website, the higher up it would appear on Google’s ranking.
As with everything else dealing with search engine optimization, this could also be exploited by black hat web designers by creating several fake sites that all lead to each other, but at least now it was much harder than before.
Where SEO Stands Today
The goal of a search engine and its spider bots is to find the websites people are looking for and to list them in order of true relevance based on the queries they type in. Thanks to the internet’s continued exponential growth, search engines are more important now than ever before, and they need to hunt through incredible amounts of information in an instant in order to accomplish all that. It’s no wonder that information parsing and search engine companies are on the cutting edge of artificial intelligence design.
But because these companies are refining their engines, and because black hat SEO techniques almost never lead to people sticking around and returning to a site later, white hat SEO tricks are becoming better able to reach the top of the rankings where they belong.
For reference, white hat SEO means optimizing a site by offering a constant stream of new, unique content that’s on point with what the site is for, and by sprinkling in a few keywords (such as “search engine optimization” for this post) that let a search engine bot understand what a certain page is all about.
There’s a lot more you can do to optimize your website for search engine bots, but hopefully you now understand a little better why SEO is as complex as it is today and why good content is better in the long run than gaming the system.
