Search Engines and Directories

Page Rank and Other Joys

Or:  Reasons to Hate Google

Well, not really. But Google certainly changed the search engine world with their PageRank algorithm. Basically, the idea was that a site with a lot of other sites linking to it was popular and therefore more relevant. I have always taken issue with the last part of that theory: popularity is not at all the same as relevance. None the less, Google itself turned that theory into a great deal of popularity -- for themselves.

Part of their success was likely the fact that the spammers were unprepared for this approach. It certainly took them a while to figure out how to get around it. See my Dubious Tricks page for some of the tricks used. As a result of some success with these tricks, Google has been forced to move away from strict PageRank to protect their market position. I'm going to cover what they are currently doing here, and maybe make a few predictions for the future, just so you can be prepared.

I'm concentrating most of this page on Google because they have the largest market share at the moment. Most of this will also apply to other search engines as well, so it's really worth doing anyway.

PageRank

Basically, and without resorting to the math here, PageRank is calculated based on the number of pages that link to you, plus the number of links that you have back to the same page. Note that this is pages, not sites. If there were only two web pages on the Internet this would be dead simple. Given the (roughly estimated) 5.5 billion pages that Google indexes, this is a major task. Few companies can afford the processing power needed to spider over 5,500,000,000 pages, then perform the complex calculations needed to rank pages this way.

The process is also complicated by a few other factors, such as link text and relevance. Google also considers whether the text in the link tags is relevant to the page being linked to. Don't worry, this is just background to explain why we are doing this. I'll cover the simple way to do this in the next few sections.

Internal Links

To get the best value out of your site, it has to have all of the pages linked to each other. Fortunately that is also good design practice. Did you see that tan bar running down the left side? (Well, it's supposed to be tan. Haven't you figured out that I'm colorblind yet?) Anyway, that bar contains convenient links to all of the pages in this article. Since it appears on every page, it effectively links every page to every other page.

This interlinking of pages spreads the ranking of your pages more-or-less evenly. It's a bit more complicated than that, but since I said I'm not going into the math here you'll just have to take my word for it. This is generally a good thing. You have a much better chance of being found if the search engines consider all of your pages to be valuable.

So now that you've seen the navigation trick, why not go steal the idea for your pages? You can have the color scheme too, and much luck to you. Better read the rest of this page first, though, because there are some more things you'll want to incorporate while you're at it.

External Links

Now here comes the fun part. Your site will never get a decent ranking unless you have other sites link to you, and have you link back to them. Don't panic; it isn't that hard. Here's what you do.

Suppose, for example, you have a genealogy site for your family. There are probably other genealogy sites out there that contain your surname. If you haven't already looked at them you probably should. It doesn't really matter if they are related or not, since they are on (generally speaking) the same topic. The topic is what matters here. So go make a list of these sites, then add in some links. See the next section for how to make your links in the most effective manner.

Now send an email to the owner/webmaster of each of those sites. Tell them that you have a site that contains one of their surnames. Tell them that you have linked to their site for additional information. Give them a link to the page where your link to their site is, and ask them to link back to you. If you think that they won't understand why you are asking, give them a link to this page as an explanation. Or write it yourself if you wish.

Now look for other sites that are related. For example, one-name sites, sites on how to do genealogy research, cemetery lists, military records, and anything else that you have used as a resource while building your site. Try to keep these as relevant to your keywords as you can. Then add some more links and send some more emails.

You don't have to do this all at once; just add a site or two whenever you find something relevant. Keep at it; this is really a case of "more is better." Just keep it relevant -- a site about fishing (unless it's by one of your relatives) is irrelevant to your family history site and will be ignored. Even if it is by one of your relatives, it's still not going to do you much good.

You will need to keep a record of who you have sent emails to. If you don't get a reply in a reasonable time, delete the link and move on. Or leave it up if you think it adds value for your visitors. Don't waste time trying to get a particular site to link to you. This is really just a numbers game. And I promised not to go into the math, so you'll just have to trust me on this one as well.

Link Text

Search engines like to think that your links are valuable and not just part of a link farm. The best way to convince them is to put the link in the text where it would naturally occur. So, if I was discussing dubious tricks for fooling search engines, I would put a link ... well, there. See? Not hard at all.

So, if my site were all about the Mulholland family, and one of the sites that I want to link to is about William MULHOLLAND; B. 11 Sept. 1855, Belfast, Ireland; M. 1890; D. 22 July 1935, then I would put the link in ... right there. (Yes, he's real. Some of my research. Nothing to see here. Moving along....) Of course, he's not very relevant to this page, so I really haven't done Family Search much of a favor with that link. Try to keep your links as much on topic as possible for best results.

Note that the link text (the part that you see in color/underlined above) is the name of the individual. If my page were about that individual, or at least partly, then his name would be in my keywords list. I want those keywords in the link text because the search engines want to see them there. So, when you are sending off your email to request a link, give them the text you want to see in that link. It's not a bad idea to give them the HTML code to just cut and past into their site. Couldn't hurt.

Needless to say, links containing text are much more valuable than links tied to a graphic. Search engines don't collect graphics, and likely couldn't read them if they did. Not that it's impossible, but the chances of this happening in the next year or so are small. You can use graphical buttons for site navigation if you wish, but put in some text links with your keywords down at the bottom of the page (for example) as well. They may just help some of your human visitors also.

The Google Bar

There is one more factor that you need to consider here: the actual page rank of the page that is linking to you. To figure this out, go to the Google website and find their Google Toolbar. No, I'm not going to link to them. They already have the highest possible PageRank; they don't need my help. Besides, they move stuff around far too often, and I'm already well behind in updating this site. Now that the download is finished, go install that toolbar into Internet Explorer.

Now open a page in IE and look at the little bar labeled "PAGERANK" in your shiny new toolbar. That's the only reason we got the thing, although it has some other nice features that you might like. The more green that is showing in that little bar, the better. If the page is poorly ranked (or just new) then the bar may be all white. If you look at google.com, you'll find it to be all green. (Should we be surprised?)

Now open each of the pages that you want to have link to you. And I do mean the exact page, not just the front page of the site. Unless you think that you can get a link from that front page, of course.

Look at the PageRank for each of those pages. The ones that have at least 40% green are now your top priority. That 40% translates to a PageRank of 4.0. Oops, there's some math. Sorry, my mistake. Anyway, as a rough rule, you want at least one page linking to you that has a PageRank of 4.0 or above, and six to ten more with any ranking at all. This really isn't that hard to do. Of course, having more links, or links with higher PageRank, is always a good thing. More is better, yet again.

The Future

Yes, I'm going to go off the deep end here and try some predictions. This isn't going to add any complexity to what you've already done, and it could really help. Since the risk is low, let's get at it.

We've seen that the search engines like link text, and we have already seen that Google (and possibly others) understand the roots of most words. What will likely come from this is ranking based on the theme of the site, or at least the theme of individual pages.

This is just a continuation of what I've already told you: keep your links relevant to the page they are on. A link to a florist's site in the middle of a page on your family history will be deemed irrelevant and (mostly) ignored. If your uncle was a florist, then you can write a page specifically about his shop and link that page to the shop's page. This is the only way that link will have any great value. This may mean that you need to rearrange some pages, or split pages up. Don't rush right out and start redesigning your site, but do keep this in mind for the future. It isn't extremely important right now, anyway.

Of course, to have the links focused on the theme of the page, the page itself has to have a specific theme. If you wander from one topic to another to another, the search engine is likely to consider your page to have little value. Since a lot of humans will make the same assessment, this is reasonably valid logic. Remember what I said about splitting up those huge, rambling pages?

There's probably not going to be a sharp dividing line between relevant and irrelevant; more likely some kind of sliding scale. Try to set a theme for each page and keep to it. The side benefit is that it will be easier for your human visitors to read as well. I personally have a hard time with huge pages full of a variety of information. I end up using page search to try to find what I came for, then leave quickly. Smaller, more focused pages I tend to read in their entirety. Not that you should take me as a representative sample.

The Search Engines

When I first wrote this article, there were half a dozen major search engines. Today, Google owns most of the field, and smaller engines have disappeared or just present somebody else's results. The number of paid advertising companies disguised as search engines has bloomed, but now seems to be fading. So here's what I would recommend you put your energies on:

Google

Still the king of the hill. I've slanted a lot of this article towards what they want. Part of this is because they still provide a majority of searches, and partly because everybody else is trying to emulate them. Or their success, anyway, which often amounts to the same thing.

Prediction: I've pretty much covered where I think Google is heading in the section above. I also expect that they will lose more of their market share, mostly due to external factors. Despite any losses, they will remain a major player for quite some time. It is still a good idea to plan on satisfying what they want for the foreseeable future.

Yahoo

Yahoo still relies somewhat on their human-generated directory. It's still a good idea to submit a personal site to them, and a business site if you can afford the fees.

Prediction: Yahoo will deemphasize their directory, largely because the number of pages available and the rapid rate of growth of the Internet have made it impossible to keep up. Their purchase of Overture, along with the Inktomi search engine, indicates the way they are going. They needed the paid advertising to stay profitable, but they also needed the search technology to remove their dependency on Google. With the decreased importance of the directory, search will become their mainstay. I can see why they would want to control that. And to stop paying Google for the use of theirs, of course. This should be interesting to watch.

MSN

They are currently a complete unknown. MSN has previously just republished the results of other search providers. They have obviously realized that they were ignoring a major market, and are now furiously working to bring up their own search engine.

Prediction: There's almost nothing to go on here. Not even a launch date, other than guesses of anywhere from July 2004 to sometime in 2005. I do know that the MSN spider has been wandering around since early February, at the very least. Judging from its behavior, they still have a lot of work to do.

When it does happen, I suspect that MSN will be pretty much following what works for everybody else. It's not impossible that they have come up with something that everyone else has overlooked, like Google did, but not likely. If you have built a good site that people would want to read, chances are that MSN will want to lead those people to your site. Still, since it's Microsoft, I expect that paid advertising will play a major part. This is certainly one to watch.

Next:

It's finally time to send out those invitations. Party time!