Search Engines and Directories

Introduction

This page is intended as a guide to the beginning webmaster. It covers the basics of search engines and directories, and how to get your web site listed on each. I'll also cover how to get listed in the right places, which could be even more important.

I'm going to try to keep this site accessible to everyone. Please be patient if you already know some of this; there's more good information coming.

Disclaimer: The actual methods that search engines use to rank pages is a jealously-guarded secret. Worse, they change frequently and without notice. There are ways to guess what criteria are being used at any given moment, but I'm not interested in doing that much work, and neither are most of you. Besides, I'd have to update this page every few days, and that's a pain. What I will tell you is the general (and easy) tricks that will always help, and the ones that will get you in trouble.

If you want to be listed Number 1 on all of the major search engines, you're looking in the wrong place. And you probably need to rethink why you're doing this.

Over 1 2 3 4 5 Billion Served

That's the whole problem: there are over 5 billion pages on the World Wide Web today. That is, 5 billion that Google admits to. Worse yet, more are being added at a furious pace. There are so many that there is no practical way to count them; we can only estimate. Without some sort of an index, it would be impossible to find anything. That's why Search Engines and Directories are so popular.

Directories are categorized listings, much like what you find in your telephone company's Yellow Pages. This familiar interface may explain why they are so popular with Web beginners. Directories are usually maintained by humans, and are therefore much harder to get listed in. They also contain a much smaller number of sites, and are more likely to be outdated. I'll cover how to get listed in directories later in this article.

Search Engines use automated software programs (called Robots or Spiders) to gather Web pages. They then use other software to determine what the important words on those pages are and to then produce an index to the Web.

Early search engines simply followed the links from a few popular pages. They then followed the links from all of the pages returned from the first run, and repeated this procedure until they had exhausted all of the links. As the size of the Web grew, this method of indexing became very time-consuming. Modern search engines use a tremendous amount of computing resources to accomplish this, and some fancy mathematical tricks to help simplify the job. I'm going to keep the math out of this, and just give you enough background for you to know what we are doing and why.

While it is possible to submit your site directly to search engines, the engines now prefer to find you by just following links. I'll be covering how to submit your site later in this article, along with ways to avoid doing all that work. Don't worry; I'll keep you busy enough with the rest of this.

Why Should I Want to be Listed?

Ok, I recognize that there may be valid reasons for not wanting to go to the extra work to promote your site. Let's see here:

My Web Site is Private.

If you've built a Web site only for the use of a few people that you know personally, then you don't need search engines. In fact, you may want to keep them out. If you really want to do this, it might be a good idea to read the section on robot exclusion. I also suggest that you password protect it, or accept that it may show up on some rogue search engines. Otherwise, you won't need to read the rest of this article. Go ahead, you're excused.

My Site has a Very Unusual Subject

This can be perfectly valid. Say that you have a genealogy page on just your family name, and your name is very unusual (mine is.) So, go to any major search engine and do a search on your surname. I suggest Google. Yes, go do it now.

Did your search returns less than 200 results? If it did, you're in. Check out the section on submitting, and you can probably forget about reading the rest of this article. Unless, of course, you want to be near the top of that list. Your choice.

However, if your family name is Smith, you'd better pay close attention. I just did a search on your name, and got 4,859,808 hits. Only the top 100 or 200 of these will ever be seen, since the search engines limit their responses. More realistically, that number is closer to 20 or 30, since most people don't go beyond the first two or three pages of search results.

If you want to be in the select number that does get seen, you'd better read everything here. I'll tell you lots of ways to get your site seen despite the odds.

I'm Willing to Take my Chances.

Some of you may not be willing to spend the time to submit. After all, it's not as glamorous as creating web pages. I've actually had people tell me that they are willing to take their chances on being found. If if this is what you want, I can only wish you luck. Don't forget to buy a lottery ticket on your way out.

Yes, I Want my Page to be Found

Now, I assume that the rest of you want other people to see what you've created. You've spent many long hours gathering the information, and many more building a Web site. So why not share it with the world? Yes, it may take a couple more hours of work. After all that time invested, isn't it worth that much?

Next:

Let's get started.