Get Ready for Search Engines

Preparing your Website for search engines means tackling the following:

• Searcher’s Needs
• Search Engine Requirements
• Meta Tags
• Content Viability
• Internal Links
• External Links
• Linkability

How Do Search Engines Work?

There are several kinds of search engines. Some collect their own data through “robots” or “spiders”, software that crawls through the web, tracking the various links between pages and sites, gathering data which the search engine filters and adds to its database. Some rely upon individuals and businesses to enter their Website’s information manually, with a staff that checks and processes the information into their database. Other search engines, called meta-search engines or metacrawlers, don’t gather their own information but search the databases of other search engines, letting them do the work. And then there are combinations of all of these. For example, Google sends out robots to gather information. They also permit manual submissions. They also search other search engines to add information to their database. This combination creates a vast resource with checks and balances so they can offer a wide range of listings and not be dependent upon a single collection.

Once the search engine has the information from a web page, it applies a software filter to clean it up and determine whether or not it is worth including in the database. Many robots and spiders are now sophisticated enough to do much of the filtering before gathering the content, saving time and effort on the other end. Therefore, you have to pass their tests in order to be considered for their listings.

When a search engine searches, these are the things it considers and gathers when visiting your pages:

• Code Viability
• Page Title
• Links to your core pages (internal links)
• Links to external web pages (external links)
• Content – Text
• Word Frequency and Density
• Meta Tags (page description)

As you can see, a search engine doesn’t care about how pretty your Website is and how many pictures or dazzling graphics you have. All it wants is information and that comes from meta tags and content text.

Web page designs now encompass more than just some words and a few pictures. A lot of designers are opting for Macromedia’s Flash and other visual expressions to showcase their business and products with dramatic visual displays, video, sound, and slide shows. Photographers are definitely taking advantage of all the bells and whistles they can for visual impact. Unfortunately, when it comes to getting picked up by search engines, if you don’t have text or some basic textual data on your page, it’s difficult for the search engines to gather information about your site. They don’t “look” at the site, they just crawl through it gathering data. No data, no collection, no listing. Even if you use splashy graphics, make sure there is some underlying code that tells the search engines what you do and why they should bother with you.

Document structure and page layout (frames, tables etc.) affect how a search engine gathers information about your page. Frames can limit the robot’s ability to index your site, as they can’t easily access your attached frames. The newer “iframe” is also ignored by robots. If you are using frames, have a “no frames” version so search engines and users with accessibility problems can still make use of your page. If you use tables to layout your page, the layout may force the key information down lower in the page away from the searching eyes of the robots. Few search engine robots go beyond 50% of your site, so if the good content is low on the page, they will miss it. Meta tags, tags with information about your web page, are meant to be found within the HEAD of your document. If a search engine requires the meta tag information and it can’t find it, you lose.

Search engines won’t check your pages for proper tags and coding, but if it isn’t correct, it can lead the robot or spider in a wrong direction or confuse it. If it has difficulty moving through your pages, it will stop and look elsewhere. A robot can spot a well designed page because it can get to the “good stuff” faster. It also gives the robot information that could help raise the rankings of the page within the search engine database. It says that this page was designed with care and attention to detail and to the standards imposed by the industry, and therefore it should have valuable content worth considering. Not every page is worthy of inclusion. The easier you make the job for the robot, the more likely they will visit you, and return.

How People Search and How They Can Find You

Did you know that 53% of Websites are found by word-of-mouth? Yes, the old gossip method still works. There are many ways of spreading the news among friends, relatives, clients, your industry, and the general public, but let’s concentrate on how Internet users can find you through search engines. After all, your goal is to be found, right?

Spamming, Flooding, and Other Search Engine Failures

A lot of amateurs, and even a few veterans, of web page design using techniques they think will get them up the rankings in search engines. They come up with all kinds of gimmicks like hiding text on a page that isn’t visible (or sometimes is) to the user filled that if they use the word “sex” or “foreplay” hidden in the code, it will increase the likelyhood of coming up on search engine searches.

These just don’t work. They are a waste of your time, our time, and the search engine’s time. Excess code that is unimportant to the page adds to the file size and slows down the loading times. Search engines know how to look for such evils as spamming and you trying these techniques.

Stick to tried and true methods and your site will get the attention it deserves. It takes time, but it will happen if you follow the simple steps outlined within our articles and other sites offering sensible techniques for attracting and keeping the attention of search engines and your audience. Play nice and play fair.

How Do You Search?

Think about how you search the web. Do you start out vague or think it through to come up with the specific phrase or term to narrow your search immediately? Do you often search within those results (refining your search) or start over? How many pages of the results do you go through before you start over or just give up? Do you start with Google or Yahoo, or another search engine? Which search engines are people using the most?

Answering these questions tells you a lot about how people search and about how people would search for you. Are the content and vague in reference to your subject? If you sell toothbrushes, but never mention “teeth”, aren’t you missing something? As professional nature photographers, we redid our web pages recently and realized that most of our web page articles and images were related to travel and not so much to nature, so we rearranged our material to be more specific to travel, with nature as a side line.

Look at Competitor’s Web Pages

Look at your Search Engine World’s Keyword Validator to find how their page measures up to yours. By what words and phrases do you think people use to find them on search engines? Experiment and try a few to see if their site comes up. Don’t know who your competitors are? Some search engines make it easy for you. Search for your product line or subject and when you see a description that matches your interests, click on “Find Similar Pages”. Also check the links offered below for the most attract attention to themselves.

Categorize Your Website

Search engines like Google, Yahoo, and Altavista offer long lists of potential sites when searched. Others categorize their listings like a directory, similar to a yellow pages directory. These include DMOZ Open Directory and All The Web. You can search their site and get specific results, but the better technique is to “drill down” through their categories, getting more and more specific as you move down through the lists. For instance, our site, Taking Your Camera on the Road, could be found in a directory under Arts, Photography, Instruction, Workshops or Travel, Activities, Photography. DMOZ requires Websites to be categorized very specifically, keeping the listings per category down to a manageable number.

Directories like DMOZ can be very helpful in determining what category your Website belongs. Browse through their categories that apply to your site and check your competition. Are they dealing with the topics you are? Then you might want to reconsider your category. Or get more specific. To help people find you, it helps to be in the category they are searching. Think about it. Where would you find your topic if you were looking in a library or bookstore?

Keep a running list of the words and phrases you see repeated over and over again. Consider how they apply to you. We’ll discuss how to maximize your coverage in a moment, so keep collecting this information as you gather it and begin the process of understanding how people will search for you.

Below are some links to help you get ready for search engines. In the next article we’ll examine how your page functions and performs under the headlights of a search engine.

Our Listings of Top Search Engines

• Search Engines: Top Search Engines Listing
• Search Engines: Directories and Indexes
• Search Engines: Specialty Search Engines and Directories
• A Winner: Awards for Websites

Search Engines – Robots and Spiders

• Enginemage’s Robots Listing
• Database of Search Engine Robots
• Crawler Alert: Be notified by Email when your web page has been spidered
• The User Agent Database (spiders, robots, harvesters, etc.)
• List of Robot Agent Strings
• The Web Robots Database
• Tool checks your spidered pages with Google.com’s database. Good for verifying titles and descriptions.
• Robots.txt Syntax Checker
• Search Engine Optimization Tools – Search Engine Spider Simulator

Search Engine – Articles about Robots and Spiders

• How to block spambots, ban spybots, and tell unwanted robots to go to hell
• How to Know When a Spider Visits
• Optimizing for Spiders
• Search Engine Test-Webpage Spider & Crawler Test
• Using Apache to Stop Bad Robots
• Search Engine Robots that Search Your Site ****
• HTML Author’s Guide to the Robots Exclusion Protocol
• A Standard for Robots Inclusion
• All About Search Indexing Robots and Spiders
• Generating Simple URLs for Search Engines Robots and Spiders

Search Engines – Cluster

• Web Search For A Planet: The Google Cluster Architecture
• Clustering With Search Engines – Article
• Clustering User Queries of a Search Engine – Article
• Google.com’s “Sets” Cluster Search Project
• Mooter – NEW and VERY UNIQUE
• Vivisimo Cluster Search Engine – one of the oldest
• Infonetware
• Queryserver
• Tenoma (part of AskJeeves)
• Wisenut

Search Engine – Reference

• Understanding Search Engine “stop words” – ignored words
• Understanding Search Results Pages
• Search Engine Results chart (where info comes from)
• Search Engine Trends

Search Engine – Resources

• Google Features and Tools (allowing searching for your pages, links to your pages, and pages like yours)
• Metaspy’s Live Search Requests
• A Survey of Google’s Page Rank
• Search Engine Watch’s Search Engine Index – statistic reports – how the search world rates

Search Engine – Validation

• AnyBrower’s Search Engine View of Your Web Page – tester

Search Engine – Comparisons

• Who Powers Whom – Search Engine Sources
• Search Engine Ratings (which are the most used search engines)
• Searchenginewatch.com’s Search Engine News and Statistics
• Reports from Searchenginewatch
• How Search Engines are Inter-Connected
• Search Engine Rankings
• Search Engine Chart
• Search Engine Statistics: Relative Size Showdown
• Search Engine Comparisons
• Search Engine Stability and Submission Times

Search Engines – Designing Web Pages and Websites for Search Engines

• How to Search the Web – Understanding How People Search
• Enginemage – Search Engine Articles and Resources
• Searchenginewatch’s Tips and Articles on Optimization
• Search Engine Features for Webmasters
• How Major Search Engines Regionalize
• Searchenginewatch Site Submission Tips and Tricks
• Article on Site Submission
• Secrets of Search Engines
• Understanding Search Engines
• Site Search Engine Saturation
• A Helpful Guide to Web Search Engines
• Search Engine Advice
• Search Engine Optimization
• Search Engine Features for Webmasters
• Accessing Your Site Indexing on Search Engines
• Search Engine Guide
• How long does it take to get listed in a search engine

Search Engines

• Understanding Alexa for Webmasters
• Aesop
• All The Web
• Alta Vista
• Ask.com
• Ask Jeeves
• Google
• Link Master
• Specialty Search Engines
• Search Engine Collosus (massive listing of search engines)
• SearchEngineWatch’s Links to Search Engines
• Search Tools Product Listings (Very comprehensive list & Includes Reviews)
• Yahoo

Other Resources

• Marketing Experiments – Research into what works and doesn’t in web page design and marketing