Communicating with Search Engines

Communicating with Search Engines

Before long tail keyword research, before keyword density analysis, before any on–page Search Engine Optimization (SEO) tasks are undertaken, to achieve a high placement — in the colloquial, to get a high ranking — on the Search Engine Results Pages (SERPs) it’s a best practice to enable communication with the search engines.

There are five search engines that have a greater than 1.0% market share of all English language searches in the US. These are Google, Yahoo, Bing, Ask, and AOL. The top three search engines (Google, Yahoo & Bing) account for almost 96%.

Please be aware, within specific genres there are additional search engines that only index activity within their sector. Examples of three such sectors would be blogs, paid content (online ads), and academic papers.

In addition, it should be noted that DMOZ, the Open Directory Project (ODP), still plays a significant role in searches conducted in non–roman character languages. In such searches Google still crawls DMOZ and lists the results.

Also, on the occasion when a web page lacks a Title element & Description meta tag and that web page has been indexed on DMOZ, many search engines will include the DMOZ results. It is, of course, a best practice to implement unique Title element & Description meta tag on each web page.

Further, if the site is in a roman–character alphabet, and each page has a unique Title element & Description meta tag, it’s recommended to block DMOZ from feeding the major search engines, esp. Yahoo & Bing.

Each search engine offers a slightly different array of services for registered webmasters. Given Google’s market share, it’s the most important and offers the most robust services, currently, 33 Webmaster Tools. Of these, nine are necessary for effective communication with Google. Bing and Yahoo each offer fewer tools. Nonetheless, the principle is the same.

Ten Steps to enable better Communication between your Web Site and Search Engines

  1. Open a Webmaster account in each search engine and complete the registration/profile for each tool. Each search engine has a different registration policy. Set up an account for each domain name and/or each email address that is to be given access.
  2.  Once a web site is public it must be verified. Verification can either be linked from an external text file or placed in a meta tag between the <head> and </head> elements in the code. Google requires verification be placed on every page. Bing and Yahoo require verification be placed on every root page. (If a web site only has a Top Level Domain (TLD) then this is the home page. If a web site has sub-domains, verification must be placed on the root page of each sub-domain.).
  3.  Each and every page must have either a robot meta tag or a link to an external robot.txt file. In its simplest form a robot.txt tells a search engine whether (or not) to index a page and whether (or not) to follow the links (to the next pages). For example: <meta name=”robots” content=”all, index, follow” /> . Primarily, this is to prevent the search engines from indexing admin pages. For instance not to index registration and e-commerce checkout pages — i.e. any age where a user has entered private or financial information.

    It is also a best practice to differentiate between the search engines; indicate specific directions for Google, Bing, and Yahoo. Google’s bot is named GOOGLEBOT. Yahoo’s is slurp. Bing’s is msnbot.

  4.  Once a site is verified, submit the domain name to the webmaster account on each search engine. While there are public submissions available, it is far more effective to login to the webmaster account and use the submission form for webmasters. Doing so will minimize the interval before the search engine’s bot indexes the site. Google only requires the TLD be submitted — although it should be noted far more emphasis is placed on the submission of XML sitemap(s). (See next step.) In addition to submitting a TLD, if a site has sub-domains, Bing and Yahoo require the root file for each sub-domain be submitted.

    Each site should have both a public and a non-public sitemap. The public sitemap is in HTML and can be accessed through a site’s menu (navigation). The non-public sitemap(s) are in XML and must be submitted to each search engine through the webmaster account. All three major search engines require a sitemap for the TLD as well as any sub-domains. Protocol for XML sitemaps is available on

  5.  Every time a page is modified, a page is removed, moved to a new directory, or new pages are added, updated sitemaps must be generated and resubmitted to the search engines.
    For dynamic sites that enable user generated content — including comments on blogs, this is best accomplished with automated scripts that will index the site, generate sitemaps and submit those sitemaps to the search engines. Google does offer a service to do this. However, there are caveats. One, your host must support Python (a scripting language). Two, when pages are modified or new pages are generated it is very important to place a robot meta tag (or a link to an external robot.txt file) that specifies if the page is to be public, i.e. crawlable. Otherwise, all pages placed on the sitemap will be indexed.

    For static sites, i.e. no user-generated content, esp. sites with no content system management (CMS) it’s best to code XML sitemaps by hand.

    On April 7th, 2010 Google added the ability to include links to media in sitemaps. This enables image, video, and audio files to be listed in sitemaps, supplying additional semantic metadata to Google.

  6.  Implement a Canonical meta tag <link rel=”canonical” href=”” /> . A canonical meta tag prevents indexing of duplicate content.
  7.  Open a Google Analytics account. If there are sub-domains, modify the javascript for sub-domains. Tag the pages with two javascripts if you want to simultaneous track TLD and sub-domain activity separately. In addition, choose and implement goals. Google now permits twenty goals per profile. To create additional goals either add more profiles or tag the action as its own page rather than as a goal.
  8.  Geo Tagging will assist Google w/placement on Google Maps. Beneficial for local search. Geo tags are used to assist Google maps with placement. As an example, below are the three meta tags I’d placed on a Woodbridge, VA store’s location page.

    <meta name=”geo.placename” content=”Woodbridge, VA, USA” />
    <meta name=”geo.position” content=”38.651789,-77.304171″ />
    <meta name=”geo.region” content=”US-VA” />

  9.  Open a Google Places (formerly, Google Local Business Center) account. Complete all of the fields. The Business Name field is limited to 80 characters. The description field is limited to 200 characters. The info in the Google Places listing is what populates the fields on Google Maps.
  10.  Register a .TEL TLD extension for your domain name. For example, in addition to having registered (or dot org, etc.) also register, . Dot Tel domain names provide an online form for contact info. The information is placed in a world-wide white pages directory. With mobile applications such as CallByName for the iPhone/iPad and Android handheld devices (smartphones) one can search any business or individual simply by speaking their name. The app searches the .TEL directory upon voice command. If a match is found (and selected by the user) then the listing is linked to Google Maps. At present this service has full functionality in three countries (USA, Canada, and China) and partial functionality in an additional 43.

In addition, on March 3rd, 2010 Google confirmed they’re now indexing asynchronous JavaScript and XML (AJAX). AJAX is the primary means by which content is fed to pages from either flat files or a database that enable updates from user interaction without reloading the entire page. Previously, content in AJAX and XML was treated as a separate page and not indexed as part of the page on which it appears. If AJAX is implemented, there guidelines to enable Google’s indexing of the content.

To check how many pages Google has indexed for a particular domain name, type into the Google search field.

A more complete review of web analytics is a larger discussion.

Trending :   Top Ten Natural Organic Search Engine Optimization (SEO) Best Practices