Advanced Web Site Structuring – Tips

Proper Web site design and structure is an integral part of any search-engine optimization campaign. The search-engine ranking algorithms are constantly evolving. More and more Web site owners are losing their spots in the rankings as their competition becomes more technically competent and adept at the technical side of search-engine optimization.

Create a Robots.txt File

Creating a robots.txt file is a way of speaking directly to the search-engine spiders when they arrive at your site. Perhaps you would rather the spiders not visit certain section of your site. Or maybe you want to instruct them to visit every single page. Other times you may want to control the frequency at which the spiders visit your site.

Also you can create a robots.txt file to prevent search-engine spiders from consuming excessive amounts of bandwidth on your server and also to prevent potential copyright infringements.

The file resides in the root directory of your Web server.

Tip: A robots.txt file can also be used to tell the search-engine spiders where a site map is located with this text: Sitemap: http://www.example.com/yoursitemap.html

You can use the robots.txt generator at: www.mcanerin.com/EN/search-engine/robots-txt.asp to simplify the robots.txt creation task.

Using the Nofollow Attribute

Attaching a nofollow attribute to a link is your way of telling the search-engine spiders that they should not follow that link or view that link as anything of significance when determining ranking.

The nofollow attribute can also be used to link to other Web sites that are not directly related to the content of you own Web site. Google has asked that all paid links be tagged with the nofollow attribute to indicate that the links should not affect ranking influence.

Tip: Essentially, adding the nofollow attribute to a link tells the search engines not to use that link as a positive factor in their ranking algorithm.

Structure URL s Correctly

Both search engines and search-engine users appreciate static-looking, descriptive URL s. The search engines also take into consideration the keywords and phrases contained within your URL s and use these to influence your rankings.

Tip: Structuring your URL s correctly provides you with benefits beyond just improving your search-engine optimization efforts. One such example is a paid search, or pay-per-click, advertising campaign. A properly categorized Web site gives you a head start in developing the keyword lists necessary to properly construct a paid search advertising campaign. You may actually see a price decrease in your paid search efforts as well.

Protect Yourself with an .htaccess File

An .htaccess file is the Apache Web server’s configuration file. It is a straightforward yet powerful text file that can accomplish a wide variety of functions.

Although normally left to expert server administrators, an .htaccess file can help you avoid several potential problems.

Rewriting URL s and redirecting Web traffic enables you to use numerous forms of protection ranging from password protecting directories, banning visitors from certain sources, and preventing bandwidth theft from image linkers.

Rules located in an .htaccess file uploaded to your images directory take precedence over the foot file. Also you may want to stop the malicious use of the Linux Wget command that can retrieve your Web site content.

You can use .htaccess to block direct linking to your images, and instead even send a replacement image.

Tip: An excellent tool located at www.htaccesstools.com allows you to generate a variety of different .htaccess code blocks.

The tool automatically generates .htaccess authentication code along with an encrypted .htpasswd file to password protect directories. This is especially important with the advent of new image search engines like Google Images. The tool also creates .htaccess code to block unwanted bots traffic or traffic from certain IP addresses. You should regularly check for excessive traffic from random bots or excessive visits from any particular IP addresses.

Using Mod_Rewrite to Rewrite URLs

The mod_rewrite module of the Apache Web server gives you the power to rewrite poorly formed URL s on the fly as well as manipulate the appearance or any file name you choose. There are tools available to simplify the complex parts.

Use mod_rewrite to easily and noninvasively restructure the URL to look like www.example.com/pets/dogs/fod.

Activate mod_rewrite by adding the text RewriteEngineOn to your Web root directory’s .htacess file. The mod_rewrite module is not present on all server, so be sure to contact your Web hosting administrator to make sure that it is installed or ask if an alternative solution is available.

Tip: Mod_rewrite can be very complex and difficult to use if you are not a skilled Web programmer. And excellent, free tool can be found at www.mod-rewrite-wizard.com. You also have the option to add any relevant keywords as a prefix or suffix.

Redirect Non-WWW Traffic to WWW

You may be amazed to learn that some search engines see the example.com and www.example.com variations of your domain name as two totally separate Web sites. This is called a www/non-www canonical issue because there is confusion as to which version is standard.

It is important to set up your server to redirect all traffic to one variation to prevent any confusion by the search engines.

Fortunately, you can easily address this problem by adding a small amount of code to your Web root directory’s .htaccess file to set up a 301 redirect from example.com to www.example.com or vice versa. A 301 redirect is also known as a permanent redirect.

301 redirects are the preferred method of redirection because Google attempts to pass all the ranking influences to the new destination.

Tip: You should always set up the 301 redirect discussed in this task, and you may be curious to see if you have a large canonical issue with your www/non-www domain entries.

To check, visit www.google.com and type site:newdomain.com into the search bar. Replace newdomain.com with your domain name.

Also take a note of the total number of pages indexed in the upper-right corner of the search results. Next, type site:www.newdomain.com into the search bar. If the results differ dramatically from the results of the previous query, you probably have a canonical issue.

Redirect with 301 Redirects

301 redirects provide you with the means to tell the search engines that not only have you moved your site, but you want all your rankings to move with it. This is a very important and often overlooked issue.

Unfortunately, search engines are not necessarily friendly when you make Web site changes. You can set up a 301 redirect to accomplish this task, and both search-engine spiders and Web browsers automatically redirect to the new location. For search-engine optimization purposes, this also preserves most if not all of your search-engine ranking factors.

If you want to move your entire Web site from one domain to another, a 301 redirect is your best bet to keep your current search-engine rankings.

Do not be alarmed if your redirect is not reflected immediately. Also, keep in mind that any changes you make to your Web site can negatively or positively influence your search-engine rankings.

Tip: Do not expect the search engines to understand your intentions. When you remove pages that are not longer necessary for your site, consider adding a 301 redirect for those pages back to your site’s home page. Otherwise, if you remove the pages and the search-engine spiders continue to crawl the missing pages, they may penalize your entire site despite the deletions being intentional.

If you own the .com, .net, .org, or other extension of your domain name, you can use 301 redirects to automatically force all .net and .org visitors to redirect to the .com extension or whatever extension you choose.  301 redirects can be used to automatically redirect any misspelled traffic to your actual domain name.

Share on Facebook