Before a website can rank well in the search engine result pages, it has to be indexed by the search engine spiders. A search engine spider (robot) will visit one of your pages and look for links on the page. It will follow the links on the page to other pages. All the web pages that are found by the robot will be added to the database (index) of the search engine. If the links are easy to follow (crawl), your site will end up with more pages in the search engine index. So it is important to improve the crawlability of your web pages.
Consider the three points below which will make your web pages easily followed by the search engine spider.
Check the directory depth of your site
You should evaluate the directory depth of your web pages. It is always good to avoid creating very deep directory structures in which your web page files are stored. The bad example will look like:
http://www.gordonchoi.com/dir1/dir2/dir3/dir4/index.html
A better example will be:
http://www.gordonchoi.com/dir1/index.html
The reasons are that,
- Deep directory structures are difficult to crawl by search engine spiders.
- Deep directory structures are rarely crawled by search engine robots.
Check the link structure of your website
It is important to create a website with link structures that can easily be crawled by search engine spiders. Some websites have been created with search engine unfriendly navigation structures. For example links in Javascript or links on image maps may not be followed by the robots.
If you cannot avoid using Javascript navigation, then the best way is to make a simple text navigation menu at the bottom of your pages.
Check your file name convention
Include your keywords in your file names, instead of naming your files as page1.html, page2.html, and so on. If you do a search on Google for any particular item, you should see that wherever your search keyword appears it is listed in the Google results in bold text. You should normally see this bold text in the title text that is displayed, and in the descriptive text that Google displays under the title text. You should also see it in the URL, if you have included your keywords in the file names. It is even better if you already have some of your keywords in your domain name.
Some webmasters do go further in separating their keywords by hyphens in the file names. This is a good practice but is not absolutely necessary anymore. Nowadays the major search engines are able to recognize keywords within longer text blocks, as this is called keyword stemming.
Check your dynamically created web pages
This is a bonus point. If you are making use of many dynamically generated pages, you may also want to read Crawlability Of Dynamically Created Pages.