Beginners guide to dynamic sitemaps

We're going to discuss in this article about a small segment of a website that is very often left behind, forgotten or not used at all. "A Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site". Google introduced this feature for developers in order to allow them to help the crawler do a better job since many websites have pages with content that are available only by some user/visitor action (form, request, entries etc.) and the search engines can't discover them by natural spidering (following links). Since MSN, Yahoo, Ask, and Google use the same protocol for indexing sitemaps we are talking about a really big step in optimizing your website and making it "crawler friendly". In my opinion, by some tests that I've run, I would say that sitemaps can have the potential of increasing your daily visitors with as much as 10% - 15% and that's a huge step. As a good note to finish this paragraph with, sitemaps won't have any effect on Google's SERPs and will not influence PageRank. Usually, sitemaps, use few details as meta data such as "urlset" which holds the whole set of url's available on the map, "url" which holds the details of each url in part, "loc" which represents the location of the url (the url itself), "lastmod" which should reflect the last date this page was updated/modified, "priority" which, in your opinion, represents the importance of that page (from 0.1 the lowest to 1.0 the highest rank) relative to the rest of your content and "changefreq" which should output how frequently that page is changed (Always, Hourly, Daily, Weekly, Monthly, Yearly, Never). The urlset, url and loc tags are required while the rest presented above are optional. All data values must be entity-escaped and the file itself must be UTF-8 encoded. There are many ways of creating a sitemap and we're going to analyze them a little bit before talking about how to build the right one for your website. Simply by looking at google's results for "google sitemap" you will notice a big number of websites that offer the service of creating one for you and with the help of some details/parameters they crawl your pages and build it up. This is a method that I recommend to static websites, with very few pages, that publish new content very rarely or never at all. I personally hate it since I have to rely on other people's software, I always have to remember to create a new file each time I update my site, all spidered pages have the same priority, I have to pay in most cases for a result that will reflect more than 500 url's spidered etc. All of this when, with a little knowledge and attention, we can create our own sitemaps that update automatically each time our database grows. By the end of this article you will know how to build the right XML sitemap for search engines that is updated dynamically using records from your database. We will start with a regular one and finish with a sitemap index in order to cover the whole subject correctly. Here's an example of a static sitemap that covers only one url:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
	<url>
		<loc>http://www.example.com/</loc>
		<lastmod>2005-01-01</lastmod>
		<changefreq>monthly</changefreq>
		<priority>0.8</priority>
	</url>  
</urlset>

Notice the opening tags which tells the header that we're dealing with an xml file, version 1.0 and that the encoding is UTF-8 in order to meet the requirements. To get the records from our database and make this file update dynamically we will have to create a new php file (name it sitemaps.php) and a recordset to help us loop through the entries from our database. The loop should start right after the opening of the "urlset" tag and finish right after the end of the last "url" record:

<?php
	$query = mysql_query ( "SELECT * FROM articles ORDER BY ID DESC" );
	$row = mysql_fetch_assoc ( $query );

	echo '<?xml version="1.0" encoding="utf-8"?>' . "\n";
?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<?php
	############# BEGIN LOOP ############
	do {
	
	$link = 'http://www.mydomain.com/articles.php?ID=' . $row['ID'];
	$lastmod = ( $row['date_updated'] ) ? $row['date_updated'] : $row['date_added'];
	$priority = 0.5;
	$changefreq = 'monthly';
	
?>
	<url>
		<loc><?=$link?></loc>
		<lastmod><?=$lastmod?></lastmod>
		<priority><?=$priority?></priority>
		<changefreq><?=$changefreq?></changefreq>
	</url>
<?php
	} while ( $row = mysql_fetch_assoc ( $query ) );
	############# END LOOP ############
?>
</urlset>

Notice I've used 0.5 as the priority of the pages since we're linking article entries and we're not expecting so much new content on those pages (maybe some comments from time to time). For "lastomd" I've used the last date the article was updated and if it wasn't at all, the date it was added to the database. Everything else is quite easy and self-explanatory. Don't forget to escape your data, especially on the "loc" tag that holds the url. Here's a PHP function that I often use:

function do_xhtml ( $string ) {
	$string = stripslashes ( $string );
	$string = str_replace ( "'", "'", $string );
	$string = str_replace ( '"', '"', $string );
	$string = str_replace ( '?', '-', $string );
	$string = str_replace ( '?', ''', $string );
	$string = str_replace ( '?', '', $string );
	$string = str_replace ( '?', '', $string );
	$string = str_replace ( '`', '', $string );
	return $string;
}

As you can see, with a few lines of code, we managed to create our own dynamic sitemap that will always be updated whenever our site grows. We no longer have to depend or spend money on other solutions and services. There are many websites that have a lot of content to link with a sitemap and only one XML file would not be enough. We're talking about thousands of pages that would create a huge sitemap. For this situation, google comes with a solution: a sitemap index that will link to other XML files. Basically, you use more sitemaps to divide your website and then link all of them into a single file. Let's take for example roScripts which has 5 sections: Scripts, Articles, Jobs, Snippets and Resources. Predicting that those segments will enjoy a rapid growth in their records, a good idea would be to create a different sitemap for each section in part and then link all of them using an index. Ok, we'll finish with a sitemap index example that we mentioned above.

<?php
	echo '<?xml version="1.0" encoding="utf-8"?>' . "\n";
?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
	<sitemap>
		<loc>http://www.mydomain.com/articles_sitemap.php</loc>
		<lastmod><?=echo date ( 'Y-m-d' )?></lastmod>
	</sitemap>
	<sitemap>
		<loc>http://www.mydomain.com/scripts_sitemap.php</loc>
		<lastmod><?=echo date ( 'Y-m-d' )?></lastmod>
	</sitemap>
	<sitemap>
		<loc>http://www.mydomain.com/resources_sitemap.php</loc>
		<lastmod><?=echo date ( 'Y-m-d' )?></lastmod>
	</sitemap>
	<sitemap>
		<loc>http://www.mydomain.com/snippets_sitemap.php</loc>
		<lastmod><?=echo date ( 'Y-m-d' )?></lastmod>
	</sitemap>
	<sitemap>
		<loc>http://www.mydomain.com/jobs_sitemap.php</loc>
		<lastmod><?=echo date ( 'Y-m-d' )?></lastmod>
	</sitemap>
</sitemapindex>

That's just about everything, remember to go to google webmasters central and add your new sitemap.

For coding help and further support please use the forum.

This posting has no comments.