SiteMap generator

« on: Mar 09, 08, 11:28 »

yacenty

Apprentice classifier

Posts: 16

Would be great to have site map generator and pinger for google, yahoo, msn and others.
I had allready such modules on my boards using IPB - it's very usefull because it's done on every defined time - my example once a day, after preparing new sitemap the script automatically ping search server.

Thanks in advance,
BR

YacentY


	Logged

Reply #1

« on: Mar 10, 08, 17:43 »

charles

Admin
Fearless Classifier

Posts: 866

Re: SiteMap generator

Have you tried gsitecrawler: http://gsitecrawler.com/ ?

Works well with Esvon as long as you have robots.txt set correctly.

hth
Charles


	Logged

ecusb.com founder and administrator (but I do not work for Esvon!)
Esvon Classifieds on: www.rentahomeabroad.com | Esvon Classifieds Manual

Reply #2

« on: Mar 10, 08, 19:57 »

yacenty

Apprentice classifier

Posts: 16

Re: SiteMap generator

but I have to run it Tongue

on IPB I have it automated, I don't need to remember about it. It's done every night and then pings google and yahoo and rest


	Logged

Reply #3

« on: Mar 18, 08, 01:51 »

RonSanSolo Offline

ecusb.com supporter
Fearless Classifier

Posts: 115

Re: SiteMap generator

Charles,
Can you explain exactly how the robots.txt file should be set "correctly" so it works with gsitecrawler?

Thanks,
Ron


	Logged

Reply #4

« on: Mar 18, 08, 11:11 »

charles

Admin
Fearless Classifier

Posts: 866

Re: SiteMap generator

I meant not leaving it blank - and using something like what isell suggested:

http://ecusb.com/a86997

Otherwise you'll get loads of duplicates and irrelevant links.

Cheers,
Charles


	Logged

ecusb.com founder and administrator (but I do not work for Esvon!)
Esvon Classifieds on: www.rentahomeabroad.com | Esvon Classifieds Manual

Reply #5

« on: Mar 25, 08, 23:40 »

RonSanSolo Offline

ecusb.com supporter
Fearless Classifier

Posts: 115

Re: SiteMap generator

Hi Charles,
I updated my robots.txt file based on Jay's sample, including the following:

Disallow /index.php*
Disallow /modules.php*

Now I'm running GSiteCrawler, which first reads the robots.txt file, and it's still crawling all the index.php? and modules.php? files on my site, including search results, logins, Favorites (with id's), Banner Ads, Recommend Us (with id's). A lot of weird stuff I didn't even know was there.

In fact, it's been running all day and the statistics are showing that it's found over 21,000 URL's, and it's still going.

I'm sitting here wondering what the heck are all these pages!

I've been running our Esvon site for only about 7 months now. We have approx. 800 categories and approx 1400 listings.

Do you have any idea why it's finding all these Login, Search Results, Recommend Us, and Favorites pages on the site? Where are all these pages coming from?

Also, why is it even crawling all the index.php? and modules.php? pages if I have the Disallow in the robots.txt file?

Thanks,
Ron


	Logged

Reply #6

« on: Mar 26, 08, 02:34 »

spanner

ecusb.com supporter
Regular classifier

Posts: 41

Re: SiteMap generator

Does anyone want to contribute to getting one made up, one that queries the database not just crawls the site?


	Logged

Reply #7

« on: Mar 15, 09, 12:25 »

mas

Apprentice classifier

Posts: 22

Re: SiteMap generator

Spanner,

I used to have open source Google Map Generator (http://sourceforge.net/projects/goog-sitemapgen/) and have coded a script to query the DB and get all the ads directly from ads table. You can set up a cron job for this code to run daily or as frequent as you want.

Then you attach the url of the output file of this script into the configuration file of the sitemap.

Good luck,
MAS

Code:

/<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 />
<title>Create URLs</title>
</head>
<body>
<?
include("include/dbcon.php");

// Make sure any results we retrieve or commands we send use the same charset and collation as the database:
$db_charset = mysql_query( "SHOW VARIABLES LIKE 'character_set_database'" );
$charset_row = mysql_fetch_assoc( $db_charset );
mysql_query( "SET NAMES '" . $charset_row['Value'] . "'" );
unset( $db_charset, $charset_row );
$db_charset = mysql_query( "SHOW VARIABLES LIKE 'character_set_database'" );
while($rows=mysql_fetch_row($db_charset))print_r($rows);
//$db_charset = mysql_query( "SHOW VARIABLES" );
//while($rows=mysql_fetch_row($db_charset)){print_r($rows); print("<br/>");}
$q="select link_id, v_title from ec3_ad";
$dbr=mysql_query($q);
$myFile = "urls.txt";
$fh = fopen($myFile, 'w') or die("can't open file");

while($row=mysql_fetch_assoc($dbr)){

$stringData = "http://www.example.com/index.php?page=out&id=";
$stringData .= $row['link_id'];
$stringData .= ".html priority=1.0\n";

fwrite($fh, $stringData);

}
fclose($fh);
die();
?>
</body>
</html>


	Logged

Reply #8

« on: Mar 27, 09, 14:44 »

spanner

ecusb.com supporter
Regular classifier

Posts: 41

Re: SiteMap generator

Thanks Mas for sharing this ... Does it do seo urls or just the default ones?


	Logged

Reply #9

« on: Mar 29, 09, 07:26 »

mas

Apprentice classifier

Posts: 22

Re: SiteMap generator

The file that contains URLs created by php script has a basic urls. But in the configuration file I've added the log directory of my webserver which suppose to contain everything including SEO urls.

I'm using now xml-sitemaps.com the stand-alone version, it gets seo URLs only. It pings 4 major search engines. but visitors to my site has decreased.

I prefer the python script much more, and I might cancel what I have now and go back to the old script if I can make it running again.


	Logged