Please advise, I would like to get started right away.

You’re not going to believe this! My website did have duplication from the hosting company, Hostway.

Since they purchased Valueweb in 2009, the traditional folder was the “web” folder. It made “siblings” to the “www” and “public_htmL” which resulted in files being placed in all of them upon updates! I had them turn it off.

Then I discovered I had front page extensions turned on even though I dont publish with it. So all those folders “_vti_Garbage…” were removed as well.

I had them make a minor revision to my htaccess with the domainaddress to www.domainaddress “redirect.”

I never knew that upon the merger of the company’s that this would result in severe duplication/triplication of files.

I’m now getting more reasonable results with your tool. Before I was getting 35k files! Now Im getting something closer to my real count. Currently its coming in with 8000 files. There are some slight duplicates but the weird thing is that they are “queued.” Also there’s a folder that shows some repetition but when I go to it, I don’t see it. Any suggestions?

All that I do on this site is 100% free and I don’t have time to create filters for visitors, sorry, but I’ve got great instructions and examples that you can learn from.

Glad to hear the hint helped :)

I meant search Google and see what sites rank high, then look at how they are set up and modify your site in a similar way – your xml sitemap should then require little tweaking :)

Best,

Jim

Thanks for an ultra prompt response. Yes indeed, we use phpBB, you are correct. As for duplicate titles, well, I’m not sure it just one reason, but there are many multipages thread, so each page has the same title, and I’m afraid that’s not something that can be avoided. As for meta descriptions, well, I was also disappointed when I saw them missing, but it wasn’t me who was setting up the forum. :) I’ll see what can be done.

Could you please tell me how to filter out SID? Actually, I set up many exclusions while building the sitemap, to filter out unneeded pages, but I can’t think of an exclusion that would weed out SID, especially that it’s different every time. A regular expression? I need a little help on this please.

I was also wondering, how deep I should allow the crawler go? The forum’s been running not even for three months and I find it hardly believable that there could be over 1ok pages, but this were the numbers popping up at level 6; is it possible that it was fetching duplicate sites?

Thanks for the hint on the soap, that’s a good one, actually. :)

“What I would do. Look up a term that you know would be hosted on phpBB, then look at their setup, titles, description tags and see how they are configured. Perhaps they’ll tell you what tweaks they made to get it that way?”

I’m apparently missing something here, look up on Google, or what? What will tell me about the tweaks? Sorry, completely not following you on this one. :)

regards,
Matt

Looks like you’re using phpBB, correct? If you run the sitemap generator and look at your titles, you’ll see a ton of duplicates – these NEED to be fixed. As for removing the multiple indexes, SID’s, etc – you’ll need to create a filter within my tool. I also see meta description tags that you also NEED. I’m not trying to sound negative, but if you want those internal pages to do well, you’ll need to address these issues.

What I would do. Look up a term that you know would be hosted on phpBB, then look at their setup, titles, description tags and see how they are configured. Perhaps they’ll tell you what tweaks they made to get it that way?

By the way, I see the term artisan shaving soap seems to be hot, not sure if you knew that, but I’d write an article on it if I were you, I’m positive it would bring visitors :)

Best,

Jim

regards,
Matt

I had 59 instances of index.php with different SIDs in the sitemap. That’s a total mess, are there any solutions?

regards,
Matt