[Case]

Slight issue (I feel) we have with our site is around 10% of our 7000+ products appear more than once.

We can't really avoid this because they fall under multiple product categories and so have unique URL's. So my question is how we can stop spiders from finding them?

I'm thinking a robots.txt file to block any instances of a duplicate? Just let them find 1 "version" of the product.

Yay or Nay? Any other alternatives?

Cheers guys

[PointandStare]

What does it matter if the products are indexed more than once?

[awebapart]

Products in multiple categories can cause some duplicate content issues with ecommerce shops, although there are probably more serious duplicate content issues you should consider first. If you have 7000+ products, have you written those product descriptions yourself or have you just copied the description from the manufacturer or wholesaler? If it is the latter then you face the duplicate content issue of your descriptions being the same as other retailer websites who have done the same thing, and possibly that same content being on the manfacturer's/wholesaler's site too, and if the other site was indexed first then your site's product descriptions could be classed as duplicate content, even if you have just the one occurrence of the product on your site.

As for products with unique descriptions appearing in multiple categories on your site, duplicate content wont be such an issue, the first product indexed shouldn't suffer any duplicate penalties, only the products that are indexed later might. In this case you may want to take action to ensure that a product in a particular area/category gets indexed first (so that the category words kick in too). You can sometimes steer the search engines to index certain new pages before other new pages by placing links to the important pages in more prominent areas on your site (e.g. home page, or latest products section), or by using a sitemap.

To prevent the indexing of certain pages, one alternative to robots.txt ignores is to use search engine unfriendly links to those pages, e.g. javascript links.

[RayB]

Originally Posted by Case

I'm thinking a robots.txt file to block any instances of a duplicate? Just let them find 1 "version" of the product.

Yay or Nay? Any other alternatives?

Cheers guys

In my opinion the best way to do this is with the following meta robots tag on the pages you want Google to ignore

<meta name="robots" content="noindex, follow" />

This way, you won't have those pages indexed - but google will still follow links on those pages and pass pagerank

[Chubster]

Spot on with the manufacturers description. I am slowly working through my 000's of products as nearly everyone in the pet trade has used the same cd to get their data onto their sites

[Case]

Yea, some good advice here. Thanks guys.