XFileSharing Pro - How to stop crawling / indexing?

Message
Author
Timmy
Posts: 146
Joined: Feb 10, 2010 8:17 pm

How to stop crawling / indexing?

#1 Postby Timmy » Jul 03, 2010 6:27 pm

hi guys,

I am receiving too many DMCA Notifications and this is BAD!
I notice that google and other popular search engines are indexing all files that are sent to my website.

For example, If search Download Rita Daniels59 in google.com, the first result is my website:

Code: Select all

Download Rita Daniels59 rar
Download File Rita Daniels59 rar. ... Maximum download size per file: 200 MB, 400 MB, Unlimited. Download speed: Very limited, Limited, Unlimited ...
www.mywebsite.com/jiubs6zeltds - Cached
How can I prevent this to happen?
Please help me...
I want to indexing all pages except files...

Thank you

komi
Posts: 161
Joined: Nov 27, 2009 12:41 pm

#2 Postby komi » Jul 03, 2010 9:55 pm

Google for robots.txt. Note that it will take time before current indexed pages are removed.

Timmy
Posts: 146
Joined: Feb 10, 2010 8:17 pm

#3 Postby Timmy » Jul 03, 2010 10:42 pm

Thank you komi!
Can you or someone suggest a robots.txt for my situation? Thank you

komi
Posts: 161
Joined: Nov 27, 2009 12:41 pm

#4 Postby komi » Jul 03, 2010 11:54 pm

Code: Select all

User-agent: *
Disallow: /
Allow: /?op=
Allow: /faq.html
Allow: /...
Allow: /...
Allow: /...
Finish it yourself.

Timmy
Posts: 146
Joined: Feb 10, 2010 8:17 pm

#5 Postby Timmy » Jul 04, 2010 2:55 pm

Thank you! :)

This is what I have:

Code: Select all

User-agent: *
Disallow: /
Allow: /index.php
Allow: /?op=
Allow: /faq.html
Allow: /login.html
Allow: /tos.html
Allow: /premium.html
Allow: /checkfiles.html
Allow: /links.html
Allow: /html.html
My question is: this will not cause problems indexing my website index?
Example: www.website.com
Or should I put a Allow: / ?

komi
Posts: 161
Joined: Nov 27, 2009 12:41 pm

#6 Postby komi » Jul 04, 2010 5:13 pm

This will do it:

Code: Select all

User-agent: *
Disallow: /
Allow: /$
Allow: /index.php
Allow: /?op=
Allow: /faq.html
Allow: /login.html
Allow: /tos.html
Allow: /premium.html
Allow: /checkfiles.html
Allow: /links.html
Allow: /html.html

dotceo
Posts: 70
Joined: Dec 30, 2009 9:34 pm

#7 Postby dotceo » Jul 04, 2010 9:03 pm

Don't know why getting a DMCA notice is bad. What's bad is if you don't comply.

Most agencies/law firms used to hired to send notifications don't go through google looking for things. Usually go through forums, news letters and other sources to find content shared.