| View previous topic :: View next topic |
| Author |
Message |
Timmy
Joined: 10 Feb 2010
Posts: 146
|
 Posted: Jul 03, 2010 6:27 pm Post subject: How to stop crawling / indexing? |
 |
|
hi guys,
I am receiving too many DMCA Notifications and this is BAD!
I notice that google and other popular search engines are indexing all files that are sent to my website.
For example, If search Download Rita Daniels59 in google.com, the first result is my website:
| Code: | Download Rita Daniels59 rar
Download File Rita Daniels59 rar. ... Maximum download size per file: 200 MB, 400 MB, Unlimited. Download speed: Very limited, Limited, Unlimited ...
www.mywebsite.com/jiubs6zeltds - Cached |
How can I prevent this to happen?
Please help me...
I want to indexing all pages except files...
Thank you |
|
| Back to top |
|
 |
komi
Joined: 27 Nov 2009
Posts: 159
|
| Posted: Jul 03, 2010 9:55 pm Post subject: |
|
|
| Google for robots.txt. Note that it will take time before current indexed pages are removed. |
|
| Back to top |
|
|
Timmy
Joined: 10 Feb 2010
Posts: 146
|
| Posted: Jul 03, 2010 10:42 pm Post subject: |
|
|
Thank you komi!
Can you or someone suggest a robots.txt for my situation? Thank you |
|
| Back to top |
|
|
komi
Joined: 27 Nov 2009
Posts: 159
|
| Posted: Jul 03, 2010 11:54 pm Post subject: |
|
|
| Code: | User-agent: *
Disallow: /
Allow: /?op=
Allow: /faq.html
Allow: /...
Allow: /...
Allow: /... |
Finish it yourself. |
|
| Back to top |
|
|
Timmy
Joined: 10 Feb 2010
Posts: 146
|
| Posted: Jul 04, 2010 2:55 pm Post subject: |
|
|
Thank you! 
This is what I have:
| Code: | User-agent: *
Disallow: /
Allow: /index.php
Allow: /?op=
Allow: /faq.html
Allow: /login.html
Allow: /tos.html
Allow: /premium.html
Allow: /checkfiles.html
Allow: /links.html
Allow: /html.html |
My question is: this will not cause problems indexing my website index?
Example: www.website.com
Or should I put a Allow: / ? |
|
| Back to top |
|
|
komi
Joined: 27 Nov 2009
Posts: 159
|
| Posted: Jul 04, 2010 5:13 pm Post subject: |
|
|
This will do it: | Code: | User-agent: *
Disallow: /
Allow: /$
Allow: /index.php
Allow: /?op=
Allow: /faq.html
Allow: /login.html
Allow: /tos.html
Allow: /premium.html
Allow: /checkfiles.html
Allow: /links.html
Allow: /html.html |
|
|
| Back to top |
|
|
dotceo
Joined: 30 Dec 2009
Posts: 70
|
| Posted: Jul 04, 2010 9:03 pm Post subject: |
|
|
Don't know why getting a DMCA notice is bad. What's bad is if you don't comply.
Most agencies/law firms used to hired to send notifications don't go through google looking for things. Usually go through forums, news letters and other sources to find content shared. |
|
| Back to top |
|
|
|
Register
Search
Profile
Log in to check your private messages
Log in
