XFileSharing Pro - Remove site from search engines

Message
Author
RockDiz
Posts: 135
Joined: May 05, 2011 4:47 pm

Remove site from search engines

#1 Postby RockDiz » May 25, 2012 4:28 am

hello,

I've tried adding robots.txt file on my server, and in the meta. Also i've removed my sitemap and disabled it in settings and it still doesn't seem to block search engines from crawling my site.

Are there any other known procedures I can do to get my site not to show up in search engines? Thanks

randy
Posts: 321
Joined: Mar 13, 2012 7:00 pm

#2 Postby randy » Oct 21, 2012 4:56 pm

could you solve the issue?

can someone help?

we have the same problem

afdah
Posts: 147
Joined: Jan 27, 2012 2:23 pm

#3 Postby afdah » Oct 21, 2012 5:43 pm

the robots.txt is there as a rule for search engine crawlers if they should crawl the site. they can choose not to follow your rules.

unfortunately, you cannot stop your site from appearing in search engines. your links will be everywhere around the web posted in forums, email, blogs for search engine robots to crawl.

randy
Posts: 321
Joined: Mar 13, 2012 7:00 pm

#4 Postby randy » Oct 22, 2012 7:55 am

of course

but what i dont understand is if new files have been uploaded as non public they also appear after a few days in google.

and they don't appear in a forum when I search them. they appear as own link in google and next to our other files like tos faq etc.

are you 100% sure that those files have been posted somewehere in a forum to appear?

afdah
Posts: 147
Joined: Jan 27, 2012 2:23 pm

#5 Postby afdah » Oct 22, 2012 8:48 am

if you are talking about recently published files, XFS got its own RSS feed for the whole world to see.

Check /catalogue.rss

randy
Posts: 321
Joined: Mar 13, 2012 7:00 pm

#6 Postby randy » Oct 22, 2012 9:04 am

aight! That's it! how can I remove it?

afdah
Posts: 147
Joined: Jan 27, 2012 2:23 pm

#7 Postby afdah » Oct 24, 2012 10:42 pm

remove /catalogue.rss

randy
Posts: 321
Joined: Mar 13, 2012 7:00 pm

#8 Postby randy » Oct 24, 2012 10:52 pm

delte the file? but it will be generated by script? doesnt script show error then?

afdah
Posts: 147
Joined: Jan 27, 2012 2:23 pm

#9 Postby afdah » Oct 24, 2012 11:47 pm

the file will not be generated by the scipt.
based, on the below code in index.cgi, if the file catalogue.rss is not found, the rest of the code of updating catalogue.rss is skipped.

Code: Select all

   if(-f "$c->{site_path}/catalogue.rss" && time-(lstat("$c->{site_path}/catalogue.rss"))[9]>3)
   {
     my $last = $db->SelectARef("SELECT file_code,file_name,file_descr,DATE_FORMAT(CONVERT_TZ(file_created, 'SYSTEM', '+0:00'),'%a, %d %b %Y %T GMT') as date FROM Files WHERE file_public=1 ORDER BY file_created DESC LIMIT 20");
     for (@$last)
     {
       $_->{download_link} = $ses->makeFileLink($_);
       $_->{download_link}=~s/\&/&/gs;
       $_->{download_link}=$ses->SecureStr($_->{download_link});
       $_->{file_name}=~s/\&/&/gs;
       $_->{file_name}=$ses->SecureStr($_->{file_name});
     }
     my $tt = $ses->CreateTemplate("feed.rss");
     $tt->param(list => $last);
     open FILE, ">$c->{site_path}/catalogue.rss";
     print FILE $tt->output;
     close FILE;
   }exit unless $ses->{dc};

randy
Posts: 321
Joined: Mar 13, 2012 7:00 pm

#10 Postby randy » Oct 24, 2012 11:49 pm

thank you mean! so i can just delete it?

afdah
Posts: 147
Joined: Jan 27, 2012 2:23 pm

#11 Postby afdah » Oct 24, 2012 11:59 pm

yes

randy
Posts: 321
Joined: Mar 13, 2012 7:00 pm

#12 Postby randy » Oct 25, 2012 12:00 am

thx man :) will leave now. leave me an email if you want! cu