XFileSharing Free - Upload failed on oversized files

Message
Author
fg
Posts: 3
Joined: Oct 01, 2012 3:45 am

Upload failed on oversized files

#1 Postby fg » Oct 01, 2012 3:54 am

When I try to upload DVD size file, it get stuck on the upload pending screen.
But I have the following error, in the apache logs :
[error] [client 192.168.123.1] Invalid Content-Length, referer: https://websitename/
[error] [client 192.168.123.1] (-3)Unknown error 18446744073709551613: Error reading request entity data, referer: https://websitename/

I also noticed that on the upload pending screen, I got "Current position: 0 / -449535 Kbytes" which puzzle me.
Note that the flength file in the temp directory contains : -460324120

Is there a problem regarding file size over 2GB for exemple?
I tried various file but some are failing even with 1.5GB files.

DimS
Posts: 20
Joined: Feb 11, 2012 8:47 am

#2 Postby DimS » Oct 01, 2012 6:39 pm

Unfortunately many browsers can't upload more 2GB
http://www.motobit.com/help/scptutl/pa98.htm

fg
Posts: 3
Joined: Oct 01, 2012 3:45 am

#3 Postby fg » Oct 01, 2012 8:14 pm

Yes.. found this also, it matches my finding about the behaviour.

The thing is when one client is doing that and stays on the "upload pending screen", for exemple, firefox keep on requesting update very fast and this is eating 40% of a CPU/core on the server.

10 clients like this and one server is down....

If I agreee the browsers should behave correctly, the server doesn't handle the error gently either.

If incorrect length is provided by the client (negative fore exemple), the server should return an error to the client and currently the upload loop tries to manage that like a normal length.

fg
Posts: 3
Joined: Oct 01, 2012 3:45 am

#4 Postby fg » Oct 02, 2012 2:05 am

As nothing interesting popped up, I dived in and found a way to handle correctly these errors.

During the upload process, when the soft is checking for file content length, I added a condition if length is negative, go into error.
In file upload.cgi, line 53

Code: Select all

if($ENV{'CONTENT_LENGTH'} > 1048576*$c->{max_upload_size} || $ENV{'CONTENT_LENGTH'} > 1048576*$c->{max_upload_filesize})
to

Code: Select all

if($ENV{'CONTENT_LENGTH'} > 1048576*$c->{max_upload_size} || $ENV{'CONTENT_LENGTH'} > 1048576*$c->{max_upload_filesize} || $ENV{'CONTENT_LENGTH'} < 0)
At the same time, as is, the software is not able to handle https correctly due to wrong url interpretation, this how to correct it.
In file index.cgi, line 35

Code: Select all

   my ($site_cgi_rel) = $c->{site_cgi}=~/^http:\/\/.+?(\/.+)/i;
to

Code: Select all

   my ($site_cgi_rel) = $c->{site_cgi}=~/^http[s]*:\/\/.+?(\/.+)/i;
Now, let's go serious about it, you really need to understand http, Apache and configuration.
My idea would be to have my users have the ability to share with third party. I don't want to control who download, it's my user's responsibility as I trust them (i.e. Enterprise, Organisation, etc...) But I need to authenticate the user only for the upload and delete operations.
I already have an authentication infrastructure (LDAP in my case), I just need to tell the software to request authentication before doing these.
I do this using Apache authentication scheme.
Btw, my advice is to grab the .htaccess contents and move it to server settings under a virtualhost. The htaccess stuff is to avoid as much as possible as a general rule. It's convenient like slippers but you don't go out in town with them, right?
So we have 3 parts to protect using apache configuration.
The admin part which are located under /admin and /?act=admin, let make it simple and make any URL containing admin protected. This translate into apache configuration as :

Code: Select all

<LocationMatch ^.*admin.*$>
    AuthType Basic
    AuthName "My Login"
    AuthBasicProvider ldap
    AuthzLDAPAuthoritative off
    AuthLDAPUrl ldap://ldap.mondomain.com:389/o=MyOrg?uid?sub
    require valid-user
</LocationMatch>
Ok, next part we want to protect is the upload. Only authenticated user should upload. Let's do the same... except that there is a js file the htdocs directory we want people to be able to get without authentication called xupload.js, let's make a bit more complicated regexp for it :

Code: Select all

<LocationMatch ^[^x]*upload.*$>
    AuthType Basic
    AuthName "My Login"
    AuthBasicProvider ldap
    AuthzLDAPAuthoritative off
    AuthLDAPUrl ldap://ldap.mondomain.com:389/o=MyOrg?uid?sub
    require valid-user
</LocationMatch>
And last but not least, we want to protect deletion. Yes, only our users should be able to delete files :

Code: Select all

<LocationMatch /del->
    AuthType Basic
    AuthName "My Login"
    AuthBasicProvider ldap
    AuthzLDAPAuthoritative off
    AuthLDAPUrl ldap://ldap.mondomain.com:389/o=MyOrg?uid?sub
    require valid-user
</LocationMatch>
You can change the authentication provider to whatever you want like local htpasswd file or even a DB, check the apache doc for that.
Ok, we have a pretty good setup here that allow our users to upload, delete file and anyone to download, add on top that the admin section is also protected.
But it's not finished, 2 things you need to know about this :
First, "AuthBasic Basic" makes client (browser) send username/password to server in plain text.... so don't do this if not on https. It's not ideal... but not to bad considering the money involved.
Second, you'll hit a new problem when a user who isn't authenticated yet is going to upload a file. When the browser begin to send the file, Apache request for authentication and cut the upload process, leaving the software lost in the wild....
How do we do correct this? Create a welcome screen to authenticate users before they reach the upload form.
Let's create a template named welcome_screen.html in cgi-bin/Templates and put the following content in

Code: Select all

<TMPL_IF msg><font class="ok"><TMPL_VAR msg></font><br><br></TMPL_IF>
<div class="rdc_box"><div class="rdc_top"><div></div></div><div class="rdc_content"><p>
<table cellpadding=0 cellspacing=0 id="div1" style="margin:5px;position:static;height: 200px;"><tr><td valign=top>
</TD></TR></Table>
<br>
<center>
Please <a href="<TMPL_VAR site_url>/upload">login</a> to use this service.
</center>
<br>
</td></tr></table>
<p></div><div class="rdc_bottom"><div></div></div></div>
</form>
Then we teach xfilesharing how to use it and when to use it, let's change the index.cgi file at line 30, it says by default, send uploadform template...
Change it to :

Code: Select all

&UploadForm if $act eq 'uploadform';
&WelcomScr;

sub WelcomScr
{
   my ($site_cgi_rel) = $c->{site_cgi}=~/^http[s]*:\/\/.+?(\/.+)/i;

   $ses->PrintTemplate("welcome_screen.html",
                       'site_cgi_rel'     => $site_cgi_rel,
                      );
}
Whooohoo... nice isn't it, don't forget to add the rule in apache configuration that the upload form isn't the default page anymore:

Code: Select all

    RewriteRule    ^admin$ /?act=admin [R]
    RewriteRule    ^upload$ /?act=uploadform [R]

What do you say?
Not bad, hey?
Ok, let's sum up, we have :
- xfilesharing free that doesn't go wild when an idiot want to upload a DVD when we say 700Mb max.
- xfilesharing free that may run under https
- xfilesharing free able to protect upload, deletion and admin through a more open authentication mechanism (LDAP)

Not bad, hey?... What's missing? I hear "How to get rid of the "Powered by" sneaky footer?"
Naaah, I'm joking.

Enjoy, and thanks for Sibsoft for that piece of software.

admin
Site Admin
Posts: 1839
Joined: Mar 22, 2006 12:32 pm

#5 Postby admin » Oct 02, 2012 2:46 am

Nice write up, fg.