UBER-UPLOADER (UU) VER 6.7.2

10:37 AM May 16, 2009

INSTALLATION

• Requirements
• Flow of Execution
• File Listing
• Initialization
• Configuration
• Installation
• Installing Multiple Config Files
• Installing Bubbly Progress Bar

HELP

• Passing Values Through UU
• Progess Bar Styles
• Security
• Debugging Tools
• Failed to Find Flength File
• Server Clusters
• How to View an Active Upload (CGI_UPLOAD_HOOK Disabled)
• How to View an Active Upload (CGI_UPLOAD_HOOK Enabled)
• Multi Simultaneous Uploads
• Foreign Language Support
• How To View Your PHP Settings
• How To View Your Perl Settings
• Apache Settings That May Affect Uploads
• PHP Settings That May Affect Uploads

1. Operating System: All BSD Platforms (FreeBSD/NetBSD/OpenBSD/Apple Mac OS X), All POSIX (Linux/BSD/UNIX-like OSes)
2. Apache web server
3. Perl with CGI, FILE::COPY, FILE::PATH, IO::File
4. PHP

-----------------
|               |                          |--->(2) UPLOAD FILE(S)---->(create upload dir)-->(upload files)---->(5) REDIRECT
| SUBMIT UPLOAD |---->(1) LINK UPLOAD -----|                                          |
|               |                          |--->(3) SET PROGRESS BAR ---------|       |
|               |                                                             |       |
| progress bar  |<-----(return upload info and start progress bar)<-----------|       |
|               |                                                                     |
| |||||||       |<--------------------->(4) GET UPLOAD STATUS <----------------------->
|               |
-----------------

The following steps will guide you throught the installation process:

1. Set the $TEMP_DIR value in ubr_upload.pl and ubr_ini.php. DO NOT MAKE THIS DIRECTORY PUBLIC! Examples:

   $TEMP_DIR = '/tmp/ubr_temp/';
   $TEMP_DIR = '/var/tmp/ubr_temp/';
   $TEMP_DIR = '/usr/home/roger/ubr_temp/';
   

2. Set the $_CONFIG['upload_dir'] value in ubr_default_config.php. May or may not be public. Examples:

   $_CONFIG['upload_dir'] = '/tmp/ubr_uploads/';
   $_CONFIG['upload_dir'] = '/var/www/html/ubr_uploads/';
   $_CONFIG['upload_dir'] = '/usr/local/www/htdocs/ubr_uploads/';
   $_CONFIG['upload_dir'] = $ENV{'DOCUMENT_ROOT'} . '/ubr_uploads/';
   

3. Set the correct path values for the scripts in ubr_ini.php. Examples:

   $_INI['path_to_upload_script']       = '/cgi-bin/ubr_upload.pl';
   $_INI['path_to_link_script']         = 'ubr_link_upload.php';
   $_INI['path_to_set_progress_script'] = 'ubr_set_progress.php';
   $_INI['path_to_get_progress_script'] = 'ubr_get_progress.php';
   $_INI['path_to_js_script']           = 'ubr_file_upload.js';
   $_INI['path_to_jquery']              = 'jquery-1.3.2.min.js';
   $_INI['path_to_css_file']            = 'ubr.css';
   

4. Upload ubr_pload.pl to your cgi-bin directory. The path will be /var/www/cgi-bin or /usr/local/apache/cgi-bin
5. Make ubr_pload.pl executable with the command chmod 755 ubr_pload.pl
6. Upload ubr_ini.php to your webspace. The path will be /var/www/html or /usr/local/apache/htdocs
7. Upload ubr_lib.php to your webspace. The path will be /var/www/html or /usr/local/apache/htdocs
8. Upload ubr_finished_lib.php to your webspace. The path will be /var/www/html or /usr/local/apache/htdocs
9. Upload ubr_default_config.php to your webspace. The path will be /var/www/html or /usr/local/apache/htdocs
10. Upload ubr_file_upload.php to your webspace. The path will be /var/www/html or /usr/local/apache/htdocs
11. Upload ubr_file_upload.js to your webspace. The path will be /var/www/html or /usr/local/apache/htdocs
12. Upload jquery-1.3.2.min.js to your webspace. The path will be /var/www/html or /usr/local/apache/htdocs
13. Upload ubr_link_upload.php to your webspace. The path will be /var/www/html or /usr/local/apache/htdocs
14. Upload ubr_set_progress.php to your webspace. The path will be /var/www/html or /usr/local/apache/htdocs
15. Upload ubr_get_progress.php to your webspace. The path will be /var/www/html or /usr/local/apache/htdocs
16. Upload ubr_finished.php to your webspace. The path will be /var/www/html or /usr/local/apache/htdocs
17. Upload ubr.css to your webspace. The path will be /var/www/html or /usr/local/apache/htdocs
18. Load ubr_file_upload.php in your web browser and attempt to do a file upload.
19. If your server is running Apache and you are using a public upload directory, it is recommended that you place a .htaccess text file in your /ubr_uploads/ directory. In that file put the single line "Options -Indexes" without the quotes. This will prevent people from getting at the raw directory listing of your upload directory. Of course, if for some reason you want people to be able to see the raw directory listing, then skip this step. Another way to lock down your upload directory is by simply placing an index.html file in your /ubr_uploads/ directory.

Your installation may look something like this:

/var/www/cgi-bin/ubr_pload.pl
/var/www/html/ubr_ini.php
/var/www/html/ubr_lib.php
/var/www/html/ubr_finished_lib.php
/var/www/html/ubr_default_config.php
/var/www/html/ubr_file_upload.php
/var/www/html/ubr_file_upload.js
/var/www/html/jquery-1.3.2.min.js
/var/www/html/ubr.css
/var/www/html/ubr_link_upload.php
/var/www/html/ubr_set_progress.php
/var/www/html/ubr_get_progress.php
/var/www/html/ubr_finished.php

Checklist

1. Make sure you set the proper (755) permissions for ubr_pload.pl
2. Make sure the path to perl is correct in ubr_pload.pl
3. Make sure you have (777) permissions on the upload and temp directories
4. Make sure all the PATH_TO values are correct in ubr_ini.php.

NOTE: UU will attempt to make the temp and upload directories but may fail due to permissions. If this occurs, you will have to make these directories manually and set the proper (777) permissions. If your upload directory is public, take this opportunity to lock it down.

• Multiple config files must be enabled by changing $_INI['multi_configs_enabled'] ini setting in ubr_ini.php.
  
• The config file must be initialized in the ubr_file_upload.php AND ubr_link_upload.php file.
  
• As of release 6.3.5 the config file name is no longer passed to the link upload script.
• As of release 3.0, UU supports multiple config files. This is a useful feature if you have several users who require to upload to several different locations. The file upload page can be coded to load different config files depending on what user loaded the page or what kind of upload the user is submitting. For example, lets say we have one user but they would like to upload images, word documents and spread sheets into different locations. In this case we need 3 more config files and a file upload page that will allow us to select a config file from a drop down and refresh the page selecting that config file. You could also load different config files based on user, you would simply put code at the top of the file upload page and link upload script that would load a config based on user id. eg.

if($_INI['multi_configs_enabled']){
	if($_SESSION['user'] == "tom"){ $config_file = 'ubr_tom_config'; }
	elseif($_SESSION['user'] == "dick"){ $config_file = 'ubr_dick_config'; }
	elseif($_SESSION['user'] == "harry"){ $config_file = 'ubr_harry_config'; }
	elseif($_SESSION['user'] == "sally"){ $config_file = 'ubr_sally_config'; }
	else{ $config_file = 'ubr_default_config'; }
}

UU supports two simple methods for transferring values through the upload process. Values passed will be available in both the ubr_upload.pl and ubr_finished.php scripts.

1. Post Values
   This method is good way to pass values if it is public and/or the end user is entering the values on the file picker page. For example lets say we want the user to enter their favorite color. Simply add the new input to the uu_upload form in the ubr_file_upload.php page. eg.

   <input type="text" name="fav_color" value="Enter favorite color">
   

   This value will be available inside the upload script ubr_upload.pl once the upload is "complete" as

   $query->param("fav_color");
   

   This value will be available inside the finished script ubr_finished.php as

   if(isset($_POST_DATA['fav_color'])){ do something; }
   

   or in the case of an array as

   if(isset($_POST_DATA['check_box'][0])){ do something; }
   

2. Config Values
   This method is good way to pass values if it is private data like a cookie or session value. For example lets say we want to pass the users password which is being stored in a session. We simply initialize a new config value and assign it the users password. This would be done in the ubr_link_upload.php script before it creates the link file for the upload. eg.

   $_CONFIG['user_password'] = $_SESSION['user_password'];
   

   This value will be available inside the upload script ubr_upload.pl once the link file is loaded as

   $config{'user_password'};
   

   This value will be available inside the finished script ubr_finished.php as

   $_CONFIG_DATA['user_password'];
   

   NOTE: You can also override config values in the ubr_link_upload.php script. eg.

   $_CONFIG['max_upload_size'] = $_SESSION['new_max_upload_size'];
   

Uber-Uploader is capable of displaying 4 different progress bar styles.

1. Default Progress Bar
   Default progress bar 'waits' for upload info and 'jumps forward' to the proper values. This is the most accurate progress bar style but it is not very sexy. If accuracy is your main concern or you may be dealing with small uploads, this is probably your best choice.
   Settings:

   $_CONFIG['cedric_progress_bar'] = 0;
   $_CONFIG['cedric_hold_to_sync'] = 0;
   $_CONFIG['bump_progress_bar'] = 0;
   

2. Cedric Progress Bar
   Cedric progress bar does 'not wait' for upload info and 'moves forward' in a smooth fashion. The rate in which it moves is continually adjusted by the upload info speed and time. It will aouto-correct itself by 'jumping forward or backward' when a reading is recieved. This style of progress bar is good if you are dealing with large uploads on a reliable line.
   Settings:

   $_CONFIG['cedric_progress_bar'] = 1;
   $_CONFIG['cedric_hold_to_sync'] = 0;
   $_CONFIG['bucket_progress_bar'] = 0;
   

3. Cedric Progress Bar With Sync Hold
   Same as Cedric progress bar but it will 'hold' and sync up with the actual upload info if it races ahead.
   Settings:

   $_CONFIG['cedric_progress_bar'] = 1;
   $_CONFIG['cedric_hold_to_sync'] = 1;
   $_CONFIG['bucket_progress_bar'] = 0;
   

4. Bucket Progress Bar
   Bucket progress bar 'waits' for upload info and 'moves forward' to the proper values. More accurate than a Cedric progress bar but tends to fall short of 100% progress on small uploads before refresh. This style of progress bar is better for large uploads.
   Settings:

   $_CONFIG['cedric_progress_bar'] = 0;
   $_CONFIG['cedric_hold_to_sync'] = 0;
   $_CONFIG['bucket_progress_bar'] = 1;
   

If your server is running Apache and you are using a public upload directory, it is recommended that you place a .htaccess text file in your /ubr_uploads/ directory. In that file put the single line "Options -Indexes" without the quotes. This will prevent people from getting at the raw directory listing of your upload directory. You may also prevent a raw file listing by simply putting a blank index.html file in the /ubr_uploads/ directory.

Address Bar Commands

www.yoursite.com/cgi-bin/ubr_upload.pl?about
www.yoursite.com/ubr_file_upload.php?about
www.yoursite.com/ubr_link_upload.php?about
www.yoursite.com/ubr_set_progress.php?about
www.yoursite.com/ubr_get_progress.php?about
www.yoursite.com/ubr_finished.php?about

Ini Settings

• $_INI['debug ajax'] (uploads possible in this mode)
  By enabling the $_INI['debug ajax'] ini setting, any call to the showDebugMessage will be displayed in an info box on the file upload page.
  This method can be used un the ubr_link_upload.php, ubr_get_progress.php and ubr_get_progress.php files.
• $_INI['debug_php'] (uploads impossible in this mode)
  Enabling the $_INI['debug_php'] ini setting will dump all your PHP info to screen when you load the file upload page.
• $_INI['debug_config'] (uploads impossible in this mode)
  Enabling the $_INI['debug_config'] ini setting will dump the config file to screen when you load the file upload page.
• $_INI['debug_upload'] (uploads impossible in this mode)
  Enabling the $_INI['debug_upload'] ini setting will dump all your Perl info to screen when you submit an upload. UPLOADS WILL FAIL IN THIS MODE!!!
• $_INI['debug_finished'] (uploads possible in this mode)
  Enabling the $_INI['debug_finished'] ini setting will dump all your XML, CONFIG, FILE and POST values to screen when the upload finished page is loaded.
• $_INI['php_error_reporting'] (uploads possible in this mode)
  Turns on PHP error_reporting(E_ALL);.
• $_INI['delete_link_file']
  Disable this ini setting if you want to examine the upload_id.link file inside the temp directory.
• $_INI['delete_redirect_file']
  Disable this ini setting if you want to examine the upload_id.redirect file inside the temp directory.
• kak statement
  Use the kak statement inside the Perl or PHP files to print data.

  kak(String, Boolean, __LINE__)
  
  String    =  The string you want to print out
  Boolean   =  Exit or contine script execution
  __LINE__  =  Line number of the print statement
  

  eg.

  &kak("This is the value of foo: $foo", 0, __LINE__);
  &kak("This is the value of Bar: $bar", 0, __LINE__);
  $foo_bar_sum = $foo + $bar;
  &kak("This is the sum of foo bar: $foo_bar_sum", 1, __LINE__);
  

The first step to solving the flength file issue is to eliminate the obvious.

1. Make sure the $TEMP_DIR setting is EXACTLY the same in the ubr_ini.php and ubr_upload.pl files.
2. Make sure the ubr_upload.pl file executes by calling it with the about command from your browser. eg. www.yoursite.com/cgi-bin/ubr_upload.pl?about
3. Max execution time of PHP or Perl scripts are set too low. See 'DEBUGGING TOOLS->INI SETTINGS->DEBUG_PHP and DEBUG_UPLOAD' section.

Assuming the script is set up properly, you are probably dealing with some kind of server side write-caching. Some servers perform write-caching which prevents writing out the flength file or the entire CGITemp file during the upload. The flength file or the CGITemp file do not actually hit the disk until the upload is complete, making it worthless for reporting on progress during the upload. So far 3 reasons have been identified along with possible fixes.

1. A mod_security module for Apache
   This may be fixed using a .htaccess file assuming your host supports them. To fix it just create a file called .htaccess (that's a period followed by "htaccess") and put the following lines in that file. Upload the file into the directory where the Uber-Uploader CGI ".pl" scripts resides, or in some directory above it (like your server's DOCUMENT_ROOT, i.e. the top-level of your webspace). .htaccess files must be uploaded as ASCII mode, not BINARY. You may need to CHMOD the .htaccess file to 644 or (RW-R--R--).

   <IfModule mod_security.c>
     # Turn off mod_security filtering.
     SecFilterEngine Off
   
     # The below probably isn't needed,
     # but better safe than sorry.
     SecFilterScanPOST Off
   </IfModule>
   

   If the above method does not work, try putting the following lines into the file

   <IfModule mod_security.c>
     SetEnvIfNoCase Content-Type \
     "^multipart/form-data;" "MODSEC_NOPOSTBUFFERING=Do not buffer file uploads"
   </IfModule>
   
   <IfModule mod_gzip.c>
     mod_gzip_on No
   </IfModule>
   

2. "Performance Cache" enabled on OS X SERVER
   If you're running OS X Server and the progress bar isn't working, it could be because of "performance caching." Apparently if ANY of your hosted sites are using performance caching, then by default, all sites (domains) will attempt to. The fix then is to disable the performance cache on all hosted sites.
3. Lighttpd web server
   Lighttpd web server is known to buffer posts. To fix add one of the following lines to the lighttpd.conf file.

   lighttd 1.5.x -> server.network-backend = "writev"
   

   or

   1.5.x -> server.network-backend = "linux-sendfile"
   

NOTE: If the above information does not solve your problem, your time is probably best spent finding a Java or Flash based uploader. Any other Perl based uploader will have the same issue.

If your site consists of several servers in a cluster, you may find that it occasionally fails to find your uploaded files. This may be due to the fact that you uploaded a file to server 'A' but were redirected to server 'B' when the upload was finished. When the 'ubr_finished.php' script is fired on server 'B' it looks in server 'B's' temp directory for the 'sid.xml' file which it cannot find because it is on server 'A'. One possible solution is setting the refresh method to 'location' and using a relative 'redirect_url' in relation to the perl script. This would force the script to redirect to the same physical machine were the upload occured.
For example if we assume our setup is like so

/cgi-bin/ubr_upload.php
/htdocs/members/ubr_finished.php

We would set the following config file setting

$_CONFIG['redirect_url']  = '../members/ubr_finished.php';

and the following ini setting

$_INI['redirect_method']  = 1;

The TEMP_DIR ini setting is assumed to be $TEMP_DIR = '/tmp/ubr_temp/';

1. Enable the $_INI['debug_ajax'] ini setting in ubr_ini.php.
2. Load ubr_file_upload.php in your web browser.
3. Open a command line (shell) to your web server and go to the TEMP_DIR directory.
4. Start a large upload.
5. Note the 32 character Upload ID (eg. 325d7d52d78083839d4f9ced94821695) that appears in the AJAX DEBUG WINDOW.
6. While your files are uploading, get a listing of your TEMP_DIR directory.
7. You should now see a directory in TEMP_DIR called 325d7d52d78083839d4f9ced94821695.dir.
8. Now get a listing of the 325d7d52d78083839d4f9ced94821695.dir directory.
9. You will see a file called 325d7d52d78083839d4f9ced94821695.flength. This is the file that contains the size of the upload in bytes.
10. You will also see a file like CGItemp3984. This is a file being uploaded. It will change in size as the file is uploaded.
11. If there is more than one upload, there will be multiple CGItemp???? files.

So with three active file uploads, the file structure will look something like this:

/tmp/ubr_temp/325d7d52d78083839d4f9ced94821695.dir/325d7d52d78083839d4f9ced94821695.flength
/tmp/ubr_temp/325d7d52d78083839d4f9ced94821695.dir/CGItemp3984
/tmp/ubr_temp/325d7d52d78083839d4f9ced94821695.dir/CGItemp3985
/tmp/ubr_temp/325d7d52d78083839d4f9ced94821695.dir/CGItemp3986

PROGRESS BAR NOTES:

• The first thing the progress bar does is look for the .flength file in the /tmp/ubr_temp/325d7d52d78083839d4f9ced94821695.dir/ directory.
  If it cannot find the .flength file after 10 seconds, it will throw the "failed to find flength file" error and quit.
• If the .flength file is found the progress bar will read the total upload size in bytes from the file. The progress bar then keeps reading the size of the /tmp/ubr_temp/325d7d52d78083839d4f9ced94821695.dir directory as files are uploaded and the directory grows in size. It keeps comparing this number to the one it got from the flength file.
  That is how the progress bar knows how much information has been uploaded.

The TEMP_DIR ini setting is assumed to be $TEMP_DIR = '/tmp/ubr_temp/';

1. Enable the $_INI['debug_ajax'] ini setting in ubr_ini.php.
2. Load ubr_file_upload.php in your web browser.
3. Open a command line (shell) to your web server and go to the TEMP_DIR directory.
4. Start a large upload.
5. Note the 32 character Upload ID (eg. 325d7d52d78083839d4f9ced94821695) that appears in the AJAX DEBUG WINDOW.
6. While your files are uploading, get a listing of your TEMP_DIR directory.
7. You should now see a directory in TEMP_DIR called 325d7d52d78083839d4f9ced94821695.dir.
8. Now get a listing of the 325d7d52d78083839d4f9ced94821695.dir directory.
9. You will see a file called 325d7d52d78083839d4f9ced94821695.flength. This is the file that contains the size of the upload in bytes.
10. You will also see a file called 325d7d52d78083839d4f9ced94821695.hook. This is the file that contains the the upload info.

The file structure will look something like this.

/tmp/ubr_temp/325d7d52d78083839d4f9ced94821695.dir/325d7d52d78083839d4f9ced94821695.flength
/tmp/ubr_temp/325d7d52d78083839d4f9ced94821695.dir/325d7d52d78083839d4f9ced94821695.hook

PROGRESS BAR NOTES:

• The first thing the progress bar does is look for the .flength file in the /tmp/ubr_temp/325d7d52d78083839d4f9ced94821695.dir/ directory.
  If it cannot find the flength file after 10 seconds, it will throw the "failed to find flength file" error and quit.
• The second thing the progress bar does is look for the .hook file in the /tmp/ubr_temp/325d7d52d78083839d4f9ced94821695.dir/ directory.
  If it cannot find the hook file after 10 seconds, it will throw the "failed to find hook file" error and quit. The hook file contains to following upload info in the following style:
  total_bytes_uploaded:number_of_files_uploaded:current_file_uploading:bytes_uploaded_of_the_current_file

UU is capable of performing multi simultaneous uploads if your web browser is configured properly. UU requires two connections per upload.
One connection is required for the upload and one connection is required for the progress bar. By default IE and Firefox ships with the ability to make several simultaneous connections. However, you can increase the the amount of simultaneous connections by changing the following values. Just 'Google' for instructions on changing these values.

• IE

  MaxConnectionsPer1_oserver
  MaxConnectionsPerServer
  

• Firefox

  network.http.max-connections
  network.http.max-connections-per-server
  network.http.max-persistent-connections-per-server
  

As of release 4.5 Uber-Uploader has changed all content-type headers to UTF-8 which will add greater support for non-latin languages. This feature is not 100% so you will need to do some of your own development. Below are some helpful links.

• http://www.phpwact.org/php/i18n/charsets
• http://www.nicknettleton.com/zine/php/php-utf-8-cheatsheet
• http://www.joelonsoftware.com/articles/Unicode.html

To see your PHP settings, enable the "DEBUG_PHP" ini setting in the "ubr_ini.php" file and load "ubr_file_upload.php" in your browser. This mode will dump all your PHP settings to screen.

To see your Perl settings, enable the "DEBUG_PERL" value in the "ubr_upload.pl" file and load "ubr_upload.pl" in your browser. This mode will dump all your Perl settings to screen. Alternatively, you enable the $_INI['debug_upload'] ini setting and post an upload. This method will dump all your Perl settings to screen along with the loaded config settings.

LimitRequestBody 
KeepAlive 
KeepAliveTimeout 
MaxClients

max_execution_time 
max_input_time 
memory_limit