split file download

[Peon]

Currently i am using curl to download a file. It stores all the content of file in a variable then write locally using fwrite function. It does the task successfully but with one limitation that is when large file is being downloaded it gives no indication to user about how much file is downloaded locally.

So, I am looking for some way so that file can be downloaded in parts or writing locally after every 10mb (or whatever) is downloaded.

Your help is appreciated.

Thanks.

[Peon]

please tell me .. this kind of solution is not possible or i need to explain the situation bit more?

your input is appreciated.

[visualAd]

You would need to use PHP's fsockopen function to download the file in parts. PHP is not really an event driven language and updating a page during download is often unsuccessful. Your PHP script would need to run-unattended and update a file or database with the status as it goes.

You can then write a script using Javascript which will periodically poll the server for the download progress. Sorry, its not an easy solution.

[Peon]

I can deal rest of the part...

but how to download file in parts using fsockopen.. i tried google but didn't find any helpful resource.

Thanks.

[visualAd]

When you open a file with fsockopen you are returned a resource which can be used with functions such as fread and fwrite. These functions allow you to read / write an arbitrary number of bytes and thus you could download 1KB at a time, updating a status in the database after every 1KB.

$hwnd = fsockopen("www.google.com", 80);

$http = "GET / HTTP/1.0\r\n";
$http .= "Host: www.google.com\r\n\r\n";

fwrite($hwnd, $http);

$data = '';
$byteCount = 0;
while(! feof($hwnd)) {
    $data .= fread($hwnd, 1024);
    $byteCount += 1024;

   // write to file or DB here. N.b: this WILL slow the download down.
} 

[Peon]

It will not unnecessarily consume up memory.. because

$data variable will store every 1kb in loop.. ??

I think.. wouldn't it be better if there could be someway to download .. suppose there is 30mb file . first download first 10mb and write then download second 10mb and write (actually append the downloaded file) and then download last 10mb and append the previously 20mb downloaded file ?

[visualAd]

Of course, you can store the data from the file however you like. I would suggest that if it is a large file you write the data straight into the destination file to avoid using copious amounts of memory. again, you can get as little or as much data in each iteration as you want; the above code is only an example so change it a you please.

A side note, the default memory limit for PHP is 8MB.

[Peon]

As file size is big so it would be better to save directly to destination file. I have confusion about appending the data in destination file... how it will be done? Because everytime 10 mb would be downloaded.. and i don't want to corrupt the file as well.

Thanks.

[visualAd]

Use fopen to open the file in write / append mode and similar to downloading the data, use fwrite to add the data to the file.

[Peon]

I think my concept with fread is little weaker..

I know fwrite will write data at then end if mode selected is append. But how fread will read data from particular location to particular bytes?

Suppose in case i want to download 100mb file and fread has to read file from 51 mb to 60mb..

[Peon]

and yes.. fsockopen open handle with hostname not file.. as i want to download file from host.. i tried to search in google but nothing helpful found.. might be my failure but i tried.

[visualAd]

and yes.. fsockopen open handle with hostname not file.. as i want to download file from host.. i tried to search in google but nothing helpful found.. might be my failure but i tried.

Please read post #5.

[visualAd]

I think my concept with fread is little weaker..

I know fwrite will write data at then end if mode selected is append. But how fread will read data from particular location to particular bytes?

Suppose in case i want to download 100mb file and fread has to read file from 51 mb to 60mb..

Fread does not provide that functionality and cannot and should not as you are reading from a pipe. In order to get a range of bytes from the web server you need to use the HTTP/1.1 protocol when making the request and set the Range header.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

Web servers are not required to support the range header so in those cases you would need to restart the download.

[Peon]

Please read post #5.

Sorry, I was about to delete that post because i found solution at
http://www.issociate.de/board/post/4...kopen()_?.html

[visualAd]

Sorry, I was about to delete that post because i found solution at
http://www.issociate.de/board/post/4...kopen()_?.html

It was in post number 5

[visualAd]

Sorry, I was about to delete that post because i found solution at
http://www.issociate.de/board/post/4...kopen()_?.html

Additionally, that algorithm is broken because it makes no allowances for chucked responses.

[Peon]

Apparently my following code is correct but it never displays "exist"

$img = "images/nav_logo3.png";

$hwnd = fsockopen("www.google.com", 80, $errno, $errstr, 30);

if (!$hwnd) 
    {
        echo "$errstr ($errno)<br />\n";
    } 
else
    {
        $out = "GET /$img / HTTP/1.0\r\n";
        $out .= "Host: www.google.com\r\n";        
        
        fwrite($hwnd, $out);
        $file_exists = false;
        while (!feof($hwnd)) 
            {                
                $header = fgets($hwnd, 128);
                if(preg_match('#HTTP/1.1 200 OK#', $header)) 
                    {
                        $file_exists = true;
                        echo "exists";
                        
                        break;
                    }
            }
        
        
    } 

What does it mean.. something is wrong with fsockopen?

because that file exists at http://www.google.com/images/nav_logo3.png
so it should return with file exist header 200.

[Peon]

btw when i tried to display $hwnd .. it contains "Resource id #2" :-s

[visualAd]

You have sent the request using HTTP 1.0, so the response received will also be HTTP 1.0 not HTTP 1.1. You also need to ensure that the HTTP request you are sending meets the protocol requirements. I.E:

METHOD /path/to/resource HTTP/1.X\r\n
Header1: value\r\n
Header2: value\r\n
\r\n
\r\n

Note the two CRLF's at the end of the request.

[Peon]

You have sent the request using HTTP 1.0, so the response received will also be HTTP 1.0 not HTTP 1.1. You also need to ensure that the HTTP request you are sending meets the protocol requirements. I.E:

METHOD /path/to/resource HTTP/1.X\r\n
Header1: value\r\n
Header2: value\r\n
\r\n
\r\n

Note the two CRLF's at the end of the request.

I am using Firefox not IE.

I also edited second HTTP/1.1 to HTTP/1.0 but still same result.

Have you tried this code.. does it print "exist" for you?

[Peon]

I have changed code to but still doesn't work.

$img = "images/nav_logo3.png";

$hwnd = fsockopen("www.google.com", 80, $errno, $errstr, 30);

if (!$hwnd) 
    {
        echo "$errstr ($errno)<br />\n";
    } 
else
    {
        $out = "GET /$img / HTTP/1.0\r\n";
        $out .= "Host: www.google.com\r\n";        
        
        fwrite($hwnd, $out);
        $file_exists = false;
        while (!feof($hwnd)) 
            {                
                $header = fgets($hwnd, 128);
                if(preg_match('#HTTP/1.0 200 OK#', $header)) 
                    {
                        $file_exists = true;
                        echo "exists";
                        
                        break;
                    }
            }
        
        
    }
$myFile = "newdir/test.gif";
$data = '';
$byteCount = 0;

$fh = fopen($myFile, 'a') or die("can't open file");

while(! feof($hwnd)) 
    {                
        $data = fread($hwnd, 100);
        $byteCount += 100;
            
        fwrite($fh, $data);     
    } 
fclose($fh); 

though file is created but of ZERO size.

[visualAd]

I am using Firefox not IE.

I also edited second HTTP/1.1 to HTTP/1.0 but still same result.

Have you tried this code.. does it print "exist" for you?

It doesn't matter what browser you use, the HTTP protocol is still the same and you are still not following it . The end of an HTTP request is identified with a blank line. You are not including a blank line at the end of your request so as far as the HTTP server at the other end is concerned, you are still sending your request.

http://www.w3.org/Protocols/rfc2616/...sec5.html#sec5

The above is the HTTP 1.1 specification pertaining to the request. You need to follow it to the letter.

[visualAd]

P.s: if you want to test your requests and view the responses then I would suggest you download a terminal program. I use putty and configure it to connect with a raw data connection to the web host on port 80.

You can format the requests in notepad, then copy and right click to paste once the connection has been established.

[Peon]

btw i displayed $header and it contains

HTTP/1.0 400 Bad Request Date: Fri, 01 Aug 2008 20:13:06 GMT Content-Type: text/html; charset=UTF-8 Server: GFE/1.3 Connection: Close Content-Length: 1344
Google
Error


Bad Request
Your client has issued a malformed or illegal request.

probably request is not being made correctly.

[visualAd]

Like I said in my previous post, you have neglected the blank line at the end of your request. There is also an issue with a URI, you have a space in the file path which should not be there.

[Peon]

My updated code is.. with this code file is created of same size 6336 bytes but its not a valid file.. and second while(! feof($hwnd)) loop doesn't stop when all 6336 bytes are read. I think i am pretty close to solution now.

$img = "images/nav_logo3.png";
$size = 6336;

$hwnd = fsockopen("www.google.com", 80, $errno, $errstr, 30);

if (!$hwnd) 
    {
        echo "$errstr ($errno)<br />\n";
    } 
else
    {
        $out = "GET /$img HTTP/1.0\r\n";
        $out .= "Host: www.google.com\r\n";        
        $out .= "\r\n";
        $out .= "\r\n";        
        
        fwrite($hwnd, $out);
        $file_exists = false;
        while (!feof($hwnd)) 
            {                
                $header = fgets($hwnd, 128);
                if(preg_match('#HTTP/1.0 200 OK#', $header)) 
                    {
                        $file_exists = true;
                        echo "exists";
                        
                        break;
                    }
            }
        
        
    }
$myFile = "newdir/test.gif";
$data = '';
$byteCount = 0;

$fh = fopen($myFile, 'a') or die("can't open file");

while(! feof($hwnd)) 
    {                
        $remsize = $size - $byteCount;        
        $bytes = $remsize > 100 ? 100 : $remsize;
        
        $data = fread($hwnd, $bytes);
            
        fwrite($fh, $data);     
    } 
fclose($fh); 

[Peon]

I have put following code to break out of while loop. But file written is not valid.. I mean its not a graphics file downloads from source correctly.

if ($byteCount == $size) break;

[visualAd]

There are two problems with your code above. The first is that you are not ignoring the HTTP response header. Like the request, the reponse has the status (which) you are checking and zero or more headers such as cookies and document information. Like the request, the response header is terminated with a blank line; thereafter comes the response body.

Below is an example of the response from the Google UK home page.

HTTP/1.0 302 Found
Location: http://www.google.co.uk/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: PREF=ID=0d75f51e34fe4354:TM=1217671182:LM=1217671182:S=xoN1ZSHmGD_5Gyx5; expires=Mon, 02-Aug-2010 09:59:42 GMT; path=/; domain=.google.com
Date: Sat, 02 Aug 2008 09:59:42 GMT
Server: gws
Content-Length: 221
Connection: Close

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
                                                                              <TITLE>302 Moved</TITLE></HEAD><BODY>
                                   <H1>302 Moved</H1>
                                                     The document has moved
                                                                           <A HREF="http://www.google.co.uk/">here</A>.
</BODY></HTML>

Again, note the blank line before the HTML starts. Instead of reading the first 128 bytes of the header; which may contin the whole head, some of the header and the body or only the partial header, you need to read lines from the response until you find a blank line:

$line ='';

while((!feof($hwnd)) && ($line!="\r\n")) {
    $line= fread($hwnd); // gets a single line when no size is specified.
} 

Once you have found the newline, the remaining data is the request body. Provided this has not been encoded in some way (which will only happen if you state in the request that you support it), it should be the byte stream of the file.

The second issue is that you are opening the file in append mode. ideally, you want to be creating a new one otherwise the data from the current download will simply be appended to that of the previous. This would of course produce a corrupt file. I suggest you open the file for writing and truncate it "wb".

[Peon]

I think here confusion lies.. because the code i posted in my second last post print "exist" so there is no problem.

Problem lies with writing of byte streams in file.

And yes.. I delete file every time manually so there is no such issue like every time data is appended to previously created file.

Have you tested my above code?
It writes file of same size but file is corrupt. (its the headache now..)

[visualAd]

I think here confusion lies.. because the code i posted in my second last post print "exist" so there is no problem.

Problem lies with writing of byte streams in file.

And yes.. I delete file every time manually so there is no such issue like every time data is appended to previously created file.

Have you tested my above code?
It writes file of same size but file is corrupt. (its the headache now..)

I have just told you why the file is corrupt and what you need to do to fix the problem. The size of the file is irrelevant, you do not need to know it, so just take it out and read 1KB at a time until you find the eof and make the changes I suggested in my previous post to ensure that the entire header has been read before hand.

while(! feof($hwnd))
    {                
        $data = fread($hwnd, 1024);
            
        fwrite($fh, $data);    
    } 

The fact that your code prints exists simply indicates that the HTTP request returned a response that contains "HTTP/1.0 200 OK", nothing more.

[Peon]

Well, I have incorporated all suggested changes very carefully.. and now written file is of more size and still corrupt.

Please have a look at my code.. (i also changed file location)

$url = "http://www.phpdig.net/images/extra/vbulletin3_logo_white2.gif";

$parsed = parse_url($url);
$host = $parsed["host"];

$img = "images/extra/vbulletin3_logo_white2.gif";

$hwnd = @fsockopen($host, 80, $errno, $errstr, 20);

if (!$hwnd) 
    {
        echo "$errstr ($errno)<br />\n";
    } 
else
    {
        $out = "GET /$img HTTP/1.1\r\n";
        $out .= "HOST: $host\r\n";        
        $out .= "Connection: close\r\n\r\n";        
        
        fwrite($hwnd, $out);
        $file_exists = false;
        
        $line ='';

        while((!feof($hwnd)) && ($line!="\r\n"))
            {                
                $line = fgets($hwnd);                                
                if(preg_match('#HTTP/1.1 200 OK#', $line)) 
                    {
                        $file_exists = true;
                        echo "exists";
                        
                        break;
                    }
            }
        
        
    }
$myFile = "newdir/test.gif";
$data = '';
$byteCount = 0;
$i = 0;

$fh = fopen($myFile, 'a') or die("can't open file");

while(!feof($hwnd)) 
    {        
        $data = fread($hwnd, 1024);                    
        fwrite($fh, $data);        
    } 
fclose($fh); 

btw you tried to execute my code? maybe something is wrong with my machine.

[visualAd]

Is there a reason why you still need the break in the first while loop before you have finished reading the header?

[visualAd]

btw you tried to execute my code? maybe something is wrong with my machine.

I am sure your machine is fine, if you make the changes I have been suggesting then you should have no problem running it. It runs here fine:

http://test.sccode.com/peon.php

[kimberly12]

Use fopen to open the file in write / append mode and similar to downloading the data, use fwrite to add the data to the file.