Download Accelerator

[winston2020]

OK, I've been trying to figure this one out for a while now. What I have right now is a program that opens multiple connections to the server (using separate threads).

Let's say for example that I open 5 threads. The program will determine the file size / 5. The last piece will contain any extra bytes that were not accounted for in the division because of rounding error.

So, the first thread spawns, and begins downloading. The second spawns, and uses the InputStream.skip() method to jump over the first thread's bytes. But since the skip() method takes the same amount of time as the actual reading of the bytes, nothing is sped up. In fact it takes longer. AND uses more bandwidth because the file has to be transferred 5 times.

So, how do programs like Download Accelerator Plus utilise multiple connections to download faster? I don't understand how they can simply begin reading at a designated spot in the file.

(I'll post my code so far if you guys want it, but it's kind of messy at the moment)

[TheCPUWizard]

Why have you created a new thread???

[winston2020]

Why have you created a new thread???

Different topic.

To Mods: you can delete this if you want, I wasn't sure wether to make a new one or not

[TheCPUWizard]

Different topic.

To Mods: you can delete this if you want, I wasn't sure wether to make a new one or not

Looks like same topic... You were using .skip() and not getting the results you wanted (parallel downloads of chunks of the same file).

How does this differ??? [it is late :1:15AM, so I may be missing something....]

[winston2020]

Looks like same topic... You were using .skip() and not getting the results you wanted (parallel downloads of chunks of the same file).

How does this differ??? [it is late :1:15AM, so I may be missing something....]

I really didn't mean to start anything here. I see it as being different because the first post was to clarify why the skip() method was taking so long, and a possible work around. I intended this one to be a discussion on the best possible algorithm for a download accelerator. If you feel it's wrong to have two threads then please, by all means have one of them removed :)

[TheCPUWizard]

We still have not actually seen the way you are using skip (unless I missed a post where you put up a minimal yet complete example....

Programs like download accellerator actually use a number of techniques depending on the capabilities of the SERVER.

We still dont know what server(s) you need to be talking to, do you control the servers, what is the nature of the url, EXACTLY how you are opening the connect...

All things that *MAY* allow you to use skip.

To understand why this is non trivial, consider if you start downloading a video. If the entire video exists when you start then 9in theory) you could download different chunks. But what if it was a live feed :eek:

Pretty difficult to start downloading what is happening now, and what is going to happen 30 minutes into the future at the same time. :lol:

BUT from a client perspective BOTH of these situations may be represented as just a file, with no way of detecting on the client side what the reality is.....

[winston2020]

We still have not actually seen the way you are using skip (unless I missed a post where you put up a minimal yet complete example....

Programs like download accellerator actually use a number of techniques depending on the capabilities of the SERVER.

We still dont know what server(s) you need to be talking to, do you control the servers, what is the nature of the url, EXACTLY how you are opening the connect...

All things that *MAY* allow you to use skip.

To understand why this is non trivial, consider if you start downloading a video. If the entire video exists when you start then 9in theory) you could download different chunks. But what if it was a live feed :eek:

Pretty difficult to start downloading what is happening now, and what is going to happen 30 minutes into the future at the same time. :lol:

BUT from a client perspective BOTH of these situations may be represented as just a file, with no way of detecting on the client side what the reality is.....

Well, my hope was that it would be able to download from *most* servers, so, no, I don't control the server. The files will already exist when I download them.

Here's a rough example:

URL url = new URL("http://somefilelocation.com/filename.ext");
URLConnection c = url.openConnection();

BufferedInputStream in = new BufferedInputStream(c.getInputStream());
BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream(new File("C:\\output.ext")));

in.skip(offset);

long bytesReadSoFar = 0;
byte[] buffer = new byte[8192];
int bytesRead = in.read(buffer);

while(bytesRead != -1 && (bytesReadSoFar < pieceSize)){
for(int i = 0; i < bytesRead; i++){
out.write(buffer[i]);
}

bytesReadSoFar += bytesRead;
bytesRead = in.read(buffer);
}

in.close();
out.flush();
out.close();


Just to note, pieceSize and offset are set in the constructor of the class. Just assume they are correct (which they are; I checked :P)