Completion of BitTorrents - hashed patch-resume

[Mikademus]

Hi!

I want to try a thought on you here, I'd prefer not to be too flamed if you don't like it...

I am using BitTorrent a lot as I'm sure a lot of you do, mostly for anime. Every so often the tracker or the last seed disappears and one is left with an incomplete file, often almost done - highly annoying, eh? I've also often found some of these files on the DC hubs I frequent.

Well, my thought is, since DC++ supports resume, and since it obviously can handle hashing (though if it uses the same hash-algo as BT i wouldn't know), it would be wonderful if DC++ could "patch-resume" the incomplete torrent file, that is fill in the blanks? The client would "resume" the first incomplete segment, then break the connection (or something) and then resume the next, and so on. I assume this would require one to manually specify (or import) a file for hash-checking / patch-resumation.

Beside allowing one to finish incomplete torrents while minimising personal and uploader bandwith usage this could also solve the annoying (old and gone?) bug with corrupted downloads.

Regards

[GargoyleMT]

I've mentioned a similar scheme before. If you have a hash for a file that you're downloading, you could import any file as a partial for it, as long as you had leaves to check against. For this to happen, DC++ would have to take a step closer to multi-source downloading, since the failed blocks could be anywhere in the file, and not contiguous.

But yes, it's possible, though not realistic in the next couple versions because of the amount of work that would have to be done on the back end.

[Lundis]

Currently there is only one way to mix DC and BT.
You can download a bit with dc and if you get tired of low speed or the only user left you can resume with bittorrent.
However you can't go back to dc with that file cause that would require a lot from the filesize and rollback to say the least. BT downloads the pieces at no particular order.

DC++ currently has hashing for the whole file, BT has hashes of small pieces (256 kb to one mb typical). So DC can't check pieces of any files and resume them.
But about future development I don't know.

[Mikademus]

Mmm, perhaps a better topic for the request would have been something like "hashed checking and resumation of downloads" - mentioning BitTorrent might have confused the idea, even though I (think I) would primarily use it to finish incomplete torrents.

A hashed check would - given that you know the hash/CRC/whatever of the file in question, which you very often do - would of course allow not only resumation of incomplete torrents but also correcting corrupted files etc.

That is, the feature would have wider uses and benefits than I though of when posting the original suggestion.

However, I do not understand why it would require stepping closer to multi-source downloading. That it would entail work on the client back end is only natural, but there is nothing that says that hashed/patch resume requires multi-source downloading, even though it would marry naturally with such functionality. In its purest form a hashed/patched resume is just a standard resume at a specified position that overwrites a certain and limited range of data without discarding the data following, which should have nothing to do with which or how many sources you download from.

[GargoyleMT]

DC++ currently has hashing for the whole file, BT has hashes of small pieces (256 kb to one mb typical). So DC can't check pieces of any files and resume them.
But about future development I don't know.

As of 0.403, the leaf hashes for the file (the hashes of the segments) are exchanged between (0.403+) users. According to BigMuscle, these aren't used for resumed transfers, but they are for normal transfers.

[Big Muscle]

but it's bad because when you download without resuming, you don't need check it.

[foxyshadis]

However, I do not understand why it would require stepping closer to multi-source downloading. That it would entail work on the client back end is only natural, but there is nothing that says that hashed/patch resume requires multi-source downloading, even though it would marry naturally with such functionality. In its purest form a hashed/patched resume is just a standard resume at a specified position that overwrites a certain and limited range of data without discarding the data following, which should have nothing to do with which or how many sources you download from.

Just to clarify what you brought up, it's not that this depends on multi-source downloading, it's that multi-source and chunked resume (I'll make up a term) both require many of the same underlying features. Once you have the ability to send pieces of files it won't matter if you use it to get a file from five people at a time or fill in missing sections from other p2p services, or both.

Major protocol changes have to be made to support each piece, and the client has much more work to track where in the file data is being filled in, which entails a great deal of coding.