Splitted hashing

[Astor]

This might sound a little bit crazy but is it possible to do a splitted hashing? I mean a hash for every MB of data and one for the entire file (if is > 1MB).

Next step would be to automatically re-download that buggy MB (instead of re-downloading the entire file).

For a 700MB data file (a large text file) file.txt and a MD5 hashing we would have: 700x32 (MD5 hashes)+700x8 (offset of hashed MB)=28kb a very small amount of data for ensuring that every file will be correctly downloaded (rollback feature deasn't work every time).

So?

[Todi]

Already in the works. Look for it in DC++ 0.402.. although 1MB is probably a bit too small, since all the partial hashes have to be saved in the hashdatabase.. it get's quite big if you have a lot of 700MB (or bigger), um, text files. Google for Tiger Tree Hashes and you'll see what makes them so useful just for this reason.

Of course, one could say that the ZBlock feature already does this.

[Astor]

Great news!

But, why too small?
1GB=5K
100GB=50k
2.000GB=1MB DB

For an ISDN connection (128kbps) re-downloading tens of megs doesn't make sense..

[Todi]

Hmm, well, PseudonympH claims that today it's 64k or 1/512th of the file, so i'll just rectract any statements i've made and go to bed now.

[GargoyleMT]

This might sound a little bit crazy but is it possible to do a splitted hashing?

It's so crazy that it's already happened. DC++ uses TTH - tiger tree hashing - which gives you the option of verifying smaller segments inside a file. The root hash is then used as a unique ID for the file. If DC++ used a straight Tiger hash algorithm, the hash would be different, and would only enable you to verify if hte download was good or not.

TTHes are used in many applications, feel free to search the fine web for more information.