Update: Downloading all archive.org metadata
Update: Downloading all archive.org metadata
Following up from my previous post.
I used the API at https://archive.org/developers/changes.html to enumerate all the item names in the archive. Currently there are over 256 million item names. However I went through a sample of them and noted the following:
- Many do not have the .torrent available because some of the files are locked due to copyright concerns, like their music collection. Ex: https://archive.org/details/lp_le-sonate-per-pianoforte-vol-1_carl-maria-von-weber-dino-ciani_0
- A lot of items have been removed from public access completely, and possibly deleted even on their storage backend. Ex: https://archive.org/details/0-5-1-0-hernan-hernandez
There are many, many items from the archive which have been removed. Much higher than I expected. If you have critical data, of course Internet Archive should never be your only backup.
I don't know the distribution of metadata and .torrent file sizes since i have not tried downloading them yet. It looks like it would require a lot of storage if there are many files or the content is huge (if only 50% of the items remain and the average .torrent + metadata is 20KB it would be over 2.5 TB to store). But on the other hand, the archive has a lot of random one off uploads that are not very big, so some metadata is 800 bytes and the torrent 3KB in those cases (only 640 GB to store if combined is 5 KB).
If you store it in compressed chunks (or on a file system that supports compression, I guess), that should be a great deal smaller, being text only.
They supposedly never delete good stuff, just make it unavailable, as you said. Maybe we'll get them in a hundred or so years!
ArchiveTeam's uploads also don't have torrent files (anymore). Their wiki says that they disabled it to lighten the load on IA's servers, as creating a .torrent file for a 10+GB upload takes considerable time, especially if it has to be redone.