Tar doesn't use any sort of index like zip does, so to extract the specified file the server side would need to parse through possibly the entire file just to see if the requested file is there, and then start streaming it. Requests for files that aren't in the tar archive would be prohibitively expensive.
There are definitely ways to do it without those problems, though. They just wouldn't be quite as simple as the approach done for supporting zip.
We considered TAR, but indexing requires reading back and decompressing the entire archive.
This may be feasible on small TAR files, and for single PutObject you could index while uploading. However for multipart objects, parts can arrive in any order so you are forced to read it back. This would lead to unpredictable response times.
Compare that to reading the directory of a zip, which maybe on big files are a couple of megabytes max.
Add to that that tar.gz will require you to decompress from the start to reach any offset. You can recompress while indexing, but an object-store mutating your data is IMO a no-no.
S3 is "eventually consistent", so I don't think indexing in the background would be such a big deal. But yeah, like I said this would only work for no-compression or those schemes that are seekable (not gzip).
In any case it is definitely a lot more work than ZIP.
No, S3, as MinIO, has a read-after-write consistency.
So indexing would block on either writes or reads until it is done. We block when doing the zip indexing, but that is much more lightweight - and we limit to 100MB ZIP directory. That way we don't risk long-blocking index operations.
IIRC gzip can't handle this, but bzip2 can; a guy I know wrote an offline Wikipedia app for the original iPhone and had to crunch things down a lot, and he used bzip2 because you can skip ahead to a chunk without having to process the previous or subsequent chunks.
Then he just had to write some code to index article names based on which chunk(s) they were in, and boom, random-access compressed archive.
This is basically exactly what we do we have created a cloud optimised tar (cotar)[1] by creating a hash index of the files inside the tar.
I work with serving tiled geospatial data [2] (Mapbox vector tiles) to our users as slippy maps where we serve millions of small (mostly <100KB) files to our users, our data only changes weekly so we precompute all the tiles and store them in a tar file in s3.
We compute a index for the tar file then use s3 range requests to serve the tiles to our users, this means we can generally fetch a tile from s3 with 2 (or 1 if the index is cached) requests to s3 (generally ~20-50ms).
To get full coverage of the world with map box vector tiles it is around 270M tiles and a ~90GB tar file which can be computed from open street map data [3]
> Though even that would only work with a subset of compression methods or no compression.
We compress the individual files as a work around, there are options for indexing a compressed (gzip) tar file but the benefits of a compressed tar vs compressed files are small for our use case
Doing S3 put requests for 260M files every week would cost around $1300 USD/week which was too much for our budget
> or in ZIP format?
We looked at zip's but due to the way the header (well central file directory) was laid out it mean that finding a specific file inside the zip would require the system to download most of the CFD.
The zip CFD is basically a list of header entries where they vary in size of 30 bytes + file_name length, to find a specific file you have to iterate the CFD until you find the file you want.
assuming you have a smallish archive (~1 million files) the CFD for the zip would be somewhere in the order of 50MB+ (depending on filename length)
Using a hash index you know exactly where in the index you need to look for the header entry, so you can use a range request to load the header entry
offset = hash(file_name) % slot_count
Another file format which is gaining popularity recently is PMTiles[1] which uses tree index, however it is specifically for tiled geospatial data.
When it is serverside, reading a 50MB CFD is a small task. And once it is read we can store the zipindex for even faster access.
We made 'zipindex' to purposely be a sparse, compact, but still reasonably fast representation of the CFD - just enough to be able to serve the file. Typically it is around a 8:1 reduction on the CFD, but it of course depends a lot on your file names as you say (the index is zstandard compressed).
Access time from fully compressed data to a random file entry is around 100ms with 1M files. Obviously if you keep the index in memory, it is much less. This time is pretty much linear which is why we recommend aiming for 10K file per archive, which makes the impact pretty minimal.
GP says not to run anything you haven't looked at yourself for the most part, paraphrasing. I doubt anyone has self audited all the software and drivers going into a desktop Linux distro.
> GP says not to run anything you haven't looked at yourself for the most part, paraphrasing.
That is a highly inaccurate paraphrase of what I said. You can tell it's inaccurate because I'm specifically suggesting that end users should be able to trust apps on the App Store without auditing the apps ourselves, as long as we trust the app authors, and that app authors should bear responsibility for what they redistribute.
I haven't used hg enough to have an opinion on it, despite several attempts... Problem is I learn bottom up, and I just haven't been able to "think in mercurial" the way I can "think in git".
I find it interesting that what git calls a commit is actually a revision (or checkpoint, snapshot, point-in-time) and what mercurial calls a revision is actually a commit (or patch, delta, changeset).
I think a lot of people think in terms of patches/changesets and I suspect (still haven't gotten far enough to confirm) hg is a toolbox for managing them in a similar way to how git is a toolbox for manipulating its snapshot based DAG.
If you're using rebase there's the `--committer-date-is-author-date` and `--ignore-date` flags. One uses the author date for both and the other uses the commit date for both.
Without using either flag rebase should update the commit date and preserve the author date.
If by rebase you meant GitHub's rebase merge option I think you're out of luck :-/
Are those really the only options? I'm trying to wrap my head around how using a fixed size thread pool for I/O automatically implies deadlocks but I just can't. Unless the threads block on completion until their results are consumed instead of just notifying and then taking the next task..
I can definitely imagine blocking happening while waiting for a worker to be available, though. Did you mean simply blocking instead of deadlock?
Thank you for humoring me. I had to sleep on it, but I can see it now. Seems like it would require a really bad design or more likely bad actors (remotes leaving dead sockets open), but it would definitely be possible.
The same scenarios would lead to resource exhaustion if the thread pool wasn't bounded.
There are definitely ways to do it without those problems, though. They just wouldn't be quite as simple as the approach done for supporting zip.