Getting more out of GZIP for web content

GZIP and Deflate, for the time being, are the only supported compression formats for serving web content. As I've shown before, it's less than ideal as a format, and has quite a bit of bloat. Thankfully, there's ways out there to serve smaller, more compact versions of GZIP files.

GZip on the web

Can be served by most web servers, although little-known-fact is that the default level is 6, which can have ramifications with your data;unless you change it to 9, you could be sending bloated files, although to be fair, my test across 100 files < 100k yielded no difference between -6 and -9

Using 7Zip to make smaller GZIP files

7zip is a pretty nice utility to generate not only LZMA compressed content, but will also generate PPM, BZIP and GZIP files as well, and often times, will produce a smaller compressed file than the original compressor codecs themselves.

If you check out the 7zip switches list, you can see that the command line arguments for generating a .gz file, at max compression:

7z.exe a -tgzip -mx9 ./outFile.gz ./infile.txt

Using Zopfli to make smaller GZIP files

Projects like Zopfli do a great job at producing smaller gzip files, yet do so at a coding level, rather than any modifications to the modeling of the data.  On average, it’s reported that Zopfli gies smaller deflate-compatible output sizes than GZIP by about 3.7-8.3%

zopfli.exe --i50 -gzip ./infile.txt

Results

Just to give a sense of things, I tested Zopfli against 7z, and got some good results:

You can see that for file sizes >5k Zopfli & 7Zip produce smaller .GZ files easily. As the file size increases you see larger savings. In general, Zopfli @ lvl 50 beats 7zip @mx9 on my small test set (with a  few edge cases)

Serving a pre-compressed file

Our end goal is this effort would be to compress all your files offline (gzip, 7zip or zopfli as you choose), upload them to a webserver, and then your page can load and receive them like normal, without any changes to your client-side code. A little bit of Google-Foo produced proof that this ability exists, specifically from Apache and Amazon Web Services. Sadly, Google App Engine removed this functionality directly (due to security concerns) so you may need to work around that part of this implementation.

If you’re a web-app developer, and have the ability to “push a build” there’s no reason to be sending extra bytes down the wire when you can easily save more data.

~Main

You can find Colt McAnlis here: