tag:blogger.com,1999:blog-11971373139918550342024-11-05T18:52:00.744-08:00The Workbench"If at first, the idea is not absurd, then there is no hope for it"Unknownnoreply@blogger.comBlogger94125tag:blogger.com,1999:blog-1197137313991855034.post-69429584652596190532017-09-30T07:08:00.003-07:002017-09-30T07:12:05.926-07:00How PNG Works<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 26pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">How PNG Works</span></div> <b id="docs-internal-guid-32218a48-d31c-2f17-1042-274f80940d35" style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <a href="https://en.wikipedia.org/wiki/Portable_Network_Graphics" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Portable Network Graphics</span></a><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (PNG) format is a lossless image format that uses a GZIP style compressor to make its data smaller. Since the image format is lossless, the compressed image quality will be identical to your source image. This is great since you can get high quality while still getting </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">some</span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> compression, but not nearly as much as with lossy formats.</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One of the biggest attractions of PNG is that it supports </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">transparency</span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. In addition to the red, green, and blue channels, it supports an alpha channel that defines which pixels to </span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">alpha-blend</span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> during rendering. This transparency support makes PNG quite attractive to the web, and to applications where you may want an image on screen that isn’t exactly rectangular. However, you are paying to have a fourth color channel though, which will increase the size of your uncompressed file.</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This means that the format itself has been extremely popular over the last 20 years or so, but as I’ve mentioned before, it’s still not the best compressor on the planet.</span></div> <h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 20pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Understanding the Compression</span></h1> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNG’s compression process is entirely lossless; meaning that the compressed file can reconstruct the source image exactly. &nbsp;Done in two stages: Prediction (aka filtering) and then Compression.</span></div> <b style="font-weight: normal;"><br /></b> <br /> <h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 16pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">#1 Filtering (prediction)</span></h2> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Delta encoding is one of the most powerful numeric compression methods out there. The basic idea is that you can represent any value as a difference from the previous value. So, </span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">[2,3,4,5,6,7,8] can become [2,1,1,1,1,1,1], where [2, 3-2=1, 4-3=1, 5-4=1, 6-5=1, 7-6=1, 8-7=1]</span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The reason this is so powerful, is that if the data is linearly correlated (that is, value has some low-difference from the previous value) then you end up transforming values of your dataset into lots of duplicate, low values, which are more compressible.</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The PNG format makes use of delta encoding in a format that it calls “filtering”. Basically, for each scan-line of pixels, a current pixel is encoded in some relation to the pixel to the left, the pixel above, and the pixel above-left.</span></div> <div class="separator" style="clear: both; text-align: center;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-weight: 400; margin-left: 1em; margin-right: 1em; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="125" src="https://lh5.googleusercontent.com/ogSB0jp0v-Fa9eaOHFvWo2acRLTlAMVXPXVOCqsv3b10VwYhPod_hFTSaCsYenzT8APsVtPFXk8h0ZBpjCPHzd1RORbgliQuKsLDRxAKlBx1OVsC_oUAQ_uAYSSO7jSRpowNsx-S" style="border: none; transform: rotate(0rad);" width="128" /></span></div> <div class="separator" style="clear: both; text-align: center;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-weight: 400; margin-left: 1em; margin-right: 1em; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> </div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For example if we encoded a given pixel by listing its difference between the average of A and B then we’d get:</span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://cdn-images-1.medium.com/max/1600/1*9DREStZ3sX45FZFt_SwDfg.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="57" data-original-width="417" height="43" src="https://cdn-images-1.medium.com/max/1600/1*9DREStZ3sX45FZFt_SwDfg.png" width="320" /></a></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <br /></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The filtering takes ABC, and uses that to </span><span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">predict</span><span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> what the value of x is. The value that we replace x with, is the difference between the prediction, and the actual value.</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now, it’s worth noting that each line may be a little different, which is why PNG allows 1 of 5 different modes to be chosen, per line:</span></div> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: white; color: #252525; font-family: Arial; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">No filtering</span></div> </li> <li dir="ltr" style="background-color: white; color: #252525; font-family: Arial; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Difference between X and A</span></div> </li> <li dir="ltr" style="background-color: white; color: #252525; font-family: Arial; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Difference between X and B</span></div> </li> <li dir="ltr" style="background-color: white; color: #252525; font-family: Arial; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Difference between X and (A + B)/2 (aka average)</span></div> </li> <li dir="ltr" style="background-color: white; color: #252525; font-family: Arial; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <a href="https://www.w3.org/TR/PNG-Filters.html" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Paeth </span></a><span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">predictor (linear function of A,B,C)</span></div> </li> </ul> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The intention here, is that each line can choose the best filtering method, for itself, such that it can produce the lowest number of unique symbols. Here’s our original example with each mode:</span></div> <b style="font-weight: normal;"><br /></b> <div class="separator" style="clear: both; text-align: center;"> <a href="https://cdn-images-1.medium.com/max/1600/1*x-yXQnUD4Vjjq10ALPyirQ.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="255" data-original-width="800" height="101" src="https://cdn-images-1.medium.com/max/1600/1*x-yXQnUD4Vjjq10ALPyirQ.png" width="320" /></a></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <br /></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And, it's important to note that these filters are done, </span><span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">per channel</span><span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> not per-pixel. Meaning the filter you use for the scan-line in a red channel, could be different that the same scanline in the blue channel. (which make sense, they will be different values, after all)</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And the PNG format has some nifty ways to pick which filter to use on a channel; While brute force is the most direct, it’s simply right out</span><span style="background-color: white; color: #252525; font-family: &quot;arial&quot;; font-size: 10.5pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Instead, the developers experimented on different image types and came up with some rules of thumb, that are close to optimal; like use None filters for palette images, and sub-8 bit grayscale images. And for the other images, choose the filter that minimizes the sum of absolute differences; Rather than using modulo-256, instead to standard signed math, then take the abs value, and add them all together for a given row, and compare the sums for the other filter types. Choose the filter that gives the smallest sum.</span></div> <h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 16pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">#2 Compression (DEFLATE)</span></h2> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Once filtering has occurred on a scan-line, it’s passed off to a descendant of the LZ77 algorithm, known as DEFLATE; This algorithm combines LZ77 coding alongside a Huffman coder. Which is almost identical to compressors like PKWARE, PKZIP, GZip etc. The implementation is out-of-the-box standard, but has some interesting caveats when it comes to image data.</span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Deflate limits match lengths between 3 and 258 symbols; which puts the maximum conceivable compression ratio around 1032:1.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If the match is less than 3 symbols, then you incur some overhead to represent the symbol</span></div> </li> </ol> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The result of these two, means that the size of your image can have an impact in if matches are found in a scanline.</span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Consider this image below, The 270x90 version is only 20k, however the 270x92 version is 2x larger.</span></div> <div class="separator" style="clear: both; text-align: center;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-weight: 400; margin-left: 1em; margin-right: 1em; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="310" src="https://lh4.googleusercontent.com/RqjPW0Jq-rC-GpVFrIS35JRI0wNKPbT9nSXlBgwN7mAOHGGCxHkLN7KPFYfhLzlGRfq7uTn3km-dq_19Q3e0CSguGnJSCmR9eNUE9nuNI2yvz6ruUYXltma_x3UAy0StT8DHa6dJ" style="border: none; transform: rotate(0rad);" width="219" /></span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> </div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Logically this seems wrong. Adding 540 pixels to an image shouldn’t cause a 2x bloat in compression. However when we look a little closer, we can see why this is occurring; The following heat map of the images represents how compressed a given pixel is. Deep blue = very compressed, yellow/red = not very compressed</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div class="separator" style="clear: both; text-align: center;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-weight: 400; margin-left: 1em; margin-right: 1em; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><img height="311" src="https://lh5.googleusercontent.com/LZUwYlTblXw3C2V2sz0_3CzSz449XvqPscDS6WqfGe_3XBJP-T1fY4jZZ1GCN52Vjj3gRjKxpqaeopmUkGrm4NPnevlX7XYCeiPKKhc9wGwqUjTuThUq-r4X4Ho1U_0EZznPgcZ1" style="border: none; transform: rotate(0rad);" width="219" /></span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> </div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What’s occurring here, is that in the smaller image, there’s more matches in the scanlines, and thus there’s better compression. However, adjusting the size, slightly, changes the type of matching that can occur.</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If you’re looking to see how well your own images compressed with PNG, check out </span><a href="http://encode.ru/threads/1725-pngthermal-pseudo-thermal-view-of-PNG-compression-efficiency" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PNGThermal</span></a><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <b style="font-weight: normal;"><br /></b> <h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 20pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Understanding the format</span></h1> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now, it’s worth noting that PNG is more than just the filtering &amp; compression stages. It’s a pretty extensible container format, that can support all sorts of image types and added data. </span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s first start with the fact that the PNG file format can support a series of chunks inside of it which can include various types of data. For example, you’ve got the </span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNG Header chunk which contains simple information about the image like width, height, bit depth &amp; colortype.</span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Image data chunk contains the actual image information itself (oh, and note, this information can exist in multiple chunks). Then for color paletted images, there’s a separate chunk for that.</span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And finally an end-of-file chunk. And those are the main chunks, but there’s also a whole slew of </span><a href="https://en.wikipedia.org/wiki/Portable_Network_Graphics#.22Chunks.22_within_the_file" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">other chunks</span></a><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> like:</span></div> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Default background color</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Chroma coordinates on how to display white-points</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Gamma spec</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Histogram info</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Text data, with language or Metadata info</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Color space information</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Stereo-image data</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A chunk which notes the last time the file was changed</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And Transparency data.</span></div> </li> </ul> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now, these chunks are where you need to watch out, because lots of junk gets put in there by your photo editing app. For example, saving a PNG file from photoshop results in a chunk that says “this image was made in photoshop” REMOVING chunks is critical to ensuring small file sizes. The image below shows you a 16x16 pixel image, saved from photoshop as a regular PNG, and the other-one, using the “export to web” option of photoshop, which removes all the gobbledygook.</span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://cdn-images-1.medium.com/max/1600/1*Pu_D0jQ8ef52C8IKFMcWYw.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="405" data-original-width="624" height="207" src="https://cdn-images-1.medium.com/max/1600/1*Pu_D0jQ8ef52C8IKFMcWYw.png" width="320" /></a></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <br /></div> <b style="font-weight: normal;"><br /></b> <br /> <h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 16pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Pixel format</span></h2> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The PNG format can ALSO support various types of pixel formats so you can create an optimal</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Indexed - 1 channel, can be 1,2,4,8 bpc</span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Grayscale - 1 channel, can be 1,2,4,8,16 bpc</span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Gray+Alpha = 2 channel, can be 8 or 16 bpc</span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Truecolor(RGB) = 3 channel, can be 8 or 16 bpc</span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RGBA = 4 channel, can be 8 or 16 bpc</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Pixel formats are selected by the author of the image file. So it’s not too interesting to discuss, other than making sure you’ve chosen the right pixel format. Storing a greyscale image as a truecolor w/ alpha, is a bad idea.</span></div> <b style="font-weight: normal;"><br /></b> <h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 20pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Story : GIF and the birth of </span><a href="http://www.libpng.org/pub/png/book/LCH-png-chapter.pdf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: &quot;arial&quot;; font-size: 20pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PNG</span></a></h1> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">When it comes to the relentless bulk of web content, images are by far the largest load bearer today (although there's an argument to be made that videos have become king as of late).</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But what's truly interesting is that as much as compression information can help solve some of the congestion, there's a massive amount of human problems that keep compression from making its way into everyone's hands. </span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s take a trip back to 1985, when Unisys filed a patent for the LZW compression algorithm.</span></div> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A few years later, when CompuServe invented the 89a format (which later became the GIF format), they used LZW as its backbone, not realizing that it was a patented. Unisys didn’t care about this until 1993, when the Netscape browser added support for the IMG HTML tag, alongside with support for the 89a format. Within a year, animated images became all the craze on the internet, and Unisys started enforcing its patent. CompuServe and Unisys eventually reached a court agreement in December of 1994, announcing that Unisys would start collecting royalties from all software that used the 89a graphics format. In the months following this decision, a group of engineers developed an entirely new, patent-free format known as the Portable Network Graphics or PNG format. </span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In 2004, the patent on LZW finally expired, but for a whole decade, the GIF/PNG image format debate was a hot one among internet folks.</span></div> <b style="font-weight: normal;"><br /></b> <br /> <h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 20pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In closing</span></h1> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNG is a very flexible and powerful image format type that’s become popular on the web due to it’s ability to support transparency. But it’s not right for _every_ situation; Basically, make sure you’re using the best tool for the job. If your image is photographic and absolute losslessness is not required, then JPG or an equivalent lossy method would be more efficient than any lossless approach. &nbsp;On the other hand If you need a smaller bit-format, or need transparency, than PNG would be a dominant win over JPG.</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And if you’re interested in seeing the most bare-bones implementation of a PNG coder, check out this </span><a href="https://gist.github.com/mmalex/908299" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">awesome gist</span></a><span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which packs the whole thing into ~40 lines of code.</span></div> <br /> <div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: &quot;arial&quot;; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Of course, then there’s WebP, which will give you both.. But that’s another video ;)</span></div> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-1197137313991855034.post-9420268060799080982014-06-05T07:08:00.001-07:002014-06-05T07:08:39.953-07:00CRABBY Vs. ICEBREAKER<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I grabbed a copy of </span><a href="https://play.google.com/store/apps/details?id=com.nitrome.icebreaker" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Icebreaker</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> the other day. Great game really. But when I opened up the APK to take a look at some of their artwork in higher detail, I found the game’s asset sizes were larger than they should be.</span></div> <b id="docs-internal-guid-217647c6-6c5b-08d8-5091-b9ee568ba47b" style="font-weight: normal;"><br /></b> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoDqZ83MVHbAEvPA3NLMsiedDXVFiNlbE4DZPBNRg1zPer5kCSs0yvU-ZFSMFqQFQxsy4hV8hKtD1tapH42pn22rAVdq1FOKhK-mRJcgyWyvf-UZwGSxflX2mqAlEdP5R9GNZLhijabFo/s1600/crb_vs_icbr.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoDqZ83MVHbAEvPA3NLMsiedDXVFiNlbE4DZPBNRg1zPer5kCSs0yvU-ZFSMFqQFQxsy4hV8hKtD1tapH42pn22rAVdq1FOKhK-mRJcgyWyvf-UZwGSxflX2mqAlEdP5R9GNZLhijabFo/s1600/crb_vs_icbr.jpg" height="282" width="400" /></a></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <br /></div> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I’ve always been a fan of the artistic style and design quality of </span><a href="http://www.nitrome.com/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Nitrome</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> games. Like most of their content, there’s lots of hand-optimized transparent sprites that are used. But I was appalled when I saw that that 20% of their asset footprint came from 32bpp PNG files. (sound took another 11% was taken by sound files, but that’s a separate article)</span></div> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As you may have known, I’ve been on a </span><a href="https://www.youtube.com/watch?v=jHXzzHElFPk" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">tirade lately</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> about the problems of </span><a href="http://mainroach.blogspot.com/2014/03/png-bloat-in-android-games.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">texture compression</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in </span><a href="http://mainroach.blogspot.com/2014/03/report-what-content-makes-up-android.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Android apps</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Seeing such a great app with obvious texture bloat problems was troubling to me.</span></div> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In March 2013, I released </span><a href="http://mainroach.github.io/crabby/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Crabby</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, an extended texture compression format meant to replace the use of simple PNG for sprite-based texture games. So I decided to run some of the assets from </span><a href="https://play.google.com/store/apps/details?id=com.nitrome.icebreaker" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Icebreaker</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> through </span><a href="http://mainroach.github.io/crabby/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Crabby</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to see what wins could be had. Here’s a small sampling of the results:</span></div> <div dir="ltr" style="margin-left: 0pt;"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="195"></col><col width="104"></col><col width="103"></col><col width="112"></col><col width="109"></col></colgroup><tbody> <tr style="height: 0px;"><td style="background-color: #cfe2f3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><br /></td><td style="background-color: #d9d2e9; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">srcfile.png</span></div> </td><td style="background-color: #d9d2e9; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">crabby.zip</span></div> </td><td style="background-color: #ead1dc; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">srcfile.gpu</span></div> </td><td style="background-color: #ead1dc; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">crabby.gpu</span></div> </td></tr> <tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">mapscreen-2cavern.png</span></div> </td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">163 k</span></div> </td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">114 k</span></div> </td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4 mb</span></div> </td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">877 k</span></div> </td></tr> <tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">active.png</span></div> </td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">121 k</span></div> </td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">221 k</span></div> </td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4 mb</span></div> </td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1.38 k</span></div> </td></tr> <tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">android_levelcomplete.png</span></div> </td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">80 k</span></div> </td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">58 k</span></div> </td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4 mb</span></div> </td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">519 k</span></div> </td></tr> <tr style="height: 0px;"><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bg_0_layer2.png</span></div> </td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">28.3 k</span></div> </td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">46 k</span></div> </td><td style="border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1.6 mb</span></div> </td><td style="background-color: #d9ead3; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">394 k</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The first 3 images in this list are sprite-sheets (the final is just a big, unique background image). I’ve noted before that passing a </span><a href="https://github.com/mainroach/crabby/issues/10" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">simple sprite-sheet to crabby does not result in savings</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This is because that most chart atlases have been rotated or translated to be offset in the atlas for packing. The result is that the 4x4 block matching mode in </span><a href="http://mainroach.github.io/crabby/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Crabby</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> doesn’t end up finding as many duplicate 4x4 blocks. As such, I wrote a quick tool to break apart the atlases, which is loosely based on a flood-fill algorithm. The end result is loose, boundary-optimized, PNG files, which help remove the 4x4 block alignment issue (I haven’t found a way to fix the rotation issue yet though, so there is some bloat in my numbers..)</span></div> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As far as distribution is concerned, you can see that simplistic</span><a href="http://mainroach.blogspot.com/2013/09/png-compression-5-simple-improvements.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"> PNG files</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> will often give wins when it comes the APK size by itself. This is roughly to be expected; We haven’t done significant lossless compression of the on-disk format, so it makes sense that the optimized Deflate algorithm would win on that part.</span></div> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">However, it’s clear that once you’re loading these textures into the GPU, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">it’s no contest</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. The raw sprite atlases are </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4MB </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">a piece, while the crabby footprint never gets above </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">900k</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This is night-and-day difference. anywhere between </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">78% savings and 99% savings on the GPU</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. For lower-powered devices, or just looking at how much content you can cram into a level, this is a large savings.</span></div> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Although </span><a href="http://mainroach.github.io/crabby/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Crabby</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is far from being a completed project, tests like these help solidify the need for such a library out there for mobile game devs. It’s hard enough to get someone to find your game, and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">decide</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to install it. They shouldn’t be turned off by the size of the package, or restrictions on GPU sizes.</span></div> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="margin-left: 0pt;"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="background-color: #f4cccc; border-bottom: solid #000000 1px; border-left: solid #000000 1px; border-right: solid #000000 1px; border-top: solid #000000 1px; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Disclaimer</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As with any comparisons, there’s some wiggle room in the numbers. </span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Firstly</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, my atlas-breaker algorithm will do it’s best to cut cut up the image into discrete chunks, but there are cases where it isolates sections of a sprite (for example, a small explosion frame) that may not be connected using a flood fill, and will extract them out into their own images. </span></div> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Secondly</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, the </span><a href="http://mainroach.github.io/crabby/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Crabby</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> mode I ran this through did not require Power-of-Two textures. You can check out the decoding examples </span><a href="https://github.com/mainroach/crabby/blob/master/decode/android/jni/main_demo.cpp#L266" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">I’ve provided to see that this mode is often available on modern mobile hardware</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. So some of my numbers will naturally be lower due to the removal of this requirement, since the source assets are all POW2 aligned.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Thirdly</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">,</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><a href="http://icebreaker/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Icebreaker</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is entirely owned by </span><a href="http://nitrome/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Nitrome</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. I have no right to the images, trademarks, distribution, or identification of any content involved with it. Also, this analysis is entirely a personal process, it’s not nothing to do with any employer, organization, or secret society that I may (or may not) be involved with.</span></div> <br /><br /><br /> <br /> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com7tag:blogger.com,1999:blog-1197137313991855034.post-91988578765171305092014-05-21T07:52:00.005-07:002014-05-21T07:55:16.343-07:00Understanding Mobile CPU throttling<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I’ve been seeing some weird CPU performance weirdness while working on a new Android project, and after digging around a bit, realized that the perf issue wasn’t in my code at all; the CPU was being throttled by the hardware, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">automatically</span></div> <div class="separator" style="clear: both; text-align: center;"> </div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9uEA5twLSjvWlPZJLm3qVLfPVvUwryeCbx4xMU9si4myIM5dbb_PffmDLRSOTHRB3VWZmNXlbQxcKgFo_LjwbWacspUZxEi5VSxTlWYfeRrI_bNcg8vJHLmU1MF4cnltJErGHzFarhD0/s1600/fig2-fast-scaling.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9uEA5twLSjvWlPZJLm3qVLfPVvUwryeCbx4xMU9si4myIM5dbb_PffmDLRSOTHRB3VWZmNXlbQxcKgFo_LjwbWacspUZxEi5VSxTlWYfeRrI_bNcg8vJHLmU1MF4cnltJErGHzFarhD0/s1600/fig2-fast-scaling.jpg" height="218" width="400" /></a></div> <b id="docs-internal-guid-6a05ebf4-1f44-6d3f-9c17-8179e5c95eb5" style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You see, pretty much all modern mobile CPUs have a process known as frequency scaling. &nbsp;That is, in order to conserve battery, most mobile hardware will reduce the amount of voltage supplied to the CPU depending on the workload that’s provided to it. &nbsp;If there’s not lots of work being done, then the CPU pulls less voltage, draining your battery less, emitting less thermal heat, and also slowing down the performance of your chip. </span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This process is basically unavoidable—most smartphones and tablets are built to run very quickly for a short amount of time. Under stress, however, they need to tone themselves down to prevent melting whatever enclosure they happen to be in. From the OEM standpoint, there are good reasons to throttle CPU cycle count, but it's a delicate balance. Throttle too much, and users won't see the benefits of that fast hardware they paid for. Don't throttle aggressively enough, and heat and battery life will both suffer. Typically, the CPU will start out at its maximum rated speed and fall off as the phone gets warmer.</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This process is generally controlled by the OEM driver, whom updates frequency based on CPU load. A process which is problematic for CPU bound games, as the simulation will often see unknown oscillations in your CPU frame time. Even worse if you’re using a the deltaTime since the last frame in some computational capacity (after getting 6 frames of 16.6ms, you’ll suddenly drop to 48ms / frame for no reason, which can cause havoc on physics simulations)</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sadly for developers, there’s no direct work around. The hardware is going to do what’s best to keep it from melting. If you need to optimize for this, you need to:</span></div> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Not let the CPU get too idle (when it’s idle, the clock cycle goes down)</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Not let the CPU get too busy (when it starts to get hot, the clock cycle also goes down)</span></div> </li> </ul> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It’s not rocket science, but finding the sweet spot in your simulation will ensure that you’re getting full potential out of your CPU on mobile devices.</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Thankfully, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">deltaTime</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> derivations of the CPU shouldn't affect your overall frame rate since the GPU will still be running @ 59.94Hz.</span></div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Which, I suppose is something, right?</span></div> <br /> <br /> <br /> <br /> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-1197137313991855034.post-18853533499360508292014-04-14T06:46:00.001-07:002014-04-14T06:46:49.303-07:004 Steps to find your performance bottleneck in Android Games.<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I’ve talked about a </span><a href="http://mainroach.blogspot.com/2013/04/faster-app-performance-with-5-simple.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">quantization of performance solutions</span></a><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> before, and even a </span><a href="http://mainroach.blogspot.com/2009/10/think-triangles-not-circles.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">more accurate way to look at algorithm complexity</span></a><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. But it’s time to get into specifics for Android Games.</span></div> <b id="docs-internal-guid-929fa229-607b-0b38-de3d-57ea0573a1cf" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We have a game. We want to know whether it is limited by CPU or GPU performance. There are three possibilities:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It could be CPU bound</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It could be GPU bound</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It could be some weird combination of the above two</span></div> </li> </ul> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If there is no direct way to measure this, (say for instance, your tool chain lacks a way to profile native code) but we can work it out by applying some detective skills.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpBaElNMekCRr3_AOoqYOp90eM6Bp4c1RbS86m34jPmj2X34UD08DMZ-FnMKaOi7nCv79JnvFyv_xAp5qNxHU4bhEBaPRE6LYE45a2Eogpc301BgwvD5WX3USRZ8FvsBdnlvEmnGjdQww/s1600/image00.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpBaElNMekCRr3_AOoqYOp90eM6Bp4c1RbS86m34jPmj2X34UD08DMZ-FnMKaOi7nCv79JnvFyv_xAp5qNxHU4bhEBaPRE6LYE45a2Eogpc301BgwvD5WX3USRZ8FvsBdnlvEmnGjdQww/s1600/image00.png" height="220" width="400" /></a></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1. Turn on data that gives you feedback about performance</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In order to find performance issues, we must get the game into a state where we will be able to notice even small changes in performance. Ideally, we wouldn’t want this state to ship with the game, so you should add a special </span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">profiling</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> configuration to your build that </span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">enables various tests, insights, and provides information about performance.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Then you’ll need some way to get feedback on performance, say, adding a </span><a href="http://en.wikipedia.org/wiki/Frame_rate" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">frame rate</span></a><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> display, using an an </span><a href="http://www.altdevblogaday.com/2012/08/21/using-chrometracing-to-view-your-inline-profiling-data/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">inline profiler</span></a><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, or going for broke and adding a </span><a href="http://www.ixbt.com/short/images/Perf_HUD5.jpg" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">performance HUD to the game</span></a><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Oh, and turning off vsync in this mode is also a good idea.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2. Walk through the optimization cycle</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To find your bottleneck, follow these 3 steps:</span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tweak &amp; watch.</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> For each subsection of your game, slightly change the workload that’s being done there. If performance varies, you've found a bottleneck.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Optimize.</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Given the bottlenecked stage, reduce its workload until performance stops improving or until you achieve your desired level of performance.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Repeat.</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Do steps 1 and 2 again until the desired performance level is reached.</span></div> </li> </ol> <b style="font-weight: normal;"><br /><span style="background-color: white; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Locating the bottleneck is half the battle in optimization, because it enables you to make intelligent decisions about focusing your actual optimization efforts. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3. Know the big-picture toggles</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Your CPU and GPU have multiple sub-stages that could be causing problems. Digging deep into each one may not be the best idea, but with a few high level workload toggles, you can quickly track down the things to interface with.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">CPU - Add a sleep function.</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Add a 1ms of work to your update loop. Does your framerate change? &nbsp;**Note, using sleep(1) would work, but might skew results, </span><a href="http://arstechnica.com/gadgets/2013/11/when-benchmarks-arent-enough-cpu-performance-in-the-nexus-5/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">since it might lower CPU frequency</span></a><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. On more recent mobile hardware, probably better to </span><a href="http://en.wikipedia.org/wiki/Spinlock" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">spin-lock</span></a><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for 1ms...</span></div> </li> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If frame rate </span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Does Not</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> change, then you are </span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GPU Bound</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. The reason the frame rate did not change was because your CPU spends time waiting on the GPU to finish it’s present function each frame. (so skip to </span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">step 4</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">). </span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If your framerate DID change, then that means the GPU is idle each frame, waiting for the CPU to finish work. Adding an extra millisecond to a frame pushes back the opengl present() return call. (so skip to </span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">step 2</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">)</span></div> </li> </ol> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">CPU - Skip an update function</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Bind a keystroke to simply skip the next update function. Your rendering will remain the same, but none of your state will be updated. &nbsp;</span></div> </li> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If your framerate changes, then you’re doing lots of work in your update function, and there could be optimizations to do there.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If your framerate doesn’t change, then you’re not pushing the limits on the CPU by itself, and changes are you might be CPU-&gt;GPU interface bound (so skip to </span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">step 3</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">)</span></div> </li> </ol> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">CPU - Skip driver calls.</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> One of my favorite performance documents, has to be this old page detailing </span><a href="http://msdn.microsoft.com/en-us/library/windows/desktop/bb172234(v=vs.85).aspx" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">How to properly profile DirectX calls on the CPU</span></a><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. In an eye-opening view, it basically gives a table of the average cycle count for any rendering API call. OpenGL is no different; CPU side calls to the OpenGLES API is considered a Driver call, and has some overhead on the CPU. Some calls are worse than others, for example, any resource locking/updating is a synchronous operation that demands access to a GPU resource, there is the potential to massively stall the GPU pipeline, which costs both CPU and GPU cycles. &nbsp;To find bottlenecks here, bind a keystroke to simply skip portions of your render phase; for example, only allow the first 2,000 api calls a frame. </span></div> </li> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Any </span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">significant</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> variance here is important, since you’re skipping GL calls, you’re also changing how much work the GPU has to perform, which can also fluctuate your framerate.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If making these changes produce no framerate modifications, then chances are you’re just not doing enough work on the CPU entirely, and as such, the GPU is getting backed up with tons of extra calls, creating a weird GPU bound paradox due to lack of CPU work.</span></div> </li> </ol> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GPU - Halve the size of your back buffer</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. If you see performance significantly change, then you are directly fragment shading bound. </span></div> </li> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If your framerate changes, fragment shaders are doing too much work, or there’s too much overdraw, etc etc. To help identify the problem, go to </span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">step 5.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If the frame rate doesn’t change, there’s something deeper going on; You’re going to need a better tool chain.</span></div> </li> </ol> <li dir="ltr" style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GPU - Reduce your Fragment Shader workload.</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Try adding a NULL shader (just draw white fragments) to your scene in place of various shaders to attempt to track down what the offending shader / instruction is.</span></div> </li> </ol> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Digging deep with better tools</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">When it doubt, nothing beats a good old-fashioned profiling tool. Once you’ve tracked down some of the sub-areas that your performance can fall, you need to start digging deeper with more advanced developer tools. For now, each chip developer has their own suite of performance tooling that allows you to track down issues on their hardware. I suggest taking a look at each set for your given game.</span></div> <br /> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="http://www.imgtec.com/powervr/insider/powervr-insider.asp" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PVR Toolchain</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="https://developer.qualcomm.com/download" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Qualcomm chips</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="https://developer.nvidia.com/tegra-android-development-pack" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">NVIDIA chips</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><a href="http://software.intel.com/en-us/vcsource/tools/intel-gpa" style="text-decoration: none;">Intel Chips</a></span></div> </li> </ul> <div> <span style="font-family: Arial;"><span style="font-size: 15px; line-height: 17.25px;"><br /></span></span></div> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-1197137313991855034.post-12556914907063087482014-04-09T06:00:00.000-07:002014-04-09T06:45:59.841-07:00Don't Alpha That Pixel!!<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white;"><a href="http://mainroach.blogspot.com/2014/03/report-what-content-makes-up-android.html" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">In my travels</span></a><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, I’ve started seeing more </span><a href="http://www.altdevblogaday.com/2012/09/17/building-an-html5-game-dont-shrug-off-atlases/" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">texture atlases</span></a><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> that waste hordes of texture space. The dominant waste comes from alpha textures being interspersed with opaque textures, effectively keeping around an extra channel of pixels for an image that isn’t needed.. Most games seem to dump all the textures into a single atlas in order to reduce the number texture swaps on the CPU. Sadly, they are wasting space by pushing all those blank pixels into memory. Because both opaque and transparent sprites are adjacent to each other in a given atlas it’s difficult to know, (unless you add a separate file listing it), what sprites need alpha blending, and which ones don’t. The end result of this, is that without extra overhead required to know when to turn on and off alpha functions, there’s a large chance that the majority of sprites you’re rendering are being done with alpha turned on, even the opaque ones.</span></span><br /> <span style="background-color: white;"><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></span><span style="background-color: white;"><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But for the mobile devices, that might be a horrible idea.</span></span></div> <b style="background-color: white; font-weight: normal;"><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="color: #222222; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0xsh_UJCWo8AGPc3pgK8h1pT-dMo46hDtP8dY9d4c1lpJAisakvhoV5HNTP5S0caz7416Pa6-pT3gt_sMQ397eRmpjt6lHq9EHsPKf0b2qjwgnsI2wWrc3u723guhNrBj71_m7VKSFlA/s1600/image00.jpg" imageanchor="1" style="background-color: white; margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0xsh_UJCWo8AGPc3pgK8h1pT-dMo46hDtP8dY9d4c1lpJAisakvhoV5HNTP5S0caz7416Pa6-pT3gt_sMQ397eRmpjt6lHq9EHsPKf0b2qjwgnsI2wWrc3u723guhNrBj71_m7VKSFlA/s1600/image00.jpg" height="185" width="320" /></a></div> <span style="background-color: white;"><b style="font-weight: normal;"><br /></b> </span><br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: white;"><span style="color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s talk about Alpha</span><span style="color: #222222; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span></h1> <span style="background-color: white;"><b style="font-weight: normal;"><br /><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> </span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white;"><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Any sort of alpha (blending and testing) must be rendered </span><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">back-to-front</span><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in order to get the desired visual result (aka, you must render the object/pixel farthest from the camera first, and the closest one last). The reason for this, is that to properly shade an alpha blended pixel, it must read from the frame buffer, do the blend, and write a new value (aka a read-modify-write or </span><a href="http://en.wikipedia.org/wiki/Render_output_unit" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">ROP: Raster Operation</span></a><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">). As such, there must already be a depth-resident pixel color in the frame buffer before doing alpha.</span></span></div> <span style="background-color: white;"><b style="font-weight: normal;"><br /><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> </span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white;"><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">On the other side of that coin, </span><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">opaque</span><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> surfaces are generally rendered in </span><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">front-to-back</span><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (ie closest to the camera first) order. This process allows fragments closer to the camera to fill the </span><a href="http://en.wikipedia.org/wiki/Z-buffering" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">z-buffer first</span></a><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, such that subsequent, farther-from-the-camera pixels can be discarded at the post vertex transform stage before needing to be shaded (which is typically a heavy performance issue).</span></span></div> <span style="background-color: white;"><b style="font-weight: normal;"><br /><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> </span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white;"><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s be clear : In order to maximize your GPU performance, you need to minimize the number of non-visible pixels you’re wasting time shading. </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Proper removal of useless fragment processing was a bane to the early 2000’s console developers, and single handedly gave rise the art of </span><a href="http://en.wikipedia.org/wiki/Deferred_shading" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Deferred Lighting</span></a><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which quickly became one of the hottest rendering trends for FPS games in the early part of this century. </span></span></div> <b style="background-color: white; font-weight: normal;"><br /><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Mobile GPUs and Tile-Based-Rendering</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white;"><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the mobile ecosystem, there’s a majority of GPU chips out there that are known as </span><a href="http://en.wikipedia.org/wiki/Tiled_rendering" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Tile Based Rendering (TBR)</span></a><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. These GPUs work by dividing the render buffer into a regular grid in order to exploit spatial coherence, hardware parallelism, and other hardware operations that allow for improved performance.</span></span></div> <span style="background-color: white;"><b style="font-weight: normal;"><br /><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> </span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white;"><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One of the more important performance &nbsp;features is a per-pixel h</span><a href="http://en.wikipedia.org/wiki/Hidden_surface_determination" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">idden surface remova</span></a><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">l during fragment shading. For a given tile, Before running any fragment shaders, the polygons are pre processed to determine which fragments potentially contribute to the final result, and only those are shaded. This removes the need to sort opaque geometry on the CPU. </span></span></div> <span style="background-color: white;"><b style="font-weight: normal;"><br /><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> </span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white;"><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">On hardware with this feature </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(not all TBR chips have this special advantage...)</span><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, using alpha-textures (beyond 1 bit alpha) causes a horde of performance problems. Directly, any pixel using alpha eliminates much of the optimizations that come from the deferred rendering portion of the TBR hardware. </span></span></div> <span style="background-color: white;"><b style="font-weight: normal;"><br /><span style="color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> </span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white;"><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As such, the presence of the GLSL </span><a href="http://www.executionunit.com/en/blog/2010/04/12/implementing-glalphafunc-opengl-es20/" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">DISCARD</span></a><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> keyword, alpha testing, </span><a href="http://en.wikipedia.org/wiki/Alpha_to_coverage" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">alpha-to-coverage</span></a><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and </span><a href="http://en.wikipedia.org/wiki/Alpha_compositing" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">alpha blending</span></a><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> will all disable the hidden surface optimization as the occluded pixel may potentially impact the final image. The result is full processing of every pixel of every polygon in a tile, regardless if it contributes to the final image.</span></span></div> <span style="background-color: white;"><b style="font-weight: normal;"><br /><span style="color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> </span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Thus, these features should only be enabled for the objects that require them, even at the cost of an extra state change.</span></div> <b style="background-color: white; font-weight: normal;"><br /><span style="color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Don’t Alpha That Pixel!</span></div> <span style="background-color: white; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The truth is that alpha functions have a plethora of performance implications in order to achieve the visual looks they are known for. At the end of the day, you want to be very judicious about what pixels you alpha blend. From a memory standpoint, storing opaque textures in alpha formats is a wasteful bloat of gpu space. From a performance standpoint, z rejection is critical to reduce the number of fragments being shaded, and overdraw is a serious performance concern on all hardware. Not keeping track of what layers are opaque, and properly rendering them in the right order, means wasting lots of fill rate on raster-op constrained devices.</span><br /> <span style="background-color: white;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span> ~Main </span><br /> <center> <span style="background-color: white;"><small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a></span> </center> Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-1197137313991855034.post-78773052937474748632014-03-24T06:46:00.001-07:002014-03-24T06:53:49.740-07:00How to move from PNG to GPU Textures on Android (and keep your sanity)<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For most Android game developers, PNGs are a simple, easy to use texture format that offers ‘good enough’ compression alongside alpha support, which is important since the dominant form of game in the </span><a href="http://mainroach.blogspot.com/2014/03/report-what-content-makes-up-android.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">mobile market is 2D based</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. But the truth is that while PNGs may help with distribution, they don’t help you when it comes to GPU residency; they are still full, fat textures eating up your available memory. But, it doesn’t have to be that way.</span></div> <div class="separator" style="clear: both; text-align: center;"> <br /></div> <br /> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEis-8a2qMr2HhdXPP6zb8sWOPMvGV9gQsbUT2Ss9iB6l5ys-gcFFgssDK5Kuclf3t78HJde51txy6BWRWooFWJhUeRyhmH11qXbWD78NdhhixSq0UoK85yMMDSx7lq6hfmO4hMQcujj4VU/s1600/image01.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEis-8a2qMr2HhdXPP6zb8sWOPMvGV9gQsbUT2Ss9iB6l5ys-gcFFgssDK5Kuclf3t78HJde51txy6BWRWooFWJhUeRyhmH11qXbWD78NdhhixSq0UoK85yMMDSx7lq6hfmO4hMQcujj4VU/s1600/image01.jpg" height="352" width="640" /></a></div> <br /> <b id="docs-internal-guid-c2259b31-f453-cacd-8c8d-a21c4657cc76" style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GPU residency is a scarce resource for Android devices. Big, uncompressed textures take up a lot of space, and make it difficult to support the federation of unique devices.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PNG to GPU</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One of the worst parts about PNG files is that they are not friendly for GPU residency. Sure they help you with distribution, but they still take up uncompressed 32bpp or 24bpp space on the GPU, which is less than ideal.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Trust me, I understand the whole GPU texture fragmentation thing. PVRTC, DXT, ETC, ASTC; it gets crazy when you try to segment your APKs based on what textures you’re using. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But this doesn’t mean you have to settle for 32bpp by default</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. There’s an array of GPU compatible formats that support smaller bytes per pixel, and work on the large majority of GPUs. For example, take a look at the 16 bits per pixel formats below:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="margin-left: 0pt;"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="160"></col><col width="139"></col><col width="323"></col></colgroup><tbody> <tr style="height: 60px;"><td style="background-color: #2364d7; border: 1px solid #ffffff; padding: 9px 9px 9px 9px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: white; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">BITS PER PIXEL</span></div> </td><td style="background-color: #2364d7; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: top;"><div dir="ltr" style="line-height: 0.9; margin-bottom: 0pt; margin-top: 9pt;"> <span style="background-color: transparent; color: white; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">CHANNELS</span></div> </td><td style="background-color: #2364d7; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: top;"><div dir="ltr" style="line-height: 0.9; margin-bottom: 0pt; margin-top: 9pt;"> <span style="background-color: transparent; color: white; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GL ENUM</span></div> </td></tr> <tr style="height: 47px;"><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">32 bpp</span></div> </td><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RGBA - 8888</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GL_UNSIGNED_BYTE</span></div> </td></tr> <tr style="height: 47px;"><td style="background-color: #efefef; border: 1px solid #ffffff; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">24 bpp</span></div> </td><td style="background-color: #eaeaea; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RGB - 888</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #eaeaea; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GL_UNSIGNED_BYTE</span></div> </td></tr> <tr style="height: 47px;"><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16 bpp</span></div> </td><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RGBA - 4444</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GL_UNSIGNED_SHORT_4_4_4_4</span></div> </td></tr> <tr style="height: 47px;"><td style="background-color: #efefef; border: 1px solid #ffffff; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16 bpp</span></div> </td><td style="background-color: #eaeaea; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RGBA - 5551</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #eaeaea; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GL_UNSIGNED_SHORT_5_5_5_1</span></div> </td></tr> <tr style="height: 47px;"><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16 bpp</span><span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #a4c2f4; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RGB - 565</span></div> </td><td style="background-color: #a4c2f4; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 19px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GL_UNSIGNED_SHORT_5_6_5</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Making these formats is pretty easy from any of the plethora of </span><a href="http://mainroach.blogspot.com/2014/03/png-bloat-in-android-games.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PNG compressors</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">; most will support this output option, heck, even </span><a href="http://www.codeandweb.com/texturepacker/documentation#output" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">texture packer will do it for you</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And as a disclaimer, it’s important to note that 16 bit pixel format is not supported by the PNG format group </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">directly</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, nor the W3C, but this doesn’t mean that </span><a href="http://r0k.us/graphics/png16.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">it’s unsupported</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in the file format itself. Really, even a fast conversion from </span><a href="http://stackoverflow.com/questions/2008842/creating-and-loading-pngs-in-rgba4444-rgba5551-for-opengl" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">32bpp to 16bpp at load time</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> would be enough of a win.</span></div> <b style="font-weight: normal;"><br /></b><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjETbAUwqHvW1cS9NGx26if-dF9vWpKM8Jis4aqp7cKErTSt1g0WZNfUMLsfGoHgmfV4idyuovYK25iM0aFabZjovuQuFIqYvLUPd_InP0xFN8R5kShKPB0gsFpD6VehGCIK0wSlKZ_7gw/s1600/image00.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjETbAUwqHvW1cS9NGx26if-dF9vWpKM8Jis4aqp7cKErTSt1g0WZNfUMLsfGoHgmfV4idyuovYK25iM0aFabZjovuQuFIqYvLUPd_InP0xFN8R5kShKPB0gsFpD6VehGCIK0wSlKZ_7gw/s1600/image00.jpg" height="261" width="640" /></a><b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One of the side effects of using 16bit textures is the insertion of visual quality issues. In the image above, you can see the side-by-side difference between RGBA32, RGB565, RGBA5551 and RGBA4444. Easily, the 4bit per channel texture produces the most inserted quantization in smooth gradients, but it’s difficult to see the difference in the high-noise areas.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">GPU Texture support on Android </span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">THe truth is that if you want </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">real</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> savings for texture distribution, you need to embrace using one of the many GPU specific texture compression formats. These formats offer supiror compression sizes, alongside hardware sampling support (so speed is not an issue). Sadly, due to the state of the industry, there’s </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">LOTS</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of options when it comes to hardware support for compressed textures. For example, here’s a few of the more popular ones.</span></div> <div dir="ltr" style="margin-left: 0pt;"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="92"></col><col width="315"></col><col width="241"></col></colgroup><tbody> <tr style="height: 60px;"><td style="background-color: #2364d7; border: 1px solid #ffffff; padding: 9px 9px 9px 9px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: right;"> <span style="background-color: transparent; color: white; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">FORMAT</span></div> </td><td style="background-color: #2364d7; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: top;"><div dir="ltr" style="line-height: 0.9; margin-bottom: 0pt; margin-top: 9pt;"> <span style="background-color: transparent; color: white; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GL ENUM</span></div> </td><td style="background-color: #2364d7; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: top;"><div dir="ltr" style="line-height: 0.9; margin-bottom: 0pt; margin-top: 9pt;"> <span style="background-color: transparent; color: white; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">SUPPORT</span></div> </td></tr> <tr style="height: 47px;"><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-left: 9pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ETC1</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GL_OES_compressed_ETC1_RGB8_texture</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This format is supported by all Android phones. But no alpha support.</span><span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> <tr style="height: 47px;"><td style="background-color: #eaeaea; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-left: 9pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ATITC</span><span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #eaeaea; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GL_AMD_compressed_ATC_texture</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #eaeaea; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Used in devices with Adreno GPU from Qualcomm (Nexus One...).</span><span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> <tr style="height: 47px;"><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-left: 9pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PVRTC</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GL_IMG_texture_compression_pvrtc</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #b3cffe; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Supported by devices with PowerVR GPUs (Nexus S, Kindle fire...).</span><span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> <tr style="height: 47px;"><td style="background-color: #eaeaea; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-left: 9pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">DXT1</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #eaeaea; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GL_EXT_texture_compression_dxt1</span><span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #eaeaea; border: 1px solid #ffffff; padding: 5px 13px 5px 13px; vertical-align: middle;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #404040; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This texture compression is used in the NVIDIA chipset integrated devices (Motorola Xoom...)</span><span style="background-color: transparent; color: #404040; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.2139636386524548; margin-bottom: 22pt; margin-top: 0pt;"> <span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So here’ becomes a disconnect when trying to determine distribution, since some devices won’t have support for your target texture type. But it’s easy to determine what texture formats are supported once the app loads, just check for the above enumerations against the GL extension string on app load. In addition, when loading your texture data, you use </span><a href="https://www.khronos.org/opengles/sdk/docs/man/xhtml/glCompressedTexImage2D.xml" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">glCompressedTexImage2D</span></a><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, instead of using </span><a href="https://www.khronos.org/opengles/sdk/docs/man/xhtml/glTexImage2D.xml" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">glTexImage2D</span></a><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <div dir="ltr" style="line-height: 1.2139636386524548; margin-bottom: 22pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQcaGDX5f6LXmRNA72aiw35C5HDTVdfEA55ZI5CzYlMF_GaDDt9v0YcBi2M1HhN0B1haRmomTCi9cHUgLbnfLq_T5XYB_w3MCHD0DizqqD_Fxt3F9h1sawWBpMHI6F9JOOcEe3tuqwZDI/s1600/image03.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQcaGDX5f6LXmRNA72aiw35C5HDTVdfEA55ZI5CzYlMF_GaDDt9v0YcBi2M1HhN0B1haRmomTCi9cHUgLbnfLq_T5XYB_w3MCHD0DizqqD_Fxt3F9h1sawWBpMHI6F9JOOcEe3tuqwZDI/s1600/image03.jpg" height="259" width="640" /></a> </span></div> <div style="text-align: center;"> <span style="line-height: 1.2139636386524548;">You can generally see that the quality difference in GPU compressed formats is high.</span></div> <br /> <div dir="ltr" style="line-height: 1.2139636386524548; margin-bottom: 22pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="margin-left: 0pt;"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="background-color: #cfe2f3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: #404040; font-family: Arial; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note : ETC1 is supported by all Android devices.</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Using PNG to avoid APK management?</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Some devs have been claiming that they use PNG in order to reduce the burden of needing to send separate APKs to devices depending on what texture format type they support. To this, I say Hogwash, and here’s way:<br class="kix-line-break" /><br class="kix-line-break" />Consider the sizes of gpu texture for this 256x256 cropping of the parrot head. You can see that if you add up the sizes of the DXT, PVR, and ETC versions of the texture together, it’s still LESS SIZE than the PNG file, both on disk, and resident in GPU memory.</span><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhl-YhCDfJstH4NKmUk_kj9vTIQAQE2_4LidWWq_I44slffJ2PqSSB9v1JSYKXS4Rsu8WHzbDynoJZdj3WDegOfd04DZUvyx_InUnY620jAJcBu-6QJMF-H53Onwc4QtpMT2SPlMLDTt2Q/s1600/image02.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhl-YhCDfJstH4NKmUk_kj9vTIQAQE2_4LidWWq_I44slffJ2PqSSB9v1JSYKXS4Rsu8WHzbDynoJZdj3WDegOfd04DZUvyx_InUnY620jAJcBu-6QJMF-H53Onwc4QtpMT2SPlMLDTt2Q/s1600/image02.png" height="324" width="640" /></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The takeaway here is that if you add up the footprint for DXT, PVR and ETC, you end up with an overall texture footprint that’s </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">lower</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> than the size of the web-exported version of the PNG file. </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">AND </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">they have better GPU residency! Or, to say it in a much more “blog friendly” way :</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="margin-left: 0pt;"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="background-color: #cfe2f3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: #404040; font-family: Arial; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Instead of using PNG, export DXT, PVR and ETC, versions of your texture. Bundle all 3 in your APK, you’ll get a smaller distribution footprint AND smaller GPU residency.</span></div> <br /> <span style="background-color: transparent; color: #404040; font-family: Arial; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: #404040; font-family: Arial; font-size: 19px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Bare minimum, use ETC1 on all opaque textures.</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Conclusion</span></div> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sure, PNG is an easy format to use; However it’s compression savings ends at distribution, meaning that you lose all those savings when it comes to GPU residency. Using 16bpp PNG textures are a quick hack, and supported by OpenglES2 directly. And if you’re feeling daring, go all the way and just bundle all your GPU specific versions of the texture into your apk; You’re still going to be saving distribution size, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> GPU residency.</span></div> <div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-1197137313991855034.post-74929210367142866782014-03-17T06:37:00.000-07:002014-03-17T07:26:41.673-07:00PNG Bloat in Android Games<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As I mentioned in <a href="http://mainroach.blogspot.com/2014/03/report-what-content-makes-up-android.html" target="_blank">my previous post</a>, PNGs on android can be pretty gnarly, and a large portion of the asset footprint, for example, over 180 APKs, here was the breakdown between texture formats:</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div class="separator" style="clear: both; text-align: center;"> </div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIgRQoencaTN7tDWyawGfVTkrmxicWiTxoncnKYCrVItgC7cwPhC8rsL9bHes4x4Ymhfi356-O1C03REcWwR6OhPNs85EwiZ1n6M1NH8Lint7Vw7tXAK83PiKV54zXstZ6BzUN44WJGcM/s1600/image00.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIgRQoencaTN7tDWyawGfVTkrmxicWiTxoncnKYCrVItgC7cwPhC8rsL9bHes4x4Ymhfi356-O1C03REcWwR6OhPNs85EwiZ1n6M1NH8Lint7Vw7tXAK83PiKV54zXstZ6BzUN44WJGcM/s1600/image00.png" height="298" width="400" /></a></div> <br /> <div class="separator" style="clear: both; text-align: center;"> <br /></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <b id="docs-internal-guid-e5742f91-d040-2a12-b69c-98f9aa38434b" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">From the graph above, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">you can see that PNG files make up 78% of the total texture footprint</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="margin-left: 0pt;"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note: I’m not saying that PNGs take up 78% of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">total</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> asset footprint, but rather in the sub-section of. </span></div> </td></tr> </tbody></table> </div> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To some, it may seem that I’m on a holy campaign to convince developers to avoid using PNGs; That’s because I am. I believe the overusage of PNG files in HTML/Android games can be directly attributed to a lack of understanding about the proper way to address / use / design your texture resources.</span></div> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Does PNG size matter for android games?</span></h1> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span id="docs-internal-guid-c911c2b2-d040-6989-0862-9a2858837eb7" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> If you consider that there seems to be some </span><a href="http://mainroach.blogspot.com/2014/03/report-what-content-makes-up-android.html" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">self-selecting market factors in place</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> for the size of assets, then you can assume that the size of your APKs matter, and therefore, reducing the size of PNGs does, infact, matter.</span><br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="font-family: 'Trebuchet MS'; font-size: 21px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Isn’t PNG a really good compression format?</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="font-family: Arial; font-size: 15px; line-height: 17.25px; white-space: pre-wrap;">matter. I've covered how <a href="http://www.html5rocks.com/en/tutorials/speed/img-compression/">complex PNG files can be</a>, their <a href="http://youtu.be/whGwm0Lky2s?t=28m23s" target="_blank">potential to be problematic for developers</a>, and how <a href="http://mainroach.blogspot.com/2013/09/png-compression-5-simple-improvements.html">import it is to consider alternatives for web developers</a>. But to further drive this point home, let's take a look at a visual example. In the image below, I've exported a pure red 16x16 24bpp image (no alpha) from Photoshop directly (using SaveAs function) vs exporting it using SaveForWeb.</span></div> <div> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> </span></h1> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_FjGFW5BTUN8D74zj3Mwp1zNmU7GIrM2reyUFheq8AGxiWMnNdgjIYqSNc4Vaq_jdVg9pnF-OycQT2O7U32L2yACEwpPtQc-lZqD5Y1YHoT7_CwwAELSUhp6kRm94hEXROGcbhODa6_I/s1600/image02.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_FjGFW5BTUN8D74zj3Mwp1zNmU7GIrM2reyUFheq8AGxiWMnNdgjIYqSNc4Vaq_jdVg9pnF-OycQT2O7U32L2yACEwpPtQc-lZqD5Y1YHoT7_CwwAELSUhp6kRm94hEXROGcbhODa6_I/s1600/image02.png" /></a></div> <b style="font-weight: normal;"><br /></b><b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><span id="docs-internal-guid-c911c2b2-d040-a4a9-efb9-48c25aa25967"></span><b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The difference here is shocking. The Web-Exported version of the PNG is 96% smaller than the basic exported counterpart. This simplistic example illustrates the worst-case scenario for PNG exportation, a simple red square can easily be more bloated than you’d expect.</span></h1> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Android AAPT</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span id="docs-internal-guid-c911c2b2-d040-b8b6-4703-a306308312e1"></span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Android toolchain folks realize that PNGs are required for icons and other in-app formats, and as such, built in PNG compression processing into the toolchain. The </span><a href="http://developer.android.com/tools/building/index.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Android Asset Packaging Tool</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (AAPT) will perform repacking of your PNG texture assets in order to attempt to reduce the filesize of your APK files. The tool itself doesn’t do anything too tricky, </span><a href="http://developer.android.com/guide/topics/resources/drawable-resource.html#Bitmap" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">keeping it classy, and simple</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">:</span></div> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="margin-left: 0pt;"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #999999; font-family: Arial; font-size: 16px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Bitmap files may be automatically </span><span style="background-color: transparent; color: #222222; font-family: Arial; font-size: 16px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">optimized with lossless image compression</span><span style="background-color: transparent; color: #999999; font-family: Arial; font-size: 16px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> by the aapt tool during the build process. For example, a true-color PNG that </span><span style="background-color: transparent; color: #222222; font-family: Arial; font-size: 16px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">does not require more than 256 colors may be converted to an 8-bit PNG with a color palette</span><span style="background-color: transparent; color: #222222; font-family: Arial; font-size: 16px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span><span style="background-color: transparent; color: #999999; font-family: Arial; font-size: 16px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This will result in an image of equal quality but which requires less memory. So be aware that the image binaries placed in this directory can change during the build.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Which is great; Images that have a low number of unique colors are automatically translated to an indexed PNG format in a lossless form. (A lossy solution would introduce noise which may be unwanted). If however, the image has </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">more</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> than 256 colors, it’s left alone.</span></div> <span id="docs-internal-guid-c911c2b2-d040-e888-d546-5e116996e8d8"><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></span> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Thanks to my previous analysis, I can see how often this is this tool is successful in finding and converting textures. When you catalog all the PNG format types that come from our sample set, you can see that there’s quite a few that have this process run on them:</span></div> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="margin-left: 0pt;"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="208"></col><col width="208"></col><col width="208"></col></colgroup><tbody> <tr style="height: 39px;"><td style="background-color: #cfe2f3; border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">FMT</span></div> </td><td style="background-color: #cfe2f3; border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">BPP</span></div> </td><td style="background-color: #cfe2f3; border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">#Files</span></div> </td></tr> <tr style="height: 36px;"><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RGBA</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">32</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">29,081</span></div> </td></tr> <tr style="height: 36px;"><td style="background-color: #f9cb9c; border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Indexed</span></div> </td><td style="background-color: #f9cb9c; border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">8+</span></div> </td><td style="background-color: #f9cb9c; border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">18,805</span></div> </td></tr> <tr style="height: 32px;"><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RA</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,651</span></div> </td></tr> <tr style="height: 35px;"><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RGB</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">24</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1055</span></div> </td></tr> <tr style="height: 32px;"><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">R</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">8</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">235</span></div> </td></tr> <tr style="height: 29px;"><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><lots formats="" of="" other=""></lots><br /> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><other></other></span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><misc></misc><br /> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><other></other></span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Open Sans'; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">894</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RGBA, 32bpp is the dominant form of PNG files being used by game developers. Which make sense, considering the dominant form of apps in my analysis were 2D games.</span></div> <b id="docs-internal-guid-c911c2b2-d041-2131-4cfb-e0504343c681" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Palletized PNG images make up the 2nd most common form of PNG, mostly due to the fact that the AAPT tool will convert PNGs into that format directly. I’m happy to see such a bulk of the PNG files falling into this format, it means that AAPT is doing it’s job, and reducing memory footprint properly.</span></div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The stark contrast between RGBA32 and RGB24 textures makes me think that developers are easily wasting space in images through atlasing. For example, textures which may be completely opaque are being grouped with transparent textures, and </span><a href="http://maximumwage.files.wordpress.com/2014/02/wtf.gif" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">wasting pixel space as a result</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">More aggressive PNG optimization</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNG’s Deflate option is a lossless encoder, but that shouldn’t stop you from embracing a lossy preprocessor if you want one. Image processing tools, can compress your PNG image in a lossy method as a pre-process before passing it off to the final PNG format. This creates a two-step process where your lossy, and lossless compression are done by two separate applications. The results are impressive, the reduced color space allows the lossless compressor to find, and make, more frequent matches in the file, yielding to better compression. </span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <span id="docs-internal-guid-c911c2b2-d041-38b9-d692-96d68bbce94f"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></span> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A quick google search yields some easy-to-grasp apps that all do more aggressive PNG processing. Any of these apps would be fine, pick one that works well in your artists’ workflow, and adopt the hell out of it.</span></div> </div> <div dir="ltr" style="margin-left: 0pt;"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="208"></col><col width="208"></col><col width="208"></col></colgroup><tbody> <tr style="height: 33px;"><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNGQuant</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ImageMagick</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">AdvDef</span></div> </td></tr> <tr style="height: 33px;"><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNGOut</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNGCrush</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">OptiPNG</span></div> </td></tr> <tr style="height: 35px;"><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">CryoPNG</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PunyPNG</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Yahoo Smush.it</span></div> </td></tr> <tr style="height: 32px;"><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNG Optimizer</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNG rewrite</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ZopfliPNG</span></div> </td></tr> <tr style="height: 33px;"><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNGWolf</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">TruePNG</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">DeflOpt</span></div> </td></tr> <tr style="height: 36px;"><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Defluff</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Huffmix</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNGKT</span></div> </td></tr> <tr style="height: 35px;"><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNGnq-s9</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Median Cut Posterizer</span></div> </td><td style="border: 1px solid #000000; padding: 9px 9px 9px 9px; vertical-align: top;"><span style="font-family: Arial, Helvetica, sans-serif;">ScriptPNG</span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <br /> <span style="font-family: Arial, Helvetica, sans-serif;">One of the most powerful (and impressive) of the lot is <b><a href="http://css-ig.net/scriptpng">ScriptPNG</a>, a tool which will cycle through several of the above applications, and choose the best one for the specific image. </b>SO yea, start there!</span><br /> <br /> <div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"> <div style="line-height: 1.15;"> <span style="font-family: 'Trebuchet MS'; font-size: 21px; line-height: 1.15; text-decoration: underline; white-space: pre-wrap;">Conclusion</span></div> </div> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Android developers do not get a free pass when it comes to incorrectly using texture files. Sure, they have a different profile when it comes to how users expect to download them, but that doesn’t forgive abuse. Plus, PNG files really only help with xfer times, not gpu residence size. So while you benefit in terms of distribution, you’re still taking up bloated space on the GPU. Even something as simple as adopting a 565 or 5551 format would cut your overall PNG texture footprint in half. </span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Remember developers, #everybitcounts</span></div> <div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com10tag:blogger.com,1999:blog-1197137313991855034.post-31276646493466021012014-03-10T07:13:00.000-07:002014-03-10T07:13:12.761-07:00Report : What content makes up an Android Game?<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; line-height: 1.15; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Lately, I’ve been seeing a lot more extended-content apps showing up in my Android game playing habits.</span><span style="font-family: Arial;"><span style="font-size: 15px; line-height: 1.15; white-space: pre-wrap;">That is, games which boast a 1.2mb download, and then require me to wait an extra 20 minutes while it brings down a brand new 1gb file after launching the game. </span><span style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;">Besides</span><span style="font-size: 15px; line-height: 1.15; white-space: pre-wrap;"> the un-announced hefty download (which, by itself has caused me to un-install a number of apps w/o ever playing them), while the games are manually downloading this content, I’m stuck with a useless phone; If i exit their app, I have to do the whole damn thing over again. Which again, leads to a quick un-install.</span></span><br /> <span style="font-family: Arial;"><span style="font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span></span> <span style="font-family: Arial;"><span style="font-size: 15px; line-height: 1.15; white-space: pre-wrap;">I've written games with 1GB of content, and most of these titles I'm playing should not, in any form, be that large when it comes to their data. What it the hell is going on with these things?</span></span></div> <b id="docs-internal-guid-e3b133b5-9ea7-9542-f9f1-9b34518064f2" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In standard engineer fashion, I decided to start understanding what the hell was going into developing these games, specifically where the content footprints for these applications was coming from. It would be extremely useful to have a global </span><a href="http://httparchive.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">httparchive</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for android apps; that type of high-level analysis of the ecosystem would be extremely useful, especially in the context of what data is the most bloated. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But sadly, for the Android ecosystem, that does not exist.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, I wrote my own.</span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsDLNMHt6-wwK0FqXdIfxJFYG4YkiVk9TD2m3Sw0cq19iLrG2IqK-8qgRUOYJ8GGtpjBrowEoEQxD-8ykShAOivi9rB6yQMeO66LpHk1-aVOrJEcwlvor11wY_wbJBaDRiI56xNMhHWbU/s1600/PackingList.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsDLNMHt6-wwK0FqXdIfxJFYG4YkiVk9TD2m3Sw0cq19iLrG2IqK-8qgRUOYJ8GGtpjBrowEoEQxD-8ykShAOivi9rB6yQMeO66LpHk1-aVOrJEcwlvor11wY_wbJBaDRiI56xNMhHWbU/s1600/PackingList.jpg" height="240" width="320" /></a></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">My test comprised of:</span></div> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Grabbing the top 60 games from Android’s Free, Paid, and Grossing categories.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Opening each one, extracting the contents, and keeping track of the file type, and size</span></div> </li> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the case of Unity or flash games, disassembling the packages properly.</span></div> </li> </ul> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Logging and making sense of all this data</span></div> </li> </ul> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <div> <span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> </div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And here’s what I found.</span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><i>Large Disclaimer : This test is not an official statement by Google, Google Play, or any of the content developers. This analysis and content is based on my own research. Also remember that content sizes mentioned only analyzes data provided directly in Google play APK files, and does not include any content that may be downloaded by the application post distribution, outside of OBB files.</i></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">1. If you want a top free game, stay around 40mb.</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Regardless of your medium, it’s pretty easy to draw correlations between app sizes and conversion rates. For instance, a user may browse away from a website if it takes too long to load (perhaps a function of the <a href="http://browserdiet.com/en/" target="_blank">2.5mb of assets it uses</a>?), or perhaps a user can’t download your app while off wi-fi, because the size of your app will <a href="http://europedia.moussis.eu/discus/discus-1343718686-976307-8045.tkl" target="_blank">incur network charges</a>. </span><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Based on my tests, you may be able to see a similar trend for Mobile App marketplaces.</span><br /> <span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"><br /></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">From the look of things, there seems to be a self-selecting market size for apps, that in general there seems to be a specific selected limit for the size of an application, against it’s purchase price:</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_ipPIeKsR_z9s5ZLRmBPpA7j3WOA-_E_hz8f5cfnQWXZRbGlDJeUlx7QiBHS-dE9V5KULE_UQ1Xn6Px_oVoqt2NDEKMAenOPb74tOCsL3wZuyb7eAdtm7tw8i8Hx9rzcIfX6K1j3x4VU/s1600/image04.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_ipPIeKsR_z9s5ZLRmBPpA7j3WOA-_E_hz8f5cfnQWXZRbGlDJeUlx7QiBHS-dE9V5KULE_UQ1Xn6Px_oVoqt2NDEKMAenOPb74tOCsL3wZuyb7eAdtm7tw8i8Hx9rzcIfX6K1j3x4VU/s1600/image04.png" height="403" width="640" /></a></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"> <div style="line-height: 1.15;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Free Apps seem to sit comfortably around 40mb; Which makes sense, I suppose, 40mb on a 3G network would take around </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>3 minutes to load (est 0.21mb/sec on 3G)</b>. Keeping your app as small as possible makes it quick and easy to install for users, of coruse decreasing the amount of friction around acquiring a user.&nbsp;</span></div> <div style="line-height: 1.15;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; line-height: 1.15; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At the complete other end of the spectrum, paid apps tend to, by their nature, define a different sort of engagement for their customers.&nbsp;</span><span style="font-family: Arial;"><span style="font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Paying more money up-front lends itself to a larger content option; that is, higher visual quality requires more content, which manifests itself in a higher asset footprint. I suppose the fact that the user has </span><span style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;">already</span> <span style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;">purchased</span><span style="font-size: 15px; line-height: 1.15; white-space: pre-wrap;"> the content means they are invested in it, and are willing to spend the time to wait for the content to magically appear on their device. What's even crazier is that <b>the Median for the top-paid is about 74MB</b>. Wow.</span></span></div> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <br /></div> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">2. Expansion files are for the premium birds</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Android’s <a href="http://developer.android.com/google/play/expansion-files.html" target="_blank">APK Expansion files</a> are a nice feature for developers looking to bring their content to Android w/o trying to cram every bit into the restrictions imposed by the base-level APK. These extra files (typically containing the .OBB extension) files were intended to be used by developers to reduce the initial download time for an app, and then bring in additional content as needed. In reality, they are now, generally being used to move past the upper limits of what the maximum APK size should be. In the chart below, you can see the breakdown of the number of apps using OBBs for their data content. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="margin-left: 0pt;"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Free</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1/ 60</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Gross</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5/ 60</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Paid</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16 / 60</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">These numbers makes sense. Normally the user-developer purchase contract for a premium game is along the lines of “I like your game, will pay for it, and am willing to wait for it to download.” As such, these tend to charge a bit more money, and as a result, have a higher asset footprint, meaning they will need to store more asset data, and more OBBs show up. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">3. APK content is mostly images.</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">When you've written seven or so game engines for various companies, you start to see a trend that regardless of your game type, the dominant asset footprint seems to always comes from textures. And it doesn't seem to matter if your app is free, or paid, the breakdown in assets seems pretty consistent:</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkDp-S9Hx2g8TplMo-MV6fNIw-1gmnIMtFEYlmxvGAYqej11ekTkTnSiMeBy1f3AYxdpP59l5E6_FE-oasHrAQQqyAjhGVelXRHnL_ynw-HzhGyZ1ZnVlJIjMHDxPYgZNUjlH9eIhzzpo/s1600/fig2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkDp-S9Hx2g8TplMo-MV6fNIw-1gmnIMtFEYlmxvGAYqej11ekTkTnSiMeBy1f3AYxdpP59l5E6_FE-oasHrAQQqyAjhGVelXRHnL_ynw-HzhGyZ1ZnVlJIjMHDxPYgZNUjlH9eIhzzpo/s1600/fig2.jpg" height="480" width="640" /></a></div> <br /> <div class="separator" style="clear: both; text-align: center;"> <br /></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Textures regularly comprise of over 50% of your asset package (for known extensions…). What I didn't expect, however, is how much overhead is generated by code files. It represents more than all the sounds, text, and font files combined! Digging deeper, it appears that this is due to shipping multiple .SO files in a given APK, one for each architecture (x86, ARM, Mips, etc).</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">4. Game developers LOVE random extensions</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So I lied a little in that last section. The real asset breakdown, according to my analysis, looks a lot more like this:</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEir4RQ69lJp-lNqyE3LJ7yEwuByakaF-mbsX7YiOIBtXlo69cKiZQmvEpulg7KWj9WewdxIoBihn3unmG7UTGiMvphGLYwJL_hp5-EOfUygJMQ0lyLnWZnZ0c4VsCEFCqB215B_uJioBPE/s1600/fig3.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEir4RQ69lJp-lNqyE3LJ7yEwuByakaF-mbsX7YiOIBtXlo69cKiZQmvEpulg7KWj9WewdxIoBihn3unmG7UTGiMvphGLYwJL_hp5-EOfUygJMQ0lyLnWZnZ0c4VsCEFCqB215B_uJioBPE/s1600/fig3.jpg" height="480" width="640" /></a></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">That massive ‘unknown’ section comes from file extensions that I didn't recognize, or couldn't determine the content type for. &nbsp;</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In my analysis of 161,225 files from 180 APK files yielded 326 unique extensions.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What the hell are game developers doing?</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sure, there’s lots of file types : Code, Text, Sounds, Textures, Levels, Physics, Videos, Meshes, Animations; Even if you had 5 variations of each of these file types, that’d still amount to about 50 file types. Where the hell is </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">326</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> files coming from? Are some developers just renaming their PNG files after their initials for no reason?</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sadly, this content skews my results in an fashion; For all I know, these could be binary files that are processing Human Genomes, or they could be Texture data..</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" />5. Developers LOVE PNG files</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">When you look at the breakdown of the image types, you end up looking at a graph that looks like this:</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvgD6dIJae40AkHhj0bMXJDCZ5VJ8Olm3wBNsFcIQVZSzrZ9rPlMf4sTcxXV5YjEOt7eSb0c5SnbDd44l-n61slCvA9mEiEMOxYvmz1B-6SFxjhyEjtczuptiJzw7xbW-Ujk2PsfCJT3w/s1600/image03.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvgD6dIJae40AkHhj0bMXJDCZ5VJ8Olm3wBNsFcIQVZSzrZ9rPlMf4sTcxXV5YjEOt7eSb0c5SnbDd44l-n61slCvA9mEiEMOxYvmz1B-6SFxjhyEjtczuptiJzw7xbW-Ujk2PsfCJT3w/s1600/image03.png" /></a></div> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There’s a couple striking things from this graph:</span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Even with the explosion of middleware out there, the number of <a href="http://software.intel.com/en-us/articles/android-texture-compression" target="_blank">GPU specific textures</a> is still in the minority. (Entirely opposite from other gaming consoles.)</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There’s way too many goddamn PNGs</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There’s not a lot of love for WebP</span></div> </li> </ol> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note, this explosion of PNG files does not simply come from the fact that Android requires .PNG files for their icon formats. No no, these numbers are from </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>developers</b> using PNG files for their games. I estimate that there's dominant reasons for this:</span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span id="docs-internal-guid-a41a0797-ac55-daaf-19f9-b912ab250f1a"></span></span><br /> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNG files are required for icons on android platform</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2D games tend to default to PNGs for their spritesheet content.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PNG natively supports alpha, which is easy for developers to understand.</span></div> </li> </ol> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To be fair, PNG holds some nice properties. It can be lossless, compressed with GZIP, most editors&nbsp;</span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">support exporting, Alpha support, and its’ decompression speed is pretty fast. B<a href="http://mainroach.blogspot.com/2013/09/png-compression-5-simple-improvements.html" target="_blank">ut, as I’ve talked about before, PNG is a less than ideal option</a>. </span><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">As a game developer, I’d work my best to put everything in a GPU format. Sadly in mobile, that’s not an easy thing to do, since there’s a massive fragmentation across devices for supported texture formats. But really, there’s no reason to leave it all in uncompressed PNG data. Come on man, that’s just mean.</span></div> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">In Conclusion</span></div> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As a developer, this type of information is quite useful to me. It gives me a sense of what type of decisions developers are making out in the ’real world’ ™. I’m really surprised about how much overhead is coming from code files</span></div> <div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com9tag:blogger.com,1999:blog-1197137313991855034.post-76390498448110655752013-09-18T07:54:00.001-07:002013-09-18T07:54:17.214-07:00Getting more out of GZIP for web content<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"> <span style="font-family: Arial;"><span style="font-size: 15px; line-height: 17px; white-space: pre-wrap;">GZIP and Deflate, for the time being, are the only supported compression formats for serving web content. As I've shown before, it's less than ideal as a format, and has quite a bit of bloat. Thankfully, there's ways out there to serve smaller, more compact <i>versions</i> of GZIP files.</span></span></div> <b id="docs-internal-guid-409c2492-317d-7994-273f-10a7f61c7698" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-tUwsWJTbmDQbAnGsR4nkldcRoVy1TCJx7E04CYf0Vjt_bHChUofFiiMhS48jdWo6WOky23OcOBrOY37_53Ubm1PciT6gIUYHngvWsG4gX601QJ8gdIdO6WdJgzJbaq8UAezJV7ltmYk/s1600/gzip_serve.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="211" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-tUwsWJTbmDQbAnGsR4nkldcRoVy1TCJx7E04CYf0Vjt_bHChUofFiiMhS48jdWo6WOky23OcOBrOY37_53Ubm1PciT6gIUYHngvWsG4gX601QJ8gdIdO6WdJgzJbaq8UAezJV7ltmYk/s320/gzip_serve.jpg" width="320" /></a></div> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GZip on the web</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Can be served by most web servers, although little-known-fact is that the default level is </span><a href="http://en.wikipedia.org/wiki/Mod_deflate" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">6</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which can have ramifications with your data;unless you change it to 9, you could be sending bloated files, although to be fair, my test across 100 files &lt; 100k yielded no difference between -6 and -9</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Using 7Zip to make smaller GZIP files</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="http://www.7-zip.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">7zip</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is a pretty nice utility to generate not only LZMA compressed content, but will also generate </span><a href="http://en.wikipedia.org/wiki/Prediction_by_partial_matching" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PPM</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://www.bzip.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">BZIP</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><a href="http://www.gzip.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">GZIP</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> files as well, and often times, will produce a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">smaller</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> compressed file than the original compressor codecs themselves.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If you check out the 7zip </span><a href="http://www.dotnetperls.com/7-zip-examples" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">switches </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">list, you can see that the command line arguments for generating a .gz file, at max compression:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 9pt; margin-left: 8pt; margin-right: 8pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7z.exe a -tgzip -mx9 ./outFile.gz ./infile.txt</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Using Zopfli to make smaller GZIP files</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Projects like </span><a href="https://code.google.com/p/zopfli/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Zopfli </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">do a great job at producing </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">smaller</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> gzip files, yet do so at a coding level, rather than any modifications to the modeling of the data. &nbsp;On average, it’s reported that Zopfli gies smaller deflate-compatible output sizes than GZIP by about </span><a href="about:blank" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">3.7-8.3%</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">zopfli.exe --i50 -gzip ./infile.txt</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Results</span></h2> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Just to give a sense of things, I tested Zopfli against 7z, and got some good results:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="335px;" src="https://lh6.googleusercontent.com/QgxZ1NoSJZCsqTdalunR2HBfaHcPD6QfkZs_1Gt7M4QvdVVuTPrql6WLoFbdijCtmuH7htIDRmtfwF026yXjjnUNfRG1_ZM0DSnY3fJox_1VdqYBrtOPzZXL" width="708px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You can see that for file sizes &gt;5k Zopfli &amp; 7Zip produce smaller .GZ files easily. As the file size increases you see larger savings. In general, Zopfli @ lvl 50 beats 7zip @mx9 on my small test set (with a &nbsp;few edge cases)</span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgog59VbVK75UJe94BzqcXHZBR_uwi-8YW3HldcrsBLk0aa1qJV_4kMEmGEsvsHu42o34C3zrx7A3Mnq_qX65xSf_b6AbVPpmYPr9J-i_2CvdmcF394lZxf-XmM59Chc1grKNS0dRmYGt0/s1600/SoC_bnr2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgog59VbVK75UJe94BzqcXHZBR_uwi-8YW3HldcrsBLk0aa1qJV_4kMEmGEsvsHu42o34C3zrx7A3Mnq_qX65xSf_b6AbVPpmYPr9J-i_2CvdmcF394lZxf-XmM59Chc1grKNS0dRmYGt0/s1600/SoC_bnr2.jpg" /></a></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Serving a pre-compressed file</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Our end goal is this effort would be to compress all your files offline (gzip, 7zip or zopfli as you choose), upload them to a webserver, and then your page can load and receive them like normal, without any changes to your client-side code. A little bit of Google-Foo produced proof that this ability exists, specifically from </span><a href="http://blog.codegrill.org/2009/07/how-to-pre-compress-static-files-in.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Apache</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><a href="https://forums.aws.amazon.com/message.jspa?messageID=137563" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Amazon Web Services</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Sadly, Google App Engine removed this functionality directly (due to security concerns) so you may need to work around that part of this implementation.</span></div> <br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If you’re a web-app developer, and have the ability to “push a build” there’s no reason to be sending extra bytes down the wire when you can easily save more data.</span></div> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-1197137313991855034.post-25895939435994430732013-09-10T08:16:00.003-07:002013-09-11T11:04:10.939-07:00Extreme Javascript Minimization<b id="docs-internal-guid-5e6e8a35-086e-743f-d03b-b9b0f1e5beaf" style="font-weight: normal;"><br /><span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Compressing javascript source code is a tricky undertaking. To my knowledge, the only domain specific javascript compressor I’ve found is </span><a href="https://www.usenix.org/conference/webapps-10/jszap-compressing-javascript-code" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">JSZap</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which is a hefty undertaking in terms of tree-teardowns. There’s also the proposals for </span><a href="http://www.w3.org/2012/11/webperf-slides-hundt.pdf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Javascript Diffing</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which would save tons of bandwidth every day for GMAIL, but that’s a very generic algorithm. Most of the web seem content to simply rely on GZIP for compression; Which seems fine when your JS is under 15k; But for a </span><a href="http://www.unrealengine.com/html5/%20" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">25mb JS file</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, you’ve got problems.</span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijGkfPpBUvG8G1JRawgXGn-gO52I1yUHoPKVSdtOFHVBF09_qo28QlEZqzAM1YE9bIBLP1lT8RNv4RaO_RyKXnUVIlZQvS66GpLtdZlf1Zx1KNIvUkHsMRVeGAF42ZZjxqDNJKDNLgFZs/s1600/image00.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="266" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijGkfPpBUvG8G1JRawgXGn-gO52I1yUHoPKVSdtOFHVBF09_qo28QlEZqzAM1YE9bIBLP1lT8RNv4RaO_RyKXnUVIlZQvS66GpLtdZlf1Zx1KNIvUkHsMRVeGAF42ZZjxqDNJKDNLgFZs/s400/image00.jpg" width="400" /></a></div> <b style="font-weight: normal;"><br /><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">BTW, I quickly want to take a second and discuss some terminology :</span></div> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Minification </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">is the act of reducing data such that it can be consumed without processing by the underlying system</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Compression </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">is the act of reducing data length such that it </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">requires</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> processing to recover it to a state in which the system can operate on it.</span></div> </li> </ul> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GZIP is a compressor; it reduces the data from its’ </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">functional</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> form to an </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">interpreted</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> form</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They may seem similar, but as I’ll show in this article, the ramifications in mind-set between the two are quite large. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Automated Javascript Minifiers</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At the top of our chart are the automated set of minifiers that fit into your production pipeline. Tools like </span><a href="https://github.com/mishoo/UglifyJS" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">UglifyJS</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://www.crockford.com/javascript/jsmin.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">JSMin</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://yui.github.io/yuicompressor/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">YUI</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://opensource.perlig.de/rjsmin/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">rJSMin</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://dojotoolkit.org/documentation/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Dojo ShrinkSafe</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://aspnet.codeplex.com/releases/view/40584" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Ajax Minifier</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and </span><a href="https://developers.google.com/closure/compiler/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Closure</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Much like we saw with CSS minifiers, </span><a href="http://www.alittlemadness.com/2010/06/03/javascript-compressor-comparison/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">there’s no dominant</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, one-size-fits-all JavaScript minifier either. Most of these systems work by compiling your Javascript into some sort of Abstract Syntax Tree representation, and then recovering from that doing various scope renaming and variable minimization. Although these automated minimizers do their job, they have recently been exposed to lack advanced optimizations that </span><a href="http://videos.sapo.pt/DsuVHAegdODkMd4Vcb0e" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">stupid robots have no idea how to do</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Although I’m sure there’s a whole separate discussion about if those hand-made adjustments hit the gzip threshold or not; but let’s skip that for now.</span></div> <br /> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The JS Demo Scene</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvqOlVvDzZxVLMV0Xjoy4Wop_rSvaUw8B5pRvFCqnXpUJunDunx_bsIpLeiIcmHTLZ0sC4ChfQjLiW2lj_Q3rtUUHUpAFsmpesuTo8tXJ-RuIQkmIhHMHQ9ob8TidKG2oRY4wYZenLi0g/s1600/image01.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvqOlVvDzZxVLMV0Xjoy4Wop_rSvaUw8B5pRvFCqnXpUJunDunx_bsIpLeiIcmHTLZ0sC4ChfQjLiW2lj_Q3rtUUHUpAFsmpesuTo8tXJ-RuIQkmIhHMHQ9ob8TidKG2oRY4wYZenLi0g/s1600/image01.jpg" /></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">With the addition of WebGL to the HTML5 stack, it’s opened up a new door for graphics-programming-hackers to bring their love of the </span><a href="http://en.wikipedia.org/wiki/Demoscene" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">demoscene</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to the web. If you’re unfamiliar, DemoScene applications attempt to pack as much excellence into as few bites as possible. One of the most popular ones is </span><a href="http://pouet.net/prod.php?which=12036" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">.kkreiger</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> who fits an entire First Person Shooter, complete with weapons, particles, textures, enemies, animations, post processing, lens flares, sound, music and physics into a single 64k executable. This is such an amazing task, especially considering that the </span><a href="http://www.youtube.com/watch?v=2NBG-sKFaB0" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">video showing off the game</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is larger than the actual game itself.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This combination of hacking, graphics, and feedback speaks to the core of many engineers, in fact, in various parts of Europe, there’s even a notion of a DemoScene </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Party </span><a href="http://upload.wikimedia.org/wikipedia/commons/e/ea/Assembly2004-areena01.jpg" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">which is taken pretty seriously</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">; awarding prizes and swag to the best demo submitted under a byte threshold.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Once these hackers set their attention towards javascript, fun things were bound to happen. Dominantly these demos extract their awesome from source-code level hacks that allow them to squeeze as much functionality into as few bytes as possible, which usually includes using procedural algorithms to generate game content. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note that the demo scene doesn’t accept browser-based GZIP as a contribution; the raw Javascript file must be under the threshold specified. This means in order for demo hackers to meet their quota, they have to push custom compressors onto their javascript </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and include the decompressor in the javascript too.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This has given rise to an awesome set of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">inline extractors</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for javascript which address a unique orthogonal need : Compress the source as much as possible, with the least number of bytes for the compressor.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="http://www.iteral.com/jscrush/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">JsCrush</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://dean.edwards.name/packer/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">jsPacker</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="https://github.com/cowboy/javascript-packify" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Packify</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> round up the top of these extractors, effectively applying a longest-string replacement process to their source (somewhat like an LZ77). More advanced versions like </span><a href="http://creativejs.com/2012/06/jsexe-javascript-compressor/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">jsEXE</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://skypher.com/index.php/2010/12/09/jssfx3/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">jsSFX3</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and </span><a href="http://crunchme.bitsnbites.eu/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CrunchMe</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> all produce more advanced versions of these, mostly due to advanced </span><a href="http://liblzg.bitsnbites.eu/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">compression codecs</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and the ability to pack javascript as </span><a href="http://blog.nihilogic.dk/2008/05/compression-using-canvas-and-png.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">GZIP compressed PNG data</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, inline.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This has brought around a whole new development experience; how low can you go</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note how these systems are true &nbsp;</span><a href="http://en.wikipedia.org/wiki/Kolmogorov_complexity" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Kolmogorov</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> measurements because the encoder exists alongside the data stream</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Extreme minification </span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPYhgTQuqWB8B7trgMq6Q1isxEpUabGhCJQ4cXsBJCP-YnxRFjijwDtWPSMSjDNvoy85WvyEvrP3mfJXCdICeDjcrhlfnAA3o4z3zCGX2XMXQXk8iDJm506XuroO5bZbjVJOjWT0VAUHM/s1600/image02.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="133" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPYhgTQuqWB8B7trgMq6Q1isxEpUabGhCJQ4cXsBJCP-YnxRFjijwDtWPSMSjDNvoy85WvyEvrP3mfJXCdICeDjcrhlfnAA3o4z3zCGX2XMXQXk8iDJm506XuroO5bZbjVJOjWT0VAUHM/s200/image02.jpg" width="200" /></a></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The demo scene folks have done a great job of compressing their JS data through various means, however there’s a new generation of JS hackers that push past those automated compression methods into hand-generated minification, which often times goes farther than any automated process.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As an example, I here is a Base64Encoder in 140bytes of javascript:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">function(a,b,c,d,e,f,g){for(g=c=e="";(d=a.charCodeAt(g))&amp;&amp;(c+=/.{8}$/(1e7+d.toString(2))),f=c.substr(g++*6,6);)e+=b[parseInt(f,2)];return e}</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Not impressed? Fine, here’s a 140byte music synth (try it </span><a href="http://www.p01.org/releases/140bytes_music_softSynth/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">here</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">):</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">function(f){for(var t=0,S='RIFF_oO_WAVEfmt '+atob('EAAAAAEAAQBAHwAAQB8AAAEACAA')+'data';++t&lt;3e5 data-blogger-escaped-eval="" data-blogger-escaped-f="" data-blogger-escaped-return="" data-blogger-escaped-s="" data-blogger-escaped-span="" data-blogger-escaped-tring.fromcharcode=""&gt;<!--3e5--></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Most of these </span><a href="about:blank" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">demos</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> adhere to a </span><a href="https://github.com/jed/140bytes/wiki/Byte-saving-techniques" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">new style of minmification</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, that, by my eyes, seems damn near impossible to automate.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For example, take this presentation on </span><a href="https://docs.google.com/a/google.com/presentation/d/1VUaCoIo5-OXoRN2yX4stjlNTxtK0A3_NZSOZnYLTyK8/present#slide=id.i0" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">JSGolf lessons</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which present how to create a </span><a href="https://gist.github.com/aemkei/1672254" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">140byte tetris game</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> :</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">function(a,b,c,d){d+=c;<br class="kix-line-break" /> &nbsp;&nbsp;&nbsp;return[d<0 a="&lt;br" amp="" b="" class="kix-line-break" d=""> &nbsp;&nbsp;&nbsp;parseInt((a|b&lt;<c br="" class="kix-line-break"> &nbsp;&nbsp;&nbsp;.toString(d=32,b=new Date&amp;2|1)<br class="kix-line-break" /> &nbsp;&nbsp;&nbsp;.replace('v',''),d):<br class="kix-line-break" /> &nbsp;&nbsp;&nbsp;a|b&lt;<d a="" b="" d="" span=""></d></c><!--0--></0></span></div> </td></tr> </tbody></table> </div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Granted, there’s some parts missing, the above code assumes that you’ve got some canvas and other data available, but the idea is there, and pretty cool. The autor from </span><a href="http://aem1k.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">aem1k.org</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> also lists some </span><a href="https://docs.google.com/a/google.com/presentation/d/1azRe7gmmfaGs_QMNIMm4NUoDylf0wXjdqGjj9wH4z-Q/present#slide=id.i0" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">great resources</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for packing numbers in JS as well.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Extreme obfuscation</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I know it’s not in the same boat, but I feel like I should mention some really awesome javascript hacks out there that you should play with. Recently, it’s been shown that there’s a transform which will take Javascript and represent it in only 6 symbols; Note that the represented version is still </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">working and valid</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Simply pass it to the browser, and things will execute as normal. Three known transforms are </span><a href="http://utf-8.jp/public/jjencode.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">JJencode</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://utf-8.jp/public/aaencode.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">AAencode </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and </span><a href="http://www.jsfuck.com/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">JSFuck</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which all encode things using various methods. In general, these provide an interesting obfuscation method for javascript, and bring up some interesting questions about compression.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Large compression wins occur when you can reduce the number of potential symbols (ie histogram reduction) alongside the number of symbol frequencies. These obfuscators clearly achieve the first method; reducing all javascript down to only 6 characters. However the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">inflation</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> caused by this process violates the 2nd process of compression; post-compressing the transformed data does yield some amazing compression </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ratios</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, however the inflation is so large that the overall file size is still larger than the original. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVmJbgTL2gvx-WIFDiwP3g4wlvR7qOMOKZw8Xk8Q30FPge0e44-fTiIsR_fDBhRKAvZbk444hwtnKkpINhyyJZghWiIICSJmRKnbtHak4ogou8UJprqMBvMv2Owr_yt8RXHn-RcxhlNz0/s1600/SoC_bnr1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVmJbgTL2gvx-WIFDiwP3g4wlvR7qOMOKZw8Xk8Q30FPge0e44-fTiIsR_fDBhRKAvZbk444hwtnKkpINhyyJZghWiIICSJmRKnbtHak4ogou8UJprqMBvMv2Owr_yt8RXHn-RcxhlNz0/s1600/SoC_bnr1.jpg" /></a></div> <b style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">How is any of this helpful?</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Although 60% of the web are images, Javascript maintains it’s spot as 2nd largest resource; And things will become increasingly difficult for Javascript file sizes, especially with technologies like Emscripten around. For example, the Javascript files for </span><a href="http://www.unrealengine.com/html5/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Unreal3’s HTML5 port</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> are around 25MB of data; Post GZipped to 5mb. That’s insanely huge; and by my estimates, as web-apps become more powerful on mobile, we’ll see these types of footprints becoming more popular over time.</span></div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <br /> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com8tag:blogger.com,1999:blog-1197137313991855034.post-4779130470577074782013-09-04T07:33:00.000-07:002013-09-04T10:08:10.177-07:00PNG Compression : 5 simple improvements<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="http://en.wikipedia.org/wiki/PNG" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PNG</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is a simple format that supports </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">transparency </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">lossless </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">compression, and used almost all over the web for it’s support of transparency. Sadly though, PNG files don’t compress as small as their JPG, or GIF counterparts, which can cause lots of headaches for HTML5 game developers. This article will look at some ways to fix that.</span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJqxiEMLcGoX2mMR1FYLHST4xscHvTJgintNHDQOyKi2UYYBX-tixiAE-5wKmv72ciIadClyLujZdD0U4Q1dy_opMFc_uqjn6jZWVFtbXauLlpqgB-ykN3ZuuJ6u3tn8PxkjMq0AckHtg/s1600/3-fingers-8ma.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJqxiEMLcGoX2mMR1FYLHST4xscHvTJgintNHDQOyKi2UYYBX-tixiAE-5wKmv72ciIadClyLujZdD0U4Q1dy_opMFc_uqjn6jZWVFtbXauLlpqgB-ykN3ZuuJ6u3tn8PxkjMq0AckHtg/s400/3-fingers-8ma.jpg" width="400" /></a></div> <b id="docs-internal-guid-18551d2e-e962-5570-25f9-25c3805ca30f" style="font-weight: normal;"><br /><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1. Use a better Deflate</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The </span><a href="http://en.wikipedia.org/wiki/DEFLATE" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Deflate</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> algorithm is what PNG uses internally to compress its’ pixel data. I’ve already talked about the fact that &nbsp;</span><a href="http://mainroach.blogspot.com/2013/07/the-state-of-web-compression_8.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Deflate/GZIP are older algorithms, and don’t compress as well as newer codecs</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Which means that your PNG images are not being compressed as small as they should be. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Generally web developers get around the limitations of GZIP by using a better GZIP encoder. That is, they manually GZIP files offline, and provide them to the server, such that it will be passed directly to the user. Advanced compressors like </span><a href="https://code.google.com/p/zopfli/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Zopfli</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><a href="http://www.7-zip.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">7zip</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> can be used to produce .gz files (AKA GZIP), and will regularly produce </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">smaller</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> GZIP files at that. These compressors gain wins with the use of advanced searching / matching algorithms coupled with data structures that utilize more memory for better pattern matching.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; line-height: 1.15; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Looking back at PNG, a great set of tools like </span><a href="http://advancemame.sourceforge.net/doc-advpng.html" style="line-height: 1.15; text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">advPNG</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; line-height: 1.15; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> have popped up with this same solution in mind : they will re-pack your PNG data using better compressors to generate a smaller PNG file.&nbsp;Another great example includes combining <a href="http://advsys.net/ken/utils.htm#pngout" target="_blank">PNGOUT</a> (</span><span style="font-family: Arial; font-size: 15px; line-height: 17px; white-space: pre-wrap;">by the legendary </span><a href="http://www.codinghorror.com/blog/2007/03/getting-the-most-out-of-png.html" style="font-family: Arial; font-size: 15px; line-height: 17px; white-space: pre-wrap;" target="_blank">Ken Silverman</a><span style="font-family: Arial;"><span style="font-size: 15px; line-height: 17px; white-space: pre-wrap;">) coupled with tools like <a href="http://optipng.sourceforge.net/" target="_blank">OptiPNG</a> or <a href="https://code.google.com/p/zopfli/" target="_blank">Zopfli</a>. Of course, each of these systems creates slightly different results, given the input systems, so it may be wise to adopt a system which will compress against multiple compressors and pick the smallest file; If you're feeling lazy, <a href="http://css-ig.net/scriptpng" target="_blank">ScriptPNG</a> will do the heavy lifting for you.</span></span><br /> <span style="font-family: Arial;"><span style="font-size: 15px; line-height: 17px; white-space: pre-wrap;"><br /></span></span> <br /> <div style="line-height: 1.15;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To be clear, if you’re not using an advanced PNG compressor to create your PNG files, then you have bloated PNGs.</span></div> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2. Improve PNG compression with a lossy pre-processor</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Most advanced image compression systems rely on a two stage algorithm, they first apply a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">lossy</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> version of compression to the source data, and then a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">lossless</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> one. In essence, the lossy transform often removes information from the stream, alongside re-configuring symbols such that the lossless phase can compress better.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sadly, PNG’s Deflate option is </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">only</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> a lossless encoder, but that shouldn’t stop you from embracing a lossy preprocessor if you want one. Image processing tools like </span><a href="http://pngmini.com/lossypng.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">ImageAlpha</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><a href="http://imageoptim.com/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">ImageOptim</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, can compress your PNG image in a lossy method as a pre-process before passing it off to the final PNG format. This creates a two-step process where your lossy, and lossless compression are done by two separate applications. The results are impressive, the reduced color space allows the lossless compressor to find, and make, more frequent matches in the file, yielding to better compression.&nbsp;</span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To be clear, if you’re not using a lossy pre-processor to create your compressed PNG files, then you have bloated PNGs.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3. Splitting your transparent layer for improved compression.</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="https://www.udacity.com/course/cs255" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HTML5 game developers</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> typically transfer more image data than your standard website, most of it being transparent frames for various sprite based flipbook animations. Sadly this process forces game devs to use the PNG format in order to get transparency for their canvas games. However some </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">cunning</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> developers have devised a few </span><a href="http://kaioa.com/node/104" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">work-arounds</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for images to get better compression </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> transparency. For example, you can split your color data, and transparency data into two separate image files (two JPGs, for example), and restoring them on the client using a CANVAS element. Although this does increase the number of requests that occur on the network, the savings in image size can be significant for developers who have tons of transparent images on their site (like). </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To be clear, if you’re not splitting your RGB and Alpha data into separate compressed data sets </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(where it makes sense)</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, then you have bloated PNGs.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVmJbgTL2gvx-WIFDiwP3g4wlvR7qOMOKZw8Xk8Q30FPge0e44-fTiIsR_fDBhRKAvZbk444hwtnKkpINhyyJZghWiIICSJmRKnbtHak4ogou8UJprqMBvMv2Owr_yt8RXHn-RcxhlNz0/s1600/SoC_bnr1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVmJbgTL2gvx-WIFDiwP3g4wlvR7qOMOKZw8Xk8Q30FPge0e44-fTiIsR_fDBhRKAvZbk444hwtnKkpINhyyJZghWiIICSJmRKnbtHak4ogou8UJprqMBvMv2Owr_yt8RXHn-RcxhlNz0/s1600/SoC_bnr1.jpg" /></a></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4. Don’t Trust Deflate</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Deflate compression algorithm is a combination of two lossless compressors, </span><a href="http://en.wikipedia.org/wiki/LZ77_and_LZ78" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">LZ77</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and </span><a href="http://en.wikipedia.org/wiki/Huffman_coding" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Huffman</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. LZ77 is a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">dictionary</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> compressor, that will work to encode a symbol (or group of symbols) by attempting to match it with symbols it has </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">already seen</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. I.E. searches all N-1 symbols to see if symbol N has ever been seen before. In order to keep things sane, the Deflate algorithm will only check the previous 32k values (this is called it’s ‘search window’). This restricted window size helps improve compression speeds, and focuses on general locality-based duplication that most image and text data tend to exhibit.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This does run into one issue however, in that if you </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">have</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> seen a symbol before (or pixel in our case) but it’s outside the 32k window, then you’re out of luck. The symbol will be encoded in it’s raw, uncompressed form, and hopefully you’ll see it again in the future to take advantage of it. This is the ebb and flow of the LZ family of algorithms.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Little known to most web developers, this can have serious ramifications for your content sizes. For example, consider the following figure:</span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0AGritkgX9kg-Ipq6ZEoVL-enZbls-I5-DYfXnb-XN2NiGIYjFBJ2JTLW_XlqTXszoiGH6OMZ_yJF37LTEW3YwB5ElupIftbzfEFYBmlebOIAQjVgYE65g3znZ8hAr24dEtpiCcJZF5c/s1600/png_kiwi.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0AGritkgX9kg-Ipq6ZEoVL-enZbls-I5-DYfXnb-XN2NiGIYjFBJ2JTLW_XlqTXszoiGH6OMZ_yJF37LTEW3YwB5ElupIftbzfEFYBmlebOIAQjVgYE65g3znZ8hAr24dEtpiCcJZF5c/s1600/png_kiwi.jpg" /></a></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <br /></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The leftmost image is a 90 x 270 image; Most likely a sprite sheet that was created, the kiwi’s in the image are exact duplicates in each tile. Note that each tile is </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">90x90</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> pixels. The 2nd image, we add 2 transparent columns to the image, making the total size of the image </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">92x270 pixels</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and each kiwi-tile is </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">92x90 pixels</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">What’s astonishing about these two images, is than when they are saved in their compressed form </span><span style="color: black; font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">the 2nd image is</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">twice as large as the first</span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Logically this seems wrong. Adding 540 pixels to an image shouldn’t cause a 2x bloat in compression. However when we look a little closer, we can see why this is occurring; The following </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">heat map</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of the images represents </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">how compressed a given pixel is</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Deep blue = very compressed, yellow/red = not very compressed</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <img height="310px;" src="https://lh5.googleusercontent.com/7F5bMUcWDvjgzN9kp-HDiXwVtj273UhUroFUtibo0-Rva8oOfyCQ4MeAusBSuJ94Iz2awmxY0oSHUJm3Vjn3Wg9w-HWbrqWRYAHhCKzSFQrptgVB5JhijUb3" width="219px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What are we looking at here? In the first image, you can see that the kiwi’s in the 2nd and 3rd tiles are defined as </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">deep blue</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> meaning that they are very well compressed; their final contribution to the file size is small. In the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">second</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> image however, you can see that the kiwis mostly green, yellow and red, meaning that they are not as well compressed.</span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It just so happens that our </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">two pixel larger image </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">exceeds the size of the lz77 window for our tiles. Meaning the lower two tiles won’t find as many matches, and won’t compress as well. Each 90x90 tile easily fits in the 32k window (90x90x4 =</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 32,400</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">), so subsequent tiles match properly. However our 92x90 tile exceeds the window (92x90x4 = </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">33,120)</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> resulting in worse compression.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now, this should be scary, as well as confusing. Firstly there’s no clear definition about what </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">really</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to do here. Ideally we’d find more icon/sprite packers would perhaps sort images based upon their histograms in an attempt to create smaller images.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To be clear, Deflate really sucks and I wish we’d move away from it on the web.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5. Don’t use PNG</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Everything I’ve covered here attempts to improve the compression of PNG through algorithms which are easily available in other codecs. The best way to improve PNG compression, is to not use PNG, instead, use </span><a href="https://developers.google.com/speed/webp/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">WebP</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. The WebP format will still be usable in HTML5 canvas, although it doesn’t have universal adoption in all web browsers just yet.</span></div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <br /> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com7tag:blogger.com,1999:blog-1197137313991855034.post-16663935447476930602013-08-27T06:36:00.005-07:002013-08-27T07:21:22.745-07:00Boosting Text Compression with Dense Codes<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Since we’re stuck with GZIP on the web-stack, it makes sense for us to investigate various ways of boosting the compression of GZIP somehow. I decided to take a deep dive into a few compression methods known as “boosters”, they are preprocessing algorithms that allow other, existing, compressors to produce better results. If this is the case, then we may have an amazing new tool in reducing the transfer sizes of web data.</span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMY-m53oiSYSMCAcrqIdfebsVnPeuLz0WlZVAhk4svFNkkiyDEyNvc5g7FWHg3pKPzavHVs4bz2zhQejyklDMUePt6ptpDNxBwh2gB-TlyGy-bnlE-aMb6Qy9AwOeQ6RZjc4ELYR2Yi-E/s1600/arrow_up_graph.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMY-m53oiSYSMCAcrqIdfebsVnPeuLz0WlZVAhk4svFNkkiyDEyNvc5g7FWHg3pKPzavHVs4bz2zhQejyklDMUePt6ptpDNxBwh2gB-TlyGy-bnlE-aMb6Qy9AwOeQ6RZjc4ELYR2Yi-E/s200/arrow_up_graph.jpg" width="200" /></a></div> <b id="docs-internal-guid-528045eb-bff2-3bd0-6ba4-534d875e2143" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Boosting compression</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I am seeking the ability to modify the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">modeling</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of the data such that GZIP(A) &gt; GZIP(X(A)) for A being 50% of the text-based files on the internet. Or rather, how can we pre-process our source input such that GZIP produces a smaller file? This is also referred to as “Boosting” in compression.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To be clear, the goal is to provide a two part compression codec</span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">An offline compressor of static text (language / platform independent)</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A runtime decompressor (in javascript)</span></div> </li> </ol> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The codec itself will comprise of a series of boosters, using GZip’s DEFLATE as the last stage of compression. We will ignore the discussion of using direct other compressors, as I’ve covered some rough performance results </span><a href="http://mainroach.blogspot.com/2013/07/the-state-of-web-compression_8.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">here</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and there are further questions about the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">size</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of the client-side javascript required to decompress, and how that can negate any compression savings (more on this process in yet another later article).</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What we’re going to be talking about today is some research that claims to boost compression of text data</span></div> <br /> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Boosting with Dense Codes</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">My internet digging yielded a set of awesome papers : </span></div> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“Boosting Text Compression with Word-based Statistical Encoding” (paper </span><a href="http://www.dcc.uchile.cl/~gnavarro/ps/tcj11.pdf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">here</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">) </span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“Word-based Statistical Compressors as Natural Language Compression Boosters” (paper </span><a href="http://www.dcc.uchile.cl/~gnavarro/ps/dcc08.1.pdf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">here</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">). </span></div> </li> </ul> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Both of which present a similar concept to boosting compression for natural language input:</span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Phase 1 Dense-Code dictionary</span></div> </li> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tokenize the text such that we’re given word-sized tokens</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Replace tokens in the source text with a modified index scheme, Called </span><a href="http://www.dcc.uchile.cl/~gnavarro/ps/spire04.3.pdf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Dense Codes</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </li> </ol> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Apply a byte-based encoder to the entire data (bzip2, gzip, lzma etc)</span></div> </li> </ol> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This really isn’t different than other word-based dictionary encoder processes, other than </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">how</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> they are encoding dictionary-references in the stream. I</span><a href="http://mainroach.blogspot.com/2013/07/css-compression-binary-css-format.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"> talked about this issue before</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">; and it’s far from being a solved problem for compression as a science.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Their results claim that their word-based system + dense code indexing provides an efficient text pre-processor for other byte-wise compressors;The table below shows the results of applying various word-based pre-processing methods combined with some common byte-wise, kth-order compressors on a single data set:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="171px;" src="https://lh4.googleusercontent.com/BS7v3PuNrFbDAuT4lK_4c7isvaJYfsiq39hHQo6GIfUSvOonnZqqKLjNl0U-5AsWsA3QzqJIk_d9oz12eixvlwLNFTaATb-xTCN2wIakV9JLjxpEe_bRGxpO" width="649px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You can see that when comparing their SCDC and ETDC codes against Tagged Huffman, or no boosting at all, that SCDC and ETDC forms easily win out by an impressive margin. What’s odd, however, is that in the </span><a href="http://vios.dc.fi.udc.es/codes/publications.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">entire corpus of papers on this topic</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (from the same authors) only a few references to the result that size(ETDC + GZIP) is less than GZIP alone for large </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">amounts </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">of data sets, but rather only a few references to a single data set. Which I found odd, as it didn’t give scope for the types of success and failures in the technique.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">My experiments with their </span><a href="http://vios.dc.fi.udc.es/codes/download.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">source code</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> may shed light into why. A random set of 40 files (comprised of popular javascript, JSON, and CSS files, along with some txt files) proved that only 18% of the files gained boosting savings by combining SCDC/ETDC with GZIP, for the majority of the files tested, simply GZIP’ing the source data provided a smaller file. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For a more conclusive result, I downloaded all text-based assets from the top 1000 sites listed on</span><a href="http://httparchive.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"> httparchive.org</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, (that is, for each listed site, cache locally all the JS, HTML, JSON, and CSS data) and ran a majority of them through the provided source code. (there were many instances where the compressor simply crashed). </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">These results were worse, where only 2% of the first 7000 javascript files yielded wins when combining GZIP with ETDC.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It’s unclear, at this point, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">what exactly</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> causes the lack of compression savings for ETDC method against these other text data files.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I hypothesize there three influencers:</span></div> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tokenization </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- JS,CSS, HTML don’t obey the same ‘natural language’ rules of english text, could improper tokenization be causing the problem?</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Coding - </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ETDC uses a type of Tagged Huffman codes called Dense Codes, perhaps LZ and Huffman don’t encode their runs that well?</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Pair-Wise Correlation </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- Perhaps the issue is that there isn’t enough mutli-word correlation in the file, such that the post-tokenized byte stream doesn’t compress as well?</span></div> </li> </ul> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s dig deeper on each one.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tokenizer</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <div class="separator" style="clear: both; text-align: center;"> </div> <br /> <img border="0" height="196" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhD8gXHLd2N1ZeF8aJ8odv8XuH0IL51dClnEtfTNgEXFvX6-c_8Z-_Qsk7Q20HzU5xX4ff52b7F9QrWfpad0OPwetNeL1bLUdZMpEgmb48yxVb8sa5A8bjC6B5-bQhiC23cLry1suwLJeM/s200/token.jpg" style="float: left;" /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It becomes apparent that the tokenization process for the provided ETDC source code is built specifically to handle natural language english text. It’s noted that of the tested files whose combined ETDC and GZIP yielded boosted savings, the majority of the file is english text. However source code for HTML5 does not follow this pattern. JS, CSS, XML and JSON have similar natural-language patterns, but do not tokenize under the same rules. &nbsp;Typically, ‘natural language’ tokenization will split an english string by spaces, using each word as a token, and simply add the spaces back in during decompression.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The designed ETDC codec </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">follows</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> this principle pattern; The expectation being that the decompressor will manually insert spaces back into the stream between each recovered token.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">CSS, JS, JSON and HTML are ‘semi natural’ in that they seem close enough to ‘natural language’ syntax to be acceptable, but their minimized forms may not exhibit that behavior. &nbsp;Spaces are actually pretty important, syntactically to these files; JS, CSS, JSON, XML etc all have explicit situations / subdata where spaces are </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">required</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for the file information to be parsed and applied correctly. As such, our algorithms must be able to take into account the ability to reconstruct spaces in some form.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As such, I investigated some alternate tokenizing processes in an attempt to find savings for these types of data.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RegEx Word based tokenization- </span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This process uses a </span><a href="http://docs.python.org/2/library/re.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">regular expression for a ‘word</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">’. This technique suffers from a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">complete removal of spaces</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, even inside of quote terminated strings and other important places that the syntax may need them. We could attempt to simply append a space to all tokens during decompression, but this breaks the syntax of css &amp; javascript when you end up placing a space between a “.” and associated identifier. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Natural Language tokenization- </span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Another approach is to group alphanumeric and non-alphanumeric runs of symbols together, using a space value as a switch delimiter. This allows us to get things like “display:linear11” to be [“display”,”:”,”linear11”] effectively switching recording when we reach a transition letter. This technique suffers from a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">complete removal of spaces</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, even inside of quote terminated strings and other important places that the syntax may need them.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Space appended tokenization- </span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Both natural language and word based tokenization suffer from the inability to recover space values for text that requires them. English language is generally homogenous with it’s need for space insertion, however for source code, it is not uniform. This technique can be applied to natural and word-based techniques, easily:</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">First split by spaces; which allows us to retain where those belong in the file; Then, for each of those tokens, do a word-based tokenization, which yields better results. The benefit of this system allows us the best of both worlds; however doesn’t produce the minimal token suite. This fails to produce the best tokenization if the input file has not been minimized, or is riddled with needed space information.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Length based tokenization - </span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Use a diminishing longest-substring search; That is, find the longest substring, and mark that as a token, and remove it from the stream; repeat this process until we only have 1 symbol streams left.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">LZ77 tokenization - </span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As an experiment, I attempted to LZ77 the input data, and used the matched/unmatched character runs as explicit tokens. This method attempted to take advantage of the fact that LZ77 exploits tokenization locality, and often is the reason (IMHO) that there are not more literary discussions about how to properly tokenize a data stream for better dictionary results.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">LZW Tokenization - </span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">THIS Suggests that LZ78 has been shown to achieve asymptotic optimality when the input is generated by an ergodic source. The suggested parsing process will first tokenize the string via LZW algorithm, and then pre-process the stream, at each step of the algorithm, the longest prefix matched in the dictionary is used.</span></div> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Results - </span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Below is a test of a few file types against each separate tokenization schemes. Note that each scheme was using the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">SCDC</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> codes.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="116"></col><col width="71"></col><col width="70"></col><col width="77"></col><col width="73"></col><col width="68"></col><col width="76"></col><col width="74"></col><col width="71"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src / gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">NL</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Space NL</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RegEx</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Space RegEx</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">LenMtc</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">lz77</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">lzw</span></div> </td></tr> <tr style="height: 33px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bootstrap.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">97,452</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">41,011</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">40,967</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">40,937</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">41,068</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">253,266</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">89,975</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fce5cd; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">36,524</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: right;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,540</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #ffe599; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,431</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16,089</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,437</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16,083</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">20,377</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">20,065</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">18,647</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">prototype.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">187,863</span><span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">94,628</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">73,748</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">80,689</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">98,157</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">n/a</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">173,930</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fce5cd; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">74,150</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: right;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">40,571</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">42,485</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">41,317</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">42,643</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">42,715</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">n/a</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">50,396</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">44,950</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">patton.txt</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">8,715</span><span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7,800</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7,117</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7,633</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7,319</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">6,983</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">8,864</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fce5cd; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5,598</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: right;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3,857</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4,813</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4,742</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4,787</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4,858</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5,313</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4,350</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4,639</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">videos.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">206,467 </span><span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">133,570</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">100,808</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">104,760</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">129,583</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">n/a</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">185,227</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fce5cd; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">90,429</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: right;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">60,761 </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">64,397</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">62,378</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">61,728</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">65,020</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">n/a</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">61,627</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">61,982</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Once the string has been tokenized with LZW, the naive parsing strategy is simply to pick the longest prefix matched in the dictionary; My results showed that to be less than ideal, instead, once the longest prefix is found, test all substrings [n0....nx] (x=len(prefix)) and select the one which produces the largest savings in the future stream (or that is seen the most times across the entire dataset). This modification allowed LZW to achieve the best entropy across all tokenizers</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In general, these modified tokenization systems </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">failed to produce a savings</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of the ETDC process, and some (like LZ77) forced worse savings. Oddly enough, &nbsp;the original paper defines the ability to remove spaces from the source stream, and re-insert them at the end. This has a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">drastic</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> impact on the performance of ETDC as a compression booster, and can save 1-4% of post-gzipped size for some files . It appears without this tokenization assumption, the text-boosting process does not yield positive results.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Codes used</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <img border="0" height="150px" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjMT_mql06sXU85iUiINLLfTf3sLLSxuQkysWqdK_MAHEp00Uu0qDkvH5y87tKFuNosziq2JynfGAU7ti7M7-cRQuXrJ7dsqFxzcGEBA2LY7AcUsjjRMV4j1eGjJJHPwZlULe1Xd1k5Rs/s320/futurama_code.jpg" style="float: right; margin: 15px;" width="150px" /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">TH and ETDC codes have specific markers that make them </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">different</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> than simple integers, yet allow us to scan and find them, at any byte stride, without additional overhead. An error in my early encoding process (listed as </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">wrong</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in the table below) yielded interesting data, using the exact dictionary index, (ie non TH’d codes) yielded an additional 1-5% savings some compressible files, yet due to it’s nature, was unrecoverable in the decompressor. That led me to wonder if the problem at hand was in fact the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">codes</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> being used. As such, I tested the system using a few separate codes.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Fib3-</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Some digging yielded research showing </span><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.112.9459&amp;rep=rep1&amp;type=pdf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">fibonacci codes to be better compressors</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> than end-tagged-dense-codes:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We see that ETDC has a consistently longer average than the Fibonacci codes, at least for m ≤ 5 and up to alphabet sizes of 108. The additional overhead may reach 33% for alphabets as</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">small as 1000 elements, but even for N = 1000000, the overhead of ETDC relative</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">to Fib2, Fib3 and Fib4 is 1, 10 and 7%, respectively.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">However the comparison presented did not directly discuss if fibonacci codes would act as a better GZIP booster; we will test that later.</span></div> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GAC-</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Going further, there exists another paper showing that a type of </span><a href="https://en.wikipedia.org/wiki/Elias_gamma_coding" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Gamma Codes</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (which has been showing up in various video codecs lately) can </span><a href="http://www.ijcsit.com/docs/Volume%202/vol2issue2/ijcsit2011020229.pdf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">beat fibonacci codes</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in terms of compression size; however a disadvantage being that variable-bit codes may produce </span><a href="http://bmagic.sourceforge.net/dGap-gamma.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">worse performance in decoding on some platforms</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">; although </span><a href="http://fgiesen.wordpress.com/2011/01/19/a-small-note-on-the-elias-gamma-cod/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">RYG produces</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> a semi-fast version, it’s </span><a href="http://lemire.me/blog/archives/2012/09/12/fast-integer-compression-decoding-billions-of-integers-per-second/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">noted that byte-wise codes will decompress faster</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Direct index &amp; bitStream</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> This mode will write the direct index to the phrases list into the symbol stream, at the min byte stride required. (ie 1 byte less than 255, 2 bytes less than 65535). In order to recover this in the decoder, we include a bit stream which denotes what indexes are 8bit / 16 bit</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here’s the results, all using the same </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RegEx Word tokenization</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, all tests were done with gzip -9. Smallest sizes are in </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bold</span></div> <b id="docs-internal-guid-528045eb-bffb-9a5b-d077-5a0b093d52ad" style="font-weight: normal;"><br /><span style="background-color: transparent; color: red; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="110"></col><col width="78"></col><col width="82"></col><col width="99"></col><col width="84"></col><col width="86"></col><col width="85"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src / gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ETDC / gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">SCDC / gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <a href="http://comjnl.oxfordjournals.org/content/53/6/701.short" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">FIB3</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> / gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <a href="http://www.ijcsit.com/docs/Volume%202/vol2issue2/ijcsit2011020229.pdf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">GAC</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> / gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">DIBS/gz</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bootstrap.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">97,452</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">43,596</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">40,937</span><span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">40,041</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">38,224</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">44,960</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: right;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,540</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,510</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,440</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">24,028</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">24,855</span><span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16,579</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">prototype.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">180,828</span><span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">81,524</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">78,598</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">77,411</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">80,916</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">84,846</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: right;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">40,317</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">43,786</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">43,704</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">52,567</span><span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">55,518</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">45,724</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">patton.txt</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">8,715</span><span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7,899</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7,634</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">6,870</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">6,655</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7,632</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: right;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3,857</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4,786</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4,785</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5,138</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5,220</span><span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5,032</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">videos.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">206,467 </span><span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">107,357</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">104,760</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">95,632</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">97,754</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">111,340</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: right;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">60,761 </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">61,768</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">61,768</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">68,431</span><span style="background-color: #fff2cc; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">72,248</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">65,587</span></div> </td></tr> </tbody></table> </div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">From this small test, you can see that FIB3 and GAC codes do, infact, create a smaller encoded file format than ETDC / SCDC, however they fails in all 4 cases to produce a smaller post gzipp’d file result.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To be fair, there’s a </span><a href="http://books.google.com/books/about/Variable_Length_Codes_for_Data_Compressi.html?id=81AfzW6vux4C" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">plethora of various binary codes</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to be tested; if the code type is the problem, I estimate that there may be files which benefit differently from different codes. Using a booster in production may mean testing an exhaustive search over the code types to see which benefit a file the best.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <b style="font-weight: normal;"><br /></b> <b style="font-weight: normal;"><br /></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Pair-Wise correlation</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <img border="0" height="99" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCLHUjgUdOpYntFMruF9-8wRX0apvf8U9HHUkTo6phLn3wyCtMFyobyJ-s7Z_2PHQtAX8Wgu9XNog5XT4h9YYIS7D3xy2U090uYYLMd_ETJmazC4aYBbmOx6VSqDUWuCmYcGmNT8BYsnA/s320/adorable_animal_pairs_02.jpg" style="float: left; margin: 15px;" width="132" /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If we assume that the tokenization is optimal, as well as the byte codes are optimal, then this suggests that a file may not allow ETDC to be a booster due to the fact the post-tokenized stream does not contain enough pair-wise correlation to be compressed with further compressors. (Ignoring, for a moment, that this data relationship is a direct result of the tokenization process... but more on that later..</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In order to test this theory, I refer to some recent </span><a href="https://www.usenix.org/system/files/conference/fast13/fast13-final38.pdf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Research from IBM</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which discusses a nice set of heuristics which gauge how compressible a piece of content is, most useful is a test determining a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">pairwise distance</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (called PWD) to random:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><img height="101px;" src="https://lh4.googleusercontent.com/qgVbijZr_VVOnLzsYgVvqCCm4_qmf_RfNVeDq3_w38F0UC6K1X4gVyO1tnv2qzg4BjxPi1PgTcARHiecBM1GVk0cET1jupe0t1OZbQsE3fQGyGaiVBKBVZmE" width="601px;" />*<span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Effectively, PWD is a the measure of how frequent the pairing of symbol AB is, when compared to the frequency of each symbol individually, against a random distribution of symbols. For example, consider this measure of entropy vs pairs-distance:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">entropy</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PWD</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">aaaaaaaaaaaaaa</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">aaaaaaabbbbbbb</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.0069</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ababababababab</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">45.562</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">string.letters</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5.700</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.033</span></div> </td></tr> <tr style="height: 4px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">string.letters + string.letters</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5.700</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.152</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">string.digits</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3.321</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.1476</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Effectively, you can see that as the grouping of pairs occur more in a string, then PWD gets higher, which, run over a number of tests, shows a direct correlation to </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">compressibility.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> You can see this best demonstrated in the fact that duplicating string.letters (a-zA-Z) in the source stream yielded no change to entropy, yet more than doubled the PWD, because the number of duplicate pair-wise symbols in the stream had increased. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">According to my tests, PWD is a pretty accurate test against how </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">compressible</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> something is, way better than </span><a href="https://en.wikipedia.org/wiki/Entropy" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Entropy</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, or </span><a href="http://en.wikipedia.org/wiki/Kolmogorov_complexity" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Kolmogorov</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> measurements. In the graph below, I took a set of random files, compressed them with GZIP, and measured them against PWD. The graph is sorted by PWD, and trend-lines have been added for each of the data sets.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><img height="235px;" src="https://lh5.googleusercontent.com/YPK7qpSkE7VkksX_LJB7_54FH-O2qvWaWOUUawGUP5fgNmH7q5uuAacOGlayH6i07xkte9V_BKZRTQmA3UwUd5-sEUlXvX1z4OWXeGjHIj4O__zE1dAwvWnE" width="608px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note, the blue PWD line is in fact, increasing along the left-hand side of the graph, the particular resolution makes it appear as though they are all 0.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We can further prove the worth of this measurement with a few simple hand-based changes to a source string. For instance:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="457"></col><col width="85"></col><col width="82"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PWD</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">entropy</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">sally_sells_sea_shells_by_the_sea_shore</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.1141</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3.0756</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">_sQRally_sQRells_sQRea_sQRheQRlls_by_theQR_sQRea_sQRhore</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.4882</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3.3336</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">_sQRSTally_sQRSTells_sQRSTea_sQRSTheQRlls_by_theQR_sQRSTea_sQRSThore</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.5883</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3.5940</span></div> </td></tr> </tbody></table> </div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Effectively, what we’ve done here is inserted two </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">new</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> symbols into the stream, first QR and then ST attached to the most common pair in the string “_s”. You can see that this changes the PWD score such that a more pairs in a stream relate to it being more compressible.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">After we have a dictionary encode stream, we can run the pair-wise-distance (PWD) test on the data to determine how compressible it is, as an estimation. Here are some results of this test: (note, these tests were taken with </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">SCDC codes</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">RegEx Word tokenization</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">)</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src.dist</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">voc.dist</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">stream.dist</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">SCDC.dist</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">savings?</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bootstrap.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3.9665</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.0523</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.0782</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.0782</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fce5cd; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Yes</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">protottype.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5.3993</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.7968</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.0042</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.0233</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">No</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">patton.txt</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.4231</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.0712</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.0231</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.0221</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">No</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">videos.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2.5114</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 0.1164</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.0622</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.0478</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">No</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Testing this on a larger set of 1000 files we find that as a general rule, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">post-encoded streams with a SCDC.dist &gt;= 0.07 generally produce boosting in compression to this technique </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(ignoring some outliers which i have no idea about..).</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Which means that the reason the ETDCs fail for compressing most things lies in the fact that the symbol stream has a higher entropy, or rather, there’s not a great deal of pair-wise correlation; The stream is just less compressible.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="138px;" src="https://lh5.googleusercontent.com/s6fHe_K-qD9sijSwgs1WtA-8mL696ZXbcT6s0PoG28LzsagVQeZBlNGoMFYqqZG7giJxCXcnTMhNbKpjTtLN_VSRDF7kQdXcKhr1IktG--ETrjBImCv4Aiqu" width="697px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Bonus Round - Other compressors?</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To be fair, when I started running my original tests on ETDC, if found a load of savings for the majority of files; However, my tests were run against </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">winzip</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, instead of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">gzip</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Switching over to a different compressor yielded different results. Just to confirm an exhaustive search, I ran some files, post ETDC against </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">other</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> compressors to see how things fared. Below is a chosen sample from my 40 file set:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">etdc.src</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">gzip</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">zopfli</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7zip</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bzip2</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">twitter.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">17741</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">49604</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">19758</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">19037</span></div> </td><td style="background-color: #ffd966; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">17399</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">19594</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">amazon.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7994</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">18148</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">8059</span></div> </td><td style="background-color: #fff2cc; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7846</span></div> </td><td style="background-color: #ffd966; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7824</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">8072</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">dyn14.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">40247</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">65712</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">40949</span></div> </td><td style="background-color: #fff2cc; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">38946</span></div> </td><td style="background-color: #ffd966; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">37790</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">41270</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">patton.txt</span></div> </td><td style="background-color: #ffd966; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3854</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">6229</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4444</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4122</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4291</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4560</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">normalize.css</span></div> </td><td style="background-color: #ffd966; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2177</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4025</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2502</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2366</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2477</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2719</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">pintrest_ducks.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">19928</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">91080</span></div> </td><td style="background-color: #fff2cc; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">17870</span></div> </td><td style="background-color: #fff2cc; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">17173</span></div> </td><td style="background-color: #ffd966; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15113</span></div> </td><td style="background-color: #fff2cc; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">17343</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There were a couple things that came out of this test; I will summarize the results for you here:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1)</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>N</b></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ot a single file thats under 20k showed savings with ETDC against any processors; Files over 40k consistently saw savings from at least one compressor.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Which puts a lower limit on our data, our files must be greater than 10k in size to get a chance to see savings from ETDC, and over 40k to see consistent results. This make sense, given the correlation of pair-wise distances would grow as more symbols are introduced</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2) Zopfli and 7zip were able to produce savings w/ ETDC where gzip failed to do so.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> This means to make ETDC a viable technique, you should use zopfli or 7zip to generate your .gzip files</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3) BZip and 7zip were able to produce savings w/ ETDC where gzip/zopfli failed to do so.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Since these compressors have a different setup and processing suite, it appears that we could find edge cases where these files produced savings for us, where gzip-based formats were unable to.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Conclusions</span></h1> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Based on all above, it appears that the original savings presented in the ETDC paper left out a great deal of important information. My intent was to validate dense codes as a valid pre-processor for text data (natural-language and not), and that research proved the topic to be tricky.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here’s the things we learned:</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Tokenization (also known as dictionary creation) process presented in the original paper, and provided source code is only applicable for space tokenized, natural-language text.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Generated token-streams do not contain enough word-based pair-wise distance from random (&gt;0.07). This is a product of the word-frequency of the source stream, post tokenization. The frequency of adjacent word-pairs have a big impact on how successful ETDC is. Sadly for non-natural-language text, this data is at the direct mercy of the tokenization process.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Tokenization process presented in the original paper(and source code) relies heavily on the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">removal of spaces</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> from the source stream. Minimized JS / CSS have most spaces-data already removed from the information; thus using this metric as a tokenization scheme may result in less than viable results.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ETDC and SCDC will </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">not</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> work on files under 10k (evar!). It’s best to apply to files over 40k just to ensure you’re getting valid savings.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Savings for ETDC can be significant. For example, pintrest_ducks.json above saw a savings of 25% over GZIP alone.</span></div> </li> </ol> <b style="font-weight: normal;"><br /></b><span style="font-family: Arial;"><span style="font-size: 15px; line-height: 17px; white-space: pre-wrap;">I have an additional hypothesis that once files get above 40k, then the particular tokenization scheme produces enough redundant data in the </span></span><i style="font-family: Arial; font-size: 15px; line-height: 17px; white-space: pre-wrap;">vocabulary</i><span style="font-family: Arial;"><span style="font-size: 15px; line-height: 17px; white-space: pre-wrap;"> data section that GZIP has </span></span><i style="font-family: Arial; font-size: 15px; line-height: 17px; white-space: pre-wrap;">something</i><span style="font-family: Arial;"><span style="font-size: 15px; line-height: 17px; white-space: pre-wrap;"> to compress, resulting in savings. If this is the case, you really do wonder how much input the tokenization scheme has on the creation of the vocab, and how minimal we can get the stream as a result. (I'll look into this when I get some more time ;)</span></span><br /> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In general, text boosting should become an active area of research moving forward. ETDC has some serious caveats to it that make it a bit tricky to work with, but if you’re sending around </span><a href="http://www.unrealengine.com/html5/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">25mb javascript</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> data, and extra 25% savings is </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">huge</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="font-family: 'Trebuchet MS'; font-size: 21px; font-weight: normal; vertical-align: baseline; white-space: pre-wrap;">Source code</span></h1> You can get the full source code to this technique, as well as all the papers from the <a href="http://vios.dc.fi.udc.es/codes/index.html" target="_blank">authors' website</a>.<br /> <br /> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-1197137313991855034.post-6429430026221299112013-08-20T06:25:00.000-07:002013-08-20T06:25:05.972-07:00CSS Compression : Property Name Enumeration<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: left;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As I talked about in </span><a href="http://mainroach.blogspot.com/2013/07/css-compression-binary-css-format.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CSS Binary Format</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, it seems that the CSS properties are quite redundant for most of the web; It would be great if we could somehow encode that data, w/o needing to list those string values in the compressed stream itself.</span></div> <br /><div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSe4OQCWHC7zAkI0O9u7JM8gMBX2WwGqoyjIA3A51pmIdayw_zPlCiNSXIIfDvs-KoiCVmZne-KiNevtK7Ht4DbOUrjd48FpdGIua9ZaootRIbSz0wYwUgmtEDXpH8gH40Y5CzZRECqrE/s1600/csswordcoloud.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="355" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSe4OQCWHC7zAkI0O9u7JM8gMBX2WwGqoyjIA3A51pmIdayw_zPlCiNSXIIfDvs-KoiCVmZne-KiNevtK7Ht4DbOUrjd48FpdGIua9ZaootRIbSz0wYwUgmtEDXpH8gH40Y5CzZRECqrE/s640/csswordcoloud.jpg" width="640" /></a></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" />You see, when creating </span><a href="http://en.wikipedia.org/wiki/Dictionary_coder" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">dictionary compression</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> algorithms, you typically need to send the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">dictionary</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> along with the</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><a href="https://en.wikipedia.org/wiki/Bitstream" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">byte stream</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> where each element of the stream contains a reference to an element in the dictionary. For some cases where the dictionary data is large enough to be encumbering, then large wins can come from figuring out how to remove the dictionary words directly from the stream, and somehow move it into the </span><a href="https://en.wikipedia.org/wiki/Codec" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">codec</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <b id="docs-internal-guid-5b022dc8-9be2-9afe-4113-81ff6a8118f7" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As it turns out, there’s a subset of CSS text-definitions that exists in the browser, that you can recover @ runtime, which could allow us to remove that data from the compressed file.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This means that we do not need to include those source strings in the file, rather we encode our file expecting that the indices to the property names point to the table we generate from the browser data.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Getting the property names on the client</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Luckily for us, calling </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">document.body.style</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> from Javascript will give you a dump of all the CSS properties that the browser supports, and after a little data-modification, can be reconstructed. (the data given back to you is not format-exact with CSS standards for some reason...)</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">for(var i in document.body.style) {</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;for p in props:</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;string = ""</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for l in p:</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if l.isupper():</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;string += "-" + l.lower()</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else:</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;string += l</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">}</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To get this working, I replace the property name in the file with a string-ized index into the array provided from </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">document.body.style</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This is easy to identify in the file, as most of them come immediately after a semicolon. How you save this information to disk is up to you, and your workflow.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Encoding the file, offline</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This process is pretty straight forward. Once we have the property names for Chrome, (or whatever browser) we can parse the input CSS file, and simply replace any CSS property names in the file with a numeric lookup, coupled with some token value to allow us to find this information later.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">for i in range(len(propTables[1])):</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;symb = propTables[1][i]</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;dat = dat.replace(symb,"^" + str(i))</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Results</span></h2> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="151"></col><col width="97"></col><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src(bytes)</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src.zip</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">pne</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">pne.zip</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1k.css</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">157</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">107</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">127</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">97</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4k.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3,330</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,068</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2,929</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,005</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15k.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">14,905</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3,016</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">12,912</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2,900</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">96k.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">97,452</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,537</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">85,087</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,046</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">219k.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">223,865</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">42,769</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">189,360</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">41,562</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You can see that this method easily saves us +10% for basic compression. &nbsp;What's nice is that we seem to</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b> keep an additional 2%-4% of GZIP compression post-transform</b></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">; Which means this is a great technique to add to your stack. Or rather, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">this is the first technique I’ve presented which acts as a text compression booster. </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I hypothesize this is due to the property names in a CSS file contributing in the minority of entropy in the file, and so removing it minimizes the file without accurately modifying the entropy. More on that later though..</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <br /></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <img height="120px;" src="https://lh6.googleusercontent.com/wdZblZ1EtQMnDFTnEB9ryyav2jRDzbhFahqQSl3QWyCdPxE4I1pe3SkTAt_-Be9t55AuxS1dgK_U7Ieph3xwXh0BNn6HE85nZwM1VQ2vBTe8amMiEy24NV8E" width="586px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></h1> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Recovering on the client side</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Firstly, i want to be clear that this particular technique requires you to</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> reconstruct the CSS file, post GZIP decompression, before handing it off to the DOM</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. For many developers, this doesn’t make sense, simply because their site loads fast enough, however for web-app developers, who may already have a site which needs heavy CSS fixes, this may be a viable path.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The decompress stage of this algorithm that occurs on the client side will have to do string-replacement / concatenations; this can be fast, or slow depending on your implementation.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Our goal is to effectively replace any “^#” tokens with the original value. To do this quickly, we :</span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tokenize the string via the “^” character</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For each token, read until you hit the “:” character</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">parseInt(substring) is the index into the client CSS list.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">outstring += doc.styles[idx] += token[colonSpot:len]</span></div> </li> </ol> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I was shocked at how fast this was, turns out that most of the web browsers are optimized for string concatenation over string-joining nowadays. You can see the results in various </span><a href="http://jsperf.com/string-concatenation/28" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">jsPerf tests</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> or </span><a href="http://jsperf.com/string-concatenation/52" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">here</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, or </span><a href="http://jsperf.com/string-concat-vs-array-join-10000/15" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">here again</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">From the book </span><a href="http://oreilly.com/server-administration/excerpts/even-faster-websites/writing-efficient-javascript.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Writing Efficient JavaScript</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: Chapter 7 :€“ Even Faster Websites</span></div> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Browser string optimizations have changed the string concatenation picture.</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Firefox was the first browser to optimize string concatenation. Beginning with version 1.0, the array technique is actually slower than using the plus operator in all cases. Other browsers have also optimized string concatenation, so Safari, Opera, Chrome, and Internet Explorer 8 also show better performance using the plus operator. Internet Explorer prior to version 8 didn’t have such an optimization, and so the array technique is always faster than the plus operator.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Of course there’s a whole separate discussion about the fastest way to get it into the dom, which I recommend using document.fragment based on </span><a href="http://jsperf.com/concat-vs-docfrag/4" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">these tests</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This works... kinda...</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">THE LARGEST DOWNSIDE to this technique, however, is that each browser's </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">document.body.style</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is different. Meaning this ties our dictionary creation to a particular rev/browser depending on how variant the result of that line of code is.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To use this technique, you need to issue different files for different browser-listing of css support. And then, as updates came out, you’d need to push a new build of the CSS to the servers.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This single issue generally kills this technique for most web-devs; there’s just too much gardening involved.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">How often do the property lists changes?</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For the sake of argument, let’s figure out </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">how often</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> CSS properties have changed in a measurable way for web browsers. Thankfully Chrome keeps a </span><a href="http://build.chromium.org/p/chromium.chrome/console" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">running list of it’s historic builds</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and considering that they run on 6 week cycles, we have the ability to test, and chart how often CSS values have changed in the past 2 years.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><img height="288px;" src="https://lh6.googleusercontent.com/FsqzZBbUYp9qAYAXM6IUrCvCIjcxQ32ERUEAx2MWOYJcl5-qhHAP6SsofkWoO8wmGrFXBeT8ggzdSI02WZl0cJd8tXVLIzVuB-957th2N5fn--2wID7zQcOP" width="537px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I was able to chart builds of Chart builds of chrome for every 3 months over the last year, You can see the large spike over 2012 as the </span><a href="https://en.wikipedia.org/wiki/Cascading_Style_Sheets#CSS_3" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CSS3 features</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> started to land in Chrome. Oddly we’ve seen a taper-off over the last few months, which I find interesting in itself; Perhaps this has to do with blink? or maybe my way of measuring CSS values is flawed with the module-based setup of </span><a href="https://en.wikipedia.org/wiki/Cascading_Style_Sheets#CSS_4" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CSS4</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">?</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Is this worth it for you?</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I’m not really sure? </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It might be worth considering the fact that you also have to reconstruct the css before passing it off to the DOM for evaluation and propagation, which could be a deal-breaker for most CSS files that are on the fast-path for ‘pixel to screen’ time. However for web-apps, this may be a win; especially considering some web-apps have 250k css files that they send down to the client.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It’s interesting to note that removing the property names for lots of files still leaves entropy in such a state that we can gain compression from GZIP on it; </span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Perhaps this fits the bill better for your mobile-specific sites? These tend to have less variance between browsers (so fewer unique CSS files to use), and usually a separate content layout, which makes more sense to attach to.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Although we are fighting an issue here, in that we can send down a smaller file to mobile, but then caching becomes an issue. That is, every time the web app boots, we need to decompress the file again from cache, which is less than ideal.</span></div> <br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You can find the source code to this experiment on my <a href="https://github.com/mainroach/compression/tree/master/css-propname-elim">github</a> account.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com5tag:blogger.com,1999:blog-1197137313991855034.post-74039987313562799772013-08-13T07:14:00.000-07:002013-08-13T07:28:59.547-07:00JSON Compression : Transpose & Binary<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">JSON data comprises a large majority of content sent around the internet, especially for social networking sites and HTML5 games. Looking at most of these types of data, it’s apparent there’s a structure layout that we can take advantage of to boost GZIP compression.</span></div> <b id="docs-internal-guid-1c7282f5-7804-fa5e-eecb-b0716c9b2ec4" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Are JSON sizes a problem?</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It’s been very clear that the latest reports from </span><a href="http://httparchive.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HTTPArchive </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">show that the web is mostly images, JS, and CSS files. But knowing how </span><a href="http://httparchive.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HTTPArchive </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">tests work, we have to assume that there’s a large amount not accounted for. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The amount of JSON data sent around the web each day remains largely unmeasured, HTTP Archive </span><a href="http://httparchive.org/trends.php#bytesJS&amp;reqJS" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">gets a general sense of it</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, that is, if a home-page receives a block of JSON, which has been transferred with a mime type that is grouped with JavaScript, then it’ll be reported there. However </span><a href="http://httparchive.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HTTPA </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">doesn’t include any non-landing-page URLs, which typically is where the bulk of JSON transfer occurs, nor does it test the amount of JSON sent to native applications every day either.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tracking it down could be easier, if all developers used the right mime-type, but as mentioned, this is not the case, as developers can easily send around JSON data as application/JavaScript or application/text, so that’s pretty hard to do.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <div class="separator" style="clear: both; text-align: center;"> </div> <br /> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCOndkFRWYwO7cVGySWI0qmAQ1avAQAu9Fg2Ck-G7nXhO7Rlw_WGoNa20J0joz5S1fZ6Uvvjs7K6ZUmpxT-vh2Xf2eCW6NZUKmWNf4DSLuPdixNrSLz1sDZR5KXSKExoiAicdnrh_mz4g/s1600/TransposeDone.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCOndkFRWYwO7cVGySWI0qmAQ1avAQAu9Fg2Ck-G7nXhO7Rlw_WGoNa20J0joz5S1fZ6Uvvjs7K6ZUmpxT-vh2Xf2eCW6NZUKmWNf4DSLuPdixNrSLz1sDZR5KXSKExoiAicdnrh_mz4g/s1600/TransposeDone.png" /></a></div> <br /></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The need to minimize JSON </span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">JSON has been adopted as a format for data transfer because, among other reasons, it is usually more compact than alternatives such as XML. It also matches the object literal syntax in JavaScript, which has had a great result of parsing support has found its way into many languages, besides JavaScript. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Even with its savings over XML, </span><a href="http://www.json.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">JSON</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> can still be considered a horribly bloated format for a number of reasons.</span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">All data must be converted to text-based encoding</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Excessive usage of quotes for property names</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">When multiple objects are serialized in the same message, the key names for each property must be repeated, even though they are the same for each object.</span></div> </li> </ol> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">With the movement towards the client-side asynchronous programming models (like </span><a href="http://en.wikipedia.org/wiki/Ajax_(programming)" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">AJAX</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">), many web applications rely on JSON transport between client and server for client-side display. </span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The client requests some block of data</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The server provides data to the client in the form of a JSON blob</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Client receives JSON, and uses it to format / display the information on the screen</span></div> </li> </ol> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You’re seeing more and more of the web work this way, including popular sites like &nbsp;</span><a href="http://pinterest.com/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Pinterest</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> who sends the client this </span><a href="https://github.com/mainroach/compression/blob/master/data-sets/json/pintrest.json" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">JSON blob</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> when you do a search for </span><a href="http://pinterest.com/search/pins/?q=ducks" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">‘ducks’</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In any case, if your application makes heavy use of JSON, then you should consider what it’s costing you, and figure out how to minimize that cost.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: red; font-family: Calibri; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Uses of JSON</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In general, JSON has three primary use cases, each of which have their own compression needs.</span><br /> <div class="separator" style="clear: both; text-align: center;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbIjLD0qKzmIAkPS-UirLP1b0VdvPZZS073vg4bIN1bkIEfYnX-F45zpV0OMTbrwgMXbj2b5-78HtcMcaCVB5kId3RhjM-HGqJKniSnDjMQJ9_7E9R9gOEEUOdj-ZO6buq5-77y3xm9vs/s1600/List-Clip1.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbIjLD0qKzmIAkPS-UirLP1b0VdvPZZS073vg4bIN1bkIEfYnX-F45zpV0OMTbrwgMXbj2b5-78HtcMcaCVB5kId3RhjM-HGqJKniSnDjMQJ9_7E9R9gOEEUOdj-ZO6buq5-77y3xm9vs/s200/List-Clip1.png" width="160" /></a></span></div> </div> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Statically built server-owned JSON data</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">These types of JSON files are typically bundled with the website, often generated from build systems.They may be used to describe a file system format, or hold templating data to send to the client. HTML5 games make heavy use of this type of data to describe their context information for maps &amp; game stats.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">These files are low-hanging-fruit for compression, as you can spend lots of offline time computing a minimal representation, allowing for the fastest distribution time to the client.</span></div> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Dynamically server-built JSON data</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For web applications that resemble storefronts, or social-media information exchange, a client object typically queries a server, asks for the results of some database operation, and receives the results in JSON form.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Typically, this type of JSON content is critical-path. While this content is being constructed, a user is generally watching a “loading” cursor of some kind. As such, &nbsp;you may need to evaluate if it’s worth compressing the data, vs just sending it to the user quickly.</span></div> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Dynamically client-built JSON data</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In many cases, the browser will send JSON information back to some server, in which case compression can be useful when the JSON consists of complex data structures.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Web games are another great user of this type of information, sending player stats, and player actions to the server, resulting in multiple megabytes of data. In that direction, using automated GZIP compression is tricky, you’ll need to include a JavaScript GZIP compressor, whose size may invalidate the savings you get from the compression you’ll perform.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Each use-case has its own specific needs and problems that need solving, but in general, we can assume some basic operational costs and reduction needs can be applied between all of them.</span></div> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Transposing JSON</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It’s apparent that there’s lots of redundancy in JSON data, especially in highly formatted text (for example, the result of a </span><a href="https://twitter.com/search?q=%23perfmatters&amp;src=typd" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Search &nbsp;for "#perfmatters" on Twitter</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">), where there are large collections of dictionary objects, each one listing the property name and value.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Effectively, a JSON object is made up of </span><a href="https://en.wikipedia.org/wiki/Attribute%E2%80%93value_pair" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">key-value pairs</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, where the “key” portion is repeated for each instance of the structure in the file, adding bloat to the file. When you have an array object, which lists a set of dictionary objects, the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">name</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of each property must be re-listed in the file for each </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">instance</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of that dictionary. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">…</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">{</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;“name”: “alex”,</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;“pos”: “AUS”,</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">},</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">{</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;“name”: “Colt”,</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;“pos”: “USA”,</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">}</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For large JSON files, which list many elements in this form, the amount of overhead per each instance of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“name”</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“pos”</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> contributes a great deal to the final byte-size.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If we group together all the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">values</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for each instance of the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“name”</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> property, and list them in an array, we reduce the instances of the “name” key down to one, as shown in the trivial example below</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">{</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;“name”: [“alex”, “colt”],</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;“pos”: [“AUS”, “USA”],</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">}</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">or a better diagram of the process:</span><img height="398px;" src="https://lh5.googleusercontent.com/3a-DkkU49DCWfHPWqKmpkVDwO4HvJ5SN31Q9TzZHt6Tp6KXDr0QQ4901xOeZaAOUt74mZjTNtPGj44JlR6tN5YjlCZoq111jj1LKi6-6K4TAzifJZeSwNK6Y" width="619px;" /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">How it works</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To accomplish this, we search the JSON object to find any array object whose member objects are </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">only</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> dictionaries. Once we find this root object, we can easily transpose it.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note that this process only works if all dictionary objects in the array have </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">exactly the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">same</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> set of property names</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, otherwise, we end up in an awkward position that we need to keep track of </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">what</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> elements had </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">what</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> properties somewhere in our data file which will bloat our results and complicate our decoder.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Recovering a transpose</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It’s worth noting that you can modify your client-side javascript to operate directly on this transposed form of the data. This would be ideal since recovering from the transpose will take time, and extra memory to do so. If, however, updating your source code to work with the format is not an option, you’ll need to reverse the transpose once the JSON reaches destination. </span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To accomplish this, the encoder can attached an “IS TRANSPOSED” flag to each object that has been transformed, which on the client, we can find quickly and reverse the effects of.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note, allowing your processed JSON data to be recoverable will decrease the compression savings you get from the &nbsp;XFJSON process. &nbsp;Working directly with the transformed data on the client can be a large win in terms of processing time (since you don’t need to waste cycles to decode it).</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Transpose results</span></h2> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">transposed</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">transposed.gz</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">amazon.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">586,750</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">54,559</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">586,742</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">55,230</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">simple.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,087</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">402</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">460</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">344</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">twitter.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">100,605</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">17,756</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">100,966</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">17,952</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">videos.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">206,467</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">60,762</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">177,164</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">58,513</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">pintrest.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">12,449 </span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,776</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">12,327</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,762</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>The best gains from XFJSON come from larger JSON blocks with lots of repeated dictionary objects. Below that, and you can see that the post-gzip’d data is inflated.</b></span><span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Going further with binary.</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRjj2uKdIExs8UlMkrdVPbdtjoLDFTCCtPwF9M8viW-MRsejY1P_Hn4DJZEruVXHN_uXBNznXRo8upHyUNIi3_9hHVf3BAkxAk1BN5kxaJJLeaDimROscD043stQYr4qy6O8K90FU6QYU/s1600/Jimmy_Kuehnle_Jimmy_in_Binary_binary_drawing_detail.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRjj2uKdIExs8UlMkrdVPbdtjoLDFTCCtPwF9M8viW-MRsejY1P_Hn4DJZEruVXHN_uXBNznXRo8upHyUNIi3_9hHVf3BAkxAk1BN5kxaJJLeaDimROscD043stQYr4qy6O8K90FU6QYU/s200/Jimmy_Kuehnle_Jimmy_in_Binary_binary_drawing_detail.jpg" width="200" /></a></div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There’s work to </span><a href="https://github.com/getify/JSON.minify" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">minimize JSON</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, or rather, </span><a href="http://bitsquid.blogspot.com/2009/10/simplified-json-notation.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">re-represent</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> it in a separate, </span><a href="http://mjtemplate.org/examples/rison.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">less verbose format</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">; Most of these alternatives work for their desired use case; they will require some client-side reconstruction of the data from their source format back to valid JSON.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Binary forms make sense once the data inside your file can be represented with less bits, if it were represented as a number, instead of a stream. For example, the string “3.141592653589793” takes </span><a href="http://www.hunlock.com/blogs/The_Complete_Javascript_Number_Reference" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">only 8 bytes of memory</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> if represented as a number, but in a JSON file, where numeric values are listed as human-readable strings, it would be 17 bytes long (or longer depending on your character encoding).</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For this reason, there exist open binary JSON formats like </span><a href="http://bsonspec.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">BSON </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and </span><a href="http://msgpack.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">MSGPACK</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which both validly and accurately pack JSON data into binary format for smaller delivery. These two formats contain more options and verbosity of encoding types, which may or may-not be needed for your application.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In order to be a single-file solution, XFJSON includes a simplistic binary-packer, so that you can get an idea of what types of savings can come from the transform (you can explore the savings and changes of the other formats if you find validity here). The XJFSON binary transform is quite simplistic, using the format types described by the Python JSON module, which makes it quite easy to determine if </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3.141592653</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is a floating point value, or a string, but ignores more advanced types like </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">DateTime </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Null</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: red; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This is a simplistic format, much like the one I used in my </span><a href="http://mainroach.blogspot.com/2013/07/css-compression-binary-css-format.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Binary CSS format</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> article, so I’ll spare you the details here ;)</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">JSON ASCII - 59 bytes</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Binary-Hex form - 41 bytes</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">{</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;“name” : “html5rocks.com” </span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;“weight” : 1.7321232</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">}</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1a00 f102 0004 006e 616d 65b0 0e00 6874</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">6d6c 3572 6f63 6b73 2e63 6f6d 0600 7765</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">6967 6874 9037 b6dd 3f</span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Transform + binary Results:</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We plot some basic results of files in the XFJSON repository. You can see how each one stacks up against various methods of encoding, and also against GZIP .</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="109"></col><col width="84"></col><col width="74"></col><col width="*"></col><col width="103"></col><col width="75"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">transposed</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">transposed.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">trans &amp; bin</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">trans &amp; bin.gz</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">amazon.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">586,750</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">54,559</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">586,742</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">55,230</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">374,326</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">54,973</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">simple.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,087</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">402</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">460</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">344</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">434</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">384</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">twitter.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">100,605</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">17,756</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">100,966</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">17,952</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">94,829</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">20,000</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">videos.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">206,467</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">60,762</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">177,164</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">58,513</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">174,585</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">60,698</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">pintrest.json</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">12,449 </span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,776</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">12,327</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,762</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">11,399</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,836</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, binary didn’t help that much. In the few cases where the post-gzip’d binary file was smaller than the non binary file, we didn’t get as much of a savings vs zipping the source file. This isn’t a surprise, we’ve seen that GZIP does not like being handed binary data. It </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">hates </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">on that.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <img height="129px;" src="https://lh3.googleusercontent.com/5ddPBSoky2hwSAyPxAnwVa3goiSgyCGEm_k379FNbqVZ55CgSZa8V024JJZp33IQRkuz6ugt3yPOgBT0hKFqtdO3rldtZiT5oxJ34jxTWrLRfE26dy2mamYj" width="631px;" /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <br /></div> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Approaching the GZIP threshold.</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">By this time, you should all be enabling GZIP to distribute your assets to users, which means that you need to be aware that in some cases, transforming a file can reduce the amount of entropy (or the amount of compressible data) in a file, such that GZIP fails to compress it as well as the source text data.</span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This is referred to as the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">gzip threshold</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> , a point in which the compressed version of the modified file is larger than just compressing the source file. This demands your attention, as there’s even edge cases where crossing the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">gzip threshold</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> can actually inflate your transformed data before sending it across the wire.</span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">With this in mind, before embracing a sweeping adoption of a compression technology, make sure you test if the compression is going to be helpful </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">on a per-file basis</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and as always, optimize the places where you can have the most impact.</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Moving forward</span></h1> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As we’ve seen, transposing JSON data can yield significant results for highly-structured data, but you have to be careful to stay on the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">positive</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> side of the GZIP threshold, otherwise you’re just bloating your file, and wasting your time.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Converting your format to binary can still yield significant wins, you can really consider it a different form of minimization, where we’re tokenizing data, such that it’s recoverable. But once again, be careful to to remove too much, otherwise you’ll lose the GZIP game again.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If you’re building JSON heavy applications, then it’s worth checking out the source code </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Encoder </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">to this article, that is available on my </span><a href="https://github.com/mainroach/compression/tree/master/xfjson" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Github</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> account. The XFJSON module which will allow you to transpose, and binary pack your JSON data using python. (I will happily accept pull requests for a javascript version ;)</span></div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-1197137313991855034.post-47028772977528866342013-08-06T06:54:00.001-07:002013-08-06T08:22:40.612-07:00CSS Compression : Block Sorting<div> <span style="color: #222222; font-family: Arial;"><span style="font-size: 15px; line-height: 17px; white-space: pre-wrap;">This week, I decided to take a stab at a different approach to compression, and attempted to create a 'text compression booster' that is, a process to modify a file such that it gzip's smaller than the original file. The perfect target? CSS Blocks.</span></span></div> <div class="separator" style="clear: both; text-align: center;"> <br /></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhg4C52CS2XNf1InfLvaGPk1oyBC2aSB0sQ0FethuEavMsmJBxX3_ON8dnROl1EVaVGuC-RNslVYL9oJ5s2RD01AYdDqtixjUIiwlBfg_s2d70nwjDsbgMZ7kt0AS4xU1xzylezzy-U70U/s1600/2274621518_395a27e62d_z.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhg4C52CS2XNf1InfLvaGPk1oyBC2aSB0sQ0FethuEavMsmJBxX3_ON8dnROl1EVaVGuC-RNslVYL9oJ5s2RD01AYdDqtixjUIiwlBfg_s2d70nwjDsbgMZ7kt0AS4xU1xzylezzy-U70U/s320/2274621518_395a27e62d_z.jpg" width="320" /></a></div> <div class="separator" style="clear: both; text-align: center;"> </div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <br /></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To this end, it’s important to know that GZIP relies on a modified</span><a href="http://en.wikipedia.org/wiki/LZ77_and_LZ78" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"> LZ77 codec</span></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which employs a </span><a href="http://www.fearme.com/misc/alg/node175.html" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">sliding-window compression</span></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> system. This type of algorithm will consider the a local set of bytes in the stream, and search a set of bytes that are listed </span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">prior</span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to the target set in an attempt to find matches.This type of system exploits locality, or rather, it assumes that similar occurrences of a string tend to be clustered in a file. As you can imagine, the number of </span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">previous</span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> bytes you check allows you find more accurate, and longer strings. In fact, some of the more successful LZ77 encoders increase the window size close to available memory limits in order to find the best matches. </span></div> <b id="docs-internal-guid-34a94331-53de-1a51-606d-3d3667fef88b" style="font-weight: normal;"><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="https://en.wikipedia.org/wiki/Cascading_Style_Sheets" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CSS</span></a><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> can be described as a block-based format; or rather, there are logical </span><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">blocks</span><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of text (grouped with brackets) whose order can be independent of each other in the file, and who’s block-contents can be ordered independently as well. Despite some </span><a href="http://mainroach.blogspot.com/2013/07/css-compression-minifier-roulette.html" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">edge cases built for cascading</span></a><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, most CSS can get away with being arbitrarily re-organized and still have valid results.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Knowing this, it seems like a logical proposal that shuffling our CSS blocks around can yield increased compression, as more correlated symbols can exist in the sliding window.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Rather than being at the mercy of the sliding window, we explicitly re-position the blocks such that we should yield the smallest result.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A proposed algorithm</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The steps:</span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -9px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">parse a CSS file into memory, and tokenize by each property block </span></div> </li> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">block name and brackets included in a block</span></div> </li> </ol> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -9px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">sort by lowest entropy (or better, estimated bits required, from entropy)</span></div> </li> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -9px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">test-string = block[0]; indexlist[0] = 0</span></div> </li> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -9px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">while not all blocks processed</span></div> </li> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">for each block</span></div> </li> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-roman; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">newString = test-string + block[i]</span></div> </li> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-roman; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">len = LZ77.compress(newString)</span></div> </li> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-roman; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">if len &lt; minLen, </span></div> </li> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">mark this block as smallest</span></div> </li> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">indexlist.append(smallestBlockIndex)</span></div> </li> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">test-string += smallestBlock.text</span></div> </li> </ol> </ol> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: lower-alpha; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">remove smallest block from the option pool</span></div> </li> </ol> </ol> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" />Effectively, </span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">attempt</span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> each block against LZ77 to see which will yield the best results, encode that as the next block.</span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhq-OdxXOSgZFOQOkfRuO18P9yrCpwsZHxbyMQqZOq_B6eGLPJ_D7f5PSwqXcUZhYt1WzmWelj387HFNLfQuRZm30a4O-NFCm8mOSAS1GuKHn-w9P56MdGgiXwlamPKYBwM5tL4gypeU0w/s1600/SoC_bnr1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhq-OdxXOSgZFOQOkfRuO18P9yrCpwsZHxbyMQqZOq_B6eGLPJ_D7f5PSwqXcUZhYt1WzmWelj387HFNLfQuRZm30a4O-NFCm8mOSAS1GuKHn-w9P56MdGgiXwlamPKYBwM5tL4gypeU0w/s1600/SoC_bnr1.jpg" /></a></div> <div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="color: #222222; font-family: Arial;"><span style="font-size: 15px; line-height: 17px;"><br class="kix-line-break" /></span></span><span style="font-family: Trebuchet MS;"><span style="font-size: 21px; line-height: 1.15;">Horrible failure</span></span></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Testing a few CSS files showed that not only did this not produce savings, but in most cases, caused a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bloat</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">file</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">gzip’d</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">resorted gzip’d</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">dyn_1.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3,126</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3,145 </span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(growth of 0.6%)</span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bootstrap.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,543</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,594</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bs3.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,686</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,801</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">10.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; line-height: 15px; white-space: pre-wrap;">61,541</span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; line-height: 15px; white-space: pre-wrap;">61,948</span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To be fair, my LZ77 parser was less than optimal, I only used 10mb of memory for the search space, so there’s a chance that it was not producing the best measurable output against what GZIP would be doing.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What’s worse is that calculating the results of the single 6k file took about 53 minutes.....</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A faster measurement</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">By re-encoding the string each block </span><a href="https://en.wikipedia.org/wiki/Permutation" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">permutation</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, we’re effectively determining if the amount of saved bytes created by each sorted pair. As the string size increases, or the number of blocks increase our algorithm gets slower. &nbsp;Ideally, we would like a way to test how the compression would respond, considering the proposed new string without having to re-encode every possible combination.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To this measure, I found a statistical algorithm that seem to fit the bill: </span><a href="http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">KLDivergence</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, from <a href="http://www.wikipedia.org/">Wikipedia</a>:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">KL measures the expected number of extra bits required to </span><a href="http://en.wikipedia.org/wiki/Huffman_coding" style="text-decoration: none;"><span style="background-color: white; color: #0b0080; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">code</span></a><span style="background-color: white; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> samples from </span><span style="background-color: white; color: black; font-family: Arial; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">P</span><span style="background-color: white; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> when using a code based on </span><span style="background-color: white; color: black; font-family: Arial; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Q</span><span style="background-color: white; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, rather than using a code based on </span><span style="background-color: white; color: black; font-family: Arial; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">P</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Effectively, we can remove the actual LZ77 compression with a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">proposed</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> one that will run execute faster, with less memory, and give measurable results.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The good news is that </span><a href="https://code.google.com/p/tackbp2011/source/browse/TAC-KBP2011/src/python-utils/LDA/kullback-leibler-divergence.py?r=100" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">a small implementation</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, re-running the tests only took 12 minutes to complete on a 305K file, and that a comparison of the LZ77 brute-force shows about 2% difference in selection against these two algorithms, meaning that they are </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">accurate enough</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to warrant valid usage. (Or rather, using these metrics produce 98% of the same decisions that the brute-force method used)</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sadly though we still saw no savings in the original file size.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Source size (Gmail10.css)</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">gmail10.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">sorted_gmail10.gz</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">312,906</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">60,541</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">61,948 </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(growth of 2%)</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Even with &gt;10k files, this re-ordering yielded no observable savings to the final file. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Why no change?</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6jAzad1ukDQE-il60rKzUaqsZMxoURW1PfB8qKYFbZrLwin9tiD7W6fVZDi7pQF4nP-kd25OD_zW6m_Ymv2v5rqzWMIoi7kEP57vtFMsyi1ZEEWnEpnsNdQ2MFbndb-uj46SpNQ7Rfgo/s1600/sorry-no-change.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="163" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6jAzad1ukDQE-il60rKzUaqsZMxoURW1PfB8qKYFbZrLwin9tiD7W6fVZDi7pQF4nP-kd25OD_zW6m_Ymv2v5rqzWMIoi7kEP57vtFMsyi1ZEEWnEpnsNdQ2MFbndb-uj46SpNQ7Rfgo/s200/sorry-no-change.jpg" width="200" /></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Recall that GZIP comprises of two parts, LZ77 and Huffman encoders. LZ77 is a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">dictionary</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> algorithm; it’s built to reduce repetitions of characters to smaller tokens. Huffman is an </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">entropy</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> encoder, where it produces the minimal bit-string based upon a frequency of a token. </span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We’ve shuffled the CSS blocks around, which did nothing to change the entropy of the stream; the letter A still occurs just as frequent as it did before. Thus we don’t expect to get any savings from the <a href="http://en.wikipedia.org/wiki/Huffman_coding">huffman</a> stage.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Really, this type of reshuffling is targeted to hit the LZ77 phase, where we’d hope that grouped symbols would find more matches in the sliding window process; however it’s worth noting that the default window size for LZ77 is about 32K - 64k (depending on your extensions); Meaning we would need to use a larger file, with a large amount of moved blocks to see a savings (assuming that blocks in the window tend to be heterogeneous).</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Conclusions</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In the end, block-sorting our file did nothing to help GZIP. &nbsp;In fact, most times it produced bloat, which is a little scary for me. The Occam's-razor assumption is that my encoder is adding bloat somehow to the file, skewing results. The other possibility suggests that there is, in fact an optimal and sub-optimal layout for CSS, otherwise why would bloat occur?</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Thinking of this a different way, </span><a href="https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transform" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Burrows Wheeler Transform</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (or BWT) is another </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">block-based</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> transform that tends to be a critical component of most </span><a href="http://www.nanozip.net/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">cutting edge compressors</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> out there (not to mention the backbone of </span><a href="http://www.bzip.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">BZIP2</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which </span><a href="http://mainroach.blogspot.com/2013/07/the-state-of-web-compression_8.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">we’ve noted </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">consistently beats GZIP in size and speed); It works by transforming the string into a </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">semi sorted</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> representation, such that the data is more compressible given statistical encoders like </span><a href="http://en.wikipedia.org/wiki/Run-length_encoding" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">RLE</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and</span><a href="http://en.wikipedia.org/wiki/Arithmetic_coding" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"> Arithmetic compressors</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, but as noted by many papers, BWT does nothing to really help LZ77 or Huffman.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This matches my tests; Both block-based sorting methods failed to produce savings as a compression booster for GZIP.</span></div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-1197137313991855034.post-75413848339500539692013-07-30T08:14:00.002-07:002013-07-30T08:14:36.498-07:00CSS Compression : A binary CSS format<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The overly verbose nature of formats like JSON and XML have given rise to </span><a href="http://msgpack.org/" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">binary versions</span></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of their data in order to save transfer size, and are used by many large companies due to their savings. Although playing “Follow the leader” isn’t my style, I decided to try my hand at a binary version of CSS would be similarly successful.</span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhe_ZDdIAFudbbybFfeoFZoZb3vYlAR_fAiXnrO9vpqi1EHa3psRMjJrgV5n7eQskeFG6PfMIoNFmvYkoaEc_p45RcaalRxj6XFfALeNDyuw4IqOHgCZkp4qdmLQnJPAZGUBqJ6PtpiaPg/s1600/bin_css.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhe_ZDdIAFudbbybFfeoFZoZb3vYlAR_fAiXnrO9vpqi1EHa3psRMjJrgV5n7eQskeFG6PfMIoNFmvYkoaEc_p45RcaalRxj6XFfALeNDyuw4IqOHgCZkp4qdmLQnJPAZGUBqJ6PtpiaPg/s200/bin_css.jpg" width="200" /></a></div> <b id="docs-internal-guid-292c32fe-301e-ccc2-6d38-3dfba17344d1" style="font-weight: normal;"><br /></b> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If you simply look at the structure of a CSS file, it looks a great deal like structured </span><a href="http://www.hexblog.com/decompilation/pix/division.gif" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">assembly code</span></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, where each line defines an operation, and parameters to that operation.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">With this in mind, it becomes naively straightforward to view a CSS file as a series of assembly operations</span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Simplistically:</span></div> <div dir="ltr"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="624"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">h1 {</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> color:black;</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> margin:15px 8px 90px;</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">}</span></div> </td></tr> </tbody></table> </div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Can be re-represented as a set of ASM-ish instructions, alongside a variable table: </span></div> <div dir="ltr"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="624"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><variable table=""></variable></span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“h1”,”black”,”15px”, …..</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><instruction stream=""></instruction></span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><load> vtable[0]</load></span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><property id=""> vtable[1]</property></span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><property id=""> vtable[2] vtable[98] vtable[1542]</property></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">That is, we redefine a CSS Name:Value pair to look like an assembly instruction with multiple inputs. Since the inputs are raw text, we keep a dictionary of them, such that the input to our assembly-style instructions is simply a pointer into the variable table.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There are a few benefits of this format:</span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -9px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We represent input data as a dictionary, so multiple uses of the information are saved. </span></div> </li> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -9px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The dictionary-keys section of the file should compress well with a dictionary + statistical encoder</span></div> </li> <li dir="ltr" style="background-color: white; color: #222222; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; margin-left: -9px; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Common human-readable items, like spaces, semicolons and brackets are removed from the format by factor of its’ layout, which will reduce size more.</span></div> </li> </ol> <b style="font-weight: normal;"><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tokenization</span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="http://fun-n-stuff.com/images/tokens.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="160" src="http://fun-n-stuff.com/images/tokens.jpg" width="200" /></a></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Tokenizing is mostly the easy part of this process. For my simple tests, I used the python </span><a href="http://pythonhosted.org/tinycss/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">tinycss</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> module, which handled all the heavy lifting of parsing the data and assigning type context for me. The </span><a href="http://pythonhosted.org/tinycss/parsing.html#quickstart" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">tokenization</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is simplistic, once tinyCSS parses the data, I crawl through, property-by-property and output an Assembly-style string.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A nice thing about the tinycss module is that it allows me to get property type information, so I can tell the difference between a value, number, and a string. During parsing, I push these values into type-specific tables (e.g. all the strings go in one vtable, all the ints in another), because this reduces the overall size of indexes that I have to use and reference in the file.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" />As noted, the entire </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">point</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of the vtable is to remove redundancy from the file; if the value “10px” is already in the table, we can encode a pointer to it, multiple places in the stream</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Dealing with lots of indexes</span></div> <b style="font-weight: normal;"><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Ideally, each line in the file is a single list of byte values, where inputs point into the dictionary table. For larger files though, our variable table can easily exceed 255 items, so we can’t simplistically push a byte-per-index to our file, and it would be less than ideal to push the max value per index, instead it would be great to only write out the bits that we need.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="http://upload.wikimedia.org/wikipedia/commons/thumb/a/a2/Fibonacci.jpg/220px-Fibonacci.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="http://upload.wikimedia.org/wikipedia/commons/thumb/a/a2/Fibonacci.jpg/220px-Fibonacci.jpg" width="148" /></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This problem is not a new one for compression, although we’d love to just write bytes where the index is less than 255, and shorts where the index is less than 65536, we run into the problem of how to tell the two apart in our byte stream effectively. Most compression systems approach this through means of </span><a href="http://en.wikipedia.org/wiki/Universal_code_(data_compression)" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Universal Codes</span></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which have been around for quite some time. These are </span><a href="http://en.wikipedia.org/wiki/Variable-length_code" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">variable-length bit-codes</span></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> that contain uniqueness and length information as properties of the code. Because of these two features, they are easily parsible, and can be recovered properly (as opposed to a raw stream of binary data). They also tend to be </span><a href="http://en.wikipedia.org/wiki/Universal_code_(data_compression)#Relationship_to_practical_compression" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">simpler and faster to decode</span></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> than their cousin, Huffman codes. These features come with a cost however, in that Universal codes </span><a href="http://en.wikipedia.org/wiki/File:Fibonacci,_Elias_Gamma,_and_Elias_Delta_encoding_schemes.GIF" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">tend to be bloated compared</span></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to their direct binary counterparts. &nbsp;For my purposes, I used a </span><a href="http://comjnl.oxfordjournals.org/content/53/6/701.short" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Fib3</span></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> universal code, which when </span><a href="http://www.amazon.com/Lossless-Compression-Communications-Networking-Multimedia/dp/0126208611" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">compared to other codes</span></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, provides a good compromise between robustness and length. (Note, there are </span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">variable byte</span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> codes: </span><a href="http://u.cs.biu.ac.il/~tomi/Postscripts/MathComp.pdf" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Tagged Huffman</span></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and </span><a href="http://www.dcc.uchile.cl/~gnavarro/ps/spire04.3.pdf" style="text-decoration: none;"><span style="background-color: white; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Dense Codes</span></a><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, however I will cover those and their effectiveness in a later article)</span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To use these codes, we tokenize our input CSS data, and count the max number of variables, we then generate a table of Fib3 codes to cover that.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note, we separate out each type into their own bucket so that indexes remain small across types. This adds about 16bits of overhead per bucket we use, simply because we need to keep track of it.</span></div> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" />The format:</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here’s the proposed format, in all it’s raw form:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="174"></col><col width="450"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Header data</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1 byte</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Control data (numeric control, etc etc)</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16 bits</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">num Declaration strings</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16 bits</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">num Property strings</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16 bits</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">num value strings </span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16 bits</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">num value numbers</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Variable Tables</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">XX bits</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">declaration string table (strings written as [len(short)][chars(byte)])</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">XX bits</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">property string table </span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">XX bits</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">value strings table</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">XX bits</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">value numbers table </span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(values written based upon the numeric control above)</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Symbol Stream</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0xFFFF 0xXX</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">fcn will always be 0xFFFF, followed by a 8 bit pointer to the decl table</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0xXXXX 0xYY</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">property single input - </span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;16 bits pointing to the static property table, </span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;8 bits pointing to the variable table, variables are all offset</span></div> <br /> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0xXXXX 0xYY [0xZZ ..]</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">property multi input</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;16 bits pointing to the static property table, <span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span></span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;8 bits for number of inputs</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;Array of bytes pointing to variable table</span></div> <br /> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Results:</span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I encoded some CSS files of various sizes to see what savings we get, and then GZIP’d the data after encoding, to see how that fares. Smallest results are in </span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bold</span><span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src.gz</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">binary</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">binary.gz</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1k.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">157</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">109</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">98</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">126</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4k.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3,330</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,073</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2,349</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,430</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15k.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">14,905</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3,016</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">9,258</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4,375</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">96k.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">97,452</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,537</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">59,877</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">20,599</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">219k.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">223,865</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">42,769</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">113,465</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #222222; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">55,873</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So you can see that the Binary CSS format easily cuts the size of the file by 30%+, but sadly, fails to produce a post-gzipp’d file that’s smaller than just gzipping the source file. Once again, we’re running into the GZIP threshold. </span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The GZIP threshold.</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The point in which a modified version of the file fails to produce a smaller post-compressed result, when compared to a compressed version of the source file.</span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Or rather, once GZIP(MODIFIED(FILE)) &gt; GZIP(FILE) then You’ve gone too far.</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This made me question those other binary formats, after all, shouldn’t you be GZIP’ing those as well? After some searching, I found no mentions of how MSGPACK stands up to gzip compression; I’m guessing they are in the same boat; the binarization process proves to remove too much information, and GZIP falls over. Although I suppose the MSGPACK folks fall into a particular use-case here, namely that in some cases, constructing the static binary JSON data is faster than gzipping things, so you can construct a light version, send it to the client ASAP. Sadly, CSS doesn’t appear to fall into that category.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">NOTE : </span><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">TinyCSS module ignores some very important things, like @media etc. I’ve carefully selected files such that this would not play a factor into the calculation</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWWQ8-vdgKxXBkUMjXAx5rCYfNgbPaZjhLqYjp8w_grKrvJi80H1MVg9K5hyTGmHoTN2sAduP0OaroHs3aHD2Ln35moiJbgJlBNC1OMMXOL91_To-1_HWtG6Jk2VO8-lflhysl3Hk0BBY/s1600/banner.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWWQ8-vdgKxXBkUMjXAx5rCYfNgbPaZjhLqYjp8w_grKrvJi80H1MVg9K5hyTGmHoTN2sAduP0OaroHs3aHD2Ln35moiJbgJlBNC1OMMXOL91_To-1_HWtG6Jk2VO8-lflhysl3Hk0BBY/s1600/banner.jpg" /></a></div> <b style="font-weight: normal;"><br /><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">An extension, could we remove the variable table?</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Although variable tables are quite common, I wanted to take a swing at seeing if we could simply enumerate all the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>potential</b></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> property names and values by frequency, and add this data to the codec, rather than the encoded data stream.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To determine if this was feasible, I cached the </span><a href="http://httparchive.org/about.php" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HAR</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> files from the top 1000 sites listed in </span><a href="http://httparchive.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HTTPArchive</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and then downloaded all the referenced CSS files, and created a large histogram across all the CSS for property names and used values. (Note, this is almost identical to my previous post on </span><a href="http://mainroach.blogspot.com/2013/07/estimating-website-transfer-costs.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">calculating website costs with HAR files</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which has source code)</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here's what I found:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The top 10 properties</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here’s the top 10 used properties, the number of times they were used, and the estimated total number of bytes that character string occupies across our corpus.</span></div> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Name</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"># appearances</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">total # bytes</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">width</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">234,872</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,174,360</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">background</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">169,695</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,696,950</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">height</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">161,969</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">971,814</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">color</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">159,828</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">799,140</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">padding</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">146,744</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,027,208</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">background-position</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">133,610</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2,538,590</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">display</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">131,790</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">922,530</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">margin</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">124,105</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">744,630</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">font-size</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">121,260</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,091,340</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">float</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">105,938</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">529,690</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The top 5 values for the top 10 properties</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For the top properties, I felt it was interesting to highlight the top values used for those properties, and how often they are used across our corpus. </span></div> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="120"></col><col width="108"></col><col width="92"></col><col width="96"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Property</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">value 1</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">value 2</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">value 3</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">value 4</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">value 5</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">width</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">100%:20896</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">auto:7485</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16px:4079</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">300px:3841</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">20px:3084</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">background</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">no-repeat:</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">66284</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0:40576</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">transparent:17279</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">left:12322</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">#fff:10449</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">height</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">20px:7318</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">100%:6236</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">auto:6018</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16px:5576</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">30px:5067</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">color</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">#fff:18420</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">#000:10273</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">#333:9303</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">#666:8515</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">#999:5906</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">padding</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0:127117</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">10px:31633</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5px:27347</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2px:12600</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4px:12117</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">background-position</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0:60397</span><span style="font-family: Arial; font-size: 15px; line-height: 1; white-space: pre-wrap;">&nbsp;</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0px:7494</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">left:6142</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">right:4346</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">-32px:3717</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">display</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">block:64186</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">none:26438</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">inline-block:22158</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">inline:14182</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">table:1899</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">margin</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0:162625</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">10px:19991</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">auto:15200</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 5px:14898</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0px:12922</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">font-size</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">12px:19524</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">11px:16837</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">14px:11748</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> 13px:7724</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">10px:5663</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">float</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">left:75766</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">right:24194</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">none:5903</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">center:29</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">inherit:15</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What is truly amazing about this table, is that somewhere in the universe, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">a thousand web developers</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> have decided that </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">padding:10px</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is a better padding amount than </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4px</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Or that </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">float:left</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is used almost twice as much as </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">float:right.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The top 20 property values</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If we transpose the data, and just look at the most used property values, independent of the property itself, we find another interesting table: </span></div> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">absolute:23268</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">pointer:11257</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">transparent:10507</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">sans-serif:10489</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">left:9498</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Arial:7055</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">block:6203</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Helvetica:5552</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">right:5319</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0:4702</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bold:4671</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">solid:4547</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1px:4324</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">no-repeat:4279</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2px:4145</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">relative:4084</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">17px:3729</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">normal:3522</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">auto:3415</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">repeat-x:3402</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What this means</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Firstly, from my direct interest of determining if there’s a way to create a </span><a href="http://en.wikipedia.org/wiki/Dictionary_coder" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">static dictionary</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for the top 16k values, this test has shown that to be less than an option, unless we implemented it directly into the browser. Looking at just the unique values, the top 1k sites lists over 151,383 unique values, which would require 32 bits to represent in lookup table, and would need to include the unique values somewhere in the file to represent.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If we were attempting to simply make a javascript library which contains this type of static dictionary, then we’re going to lose more in the existence of the data in JS, than what we would save by removing it from the CSS files.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">And really, this is a skewed result for developers, directly. </span><a href="http://httparchive.org/interesting.php" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HTTPArchive lists that the average website in their test suite contains about 15 JavaScript files an 5 css files</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, meaning that even if we did enumerate each property name for CSS, we wouldn’t get much savings for this single developer; I mean IF you just count the characters, the top 10 property names, across 1000 websites only account for &nbsp;~11mb of traffic</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Effectively, this means that we need to put our variable data in the file, and point to it from each line in our binary format.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The code</span></h2> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You can find the </span><a href="https://github.com/mainroach/compression/tree/master/css-binary-fmt" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">encoder for this experiment</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> on my </span><a href="https://github.com/mainroach" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">github</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> account. Since the testing showed negative gains, I did not spend the time to write the decoder; I offer the code warranty-free as a proof of concept for anyone out there who’s got better ideas.</span></div> <div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-1197137313991855034.post-59405245220467268962013-07-26T09:58:00.002-07:002013-07-26T10:01:10.600-07:00CSS Compression : Genetic Minification<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I already looked at why most minifiers </span><a href="http://mainroach.blogspot.com/2013/07/css-compression-minifier-roulette.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">didn’t go all the way to the edge</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> when it comes to making huge changes to code, and when trying to write a minifier that intentionally broke ALL the rules, just to save things, I ran across this</span><a href="http://www.ryansb.com/2013/04/genetic-algorithms-and-css/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"> awesome piece of research</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which uses </span><a href="http://en.wikipedia.org/wiki/Genetic_algorithm" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Genetic Algorithms</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to minimize CSS files. </span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh84fk17s3n3mWPYtkJdWHf5cY3zCVDQP73iIpdoPO-xv25OwoiBcUqhawygG-LBpAaMmOd5VWZIdsniU3T-BKxodmFiOyB_9XAAb8TNBQUnlZ4SbaiK702LGM-Y2XB3deLQv_PiDr6gbQ/s1600/Collector2.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="242" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh84fk17s3n3mWPYtkJdWHf5cY3zCVDQP73iIpdoPO-xv25OwoiBcUqhawygG-LBpAaMmOd5VWZIdsniU3T-BKxodmFiOyB_9XAAb8TNBQUnlZ4SbaiK702LGM-Y2XB3deLQv_PiDr6gbQ/s320/Collector2.JPG" width="320" /></a></div> <b id="docs-internal-guid-6f699d5c-1be7-50d4-3055-70aeddfd1eeb" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div class="separator" style="clear: both; text-align: center;"> </div> <b style="font-weight: normal;"><br /></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The first implementation I found about </span><a href="http://xkcd.com/534/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">GA</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">s using to minimize CSS was from &nbsp;</span><a href="http://friggeri.net/blog/a-genetic-approach-to-css-compression/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Adrien Friggeri</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, who’s open source implementation was posted </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">9.12.2012</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. It seems that Ryan Brown </span><a href="https://github.com/ryansb/genetic-css/blob/master/css.py" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">copied </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">the source code </span><a href="https://github.com/afriggeri/css-bipartite-compression/blob/master/css.py" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">identically</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> from Adrien’s repo (after pushing a few </span><a href="https://github.com/afriggeri/css-bipartite-compression/commits/master/css.py" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">useless commits</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to it), and slapped a GNU license on it, posting to his repo on </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4.18.2013, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">and then presented the work, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">without credit to Adrien</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> @ </span><a href="http://pyvideo.org/video/1808/python-css-and-genetic-algorithms" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PyCon</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To my knowledge, Adrien’s minifier the first non-syntax-only based minifier in existence, which is pretty exciting stuff. One of the cool things about the CSS standard is that file-order is only needed in some cases; otherwise, you're free to reorder parts of the file as you need. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As Adrien describes it, a CSS file can be viewed as a graph of connections, in which can be re-interpreted as a structure for genetic minimization:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In terms of </span><a href="http://en.wikipedia.org/wiki/Graph_theory" style="text-decoration: none;"><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">graph</span></a><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, a rule can be viewed a complete bipartite graph – also called a biclique. </span><a href="http://en.wikipedia.org/wiki/Bipartite_graph" style="text-decoration: none;"><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Bipartite graphs</span></a><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> are a specific kind of graphs which nodes can be decomposed into two disjoints sets such that two nodes in a same set are not connected, therefore those graphs are usually written </span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">B</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">=(</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">L</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">,</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">R</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">,</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">E</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">), where </span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">L</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">,</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">R</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (left/right) are the disjoint sets of nodes and </span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">E</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> the edges. In this context, a</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">complete</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> bipartite graph is a bipartite graph where every node in one set are connected to every node in the other set.</span><span style="background-color: transparent; color: black; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">or for the more human of us</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Using such a representation allows us to formalize our initial compression problem: compressing a stylesheet be rewriting/reordering rules boils down to finding a set of bicliques which </span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">covers</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> the whole bipartite graph, such that the</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“cost”</span><span style="background-color: transparent; color: #333333; font-family: 'Helvetica Neue'; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of that covering is minimal.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The inner loop of the process boils down to this:</span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Randomly mutate members of the population</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Test all the members of the population against a fitness function</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Keep the ‘most fit’ member of the environment stationery for this next round</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Repeat</span></div> </li> </ol> <div class="separator" style="clear: both; text-align: center;"> <br /></div> <div class="separator" style="clear: both; text-align: center;"> <br /></div> <div class="separator" style="clear: both; text-align: center;"> </div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWWQ8-vdgKxXBkUMjXAx5rCYfNgbPaZjhLqYjp8w_grKrvJi80H1MVg9K5hyTGmHoTN2sAduP0OaroHs3aHD2Ln35moiJbgJlBNC1OMMXOL91_To-1_HWtG6Jk2VO8-lflhysl3Hk0BBY/s1600/banner.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWWQ8-vdgKxXBkUMjXAx5rCYfNgbPaZjhLqYjp8w_grKrvJi80H1MVg9K5hyTGmHoTN2sAduP0OaroHs3aHD2Ln35moiJbgJlBNC1OMMXOL91_To-1_HWtG6Jk2VO8-lflhysl3Hk0BBY/s1600/banner.jpg" /></a></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The problem with being awesome.</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Firstly, this research is groundbreaking, and should set up lots of other people to investigate the same things in the future.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sadly though, this algorithm is </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">so efficient</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in it’s process that it removes a huge amount of entropy and duplicate symbols from the file, that it passes the GZIP threshold. That is, the point in which a modified version of the file fails to produce a SMALLER post-compressed result, when compared to a simple compression of the original file.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The GZIP threshold.</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The point in which a modified version of the file fails to produce a smaller post-compressed result, when compared to a compressed version of the source file.</span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Or rather, once GZIP(MODIFIED(FILE)) &gt; GZIP(FILE) then You’ve gone too far.</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You can see confirmation of this with some of the example files, run through various compressors :</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="177"></col><col width="*"></col><col width="135"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">src</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">gzip</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bzip2</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">LZMA</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bootstrap.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">97,452</span></div> </td><td style="background-color: #d9ead3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,837</span></div> </td><td style="background-color: #d9ead3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">13,144</span></div> </td><td style="background-color: #d9ead3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">13,725</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bootstrap-gbc.css</span></div> </td><td style="background-color: #d9ead3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">95,002</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">18,188</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">13,707</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16,066</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">friggeri.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">18,712</span></div> </td><td style="background-color: #d9ead3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4,022</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3,868</span></div> </td><td style="background-color: #d9ead3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3,770</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">friggeri-gbc.css</span></div> </td><td style="background-color: #d9ead3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16,663</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4,257</span></div> </td><td style="background-color: #d9ead3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3,860</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4,176</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">stackoverflow.css</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">90,881</span></div> </td><td style="background-color: #d9ead3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">18,877</span></div> </td><td style="background-color: #d9ead3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,720</span></div> </td><td style="background-color: #d9ead3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">17,045</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">stackoverflow-gbc.css</span></div> </td><td style="background-color: #d9ead3; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">88,565</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">20,174</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">15,733</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">18,204</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Dammit Jim.</span></h1> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Ran this across ~200 files, saw the same growth types. Which is really saddening. I even tried a variation, where I attempted to bail-out early if the savings crossed the GZIP threshold (assuming that there was a point in which savings did occur, and we simply needed to avoid crossing into negative savings) </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sadly, these tests proved invalid; One simple generation of minimization caused a growth in the post-gzip’d size.</span></div> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Closing thoughts</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This process proves that GZIP is really touchy, I mean </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">really goddamn touchy</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. The graph above shows that 200 bytes removed from a CSS file in this process produced a negative savings, and as </span><a href="http://www.phpied.com/css-minifiers-comparison/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">we’ve already seen</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> that the smallest minimized file doesn’t always produce the smallest GZip’d data.&nbsp;</span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This scares me.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">How many files on the internet are being inflated by GZip?</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I feel there’s needed research addressing this type of problem; Some heuristics modifications are occurring which are causing the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">types of changes</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> that minifiers and this GA algo is making to cause worse GZIP performance. This demands more of my attention.</span></div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-1197137313991855034.post-43732192793089924702013-07-23T10:46:00.002-07:002013-07-23T10:46:38.575-07:00CSS Compression : Minifier Roulette<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I’ve always wondered why there’s a plethora of </span><a href="http://designshack.net/articles/css/18-css-compression-tools-and-techniques/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CSS minifiers out there</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, it seems the one place where the “open web” just creates confusion for most developers. &nbsp;You would think that by now, CSS minification would be a solved problem, and that the only variance between CSS minifiers would be reduced to the minimization performance and how well it integrates into your build process. </span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="color: black; font-family: Arial; font-weight: bold; vertical-align: baseline;"><br /></span></span> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="color: black; font-family: Arial; font-weight: bold; vertical-align: baseline;">It turns out that this is far from reality.</span><span style="color: black; font-family: Arial; vertical-align: baseline;"> </span></span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUFD2BfpJNV-Ibnm-giDIPAEwugpyWY0yi84PVCC1wRBmCQIyZBoVNEcWZ0jt_BVf2y_7vZ0-BJ6J1YUEELKowTUpBPQ5uuB0mHcWYIUPvcBatseUEhWt8L7BTj31mWYY6cAq6cSn-sME/s1600/HD_picture_of_orginal_roulette.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUFD2BfpJNV-Ibnm-giDIPAEwugpyWY0yi84PVCC1wRBmCQIyZBoVNEcWZ0jt_BVf2y_7vZ0-BJ6J1YUEELKowTUpBPQ5uuB0mHcWYIUPvcBatseUEhWt8L7BTj31mWYY6cAq6cSn-sME/s400/HD_picture_of_orginal_roulette.jpg" width="400" /></a></div> <b id="docs-internal-guid-0b2e91bb-0c99-794a-fb43-8883cafe8fb4" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">CSS minification does not appear to be a solved, or even agreed-upon subject: There is already a </span><a href="https://www.google.com/search?q=css+minifier&amp;oq=css+minifier" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">plethora</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of CSS minifiers out there, and some developers are even resorting to </span><a href="http://lesscss.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">other versions</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of </span><a href="http://sass-lang.com/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CSS languages</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to help author and generate more effective CSS code. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What’s </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">really</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> scary is that each one seems to have a different set of features, and huge variation in minimization results. For example, taking a look at these three tests [</span><a href="http://www.codenothing.com/benchmarks/css-compressor-3.0/full.html" style="text-decoration: none;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">1</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;</span><a href="http://www.adamkoch.com/2010/01/28/comparison-of-10-css-compressors/" style="text-decoration: none;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">2</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;</span><a href="http://www.phpied.com/css-minifiers-comparison/" style="text-decoration: none;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">3</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> ] comparing the top minifiers, they can’t even agree on the same feature set, output size or corpus of testing files.. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Considering that CSS is a semi-small </span><a href="http://www.w3.org/Style/CSS/Overview.en.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">set of known combinations</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, it seems like our understanding of minimization of that data shouldn’t be so disjoint.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Why there’s differences</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I did a quick </span><a href="https://twitter.com/ChromiumDev" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">around-the-office</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> survey of why a developer chose one CSS minifier over the other, and mostly it came down to:</span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It fits in my existing work flow</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It doesn’t break anything</span></div> </li> </ol> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The first option really doesn’t have much bearing to why minifier have different results; Your source CSS data should get the same treatment regardless of what language or package it comes in. It should make your site more awesome regardless of if it’s </span><a href="http://jekyllrb.com/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">jeykll</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://yeoman.io/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">yeoman</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> or </span><a href="http://twitter.github.io/bootstrap/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">bootstrap</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It turns out that the 2nd one then has the biggest impact on things. </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Most devs use a CSS minifier that’s </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">easy</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to use, and gives them </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">good enough</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> results that they can measure and doesn’t break anything</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Note, this is quite different than </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">the most optimal </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">output.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Directly, most developers want to optimizations to be automatic part of their tool chain that gives them savings, without having to think too much; If something breaks, that requires thinking. That’s why the more aggressive CSS minifiers (which give the best savings) are often not used in most production pipelines: they can break things.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What kind of breaks</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">But let’s take a step back; What types of dangerous breaks are there, and what types of savings to they provide?</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In order to get a better sense of things, here’s a quick </span><a href="http://www.adamkoch.com/2010/01/28/comparison-of-10-css-compressors/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">breakdown</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of the types of optimizations a CSS minifier will make, and how safe / unsafe they are.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Safe</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Semi-Safe</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Not Safe</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Removes comments</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Removes whitespace</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Removes empty blocks</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Removes decimals</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Removes units from zero values</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Removes quotes from URIs</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Reduces margin, padding, border-width</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Reduces hex colors</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Swaps colors by hex or name, whichever is shorter</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Converts RGB to hex</span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Removes duplicate rules</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Removes overwritten rules</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Reduces font weight</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Changes color names to lowercase</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Changes color hex values to lowercase</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Converts tag names to lowercase</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Combines styles for the same element</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Combines font styles</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Combines margins, paddings, borders</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Combines same styles from different elements</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Some of the semi-safe changes seem odd, but make sense. For example, changing color names and hex colors to lower-case, will reduce the number of unique symbols seen in the stream, which changes the entropy of the stream when going through GZIP compression.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Some of the ‘Not safe” optimizations are explained over at </span><a href="http://www.lotterypost.com/css-compress.aspx" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">lotterypost</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, where the dominant reason for avoiding them is that it changes how the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">cascade</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> of properties is applied to a sheet. For example:</span></div> <b id="docs-internal-guid-0b2e91bb-0c9a-2463-2a06-6056f4264f2a" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="310"></col><col width="314"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Combine individual </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">margin</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">padding</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, or </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">border styles</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> into a single property.</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Why Not?</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">margin-top: 10px;</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">margin-right: 0;</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">margin-bottom: 8px;</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">margin-left: 30px;</span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Becomes</span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">margin: 10px 0 8px 30px;</span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The "C" in CSS is extremely important: it stands for "Cascading", and means that the style rules are applied (and re-applied) in the order they appear in the page and in all the page's external CSS files. &nbsp;</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Changing the order of the styles can easily disrupt the cascade</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and thus change the entire layout and/or style — and that is the problem with combining individual margin, padding, and/or border styles: it re-writes to order of the cascade, because rarely are they all specified perfectly together like in the example.</span></div> </td></tr> </tbody></table> </div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What’s quite interesting is that the </span><a href="http://www.lotterypost.com/css-compress.aspx" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">lotterypost</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, (as well as many other minifiers ) </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">exclude</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> these optimizations entirely, simply because it </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">could</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">break </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">someone’s</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">css.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I view this as a </span><a href="http://en.wikipedia.org/wiki/Throw_out_the_baby_with_the_bath_water" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">throw-the-baby-out-with-the-bathwater</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> methodology; There could be savings, if you’re willing to compromise. For example, the above example reduces the size of byte from 77 to 24 (a 31% savings). </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">IF</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> this combination of cascades won't matter to you, then this change seems like a reasonable adaptation to make. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What if we broke stuff, the right way.</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This is where I think most developers are missing the boat;</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> You don’t want the “Always Safest” minifier, you want the “Smallest producing minifier that breaks things </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">that your site doesn’t use</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.”</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Which turns out to be a big difference. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If we want to go to the extreme, we may find great savings by compromising, and building a CSS minifier chain that works this way, requires two things:</span></div> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A minifier which will detect Safe, Semi-Safe and Not Safe optimizations</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A feedback loop letting you know how those changes break your site.</span></div> </li> </ol> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Detecting unsafe optimizations</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You’ll find some <a href="https://github.com/mainroach/compression/tree/master/css-danger-min">example code</a> on my <a href="https://github.com/mainroach/compression">github</a> account which will parse a given CSS block, and identify multiple </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">semi-safe</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">unsafe </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">targets for potential optimization. Effectively, it uses </span><a href="http://www.glazman.org/JSCSSP/freshmeat.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">jsCSSP</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to parse the CSS block in JavaScript, and then per-block, identify if it’s a target for potential savings. For about 3 hours worth of work, I could semi-accurately identify over 500 potential optimizations for a CSS file from </span><a href="https://plus.google.com/u/0/+ColtMcAnlis/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Google Plus</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">; Most coming in the form of :</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.U-L-Ba button:active</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">{-webkit-box-shadow:0 1px 1px rgba(0,0,0,0.1);</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">box-shadow:0 1px 1px rgba(0,0,0,0.1);</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">background-color:#fff;</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">background-image:-webkit-linear-gradient(top,transparent,transparent);</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">background-image:linear-gradient(top,transparent,transparent);</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">border:1px solid #b8b8b8;color:#404040;</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">-webkit-box-shadow:inset 0 2px 0 #ddd;</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">box-shadow:inset 0 2px 0 #ddd</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">}</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So yes, we can accurately find that this CSS rule is re-setting the background-image property. Weather you should </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">remove</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> this or not, brings up a whole separate discussion about how non-standardization can hurt us just as much, considering that we’re not sending down more bytes on the wire...</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Detecting how much we broke things</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As noted, most web developers follow the path of least resistance towards CSS minimization. This reasons to believe that most developers care more about the automation process than the savings involved (as long as the savings are semi-effective). It would be great then, if we could embrace some of these changes AND get feedback in an automated fashion to determine how breaks occur, and how dangerous they are.</span></div> <div class="separator" style="clear: both; text-align: right;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwPdheubxu209zsAqw9jHdSPbmn6iAENk9z834hDeBwXcku6lv5enHK7-AY6WAhshKYZJu3YLuTeXT2qWTYsRoU6rqYitbUXR8fI3JXrsDUBkLIGCI00a5SBkt-WQIe9CrmBUs28M686Y/s1600/6811f34e-e190-42b6-9b3e-b290424808d2.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="150" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwPdheubxu209zsAqw9jHdSPbmn6iAENk9z834hDeBwXcku6lv5enHK7-AY6WAhshKYZJu3YLuTeXT2qWTYsRoU6rqYitbUXR8fI3JXrsDUBkLIGCI00a5SBkt-WQIe9CrmBUs28M686Y/s200/6811f34e-e190-42b6-9b3e-b290424808d2.jpg" width="200" /></a></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At the core, we have basic systems like </span><a href="http://jigsaw.w3.org/css-validator/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CSS validator</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and </span><a href="http://csslint.net/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CSS Lint</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> testing, which will at least let you know that your code is </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">compliant</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to standards. Any CSS minifier should always produce valid CSS as a core feature. The real test comes in determining if you’re compliant </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">syntactically</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, but broken </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">visually</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in which case you’d need a setup described in “</span><a href="http://tldr.huddle.com/blog/css-testing/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Css Regression Testing</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">” which points to a great new tool called </span><a href="https://github.com/Huddle/PhantomCSS" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PhantomCSS</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">; a tool which will let you take screenshots of your page after CSS changes, and determine how many pixels have been modified.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Of course there’s that whole “Even though it’s a web standard it’s not really standardized across all browsers” thing. Which could complicate things even more, considering that you may be syntactically complaint against the standard, but visually non-compliant in a specific browser-revision, simply because of implementation issues around a subset of properties.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One Minifier to rule them all?</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There’s a separate debate about how useful this type of deep-control is. I mean, how many bytes are you </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">really</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> saving for all this extra brain-work &amp; integration? For the most performance, and cost conscious developers, I expect this type of control to be a viable, embraced solution to consider for their design pipeline. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For others, I’m sure this is just noise. </span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" />In any case, I recommend that you test this method out, determine if it works for you, and embrace some awesome.</span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWWQ8-vdgKxXBkUMjXAx5rCYfNgbPaZjhLqYjp8w_grKrvJi80H1MVg9K5hyTGmHoTN2sAduP0OaroHs3aHD2Ln35moiJbgJlBNC1OMMXOL91_To-1_HWtG6Jk2VO8-lflhysl3Hk0BBY/s1600/banner.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWWQ8-vdgKxXBkUMjXAx5rCYfNgbPaZjhLqYjp8w_grKrvJi80H1MVg9K5hyTGmHoTN2sAduP0OaroHs3aHD2Ln35moiJbgJlBNC1OMMXOL91_To-1_HWtG6Jk2VO8-lflhysl3Hk0BBY/s1600/banner.jpg" /></a></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">P.S. The GZIP threshold</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One of the things I love about </span><a href="http://www.phpied.com/css-minifiers-comparison/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">this test </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">in particular, is that it actually shows the savings of minification </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">post gzip</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This is an important metric, because as you can see, there are some cases where the encoder that produces the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">smallest</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> CSS file failed to produce the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">smallest</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> post-gzip’d data:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="72"></col><col width="129"></col><col width="138"></col><col width="127"></col><col width="147"></col></colgroup><tbody> <tr style="height: 0px;"><td style="background-color: #c1e56d; border: 1px solid #000000; padding: 5px 5px 5px 5px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"> <span style="background-color: #c1e56d; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">File</span><span style="background-color: #fefefe; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #c1e56d; border: 1px solid #000000; padding: 5px 5px 5px 5px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"> <span style="background-color: #c1e56d; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Original</span><span style="background-color: #fefefe; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #c1e56d; border: 1px solid #000000; padding: 5px 5px 5px 5px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"> <span style="background-color: #c1e56d; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">cssminjs</span><span style="background-color: #fefefe; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #c1e56d; border: 1px solid #000000; padding: 5px 5px 5px 5px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"> <span style="background-color: #c1e56d; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">CSSTidy-small</span><span style="background-color: #fefefe; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #c1e56d; border: 1px solid #000000; padding: 5px 5px 5px 5px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"> <span style="background-color: #c1e56d; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ajaxmin</span><span style="background-color: #fefefe; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt;"> <a href="http://csszengarden.com/001/001.css" style="text-decoration: none;"><span style="background-color: #fefefe; color: #3980c0; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">001.css</span></a></div> <br /> <span style="background-color: #fefefe; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fefefe; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4832b 100.00%</span></div> <br /> <span style="background-color: white; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fefefe; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2650b 54.84%</span><span style="background-color: #f5b973; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #f5b973; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #f5b973; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2586b &nbsp;&nbsp;53.52%</span><span style="background-color: white; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #fefefe; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2653b &nbsp;&nbsp;54.90%</span></div> <br /> <span style="background-color: white; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt; text-align: right;"> <span style="background-color: #fefefe; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">001.gz</span></div> <br /> <span style="background-color: #fefefe; color: #3980c0; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1781b &nbsp;&nbsp;36.86%</span><span style="background-color: #fefefe; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="background-color: #f5b973; border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: #f5b973; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">888b &nbsp;&nbsp;18.38%</span><span style="background-color: #fefefe; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">902b &nbsp;&nbsp;18.67%</span></div> <br /> <span style="background-color: #f5b973; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.7173295454545454; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: white; color: #333333; font-family: Verdana; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">892b &nbsp;&nbsp;18.46%</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This is what I call the “GZIP THRESHOLD” and you’ll see this term a great deal this summer.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><h1 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The GZIP threshold.</span></h1> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The point in which a modified version of the file fails to produce a smaller post-compressed result, when compared to a compressed version of the source file.</span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Or rather, once GZIP(MODIFIED(FILE)) &gt; GZIP(FILE) then You've gone too far.</span><span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 21px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> </td></tr> </tbody></table> </div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This makes me wonder, at it’s core, how many files on the internet have been naively minimized </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">past</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> the GZIP threshold, producing larger files than they need, and blindly sent around the internet...</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-1197137313991855034.post-35258551516354475922013-07-17T06:27:00.002-07:002013-07-17T06:42:07.683-07:00Estimating website transfer costs<br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <div class="separator" style="clear: both; text-align: center;"> </div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The feedback for “SUMMER OF COMPRESSION” has been pretty good so far, although with any endeavor, there’s a set of nay-sayers. </span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If most of the web is images, then why do we care about text compression?</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <b id="docs-internal-guid-7f7c6601-ce76-63be-8d77-0eb879b1525e" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Well, let’s ignore for a moment all the vague hand-waving discussions about how smaller-files influence load-times, which influence user experience, which influence revenue, and instead skip right to the revenue part. </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s figure out how to calculate the cost of distributing the text portion of websites.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <div class="separator" style="clear: both; text-align: center;"> </div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div class="separator" style="clear: both; text-align: center;"> </div> <div class="separator" style="clear: both; text-align: center;"> </div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVIhLK9JQYbi0Yw5N1xUdW2dPwCWBHolrAhWYWFTVO_q-SfLZOYB7XA3RIpbj8VvFXB6DoY8y5lR9iCt6-TljCkjzgAZhdZhY05mvYzx_lICML8HiegJhtCA1IbkcDubvrnEnVB4U2nXQ/s1600/Baby-Sitting-Holding-Money-Shocked.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVIhLK9JQYbi0Yw5N1xUdW2dPwCWBHolrAhWYWFTVO_q-SfLZOYB7XA3RIpbj8VvFXB6DoY8y5lR9iCt6-TljCkjzgAZhdZhY05mvYzx_lICML8HiegJhtCA1IbkcDubvrnEnVB4U2nXQ/s320/Baby-Sitting-Holding-Money-Shocked.jpg" width="320" /></a></div> <b style="font-weight: normal;"><br /></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Calculating Costs</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In order to calculate your costs, you need a few numbers.</span></div> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Size of your site, in bytes.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cost to transfer your site to a single user.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Visitors per day (ie traffic).</span></div> </li> </ul> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">How to measure your site size</span></h2> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <b id="docs-internal-guid-7f7c6601-eccd-9d13-c76e-692fb5a5e2cb" style="font-weight: normal;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At tjer core, most websites are composed of Text, Images, and other junk. And thankfully </span><a href="http://httparchive.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HTTP archive</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> has been doing a great job in reporting the breakdown and byte-content-sizes of assets on the web. The </span><a href="http://www.softwareishard.com/blog/har-12-spec/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HTTP Archive file</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> format, which basically tracks all inbound and outbound requests from a browser on page load, so that you can see what data is being asked for. There’s also a great resource at </span><a href="http://www.webpagetest.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Webpagetest.org</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which will show you your </span><a href="http://www.webpagetest.org/breakdown.php?test=130711_W7_d44f849315ee7eeb160a2913b1ce9257&amp;run=2&amp;cached=0" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">content breakdown</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, but doesn’t separate between what’s been gzipp’d and what hasn’t been.</span></div> <br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></h2> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Generating HAR files</span></h3> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <b style="font-weight: normal;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Generating a HAR file is pretty easy, for example Chrome will allow you to </span><a href="https://support.zendesk.com/entries/22914301-Generating-a-HAR-file-for-troubleshooting" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">simply right-click on the network tab to download the page content in a HAR format</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, or if you’re doing debugging on a mobile device, you can also use </span><a href="https://github.com/cyrus-and/chrome-har-capturer" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Chrome to grab HAR traces from remote instances</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, allowing you to even grab them from a mobile instance of Chrome.</span></div> <br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If, however, you’d like a more automated process to test content distribution costs, then you may need to use </span><a href="http://phantomjs.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PhantomJS</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which is essentially a headless browser instance (see this post on </span><a href="http://www.igvita.com/2012/08/28/web-performance-power-tool-http-archive-har/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">how to use it</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">). If you’re super-lucky, and you’re targeting a site which is extremely popular, you can just </span><a href="http://httparchive.org/viewsite.php?pageid=9304633#downloads" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">grab an existing HAR file from HTTP Archive</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></h2> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">An issue with HAR files</span></h3> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <b style="font-weight: normal;"><ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Missing Content - </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Not every HAR capture is created equal, </span><a href="http://phantomjs.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PhantomJS </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">for instance, won’t list </span><a href="https://en.wikipedia.org/wiki/SWF" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">SWF </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">data.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Not Mobile Safe</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">-</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><a href="http://phantomjs.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PhantomJS</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><a href="http://httparchive.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HTTPArchive</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> use a desktop </span><a href="http://en.wikipedia.org/wiki/User_agent" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">UA</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> string, so the report data is not for mobile devices. You’ll need to use the </span><a href="https://developers.google.com/chrome-developer-tools/docs/remote-debugging" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">chrome-remote-debugging</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> hacks to gather than data.</span></div> </li> <li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: decimal; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Skewed Content</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">-</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Most of these grabbers will only capture assets until the ‘</span><a href="http://www.w3schools.com/jsref/event_onload.asp" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">page load</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">’ event fires. So any content after that event, or any lazy-load content, appear to be ignored. Some frameworks, like </span><a href="http://www.chromium.org/developers/telemetry" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Telemetry</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, will allow you to script loading a page, scrolling to the bottom, and then back up. With this type of setup, you would be able to get the TRUE cost of your full site.</span></div> </li> </ol> <br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></h2> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Getting the size of your site from HAR</span></h3> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <b style="font-weight: normal;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A HAR file will list out all the resources that have been requested from your site during its loading process. As such, you can simply scrape the HAR JSON data to determine the data types and sizes of assets requested from your page. This will give you a footprint size for a single instance of a visitor hitting your site.</span></div> <br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></h2> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWWQ8-vdgKxXBkUMjXAx5rCYfNgbPaZjhLqYjp8w_grKrvJi80H1MVg9K5hyTGmHoTN2sAduP0OaroHs3aHD2Ln35moiJbgJlBNC1OMMXOL91_To-1_HWtG6Jk2VO8-lflhysl3Hk0BBY/s1600/banner.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWWQ8-vdgKxXBkUMjXAx5rCYfNgbPaZjhLqYjp8w_grKrvJi80H1MVg9K5hyTGmHoTN2sAduP0OaroHs3aHD2Ln35moiJbgJlBNC1OMMXOL91_To-1_HWtG6Jk2VO8-lflhysl3Hk0BBY/s1600/banner.jpg" /></a></div> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Estimating traffic</span></h2> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <b style="font-weight: normal;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If you own your site, &nbsp;you have very detailed statistics. If not, you can look at sites like </span><a href="http://www.alexa.com/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">alexa.com</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which tracks some heuristics. (here’s the page for </span><a href="http://www.alexa.com/siteinfo/amazon.com" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">amazon.com</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">). Once you get the alexa number, you can use some estimated </span><a href="http://stackoverflow.com/questions/1485739/how-to-calculate-pageviews-of-a-site-using-alexa-stats-formula" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">basic heuristics</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to get a rough estimate of the number of users per-day</span></div> <br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b></h2> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Estimating costs</span></h2> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="font-weight: normal;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">It’s tough to know what exact distribution platform each site is being served off of, but you can make some estimates. For example, you can use the </span><a href="https://cloud.google.com/pricing/" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Google App Engine costs</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> to get a ballpark number.</span></div> <br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">From there, calculating the yearly-cost is is a pretty straightforward equation. Once we have cost-per-user, and users-per-day, we can refactor those two numbers into whatever type of billing system you have. App Engine, for example, charges per GB/mo, which allows us to bin our general cost per-month to estimate an overall cost.</span></div> </span></h2> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">General equation:</span></div> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">monthly cost = (num_users_month * bytes_per_user) / 1gb * cost_to_xfer_1gb</span></div> </td></tr> </tbody></table> </div> </span></h2> <div> <span style="font-weight: normal;"><br /></span></div> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">An Example</span></h2> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Amazon.com -</span></h3> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Est. # Users : 324,142,753</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Est. $ / gig : 0.12</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Assets: cost per year</span><br /> <b id="docs-internal-guid-7f7c6601-eccc-df12-7fb3-8da0d7d719ce" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">images</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">$12,008,564</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">908,198 bytes</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">compressed images</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">$39,825</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3,012 bytes</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>Text</b></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>$36,969</b></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>2796 bytes</b></span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>compressed text</b></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>$4,105,200</b></span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><b>310,473 bytes</b></span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">flash</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">$449,231</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">33,975 bytes</span></div> </td></tr> </tbody></table> </div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Est. Total size <span class="Apple-tab-span" style="white-space: pre;"> </span>: 1,258,454 bytes</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Est. Total cost <span class="Apple-tab-span" style="white-space: pre;"> </span>: $16,639,789</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, that’s still ~4 million a year to distribute GZIP’d text data </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">just for the homepage</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. A 10% reduction in text size would save ~$400k a year </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">just for the homepage.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> I’m sure anyone owning </span><a href="https://www.google.com/finance?q=NASDAQ:AMZN" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">amazon.com stock</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> would surely like an extra improvement.</span></div> <b id="docs-internal-guid-7f7c6601-ecca-a668-c78c-b056d0623a51" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The Caveats</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Firstly</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, this number only represents people </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">who hit amazon.com directly</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, it doesn’t take into account any sub-pages (amazon.com/blah.html for instance) and also ignores international domains (amazon.uk for instance). So, if you </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">hypothetically</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> consider that the traffic to a sub-page on amazon is 100% of the traffic to the homepage (since so much traffic comes to direct-product pages from google.com searches), then you could </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">hypothetically</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">assume that</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">each individual page of amazon.com costs about $4 million a year to transfer GZIP’d text</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.... Hypothetically.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Secondly</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, Of course the estimated users, are in fact </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">estimated</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><span style="font-family: Arial; font-size: 15px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">Thirdly</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, this estimate is obviously worst-case scenario. It would be great to factor in all the multi-tier caching that occurs for your resource all over the internet, but that’s difficult, if-not impossible to calculate properly. For a single user instance, &nbsp;Yahoo’s last report estimated that the number of users hitting your cached site is about </span><a href="http://yuiblog.com/blog/2007/01/04/performance-research-part-2/" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">about 50%</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> , while WebpageTest.org </span><a href="http://www.webpagetest.org/experiment/" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">recently noted</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> that ~75% of visitors on any given day are hitting the site w/o the cache being populated. So I’m sure the true-cost to distribute these assets are much lower than the estimated costs.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Thinking Mobile</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Up to this point, we’ve talked about how much it costs you, the developer, to send your page to visitors, however page-weight has a massive impact on load times and user retention, which also hurt your bottom line.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What’s worse is that </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">users have to pay to download your website</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Even most ‘unlimited’ data plans for mobile, </span><a href="http://news.cnet.com/8301-1035_3-57564716-94/unlimited-verizon-data-customers-beware-make-sure-your-next-phone-is-4g/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">aren’t truly ‘unlimited free.’</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Most of them will charge a flat rate for access up to 2GB or so, and passing that amount can cost more money. Not to mention that there’s many areas of the world w/o these types of ‘all you can eat’ billing plans, where cost-to-download information is a serious concern for users, as a </span><a href="http://www2.research.att.com/~rjana/MobEA-IV/PAPERS/MobEA_IV-Paper_7.pdf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">research paper by AT&amp;T</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> highlighted: </span></div> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">mobile data traffic cost problems are severely hindering the use of mobile services on handheld devices today</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">…</span></div> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It was interesting that users try to understand the billing rules even when the carrier does not provide this information. Based on connection indicators, information visible on the phone bill, and previous experience on data traffic billing, people create a perceived billing model that affects their mobile browsing usage patterns. </span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The folks over at </span><a href="http://mobiforge.com/designing/blog/performance-money-part-1-end-users-wallet" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">mobiforge</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> provided even more context, putting the entire argument in terms of costs:</span></div> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Vodafone Ireland charge </span><a href="http://www.vodafone.ie/planscosts/paymonthly/roaming/map/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">$7.96/MB</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, Vodafone UK charge over </span><a href="http://www.vodafone.co.uk/shop/pay-monthly/travelling-abroad/internet-abroad/index.htm" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">$1/MB</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for users roaming in the EU. AT&amp;T charge as much as </span><a href="http://www.wireless.att.com/learn/international/roaming/affordable-world-packages.jsp?wtSlotClick=1-006RPJ-0-1&amp;WT.svl=calltoaction&amp;locale=en_US#data" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">$19.97/MB</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for roaming data in certain countries. Taking some of the examples from Guy's </span><a href="http://www.guypo.com/wp-content/uploads/2013/03/RWD-perf-2013-results.xlsx" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">spreadsheet</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and using the AT&amp;T roaming tariff, is a peak at 1pictureaday.com really worth $178? Do you really need to visit thenextweb.com for $44 or vogue.co.uk for $65? At $17 microsoft.com is a relative bargain.</span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For roaming users these page weights are prohibitive for all but the most essential tasks.</span></div> <br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The code</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You can find source code to automate all this on my </span><a href="https://github.com/mainroach/compression/tree/master/est-page-cost" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">github</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> account.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; vertical-align: baseline; white-space: pre-wrap;"><b>fetchHAR.py</b></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> will fetch/generate a HAR file from either HTTP Archive, or PhantomJS</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; vertical-align: baseline; white-space: pre-wrap;"><b>calcHARCosts.py</b></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> will read in a specific HAR file, # of daily users, and estimated cost-per-gig to report to you a breakdown of how much the site costs to distribute.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Moving forward</span></h2> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Armed with the ability to estimate your cost, and the cost of random websites, you should be able to get a better understanding of how costly these operations are, based upon the size of your content that you’re transferring. Not just yourself, but also for your users.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I found this great quote in Eric Schmidt’s “<a href="http://www.amazon.com/The-New-Digital-Age-Reshaping/dp/0307957136">New Digital Age</a>” that I thought was quite relevant:</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There are already more than 650 million mobile phone users in Africa, and close to &nbsp;3 billion across Asia. The majority of these people are using basic-feature phones-voice calls and text messages only-because the cost of data service in their countries is often prohibitively expensive, so that even those who can buy web-enabled phones or smart phones cannot use them affordably. This will change, and when it does the smartphone revolution will profoundly benefit these populations.</span></div> </td></tr> </tbody></table> </div> <br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <br /> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com2tag:blogger.com,1999:blog-1197137313991855034.post-47633584301152599962013-07-08T11:25:00.000-07:002013-07-08T15:26:12.173-07:00The state of web compression<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCNU1iy2mfuarw1o44SA3WfSox7DW4h7hFuWCVTI0pSssWfW7Q_6uFfIDfs30rdaODsWIQb5DViYFcB8eN2p_lGU3KHl5hPEx6OKoVoeNcoArMyGVSDjGGXgXy1HWNbQTTNuNXsrmyFEo/s1600/banner.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCNU1iy2mfuarw1o44SA3WfSox7DW4h7hFuWCVTI0pSssWfW7Q_6uFfIDfs30rdaODsWIQb5DViYFcB8eN2p_lGU3KHl5hPEx6OKoVoeNcoArMyGVSDjGGXgXy1HWNbQTTNuNXsrmyFEo/s1600/banner.jpg" /></a></div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span id="goog_943749653"></span><span id="goog_943749654"></span><br /></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Most people pick a good set of books for summer reading, but true to my nature, I’ve decided to spend the summer researching Compression technologies and how it applies to the web. Compression has always had a special place in my heart, starting back with my research on </span><a href="http://www.gdcvault.com/play/1http://www.gdcvault.com/play/1277/HALO-WARS-The-Terrain-of277/HALO-WARS-The-Terrain-of" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Halo Wars</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, mostly focusing on </span><a href="http://www.youtube.com/watch?v=7bJ-D1xXEeg" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">texture compression</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, so it seems natural to spend my summer focusing on something which reminds me of the early days of programming.</span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <b id="docs-internal-guid-662d6002-bf82-5ab4-e940-051fb0a718c8" style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So let’s kick off with an interesting look at the state of compression on the web today.</span><br /> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The web today</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It’s been noted that the average website is, in fact, </span><a href="http://www.webperformancetoday.com/2013/06/05/web-page-growth-2010-2013/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">getting bigger</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, even with overall connectivity connected </span><a href="http://www.stevesouders.com/blog/2013/05/09/how-fast-are-we-going-now/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">speeds increasing</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, we see that mobile speeds are still averaging out @ </span><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><a href="http://stevesouders.com/images/akamai-mobile-conn-speed.png" style="text-decoration: none;">about 1.5Mbps (in the US)</a>.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We’re also moving into a new world of web-app information, where technologies like Emscripten allow C++ code to run inside the web-browser, but even the </span><a href="http://www.unrealengine.com/html5/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">UE3 Epic Citadel demo</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is ~25MB of javascript data alone (not to mention the 86mb of content that comes along with it)</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Ignoring the outliers, we can see the the largest increase in sizes come from </span><a href="http://httparchive.org/interesting.php#bytesperpage" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">images</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which can spell bad news for mobile, where a picture </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">literally</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><a href="http://www.slideshare.net/guypod/a-picture-costs-a-thousand-words18062013" style="text-decoration: none;">costs a thousand words</a>. </span><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The browsers of the web generally support &nbsp;</span><a href="http://en.wikipedia.org/wiki/PNG" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PNG</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://en.wikipedia.org/wiki/Graphics_Interchange_Format" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">GIF</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="https://en.wikipedia.org/wiki/JPEG" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">JPG</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, and the newly anointed </span><a href="https://developers.google.com/speed/webp/" style="line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">WebP</span></a><span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. Each one effectively optimizes lossy compression algorithms to minimize size; which equates to removing image quality for the sake of less bytes.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Text data, on the other hand, does not allow such lossy compression. If we remove even 2 bits of precision in ASCII, we no longer have readable text. The web has responded, in turn, realizing that most text data for the web is overly verbose. This birthed the entire </span><a href="http://en.wikipedia.org/wiki/Minification_(programming)" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">minimization craze</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which effectively reduces the amount of verbose, and overly redundant characters in a file, while still allowing it to conform with the file-type standardization process, and can be passed directly to the </span><a href="https://code.google.com/p/v8/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">underlying processing systems</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Minimization is not enough, obviously, since we the resulting data must still be human readable, in text format, which means we need to couple minimization with some sort of web-based compression.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What tools do we have?</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There are two commonly used </span><a href="http://en.wikipedia.org/wiki/HTTP_compression" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HTTP compression</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> schemes on the web today: </span><a href="http://en.wikipedia.org/wiki/DEFLATE" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">DEFLATE</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and </span><a href="http://en.wikipedia.org/wiki/Gzip" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">GZIP</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="http://en.wikipedia.org/wiki/DEFLATE" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">DEFLATE</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is a very popular compression algorithm which generally wraps up data using &nbsp;</span><a href="http://en.wikipedia.org/wiki/LZ77_and_LZ78" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">LZ77 </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">algorithm and </span><a href="http://en.wikipedia.org/wiki/Huffman_coding" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Huffman coding</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. We can describe this process as ‘Pairing a Dictionary transform with a statistical compressor’ which, at it’s core is as simple as you can get for decent savings.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="http://en.wikipedia.org/wiki/Gzip" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">GZIP</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is a file format which will use DEFLATE internally, along-side some interesting blocking, filtering heuristics, a header and a checksum. The two perform almost the same, but in some cases the additional blocking and heuristics that GZIP uses can help it achieve better compression ratios than DEFLATE alone, where there it can block out areas of high or low entropy.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The web stack has done its best to make the usage of these technologies semi-automatic, pushing the </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">actual</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> compression of files to the distribution server (Both algorithms are quite fast at both compression and decompression speeds, which make them great candidates to work server side). </span><a href="http://www.webcodingtech.com/php/gzip-compression.php" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PHP</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://www.askapache.com/htaccess/apache-speed-compression.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">apache</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, even </span><a href="https://developers.google.com/appengine/kb/general#compression" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Google App Engine</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> all support GZIp, and will compress files on your behalf, and allow you to set flags in HTTP headers to describe how the traffic is transferred</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <a href="http://en.wikipedia.org/wiki/SPDY" style="text-decoration: none;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Mode</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">rn transfer protocols like </span><a href="http://en.wikipedia.org/wiki/SPDY" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">SPDY</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and </span><a href="http://en.wikipedia.org/wiki/QUIC" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">QUIC</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> will do some sort of header redundancy elimination, or header compression via GZIP, making most of the web-stack rely entirely on this single compression algorithm.</span></div> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Is GZIP good enough?</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">By age, Gzip is about 20 years old now; it can legally drink in the USA come October. For the most part, it can be considered a great ‘compromise’ algorithm in terms of compression ratio to decompression speed, but it is not perfect.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For example, if you only consider the </span><a href="https://en.wikipedia.org/wiki/Static_web_page" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">static parts of webpages</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, (that is, generated once, and distributed many times) then we could do quite a bit better.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For instance, many developers don’t send large sums of dynamic data to the client that need to be created on-demand; rather most of it is statically served websites, which could easily be compressed offline (where we have all the time and power to compute) and then served multiple times to developers. So </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">compression speed</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> shouldn't really be an argument </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">for</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> GZIP, simply because that’s really only a subset of the data that’s being moved around on the net.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Therefore, we should properly weigh GZip’s contribution on % compression and decompression speed against other archival compressors, to see how it stands up.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To get a sense on where GZIP stands in the scheme-of-things, let’s consider the results from the Large </span><a href="http://mattmahoney.net/dc/text.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Text Compression Benchmark</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, we see that </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GZIP sits at about #122 in terms of compression size. </span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col><col width="*"></col><col width="*"></col><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Mode</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Bytes</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">%smaller</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Decomp time MB/sec</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">NanoZip</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">18,723,846</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">81%</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.16533751</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">NanoZip PLTZ</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">20,537,902</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">79%</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">6.528823217</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">7-Zip</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">21,197,559</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">78%</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.374362363</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">LZTurbo</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">24,416,777</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">75%</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">25.8729479</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">LZHAM</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">24,954,329</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">75%</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">26.4425585</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">BZIP2</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">29,008,736</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">70%</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2.144564843</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Crush</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">32,577,338</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">67%</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">77.67042637</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">GZIP</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">36,445,248</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">63%</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">20.44523351</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">source</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">100,000,000</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0%</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">xxxxx</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It’s nice to see GZIP’s decompression speed is one of the higher of the group; But notice that some packages, like </span><a href="https://code.google.com/p/lzham/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">LZHAM</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> are smaller, AND faster.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It’s also worth noting that the LTCB is focused only on </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">text</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> compression, in which some cases there’s algorithms like </span><a href="http://en.wikipedia.org/wiki/Prediction_by_partial_matching" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">PPM</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> which are known to blow-out the LZ77 compressor that little-old-gzip uses. If we consider that image data is already taken care of by advanced compressors, then shouldn't we employ a text-specific compressor for the rest?</span></div> <b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Could we hack it?</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There’s lots of momentum around GZIP, and any time you challenge the existing tech-stack for the web, you’re bound to get push-back. So lets assume that we wanted to get LZHAM added to the tech stac. By any measurement, getting </span><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><a href="http://www.w3.org/" style="text-decoration: none;">W3C</a></span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to standardize this change would take a while. Which means if we want better compression, instead of relying on native browser support, we'd just have to handle compression ourselves in JavaScript.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This would mean we compress the static assets offline with a compressor, and decompressor in JS, on the client @ run-time. To determine if this is viable, we should take a look at how long these archivers’ perform in their </span><a href="https://github.com/cscott/compressjs" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Javascript implementation</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none;"><colgroup><col width="189"></col><col width="435"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">codec</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">MB/Sec</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ppm.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.009683293</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">LZMA.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.007104233</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bzip2.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.025879408</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bwtc.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.045328359</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bzip2.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.030882925</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">lzp3.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.26450156</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">bwtc.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.053238127</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">lzjbr (lvl 9).js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.171373785</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">lzjbr (lvl 1).js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.206384499</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">context1.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.182398396</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">huffman.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.198333123</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">fenwick.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.412231682</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">mtf.js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.708136208</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">lzjb (lvl 9).js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0.972261002</span></div> </td></tr> <tr style="height: 0px;"><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">lzjb (lvl 0).js</span></div> </td><td style="border: 1px solid #000000; padding: 7px 7px 7px 7px; vertical-align: top;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2.21074193</span></div> </td></tr> </tbody></table> </div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You can see that these numbers are less than ideal, especially compared to their native counterparts. Which means if you WANTED to compress your data smaller than GZIP, you’d still run into issues with decompression time. The only places where this would make sense are </span><a href="http://www.digitaltrends.com/computing/who-has-the-fastest-internet-report-sheds-light-on-most-and-least-connected-countries/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">low-connectivity areas</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, where the power of the device usually surpasses the data on the medium, or distribution scenarios where </span><a href="http://blog.kissmetrics.com/loading-time/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">you’re not burdened by loading in 3 seconds or less.</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One size doesn’t fit all</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Lots of arguments for </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">not</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> changing gzip have to do with conversations about it being ‘good enough for everyone’ But once again, I think this is a generalization. For instance, if you’re an HTML5 game, with a loading-bar, maybe you have an extra 1-2 seconds to load your scripts for the first time, and write them to disk for the user; Since you send 25MB in Javascript &amp; text data to each user, it might be worth a lot of $$ for your company to simply use one of the other compression algorithms which get better compression on the wire.</span></div> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The fact is, GZIP is old, bloated, and we need something better, and I’m going to take the rest of the summer to see if we can fix that.</span></div> <br /> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com3tag:blogger.com,1999:blog-1197137313991855034.post-64020305655813453872013-05-03T06:49:00.000-07:002013-05-07T14:59:25.117-07:00#PERFMATTERS at Google IO 2013With <a href="https://developers.google.com/events/io/">Google I/O 2013</a> showing up less than two weeks away the sessions have been announced; and you can already tell that this year that for Chrome it's <b>all about performance.&nbsp;</b><br /> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYFzpvxp23BJSfOGlo32tlX5uvvlV39pPy3kreIrAsDTWJeZ6SyAKLrJAEXnwzVEZwlT_l4sRJi6uEflrJWOF5pKPx3Z-K3MoOoFbiKbbN6Qz_n7BlZnmkbx1LLgI_SOLfDK98E4JLx4Q/s1600/perfmatters.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="198" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYFzpvxp23BJSfOGlo32tlX5uvvlV39pPy3kreIrAsDTWJeZ6SyAKLrJAEXnwzVEZwlT_l4sRJi6uEflrJWOF5pKPx3Z-K3MoOoFbiKbbN6Qz_n7BlZnmkbx1LLgI_SOLfDK98E4JLx4Q/s320/perfmatters.jpg" width="320" /></a></div> <br /> And Lately, you've been noticing a bald guy&nbsp;<a href="http://aerotwist.com/">#1</a> and <a href="http://mainroach.blogspot.com/">#2</a> giving&nbsp;<a href="http://aerotwist.com/blog/dont-guess-it-test-it/">some</a> <a href="http://www.youtube.com/watch?v=0UNWi7FA36M&amp;feature=youtu.be">talks </a>about the fact that "Web Performance" does not just include <a href="http://www.igvita.com/2013/01/15/faster-websites-crash-course-on-web-performance/">Networking</a>, but also <a href="http://www.html5rocks.com/en/tutorials/speed/css-paint-times/">Rendering</a>&nbsp;and <a href="http://coding.smashingmagazine.com/2012/11/05/writing-fast-memory-efficient-javascript/">Compute</a>&nbsp;performance, and at the conference this year, you can see that in full force.<br /> <br /> Here's my top talks that you should keep an eye on:<br /> <h3> #perfmatters for Networking</h3> <ul> <li><a href="https://developers.google.com/events/io/sessions/325313237">Network/Computer/Render: Automating Performance Best Practices with PageSpeed</a></li> <li><a href="https://developers.google.com/events/io/sessions/325128936">Instant Mobile Websites: Techniques and Best Practices</a></li> <li><a href="https://developers.google.com/events/io/sessions/462268217">WebP: Deploying Faster, Smaller, and More Beautiful Images</a></li> <li><a href="https://developers.google.com/events/io/sessions/327864330">Mobile Performance from the Radio Up: Battery, Latency and Bandwidth Optimization</a></li> </ul> <br /> <h3> #perfmatters for Rendering</h3> <ul> <li><a href="https://developers.google.com/events/io/sessions/325933151">Jank Free: Chrome Rendering Performance</a></li> <li><a href="https://developers.google.com/events/io/sessions/325091862">Web Page Design with the GPU in Mind</a></li> <li><a href="https://developers.google.com/events/io/sessions/325206725">Chrome DevTools Revolutions 2013</a></li> <li><a href="https://developers.google.com/events/io/sessions/324511365">True Grit: Debugging CSS &amp; Render Performance</a></li> </ul> <div> <h3> #perfmatters for Compute</h3> <ul> <li><a href="https://developers.google.com/events/io/sessions/325547004">A Trip Down Memory Lane with Gmail and DevTools</a></li> <li><a href="https://developers.google.com/events/io/sessions/324431687">Web Languages and VMs: Fast Code is Always in Fashion</a></li> <li><a href="https://developers.google.com/events/io/sessions/324908972">Accelerating Oz with V8: Follow the Yellow Brick Road to JavaScript Performance</a></li> </ul> </div> <br /> Interested in keeping up to speed with the latest in web performance? Make sure you follow the #perfmatters tag on <a href="https://twitter.com/search?q=%23perfmatters">Twitter</a> and <a href="https://plus.google.com/s/%23perfmatters">Google+</a>, and check out the <a href="https://plus.google.com/communities/113993151878673122189/stream/fd0cb931-79f1-4f9d-a277-d5da56f4ca94">Web Performance G+ community</a> for more!<br /> <br /> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-1197137313991855034.post-62780433737920572342013-04-23T07:06:00.001-07:002013-04-23T07:06:22.641-07:00Faster App Performance with 5 Simple Algorithms.<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One of the primary reasons that making high performance software is so difficult is that it requires a huge amount of insight and intuition. Understanding the nuances of the technology stack, and how subtle coding modifications can have an impact on performance take years of repetition to master and understand. </span></div> <b id="docs-internal-guid-5ba34f65-3729-a1ca-b9d3-c23b85b6b9df" style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In my professional career, I’ve found that code can be complex, but the computational methodology is pretty consistent. After years of increasing the performance of applications (as my day job) I realized that there’s a boiled-down core set of algorithms that make up the base of all other performance optimizations. When you’re looking for how to increase the performance of your application, make sure you run it through this checklist of : </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ABC-CT</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody> <tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYQ8m5WpgXRNGnCOuKhm1nU8QvRflNUUs2RsKcMca9AtPHpOo9klJk89K13M7HoZJph3I0FcrFmXBvxM-WeBTwuwGphInFkkTz5JXCmrnXalya045caTCgHDjXhekcLbLN0oixR_wP6wM/s1600/perf_ignorance.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYQ8m5WpgXRNGnCOuKhm1nU8QvRflNUUs2RsKcMca9AtPHpOo9klJk89K13M7HoZJph3I0FcrFmXBvxM-WeBTwuwGphInFkkTz5JXCmrnXalya045caTCgHDjXhekcLbLN0oixR_wP6wM/s400/perf_ignorance.jpg" width="400" /></a></td></tr> <tr><td class="tr-caption" style="text-align: center;">I'm known for my subtle manners.</td></tr> </tbody></table> <b style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;"><br /><span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: 16px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h3> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: large; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Approximate&nbsp;</span></h3> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhX-LMLPckvDDOfU7OcbXBtJcHwrIOZnCLoZNF01ZhgV449YNiCsQU9Gy_dwTZIMy3PK2IvkmEyRELR4HnVrehJsYfbsK0eBAFGd09YgxbETGe4CM12CJ7GNwUOPLS4nspKP5PJAOcq4pE/s1600/lowres.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhX-LMLPckvDDOfU7OcbXBtJcHwrIOZnCLoZNF01ZhgV449YNiCsQU9Gy_dwTZIMy3PK2IvkmEyRELR4HnVrehJsYfbsK0eBAFGd09YgxbETGe4CM12CJ7GNwUOPLS4nspKP5PJAOcq4pE/s1600/lowres.jpg" /></a></div> <h3> <span style="color: black; font-family: Arial; font-size: 15px; font-weight: normal; text-indent: 36pt; vertical-align: baseline; white-space: pre-wrap;">In many cases, our algorithms contain the ability to produce a data set with amazing fidelity, such that it mirrors expected real-world values. For example, </span><a href="http://home.comcast.net/~tom_forsyth/blog.wiki.html#%5B%5BA%20matter%20of%20precision%5D%5D" style="line-height: 1.15; text-decoration: none; text-indent: 36pt;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">using 64 bits of fixed-point precision</span></a><span style="color: black; font-family: Arial; font-size: 15px; font-weight: normal; text-indent: 36pt; vertical-align: baseline; white-space: pre-wrap;"> gets you to the furthest distance of </span><a href="http://www.teefury.com/archive/1600/Pluto___Never_Forget_/" style="line-height: 1.15; text-decoration: none; text-indent: 36pt;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Pluto</span></a><span style="color: black; font-family: Arial; font-size: 15px; font-weight: normal; text-indent: 36pt; vertical-align: baseline; white-space: pre-wrap;"> from the Sun (7.4 billion km) with sub-micrometer precision. There exists cases however, where the precision of computed data-sets do not contribute to the final computed result in a significant way ( It’s either lost in the noise, or the detail is not noticeable to the output). As such you can be spending computational time in order to calculate a result which does not enhance the final result. Approximation is useful when the computed data-set does not require the fidelity available to be computed by the computation. In some cases, we can allow our task to compute a less accurate version of a data-set, at the result of significant performance improvements. &nbsp;</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <a href="http://www.youtube.com/watch?v=v1hoVCZZOd0" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Physics engines</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> are a perfect example of this; An accurate, real-world representation of the physical properties of the environment would be too computationally expensive to calculate, and worse, would not aid in the experience computed. </span><a href="http://box2d.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Box2D </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">is a great example of a system which appears to be physically accurate, but whose precision is low enough to appear valid w/o needing physically accurate results.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You can also take a look at real-time 3D rendering, where we often render<a href="http://en.wikipedia.org/wiki/Level_of_detail"> lower resolution meshes</a> to represent higher-order surfaces in order to keep immersion w/o busting framerate.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Take a look at your data requirements, are you spending cycles computing results that don’t matter to the final representation? If so, you might be able to approximate the result for better performance.</span></div> <b style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: large; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Batch</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Most computational processing is made up of tasks which are executed multiple times across many pieces of data. Each task has a specific amount of operational overhead associated with each execution of that task and when executed multiple times the compounded operational overhead can be a significant hit to performance. Regardless of the performance of the computation itself, it can be useful to eliminate the overhead of per-instance calls through batching. This process reduces overall cost by grouping the computational target of the task together, reducing the per-execution overhead where tasks can be batched together.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Great examples of this are technologies like </span><a href="http://en.wikipedia.org/wiki/SIMD" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">SIMD</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, which provides additional hardware on your CPU to all you to batch together floating point operations to reduce wattage and execution drain. </span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note that there are many situations where batching does not net you a clear win. If the number of execution instances are low, then the overhead of those instances will not be significant enough to warrant the work involved to eliminate them.</span></div> <b style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: large; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cache</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There goal of most task-based processing is to compute some data set (in one form or another). In the cases where that data is constant over multiple computations, it makes sense then to save processing power, compute the data set a single time, store the result, and allow subsequent execution instances to look up the precomputed results than compute the data themselves. </span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One of my favorite examples of this was the specular shader in </span><a href="http://www.idsoftware.com/games/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">DOOM3</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Computing specular results of a complex lighting equation can be computationally expensive when ran per GPU fragment, especially on entry-level or lower-end hardware. Rather than allowing full computation of specular values across their environment, the artists pre-computed a set of specular values and allowed a small piece of math inside the fragment processor to fetch the correct specular index from the </span><a href="http://en.wikipedia.org/wiki/Lookup_table" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">lookup table</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This saved GPU time by reducing the number of floating point operations that occurred during the fragment shader.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Note that caching is one of those algorithms which has an inherent trade-off. You’re directly trading runtime processing speed for memory footprint. The more complex data you’re caching / precomputing can result in larger memory footprints, which may not be a correct trade-off on some embedded systems. </span></div> <b style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: large; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Cull</span></h3> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZqQHk4iw2q_D8LToKxKCOy-A97NKPrcI3kRHgnA1AiPyRXF29GmyzhZSsBJQIhyphenhyphenjYbrtfz5G635qtJ6QomuRNzfC5HC-D3oqXe-GatSYlSYbH-wuiK9JyvHtTs4oY_DhhUO5fCs9E7uo/s1600/cull.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZqQHk4iw2q_D8LToKxKCOy-A97NKPrcI3kRHgnA1AiPyRXF29GmyzhZSsBJQIhyphenhyphenjYbrtfz5G635qtJ6QomuRNzfC5HC-D3oqXe-GatSYlSYbH-wuiK9JyvHtTs4oY_DhhUO5fCs9E7uo/s1600/cull.jpg" /></a></div> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="font-family: Arial; font-size: 15px; font-weight: normal; line-height: 1.15; text-indent: 36pt; white-space: pre-wrap;">If you have a view of object processing in your code, then some objects may not need to be processed at various execution points. For example, if you have an object that is hidden behind a wall, it won’t contribute to the final pixels on the screen; So by drawing the object, you are wasting performance. Culling is the concept of avoiding processing of objects based upon their need to contribute to the final data set.</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The easiest examples of this come from the graphics community in the form of </span><a href="https://en.wikipedia.org/wiki/Frustum_culling" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Frustum culling</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and</span><a href="http://en.wikipedia.org/wiki/Hidden_surface_determination" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"> Occlusion culling</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. </span><a href="http://www.altdevblogaday.com/2012/04/14/software-rasterizer-part-1/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Graphics Rendering</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> works by </span><a href="http://en.wikipedia.org/wiki/Perspective_projection#Perspective_projection" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">transforming world-space</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> mesh/texture primitives into a parameterized 2D pixel grid (ie your monitor), and just like the real 3D environments we all live in, some objects can be hidden from view. This means those objects do not contribute to the final 2D pixel grid, and thus moving forward with rendering those items will be wasting computation.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It’s worth noting that Culling (as an optimization) only works when the performance associated with determining if an object should be culled is lower than the performance cost to process the object anyhow. If you’re spending 12ms a frame to save you 4ms, then it’s not the right use of your time.</span></div> <b style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h3 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 8pt;"> <span style="background-color: transparent; color: #666666; font-family: 'Trebuchet MS'; font-size: large; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Thread</span></h3> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Some computation is simply slower than others, and even after adopting all the previous methods, can still be slow. It’s times like that where you can rely on the trending growth of </span><a href="http://en.wikipedia.org/wiki/Moore's_law" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">computational power</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, and split your computation off onto multiple execution elements that can operate in parallel. By allowing multiple processors, or threads of execution, </span><a href="http://marjan.fesb.hr/~psarajce/images/Fork_Join.png" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">to operate on your dataset in paralle</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">l, you reduce the overall time required to complete the task.</span></div> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><a href="http://www.amazon.com/The-Multiprocessor-Programming-Maurice-Herlihy/dp/0123705916" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Putting threading into your system</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> is not for the light-at-heart. It requires you not only to adjust your view of computational complexity, but also address your platforms </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">memory model</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in order to properly achieve it. This means that you quickly end up going down the rabbit hole if you’re not properly understanding the problem, or worse, you don’t understand what you’re trying to accomplish. In these cases, </span><a href="http://threadingbuildingblocks.org/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">using existing threading libraries</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> can be a huge win, as they handle most of the heavy lifting of primitives, alongside platform-specific issues that can arise.</span></div> <b style="-webkit-text-stroke-width: 0px; color: black; font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;"> <span style="background-color: transparent; color: black; font-family: 'Trebuchet MS'; font-size: 17px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Keep Calm and Profile it.</span></h2> <br /> <div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I’ve tried my best to highlight the instances where each of these techniques are a clear win, and a clear loss. With all applications, you need to ensure that you’re properly profiling your code to determine what the true-core causes are for performance problems before adopting any of the above techniques. In general, as you find slow-performing components of your system, make sure that you think about each of the options in </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ABC-CT</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to chart a path to success.</span></div> <div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div> <span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-1197137313991855034.post-6633186275254682832013-04-12T15:26:00.004-07:002013-04-12T15:27:04.045-07:00Great books for game developers<br /> &nbsp;One of the top questions that came out of my <a href="https://www.udacity.com/course/cs255">UDACITY course on HTML5 Game Development</a> was <i>"what books to you recommend for game development?"</i> Behold, here's my personal list of awesome.<br /> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVaxkd6QjP5qVJLttSDa7hQXTYNk_g9b263NvfoT882ITiT9iX2SbSkH9v5aJ7q_Yp76xNfjsHQZ_GSaOLQuehccbjP2OkNWQmsMv1uSvgoTG5H8QOqyGxJ13Sr97Apnxy_-_pHVJAYKo/s1600/4558_original.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="250" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVaxkd6QjP5qVJLttSDa7hQXTYNk_g9b263NvfoT882ITiT9iX2SbSkH9v5aJ7q_Yp76xNfjsHQZ_GSaOLQuehccbjP2OkNWQmsMv1uSvgoTG5H8QOqyGxJ13Sr97Apnxy_-_pHVJAYKo/s320/4558_original.jpg" width="320" /></a></div> <br style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 18px;" /> <br /> <ul> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">The Art of Multiprocessor programming -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/Ow9lv" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/Ow9lv</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Data Compression: The complete reference -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/tuWyz" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/tuWyz</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Real Time Rendering -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/M84Lm" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/M84Lm</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Garbage Collection : Algorithms for Automatic Dynamic Memory Management -</span><a class="ot-anchor" href="http://goo.gl/Zfnsn" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/Zfnsn</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Tricks of the 3D game programming Gurus -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/SHWhf" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/SHWhf</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Game Coding Complete -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/edYGd" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/edYGd</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">AI Game Engine Programming -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/OnwRS" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/OnwRS</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Programming interviews exposed -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/BnmXv" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/BnmXv</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">How to move Mt. Fuji -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/CsXSg" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/CsXSg</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">C++ for game programmers -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/Nnbu8" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/Nnbu8</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">3D Math Primer for Graphics and Games development -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/yWyMY" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/yWyMY</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Physics for Game developers -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/fMWCq" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/fMWCq</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">AI Programming wisdom (series) -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/sgiO0" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/sgiO0</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Game Engine Gems (series) -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/8ru6p" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/8ru6p</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">GPU Gems (series) -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/doJ0C" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/doJ0C</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Texturing and modeling : A procedural approach -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/Q6DHs" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/Q6DHs</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Graphics Gems (series source code) -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/MJpXZ" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/MJpXZ</a><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;"></span></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">GPU Pro (series) -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/xnGGS" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/xnGGS</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Physically based rendering -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/lLFYk" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/lLFYk</a><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;"></span></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Game Programming Gems (series) -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/KPVXo" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/KPVXo</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">ShaderX (series) -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/Q8PVx" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/Q8PVx</a></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Collision Detection in Interactive 3D Environments -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/gDJ4E" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">http://goo.gl/gDJ4E</a><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;"></span></li> <li><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Game Engine Architecture -&nbsp;</span><a class="ot-anchor" href="http://goo.gl/28VM" rel="nofollow" style="cursor: pointer; font-family: arial, sans-serif; font-size: 13px; line-height: 18px; text-decoration: none;">http://goo.gl/28VM</a><span style="font-family: arial, sans-serif; font-size: 13px; line-height: 18px;"></span></li> <li><span style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 18px;">Real-Time Collision Detection -<span id="goog_1485348340"></span></span><span style="font-family: arial, sans-serif; font-size: x-small;"><span style="line-height: 18px;"><a href="http://goo.gl/mDSCX">http://goo.gl/mDSCX</a></span></span></li> </ul> <br /> <br /> <br /> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-1197137313991855034.post-3587680242721006572013-02-27T09:33:00.000-08:002013-02-27T09:34:22.146-08:00Fast HTML5 canvas : Cached Canvas Tiles<b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">Now that my <a href="https://www.udacity.com/course/cs255">UDACITY </a>course is up and in the wild, I felt it was time to look back at some research data I did while doing the rendering engine for GRITS; This is a multi-article series, and today, we're talking about how to reduce runtime performance overhead by pre-caching your image tiles, cutting down the number of draws-per-frame.</span></span></b></b><br /> <h2 dir="ltr"> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">Before we begin..</span></b></b></h2> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: decimal; vertical-align: baseline;"><b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><b style="font-weight: normal;"><span style="vertical-align: baseline; white-space: pre-wrap;">Read the <a href="http://mainroach.blogspot.com/2013/02/fast-html5-canvas-rendering-tiled-maps.html">first article</a>.</span></b></b></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: decimal; vertical-align: baseline;"><b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="vertical-align: baseline; white-space: pre-wrap;">You should watch my talk on </span><a href="http://www.youtube.com/watch?v=Prkyd5n0P7k"><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;">GRITS</span></a><span style="vertical-align: baseline; white-space: pre-wrap;">, or even look at the </span><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;"><a href="https://code.google.com/p/gritsgame/">GRITS code</a></span></b></b></li> </ol> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><span style="font-family: Arial; font-size: 19px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h2 dir="ltr"> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">Fast Canvas rendering with GRITS</span></b></h2> <div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">At Google I/O 2012 I presented a nice little research project of mine called GRITS; it was an attempt to take a hard look at HTML5 gaming, from the viewpoint of serious game developers. In the rendering of GRITS, we quickly ran into an issue of fast canvas rendering, and handled it in a different way, by trading memory for performance.</span></b></div> <div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <span id="internal-source-marker_0.6028940638061613"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">More specifically, the issue was that we received a much-much better looking map from our artist friend </span><a href="http://www.fuzzycubesoftware.com/"><span style="color: #1155cc; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Pete Parisi</span></a><span style="font-family: Arial; vertical-align: baseline;"><span style="font-size: 15px; white-space: pre-wrap;">, which included a whole new set of layers, and additional tiles placed. For most of our desktop machines, this </span><span style="font-size: 15px; white-space: pre-wrap;">wasn't</span><span style="font-size: 15px; white-space: pre-wrap;"> an issue, but for some of our laptops, we ended up seeing a large performance burden with the change.</span></span></span></div> <div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The burden came from the fact that each tile required an extra amount of rendering to occur in order to create the final image. That is, each tile-area on the screen had 4-or-5 image tiles placed on it. Effectively, our draw count, per-frame, quadrupled with the new map.</span></b></div> <br /> <div style="text-align: center;"> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><img height="195px;" src="https://lh3.googleusercontent.com/E5A2NKR0kRmgvcdO-DAUnECmVDjzgQISZr32uw4pMgLRkrnqSQAj40eIbdfv3M_UH9gtnBUv8MGXebtHfNITwtnFnfJdljhLcVWLoEBzpfRrFU9TMfQ" width="245px;" /></b></div> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Our solution involved taking the raw map rendering data, and using a concept of off-screen canvas rendering to reduce the number of draw calls. Effectively, we divided the entire map into 1024x1024 canvases and rendered the map into each canvas, storing it for future usage. For GRITS the bulk of performance strain wrt rendering was from the background / map rendering, as such, with this method, we were able to reduce the number of draws-per-frame from 1k down to about 6 at worst (for the background). </span></b><br /> <div style="text-align: center;"> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg37KPQVpHjKSUIcaoKyztn5jut4T-N6iYYmepgR8i7Vu9VII9MbPEnOWjWpWerx6ZniWdHBcbW1aNVCX085RdN59j6IxyshXPvdn2MR2zFQ7XHTh-7nFSP67G0HSQAUdPVUPFp2OCNmek/s1600/fullmap1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="178" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg37KPQVpHjKSUIcaoKyztn5jut4T-N6iYYmepgR8i7Vu9VII9MbPEnOWjWpWerx6ZniWdHBcbW1aNVCX085RdN59j6IxyshXPvdn2MR2zFQ7XHTh-7nFSP67G0HSQAUdPVUPFp2OCNmek/s320/fullmap1.jpg" width="320" /></a></div> <br /></div> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Per frame, we’d clear the screen and draw the background tiles, followed by the animated sprites on top of it.</span></b></div> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> </b> <br /> <div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt; text-indent: 36pt;"> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The result was a trade-off between memory and draw-call performance. Yes, we were churning up more memory for all of those 4mb canvases, but we had reduced the draw call &nbsp;to 1-6 per frame (down from ~200) which shown great performance improvements.</span></b></div> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr"> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> <span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">Creating a CanvasTile</span></b></h2> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">To help with this process, we created a new concept called &nbsp;‘canvasTile’ this will represent a canvas object that we render into and use later.</span> </b><br /> <pre><b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;">var CanvasTile = Class.extend({ x:0, y:0, w:100, h:100, cvsHdl:null, ctx:null, isFree:true, numFramesNoVisible:0, //----------------------------- create:function(width,height) { this.x = -1; this.y = -1; this.w = width; this.h = height; //create the canvas object here var can2 = document.createElement('canvas'); can2.width = width; can2.height = height; this.cvsHdl = can2; this.ctx = can2.getContext('2d'); this.isFree = true; } }); </b></pre> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> </b> <br /> <div dir="ltr"> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><br /></b></div> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> <br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h2 dir="ltr"> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> <span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">Filling the cache</span></b></h2> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Now that we have the basic object, we need to create a 2D array of them that cover the world-space map correctly. To do this, we divide the size of the map (on each axis) by the size of each canvasTile (which is tunable) to get the number of canvasTiles along that axis. We store these in an array for retrieval later.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Most importantly, is that once we create them, we call </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">fillCanvasTile</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> which will do all the work to fill in the canvas with the proper tile data.</span> </b><br /> <pre><b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;">//--------------------------- preDrawCache:function() { var xCanvasCount = 1 + Math.floor(this.pixelSize.x / this.canvasTileSize.x); var yCanvasCount = 1 + Math.floor(this.pixelSize.y / this.canvasTileSize.y); var numSubCanv = xCanvasCount*yCanvasCount; for(var yC = 0; yC &lt; yCanvasCount; yC ++) { for(var xC = 0; xC &lt; xCanvasCount; xC ++) { var k = new CanvasTile(); k.create(this.canvasTileSize.x,this.canvasTileSize.y); k.x = xC * this.canvasTileSize.x; k.y = yC * this.canvasTileSize.y; this.canvasTileArray.push(k); this.fillCanvasTile(k); } } gMap.fullyLoaded = true; }, </b></pre> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> </b> <br /> <div dir="ltr"> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">To fill the canvasTile object, we need to modify our rendering function from the last article. Before we were taking into account the entire viewrect when rendering; we can extend this concept, by culling against the suggested canvasTile bounds rather than the viewRect itself. This allows us to easily re-use our existing code to fill our new canvas tiles.</span></b></b><br /> <pre><b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> //------------------------------------ fillCanvasTile:function(ctile) { var ctx = ctile.ctx; ctx.fillRect(0,0,ctile.w, ctile.h); var vRect={ top:ctile.y, left:ctile.x, bottom:ctile.y+ctile.h, right:ctile.x+ctile.w}; for (var layerIdx = 0; layerIdx $lt; this.currMapData.layers.length; layerIdx++) { if (this.currMapData.layers[layerIdx].type != "tilelayer") continue; ..... var visible= this.intersectRect(vRect, {top:worldY,left:worldX,bottom:worldY + this.tileSize.y,right:worldX + this.tileSize.x}); if(!visible) continue; ctx.drawImage(tPKT.img, tPKT.px, tPKT.py, this.tileSize.x, this.tileSize.y, worldX - vRect.left, worldY - vRect.top, this.tileSize.x, this.tileSize.y); </b></pre> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"><b style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b> <b style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b></b></div> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> </b> <br /> <div dir="ltr"> </div> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span> </b><br /> <h3 dir="ltr"> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> <span style="color: #666666; font-family: Arial; font-size: 16px; vertical-align: baseline; white-space: pre-wrap;">Draw!</span></b></h3> <b id="internal-source-marker_0.6028940638061613" style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">All the creation and filling of canvasTiles occur at init time for our app; Later on, to render, we simply need to determine if a given canvasTile is visible to the viewRect (using the same box-box intersection code we introduced last article) and if it is, draw it as though it were any other tile.</span> <pre>//--------------------------- draw: function (ctx) { // if(!this.fullyLoaded) return; //aabb test to see if our view-rect intersects with this canvas. for(var q =0; q &lt; this.canvasTileArray.length; q++) { var r1 = this.canvasTileArray[q]; if(r1.isVisible()) ctx.drawImage(r1.cvsHdl, r1.x - this.viewRect.x, r1.y - this.viewRect.y); } }, </pre> <div dir="ltr"> </div> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span> <h3 dir="ltr"> <span style="color: #666666; font-family: Arial; font-size: 16px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></h3> <h3 dir="ltr"> <span style="color: #666666; font-family: Arial; font-size: 16px; vertical-align: baseline; white-space: pre-wrap;">Results</span></h3> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The performance differences were directly seen, as our FPS on lower-end machines shot through the roof, although our memory overhead was also increased; this kept us from shipping the larger version of the map to players, as some low-end machines would choke out from the extra memory requirements.&nbsp;</span></b><br /> <div> <b style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b></div> <div> <b style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">We tried a middle-ground solution; one that would only use a static-pool of canvasTiles that we’d fill in as-needed; but this resulted in the </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">worst</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> of both worlds; The memory requirements were still pretty high to get a good number of tiles, and we were having to fill in a large amount per frame, which would cause hitching. less than ideal :( </span> </b><br /> <h2 dir="ltr"> <b style="font-weight: normal;"> <span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">Source</span></b></h2> <b style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">You can find the source for the GRITS tiling, as well as the free-list tiling mentioned in this article on my <a href="https://github.com/mainroach/sandbox/tree/master/canvas_tiled_rendering">Github</a>.</span> </b><br /> <br /> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> </div> Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1197137313991855034.post-43296805750741895692013-02-21T11:30:00.001-08:002013-02-21T11:35:19.491-08:00Fast HTML5 canvas : Rendering TILED maps<b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"></b><br /> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">Now that my <a href="https://www.udacity.com/course/cs255">UDACITY </a>course is up and in the wild, I felt it was time to look back at some research data I did while doing the rendering engine for GRITS; This is a multi-article series, and today, we're starting with the basics of how you load and render a TILED map to the screen.</span></span></b><br /> <h2 dir="ltr"> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">Before we begin..</span></b></h2> <ol style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: decimal; vertical-align: baseline;"><b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="vertical-align: baseline; white-space: pre-wrap;">You should read my article on </span><a href="http://www.altdevblogaday.com/2012/09/17/building-an-html5-game-dont-shrug-off-atlases/"><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;">atlasing</span></a><span style="vertical-align: baseline; white-space: pre-wrap;"></span></b></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: decimal; vertical-align: baseline;"><b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="vertical-align: baseline; white-space: pre-wrap;">You should watch my talk on </span><a href="http://www.youtube.com/watch?v=Prkyd5n0P7k"><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;">GRITS</span></a><span style="vertical-align: baseline; white-space: pre-wrap;">, or even look at the </span><a href="https://code.google.com/p/gritsgame/"><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;">GRITS code</span></a><span style="vertical-align: baseline; white-space: pre-wrap;"></span></b></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: decimal; vertical-align: baseline;"><b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="vertical-align: baseline; white-space: pre-wrap;">I’m using a concept of Object Oriented programming in this discussion, using Class object based on </span><a href="http://ejohn.org/blog/simple-javascript-inheritance/"><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;">John Resigs code</span></a><span style="vertical-align: baseline; white-space: pre-wrap;"></span></b></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: decimal; vertical-align: baseline;"><b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="vertical-align: baseline; white-space: pre-wrap;">Make sure you’ve played around with the TILED editor</span></b></li> </ol> <h2 dir="ltr"> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">Loading TILED data</span></b></h2> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div style="text-align: left;"> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="vertical-align: baseline;"><span class="Apple-tab-span" style="white-space: pre;"> </span>As mentioned in the 2012 talk on GRITS, <a href="http://www.mapeditor.org/">TILED</a> is a great 2D editor for world data, AND lucky for us, it outputs to JSON as a format.&nbsp;</span></b></span></b></div> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span></b> <br /> <div style="text-align: left;"> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><b style="font-weight: normal;"><span style="vertical-align: baseline;"><br /></span></b></span></b></div> <div class="separator" style="clear: both; text-align: center;"> <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioFaZSDfn4PkQMGgGERaBRz7l7AvJDlFy7mLgexP0pcAw2fwTFQZ6SktUbYYE6-CeXaMqSW88zUjJqDX8sgYeXNcq6U_Y0n9T_EpAhlvWl6TqVsCAVL4btPpNF_kHeSSIAnKm1azuItQI/s1600/tiled0.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="178" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioFaZSDfn4PkQMGgGERaBRz7l7AvJDlFy7mLgexP0pcAw2fwTFQZ6SktUbYYE6-CeXaMqSW88zUjJqDX8sgYeXNcq6U_Y0n9T_EpAhlvWl6TqVsCAVL4btPpNF_kHeSSIAnKm1azuItQI/s320/tiled0.jpg" width="320" /></a></div> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span></b> <br /> <div style="text-align: center;"> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><b style="font-weight: normal;"></b></span></b></div> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><b style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b> Here’s how we read in the TILED JSON data.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">TILED JSON File Layout, in condensed form</span></b><br /> <ul style="margin-bottom: 0pt; margin-top: 0pt;"><b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">height,width,orientation, tileheight, tilewidth,version</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">properties dictionary (name-value pairs)</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">layers array</span></li> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">data : array of integers specifying what tile to draw</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">layer name,</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">width,height, visible, opacity</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">layer type (is this a tile layer, or object layer?)</span></li> </ul> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">tilesets array</span></li> <ul style="margin-bottom: 0pt; margin-top: 0pt;"> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">image (ie grits_master.png)</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">firstgid - Important, this lets us know what tile we’re referencing in the data!</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: circle; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">imgwidth, imgheight,margin,spacing, tilewidth, tileheight</span></li> </ul> </b></ul> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h3 dir="ltr"> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"> <span style="color: #666666; font-family: Arial; font-size: 16px; vertical-align: baseline; white-space: pre-wrap;">1. Fetch the data from the server</span></b></h3> <b id="internal-source-marker_0.19405772886238992" style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">I expect that your JSON data will be placed on your server, and that you’ll need to fetch it through some method of </span><a href="http://en.wikipedia.org/wiki/XMLHttpRequest"><span style="color: #1155cc; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">XHR</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">, namely we define a </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">load</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> function that takes in a location to a map JSON file to load: </span></b><br /> <pre class="prettyprint linenums"><code class="language-js"> </code><div dir="ltr" style="font-family: 'Times New Roman'; white-space: normal;"> <pre class="prettyprint linenums"><code class="language-js">load: function (map) { xhrGet(map, false,function(data) { <span class="nocode">//one the file has been fetched from the server, send it to our map to parse</span> gMap.parseMapJSON(data.response); });</code></pre> <table style="border-collapse: collapse; border: none; width: 624px;"><tbody></tbody></table> </div> <div style="font-family: 'Times New Roman'; white-space: normal;"> </div> <div style="font-family: 'Times New Roman'; white-space: normal;"> <span style="color: #666666; font-family: Arial; font-size: 16px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"></span> </div> </pre> <div> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div> <div> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">xhrGet is a simple function that wraps up a XHR call to our server to grab a set of data; The XHRGet function below was scraped from the </span><a href="https://code.google.com/p/gritsgame/source/browse/src/client/scripts/pregame/xhr.js"><span style="color: #1155cc; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">GRITS codebase</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> for ease-of-use.</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <h3 dir="ltr"> <span style="color: #666666; font-family: Arial; font-size: 16px; vertical-align: baseline; white-space: pre-wrap;">2. Parsing the map data</span></h3> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Parsing the map data is quite simplistic; Once we’ve received the JSON data, we need to convert it to a javascript object using the </span><a href="https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Global_Objects/JSON/parse"><span style="color: #1155cc; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">JSON.parse</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> method. For the sake of ease, we cache some of the values direction so that we can fetch them later w/o an indirection:</span><br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span style="font-size: x-small;"><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> //---------------------------</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;parseMapJSON:function(mapJSON)</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;{</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;this.currMapData = JSON.parse( mapJSON );</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;var map = this.currMapData<span style="color: #38761d;">;//this is just for syntactical sugar</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;this.numXTiles = map.width;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;this.numYTiles = map.height;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;this.tileSize.x = map.tilewidth;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;this.tileSize.y = map.tileheight;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;this.pixelSize.x = this.numXTiles * this.tileSize.x;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;this.pixelSize.y = this.numYTiles * this.tileSize.y;</span></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <span style="color: #666666; font-family: Arial; font-size: 16px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <br /> <span style="color: #666666; font-family: Arial; font-size: 16px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;"></span><br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">In TILED, an artist can drop in a group of images, and start placing tiles from any of the images onto the map as they see fit. During the loading of our tiled data, we need to also load the atlases that were used to create the map, listed inside the .JSON file. </span><br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Since image loading is async, and we don’t know what tiles the artist might have defined in the map format, then we have to read in the tile-sheet data, and store it so we can access it later.</span><br /> <div> <span style="font-family: Arial;"><span style="font-size: 15px; white-space: pre-wrap;"><br /></span></span><b style="font-weight: normal;"></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span style="font-size: x-small;"><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"><span style="color: #38761d;">//load our tilesets if we are a client.</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;var gMap = this;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;gMap.imgLoadCount = 0; <span style="color: #38761d;">//reset our image loading counter</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;for (var i = 0; i &lt; map.tilesets.length; i++) </span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;{</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//load each image and store a handle to it</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var img = new Image();</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;img.onload = new function() </span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>{gMap.imgLoadCount++;}; <span style="color: #b6d7a8;">&nbsp;</span><span style="color: #38761d;">//once the image is loaded, increase a global counter</span></span><span style="color: #38761d;"><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//NOTE that the TILED data puts some gnarly relative path data into the file</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//let’s get rid of that, since our directory structure in the shipping product is not the same</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//as in the editor layout.</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;img.src = "../data/" + map.tilesets[i].image.replace(/^.*[\\\/]/, '');</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #38761d;">//store this data in a way that makes it easy to access later:</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var ts = {</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"firstgid": map.tilesets[i].firstgid,</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"image": img,</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"imageheight": map.tilesets[i].imageheight,</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"imagewidth": map.tilesets[i].imagewidth,</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"name": map.tilesets[i].name,</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"numXTiles": Math.floor(map.tilesets[i].imagewidth / this.tileSize.x),</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"numYTiles": Math.floor(map.tilesets[i].imageheight / this.tileSize.y)</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;};</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;this.tileSets.push(ts);</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;<span style="color: #38761d;">&nbsp;&nbsp;&nbsp;//kick off a timer to check if all the images are loaded. If so, move forward.</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;checkWait(</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;function()</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return gMap.imgLoadCount == gMap.tileSets.length;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;},</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;function () </span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;gMap.fullyLoaded = true;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;});</span></span><br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <b style="font-weight: normal;"> <br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">The </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">checkWait</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> function is a nice little helper that will kick off a timer to check for the results of a function periodically. Once the test function returns TRUE, it will call the result function</span></b><br /> <h2 dir="ltr"> <b style="font-weight: normal;"> <span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">Rendering Tiled Data</span></b></h2> <b style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Rendering the data is pretty straight-forward, We walk through each layer, and each tile in that layer, and draw the specified sprite to the screen. Dominantly, the most complex piece of logic we have here is actually determining where this tile should reside in world-space, given that we only have an index to it’s location in the layer data.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: x-small;"><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">draw: function (ctx) </span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;{ </span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span>if(!this.fullyLoaded) return; //sanity check!</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;for (var layerIdx = 0; layerIdx &lt; this.currMapData.layers.length; layerIdx++) </span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span>{</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (this.currMapData.layers[layerIdx].type != "tilelayer") continue;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var dat = this.currMapData.layers[layerIdx].data;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #38761d;">&nbsp;//for each tile in this layer</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for (var tileIDX = 0; tileIDX &lt; dat.length; tileIDX++) {</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var tID = dat[tileIDX];</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (tID == 0) continue;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;<span style="color: #38761d;">&nbsp;&nbsp;&nbsp;//find what the tileIndexOffset is for this layer</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var tPKT = this.getTilePacket(tID);</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;<span style="color: #38761d;">&nbsp;&nbsp;&nbsp;&nbsp;//figure out what our X,Y coords are to render this tile in world-space</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var worldX = Math.floor(tileIDX % this.numXTiles) * this.tileSize.x;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var worldY = Math.floor(tileIDX / this.numXTiles) * this.tileSize.y;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #38761d;">&nbsp;// Nine arguments: the element, source (x,y) coordinates, source width and </span></span><span style="color: #38761d;"><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// height (for cropping), destination (x,y) coordinates, and destination width </span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// and height (resize).</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ctx.drawImage(tPKT.img, tPKT.px, tPKT.py, this.tileSize.x, this.tileSize.y, worldX, worldY, this.tileSize.x, this.tileSize.y);</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;}</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;},</span></span><br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <b style="font-weight: normal;"> <br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="color: #666666; font-family: Arial; font-size: 16px; font-weight: bold; vertical-align: baseline; white-space: pre-wrap;">What tile to render</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">If you look at a chunk of the layer data from the tiled map:</span><br /><span style="font-family: Arial; font-size: 13px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;"> "data":[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167, 167,</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">0 represents that there’s no title specified for that layer.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Any non-zero number however, represents a tile that has been placed at that location from an atlas in the image map. Remember that in the tilesets array specifies an image, and a starting offset for that image. So </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">167</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> means </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">image starting with base index 1, tileindex 166</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">. For any given tile in our array, we need to determine what atlas it’s from, and what its’ X,Y coords are in that atlas; We can simplify all this work with a small function to do this for us.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: x-small;"><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">//---------------------------</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;getTilePacket: function (tileIndex) {</span><br /><span style="color: #6aa84f; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">//the paket data we’ll return to the calling function</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;var pkt = {</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"img": null,</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"px": 0,</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"py": 0</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;};</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="color: #6aa84f; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">//given an index, loop through all tileSets and determine which one we’re looking for</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;var i = 0;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;for (i = this.tileSets.length - 1; i &gt;= 0; i--) {</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (this.tileSets[i].firstgid &lt;= tileIndex) break;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;}</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;pkt.img = this.tileSets[i].image;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="color: #38761d; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">//figure out what x/y is for the given tileIndex</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;var localIdx = tileIndex - this.tileSets[i].firstgid;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;var lTileX = Math.floor(localIdx % this.tileSets[i].numXTiles);</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;var lTileY = Math.floor(localIdx / this.tileSets[i].numXTiles);</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;pkt.px = (lTileX * this.tileSize.x);</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;pkt.py = (lTileY * this.tileSize.y);</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;return pkt;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;},</span></span><br /> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <b style="font-weight: normal;"> <br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span></b><br /> <h3 dir="ltr"> <b style="font-weight: normal;"> <span style="color: #666666; font-family: Arial; font-size: 16px; vertical-align: baseline; white-space: pre-wrap;">Coordinate spaces and ViewRect</span></b></h3> <b style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">With our current code, we run into the issue of coordinate spaces. You see, the canvas has coordinates [0,width] and [0,height], which is not equal to the coordinates of the map, [0, map width] (aka world space-coordinates). As such, we end up only drawing part of the map whose coordinates are identical to the canvas coordinates; this of course is less than our desired output, as we want the ability to render parts of the map, wherever our player happens to be.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">For this, we introduce the concept of a view-rect; or rather, a rectangle that defines what part of the map, in world-coordinates, that is currently visible in canvas-coordinates. Another way to put this, is that we use the view-rect to </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">map</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> world-space to view-space.</span></b></div> <div> <b style="font-weight: normal;"><span style="font-family: Arial;"><span style="font-size: 15px; white-space: pre-wrap;"><br /></span></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-size: x-small;"><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">draw: function (ctx) </span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;{ //</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span><span style="color: #38761d;">//....</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #38761d;">&nbsp;//test if this tile is within our world bounds</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var worldX = Math.floor(tileIDX % this.numXTiles) * this.tileSize.x;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var worldY = Math.floor(tileIDX / this.numXTiles) * this.tileSize.y;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;</span><span style="color: red; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if ((worldX + this.tileSize.x) &lt; this.viewRect.x || (worldY + this.tileSize.y) &lt; this.viewRect.y || worldX &gt; this.viewRect.x + this.viewRect.w || worldY &gt; this.viewRect.y + this.viewRect.h) continue;</span><br /><span style="color: red; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="color: red; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//adjust all the visible tiles to draw at canvas origin.</span><br /><span style="color: red; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;worldX -= this.viewRect.x;</span><br /><span style="color: red; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;worldY -= this.viewRect.y;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: #38761d;">&nbsp;// Nine arguments: the element, source (x,y) coordinates, source width and </span></span><span style="color: #38761d;"><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// height (for cropping), destination (x,y) coordinates, and destination width </span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// and height (resize).</span></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ctx.drawImage(tPKT.img, tPKT.px, tPKT.py, this.tileSize.x, this.tileSize.y, worldX, worldY, this.tileSize.x, this.tileSize.y);</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;&nbsp;&nbsp;}</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;"> &nbsp;},</span></span><br /> <span style="color: red; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></td></tr> </tbody></table> </div> <b style="font-weight: normal;"> <br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Another added benefit of the view-rect, is that it allows us to do visibility culling on our tiles, such that we only render the tiles that reside visible to the user, rather than drawing every tile in the map, regardless if its’ on screen or not.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">To use the view-rect properly, we allow the </span><span style="font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">update</span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"> function of our little example to modify the view-rect based upon the position of the player in the world:</span></b></div> <div> <b style="font-weight: normal;"><span style="font-family: Arial;"><span style="font-size: 15px; white-space: pre-wrap;"><br /></span></span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <div dir="ltr"> <table style="border-collapse: collapse; border: none; width: 624px;"><colgroup><col width="*"></col></colgroup><tbody> <tr style="height: 0px;"><td style="border: 1px solid rgb(0, 0, 0); padding: 7px; vertical-align: top;"><span style="font-size: x-small;"><span style="color: #38761d; font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">//make sure the player is at the center of the screen</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">gMap.viewRect.x = (pPos.x - ( canvas_width / 2 ) );</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">gMap.viewRect.y = (pPos.y - ( canvas_height / 2 ));</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">gMap.viewRect.w = canvas_width;</span><br /><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">gMap.viewRect.h = canvas_height;</span></span></td></tr> </tbody></table> </div> <b style="font-weight: normal;"> <br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b><br /> <h3 dir="ltr"> <b style="font-weight: normal;"> <span style="color: #666666; font-family: Arial; font-size: 16px; vertical-align: baseline; white-space: pre-wrap;">What about rotations?</span></b></h3> <b style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">Unknown to most, TILED does support rotating/flipping tiles, although using it in rendering takes a bit of code change on our side.</span><br /><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">When a tile is rotated in TILED, it’s tile-index value in JSON changes, effectively, it appears that the app will set the high-4 bits to a specific value representing the state of the tile. So if you make the top 4 bits of a 32 bit integer, you’ll see:</span></b><br /> <ul style="margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Normal = 0,</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Rot 90 = 0xA0000000,</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Rot 180 = 0xC0000000,</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Rot 270 = 0x60000000,</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Flip Y = 0x40000000,</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Flip X = 0x80000000,</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Flip Y &amp; Rot90 = 0x20000000,</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Flip X &amp; Rot90 = 0xE0000000,</span></li> </b></ul> <b style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" />Note that to render these tiles, you’ll need to rotate them using basic principles of linear mathematics:</span></b><br /> <ul style="margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Create a 2D matrix to translate the tile to it’s center (-32,-32 or something)</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Augment the matrix by another which rotates / flips it</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">Augment the matrix by another which translates it to the world-space location</span></li> <li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline; white-space: pre-wrap;">call drawImage with 0,0 as the world-space location.</span></li> </b></ul> <b style="font-weight: normal;"> <span style="color: red; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></b> <br /> <h2 dir="ltr"> <b style="font-weight: normal;"> <span style="font-family: Arial; font-size: 19px; vertical-align: baseline; white-space: pre-wrap;">Source</span></b></h2> <b style="font-weight: normal;"> <span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;">You can find the source to this article on my <a href="https://github.com/mainroach/sandbox/tree/master/canvas_tiled_rendering">Github</a></span></b><br /> <b style="font-weight: normal;"><span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></b> ~Main <br /> <center> <small style="text-align: -webkit-center;"><i><br class="Apple-interchange-newline" />You can find Colt McAnlis here:</i></small><br style="text-align: -webkit-center;" /><a href="https://plus.google.com/u/0/105062545746290691206/posts" style="text-align: -webkit-center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2Xjim-nDdaZQw5H1ZJmXolh5BtgkNO63OmInMCcjUlt4owM9aVwC6YvIZyUbwW4BonKONx9Md4DBzSkRxg9HO1bHAahhSaKppusVNZ9wEfNql71aY0XOH66ZFkZ3bMyQIAubwA6Nu9uA/s400/google_plus_32.png" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="http://www.linkedin.com/in/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh19HK_UGyRbDZ_r4c6QCGF0UpVg21rNCE-VGWb_-PzbqCccJ6dgDVndUE6InnBj-nPRAyhMQAbYYk27SGBpbBa21cGH6bCKQSVu-AMHnTqpyEiK2OC0u3GBdG1UCpQFT2Tc0EVGmLa9_g/s400/linkedin_32.png" target="_blank" /></a><span style="text-align: -webkit-center;">&nbsp;</span><a href="https://twitter.com/#!/duhroach" style="text-align: -webkit-center;" target="_blank"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1aH7qGyYhalVNMvwl4jIGATbkL8NUkfRjDFZwVO1n32ANWufQWGkEx0KtPTWJ97yHW8Y7DXJixmu9NDwYOArkVajhz2Z7WSjYP9P8fI-nnLUCGQc0nb4WRqVTiLUOB-2pzvl2qxM8vJg/s400/twitter_32.png" /></a> </center> </div> </div> Unknownnoreply@blogger.com1