I heard something about MsgPack that can be used for compressing json messages, but I have a question about that, why not simply using Gzip/Deflate instead of MsgPack? I think it's so better to debugging also.
MsgPack for compressing json data, why not using gzip/deflate
compressiondeflategzipmsgpack
Related Solutions
Why use deflate instead of gzip for text files served by Apache?
The simple answer is don't.
RFC 2616 defines deflate as:
deflate The "zlib" format defined in RFC 1950 in combination with the "deflate" compression mechanism described in RFC 1951
The zlib format is defined in RFC 1950 as :
0 1
+---+---+
|CMF|FLG| (more-->)
+---+---+
0 1 2 3
+---+---+---+---+
| DICTID | (more-->)
+---+---+---+---+
+=====================+---+---+---+---+
|...compressed data...| ADLER32 |
+=====================+---+---+---+---+
So, a few headers and an ADLER32 checksum
RFC 2616 defines gzip as:
gzip An encoding format produced by the file compression program "gzip" (GNU zip) as described in RFC 1952 [25]. This format is a Lempel-Ziv coding (LZ77) with a 32 bit CRC.
RFC 1952 defines the compressed data as:
The format presently uses the DEFLATE method of compression but can be easily extended to use other compression methods.
CRC-32 is slower than ADLER32
Compared to a cyclic redundancy check of the same length, it trades reliability for speed (preferring the latter).
So ... we have 2 compression mechanisms that use the same algorithm for compression, but a different algorithm for headers and checksum.
Now, the underlying TCP packets are already pretty reliable, so the issue here is not Adler 32 vs CRC-32 that GZIP uses.
Turns out many browsers over the years implemented an incorrect deflate algorithm. Instead of expecting the zlib header in RFC 1950 they simply expected the compressed payload. Similarly various web servers made the same mistake.
So, over the years browsers started implementing a fuzzy logic deflate implementation, they try for zlib header and adler checksum, if that fails they try for payload.
The result of having complex logic like that is that it is often broken. Verve Studio have a user contributed test section that show how bad the situation is.
For example: deflate works in Safari 4.0 but is broken in Safari 5.1, it also always has issues on IE.
So, best thing to do is avoid deflate altogether, the minor speed boost (due to adler 32) is not worth the risk of broken payloads.
UPDATE: Browsers have been dropping support for raw deflate. zOompf has completed some very thorough research on this very topic here. Unfortunately, it appears that raw deflate is NOT safe to use.
Check http://www.vervestudios.co/projects/compression-tests/results for more results.
Here are the browsers that have been tested:
/* Browser DEFLATE ZLIB */
XP Internet Explorer 6 PASS FAIL
XP Internet Explorer 7 PASS FAIL
XP Internet Explorer 8 PASS FAIL
Vista Internet Explorer 8 PASS FAIL
XP Firefox 3.6.* PASS PASS
XP Firefox 3.5.3 PASS PASS
XP Firefox 3.0.14 PASS PASS
Win 7 Firefox 3.6.* PASS PASS
Vista Firefox 3.6.* PASS PASS
Vista Firefox 3.5.3 PASS PASS
XP Safari 3 PASS PASS
XP Safari 4 PASS PASS
XP Chrome 3.0.195.27 PASS PASS
XP Opera 9 PASS PASS
XP Opera 10 PASS PASS
XP Sea Monkey 1.1.8 PASS PASS
Android 1.6 Browser (v4)* N/A N/A
OS-X Safari 4 PASS PASS
OS X Chrome 7.0.517.44 PASS PASS
OS X Opera 10.63 PASS PASS
iPhone 3.1 Safari PASS PASS
* Android Sends HTTP request header "Accept-Encoding: gzip". Deflate is not permitted.
I conclude that we can always send raw DEFLATE (when the HTTP request header "Accept-Encoding" contains "deflate") and the browser will be able to correctly interpret the encoded data. Can someone prove this wrong?
note: .NET's native implementation of DEFLATE (System.IO.Compression.DeflateStream) is raw DEFLATE. It also sucks. Please use zlib.net for all of your .NET deflating needs.
Best Answer
msgpack is an efficient binary serialization format that you may want to choose as an alternative to JSON according to your use case.
Even if it is advertised as like JSON, but fast and small, it is not intended to compress JSON data. Once again think of it as an alternative, and keep in mind that as stated by its author:
I would greatly recommend you to read this article - My thoughts on MessagePack, written by msgpack's author, that includes everything you need to know vs. recommended msgpack use cases, pros/cons and other useful pointers related to msgpack and JSON.