A while ago I blogged about [supporting BSON][1] at your Web API end points. A good alternative to both JSON and BSON has, for quite a while now, been [MessagePack][2].
The main problem with supporting MessagePack in your Web API has been the fact that MessagePack required a strongly typed serializer (in other words, you needed to tell the serializer what type you serialize and deserialize at compile time). There was no easy way to provide support for boxed objects (untyped context) - and Web API media type formatters run against such generic object instances. In face, to be precise, writing to a stream (serializing) has never been a big problem, but reading was very complicated. An easy solution was a type whitelist and a massive if-else block, but such approaches are hardly good ideas.
Four days ago, Yusuke Fujiwara, one of the wizards behind MessagePack for CLI, [added support][3] for untyped serializer factory, effectively inviting all of us to start using MessagePack in Web API.
What is MessagePack? π
MessagePack is a binary serialization format. It’s very fast, and very space efficient - it’s more compact than both JSON and BSON. Moreover, it can serialize pretty much anything that can be represented in a JSON, and, through external ports, is supported in all mainstream languages.
There are quite a few reasons for these performance/space advantages, mainly due to the way common data types are being stored, for example:
- JSON uses 4 bytes to represent null, MessagePack only requires 1 byte
- JSON uses 2 bytes to represent a typical int, MEssagePack requires 1 byte
and so on. - Also, since it’s binary, it’s faster to read and write than JSON. There is a great article available [here][4], showing how well MessagePack performs in comparison to JSON and even the “sexy” Protocol Buffers
You can read more about MessagePack at the [official website][2].
Prerequisites π
To get started you’d need the MessagePack for CLI library, but not the one from Nuget (that one is old and only supports strongly typed serialization and deserialization) but the latest one. You can either [grab it from GitHub][5] and build yourself, or you could download the ZIP-ped DLL directly, from [my dropbox][6].
Creating the formatter π
If you follow this blog (or you are semi-familiar with Web API), MediaTypeFormatters should be no mystery to you. They allow us to support various media types and expose data via our Web API endpoint in various formats.
Let’s create the formatter:
public class MessagePackMediaTypeFormatter : MediaTypeFormatter
{
private const string mime = "application/x-msgpack";
Func<Type, bool> IsAllowedType = (t) =>
{
if (!t.IsAbstract && !t.IsInterface && t != null && !t.IsNotPublic)
return true;
if (typeof(IEnumerable).IsAssignableFrom(t))
return true;
return false;
};
public MessagePackMediaTypeFormatter()
{
SupportedMediaTypes.Add(new MediaTypeHeaderValue(mime));
}
public override bool CanReadType(Type type)
{
if (type == null) throw new ArgumentNullException("type is null");
return IsAllowedType(type);
}
public override bool CanWriteType(Type type)
{
if (type == null) throw new ArgumentNullException("Type is null");
return IsAllowedType(type);
}
public override Task WriteToStreamAsync(Type type, object value, Stream stream, HttpContent content, TransportContext transportContext)
{
//TODO
}
public override Task
<object>
ReadFromStreamAsync(Type type, Stream stream, HttpContent content, IFormatterLogger formatterLogger)<br /> {<br /> //TODO<br /> }<br /> }<br /> ```</p>
<p>
I left the two core methods empty on purpose for now. For now we have the supported media type headers - <i>application/x-msgpack</i> and the serialization rules - we only support public, non-abstract, non-interface types - except IEnumerable.
</p>
<h3>
Write to stream
</h3>
<p>
```csharp
<br /> public override Task WriteToStreamAsync(Type type, object value, Stream stream, HttpContent content, TransportContext transportContext)<br /> {<br /> if (type == null) throw new ArgumentNullException("type is null");<br /> if (stream == null) throw new ArgumentNullException("Write stream is null");
</p>
<p>
var tcs = new TaskCompletionSource
<object>
();</p>
<p>
if (typeof(IEnumerable).IsAssignableFrom(type)) {<br /> value = (value as IEnumerable
<object>
).ToList();<br /> }</p>
<p>
var serializer = MessagePackSerializer.Create<dynamic>();<br /> serializer.Pack(stream, value);
</p>
<p>
tcs.SetResult(null);<br /> return tcs.Task;<br /> }<br /> ```
</p>
<p>
In this case I'm actually using a strongly typed serializer, against <i>object</i> (or rather, against <i>dynamic</i>, but that's effectively object as well). As I wrote in the beginning, that feature has always worked without problems, it's the read from stream that was problematic. Not much exciting happens here besides that; we call the void <i>Pack</i> on the serializer, which takes care of writing to our output stream.
</p>
<p>
Note, because MessagePack is having troubles with interface-based collections, we flush them to list if we deal with such.
</p>
<h3>
Read from stream
</h3>
<p>
```csharp
<br /> public override Task
<object>
ReadFromStreamAsync(Type type, Stream stream, HttpContent content, IFormatterLogger formatterLogger)<br /> {<br /> var tcs = new TaskCompletionSource
<object>
();<br /> if (content.Headers != null && content.Headers.ContentLength == 0) return null;<br /> try<br /> {<br /> var serializer = MessagePackSerializer.Create(type);<br /> object result;</p>
<p>
using (var unpacker = Unpacker.Create(stream))<br /> {<br /> unpacker.Read();<br /> result = serializer.UnpackFrom(unpacker);<br /> }<br /> tcs.SetResult(result);<br /> }<br /> catch (Exception e)<br /> {<br /> if (formatterLogger == null) throw;<br /> formatterLogger.LogError(String.Empty, e.Message);<br /> tcs.SetResult(GetDefaultValueForType(type));<br /> }
</p>
<p>
return tcs.Task;<br /> }<br /> ```
</p>
<p>
In the read portion, we use the new untyped <i>MessagePackSerializer</i>, and in conjuction with an <i>Unpacker</i> object, we read the stream into the generic <i>object</i>.
</p>
<p>
That stream becomes an <i>object</i> but will be unboxed by Web API into our CLR type.
</p>
<h3>
Using it
</h3>
<p>
Now a quick example on how you'd use if from your .NET client. For the record, this is the simple model I used:<br /> ```csharp
<br /> public class Url<br /> {<br /> public int UrlId { get; set; }<br /> public string Address { get; set; }<br /> public string Title { get; set; }<br /> public string Description { get; set; }<br /> public DateTime CreatedAt { get; set; }<br /> public string CreatedBy { get; set; }<br /> }<br /> ```
</p>
<p>
```csharp
<br /> var client = new HttpClient();<br /> var request = new HttpRequestMessage(HttpMethod.Get, "http://localhost:49745/api/values/1");<br /> request.Headers.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/x-msgpack"));<br /> var result = client.SendAsync(request).Result;<br /> var serializer = MessagePackSerializer.Create<WebApi.MessagePack.Models.Url>();<br /> Url data = serializer.Unpack(result.Content.ReadAsStreamAsync().Result);<br /> ```
</p>
<p>
Note, in this case, since we are consuimg it from the client, we can easily use the typed serializer - in this case against my test <i>Url</i> model (I always use "var", but on the last line I purposefully used the Type indicator, to show at which point our CLR object emerges).
</p>
<p>
In a similar manner, we'd retrieve a collection:<br /> ```csharp
<br /> var client = new HttpClient();<br /> var request = new HttpRequestMessage(HttpMethod.Get, "http://localhost:49745/api/values/");<br /> request.Headers.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/x-msgpack"));<br /> var result = client.SendAsync(request).Result;<br /> var serializer = MessagePackSerializer.Create<List<WebApi.MessagePack.Models.Url>>();<br /> List<Url> data = serializer.Unpack(result.Content.ReadAsStreamAsync().Result);<br /> ```
</p>
<p>
Finally, if we want to send some data to the Web API in MessagePAck format:<br /> ```csharp
<br /> var request = new HttpRequestMessage(HttpMethod.Post, "http://localhost:49745/api/values/");<br /> request.Content = new ObjectContent<Url>(<br /> new Url() {<br /> Address = "http://www.strathewb.com",<br /> Title = "Test",<br /> CreatedAt = DateTime.Now,<br /> CreatedBy = "Filip",<br /> Description = "dummy" },<br /> new MessagePackMediaTypeFormatter());<br /> request.Content.Headers.ContentType.MediaType = "application/x-msgpack";<br /> StatusCode result5 = client.SendAsync(request5).Result.StatusCode; //204, POST is void<br /> ```
</p>
<p>
In this case, we use our <i>MediaTypeFormatter</i> to help us with serialization to send over the wire.
</p>
<p>
As you can see, even with a small amounts of data - retrieving a collection of 5 objects:
</p>
<p>
<a href="/images/2012/09/comparison.png"><img src="/images/2012/09/comparison.png" alt="" title="comparison" width="620" height="50" class="aligncenter size-full wp-image-505" /></a><br /> 1st MessagePack, 2nd JSON, 3rd XML
</p>
<p>
and retreving a single object and again 5 objects:
</p>
<p>
<a href="/images/2012/09/comparison2.png"><img src="/images/2012/09/comparison2.png" alt="" title="comparison2" width="413" height="111" class="aligncenter size-full wp-image-506" srcset="/images/2012/09/comparison2.png 413w, /images/2012/09/comparison2-300x81.png 300w" sizes="(max-width: 413px) 100vw, 413px" /></a>
</p>
<p>
MessagePack outperforms JSON in terms of size by about 15%. The gain would be even bigger if we used arrays and different types of ints in the model, as that's the stuff MessagePack excells at.
</p>
<h3>
Summary and source
</h3>
<p>
MessagePack is a really viable alternative to JSON. Moreover, with the way Web API works, it's really not a big effort to throw in support for MessagePack in your API as an extra. I have used MessagePack to communicate between .NET clients and it has always worked great.
</p>
<p>
There are implementations of MessagePack serializers for all major languages, <a href="https://github.com/msgpack/msgpack-javascript">including JavaScript</a>, so theoretically you could use that even in the browser (although, to be honest I have never tried that).
</p>
<p>
Anyway, I hope you'd find this post useful!
</p>
<p>
- <a href="https://gist.github.com/3693339">source (gist)</a>
</p>
[1]: /2012/07/bson-binary-json-and-how-your-web-api-can-be-even-faster/
[2]: http://msgpack.org/
[3]: https://github.com/msgpack/msgpack-cli/commit/8c8b4d7a4f74b96156226a4034f95b30f6a69d2d
[4]: http://www.rubyinside.com/messagepack-binary-object-serialization-3150.html
[5]: https://github.com/msgpack/msgpack-cli/
[6]: https://dl.dropbox.com/u/23961623/MsgPack-20120910.zip