Transparent compression of RabbitMQ messages with Sneakers

When you’re sending a lot of messages into RabbitMQ message broker and have started to experience RabbitMQ slowdowns, maybe it’s time to think about message compression. Good news: if you’re using Sneakers, you can do it without any ado!

The main idea is to reduce the amount of data RabbitMQ operates by compressing and decompressing messages on the client’s side.

The great thing about Sneakers is that it allows us to define our own content type and implement serializer and deserializer causing the work to be as easy as a pie.

In the code snippet below we define a custom content type application/json+gzip and two lambdas, one to convert data string into its compressed representation and another one to turn compressed data back into the string it used to be. Here I use Zlib::GzipWriter for compression and Zlib::GzipReader for decompression but implementation is totally up to you.

Sneakers::ContentType.register(
  content_type: 'application/json+gzip',
  serializer: lambda do |payload|
    io = StringIO.new('w')
    w = Zlib::GzipWriter.new(io)
    w.write(payload)
    w.close
    w.string
  end,
  deserializer: lambda do |payload|
    io = StringIO.new(payload, 'rb')
    Zlib::GzipReader.new(io).read
  end
)

Make sure you’ve implemented custom content type support in the message publisher and the message consumer both!

Place this code into initializers/sneakers.rb if you are using Rails, or another suitable place if you don’t.

P.S.

A few months later, I discovered Google’s Brotli compression might be up to several times more effective for compressing json than the Gzip compression. Thanks to an awesome brotli gem, making use of it is a piece of cake:

Sneakers::ContentType.register(
  content_type: 'application/json+br',
  serializer: lambda do |payload|
    Brotli.deflate(payload)
  end,
  deserializer: lambda do |payload|
    Brotli.inflate(payload)
  end
)