Java – Achieving JMS/AMQP messaging patterns using Redis

amqpjavajmsmessagingredis

This question arises as I came across some mentions (such as this), about using a Messaging software such as ZeroMQ alongwith Redis, but I keep hearing about Redis itself used a messaging-system. So, if Redis is used along with other messaging systems, does it mean Redis has some serious deficiencies when used as a messaging system by itself ?

While the use of Redis for caching and pub/sub is clear to me, it is not clear if Redis can be used in place of a full-fledged messaging system such as JMS, AMQP or ZeroMQ.
Leaving alone the standards-compliance aspect and only concentrating on the functionality/features, does Redis provide support for all the messaging patterns/models required of a messaging system ?

The messaging patterns I am talking about are :

RPC/Request-reply (an
example
using ActiveMQ/JMS and another using RabbitMQ/AMQP)
Pipeline/Work queues (once and atmost once consumption of each message)
Broadcast (everyone subscribed to the channel)
Multicast (filtering of messages at server based on consumers' selectors)
Any other messaging pattern ?

If yes, then Redis seems to solve two (possibly more) aspects at once: Caching and Messaging.

I am looking at this in the context of a web-application backed by a Java/Java EE server.
And I am looking at this not from a proof-of-concept point-of-view but from a large-scale software development angle.

Edit1:

user:791406 asked a valid question:

"Who cares if redis supports those patterns; will redis meet your SLA
and QoS needs?"

I thought it is better to give this detail as part of the question instead of in the comments section.

My current needs are less to do with SLA and QOS and more to do with choosing a tool for my job (messaging) that I can use even when my requirements grow (reasonably) in future.
I am starting with simplistic requirements at first and we all know requirements tend to grow.
And NO, I am not looking for a one tool that does it all.
I just want to know if Redis fulfills the usual requirements expected out of a messaging system, like ActiveMQ/RabbitMQ does. Ofcourse, if my SLA/QOS needs are extreme/eccentric, I would need to get a special tool for satisfying that. For ex: In some cases ZeroMQ could be chosen over RabbitMQ due to specific SLA requirements. I am not talking about such special requirements. I am focusing on average enterprise requirements.

I was afraid (based on my little understanding) that eventhough redis could be used as a basic tool for my messaging needs of today, it might be the wrong tool for a real messaging job in future. I have experiences with messaging systems like ActiveMQ/RabbitMQ and know that they could be used for simple to (reasonably) complex messaging needs.

Edit2:

The redis website mentions "Redis is often used as a messaging server" but how to achieve the messaging patterns is not clear.
Salvatore sanfilippo mentions Redis users tend to use it as a database, as a messaging bus, or as a cache. To what extent can it serve as a "messaging bus" is not clear.
When I was trying to find out what messaging requirements of JMS that redis doesnt support, I came across something that Redis supports but JMS doesnt: Pattern-matching subscriptions i.e Clients may subscribe to glob-style patterns in order to receive all the messages sent to channel names matching a given pattern.

Conclusion:

I have decided to use JMS for my messaging needs and use Redis for Caching.

Best Answer

What are you needs?

I think the question you should be asking yourself is "what quality of messaging do I need to support my application?" before deciding on a messaging platform. Who cares if redis supports those patterns; will redis meet your SLA and QoS needs? Focus on that first, then make your technology decision based on that assessment.

Having said that, I'll offer my input on how you can make that decision...

Highly-Reliable/Persistent/Durable Messaging
Let's take an extreme case: say you're building a trading or financial application. Such applications demand strict SLA's where message persistence, reliability, exactly-once delivery and durability are paramount. Using redis as the messaging backbone for this case is probably a bad choice, lots of reasons why...

message redelivery (when the sh*t hits the fan)
message store replication when redis goes down
message transactions (redis can't do XA)
producer/subscriber fault tolerance and disconnection resilience
message order sequencing
sending messages when broker is down (store-and-forward)
single-threaded redis may become bottleneck

If your system has strict SLA's, some or all of these issues will most definitely arise, so how will you handle these limitations? You can implement custom code around redis to address some issues, but why bother when mature messaging platforms like ActiveMq, WebsphereMQ, and WebLogic JMS offer persistence, reliability, and fault tolerance? You said you're on a Java/Java EE stack, so you're in a position to use some of the most robust messaging frameworks available, either open source or commercial. If you're doing financial transactions, then you need to consider these options.

High Performance/Large Distributed Systems Messaging
If you're building a social network or gaming platform where you want performance over reliability, ZeroMq is probably a good fit. It's a socket communication library wrapped in a messaging-like API. It's decentralized (no broker), very fast, and highly resilient and fault tolerant. If you need to do things like N-to-N pub/sub with broker intermediaries, flow control, message persistence, or point-to-point synchronization, ZeroMq offers the necessary facilities and code samples to do it all with minimal code while avoiding building solutions from the ground up. It's written in C but has client libraries for nearly every popular language.

Hope that helps...

Related Solutions

How to atomically delete keys matching a pattern using Redis

Execute in bash:

redis-cli KEYS "prefix:*" | xargs redis-cli DEL

UPDATE

Ok, i understood. What about this way: store current additional incremental prefix and add it to all your keys. For example:

You have values like this:

prefix_prefix_actuall = 2
prefix:2:1 = 4
prefix:2:2 = 10

When you need to purge data, you change prefix_actuall first (for example set prefix_prefix_actuall = 3), so your application will write new data to keys prefix:3:1 and prefix:3:2. Then you can safely take old values from prefix:2:1 and prefix:2:2 and purge old keys.

Java – When to use actors instead of messaging solutions such as WebSphere MQ or Tibco Rendezvous

First off the "older" message systems (MQ) are older in implementation but they are a newer in engineering idea of: transactional persistent queues. Scala Actors and Akka maybe a newer implementation but are built on an older concurrency model of Actors.

The two models however end up being very similar in practice because they both are event message based: See my answer to RabbitMQ vs Akka.

If you're going to code only for the JVM then Akka is probably a good choice. Otherwise I would use RabbitMQ.

Also if you're a Scala developer, then Akka should be a no-brainer. However Akka's Java bindings are not very Java-ish and require casting due to Scala's type system.

Also in Java people don't typically make immutable objects which I recommend you do for messaging. Consequently its very easy in Java to accidentally do something using Akka that will not scale (using mutable objects for messages, relying on weird closure callback state). With MQ this is not a problem because the messages are always serialized at the cost of speed. With Akka they are generally not.

Akka also scales better with large amount of consumers than most MQ. This is because for most MQ (JMS, AMQP) clients every queue connection requires a thread... thus lots of queues == lots of permanently running threads. This is mainly a client issue though. I think ActiveMQ Apollo has a non-blocking dispatcher that purportedly fixes that issue for AMQP. The RabbitMQ client has channels that allow you to combine multiple consumers but there are still issues with large number of consumers potentially causing deadlocks or connections to die so generally more threads are added to avoid this issue.

That being said Akka's remoting is rather new and probably still doesn't offer all the reliable message guarantees and QoS that traditional message queues provide (but that is changing everyday). Its also generally peer-to-peer but does I think support server-to-peer which is generally what most MQ systems do (ie single point of failure) but there are MQ systems that are peer-to-peer (RabbitMQ is server-to-peer).

Finally RabbitMQ and Akka actually make a good pair. You can use Akka as a wrapper to RabbitMQ particularly since RabbitMQ does not help you with handling the consumption of messages and routing the messages locally (in a single JVM).

When to choose Akka

Have lots of consumers (think millions).
Need low latency
Open to the Actor concurrency model

Example system: An interactive real time chat system

When to choose MQ

Need to integrate with lots of different systems (ie non JVM)
Message reliability is more important than latency
Would like more tools and admin UI
Because of previous points better for long running tasks
Would like to use a different concurrency model than Actors

Example system: A scheduled transactional batch processing system

EDIT based on concerned comments

I made an assumption that the OP was concerned with distributed processing which both Akka and Message Queues can handle. That is I assumed he was talking about distributed Akka. Using Akka for local concurrency is an apples to orange comparison to most message queues. I say most because you can apply the message queue model locally as a concurrency model (ie topic, queues, exchanges) which both the Reactor library and simple-react do.

Picking the right concurrency model/library is very important for low latency applications. A distributed processing solution such as a message queue is generally not ideal because the routing is almost always done over the wire which is obviously slower than within application and thus Akka would be a superior choice. However I believe some proprietary MQ technologies allow for local routing. Also as I mentioned earlier most MQ clients are pretty stupid about threading and do not rely on non-blocking IO and have a thread per connection/queue/channel... ironically non-blocking io is not always low latency but is generally more resource efficient.

As you can see the topic of distributed programming and concurrent programming is rather large and changing everyday so my original intention was not confuse but rather focus on one particular area of distributed message processing which is what I though the OP was concerned with. In terms of concurrency one might want to focus their searches on "reactive" programming (RFP / streams) which is a "newer" but similar model to the actor model and message queue model of which all of these models can be generally combined because they are event based.

Best Answer

Related Solutions

How to atomically delete keys matching a pattern using Redis

Java – When to use actors instead of messaging solutions such as WebSphere MQ or Tibco Rendezvous

Related Topic