Message Broker for Microservices

Microservices uses asynchronous communication commonly through message broker. Each Service handles and collaborates with other services in order to perform on a given request coming from the Client via inter-communication among them. A broker ensures communication between different microservices is reliable and stable, that the messages are managed and monitored within the system and that messages don’t get lost. There are a few message brokers you can choose from, varying in scale and data capabilities. This post will compare the three most popular brokers:

• RabbitMQ,

• Kafka, and

• Redis.

What is Microservices Communication?

There are two common ways microservices communicate with each other: Synchronous and Asynchronous.

In Synchronous communication, The requester/consumer/caller waits for a response before sending the next message - preferably operates as REST over HTTP. In Asynchronous communication, the messages are sent without waiting for a response. This is suitable for a distributed system and usually requires a message broker to manage the messages.

Styles of Communication

• Notifications — a sender sends a message to a recipient but does not expect any/nor even get any reply.

• Request/response — a service sends a request message to a recipient and expects to receive a reply of the message promptly.

• Request/asynchronous response — a service sends a request message to a recipient and expects to receive a reply of the message eventually.

• Publish/subscribe — a service publishes a message to zero or more recipients

• Publish/asynchronous response — a service publishes a request to one or recipients, some of whom send back a reply but some don’t.

Choosing the right type of communication involves on various parameters :

• Structure of microservice,

• infrastructure that in place,

• latency acceptance,

• scale requirement,

• dependencies, and

• purpose of communication

Advantages of Asynchronous Communication

• They are non-blocking by definition.

• It supports scale better than synchronous operations

• Provides recovery mechanism on microservice event crashes

• Don't really know about services involved each other

• Better decoupling of services

• Increases the capability of central discovery, monitoring, load-balancing, and even policy enforcer

• Enhances flexibility and scalability

Asynchronous is usually managed through message broker. Choosing the right the message broker should have below considerations

1. Scaling need of the Broker - Number of messages sent per second

2. Data Persistence - Message recoverability on failure

3. Consumer Capability - Managing the toplogies of the consumer structure. i.e.., One - to - one and/or one-to-many consumers

RabbitMQ

It was released in 2007 and one of the first common message brokers to be created. It's an open source that delivers messages through point-to-point and publish-subscribe mechanism through AMQP (Advanced Message Queuing Protocol). It is not part of any cloud platform serice provides as SaaS unlike Kafka. It supports all major languages - Python, Java, .Net, PHP, Ruby, Javascript, Go etc..,

RabbitMQ expects some performance issues when in the persistent mode.

Where to use: complex routing and support when the required rate is not high.

Kakfa

It was created by LinkedIn in 2011 to handle high throughput and low latency. Kafka replicates the publish-subscribe service for distrbuted streaming platform. It provides data persistency and store streams of records. It has a managed SaaS on Cloud platforms. It supports all major programming languages - Python, Java, C,C++, Javascript, Go, Ruby, .Net, PHP etc,,

Where to use : Large amounts of Data. It is a high throughput distributed queue and ideal to use for one- to -many use cases where persistene is requires

Redis

Redis is differnet from the other message brokers. It is an in-memory data store that can be used as either a high-performance key-value store or a message broker. It has no persistency and it just dumps the data memory into a Disk/Db. It is suited well for real -time processing.

Where to use: Largely suited for short-lived messages. It is the perfect candidate for short retention messages where persistence isn’t so important and you can tolerate some loss

Any message queue that allows publishing messages decoupled from consuming them is effectively acting as a storage system for the in-flight messages. What is different about Kafka is that it is a very good storage system. It provides data persistency and stores streams of records that render it capable of exchanging quality