Rebalancer for RabbitMQ, SQS, etc.

Rebalancer 6.0 (forked from Jack Vanlightly and improved)

    Create Kafka style consumer groups in other technologies. Rebalancer was born of the need for consumer groups with RabbitMQ. But Rebalancer is completely technology-agnostic and will balance activity over any group of resources across a group of participating nodes.

Use cases

• Create Kafka-like "consumer groups" with messaging technologies like RabbitMQ, SQS, etc. On RabbitMQ 3.8+ read RabbitMQ: ordered consumer groups with Rebalancer or with super streams first: single active consumer and super streams now cover much of this natively.
• Consume a group of resources such as file shares, FTPs, S3 buckets between the instances of a scaled-out application.
• Single Active-Consumer / Active-Backup.
• Create an application cluster that consumes a single resource in a highly available manner. The cluster leader (Coordinator) consumes the single resource, and the other nodes (Followers) remain idle in backup in case the leader dies.

Balancing assignment of multiple resources over multiple nodes

Consuming from a single resource with backup nodes in case of failure

Concepts

    The repository contains Rebalancer's .NET implementation:

• protocols for resource-allocation algorithms
• TLA+ specifications that verify the protocols
• code libraries for the supported backends

    The important terms are:

• Resource Group = Group of Nodes + Group of Resources

    When a node becomes active, it notifies the other nodes. One node is the Coordinator (leader) and the rest are Followers. The Coordinator has the job to assign resources to nodes. It monitors the coming and going of nodes, as well as changes to the number of resources available to the resource group. When any change happens to the nodes or resources, the Coordinator triggers a rebalancing.

    ZooKeeper supports two rebalancing modes: a resource barrier and a global barrier. In ZooKeeper's global-barrier mode:

1. The Coordinator orders all Followers to stop activity.
2. Once activity has stopped, the Coordinator distributes the resource identifiers equally between the Followers and itself.
3. The final step is that the Coordinator notifies each Follower of the resources it has been assigned and can start its activity (consuming, reading, writing, etc.).

    SQL Server, PostgreSQL and Redis share one lease-based coordinator/follower protocol (Rebalancer.LeaseCoordination): the Coordinator holds a lease with a fencing token, stops every Follower, then grants each one its resources, and every write is guarded by that lease. It is described in the architecture overview.

    Leader election determines who the Coordinator is. If the Coordinator dies, then a Follower takes its place. Leader election and metadata storage are performed via a backing metadata store; this library implements ZooKeeper, Redis (also on Valkey), SQL Server (serializable transactions) and PostgreSQL (row locks). Other consensus services (etcd, Consul) or RDBMSs (Oracle, MySQL) would fit the same pattern but are not implemented. All communication between nodes is also performed via this backing metadata store.

Languages and Backends

    This repository is the .NET implementation: Rebalancer.Core, Rebalancer.LeaseCoordination (shared by SQL Server, Redis and PostgreSQL), one package for each of the four backends (Rebalancer.ZooKeeper, Rebalancer.Redis, Rebalancer.SqlServer, Rebalancer.PostgreSql), and optional packages for dependency injection (Rebalancer.Extensions.DependencyInjection) and health checks (Rebalancer.Extensions.Diagnostics.HealthChecks). NuGet installs Rebalancer.LeaseCoordination with the SQL Server, Redis and PostgreSQL packages.

    Supported servers: Redis (tested on 7.0) and Valkey (tested on 8.1), SQL Server (tested on 2022), PostgreSQL (tested on 17.8) and ZooKeeper (tested on 3.8). The packages target .NET 10. Other languages would need their own implementation of the protocol.

Getting started

    Reference Rebalancer.Extensions.DependencyInjection and register the backend you use, for example:

var services = new ServiceCollection();
services.AddRebalancerSqlServer(connectionString);
await using var provider = services.BuildServiceProvider();
var client = provider.GetRequiredService<RebalancerClient>();

    Alternatively use AddRebalancerPostgreSql, AddRebalancerRedis or AddRebalancerZooKeeper. Without dependency injection, pass the provider explicitly: new RebalancerClient(new SqlServerProvider(connectionString)). A client raises OnAssignment with the resources it now owns, OnUnassignment before they move elsewhere, and OnAborted on errors.

Metrics, traces and health

    Rebalancer emits metrics and traces through System.Diagnostics; collect them with OpenTelemetry:

builder.Services.AddOpenTelemetry()
    .WithMetrics(m => m.AddMeter(RebalancerTelemetry.MeterName))
    .WithTracing(t => t.AddSource(RebalancerTelemetry.ActivitySourceName));

    The package Rebalancer.Extensions.Diagnostics.HealthChecks adds a health check for a client your host started: services.AddHealthChecks().AddRebalancerClient("rebalancer", client).

Verified and tested

• The coordination protocols are specified in TLA+ and model-checked with TLC: no resource is processed by two nodes at once. A node that stalls longer than its lease (a long GC pause, a frozen VM) stops at its first heartbeat after it resumes, so keep the lease expiry well above the longest pause a node can have.
• Every backend runs the same integration scenarios against real servers (Docker): assignment safety, paused and crashed nodes, slow and lossy networks, and randomised runs that check what nodes actually process.
• Every release is gated by the unit tests and a dependency vulnerability scan; pull requests also run the integration suites.

What's new in 6.0

• Redis and Valkey: every guarded write is a server-side Lua script, so a group settles reliably even over slow links. A per-group ACL user needs +eval +evalsha; see the migration guide.
• Eight types that were never part of the supported API (deprecated in 5.2) are now internal.
• Earlier: a PostgreSQL backend and Valkey support (5.0), shared lease coordination for SQL Server, PostgreSQL and Redis (5.0), and metrics, traces and health checks (4.1). All release notes are in the repository.

    The source code can be found on GitHub at:

https://github.com/dradoaica/Rebalancer

Enjoy (ง'̀-'́)ง