Parameters and Policies

Overview

While much of the configuration for RabbitMQ lives in the configuration file, some things do not mesh well with the use of a configuration file:

• If they need to be the same across all nodes in a cluster
• If they are likely to change at run time

RabbitMQ calls these items parameters. Parameters can be set by invoking rabbitmqctl or through the HTTP API.

There are two kinds of parameters: vhost-scoped parameters and global parameters. Vhost-scoped parameters are tied to a virtual host and consist of a component name, a name and a value.

Global parameters are not tied to a particular virtual host and they consist of a name and value.

One special case of parameters usage is policies.

Policies is the recommended way of specifying optional arguments for groups of queues and exchanges, as well as plugins such as Federation and Shovel.

Policies are vhost-scoped.

Operator policies allow cluster operators override certain arguments defined in regular policies. This special policy type was designed for defining guardrails, for example, limiting maximum queue size, enforcing a quorum queue and stream initial replication factor, and so on.

Operator policies are particularly important in environments where RabbitMQ is offered as a service, that is, where it is operated by one team but consumed by multiple other teams or external customers.

Global and Per-vhost Parameters

As stated above, there are vhost-scoped parameters and global parameters. An example of vhost-scoped parameter is a federation upstream: it targets a component (federation-upstream), it has a name that identifies it, it's tied to a virtual host (federation links will target some resources of this virtual host), and its value defines connection parameters to an upstream broker.

Vhost-scoped parameters can be set, cleared and listed:

rabbitmqctl	rabbitmqctl set_parameter [-p vhost] <component_name> <name> <value> rabbitmqctl clear_parameter [-p vhost] <component_name> <name> rabbitmqctl list_parameters [-p vhost]
HTTP API	PUT /api/parameters/{component_name}/{vhost}/{name} DELETE /api/parameters/{component_name}/{vhost}/{name} GET /api/parameters

Global parameters is the other kind of parameters. An example of a global parameter is the name of the cluster. Global parameters can be set, cleared and listed:

rabbitmqctl	rabbitmqctl set_global_parameter <name> <value> rabbitmqctl clear_global_parameter <name> rabbitmqctl list_global_parameters
HTTP API	PUT /api/global-parameters/name DELETE /api/global-parameters/name GET /api/global-parameters

Since a parameter value is a JSON document, you will usually need to quote it when creating one on the command line with rabbitmqctl. On Unix it is usually easiest to quote the whole document with single quotes, and use double quotes within it. On Windows you will have to escape every double quote. We give examples for both Unix and Windows for this reason.

Parameters reside in the database used by RabbitMQ for definitions of virtual hosts, exchanges, queues, bindings, users and permissions. Parameters are exported along with other object definitions by the management plugin's export feature.

Vhost-scoped parameters are used by the federation and shovel plugins. Global parameters are used by the MQTT plugin.

Policies

Policies is the recommended way of configuring optional arguments for queues, exchanges, and some plugins.

Two notable exceptions are the queue type and the maximum number of priorities of classic queues. Those values intentionally cannot be configured by policies: their values are fixed at queue declaration time.

Why Policies Exist

Before we explain what policies are and how to use them it would be helpful to explain why they were introduced to RabbitMQ.

In addition to mandatory properties (e.g. durable or exclusive), queues and exchanges in RabbitMQ have optional arguments, sometimes referred to as x-arguments.

Those are provided by clients when they declare queues (exchanges) and control various optional features, such as queue length limit or TTL.

Client-controlled properties in some of the protocols RabbitMQ supports generally work well but they can be inflexible: updating TTL values or mirroring parameters that way required application changes, redeployment and queue re-declaration (which involves deletion). In addition, there is no way to control the extra arguments for groups of queues and exchanges. Policies were introduced to address the above pain points.

A policy matches one or more queues by name (using a regular expression pattern) and appends its definition (a map of optional arguments) to the x-arguments of the matching queues. In other words, it is possible to configure x-arguments for multiple queues at once with a policy, and update them all at once by updating policy definition.

In modern versions of RabbitMQ the set of features which can be controlled by policy is not the same as the set of features which can be controlled by client-provided arguments.

How Policies Work

Key policy attributes are

• name: it can be anything but ASCII-based names without spaces are recommended
• pattern: a regular expression that matches one or more queue (exchange) names. Any regular expression can be used.
• definition: a set of key/value pairs (think a JSON document) that will be injected into the map of optional arguments of the matching queues and exchanges
• policy priority used to determine which policy should be applied to a queue or exchange if multiple policies match its name

Policies automatically match against exchanges and queues, and help determine how they behave. Each exchange or queue will have at most one policy matching (see Combining Policy Definitions below), and each policy then injects a set of key-value pairs (policy definition) on to the matching queues (exchanges).

Policies can match only queues of a specific type, all queues, only exchanges, or all queues and exchanges. This is controlled using the apply-to flag when a policy is created.

Policies can change at any time. When a policy definition is updated, its effect on matching exchanges and queues will be reapplied. Usually it happens instantaneously but for very busy queues can take a bit of time (say, a few seconds).

Policies are matched and applied every time an exchange or queue is created, not just when the policy is created.

Policies can be used to configure

• federation
• alternate exchanges
• dead lettering,
• per-queue TTLs
• queue length limit

and other features.

An example of defining a policy looks like:

rabbitmqctl	rabbitmqctl set_policy federate-me \ "^federated\." '{"federation-upstream-set":"all"}' \ --priority 1 \ --apply-to exchanges
rabbitmqctl (Windows)	rabbitmqctl.bat set_policy federate-me ^ "^federated\." "{""federation-upstream-set"":""all""}" ^ --priority 1 ^ --apply-to exchanges
HTTP API	PUT /api/policies/%2f/federate-me {"pattern": "^federated\.", "definition": {"federation-upstream-set":"all"}, "priority": 1, "apply-to": "exchanges"}
Web UI	Navigate to Admin > Policies > Add / update a policy. Enter "federate-me" next to Name, "^federated." next to Pattern, and select "Exchanges" next to Apply to. Enter "federation-upstream-set" = "all" in the first line next to Policy. Click Add policy.

This matches the value "all" with the key "federation-upstream-set" for all exchanges with names beginning with "federated.", in the virtual host "/".

The "pattern" argument is a regular expression used to match exchange or queue names.

In the event that more than one policy can match a given exchange or queue, the policy with the greatest priority applies.

The "apply-to" argument can be one of the following:

• "exchanges", applies to exchanges only
• "queues", applies to all types of queues, including streams
• "classic_queues", applies to classic queues only
• "quorum_queues", applies to quorum queues only
• "streams", applies to streams only
• "all", applies to all exchanges and queues (including streams)

The "apply-to" and "priority" settings are optional. The default values are "all" and "0" respectively.

Policy Priorities

Policy patterns are matched against exchange and queue names to determine what policy (if any) should then inject a set of key-value pairs (the definition of that policy) into the optional arguments of matching queues (exchanges).

At most one policy matches a queue or exchange. Since multiple policies can match a single name, a mechanism is needed to resolve such policy conflicts. This mechanism is called policy priorities. Every policy has a a numeric priority associated with it. This priority can be specified when declaring a policy. If not explicitly provided, the priority of 0 will be used.

Matching policies are then sorted by priority and the one with the highest priority will take effect.

When multiple policies match an entity and they all have equal priorities, the effective one will be chosen undeterministically. Such cases should be avoided by paying attention to what priorities various policies use.

Combining Policy Definitions

In some cases we might want to apply more than one policy definition to a resource. For example we might need a queue to be federated and has message TTL. At most one policy will apply to a resource at any given time, but we can apply multiple definitions in that policy.

A federation policy definition would require an upstream set to be specified, so we would need the federation-upstream-set key in our definition. On the other hand to define some queues as TTL-enabled, we would need the TTL-related keys key to be defined as well for the policy. The policy definition is just a JSON object and can have multiple keys combined in the same policy definition.

Here's an example:

rabbitmqctl	rabbitmqctl set_policy ttl-fed \ "^tf\." '{"federation-upstream-set":"all", "message-ttl":60000}' \ --priority 1 \ --apply-to queues
rabbitmqctl (Windows)	rabbitmqctl set_policy ttl-fed ^ "^tf\." "{""federation-upstream-set"":""all"", ""message-ttl"":60000}" ^ --priority 1 ^ --apply-to queues
HTTP API	PUT /api/policies/%2f/ttl-fed {"pattern": "^tf\.", "definition": {"federation-upstream-set":"all", "message-ttl":60000}, "priority": 1, "apply-to": "queues"}
Web UI	Navigate to Admin > Policies > Add / update a policy. Enter "ttl-fed" next to Name, "^tf." next to Pattern, and select "Queues" next to Apply to. Enter "federation-upstream-set" = "all" in the first line next to Policy. Enter "ha-mode" = "exactly" and "ha-params" = 2 on the following form lines. Click Add policy.

By doing that all the queues matched by the pattern "^tf\." will have the "federation-upstream-set" and the policy definitions applied to them.

Operator Policies

Difference From Regular Policies

Sometimes it is necessary for the operator to enforce certain policies. For example, it may be desirable to force queue TTL but still let other users manage policies. Operator policies allow for that.

Operator policies are much like regular ones but their definitions are used differently. They are merged with regular policy definitions before the result is applied to matching queues.

Because operator policies can unexpectedly change queue attributes and, in turn, application assumptions and semantics, they are limited only to a few arguments:

	Classic	Quorum	Stream
delivery-limit		✓
expires	✓	✓
ha-mode	✓
ha-params	✓
ha-sync-mode	✓
max-in-memory-bytes		✓
max-in-memory-length		✓
max-length	✓	✓
max-length-bytes	✓	✓	✓
message-ttl	✓	✓
target-group-size		✓

Conflict Resolution with Regular Policies

An operator policy and a regular policy can contain the same keys in their definitions. When it happens, the more conservative value is chosen as effective. For example, if a matching operator policy definition sets max-length to 50 and a matching regular policy definition uses the value of 100, the value of 50 will be used. If, however, regular policy's value was 20, it would be used. Operator policies, therefore, don't just overwrite regular policy values. They enforce limits but try to not override user-provided policies where possible.

	Classic	Quorum	Stream
delivery-limit		lesser value
expires	lesser value	lesser value of the two policies policy precedence over queue arguments
ha-mode	The order of preference: all exactly nodes
ha-params	greater value when both values are integers integer value when only one value is an integer longer list when both values are a list of nodes operator policy value when both values are an equal length list of nodes
ha-sync-mode	operator policy value
max-in-memory-bytes		lesser value
max-in-memory-length		lesser value
max-length	lesser value	lesser value
max-length-bytes	lesser value	lesser value	lesser value of the two policies policy precedence over queue arguments
message-ttl	lesser value	lesser value
target-group-size		greater value

When the same key is provided by both client-provided x-arguments and by a user policy, the former take precedence.

However, if an operator policy is also used, that will take precedence over the client-provided arguments, too. Operator policies are a protection mechanism and override client-provided values and user policy values.

Use operator policies to introduce guardrails for application-controlled parameters related to resource use (e.g. peak disk space usage).

Defining Operator Policies

Operator policies are defined in a way very similar to regular (user) policies. When rabbitmqctl is used, the command name is set_operator_policy instead of set_policy. In the HTTP API, /api/policies/ in request path becomes /api/operator-policies/:

rabbitmqctl	rabbitmqctl set_operator_policy transient-queue-ttl \ "^amq\." '{"expires":1800000}' \ --priority 1 \ --apply-to queues
rabbitmqctl (Windows)	rabbitmqctl.bat set_operator_policy transient-queue-ttl ^ "^amq\." "{""expires"": 1800000}" ^ --priority 1 ^ --apply-to queues
HTTP API	PUT /api/operator-policies/%2f/transient-queue-ttl {"pattern": "^amq\.", "definition": {"expires": 1800000}, "priority": 1, "apply-to": "queues"}
Web UI	Navigate to Admin > Policies > Add / update an operator policy. Enter "transient-queue-ttl" next to Name, "^amq." next to Pattern, and select "Queues" next to Apply to. Enter "expires" = 1800000 in the first line next to Policy. Click Add policy.

How to Disable Operator Policy Changes

Modification of operator policies via the HTTP API and Web UI can be disabled in configuration. This makes operator policies read-only for all users via the HTTP API and Web UI.

management.restrictions.operator_policy_changes.disabled = true