RabbitMQ AMQP.BasicProperties.Builder values

Question

In the RabbitMQ/AMQP Java client, you can create an AMQP.BasicProperties.Builder, and use it to build() an instance of AMQP.BasicProperties. This built properties instance can then be used for all sorts of important things. There are lots of "builder"-style methods available on this builder class:

BasicProperties.Builder propsBuilder = new BasicProperties.Builder();
propsBuilder
    .appId(???)
    .clusterId(???)
    .contentEncoding(???)
    .contentType(???)
    .correlationId(???)
    .deliveryMode(2)
    .expiration(???)
    .headers(???)
    .messageId(???)
    .priority(???)
    .replyTo(???)
    .timestamp(???)
    .type(???)
    .userId(???);

I'm looking for what fields these builer methods help "build-up", and most importantly, what valid values exist for each field. For instance, what is a clusterId, and what are its valid values? What is type, and what are its valid values? Etc.

I have spent all morning scouring:

• The Java client documentation; and
• The Javadocs; and
• The RabbitMQ full reference guide; and
• The AMQP specification

In all these docs, I cannot find clear definitions (besides some vague explanation of what priority, contentEncoding and deliveryMode are) of what each of these fields are, and what their valid values are. Does anybody know? More importantly, does anybody know where these are even documented? Thanks in advance!

Solution 1

Usually I use very simple approach to memorize something. I will provide all details below, but here is a simple picture of BasicProperties field and values. I've also tried to properly highlight queue/server and application context.

If you want me to enhance it a bit - just drop a small comment. What I really want is to provide some visual key and simplify understanding.

High-level description (source 1, source 2):

Please note Clust ID has been deprecated, so I will exclude it.

• Application ID - Identifier of the application that produced the message.
  • Context: application use
  • Value: Can be any string.
• Content Encoding - Message content encoding
  • Context: application use
  • Value: MIME content encoding (e.g. gzip)
• Content Type - Message content type
  • Context: application use
  • Value: MIME content type (e.g. application/json)
• Correlation ID - Message correlated to this one, e.g. what request this message is a reply to. Applications are encouraged to use this attribute instead of putting this information into the message payload.
  • Context: application use
  • Value: any value
• Delivery mode - Should the message be persisted to disk?
  • Context: queue implementation use
  • Value: non-persistent (1) or persistent (2)
• Expiration - Expiration time after which the message will be deleted. The value of the expiration field describes the TTL period in milliseconds. Please see details below.
  • Context: queue implementation use
• Headers - Arbitrary application-specific message headers.
  • Context: application use
• Message ID - Message identifier as a string. If applications need to identify messages, it is recommended that they use this attribute instead of putting it into the message payload.
  • Context: application use
  • Value: any value
• Priority - Message priority.
  • Context: queue implementation use
  • Values: 0 to 9
• ReplyTo - Queue name other apps should send the response to. Commonly used to name a reply queue (or any other identifier that helps a consumer application to direct its response). Applications are encouraged to use this attribute instead of putting this information into the message payload.
  • Context: application use
  • Value: any value
• Time-stamp - Timestamp of the moment when message was sent.
  • Context: application use
  • Value: Seconds since the Epoch.
• Type - Message type, e.g. what type of event or command this message represents. Recommended to be used by applications instead of including this information into the message payload.
  • Context: application use
  • Value: Can be any string.
• User ID - Optional user ID. Verified by RabbitMQ against the actual connection username.
  • Context: queue implementation use
  • Value: Should be authenticated user.

BTW, I've finally managed to review latest sever code (rabbitmq-server-3.1.5), there is an example in rabbit_stomp_test_util.erl:

                content_type     = <<"text/plain">>,
                content_encoding = <<"UTF-8">>,
                delivery_mode    = 2,
                priority         = 1,
                correlation_id   = <<"123">>,
                reply_to         = <<"something">>,
                expiration       = <<"my-expiration">>,
                message_id       = <<"M123">>,
                timestamp        = 123456,
                type             = <<"freshly-squeezed">>,
                user_id          = <<"joe">>,
                app_id           = <<"joe's app">>,
                headers          = [{<<"str">>, longstr, <<"foo">>},
                                    {<<"int">>, longstr, <<"123">>}]

Good to know somebody wants to know all the details. Because it is much better to use well-known message attributes when possible instead of placing information in the message body. BTW, basic message properties are far from being clear and useful. I would say it is better to use a custom one.

Good example (source)

Update - Expiration field

Important note: expiration belongs to queue context. So message might be dropped by the servers.

README says the following:

expiration is a shortstr; since RabbitMQ will expect this to be an encoded string, we translate a ttl to the string representation of its integer value.

Sources:

• Additional source 1
• Additional source 2

Solution 2

At time of writing:

1. The latest AMQP standard is AMQP 1.0 OASIS Standard.
2. The latest version of RabbitMQ is 3.1.5 (server and client), which claims to support AMQP 0.9.1 (pdf and XML schemas zipped).
3. RabbitMQ provides it's own description of the protocol as XML schema including extensions (i.e. non-standard), plus XML schema without extensions (which is identical to the schema linked via (2)) and pdf doc.

In this answer:

• links in (3) are the primary source of detail
• (2) pdf doc is used as secondary detail if (3) is inadequate
• The source code (java client, erlang server) is used as tertiary detail if (2) is inadequate.
• (1) is generally not used - the protocol and schema have been (fairly) significantly evolved for/by OASIS and should apply to future versions of RabbitMQ, but do not apply now. The two exceptions where (1) was used was for textual descriptions of contentType and contentEncoding - which is safe, because these are standard fields with good descriptions in AMQP 1.0.

The following text is paraphrased from these sources by me to make a little more concise or clear.

• content-type (AMQP XML type="shortstr"; java type="String"): Optional. The RFC-2046 MIME type for the message’s application-data section (body). Can contain a charset parameter defining the character encoding used: e.g., ’text/plain; charset=“utf-8”’. Where the content type is unknown the content-type SHOULD NOT be set, allowing the recipient to determine the actual type. Where the section is known to be truly opaque binary data, the content-type SHOULD be set to application/octet-stream.
• content-encoding (AMQP XML type="shortstr"; java type="String"): Optional. When present, describes additional content encodings applied to the application-data, and thus what decoding mechanisms need to be applied in order to obtain the media-type referenced by the content-type header field. Primarily used to allow a document to be compressed without losing the identity of its underlying content type. A modifier to the content-type, interpreted as per section 3.5 of RFC 2616. Valid content-encodings are registered at IANA. Implementations SHOULD NOT use the compress encoding, except as to remain compatible with messages originally sent with other protocols, e.g. HTTP or SMTP. Implementations SHOULD NOT specify multiple content-encoding values except as to be compatible with messages originally sent with other protocols, e.g. HTTP or SMTP.
• headers (AMQP XML type="table"; java type="Map"): Optional. An application-specified list of header parameters and their values. These may be setup for application-only use. Additionally, it is possible to create queues with "Header Exchange Type" - when the queue is created, it is given a series of header property names to match, each with optional values to be matched, so that routing to this queue occurs via header-matching.
• deliveryMode (RabbitMQ XML type="octet"; java type="Integer"): 1 (non-persistent) or 2 (persistent). Only works for queues that implement persistence. A persistent message is held securely on disk and guaranteed to be delivered even if there is a serious network failure, server crash, overflow etc.
• priority (AMQP XML type="octet"; java type="Integer"): The relative message priority (0 to 9). A high priority message is [MAY BE?? - GB] sent ahead of lower priority messages waiting in the same message queue. When messages must be discarded in order to maintain a specific service quality level the server will first discard low-priority messages. Only works for queues that implement priorities.
• correlation-id (AMQP XML type="octet"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. A client-specific id that can be used to mark or identify messages between clients.
• replyTo (AMQP XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour but may hold the name of a private response queue, when used in request messages. The address of the node to send replies to.
• expiration (AMQP XML type="shortstr"; java type="String"): Optional. RabbitMQ AMQP 0.9.1 schema from (3) states "For implementation use, no formal behaviour". The AMQP 0.9.1 schema pdf from (2) states an absolute time when this message is considered to be expired. However, both these descriptions must be ignored because this TTL link and the client/server code indicate the following is true. From the client, expiration is only be populated via custom application initialisation of BasicProperties. At the server, this is used to determine TTL from the point the message is received at the server, prior to queuing. The server selects TTL as the minimum of (1) message TTL (client BasicProperties expiration as a relative time in milliseconds) and (2) queue TTL (configured x-message-ttl in milliseconds). Format: string quoted integer representing number of milliseconds; time of expiry from message being received at server.
• message-id (AMQP XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. If set, the message producer should set it to a globally unique value. In future (AMQP 1.0), a broker MAY discard a message as a duplicate if the value of the message-id matches that of a previously received message sent to the same node.
• timestamp (AMQP XML type="timestamp"; java type="java.util.Date"): Optional. For application use, no formal (RabbitMQ) behaviour. An absolute time when this message was created.
• type (AMQP XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. [Describes the message as being of / belonging to an application-specific "type" or "form" or "business transaction" - GB]
• userId (AMQP XML type="shortstr"; java type="String"): Optional. XML Schema states "For application use, no formal (RabbitMQ) behaviour" - but I believe this has changed in the latest release (read on). If set, the client sets this value as identity of the user responsible for producing the message. From RabbitMQ: If this property is set by a publisher, its value must be equal to the name of the user used to open the connection (i.e. validation occurs to ensure it is the connected/authenticated user). If the user-id property is not set, the publisher's identity remains private.
• appId (RabbitMQ XML type="shortstr"; java type="String"): Optional. For application use, no formal (RabbitMQ) behaviour. The creating application id. Can be populated by producers and read by consumers. (Looking at R-MQ server code, this is not used at all by the server, although the "webmachine-wrapper" plugin provides a script and matching templates to create a webmachine - where an admin can provide an appId to the script.)
• cluster Id (RabbitMQ XML type="N/A"; java type="String"): Deprecated in AMQP 0.9.1 - i.e. not used. In previous versions, was the intra-cluster routing identifier, for use by cluster applications, which should not be used by client applications (i.e. not populated). However, this has been deprecated and removed from the current schema and is not used by R-MQ server code.

As you can see above, the vast majority of these properties do not have enumerated / constrained / recommended values because they are "application use only" and are not used by RabbitMQ. So you have an easy job. You're free to write/read values that are useful to your application - as long as they match datatype and compile :). ContentType and contentEncoding are as per standard HTTP use. DeliveryMode and priority are constrained numbers.

Note: Useful, but simple constants for AMQP.BasicProperties are available in class MessageProperties.

Cheers :)

UPDATE TO POST:

With many thanks to Renat (see comments), have looked at erlang server code in rabbit_amqqueue_process.erl and documentation at RabbitMQ TTL Extensions to AMQP. Message expiration (time-to-live) can be specified

• per queue via:

  Map<String, Object> args = new HashMap<String, Object>();
  args.put("x-message-ttl", 60000);
  channel.queueDeclare("myqueue", false, false, false, args);
  

• or per message via:

  byte[] messageBodyBytes = "Hello, world!".getBytes();
  AMQP.BasicProperties properties = new AMQP.BasicProperties();
  properties.setExpiration("60000");
  channel.basicPublish("my-exchange", "routing-key", properties, messageBodyBytes);
  

Here, the ttl/expiration is in millisecs, so 60 sec in each case. Have updated above definition of expiration to reflect this.

Comments

[PedroD]

One day, people who write these APIs for Java will find out about ENUMS.

[Renat Gilmanov]

@Glen_Best, expiration is not for application use. Please check rabbit_amqqueue_process.erl or see my update.

[Glen Best]

Have checked the code - you're correct. Edited post. Thanks!

[Glen Best]

Clarification: is for application use as per rabbitmq.com/ttl.html; my explanation indicated no formal processing by Rabbit - there is such. i.e. RabbitMQ 0.9.1 schema comment is out of date / wrong.

[Slaven Rezic]

The value descriptions for Content Encoding and Content Type have to be swapped.

[Renat Gilmanov]

@SlavenRezic, you are very right. Sorry for the mistake and thank you.

[Max Barraclough]

Great answer, thanks. The 'Additional source 2' link is broken though.

[Thomas]

at the Time-stamp field the type is really a timestamp not a number (containing milisecs), the rabbit admin page displaying it as number but if you are receiving the msg you can see it, and the declaration tell you too that he expecting a timestamp