I am working with Google Cloud Pub/Sub, and I am trying to ensure that messages with the same orderingKey are processed in order, as stated in the official documentation. However, I notice that messages with the same orderingKey are delivered in parallel, even though I haven’t yet called message.ack() for the previous message.

Here are the details of my configuration and implementation:

1. Subscription: Created with --enable-message-ordering to respect message order.
2. Message Publishing: Messages are published with the exact same orderingKey.
3. Subscriber: I am using a Node.js subscriber with explicit message handling and asynchronous processing. I ensure that message.ack() is only called after the message has been fully processed.

Code Example:

Publishing:

const { PubSub } = require("@google-cloud/pubsub");
const pubSubClient = new PubSub();

const publishMessage = async (orderingKey, data) => {
  const dataBuffer = Buffer.from(JSON.stringify(data));
  await pubSubClient
    .topic("YOUR_TOPIC_NAME")
    .publishMessage({
      data: dataBuffer,
      orderingKey: orderingKey,
    });
};

publishMessage("my-key", { orderId: "123" });

Subscriber:

const subscription = pubSubClient.subscription("YOUR_SUBSCRIPTION_NAME");

subscription.on("message", async (message) => {
  try {
    console.log(`Message received: ${message.id}, orderingKey: ${message.attributes.orderingKey}`);
    const data = JSON.parse(message.data);

    // Simulate long processing
    await new Promise((resolve) => setTimeout(resolve, 5000));
    console.log(`Message processed: ${data.orderId}`);
  } catch (err) {
    console.error(`Error: ${err}`);
    return;
  } finally {
    console.log(`Calling message.ack() for: ${message.id}`);
    message.ack();
  }
});

Problem:

Despite this configuration:

Messages with the same orderingKey are delivered and executed in parallel. This happens even though I haven’t yet called message.ack().

What I Want:

I want to ensure that Google Pub/Sub processes messages with the same orderingKey sequentially, meaning:

The next message is only delivered after the previous one has been marked as ack().

Questions:

1. Why does Pub/Sub deliver messages with the same orderingKey in parallel in my case?
2. How can I configure my subscriber or system to ensure that messages are strictly sequential with orderingKey?

Additional Context:

I am using Node.js. The subscription was created with --enable-message-ordering. I want to avoid adding local logic (like a queue) to manage this manually.

I am working with Google Cloud Pub/Sub, and I am trying to ensure that messages with the same orderingKey are processed in order, as stated in the official documentation. However, I notice that messages with the same orderingKey are delivered in parallel, even though I haven’t yet called message.ack() for the previous message.

Here are the details of my configuration and implementation:

1. Subscription: Created with --enable-message-ordering to respect message order.
2. Message Publishing: Messages are published with the exact same orderingKey.
3. Subscriber: I am using a Node.js subscriber with explicit message handling and asynchronous processing. I ensure that message.ack() is only called after the message has been fully processed.

Code Example:

Publishing:

const { PubSub } = require("@google-cloud/pubsub");
const pubSubClient = new PubSub();

const publishMessage = async (orderingKey, data) => {
  const dataBuffer = Buffer.from(JSON.stringify(data));
  await pubSubClient
    .topic("YOUR_TOPIC_NAME")
    .publishMessage({
      data: dataBuffer,
      orderingKey: orderingKey,
    });
};

publishMessage("my-key", { orderId: "123" });

Subscriber:

const subscription = pubSubClient.subscription("YOUR_SUBSCRIPTION_NAME");

subscription.on("message", async (message) => {
  try {
    console.log(`Message received: ${message.id}, orderingKey: ${message.attributes.orderingKey}`);
    const data = JSON.parse(message.data);

    // Simulate long processing
    await new Promise((resolve) => setTimeout(resolve, 5000));
    console.log(`Message processed: ${data.orderId}`);
  } catch (err) {
    console.error(`Error: ${err}`);
    return;
  } finally {
    console.log(`Calling message.ack() for: ${message.id}`);
    message.ack();
  }
});

Problem:

Despite this configuration:

Messages with the same orderingKey are delivered and executed in parallel. This happens even though I haven’t yet called message.ack().

What I Want:

I want to ensure that Google Pub/Sub processes messages with the same orderingKey sequentially, meaning:

The next message is only delivered after the previous one has been marked as ack().

Questions:

1. Why does Pub/Sub deliver messages with the same orderingKey in parallel in my case?
2. How can I configure my subscriber or system to ensure that messages are strictly sequential with orderingKey?

Additional Context:

I am using Node.js. The subscription was created with --enable-message-ordering. I want to avoid adding local logic (like a queue) to manage this manually.

• javascript
• google-cloud-platform
• google-cloud-functions
• google-cloud-pubsub
• publish-subscribe

1 Answer 1

The guarantee that Pub/Sub ordering provides is not that the next message for a key will be delivered only after the previous one is acknowledged, but that the next message for a key will be delivered after the previous one has been sent to the user callback and any synchronous work has completed. There are some more details about this behavior for Node specifically in GitHub:

This is unfortunately a "works as intended", because the guarantee in the library is that you'll receive all of the messages in order within one subscriber RPC callback (meaning, the code will not return to the event loop until it's called the user callback for each message in order). But Node makes that guarantee somewhat less useful because of the asynchrony.

In similar cases, we've been recommending using a queue on the user side, just pushing all of the messages for that key into the queue and then executing their callbacks one at a time.