[Question] Recommended way to publish 100K messages in a batch ?

[geminigeek]

hi,

can you suggest a recommended way to publish large amount of messages i want to publish about 100K messages.
in the documentation here https://lavinmq.com/documentation/nodejs-sample-code , i can see this example is it the correct way for doing 100K messages, with a loop ?

 //Publish a message to the exchange
    async function sendToQueue(routingKey, body) {
      await channel.basicPublish("", routingKey, body)
      console.log("[📥] Message sent to queue", body)
    }
  
    //Send some messages to the queue
    sendToQueue("hello_world", "Hello World");
    sendToQueue("hello_world", "Hello World");
    sendToQueue("wrong_routing_key", "Hello World");

[baelter]

What qos do you need? Consider publisher confirms if you need guarantiees.

e.g.

channel.confirm_select
batched_messages.forEach(message => {
  channel.basicPublish(exchange, routingKey, message)
})
channel.wait_for_confirms // Will raise if any message in batch was NACKed, you may want to recover channel and republish batch.

batch size 1 => lowest performance, delivery with no duplicates guaranteed.
batch size > 1 => more performance, play around to find a good value for you. Possible message duplication.

[geminigeek]

hi,

thanks for the reply, i managed to published 100k messages , currently i do not require publish confirmation, but making batches of 1000 is ideal when doing in batches in my case,

i have one more query what is the compatible function in nodejs for "channel.wait_for_confirms" , i could not find this in api doc page here https://cloudamqp.github.io/amqp-client.js/classes/AMQPChannel.html

[baelter]

Sorry got this confused with another client we maintain :)
wait_for_confirms does not exists in this client. basicPublish will return a promise that is resolved when the publish is confirmed. So something like:

channel.confirmSelect
confirmPromises = []
batched_messages.forEach(message => {
  confirmPromises.push(channel.basicPublish(exchange, routingKey, message))
})
Promise.all(confirmPromises)