connecting emqx to hstreamdb

[mntnclmbr]

Hi, this link (https://www.emqx.com/en/blog/integration-practice-of-emqx-and-hstreamdb#start-emqx-and-hstreamdb) seems to be fairly outdated . The yaml file for docker does not setup the httpServerPort anymore and only sets up 6570 from what I can see. Is there an updated SOP version that we can follow to link EMQX 5.2 with the latest hstream release ?

[alissa-tung]

Here are some steps we can follow to persist EMQX data to HStreamDB.
https://docs.emqx.com/en/enterprise/v4.4/rule/backend_hstreamdb.html
We are currently not supporting EMQX 5.2 with the latest HStreamDB. The above docs are for EMQX Enterprise 4.4, and we are going to make more improvements in the integration in 4.5. Feel free to ask anything when encountering any problems.

You can try EMQX Enterprise for free https://www.emqx.com/en/try?product=enterprise here. The license will be sent to your email immediately https://www.emqx.com/en/apply-licenses/emqx.

Thank you for using EMQX and HStreamDB. We are sorry for the outdated blog post. Thank you for your feedback. We will mark it as outdated and attach the link to the latest EMQX HStreamDB Integration docs.

[alissa-tung]

FYI, The previous implementation is based on the EMQX Webhook Plugin and HStreamDB HTTP Services, which is less efficient than the action persist to HStreamDB in EMQX Enterprise.

So there is no need for an HTTP server port (no need for separate HTTP Services) in the EMQX Enterprise setup. The URL passed to EMQX Dashboard when creating a resource is the HStreamDB server URL, which has been started in the Docker quick start of the form http://your_hostname_here:your_port, and the port is default to 6570, that is, which you can get from the docker ps outputs.

[mntnclmbr]

Thanks for the quick response. I was planning to use the open source version and not the EMQX enterprise setup.
Can I assume using an exhook plugin with a publish hook in a python setup might be able to do what the "persist to HStreamDB" plugin does ?

Also, is there a server estimate ( like https://www.emqx.com/en/server-estimate) for hstream ?

[alissa-tung]

for now as we build up our system, we will probably be making sure we can support about 800K active mqtt connections (with 1:1 streams in hstream) and up to 80k msgs/second( on the high end) we have emqx in a 3 machine cluster of m6g.2xlarge

Sorry for the late reply. I am wondering how many topics do you have in total in your service? (there are 800k MQTT connections, can I assume there are 800k topics (there is no restrict on this) )

[mntnclmbr]

yes, there would be 800K topics (plus potentially one material view on each but no active subscription on the actual stream)in this case.

[alissa-tung]

Oh thanks. In addition, can I ask about the average payload size for each payload?

[mntnclmbr]

the average payload would be around 1KB in size.

[alissa-tung]

the average payload would be around 1KB in size.

Thanks for you patience. I think 3 instances of 8c32g will be enough for deploy a HStream cluster.
We also have some deploy tools for deploy cluster on cloud. https://github.com/hstreamdb/deployment-tool