20.11 MQTT Broker

As mentioned in 20.07 node MQTT, I use [[20.11 MQTT Broker#^35b724|https://test.mosquitto.org]] as MQTT broker. But what if you want your own?

Luckily mosquitto has an answer for us there.

Info

If you have outgrown this public server and would like your own, you have a few options.

• A.Host your own broker - this is fairly straightforward to do and will run on any general purpose cloud instance or local server.
• B.Use a hosted service - Cedalo offers managed hosted Mosquitto instances.
• C.Use a supported on-premise broker - Cedalo will support your use of Mosquitto, offer consulting for building your system or developing features to your requirements, and have premium features available for Mosquitto.

A.Host your own broker

Aedes I couldn’t get to work.

Mosquitto self host installation

1. go to https://mosquitto.org/download/
2. select the right download, I use mosquitto-2.0.18-install-windows-x64.exe
3. execute the .exe
4. folstallation proces
5. press Win and type “services”
6. select the sytem app called “services”
7. look for mosquitto
8. look for “mosquitto Broker”
9. status should not be running, so we follow the following steps:
10. press Win and type “cmd”
11. select the system app called “command prompt”
12. select the system app called “command prompt”, run it as an administrator
13. navigate to the folder where ytou installed mosquitto, in my case cd "C:\Program Files\mosquitto"
14. type net start mosquitto
15. if the message is The Mosquitto Broker service was started successfully. we can continue with the next steps
16. go back to the services app to double check if the “mosquitto Broker” is running
17. go back to the services app to double check if the “mosquitto Broker” is running (don’t forget to press the refresh button at the top left)
18. go back to command prompt and verify if it is running by typing: netstat -a
19. 127.0.0.1:1883 is the MQTT borker default port
20. change the url from mqtt://test.mosquitto.org to mqtt://localhost in the code Figured it out thanks to this Youtube video.

You would need to enable port forwarding to be able to connect to this from other networks.

limits

To test the broker, I was spamming it with node worker threads and listening to all the messages. At first I received all messages. But after spamming it with messages. My spammer made X worker threads, and then all these threads send X messages.

Conditions: I used a spammer.js which makes X worker threads who all send X messages, and used the topicfinder.js to see if I received em all, these are the results:

• At 111 workers and 1000 messages per worker I started getting data loss. But then when I scale back down to 101w x 1000m I still have data loss, which I didn’t have beforehand.
• At 70w and 1000m seems to be fine again.
• At 1001w and 10m it had a strain on my pc, but it did the job. But 570w sent 20m instead of 10m, atleast the scanner thought so.

Conditions: I created a program to spam MQTT, and write down once it receives the messages, what delay, how many it received, and how many it missed. It uses worker threads. It is expected to lose a bit faster then the above method because it has logic, memory for dates etc. The above method had way less logic.

• 50w + 500m + 0d = everything always arrives in 6s, 5s average.
• 50w + 750m + 0d = most of the time everything arrives 7s, but sometimes it loses hundreds.
• 50w + 1000m + 10d = everything always arrives in 13s, 12s average.
• 50w + 2000m + 10d = everything always arrives in 24s, 23s average.
• 50w + 3000m + 10d = everything always arrives in 34s, 33s average.
• 50w + 10000m + 10d = everything always arrives in 104s, 103.5s average.
• 100w + 1000m + 10d = not even 1 worker has all their messages received. They all space out at above 800m.
• 100w + 1000m + 20d = not even 1 worker has all their messages received. They all space out at above 900m.
• 100w + 1000m + 25d = everything always arrives in 34s, 32.5s average.
• 100w + 1000m + 21d = everything always arrives in 28s, 27.5s average.
• 100w + 2000m + 21d = everything always arrives in 50s, 49s average.
• 100w + 3000m + 21d = everything always arrives in 71s, 70s average.

What I am understanding from this is:

• If you have devices that are constantly sending data, it will be able to handle 50 messages at once, as long as the next 50 messages is 10ms away. Meaning it takes between 0 and 10ms to handle 50 messages, by sending faster, eventually the queue will run out.
• By taking these measurements I can calculate that you can send somewhere around 3125m/s, 4166m/s, 4807m/s, 4285m/s, 4,081m/s, 3636m/s. Calculation: W * M / S = M/S.

The bigger the message being sent, the bigger the chance of losing messages.

• 100w + 1000m + 25d had a chance of losing messages when I was sending 279 characters instead of 20 characters. This means that all the above testing will have different results with more than 20 characters. Same testing as before, but with ±max characters per message:
• 50w + 500m + 0d = everything always arrives in 7s, 6s average. adds 1s
• 50w + 750m + 0d = most of the time everything arrives 7s, but sometimes it loses hundreds. same result
• 50w + 1000m + 10d = Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching memory limit: JS heap out of memory OOPS. This is probably caused because of the artificial delay. Same testing as before, but with 35108863 characters instead of 67108863:
• 50w + 500m + 0d = sometimes 6s, but sometimes it loses ±1500m.
• 50w + 750m + 0d = most of the time everything arrives 7s, but sometimes it loses hundreds. same result
• 50w + 1000m + 10d = slow, because it loses a lot of memory. Because everything is done on my laptop. Which means inaccurate testing.

I found the following (default) limits in the dosc:

• max_queued_messages = 1000
• message_size_limit = 268435455 bytes
• max_inflight_messages = 20 (messages it can handle at the same time) This is all changeable.

EMQX self host installation

You need docker to install it on windows.

limits

The Github says it can handle 1M messages per second

B.Use a hosted service

hosts

these are the top results I found.

• cedalo.com hosts the mosquitto broker online. There is a free trial. They have different pricing tiers.
• hiveMQ I think it is still local hosting? I need to download a zip folder. I’ll try later. Cause it needs contact info etc.
• vimexx is normal webhosting, IDK why it came up as MQTT broker.
• cloudMQTT I don’t see a free trial
• MyQttHub.com has max messages per month limit per plan.
• stackhero has free tier.

C.Use a supported on-premise broker

This is just A.Host your own broker but then cedalo.com helps us with the installation and optimisation etc. But we would still run it ourselves.

MQTT Broker list

• mosquitto is public & self host
• aedes is self host. - didn’t work.
• emqx is self hosted. Said to be the most scalable. You can also let others host it for you, said to be free (but I haven’t tested because I had to login using phone number)

Summary

I currently only have installed mosquitto. mosquitto has a speed limit of something between 4000m/s and 5000m/s. This can change depending on message size. And I can’t fully test the real limits because when I try, my laptop memory will make the results inaccurate.

You can host it on the cloud, "for free" with emqx, but I haven’t tried because it requires a phone number. If this works, they say they can handle 1M messages per second. You can also deploy emqx with docker, but I don’t know how docker works. So before I start doing that, I want to know if mosquitto is enough.