Lag when MQTT Server is down (possible workaround)

[ascillato]

Hi

It is a known Tasmota issue (Wiki) that when a Sonoff try to connect to a MQTT server that is DOWN (IP and Port, both down), the pubsubclient library hangs for 5 second on each reconnection attempt making a Sonoff unresponsive.

So, looking for a work around for this, found this other MQTT Library (https://github.com/i-n-g-o/esp-mqtt-arduino). I tested on ESP8266 and with Home Assistant, and it has no lag when losing connection to MQTT Server. It has a different approach for conecting to an MQTT Server that is not using the wificlient library like pubsubclient.

What about changing the library so as to make Tasmota faster?

[Jason2866]

Hello, no offend
what is the use case of a mqtt device (sonoff) without a mqtt broker?
Users who want to use it without mqtt can turn it permant off (for example to use with Alexa)

[ascillato]

Hi, no offend taken. I just post an idea to be discussed ;)

May be I wasn't clear. I'm not saying of taking out MQTT from tasmota. No. That's not the idea. You are right. Will be useless that. What I'm saying is changing the MQTT library from pubsubclient to esp-mqtt-arduino. Just using another library for MQTT on Tasmota Firmware.

The actual library (pubsubclient) is a generic MQTT library to be used for any Arduino compatible device. So it is NOT optimized for ESP8266. However, esp-mqtt-arduino library is a MQTT library optimized for ESP8266.

So, I tested both libraries and pubsubclient has a 5 second lag when there is not a MQTT server active while the other has not that bug and remains fast responsive.

When I say MQTT Server Down I meant the case where you are using the MQTT feature and because a Wifi disconnection or a problem in a MQTT server, a Sonoff disconnects from the server. Using pubsubclient, Tasmota gets unresponsive for 5 seconds each 10 seconds. With the other library, this bug should be resolved.

So, @arendst, What do you think about changing the library from pubsubclient to esp-mqtt-arduino?

[arendst]

I'll take a look at it.

[emontnemery]

This sounds really nice! WAF is very low when the wall switches are not working because the WIFI is acting up..

[arendst]

UPDATE:

Installed patched esp-mqtt-arduino as the original does not support LWT like Tasmota does and I want it to be compatible with PubSubClient. It adds 5k to the code base while using the same amount of ram so that's fine. It does not support TLS using WifiClientSecure as PubSubClient does so it is only a replacement for non-TLS configurations. Blocking is indeed gone so that's positive.

I also tried ESP8266MQTTClient but that only seems to support TLS which is very code hungry: +52k code and exceptions everywhere. So that's no way to go.

I'll tune/rewrite/strip the current esp-mqtt-arduino library and will release a TasmotaMQTT library being a wrapper around slightly modified TuanPmt code and to be used on non-TLS sytems.

TO BE CONTINUED.

[ascillato]

Work really well !!! Thanks 👍

@arendst please put the TasmotaMQTT as default option. Works really good.

[ascillato]

Issue solved. Closing...

[jeylites]

@ascillato
A noob question is this alternative by default in v5.12.0d or something has to be done in the script for it to taken effect?

Update:

Nevermind, I found it in user config.h

/*********************************************************************************************\
 * Select ONE of possible MQTT library types below
\*********************************************************************************************/
// Default MQTT driver for both non-TLS and TLS connections. Blocks network if MQTT server is unavailable.
#define MQTT_LIBRARY_TYPE      1                 // Use PubSubClient library
// Alternative MQTT driver does not block network when MQTT server is unavailable. No TLS support
//#define MQTT_LIBRARY_TYPE      2                 // Use TasmotaMqtt library (+4k4 code, +4k mem) - non-TLS only
// Alternative MQTT driver does not block network when MQTT server is unavailable. No TLS support
//#define MQTT_LIBRARY_TYPE      3                 // Use (patched) esp-mqtt-arduino library (+4k8 code, +4k mem) - non-TLS only

[ascillato]

great!!! 👍

[Kepete]

Run into the same lag problem when MQTT Broker is down. So I changed the Default MQTT driver to
MQTT_TASMOTAMQTT. But it didn't help, the lag is still present.
Version 6.2.1

[ascillato2]

@Kepete

Use the last Tasmota version without modifications. It doesn't have any lag.

The lags were solved with newer Pubsubclient versions already available in libs folder. If you try the precompiled bins (http://thehackbox.org/tasmota/) you will not have any issue. If you compile by yourself, update Tasmota to last (v6.3.0.7 is now) and also update all the libraries (delete the previous you have first)

TasmotaMQTT is deprecated and has some restart with exception issues. Not recommended. The new Pubsubclient works fine.

(BTW you were writing in a very old closed issue)

[Kepete]

Thank you for quick help.
Didn't see a point in opening a new issue if it is already discussed here.

Downloaded the newest v6.3.0 precompiled software from https://github.com/arendst/Sonoff-Tasmota/releases. Did an OTA update from the web-interface and tested it.
Still same lag when MQTT broker is down.

[ascillato]

Seems that you are having another issue.

Did you check the console output? may be your device is rebooting due to a problem, not lagging

[ascillato]

To discard other issues, please, upgrade to last version 6.3.0.7 (http://thehackbox.org/tasmota/) that have several improvements on performance.

[ascillato]

Also, if you can, please try to erase all before flash.

If after all that, you still experience the lag, Please, open a new issue but with the troubleshooting template completed and the output of your console. Thanks

[khriss75]

Precompiled 6.6.0 version still having lag with mqtt server down.

[Kepete]

@khriss75 I eventually fixed the problem. I think MQTT_TASMOTAMQTT helps, but make sure to erase all flash before updating.
I think that was the issue why at first it didn't help for me.

[khriss75]

Thanks Kepete, I'm searcing to define MQTT_TASMOTAMQTT in "my user config" tab but I can't find the mqtt_tasmota line #define
I'm using 6.6.0 version.

[Jason2866]

MQTT_TASMOTAMQTT is not used anymore....
PubSubClient-EspEasy is the one and only mqtt version used with Tasmota.
Blocking is solved!

[khriss75]

Thanks Janson2866 for your replay, but I'm still having issue! As described I downloaded latest version (6.6.0) PRECOMPILED using on ESP8266. If I switch off my mqtt server I have a lag for about 3-4 seconds every time retry mqtt connection. During mqtt connection my buttons/relays don't react. I can provide a video of this issue. Tasmota 6.6.0 precompiled + NODEMCU.

[Jason2866]

Mhh, why does you mqtt server goes offline? Yes i know bad things can happen, but normally it should run. And for the times it doesnt run you have some side effects

[khriss75]

Mhh, imagine a door sensor (magnetic switch), and siren. Mqtt connected: If someone open and close the door siren start and send mqtt to my smartphone.
Now, mqtt server down, if someone open and close the door during reconnection, siren stay off and obviously no mqtt message sent. But I'd like the siren start when door is opened.
I know, this is a stupid example, but I'd like that the loop don't stop during mqtt reconnection.
Real example: wind sensor send a signal in order tho close the sun shade. If is in "mqtt reconnection time" the sun shade is not closed... Mqtt broker/server could be down for so many reason...

[ascillato]

Hi, will investigate your issue.

[khriss75]

Thank you so much Adrian. If you need some other info ask to me. I can provide a video.

[ascillato]

Using Tasmota from today (6.6.0.6):

The lag CAN NOT be reproduced on Core 2.3.0
The lag CAN NOT be reproduced on Core 2.4.2
The lag CAN NOT be reproduced on Core 2.5.2
The lag CAN be reproduced on Core pre2.6.0
The lag CAN be reproduced on Core STAGE

So, bad news. Confirmed Issue and core related. @khriss75 Please, use Tasmota precompiled bins with core 2.3.0 (like http://thehackbox.org/tasmota/020300/sonoff.bin) to avoid this issue until a solution could be found for the core.

For core 2.3.0, 2.4.2, 2.5.2 there is no lag issue.
For core pre2.6.0 and STAGE(under development), the unresponsive time (lag) is of 5 seconds while trying to reconnect to the MQTT broker and this happens every 10 seconds.

The issue seems to be inside the new ArduinoESP8266 WIFICLIENT Library. Will report to Arduino Esp8266 devs.

Thanks for reporting.

[ascillato]

The wificlient library from Arduino didn't change from 2.5.2 to STAGE. It is the same.

So, it is something in the SDK ?

[Jason2866]

With core pre2.6 and feature/stage nonos-sdk 22y is standard and different to all other core builds
This closed source code solved mqtt reconnects and maybe introduced the lag.
Needs testing if reverting back to nonos-sdk 22x solves lag. This will reintroduce mqtt disconnects...
Mhh... But we do NOT offer precompiled Tasmota versions with core pre2.6 and feature/stage!
So the problem of @khriss75 has to be elsewhere

[ascillato]

As your suggestion, tested SDK 22x and the lags is there in core STAGE. So, it is not SDK related due sdk22y and sdk22x are used in core 2.5.2 too and with that core there is no lag.

So, the issue seems to be inside the core stage or in the way that pubsubclient-espeasy uses wificlient.

In the precompiled bins (core 2.3.0, 2.4.2 and 2.5.2) I could not reproduce any lag.

[ascillato]

Hi,

I have just found the issue in the Arduino core that produces this 5 seconds hung when the MQTT broker goes down.

The issue appears with this PR: esp8266/Arduino#6213

I have just proposed the PR: esp8266/Arduino#6440 in order to fix this hung.

[khriss75]

Using Tasmota from today (6.6.0.6):

The lag CAN NOT be reproduced on Core 2.3.0
The lag CAN NOT be reproduced on Core 2.4.2
The lag CAN NOT be reproduced on Core 2.5.2
The lag CAN be reproduced on Core pre2.6.0
The lag CAN be reproduced on Core STAGE

So, bad news. Confirmed Issue and core related. @khriss75 Please, use Tasmota precompiled bins with core 2.3.0 (like http://thehackbox.org/tasmota/020300/sonoff.bin) to avoid this issue until a solution could be found for the core.

Thanks for reporting.

@ascillato , I'm still having problem with the precompiled core 2.3.0 linked. I erased my nodemcu and flashed the linked .bin but the lag persist.

[khriss75]

Other info:
if error connection "rc -5" no lag
if error connection "rc -2" lag

[Jason2866]

Cant reproduce the lag in my setup.
With the fix from @ascillato core pre 2.6.0 has no lag too.

[khriss75]

Jason, thanks for interesting!
_00:00:00 CFG: Loaded from flash at FA, Count 104
00:00:00 Project sonoff Sonoff Version 6.6.0.6(5cb863d-sonoff)-2_3_0
00:00:00 WIF: Connecting to AP1 TISCALI_2340 in mode 11N as sonoff-3203...
00:00:04 WIF: Connected
00:00:04 HTP: Web server active on sonoff-3203 with IP address 192.168.178.213
00:00:05 MQT: Attempting connection...
13:29:33 MQT: Connect failed to 192.168.178.85:1883, rc -2. Retry in 10 sec
_
Here is serial from termite, I think I'm using the correct version (2.3.0).

Just to test as my configuration:

Generic Module.
D3 Button1
D4 Button2
D2 Button3
D1 Button4

D6 Relay1
D7 Relay2
D5 Relay3
D8 Relay4

Setoption13 1

As you can see from console, at mqtt connection tentative there are about 5 seconds:

Can you send me a link of version without issue (You have tested)?
Do I need to configure some other?
.bin uploaded with "Nodemcu Firmware programmer" tool, Flash size 4MByte, Flash Speed 40MHz, DOUT.

[Jason2866]

You have a general problem in your wifi / mqtt setup.
https://github.com/arendst/Sonoff-Tasmota/wiki/MQTT-Overview#status-codes
Tasmota firmware is not the problem.
I would try a different mqtt broker, to see if issue is gone.
firmware.zip

[ascillato]

Fix is already merged into Arduino Core STAGE. esp8266/Arduino#6454

[ascillato]

If anyone is having any blocking or lag, please open a new issue with the troubleshooting template complete or address it to the Tasmota Support Chat. Thanks.