Customer Issue:
In an EcoStruxure Power Operation (EPO) SCADA system runtime, there were frequent communication failures with Modbus RS485 serial devices connected through Modbus TCP/IP gateways. A driver from the MODNET family was used to facilitate communication with these devices. In this architecture, the SCADA system acted as a Modbus TCP/IP Client, while the field devices functioned as Modbus TCP/IP Servers through the gateways.
Resolution:
Several attempts made on both SCADA side and Gateway side were not helping at all, such as:
- increasing the Polling time (Cache Time), Timeout, Retries and applying a Driver Error Masking (Citect's KB - Q5923 Driver Error 00008203 (0x0000200b) with MODNET driver when connecting via a gateway) on SCADA side,
- increasing the Response Time of Serial Port on the Modbus TCP/IP gateway side.
The gateway was replying with exception responses as if the source devices were offline: GATEWAY_TARGET_DEVICE_FAILED_TO_RESPOND_EXCEPTION = 0xB
It was happening several times during the runtime, not periodically, however, long enough to put these IODevices offline and trigger communication failure alarms.
When analysing the Modbus TCP/IP frames, it was observed that the Exception Responses were not always to the same register(s) Modbus Read Request, and were not successively persisting more than one Modbus Response, meaning that the next Modbus Read Request from SCADA was answered without error.
It looked like a problem on the Modbus Server devices.
The most efficient way to fix it was by replacing the IODevice Protocol MODNET with PwrModbus and enabling RetryException in [PWRMODBUS] section of citect.ini:
[PWRMODBUS]
RetryException = 1
Note: RetryException = 0 by default, meaning that retries on exceptions are disabled. The RetryException is a feature of the PwrModbus driver, which was not available in the MODNET driver.
More info about this parameter (which exception codes are retried) can be found in the PwrModbus Driver's help (C:\Program Files (x86)\Schneider Electric\Power Operation\v2022\bin\PWRMODBUSDriverhelp.chm).
The number of default retries for PwrModbus driver is 3 and is handled by the "[PWRMODBUS] retry" parameter.
In this system, a [PWRMODBUS] retry = 2 worked well, however it is parameter that can be tuned.
This resolution applies to a SCADA system matching all the conditions below:
- any Power Operation versions higher than 7.40 (Power SCADA Expert, Power SCADA Operation, Power Operation) patched or unpatched with Cumulative Updates
- any PwrModbus and MODNET driver versions compatible with the above.
In an EcoStruxure Power Operation (EPO) SCADA system runtime, there were frequent communication failures with Modbus RS485 serial devices connected through Modbus TCP/IP gateways. A driver from the MODNET family was used to facilitate communication with these devices. In this architecture, the SCADA system acted as a Modbus TCP/IP Client, while the field devices functioned as Modbus TCP/IP Servers through the gateways.
Resolution:
Several attempts made on both SCADA side and Gateway side were not helping at all, such as:
- increasing the Polling time (Cache Time), Timeout, Retries and applying a Driver Error Masking (Citect's KB - Q5923 Driver Error 00008203 (0x0000200b) with MODNET driver when connecting via a gateway) on SCADA side,
- increasing the Response Time of Serial Port on the Modbus TCP/IP gateway side.
The gateway was replying with exception responses as if the source devices were offline: GATEWAY_TARGET_DEVICE_FAILED_TO_RESPOND_EXCEPTION = 0xB
It was happening several times during the runtime, not periodically, however, long enough to put these IODevices offline and trigger communication failure alarms.
When analysing the Modbus TCP/IP frames, it was observed that the Exception Responses were not always to the same register(s) Modbus Read Request, and were not successively persisting more than one Modbus Response, meaning that the next Modbus Read Request from SCADA was answered without error.
It looked like a problem on the Modbus Server devices.
The most efficient way to fix it was by replacing the IODevice Protocol MODNET with PwrModbus and enabling RetryException in [PWRMODBUS] section of citect.ini:
[PWRMODBUS]
RetryException = 1
Note: RetryException = 0 by default, meaning that retries on exceptions are disabled. The RetryException is a feature of the PwrModbus driver, which was not available in the MODNET driver.
More info about this parameter (which exception codes are retried) can be found in the PwrModbus Driver's help (C:\Program Files (x86)\Schneider Electric\Power Operation\v2022\bin\PWRMODBUSDriverhelp.chm).
The number of default retries for PwrModbus driver is 3 and is handled by the "[PWRMODBUS] retry" parameter.
In this system, a [PWRMODBUS] retry = 2 worked well, however it is parameter that can be tuned.
This resolution applies to a SCADA system matching all the conditions below:
- any Power Operation versions higher than 7.40 (Power SCADA Expert, Power SCADA Operation, Power Operation) patched or unpatched with Cumulative Updates
- any PwrModbus and MODNET driver versions compatible with the above.
