Issue
An Ethernet-connected meter appears to be connected in management console, but is marked offline in Diagnostics Viewer, real-time data cannot be displayed for the meter in Vista, and logged data is not being downloaded from the meter. Testing the connection with ping or by connecting with ION Setup is usually successful. Disconnecting the meter and reconnecting in Management Console usually restores communications to the meter, but sometimes either the PC or the meter must be rebooted.
Note: All references to ION Enterprise are synonyms with System Power Monitoring and Power Monitoring Expert
Symptoms
This article refers to a very specific condition which can be confirmed through the following symptoms:
- Real-time views of the meter in Vista show yellow boxes indicating a meter that is off-line
- The device shows a site status of Connected in Management Console
- Diagnostic Viewer (Communications Diagnostics > Site Overview) shows:
- Device status as offline.
- Good Responses counter is no longer increasing
- Bad Responses is slowly increasing.
- Last Attempt is usually the current time, but
- Last Response is some time in the past. (Normally, attempt time and response time are the same timestamp, or apart by a few seconds)
- A netstat command shows that a connection is still established between ION Enterprise and the meter. At a command prompt, type
netstat an | find <ip address of meter>
Example output from netstat for meter at 10.168.80.148:
TCP 10.168.81.154:2694 10.168.80.148:7700 ESTABLISHED
- To definitively confirm this problem, disconnect the affected meter using Management Console, and reconnect. Repeat step 4, and the netstat command will display more than one connection, including the old one in the state FIN_WAIT_1 (an orphaned connection)
TCP 10.168.81.154:2813 10.168.80.148:7700 ESTABLISHED
- It may be possible that you cannot re-connect to the meter using this method. This usually occurs when all four connections to the meter are now used, often from multiple instances of ION Enterprise. In this case, there are two typical resolutions:
- Reset (power down/up) the meter to clear all connections. Site Server will reestablish the connection on restart.
- Reboot all ION Enterprise servers connected to the meter.
Product Line
ION7550/7650
ION7550 RTU
ION8600
ION8800
ION7500/7600
ION7500 RTU
ION8300/8400/8500
Siemans9510/9610
Siemans9510 ADR
Siemans9500/9600
Siemans9500 ADR
ION Enterprise 5.x, 6.x
StruxureWare Power Monitoring 7.x
Power Monitoring Expert 7.2.x, 8.x
Environment
Ethernet Meters
Cause
The meter enters this state due to a unexpected condition in the TCP/IP stack code in firmware. Generally, users will see this condition only when:
- The meter has been continually connected to the server for a long period of time, and/or
- A large volume of data has been sent FROM the PC to the meter. This is most typically the case with deployments that result in sustained high polling rates. Examples include, continuous real-time polling of all meters in a system for indefinite periods of time (i.e. leaving Vista diagrams open to meters for an extended period of time), or the use of custom meter frameworks that introduce rapid logging (causing the Log Inserter to continuously download records at a high rate)
Resolution
Firmware upgrades are required in order to eliminate this condition. Upgrades at or above the following version numbers will resolve the condition:
- ION7550/7650 V350 (V340 also resolved this problem)
- ION7550 RTU V222
- ION8600: V321
- ION8800: V331
- ION7500/7600 V254
- ION7500 RTU V122
- ION8300/8400/8500 A/B: V265(A), V281(B)
- Siemans9510/9610 V350 (V340 also resolved this problem)
- Siemans9510 ADR V222
- Siemans9500/9600 V254
- Siemans9500 ADR V122
Workaround
As described in symptoms, the meter connection can be re-established by disconnecting and reconnecting the meter in Management Console. This will leave an orphaned connection as described. If the customer is experiencing a large number of these orphaned connections, a third-party tool such as SysInternals TCPView tool can be used to identify these connections, and then close them.
Alternately, the resolutions described above can also be used:
- Reset (power down/up) the meter to clear all connections. Site Server will reestablish the connection on restart.
- Reboot all ION Enterprise servers connected to the meter.