Goals and Symptoms
Overview
The following issues were found during August and September, 2005 on the 8500 line of meters. The V262 release notes contain the information required by TS to resolve these issues this article is focused on providing a technical record of what the problems were and what was done to fix them.
Keywords
I2C, IIC, -97, -105, Modem
Causes and Fixes
Stale Data Bug Description
Symptoms:
DSP temperature of 126 degC.
Root Cause:
Unknown. While this is still under investigation, it is thought that the communications processor module (CPM) in the PowerPC chip is malfunctioning.
Failure Mechanism:
An I2C transaction does not complete successfully. This is evident by viewing the I2C nets with an oscilloscope during I2C transactions. The CPM shows no indication that the I2C transaction has failed, so the meter firmware uses the information in the location that I2C responses are stored. Due to the I2C transaction failure, there is no new data so the data from the last successful transaction is used (i.e. stale data). 126 is the write address of the I/O expander chip on the communication card.
Resolution(s):
The following resolution was implemented in V262 for the 8500 line and V320 for the 7550/7650 line. At the time of writing, the fix had not yet been put in the 8800 code or the 8600 code. To avoid using stale data, a the I2C receive buffer was seeded with a value that is known to not be a valid response from any of the I2C devices. Before using any data from the I2C receive buffer, the firmware now checks to see if the value in the buffer is the illegal value. The illegal value was chosen as 0x80. This value is not within the range of possible temperature values that can be returned from the LM75 devices that we use, nor is this value a valid response from any of the I/O expander devices we use. Consideration must be given when adding new I2C functionality to meters to respect this constraint.
Other Notes:
Tracker SCR = 8500 Specific #1200. Also logged in all other PowerPC issue databases although there has been no evidence of this issue on the MPC823.
Two Byte Return Bug Description
Symptoms:
This issue can occur with the temperature APS_SENSORS or the I/O expander chip on the communication card. If it occurs with the DSP temperature sensor, a temperature of -97degC will be detected. If it occurs with the LCD temperature sensor, a temperature of -105degC will be detected. If the problem occurs with the I/O expander chip, the following symptoms can be expected:
Comm Channel | Symptom with Standard Comm Board (14000-0060) | Symptom with Modem2 Comm Board (14000-0070) |
Com1 232/485 | Will work. | Will work. |
Com2 485 | Will work as long as there is no modem on the card. | N/A for this board. |
Com3 Optical Port | Will work. | Will work. |
Com4 485 | N/A for this board. | Will not work. |
Modem | Will not work. | Will not work. |
Ethernet | Will work if present. | N/A for this board. |
Root Cause:
Unknown. While this is still under investigation, it is thought that the communications processor module (CPM) in the PowerPC chip is malfunctioning.
Failure Mechanism:
A normal I2C read transaction returns the data read from the slave device into the I2C receive buffer. This failure causes the data as well as the I2C address of the slave to be returned into the receive buffer. The firmware was designed to work when only the I2C slave device data was returned so if both values are returned erroneous temperatures and/or communication option detection issues arise.
Resolution(s):
The following patch was added to the 8500 V262 firmware release: For I/O expander chip device read transactions (which detect some of the communication options), the length of data that is written to the I2C receive buffer is used to detect if the data has been returned or if the data and the address have been returned. The firmware then chooses the appropriate memory location so that this bug has no impact on the meter functionality. Note: this fix only patches the I/O expander device functionality and not the temperature sensor functionality so -97 and -105degC temperatures will still be present with V262 firmware. The same resolution could be applied to the temperature sensor devices in future releases of the code.
Other Notes:
Tracker SCR = 8500 Specific #1201. Also logged in all other PowerPC issue databases although there has been no evidence of this issue on the MPC823.