Document fins/I0760-1
FIN #: I0760-1
SYNOPSIS: Too many Memory DIMMs are being unnecessarily replaced on UltraSPARC
III family of systems utilizing NG-DIMM memory
DATE: Aug/04/02
KEYWORDS: Too many Memory DIMMs are being unnecessarily replaced on UltraSPARC
III family of systems utilizing NG-DIMM memory
---------------------------------------------------------------------
- Sun Proprietary/Confidential: Internal Use Only -
---------------------------------------------------------------------
FIELD INFORMATION NOTICE
(For Authorized Distribution by SunService)
SYNOPSIS: Too many Memory DIMMs are being unnecessarily replaced on
UltraSPARC III family of systems utilizing NG-DIMM memory.
Sun Alert: No
TOP FIN/FCO REPORT: No
PRODUCT_REFERENCE: UltraSPARC III family of systems utilizing NG-DIMM memory
PRODUCT CATEGORY: Server / Service
PRODUCTS AFFECTED:
Systems Affected
----------------
Mkt_ID Platform Model Description Serial Number
------ -------- ----- ----------- -------------
- S8 ALL Sun Fire 3800 -
- S12 ALL Sun Fire 4800 -
- S12i ALL Sun Fire 4810 -
- S24 ALL Sun Fire 6800 -
- F15K ALL Sun Fire 15000 -
- A28 ALL Sun Blade 1000 -
- A35 ALL Sun Fire 280R -
- A30 ALL Sun Fire V880 -
- N28 ALL Netra 20 -
X-Options Affected
------------------
Mkt_ID Platform Model Description Serial Number
------ -------- ----- ----------- -------------
- - - - -
PART NUMBERS AFFECTED:
Part Number Description Model
----------- ----------- -----
- - -
REFERENCES:
N/A
PROBLEM DESCRIPTION:
Memory components (DIMMs) for UltraSPARC III family of systems
utilizing NG-DIMM memory are being returned from the field as failed
based on Correctable Error (CE) reports. However, upon failure
analysis, most of these memory parts show No Trouble Found (NTF). The
intent of this FIN is to provide Sun Service Representatives an
overview of ECC and to give criteria for replacing DIMMs. This FIN is
expected to reduce or eliminate unnecessarily replaced DIMMs.
For the UltraSPARC III family of systems utilizing NG-DIMM memory, it
has been reported that DIMMs are suspected to have failed for ECC
errors and are being replaced in the field unnecessarily. This may
partially be caused by a lack of understanding by Field Engineers of
what actually constitutes ECC, what are the definitions of different
terms related to ECC, and what is the criteria to determine when ECC
errors are to be considered excessive.
Failure analysis on suspected failed DIMMs, which are returned from the
field, has determined that nearly 100% turn out to be NTF. This is
causing a substantial impact on the valuable resources of Engineering,
Operations and Service. Further, the cost of procuring additional
DIMMs in order to maintain Target Stocking Levels (TSL) in the field
is very high.
The following ECC overview should help in providing an understanding
of this issue:
An Overview of ECC
Introduction:
=============
The recent launch of the UltraSPARC III family of systems utilizing
NG-DIMM memory, coupled with recent changes that have been
implemented and released in the Solaris 8 Operating Environment,
has led to some confusion about reported ECC errors and whether
these events are indications of hardware that needs replacement.
This document is intended to give a brief overview of ECC to
explain why these events occur and what action, if any, should be
taken when they do.
The scope of this discussion is limited to soft errors that occur
in memory and how they are reported by Solaris. It does not
account for hard errors or errors that occur while data travels
through the system interconnect. It also does not account for
information made available to the service processor. As such, the
concepts discussed here can be applied to any system, not just
UltraSPARC III family of systems utilizing NG-DIMM memory. For
this discussion, soft errors are transient errors in memory that
can be corrected by rewriting the affected memory cell. Hard
errors occur when a cell is permanently damaged and cannot hold the
correct information. Sometimes the cell can be stuck at "0",
sometimes it can be stuck at "1".
ECC Concepts:
=============
Any non-persistent storage device, whether it be Dynamic Random
Access Memory (DRAM) used for main memory or Static Random Access
Memory (SRAM) used for caches, is subject to occasional incidences
of data loss due to the impact of energetic alpha particles or cosmic
rays. This data loss manifests itself in the changing of the value
stored in the memory location affected by the collision. Typically
only a single bit is affected, but there is a small probability (<10%)
that multiple cells can be upset.
When a bit flips due to this phenomenon, it is referred to as a soft
error. This is to distinguish it from a hard error resulting from
faulty hardware. These soft errors happen at a rate, called the soft
error rate, that can be predicted as a function of the memory density,
the memory technology, and the geographic location of the memory system.
ECC was invented to facilitate the survival of these naturally
occurring losses of data. The concept is that every word of data stored
in memory also has check information stored along with it. This check
information serves two purposes. First, when a word of data is read out
of memory, the check information can be used to detect if any of the
bits of the word have changed, and whether just a single bit has
changed or more than one bit has changed. Second, in the event that a
single bit has changed, the check information can be used to determine
which bit in the word changed and therefore correct the word by
flipping this bit back to its complementary value.
When an ECC check mechanism has detected that one or more bits in a
word of data has changed, this is broadly categorized as an ECC error.
These errors can be further categorized as a function of the number of
bits in error. Because ECC can correct single bit flips, single bit
errors are referred to as Correctable Errors (CEs). Multi-bit errors
are referred to as Uncorrectable Errors (UEs).
Solaris Behavior:
=================
When a CE is detected, the device that read the word and detected the
event can certainly correct the data and continue on unimpeded. However,
this does not address the fact that the referenced word is still
resident in memory uncorrected, i.e., a subsequent read of this word
would result in another CE event. If, over time, this word in memory
is never corrected, the possibility starts to arise that another bit
may flip in the same word. This would lead to a UE event. To avoid
this possibility, the detection of a CE causes a trap to Solaris.
The resultant error handling code scrubs the affected memory word by
writing the corrected word back into memory.
As part of handling the error, Solaris will proceed to log a fair
amount of diagnostic information. One such event log, taken from a
Sun Fire 6800 running Solaris 8, looks like the following:
Oct 25 09:06:25 wpc26 SUNW,UltraSPARC-III: [ID 796192 kern.notice]
NOTICE:[AFT0]
Corrected system bus (CE) Event on CPU18 at TL=0, errID 0x0000c9b9.19d92690
Oct 25 09:06:25 wpc26 AFSR 0x00000002<CE>.00000097 AFAR
0x00000001.04bdf7d0
Oct 25 09:06:25 wpc26 Fault_PC 0x10024a74 Esynd 0x0097 /N0/SB5/P3/B0/D2 J16500
Oct 25 09:06:25 wpc26 SUNW,UltraSPARC-III: [ID 154767 kern.notice] [AFT0] errID
0x0000c9b9.19d92690 Corrected Memory Error on /N0/SB5/P3/B0/D2 J16500 is
Persistent
Oct 25 09:06:25 wpc26 SUNW,UltraSPARC-III: [ID 682217 kern.notice] [AFT0] errID
0x0000c9b9.19d92690 Data Bit 3 was in error and corrected
Oct 25 09:06:25 wpc26 SUNW,UltraSPARC-III: [ID 422650 kern.info] [AFT2] errID
0x0000c9b9.19d92690 E$tag PA=0x00000000.00bdf7c0 does not match
AFAR=0x00000001.04bdf7c0
Oct 25 09:06:25 wpc26 SUNW,UltraSPARC-III: [ID 904800 kern.info] [AFT2] errID
0x0000c9b9.19d92690 PA=0x00000000.00bdf7c0
Oct 25 09:06:25 wpc26 E$tag 0x00000000.01000001 E$state_7 Invalid
Oct 25 09:06:25 wpc26 SUNW,UltraSPARC-III: [ID 895151 kern.info] [AFT2] E$Data
(0x00)
0x5a8d0016.00000a20 0x20202020.37333231 ECC 0x128
Oct 25 09:06:25 wpc26 SUNW,UltraSPARC-III: [ID 895151 kern.info] [AFT2] E$Data
(0x10)
0x37333330.32062c00 0x5a8f000c.00000a20 ECC 0x1f6
Oct 25 09:06:25 wpc26 SUNW,UltraSPARC-III: [ID 895151 kern.info] [AFT2] E$Data
(0x30)
0x20202020.37333330 0x34062c00.5a8f000d ECC 0x1fc
Oct 25 09:06:25 wpc26 SUNW,UltraSPARC-III: [ID 929717 kern.info] [AFT2] D$ data
not available
Oct 25 09:06:25 wpc26 SUNW,UltraSPARC-III: [ID 335345 kern.info] [AFT2] I$ data
not available
For the case of a CE, the lines tagged with AFT0 are the most important
ones to examine. The lines tagged with AFT2 provide data for detailed
diagnostic evaluation, which is not expected to occur in the field.
Points that need explanation are the following:
1. The event was detected by CPU18 (line 2). All this really means is
that CPU18 is the processor that took the trap, thus invoking the
Solaris CE error handling code.
2. The DIMM containing the affected memory location is /N0/SB5/P3/B0/D2,
which has a reference designator on the Sun Fire system board of
J16500 (lines 4 and 6). This is the important information, not by
itself, but in conjunction with other events reported over time, as
will be described in the next section.
3. Solaris describes this event as Persistent (line 6). The Solaris
error handling code provides a disposition code as a result of the
scrub operation. This disposition is one of Intermittent,
Persistent, or Sticky. The definition of each of these codes is:
o Intermittent means the error was not detected on a reread of the
affected memory location.
o Persistent means the error was detected again on a reread of the
affected memory location but the scrub operation corrected it.
o Sticky means that the error still exists in memory even after the
scrub operation. These events should be investigated further to
determine if some hardware replacement is necessary since this is
indicative of a hard failure.
Servicing Memory Based on Soft Errors:
--------------------------------------
As discussed earlier, soft errors are naturally occurring events. As
such, a single report of a CE should not be the basis for servicing/
replacing a memory device. In fact, one should expect the number of CEs
reported by a system to correlate with the soft error rate that can be
predicted by the amount of memory in the system and the geographic
location of the system. Rather than going through system specific
calculations to determine acceptable soft error rates, the guideline
that is recommended for servicing of memory in the presence of soft
errors is the following:
NOTE: Three or more CE's attributed to the same memory module within
a 24 hour period is not acceptable.
Effective with Solaris 9 KU1 and Solaris 8 KU16, there is now
functionality implemented in Solaris that notifies the administrators of
excessive CE events.
In Solaris 9 KU1 and Solaris 8 KU16, two new tunables, ecc_softerr_limit
and ecc_softerr_interval, are introduced. A per-DIMM count is kept for
the number of intermittent CEs that occurred. This count is decremented
every (ecc_softerr_interval/ecc_softerr_limit) seconds. If the count ever
exceeds ecc_softerr_limit, the following message is printed out:
Mar 22 13:12:31 wpc26 unix: WARNING: [AFT0] 3 soft errors in less than
24:00 (hh:mm) detected from Memory Module U1004
The default values are 1440 seconds (24 hours) for ecc_softerr_interval
and 2 for ecc_softerr_limit. These values can be changed by adding
entries in /etc/system. For example:
set ecc_softerr_interval=2880
set ecc_softerr_limit=4
Note that ecc_softerr_interval is defined in seconds.
IMPLEMENTATION:
---
| | MANDATORY (Fully Pro-Active)
---
---
| | CONTROLLED PRO-ACTIVE (per Sun Geo Plan)
---
---
| X | REACTIVE (As Required)
---
CORRECTIVE ACTION:
The following recommendation is provided as a guideline for authorized
Enterprise Services Field Representatives who may encounter the above
mentioned situation.
For the three categories of ECC memory errors that Solaris reports
(Intermittent, Persistent, Sticky), the following guidelines should be
followed for replacement of DIMMs on Sun Fire Midframe and High-End
servers.
. Intermittent: Check for the reporting of parity errors, otherwise
ignore.
. Persistent: Replace DIMM if a message is output to the console
warning of excess errors.
. Sticky: Replace DIMM on first occurrence.
COMMENTS:
============================================================================
Implementation Footnote:
i) In case of MANDATORY FINs, Enterprise Services will attempt to
contact all affected customers to recommend implementation of
the FIN.
ii) For CONTROLLED PROACTIVE FINs, Enterprise Services mission critical
support teams will recommend implementation of the FIN (to their
respective accounts), at the convenience of the customer.
iii) For REACTIVE FINs, Enterprise Services will implement the FIN as the
need arises.
----------------------------------------------------------------------------
All released FINs and FCOs can be accessed using your favorite network
browser as follows:
SunWeb Access:
--------------
* Access the top level URL of http://sdpsweb.ebay/FIN_FCO/
* From there, select the appropriate link to query or browse the FIN and
FCO Homepage collections.
SunSolve Online Access:
-----------------------
* Access the SunSolve Online URL at http://sunsolve.Corp/
* From there, select the appropriate link to browse the FIN or FCO index.
Internet Access:
----------------
* Access the top level URL of https://infoserver.Sun.COM
--------------------------------------------------------------------------
General:
--------
* Send questions or comments to finfco-manager@Sun.COM
--------------------------------------------------------------------------
Copyright (c) 1997-2003 Sun Microsystems, Inc.