Document fcos/A0199-2


FCO #: A0199-2

SYNOPSIS: Seagate Cheetah 3 FCAL disk drives in A5x00 and T3 Storage Arrays may
          experience higher than expected failure rate

DATE: Sep/27/2002

KEYWORDS: Seagate Cheetah 3 FCAL disk drives in A5x00 and T3 Storage Arrays may
          experience higher than expected failure rate

----------------------------------------------------------------------------
            - Sun Proprietary/Confidential: Internal Use Only -
----------------------------------------------------------------------------

                             FIELD CHANGE ORDER
            (For Authorized Distribution by Enterprise Services)
            
FCO #: A0199-2                                             Date: Sep/27/2002

SYNOPSIS: Seagate Cheetah 3 FCAL disk drives in A5x00 and T3 Storage Arrays
          may experience higher than expected failure rate when left in an
          "idle" state for extended periods.
                    
Sun Alert: No

TOP FIN/FCO REPORT: Yes

PRODUCT_REFERENCE:  Seagate FCAL Cheetah 3 Disk Drive

PRODUCT CATEGORY: Storage / Disk

PRODUCT AFFECTED:

Systems Affected:
 
Mkt_ID   Platform    Model   Description                 Serial Number
------   --------    -----   -----------                 -------------
-         Anysys      All    System Platform Independent       -

X-Options Affected:
 
Mkt_ID   Platform    Model   Description            Serial Number
------   --------    -----   -----------	    -------------
T3          -         All    T3 StorEdge Array            -
T3+         -         All    T3+ StorEdge Array           -
A5200       -         All    A5200 StorEdge Array         -
A5100       -         All    A5100 StorEdge Array         -


AFFECTED PARTS:

Part Number	Description             		Model
-----------	-----------	  			-----
540-4440-01	18GB 10K FCAL Disk (T3 FRU)  		ST318203FC (Cheetah 3)
            
540-4673-01	18GB 10K FCAL Disk (SPUD Bracket) 	ST318203FC (Cheetah 3)
             
540-4367-01	36GB 10K FCAL Disk (T3 FRU) 		ST136403FC (Cheetah 3)
             
540-4191-01	18GB 10K FCAL Disk (SPUD Bracket) 	ST318203FC (Cheetah 3)
            
540-3869-01	9GB 10K FCAL Disk (SPUD Bracket)	ST39103FC  (Cheetah 3)

540-4192-01	36GB 10K FCAL Disk (SPUD Bracket)	ST136403FC (Cheetah 3)


(SCSI Devices)
Type   Vendor    Model     SerialNumber(Min)   SerialNumber(Max)   Firmware
----   ------    -------   -----------------   -----------------   --------
Disk   Seagate   ST318203FC       -                   -            < D94A
Disk   Seagate   ST136403FC       -                   -            < D94A
Disk   Seagate   ST39103FC        -                   -            < D94A


REFERENCES:
  ESC: 537137               
  ESC: 536084
  ESC: 534571   
  ESC: 535693     
  DPCO: 312   
  DPCO: 322   
  PatchID: 111535 (or latest revision)


PROBLEM DESCRIPTION:

Change History 
--------------
A0199-2

 Date Modified: Sep/27/2002
 Updates: SYNOPSIS, REFERENCES, PROBLEM DESCRIPTION, SPECIAL CONSIDERATIONS
 . SYNOPSIS: Added reference to A5x00 and T3 Storage Arrays
 . REFERENCES: PatchID revision change from -01 to -03
 . PROBLEM DESCRIPTION: Added Note about internal system drives
 . SPECIAL CONSIDERATIONS: PatchID rev change from -01 to -03

--------------

Seagate Cheetah 3 FCAL disk drives may experience higher than expected
failure rate when left in an "idle" state for extended periods of time.
The experienced failure mode is Offline or Not Ready.

Root Cause of this issue is due to low resistance, intermittent
shorts between adjacent pins in flash ROM memory chips that use a
red phosphorous inorganic fire retardant molding compound.  These
shorts are caused by electro-migration of metal across adjacent
pins of these memory chips.

The typical senario involves drives that go into an "idle" state,
with power applied, for extended periods of time (usually greater
than 6 months).  A small percentage of these drives may have
experienced this failure already, but it will not be evident until
a data transfer is initiated.  Until then the failed drive appears
to be working and may even respond to a Test Unit Ready command.

Note: Due to the operating profile of internal system drives (no
      prolonged idle time), these drives are not exposed to this
      issue.

New firmware has been developed that decreases the risk of this
electro-migration from occuring by exercising the drives, thereby
avoiding extended idle times.  This firmware also will reveal any
Seagate Cheetah 3 drive sitting in an idle-failed state.

This FCO explains how to determine potential customer exposure, how
to install new firmware that will run a reliability check (which
will "wake-up" any Seagate Cheetah 3 drives out of an "idle"
state),
and how to plan for possible failed drives that will be evident
after the reliability check is run.  

Please follow the directions listed in the SPECIAL INSTRUCTIONS and
CORRECTIVE ACTION sections of this FCO.  It is extremely important
that you read and understand the Question and Answer document as
well as the White Paper (both of which are referenced in this FCO),
prior to implementing any corrective action.

For those customers who are considered secure sites and don't allow
for the removal of disk drives from the site please refer to the
COMMENTS section at the end of this FCO.

Example Error Messages
----------------------

T3 Purple Error Messages (Typical):

 May 07 22:42:29 ISR1[1]: N: u1d6 SVD_RETRY: Retries Exhausted (ccb =
0x1384034)
 May 07 22:42:29 ISR1[1]: N: u1d6 SVD_DONE: Command Error = 0x4
 May 07 22:42:29 ANNT[1]: W: u1d6: Failed
 May 07 22:42:31 LPCT[1]: N: u1d6: Bypassed on loop 1
 May 07 22:42:31 ISR1[1]: N: u1d6 SVD_RETRY: Retries Exhausted (ccb =
0x13918f4)
 May 07 22:42:31 ISR1[1]: N: u1d6 SVD_DONE: Command Error = 0x4
 May 07 22:42:31 snmp[1]: N: u1d6 ioctl disk failed err=1
 May 07 22:42:31 LPCT[1]: N: u1d6: Bypassed on loop 2

 Mar 23 23:01:16 ISR1[2]: N: u1d7 SVD_DONE: Command Error = 0x3
 Mar 23 23:01:16 ISR1[2]: W: u1d7 SCSI Disk Error Occurred (path = 0x0)
 Mar 23 23:01:16 ISR1[2]: W: Sense Key = 0x2, Asc = 0x4, Ascq = 0x2
 Mar 23 23:01:16 ISR1[2]: W: Sense Data Description = Logical Unit Not Ready,

A5x00 Photon error messages (Typical):

 socal3: port 0: Fibre Channel is OFFLINE
 WARNING: /sbus@b,0/SUNW,socal@0,0/sf@1,0/ssd@w2100002037aef8c8,0 (ssd164):
 SCSI transport failed: reason 'tran_err': retrying command

 socal3: port 1: Fibre Channel is OFFLINE
 WARNING: /sbus@3,0/SUNW,socal@0,0/sf@1,0/ssd@w21000020378837f2,0 (ssd171):
 SCSI transport failed: reason 'timeout': giving up

 May 20 10:59:58 gmi2wap011 scsi: [ID 107833 kern.notice] Sense Key: Not Ready
 May 20 10:59:58 gmi2wap011 scsi: [ID 107833 kern.notice] ASC: 0x4 (), ASCQ: 
0x1, FRU: 0x2

Specific environmental conditions which exacerbate drive failures are;

  . exposure to high humidity
  . prolonged drive idle times of greater than 5,000 hours

Long periods of drive idle or constant dwell, with power on, provides a
voltage bias which promotes electro-migration of silver/copper between
the failing, adjacent pins on the chip package.

For more information reference the White Paper at the following URL:

  http://sdpsweb.EBay/FIN_FCO/FCO/FCO_A0199-1_Dir/WhitePaper.ps

A Question and Answer document has been provided to help the field
understand and properly communicate the issue to Sun's customers.
This document can be found at the following URL:

  http://sdpsweb.EBay/FIN_FCO/FCO/FCO_A0199-1_Dir/QandA.ps

A Sun legal approved Customer Letter can be found at the following URL:

  http://sdpsweb.EBay/FIN_FCO/FCO/FCO_A0199-1_Dir/CustomerLetter.sxw

Note: To view document click on the above URL, then save to your local
      disk using your Netscape 'file' button and select 'save as', then
      open file locally using StarOffice.

All field RSLs were purged via DPCO 312 starting on June 19, 2002.
Corrective action was made available by releasing patch 111535
on September 24, 2002.


PLANNED IMPLEMENTATION COMPLETION DATE: February 28, 2003


IMPLEMENTATION:

 ---
|   |   MANDATORY (Fully Pro-Active)
 ---

 ---
| X |   CONTROLLED PRO-ACTIVE (per Sun Geo Plan)
 ---

 ---
|   |   UPON FAILURE
 ---

REPLACEMENT TIME ESTIMATE: 2 hours


SPECIAL CONSIDERATIONS:

This FCO will have a timezone phased release based on material
readiness as follows:

US/Canada       August 30, 2002
EMEA            September 11, 2002
Ltn America     September 11, 2002
APac/Japan      September 18, 2002


IMPORTANT: Implementation of this FCO must be performed in two steps:

1. Controlled Proactive Reliability Screen of customer system(s),
   and Patch installation - PatchID 111535 (or later).
   
2. Replacement of failed disks.  Only disks identified as faulty
   by the installed patch should be replaced.
   
   Proactive swap of disk drives is NOT authorized by this FCO.

 ###########################################################

     ***  Seagate Cheetah 3 Disk Drive Health Check  ***

 ###########################################################

The FCO corrective action consists of a short term immediate action, and a
longer term monitoring action.  Prior to performing the firmware install, or
prior to initiating the Seagate Cheetah 3 disk drives HEALTHCHECK and SCREEN
ensure you have:

        A. Read the complete FCO Instructions

        B. Understand the disk drive reliability screen

        C. Have advised your customer of possible results from installing
           the firmware provided via this FCO

        D. Are prepared, from a time and material perspective (meaning the
           customer has provided an adequate maintenance window, and you
           have staged a sufficient number of spares on site) prior to
           implementing the on-site firmware upgrade.

+ Short Term Immediate Action:
  ----------------------------

OVERVIEW:

        A. PREPARE FOR DISK DRIVE RELIABILITY SCREEN:

                A1. IDENTIFY SUSPECT DRIVE POPLULATION AT CUSTOMER SITE
                A2. DEFINE RELIABILITY HISTORY PROFILE
                A3. DETERMINE IF CUSTOMER IS EXPERIENCING HIGHER THAN
                    EXPECTED FAILURE RATE
                A4. CALCULATE NUMBER OF SPARES REQUIRED TO SUPPORT INSTALL
                    OF HEALTHCHECK SCRIPT AND FIRMWARE

        B. PLAN/PREPARE FOR FIRMWARE INSTALL

                B1. INSTALL NEW FIRMWARD
                B2. IDENTIFY ANY LATENT (HIDDEN) FAILED DRIVES
                B3. SWAP ANY FAILED DRIVES WITH SPARE DRIVES
                B4. RERUN THE FW SCRIPT TO UPDATE THE FW OF ANY REPLACED DRIVES
                B5. VERIFY PROPER OPERATION OF SYSTEM. 


+ Long Term Monitoring Action:
  ---------------------------

        A. MONITOR THE SITE FOR CHEETAH 3 DRIVE FAILURES
        B. IF FAILURE RATE REMAINS ABOVE EXPECTED LEVELS ESCALATE SERVICE ORDER
           PER LOCAL GUIDELINES (reference #6 for failure rates)

NOTE:
-----

The firmware provides several fixes, including an on the drive function
to exercise flash ROM chip pins during drive idle periods.  This fix
addresses a contributing factor to Cheetah 3 drive failures; extensive
drive idle time.

Engineering review of the initial customer sites reporting failed
Cheetah 3 disk drives revealed the following:

        - Engineering failure analysis of Cheetah 3 drives with failed
          flash ROM chips ALL displayed a similar drive usage pattern;

                - Extensive idle time (7,000 hours or longer) followed by;
                - A sudden increase in I/O activity.

        - Each customer site contained a small percentage of drives that
          had been in a idle mode for thousands of hours

        - Idle drives can be the function of:

                - data base access; the data is there, but is not accessed
                - striping format and read/writes
                - test systems

Due to the engineering findings:

        - Be advised that your customer may have suspect drives.
        - Suspect drives are drives that have been sitting idle for long 
          periods of time.
        - Running the Healthcheck script will "wake-up" idle drives,
and 
          will reveal any Seagate Cheetah 3 drive sitting in an idle-failed 
          state.
        - Before running the Healthcheck script it is critical that you 
          evaluate site reliability, complete analysis, and have an 
          appropriate number of spares on site when performing the firmware 
          install.


A. PREPARE FOR DISK DRIVE RELIABILITY SCREEN:

A1. IDENTIFY SUSPECT DRIVE POPLULATION AT CUSTOMER SITE
------------------------------------------------------------------------

            ***  Seagate Cheetah 3 Disk Drive Reliability Screen  ***

------------------------------------------------------------------------
#1.  Define the customer's install base by part number.  Use explorer or
     T3 extractors to identify the part numbers.

 Product/model          Part Number        Model    Quantity   Months of
                                                               Service

Servers/E3500           540-3869-01     (9.1GB)    ___________ ___________
ST39103FC
Servers/E3500           540-4191-01     (18.2GB)   ___________ ___________
ST318203FC
Servers/E3500           540-4673-01     (18.2GB)   ___________ ___________
ST318203FC
Servers/280R            540-4191-01     (18.2GB)   ___________ ___________
ST318203FC
Servers/SB 1000/2000    540-4673-01     (18.2GB)   ___________ ___________
ST318203FC
A5000 Array             540-4192-01     (36.4GB)   ___________ ___________
ST136403FC
A5100 Array             540-4192-01     (36.4GB)   ___________ ___________
ST136403FC
A5200 Array             540-3869-01     (9.1GB)    ___________ ___________
ST39103FC
T3                      540-4440-01     (18.2GB)   ___________ ___________
ST318203FC
T3                      540-4367-01     (36.4GB)   ___________ ___________
ST136403FC

    NOTE: You must run explorer script to identify all Seagate drives.

A2. DEFINE RELIABILITY HISTORY PROFILE
---------------------------------------

#2.  List the number of failures by part number and manufacturer:

Part Number/     Capacity  Quantity      Model #         Failure Mode
Model

540-3869-01     (9.1GB)    _________    ______________  _____________
ST39103FC
540-4191-01     (18.2GB)   _________    ______________  _____________
ST318203FC
540-4673-01     (18.2GB)   _________    ______________  _____________
ST318203FC
540-4192-01     (36.4GB)   _________    ______________  _____________
ST136403FC
540-4440-01     (18.2GB)   _________    ______________  _____________
ST318203FC
540-4367-01     (36.4GB)   _________    ______________  _____________
ST136403FC

A3. DETERMINE IF CUSTOMER IS EXPERIENCING HIGHER THAN EXPECTED FAILURE RATE
---------------------------------------------------------------------------

#3. Does the number of drive failures exceed the expected failure rate?
    (See chart below)

    Note: an example follows.

 |------------------------------------------------------------
 | # of drives    | 3 months | 6 months| 9 months| 12 months |
 |  on site       |          |         |         |           |
 |===========================================================|
 |     300        |    1-2   |  3-4    |    4-5  |     6     |
 |----------------|----------|---------+---------+-----------|
 |     400        |     2    |   4     |      6  |     8     |
 |----------------|----------|---------|---------|-----------|
 |     500        |    2-3   |  5-6    |   7-8   |     10    |
 |----------------|----------|---------|---------|-----------|
 |    1,000       |     5    |   10    |     15  |     20    |
 |----------------|----------|---------|---------|-----------|
 |    1,500       |    8-9   |  14-15  |     24  |     30    |
 |----------------|----------|---------|---------|-----------|
 |total  failures |          |         |         |           |
 |-----------------------------------------------------------|

 Example:

 In this example customer X has the following Seagate Cheetah 3
 drives in the data center:

 Quantity   drive type            install          # of
                                  time             failures

 300        540-3869-01 (9.1GB)   approx 9 months     9

 500        540-4191-01 (18.2GB)  approx 9 months    14

 400        540-4192-01 (36.4GB)  approx 12 months    6

 That's 300 9GB drives installed for nine months, and 500 18GB
 drives installed for nine months.  The customer also has 400
 36GB drives that have been installed for about twelve months.

 Example: Fill in a chart defining your customer's information:

                   -------------------------------------------
                  | List # of failures during install period |
 |------------------------------------------------------------
 | # of drives    | 3 months | 6 months| 9 months| 12 months |
 |  on site       |          |         |         |           |
 |===========================================================|
 |     300        |          |         |     9   |           |
 |----------------|----------|---------|---------|-----------|
 |     400        |          |         |         |     6     |
 |----------------|----------|---------|---------|-----------|
 |     500        |          |         |    14   |           |
 |----------------|----------|---------|---------|-----------|
 |    1,000       |          |         |         |           |
 |----------------|----------|---------|---------|-----------|
 |    1,500       |          |         |         |           |
 |----------------|----------|---------|---------|-----------|
 |total  failures |          |         |    23   |     6     |
 |-----------------------------------------------------------|

 Comparing the customer drive reliability profile with the
 expected chart below.
 --------------------------------------------------------------
 | Drives/install time  | Expected failures | Actual failures |
 |====================================================+=======|
 | 300/9 months  9GB    |    4-5            |       9         |
 |----------------------|-------------------|-----------------|
 | 500/9 months  18GB   |    7-8            |      14         |
 |----------------------|-------------------|-----------------|
 | 400/12 months 36GB   |     8             |      6          |
 |------------------------------------------------------------|

 Looking at the numbers we can now answer the question, "Does the number
 of drive failures exceed the expected failure rate?"

 In this example the answer is YES and NO.

   We would expect no more than 4-5 9GB drives to fail in 9 months,
   the customer had 9.  YES the failure rate exceeds expected.

   We would expect no more than 7-8 18GB drives to fail in 9 months,
   the customer had 14.  YES the failure rate exceeds expected.

   We would expect no more than 8 36GBB drives to fail in 12 months,
   the customer had 6.  NO the failure rate does not exceed expected.


#4. Have the majority of the drives failed due to 'logical unit not
    ready' or 'drive not ready' errors? Y or N

    If yes to # 4, it is recommended you proceed with the Reliability
    Screen and Healthcheck.

    If no, reevaluate the situation, escalate per local guidelines.

A4. CALCULATE NUMBER OF SPARES REQUIRED TO SUPPORT INSTALL OF HEALTHCHECK
    SCRIPT AND FIRMWARE
--------------------------------------------------------------------

#5. Based on the reliability information, calculate the expected number
    of drives that may be sitting in an idle-failed state.

#5A. Calculate failure rate compared to expected failure rate:

Part Number/   Capacity   QTY on    Recorded    Expected #   Results**:
Model #                   site      Failures    Failures     Less than/
                                                             equal to/
                                                             greater than
                                                             expected

540-3869-01/    (9.1GB)    _____    _________   __________   _____________
ST39103FC
540-4191-01/    (18.2GB)   _____    _________   __________   _____________
ST318203FC
540-4673-01/    (18.2GB)   _____    _________   __________   _____________
ST318203FC
540-4192-01/    (36.4GB)   _____    _________   __________   _____________
ST136403FC
540-4440-01/    (18.2GB)   _____    _________   __________   _____________
ST318203FC
540-4367-01/    (36.4GB)   _____    _________   __________   _____________
ST136403FC

**Results category:

   from 0 - 1.4            - Failure rate better than expected:


   from 1.5 - 2.8          - Failure rate equal to expected:


   greater than 2.9        - Failure rate greater than expected:


EXAMPLE:

Quantity   drive type            install            # of
                                  time             failures
--------   ----------		 -------	   --------
 300       540-3869-01 (9.1GB)   approx 9 months      9

 500       540-4191-01 (18.2GB)  approx 9 months     14

 400       540-4192-01 (36.4GB)  approx 12 months     6



Part Number/   Capacity   QTY on    Recorded    Expected #   Results:
Model #                   site      Failures    Failures     Less than/
                                                             equal to/
                                                             greater than
                                                             expected
------------   --------   ------    --------	--------     ------------

540-3869-01/    (9.1GB)    300        9            8        Greater than
ST39103FC                                                   expected;
                                                            9/8 = 1.125 times
                                                            expected

540-4440-01/    (18.2GB)   500        14          7-8       Greater than
ST318203FC                                                  expected;
                                                            14/7 = 2 times
                                                            expected

540-4367-01/    (36.4GB)   400        6           7-8       Better than
ST136403FC                                                  expected;
                                                            6/8 = .75 times
                                                            expected

#6. Calculate forecasted number of possible idle-failed state disk drives:

        Forecast factors

                Failure rate better than expected:      1% or .01

                        less than 1.0

                Failure rate equal to expected:         2% or .02

                        from 1.5 - 2.8

                Failure rate greater than expected:     4% or .04

                        greater than 2.9


        Multiply the drive population by the applicable forecast factor.


Part Number       Model   Drive        Forecast Factor  Results:
                          Population   Failures %       Drives required on
                                                        site to install new
                                                        firmware
-----------	--------  -----------  --------------   --------------

540-3869-01/    (9.1GB)    _________    See #6 above    _____________
ST39103FC
540-4191-01/    (18.2GB)   _________    See #6 above    _____________
ST318203FC
540-4673-01/    (18.2GB)   _________    See #6 above    _____________
ST318203FC
540-4192-01/    (36.4GB)   _________    See #6 above    _____________
ST136403FC
540-4440-01/    (18.2GB)   _________    See #6 above    _____________
ST318203FC
540-4367-01/    (36.4GB)   _________    See #6 above    _____________
ST136403FC

FORMULA: (Drive Population) x (Forecast Factor Failures) = Drives required for
                                                           on site

EXAMPLE:

Customer has 400 9GB drives, 300 18GB drives, and 500 36GB drives.

Part Number       Model   Drive        Forecast Factor  Results:
                          Population   Failures         Drives required on
                                                        site to install new
                                                        firmware
------------   --------   ------        ----------	-----------------

540-3869-01/    (9.1GB)    400          4% or .04               16

540-4440-01     (18.2GB)   300          2% or .02                6

540-4367-01     (36.4GB)   500          4% or .04               20

in this case:

(Drive Population) x (Forecast Factor Failures) = Drives required on-site

        400        x    .04             =                       16

        300        x    .02             =                        6

        500        x    .04             =                       20
                                                        -------------------

                Total number of spares to have on-site          42

Advise your customer that these results equal the forecasted number of
idle-failed state disk drives at the site.  This is the number of
failed drives that may be discovered when installing the new firmware.


B. PLAN/PREPARE FOR FIRMWARE INSTALL


B1. INSTALL NEW FIRMWARD - PatchID 111535 (or latest revision)
--------------------------------------------

#1. Ensure the number of spares required to support the firmware install
    are available and on-site.

#2. Install Cheetah 3 disk drive healthcheck script.

Reference: Patch-ID# 111535 (or latest revision)
Keywords:  ST39103FC ST318203FC ST136403FC 9GB 18GB 36GB disk
           firmware FC-AL

B2. IDENTIFY ANY LATENT (HIDDEN) FAILED DRIVES / MONITOR FOR ERRORS
-------------------------------------------------------------------

#3. Monitor systems for reported drive failures.

B3. SWAP ANY FAILED DRIVES WITH SPARE DRIVES
--------------------------------------------

#4. Replace failed drives as required.


B4. RERUN THE FW SCRIPT TO UPDATE THE FW OF ANY REPLACED DRIVES
---------------------------------------------------------------

Reference: Patch-ID# 111535 (or latest revision)
Keywords:  ST39103FC ST318203FC ST136403FC 9GB 18GB 36GB disk
           firmware FC-AL
           
Only required for any replaced drives.


B5. VERIFY PROPER OPERATION OF SYSTEM
-------------------------------------

#5. Restore target configuration and data (if neccessary), and verify
    proper operation of system.


+ Long Term Monitoring Action:
  ---------------------------

#1. MONITOR THE SITE FOR CHEETAH 3 FCAL DRIVE FAILURES.

#2. SWAP DRIVES (IF REQUIRED).

#3. SUBMIT DRIVES FOR CPAS AND OPEN AN ESCALATION IF FAILURE
    RATE REMAINS ABOVE EXPECTED LEVELS.

           Use the table below to evaluate if drive failure
           rates are above expectations:
           
                |---------------------------|
                | # of drives    |  2 months|
                |  on site       |          |
                |============================
                |     300        |    6     |
                |----------------|----------|
                |     400        |    8     |
                |----------------|----------|
                |     500        |   10     |
                |----------------|----------|
                |    1,000       |   20     |
                |----------------|----------|
                |    1,500       |   30     |
                |----------------|----------|
                |total  failures |          |
                |----------------------------


CORRECTIVE ACTION:

IMPORTANT! Please follow the Disk Drive Health Check process
listed under the Special Consideration section of this FCO
prior to implementing any other corrective action.

Implementation of this FCO must be performed in two steps:

1. Controlled Proactive Reliability Screen of customer system(s),
   and Patch installation - PatchID 111535 (or later).
   
2. Replacement of failed disks.  Only disks identified as faulty
   by the installed patch should be replaced.
   
*** Proactive swap of disk drives is NOT authorized by this FCO. ***

Upon failure replace as follows;

replace 540-3869-01 with 540-3869-01 (having DPCO 312 label)
replace 540-4191-01 with 540-4191-01 (having DPCO 312 label)
replace 540-4673-01 with 540-4673-01 (having DPCO 312 label)
replace 540-4192-01 with 540-4192-01 (having DPCO 312 label)
replace 540-4440-01 with 540-4440-01 (having DPCO 312 label)
replace 540-4367-01 with 540-4367-01 (having DPCO 312 label)


COMMENTS:


SECURE SITE ACTIVITY:

Please follow standard, local procedures for Secure Site replacements.


BILLING TYPE:

 Warranty: Sun will provide parts at no charge under Warranty
           Service. On-Site Labor Rates are based on how the
           system was initially installed.

 Contract: Sun will provide parts at no charge. On-Site Labor Rates
           are based on the type of service contract.

 Non Contract: Sun will provide parts at no charge. Installation by
               Sun is available based on the On-Site Labor Rates
               defined in the Price List.

--------------------------------------------------------------------------
Implementation Footnote:
________________________

i)   In case of Mandatory FCOs, Sun Services will attempt to contact
      all known customers to recommend the part upgrade.

ii)  For controlled proactive swap FCOs, Sun Services mission critical
     support teams will initiate proactive swap efforts for their respective
     accounts, as required.

iii) For Replace upon Failure FCOs, Sun Services partners will implement
     the necessary corrective actions as and when they are required.

--------------------------------------------------------------------------

All released FINs and FCOs can be accessed using your favorite network
browser as follows:

SunWeb Access:
______________

* Access the top level URL of http://sdpsweb.EBay/FIN_FCO/

* From there, select the appropriate link to query or browse the FIN and
  FCO Homepage collections.

SunSolve Online Access:
_______________________

* Access the SunSolve Online URL at http://sunsolve.Central/

* From there, select the appropriate link to browse the FIN or FCO index.

Internet Access:
_______________

* Access the top level URL of https://infoserver.Sun.COM

--------------------------------------------------------------------------
General:
________

Send questions or comments to finfco-manager@sun.com

---------------------------------------------------------------------------


Copyright (c) 1997-2003 Sun Microsystems, Inc.