Document fins/I0870-1
FIN #: I0870-1
SYNOPSIS: Sun StorEdge 69x0 Series in a clustered environment may encounter
SCSI-2 Reservation issues
DATE: Aug/27/02
KEYWORDS: Sun StorEdge 69x0 Series in a clustered environment may encounter
SCSI-2 Reservation issues
---------------------------------------------------------------------
- Sun Proprietary/Confidential: Internal Use Only -
---------------------------------------------------------------------
FIELD INFORMATION NOTICE
(For Authorized Distribution by SunService)
SYNOPSIS: Sun StorEdge 69x0 Series in a clustered environment may
encounter SCSI-2 Reservation issues.
SunAlert: Yes
TOP FIN/FCO REPORT: Yes
PRODUCT_REFERENCE: Sun StorEdge 69x0 Arrays
PRODUCT CATEGORY: Storage / SW Admin
PRODUCTS AFFECTED:
Systems Affected:
-----------------
Mkt_ID Platform Model Description Serial Number
------ -------- ----- ----------- -------------
- ANYSYS - System Platform Independent -
X-Options Affected:
-------------------
Mkt_ID Platform Model Description Serial Number
------ -------- ----- ----------- -------------
- 6910 - Sun StorEdge 6910 Array -
- 6960 - Sun StorEdge 6960 Array -
PART NUMBERS AFFECTED:
Part Number Description Model
----------- ----------- -----
- - -
REFERENCES:
BugId: 4685803: scsi 2 test failed on Vicom with ioctl(MHIOCTKOWN)
returns -1 and errno 13.
4709377: MHIOCSTATUS return value unexpected indicating access
to a lun.
4709898: qlc fw: task mgmt operations complete before completion
of actual xchg.
PatchId: 
113247:
112580: SUNWstade Brocade, T3 2.1 F/W, 3900SL and 6900SL
Support patch.
112276 or higher: T3+ 2.01.00: System Firmware Update.
Manual: 816-3244-10: Sun StorEdge 3900 and 6900 Series Installation and
Service Manual.
816-5253-10: Sun StorEdge"3900 and 6900 Series 1.1 Reference and
Service Manual.
SunAlert: 46500
PROBLEM DESCRIPTION:
Sun StorEdge 6900 Series storage used in a clustered environment may
encounter a race condition which can cause the storage unit to not
handle SCSI-2 disk reservations properly. When this happens, the Sun
StorEdge 6900 Series storage becomes inaccessible to the cluster
nodes.
This issue affects any StorEdge 69x0 array with Service Processor Image
version 2.1.1 or below which is used in a clustered configuration. Due
to the conditions needed for this issue to occur, the probability of
failure in the field is less than 1% of the installed base.
To detemine the revision of Service Processor Image for a StorEdge 69x0
array, log on to the Service Processor and type:
cat /etc/motd
Affected 69x0 arrays will exhibit errors similar to the following found
in the /var/adm/messages file of one of the cluster nodes:
Jun 18 15:12:41 phys-bazaar-1 fp: [ID 517869 kern.warning] WARNING:
fp(2): GPN_ID for D_ID=108000 failed
Jun 18 15:12:41 phys-bazaar-1 fp: [ID 517869 kern.warning] WARNING:
fp(2): N_x Port with D_ID=108000, PWWN=2b00006022004216 disappeared
from fabric
Jun 18 15:12:51 phys-bazaar-1 fp: [ID 517869 kern.warning] WARNING:
fp(2): N_x Port with D_ID=108000, PWWN=2b00006022004216 reappeared
in fabric
Jun 18 15:12:51 phys-bazaar-1 scsi: [ID 243001 kern.info]
/ssm@0,0/pci@19,700000/pci@2/SUNW,qlc@4/fp@0,0 (fcp2):
Jun 18 15:12:51 phys-bazaar-1 Dynamically discovered 18 LUNs for
D_ID=108000
Jun 18 15:13:02 phys-bazaar-1 fp: [ID 517869 kern.warning] WARNING:
fp(0): GPN_ID for D_ID=104000 failed
Jun 18 15:13:02 phys-bazaar-1 fp: [ID 517869 kern.warning] WARNING:
fp(0): N_x
Jun 18 15:13:17 phys-bazaar-1 fp: [ID 517869 kern.warning] WARNING:
fp(0): N_x Port with D_ID=104000, PWWN=2b00006022004211 reappeared
in fabric
Jun 18 15:13:18 phys-bazaar-1 scsi: [ID 243001 kern.info]
/ssm@0,0/pci@18,700000/pci@2/SUNW,qlc@4/fp@0,0 (fcp0):
Jun 18 15:13:18 phys-bazaar-1 Dynamically discovered 18 LUNs for
D_ID=104000
Jun 18 15:15:50 phys-bazaar-1 scsi_vhci: [ID 734749 kern.warning] WARNING:
vhci_scsi_reset 0x1
Jun 18 15:15:50 phys-bazaar-1 scsi: [ID 243001 kern.warning] WARNING:
/ssm@0,0/pci@18,700000/pci@2/SUNW,qlc@4/fp@0,0 (fcp0):
Jun 18 15:15:50 phys-bazaar-1 FCP: WWN 0x2b00006022004211 reset
successfully
Jun 18 15:15:50 phys-bazaar-1 fp: [ID 517869 kern.warning] WARNING:
fp(2): GPN_ID for D_ID=108000 failed
Jun 18 15:15:50 phys-bazaar-1 fp: [ID 517869 kern.warning] WARNING:
fp(2): N_x Port with D_ID=108000, PWWN=2b00006022004216 disappeared
from fabric
This problem occurs due to the existence of a race condition which can
cause the storage unit to not properly handle SCSI-2 disk
reservations. If two nodes in a cluster simultaneously send SCSI-2
reservation commands to the same 69x0 array, the Storage Virtualization
Engine (SVE) is unable to handle the requests due to a firmware issue.
This issue is addressed in Service Processor Image version 2.1.2.
Arrays with Image versions less than 2.1.1 must first upgrade to 2.1.1
before upgrading to 2.1.2.
IMPLEMENTATION:
---
| | MANDATORY (Fully Proactive)
---
---
| X | CONTROLLED PROACTIVE (per Sun Geo Plan)
---
---
| | REACTIVE (As Required)
---
CORRECTIVE ACTION:
The following recommendation is provided as a guideline for authorized
Sun Services Field Representatives who may encounter the above
mentioned problem.
1) Upgrade the 69x0 Service Processor Image from version 2.1.1 to 2.1.2
using the "2.1.1 to 2.1.2 Storage Service Processor Image Upgrade
1.0"
which can be found at http://edist.central (go to the "Network Storage /
Software:All" Edist Repository). This Image Upgrade also includes and
automatically installs the following three patches:
112276.porsche
112580
113247
2) Arrays with Image version less than 2.1.1 must first upgrade to 2.1.1
using "2.0.3 to 2.1.1 Storage Service Processor Image Upgrade" found
at http://edist.central/ .
Installation Instructions for 2.1.1 to 2.1.2 Upgrade
----------------------------------------------------
NOTE: Prior to performing a firmware upgrade you must backup the SVE
configuration.
To access the patch, go to:
http://edist.central/
2.1.1 to 2.1.2 upgrade:
-----------------------
Read this entire document before attempting to perform this upgrade.
For information on what has changed in the 2.1.2 build, refer to:
/tmp/jass-0.3/Files/export/README.txt. After the upgrade is performed the
README.txt file will reside at: /export/README.txt
It is necessary for the Service Processor to complete a reboot for
the functionality of 2.1.2 to take effect. If the upgrade process is
terminated prior to the reboot it is the responsibility if the user to
initiate a reboot process.
UPGRADE PREREQUISITES
---------------------
This upgrade assumes the Service Processor has the 2.1.1 image revision
installed. To verify this, log on to the Service Processor and type:
cat /etc/motd. The response should indicate that the revision of the
Service Processor image is 2.1.1. If the revision reflects anything
other than 2.1.1, DO NOT INSTALL THIS PATCH.
In order to perform this upgrade, it is necessary to be connected via
the console of the Service Processor and logged in as the root user. Be
sure that the user cmdadm is not logged into the Service Processor
being upgraded as this will cause the upgrade to fail.
Finally, be sure to read this file completely before performing this
upgrade. The file contains important, up-to-date information related
to performing a successful Service Processor upgrade.
NOTES
-----
You must be logged onto the Service Processor via the console port and
be logged in as the root user to perform this installation. You must
also verify that no one is logged in as the cmdadm user as this will
cause the upgrade to be unsuccessful.
A log file is created in the /var/tmp directory. The name of the log
file is: 2.1.2-upgrade.log
The upgrade process makes a back copy of /etc/shadow, /etc/passwd,
/etc/inet/hosts, /etc/ethers, /etc/nsswitch.conf, and /etc/groups. The
backup for each of these files is contained in the original directory
and is named: <filename>.2.1.2.upgrade.bak where <filename> is
the
original name of the file, including extension if applicable. These
backup files may be used to assist in restoring site specific
configuration information that may otherwise be lost in the upgrade
process.
This script requires user interaction. Do not start and forget this
script as it will not complete without responses from the user.
Installation Instructions
-------------------------
During the upgrade process it will be necessary to answer a number
of questions. Follow the instructions below:
(HINT: For places which require input, look for the word USER)
Place the file upgrade.tar into the /tmp directory on the service processor.
To run the upgrade process,
USER: type the following command: ./upgrade.sh
This script requires user interaction. Do not start and forget this script
as it will not complete without responses from the user.
Press Enter or Return to continue.
USER: Press the Enter or Return key
Checking current version...
Patch process started
Backing up files...
Removing superseded files...
Removing superseded directories...
Installing VE Patch...
x 113247, 0 bytes, 0 tape blocks
x 113247/README.113247-01, 1535 bytes, 3 tape blocks
x 113247/.diPatch, 76 bytes, 1 tape blocks
x 113247/patchinfo, 159 bytes, 1 tape blocks
...
Installing StorADE Patch...
x 112580, 0 bytes, 0 tape blocks
x 112580/README.112580-03, 9306 bytes, 19 tape blocks
x 112580/.diPatch, 76 bytes, 1 tape blocks
x 112580/patchinfo, 133 bytes, 1 tape blocks
...
Expanding 112276...
x 112276, 0 bytes, 0 tape blocks
x 112276/disk, 0 bytes, 0 tape blocks
x 112276/disk/S318203_D94A, 439424 bytes, 859 tape blocks
x 112276/disk/S318304_A726, 570496 bytes, 1115 tape blocks
...
Expanding 112276.porsche...
x 112276.porsche/README.112276-04, 27299 bytes, 54 tape blocks
x 112276.porsche/files.tar, 851456 bytes, 1663 tape blocks
x 112276.porsche/nb211.bin, 2579724 bytes, 5039 tape blocks
x 112276.porsche/patchtoc, 20 bytes, 1 tape blocks
x 112276.porsche/t3.sh, 6074 bytes, 12 tape blocks
Running ras_install ...
**** Installing the Package and Crons ****
*** Master Install ***
This script will now add the inet service to the inetd config file.
When this scripts ends, go to the IP Name/Address of the HOST configured
as MASTER, port 7654, with a browser to complete the configuration.
/etc/services is now updated.
/etc/inetd.conf is now updated.
** cron installed.
. Resetting the inetd services to see the new rashttp service.
. Testing access to the webserver, (this will timeout after 30 secs) ...
1/6 attempting to contact agent service...
*** Contacted agent service (830bd0e7).
SUNWstade installed properly!
To complete the configuration, point your browser to:
http://<hostname>:7654. Use the browser only after package
has been installed on all Master and Slave hosts.
What is the SE Series model that this upgrade is being done on?
Supported models are :
3910 ( 2 8-Port Switches NO Virtualization Engines )
3960 ( 2 16-Port Switches NO Virtualization Engines )
6910 ( 4 8-Port Switches With Virtualization Engines )
6960 ( 4 16-Port Switches With Virtualization Engines )
The SE Series model is :
USER: enter the correct model number from the menu
Running config_solution on this SE Series <model number>
If this step is unsuccessful, the following prompt will appear:
It appears that config_solution did not complete successfully. This
may be due to the incorrect model number being entered.
Config_solution must complete without error to complete the 2.1.2
upgrade.
Press the Enter or Return key to retry config_solution.
USER: Press the Enter or Return key and re-enter the correct model number.
When this step is completed successfully, the following prompt will appear:
Restoring files...
Performing final package check...
The Service Processor must be rebooted for the changes made in this
upgrade to take effect. Rebooting the Service Processor will have no
effect on the customer's data access.
Press 'y' to reboot now or 'n' to reboot later.
USER: Select either 'y' to reboot now or n to reboot later
If any key other than 'y' or n is selected, the following will appear:
You must select either 'y' to reboot now or 'n' to reboot later.
Press the Enter or Return key to retry.
USER: press Enter or Return to answer the reboot question again.
If 'n' is selected, the following will appear:
You have elected not to reboot the Service Processor at this time.
The changes made in this upgrade will not take effect until the
Service Processor is rebooted.
If 'y' is selected, the following will appear:
The Service Processor will reboot now.
The Service Processor will now reboot itself. The reboot completes the
2.1.2 Service Processor upgrade process.
Once the Service Processor upgrade process is completed, it is
necessary to upgrade the T3+ and VE firmware. This is a manual process
explained in the Sun StorEdge(tm) 3900 and 6900 Series 1.1 Reference
and Service Manual. Please refer to the appropriate section of this
manual for details on how to perform the firmware upgrades on these
devices.
The firmware for the T3+ is located on the Service Processor in:
/export/patches/112276-04.porsche
The file is named: nb211.bin
The nb211.bin file is also copied to the /tftpboot directory.
The firmware for the VE is located on the Service Processor in:
/opt/svengine/firmware The file is named: fw8_18.ima
COMMENTS:
None
============================================================================
Implementation Footnote:
i) In case of MANDATORY FINs, Enterprise Services will attempt to
contact all affected customers to recommend implementation of
the FIN.
ii) For CONTROLLED PROACTIVE FINs, Enterprise Services mission critical
support teams will recommend implementation of the FIN (to their
respective accounts), at the convenience of the customer.
iii) For REACTIVE FINs, Enterprise Services will implement the FIN as the
need arises.
----------------------------------------------------------------------------
All released FINs and FCOs can be accessed using your favorite network
browser as follows:
SunWeb Access:
--------------
* Access the top level URL of http://sdpsweb.ebay/FIN_FCO/
* From there, select the appropriate link to query or browse the FIN and
FCO Homepage collections.
SunSolve Online Access:
-----------------------
* Access the SunSolve Online URL at http://sunsolve.Corp/
* From there, select the appropriate link to browse the FIN or FCO index.
Internet Access:
----------------
* Access the top level URL of https://infoserver.Sun.COM
--------------------------------------------------------------------------
General:
--------
* Send questions or comments to finfco-manager@sdpsweb.EBay
--------------------------------------------------------------------------
Copyright (c) 1997-2003 Sun Microsystems, Inc.