Document fins/I0665-1
FIN #: I0665-1
SYNOPSIS: PCI Bus configurations on Ultra 80 system based boards
DATE: Apr/18/01
KEYWORDS: PCI Bus configurations on Ultra 80 system based boards
---------------------------------------------------------------------
- Sun Proprietary/Confidential: Internal Use Only -
---------------------------------------------------------------------
FIELD INFORMATION NOTICE
(For Authorized Distribution by SunService)
SYNOPSIS: Ultra 80 system board based systems may panic at boot time
with certain PCI Bus configurations.
TOP FIN/FCO REPORT: No
PRODUCT_REFERENCE: PCI card configuration
PRODUCT CATEGORY: Server / Documentation
PRODUCTS AFFECTED:
Mkt_ID Platform Model Description Serial Number
------ -------- ----- ----------- -------------
Systems Affected
----------------
- A27 ALL Ultra 80 -
- A33 ALL Enterprise 420R -
- N14 ALL Netra t 1405 -
- N15 ALL Netra t 1400 -
X-Options Affected
------------------
X1034 - - Quad Fast Ethernet card -
X1141 - - GBE/P Ethernet card -
PART NUMBERS AFFECTED:
Part Number Description Model
----------- ----------- -----
501-4366 ASSY PCI CARD TESTED -
501-5406 ASSY PCI CARD W/LUCENT TESTED -
501-4373 ASSY GEM-PCI CARD MMF TESTED -
REFERENCES:
BugID: 4424939 - U80 panics during Psycho reset.
4393940 - Netra T 1405/U80 panics intermittently.
4342026 - Psycho intermittently violates PCI spec for signaling
64-bit bus @ end of reset.
4339479 - U80 getting panic on PCI bus 3 error(s).
ESC: 528472
DOC: 805-5938: GBE/P Product Note.
MANUAL: 805-2785: GBE/P Installation and User Guide.
805-1797: QFE Installation and User Guide.
806-0576: Netra 140x Service and Reference Manual.
805-6618: Ultra 80 Service Manual.
806-1080: E420R Service Manual.
PROBLEM DESCRIPTION:
Incorrect slot positioning of the GBE/P related to the QFE board can
cause panics on Ultra 80 systems. Mixture of a QFE and a GBE/P on the
same bus, would cause the system may suffer a SERR error and panic the
system.
Ultra 80, E420R, and Netra t 1400/1405 systems may panic at boot time
and drop to the "ok" prompt with certain PCI card configurations. The
problem occurs when a QFE card and another 64-bit capable card such as
GBE/P Ethernet card are installed in adjacent 64-bit slots in the 33MHz
PCI bus of a U80 or related platform.
Typical failure:
----------------
Solaris 5.7 Version Generic_106541-04,
panic[cpu1]/thread=2a100003d60:
pci0: PCI SERRpci0: PBM detected parity error.
Fast Data Access MMU Miss
ok
Failing Configuration:
----------------------
----------------------------------------------------
| Board location | slot | bus | Mhz | mode | Card |
|----------------------------------------------------|
| J1801 | 1 | A | 66 | 64bit | GBE/P |
| J2001 | 2 | B | 33 | 64bit | QFE |
| J1901 | 3 | B | 33 | 64bit | GBE/P |
| J4701 | 4 | B | 33 | 32bit | - |
----------------------------------------------------
The probability of the problem occurring is very low. However if it
does occur, all service will be lost until the machine is rebooted.
This failure has been reproduced on Solaris 2.6 and Solaris 7. Due to
improvements in Solaris 8, the fault is detected and the operating system
panics but does not drop to the ok prompt. It reboots successfully.
The system halts at the ok prompt since the OS did not successfully
panic, passing on handling of a fast data access MMU miss to OBP when
at TL=0x2. OBP can only handle MMU misses from TL=0. MMU misses from
TL=0x2 should not have been triggered by Solaris.
The problem occurs due to a RESET timing bug in the Psycho chip.
During the RESET cycle, the PCI bus is sized, meaning any card
installed in that bus at RESET time will configure itself as either
being in a 64-bit slot (if REQ64 is asserted) or in a 32-bit slot (if
REQ64 is not asserted). Psycho MAY de-assert REQ64 too early due to an
internal timing problem; when this occurs, the PCI cards may
incorrectly configure themselves as being installed in a 32-bit PCI
slot instead of a 64-bit slot. In effect, the second PCI card (i.e.,
GBE) gets configured as a 32-bit card in this case.
When the GBE card sees a 64-bit PCI transfer on the bus,
corresponding to the first QFE DMA data transfer after the driver is
loaded, GBE drives the upper 32 Address/Data lines to 0. This can
happen even before GBE has been configured and the GBE driver gets
loaded. This contends with the QFE data being sent to Psycho, and
causes an apparent parity error. The parity error is caused by bus
contention from the second PCI card (GBE) and not from QFE itself (QFE
generated parity is, in fact, correct).
It is not possible to fix this problem without a respin to this ASIC
and will not be done. The only way to avoid this issue is through
correct PCI card placement.
IMPLEMENTATION:
---
| | MANDATORY (Fully Pro-Active)
---
---
| | CONTROLLED PRO-ACTIVE (per Sun Geo Plan)
---
---
| X | REACTIVE (As Required)
---
CORRECTIVE ACTION:
An Authorized Enterprise Field Service Representative may avoid the
above mentioned problems by following the recommendations as shown
below.
When installing this mixture of PCI QFE and GBE/P cards particular
attention must be made to slot assignment. The slot number is the
physical position as printed on the printed circuit board and not the
Unix identification.
The following guideline should be adhered to:
1. Install QFE card in slot 1 only.
2. Install GBE/P cards in either slots 2 and 3. Do not install
GBE/P card in slot 1.
3. Any 64-bit card can be used in a 32-bit slot (slot 4) provided
there is no mechanical interference from adjacent components
next to the slot.
The preferred slot configuration is shown in the following table:
----------------------------------------------------
| Board location | slot | bus | Mhz | mode | Card |
|====================================================|
| J1801 | 1 | A | 66 | 64bit | QFE |
| J2001 | 2 | B | 33 | 64bit | GBE/P |
| J1901 | 3 | B | 33 | 64bit | GBE/P |
| J4701 | 4 | B | 33 | 32bit | - |
----------------------------------------------------
In addition to the physical PCI card positioning, system performance
tuning may be undertaken to obtain maximum throughput for this
configuration. Typical setting are shown below but any changes are
dependent on the customers requirements. Refer to chapter 3 of the
GBE/P Installation and User Guide (805-2785), and chapter 3 of the QFE
Installation and User Guide (805-1797) for detailed information about
setting device driver parameters.
ndd -set /dev/ge instance 0
ndd -set /dev/ge adv_pauseTX 1
ndd -set /dev/ge adv_1000autoneg_cap 1
ndd -set /dev/ge instance 2
ndd -set /dev/ge adv_pauseTX 1
ndd -set /dev/ge adv_1000autoneg_cap 1
sq_max_size=500 (/etc/system),
COMMENTS:
-----------------------------------------------------------------------------
Implementation Footnote:
i) In case of MANDATORY FINs, Enterprise Services will attempt to
contact all affected customers to recommend implementation of
the FIN.
ii) For CONTROLLED PROACTIVE FINs, Enterprise Services mission critical
support teams will recommend implementation of the FIN (to their
respective accounts), at the convenience of the customer.
iii) For REACTIVE FINs, Enterprise Services will implement the FIN as the
need arises.
----------------------------------------------------------------------------
All released FINs and FCOs can be accessed using your favorite network
browser as follows:
SunWeb Access:
--------------
* Access the top level URL of http://sdpsweb.ebay/FIN_FCO/
* From there, select the appropriate link to query or browse the FIN and
FCO Homepage collections.
SunSolve Online Access:
-----------------------
* Access the SunSolve Online URL at http://sunsolve.Corp/
* From there, select the appropriate link to browse the FIN or FCO index.
Supporting Documents:
---------------------
* Supporting documents for FIN/FCOs can be found on Edist. Edist can be
accessed internally at the following URL: http://edist.corp/.
* From there, follow the hyperlink path of "Enterprise Services Documenta-
tion" and click on "FIN & FCO attachments", then choose the
appropriate
folder, FIN or FCO. This will display supporting directories/files for
FINs or FCOs.
Internet Access:
----------------
* Access the top level URL of https://infoserver.Sun.COM
--------------------------------------------------------------------------
General:
--------
* Send questions or comments to finfco-manager@Sun.COM
---------------------------------------------------------------------------
Copyright (c) 1997-2003 Sun Microsystems, Inc.