Document intinfodoc/49667

InfoDoc ID		Synopsis		Date
49667		Sun Fire[TM] 12K/15K: Dynamic Reconfiguration Considerations		21 Jan 2003

Status

Issued

Description

- Title:

Sun Fire[TM]12K/15K Dynamic Reconfiguration Considerations

- Description:

Dynamic Reconfiguration (DR) is a powerful and useful tool for allocating/
deallocating resources to/from a domain without interruption. Also, it is
useful when servicing problem components in the system. However, some
configurations are better suited to maximize the effectiveness of DR than
others. Because of this, to fully appreciate the impact of a DR operation
on a domain, detailed familiarity with the domain's configuration and
application workload is required.

The sections below are intended to delineate important points for various
DR operations which should be known and understood prior to performing a DR
operation. This document is not intended as a tutorial on using DR, command
syntax, etc. It may be helpful as a starting point for development of site
specific DR practices and/or procedures.

- Body Section:
TERMS

When describing DR operations, this document uses the following terms:

Attach Adding an SB/IO board to a running domain.
Detach Removing an SB/IO board from a running domain.
Hot-plug Adding/removing a PCI adapter from a running domain.
Slot 0 DR DR involving a board that physically resides in Slot 0
of an expander.
Slot 1 DR DR involving a board that physically resides in Slot 1
of an expander.

ASSUMPTIONS

It is a basic assumption that both the System Controller (SC) and Domain
are running the minimum Solaris[TM]/SMS versions and patches required to support
DR. At the time of this writing, the minimum requirements for DR are:

* Minimum patch levels required for SMS and Solaris are detailed
in "Sun Fire 15K/12K DR Installation Guide and Release Notes"

The installation guide lists the minimum set of patches required. As a
best practice, having the most recent patch levels is preferred. When
appropriate, this document lists minimum desired patch revisions based
on experienced customer problems. However, it is not all-encompassing
and does not preclude the need for proper patch management for a 12K/15K
system.

Furthermore, it is assumed that the 12K/15K hardware has undergone
FCOs A0192 and A0193. These FCOs upgrade the AXQ and Schizo ASICs to
acceptable levels. Slot 1 DR requires AXQ 6.1 and Schizo 2.3 at a minimum.

Additionally, it is assumed for any attach operations that the hardware
being introduced into the system is fault free. POST may detect faults
during the attach process which must be corrected by Sun personnel. This
document does not discuss diagnosis of failures and/or replacement
procedures.

ATTACH OPERATIONS

Prior to executing an attach operation for a component, review the notes
and points listed below to minimize potential problems with the operation.

In general, when executing the attach, the POST diagnostic level can be
specified by:

a. On the cfgadm/rcfgadm command line, using the '-o platform=diag=<level>'
option to specify an alternative level.
b. In the domain/platform .postrc file, using the 'dash_H_level <level>'
directive to specify an alternative level.

The 'dash_H_level' option is preferred as it applies to all DR interfaces
(addboard/cfgadm/moveboard/rcfgadm). The various POST levels are described at
http://product-support/starcat/tools/post_stages.html.

Slot 0 CPU/Memory Board
-----------------------

1. Ensure the CPU/Memory board is flashupdated to LPOST version
5.13.4 or higher.

Lesser versions of LPOST are exposed to bug 4728549, which may
cause the target domain to hang. Refer to FIN I0895 for details.
As a best practice, the board being introduced should be flashed
to the same version used in the rest of the target domain.

Slot 1 MaxCPU Board
-------------------

1. Ensure the MaxCPU board is flashupdated to LPOST version 5.13.4
or higher.

Slot 1 HPCI/HPCI+ Board
-----------------------
1. Can the application(s) in the target domain tolerate 1 less
processor?

IO boards do not have processors or physical memory. POST requires
a processor and a small amount of memory to execute its tests.
Prior to POST running, Solaris "loans" POST a processor and memory
for testing. During POST execution, this processor and memory is not
available to the domain.

2. Are there bound processes to all Slot 0 CPUs on the domain?

The logic Solaris uses to select a "loaner" processor for IO attach
operations is to step through all the Slot 0 CPUs in the system.
Slot 1 CPUs (i.e., MaxCPU) are ignored. If a CPU has processes bound
to it, that CPU cannot be offlined and attach logic considers it
unavailable. If all CPUs in the domain have bound processes, selection
of a "loaner" processor fails.

DR will not automatically rebind processes to other CPUs. This must be
done by an administrator. Whether to unbind/rebind the processes, and
to which of the remaining CPUs in the domain, is a decision that must
be made by someone knowledgeable of the application(s) on the domain.
To locate and rebind bound processes, use the 'pbind' command.

3. Are the PCI adapters in the board qualified for hot-plug?

For Sun adapters, a list of adapters qualified is maintained by
15K marketing in "Sun Fire 12K-15K Server I/O Support Matrix". The
'DR' column indicates hot-plug compliance and support.

4. Are the PCI adapters in the board known, good adapters?

It must be noted that POST does no testing, stressing or verification
of the PCI adapters present in the IO board. If the adapters have
not been stress tested by other means (SunVTS, etc.), a card with a
fault may be introduced into the system.

5. Does the IO board contain 3rd party adapters?

Sun does not qualify all vendor PCI adapters. Refer to the THIRD PARTY
STATEMENT below.

6. Followup configuration

After I/O devices are attached into a domain, followup configuration
is likely required (network plumbing, file system creation, etc.).

DETACH OPERATIONS

Prior to executing a detach operation for a component, review the notes
and points listed below to minimize potential problems with the operation.

Slot 0 CPU/Memory Board
-----------------------

1. Does the board being detached have processes bound to its CPUs?

CPUs with bound processes cannot be detached. DR will not automatically
rebind processes to other CPUs. This must be done by an administrator.
Whether to unbind/rebind the processes, and to which of the remaining
CPUs in the domain, is a decision that must be made by someone knowledgeable
of the application(s) on the domain. To locate and rebind bound processes,
use the 'pbind' command.

2. Can the application(s) on the domain tolerate fewer processors and
less memory?

A detach of a CPU/Memory will reduce the domain in both memory capacity
and processing power. This may have an impact on domain application(s).
A prior workload baseline may help determine if the application(s)
performance will suffer with less resources.

To avoid the tradeoff, if available, a different CPU/Memory board could
be attached to the domain prior to detaching a board. This would also
provide CPUs to shift bound processes to.

3. Does the board being detached contain Intimate Shared Memory (ISM) pages?

ISM is extensively used in database applications (Oracle[R], Informix, Sybase,
etc.) ISM pages cannot be paged out, and therefore must be relocated to
other physical memory as part of a detach process. ISM pages are reported
by 'cfgadm' as permanent memory. See SRDB 48157 to determine where permanent
memory is located.

During the detach process, database performance may be impacted as ISM
pages are relocated. The detach process itself may also be lengthy.
Refer to bug 4632219 for details.

4. Does the board being detached contain kernel memory?

The memory used for the kernel cannot be paged out. Therefore, when
detaching kernel memory, the OS must be temporarily suspended while
kernel memory is relocated to another CPU/Memory board.
Kernel memory is reported by 'cfgadm' as permanent memory. Refer to
SRDB 48157 to determine where permanent memory is located and if that
memory is kernel memory. If kernel memory is present, there are some
additional items to consider prior to executing the detach operation:

o Is a sufficiently equipped "target" board for permanent memory
available?

Permanent memory must be relocated to another board as a single,
contiguous slice of physical memory. This requires another board
in the system that can receive the permanent memory. Refer to SRDB
48157 for how DR selects a target board for kernel relocation.

o Are there real time (RT) processes on the system?

The most common real time process is NTP (Network Time Protocol)
which is real time by default. But, other application(s) may be
scheduled as real time.

Since the OS must be temporarily suspended during a permanent
memory detach, running processes will no longer be real time.
In order to perform the detach, all real time processes must be
changed to a non-real time scheduling class. See priocntl(1).
Whether or not a process can be temporarily changed to a non-real
time scheduling class must be determined by an administrator with
knowledge of the application(s) requirements. Time sensitive
applications may not tolerate a suspension, in which case a detach
of permanent memory is not possible.

o Is MPxIO active in the system?

Ensure the appropriate kernel level is applied to the system to avoid
bug 4649851. This bug is addressed in Solaris 8 KU-15 (and higher)
and Solaris 9 KU-01 (and higher).

o Are 3rd party adapters and/or driver software present in the domain?

Sun does not qualify all vendor PCI adapters and/or 3rd party driver
software with DR. Refer to the THIRD PARTY STATEMENT below.

o Is the domain part of a Sun[TM] Cluster?

Sun Cluster does not support detach operations on permanent memory
and the system will prevent such operations.

Slot 1 MaxCPU Board
-------------------

1. Does the board being detached have processes bound to its CPUs?

2. Can the application(s) on the domain tolerate fewer processors?

A detach of a MaxCPU will reduce the domain's processing power. This
may have an impact on domain application(s). A prior workload baseline
may help determine if the application(s) performance will suffer with
less resources.

To avoid the tradeoff, if available, a different MaxCPU board could be
attached to the domain prior to detaching a board. This would also
provide CPUs to shift bound processes to.

Slot 1 HPCI/HPCI+ Board
-----------------------

1. Are the PCI adapters in the IO board qualified for DR?

For Sun adapters, a list of adapters qualified is maintained by
15K marketing in "Sun Fire 12K-15K Server I/O Support Matrix". The
'DR' column indicates hot-plug compliance and support.

2. For each PCI adapter in the IO board, does an alternate path to
its storage/network exist?

Detaching an IO board obviously removes pathways to storage devices
and networks. An alternate path to the storage/network must be
maintained to support running application(s). This is typically
accomplished by multi-pathing software (MPxIO, IPMP, etc.).

It is also required that the multi-pathing software be DR safe.
For MPxIO, ensure the appropriate kernel level is applied to the
system to avoid bug 4649851. This bug is addressed in Solaris 8
KU-15 (and higher) and Solaris 9 KU-01 (and higher).

Sun does not qualify all vendor multi-pathing software. Refer to
the THIRD PARTY STATEMENT below.

3. Can the application tolerate single pathways to storage/networks?

For I/O intensive application(s), ensure the bandwidth provided by
a single pathway to storage/networks is sufficient for the running
application(s).

4. Are any PCI adapters in the IO board 3rd party?

Sun does not qualify all vendor PCI adapters. Refer to the THIRD PARTY
STATEMENT below.

5. Is the domain part of a Sun Cluster?

Sun Cluster has additional restrictions for DR operations on global
devices, quorum devices, private cluster interconnects, and public
network interfaces. Refer to the Sun Cluster 3.0 System Administration
Guide and Release Notes.

HOT-PLUG OPERATIONS

Prior to executing a hot-plug operation for an adapter, review the notes
and points listed below to minimize potential problems with the operation.

Adding An Adapter
-----------------

1. Is the PCI adapter qualified for hot-plug?

For Sun adapters, a list of adapters qualified is maintained by
15K marketing in "Sun Fire 12K-15K Server I/O Support Matrix". The
'DR' column indicates hot-plug compliance and support.

2. Is the PCI adapter a known, good adapter?

It must be noted that the hot-plug procedure does no testing,
stressing or verification of the PCI adapter. If the adapter has
not been stress tested by other means (SunVTS, etc.) a fault may
be introduced into the system.

3. Is the adapter 3rd party?

Sun does not qualify all vendor PCI adapters. Refer to the THIRD PARTY
STATEMENT below.

4. Followup configuration

After an adapter is hot-plugged into a domain, followup configuration
is likely required (network plumbing, file system creation, etc.). Also
be aware of bug 4721698. After an adapter is added, the GDCD is not
updated so reboots (hpost -Q) do not recognize the newly added adapter.

Removing An Adapter
-------------------

1. Is the PCI adapter qualified for hot-plug?

For Sun adapters, a list of adapters qualified is maintained by
15K marketing in "Sun Fire 12K-15K Server I/O Support Matrix". The
'DR' column indicates hot-plug compliance and support.

2. Does an alternate path to this adapter's storage/network exist?

Detaching an adapter obviously removes a pathway to storage devices
and/or networks. An alternate path to the storage/network must be
maintained to support running application(s). This is typically
accomplished by multi-pathing software (MPxIO, IPMP, etc.).

Sun does not qualify all vendor multi-pathing software. Refer to
the THIRD PARTY STATEMENT below.

3. Can the application tolerate a single pathway to the storage/network?

For I/O intensive application(s), ensure the bandwidth provided by
a single pathway to the storage/networks is sufficient for the
running application(s).

4. Is the PCI adapter 3rd party?

Sun does not qualify all vendor PCI adapters. Refer to the THIRD PARTY
STATEMENT below.

- Background Information:

THIRD PARTY STATEMENT

If a 3rd party adapter is being attached, detached, or hot-plugged, it is
the responsibility of the owner of the system to ensure the adapter, as
well as any associated device drivers and firmware are DR safe. This also
applies for 3rd party software drivers (i.e. multipathing software) if
a detach requires an OS suspension. Consult with the vendor of the adapter
to ensure their driver stack and adapter firmware level are DR compliant.

3rd party components known to have issues:
- Emulex driver stack 5.00 (or lower). See Sun BugID 4725681.

ADDITIONAL KNOWN ISSUES

4785231 - If a psradm command is issued before running cfgadm, it
may hang all cfgadm processes, effectively preventing any
DR operations.

Workaround: Run cfgadm before psradm.

        FEEDBACK

        This is a living document. As features/requirements change, all attempts to
        keep this document current will be made. If while using its content, an
        oversight or discrepancy is noted, contact the submitter.

- References

        Sun Fire 15K/12K DR Installation Guide and Release Notes (816-5080)
        Sun Fire 15K/12K DR User Guide (816-5075)
        Sun Cluster 3.0 12/01 System Administration Guide (816-2026)
        SRDB 48157
        http://systems.corp/products/servers/docs/SF_Highend_IO_Support_Matrices_latest.pdf

- Keywords

        15K, 12K, SF15K, SF12K, Sun Fire 15K, Enterprise, Server, Sun Fire 12K,
        Dynamic Reconfiguration, DR, best practices

SUBMITTER: Scott Davenport BUG REPORT ID: 4728549, 4632219, 4649851, 4721698, 4649851, 4725681., 4785231 APPLIES TO: Hardware/Sun Fire /15000, Hardware/Sun Fire /12000 ATTACHMENTS: