Solaris Network Multipathing
The following components implement Solaris network multipathing:
The in.mpathd daemon detects failures and implements various policies for failover and failback. After in.mpathd detects a failure or repair, in.mpathd sends an ioctl to do the failover or failback. IP, which implements the ioctl, does the network access failover transparently and automatically.
Caution - Do not use Alternate Pathing while using IP Network Multipathing
on the same set of NICs. Likewise, you should not use IP Network Multipathing
while you are using Alternate Pathing. You can use Alternate Pathing and IP
Network Multipathing at the same time on different sets of NICs.
Detecting Physical Interface Failures
The in.mpathd daemon can detect interface failure and repair by two methods. In the first method, the daemon sends and receives ICMP echo probes through the interface. In the second method, the daemon monitors the RUNNING flag on the interface. The link state on some models of network interface cards is reflected by the RUNNING flag. Consequently, when the link fails, the failure is detected much sooner. An interface is considered to have failed if either of the previous two methods indicates failure. An interface is considered repaired only if both methods indicate that the interface is repaired.
The in.mpathd daemon sends ICMP echo probes to the targets that are connected to the link on all the interfaces that belong to a group to detect failures and repair. After you add an interface to a multipathing group and assign a test address, the daemon sends probes to detect failures on all the interfaces of the multipathing group. "How to Configure a Multipathing Interface Group With Two Interfaces" describes the steps you perform to configure test address and groups.
Because in.mpathd determines which targets to probe dynamically, you cannot configure the targets. Routers that are connected to the link are chosen as targets for probing. If no routers exist on the link, arbitrary hosts on the link are chosen. A multicast packet that is sent to the "all hosts" multicast address (224.0.0.1 in IPv4 and ff02::1 in IPv6) determines the arbitrary hosts. The first few hosts that respond to the echo packets are chosen as targets for probing. If in.mpathd cannot find routers or hosts that responded to ICMP echo packets, in.mpathd cannot detect failures.
To ensure that each NIC in the group functions properly, in.mpathd probes all the targets separately through all the interfaces in the multipathing group. If no replies are made to five consecutive probes, in.mpathd considers the interface to have failed. The probing rate depends on the failure detection time (FDT). The default value for failure detection time is 10 seconds. The in.mpathd(1M) man page describes how to change the failure detection time. For a failure detection time of 10 seconds, the probing rate is approximately one probe every two seconds.
The failure detection time only applies to the ICMP echo probe method of detecting failures. If link failure results in the clearing of the RUNNING flag for an interface, the in.mpathd daemon responds immediately to the change in the flag status.
After a failure is detected, failover of all network access occurs from the failed interface to another functional interface in the group. If you have configured a standby interface, in.mpathd chooses the standby interface for failover of IP addresses, broadcasts, and multicast memberships. If you have not configured a standby interface, in.mpathd chooses the interface with the least number of IP addresses.
Physical interfaces in the same group that are not present at system boot represent a special instance of failure detection. The startup script /etc/init.d/network detects these types of failure. This type of failure displays error messages similar to the following:
moving addresses from failed IPv4 interfaces: hme0 (moved to hme1) moving addresses from failed IPv6 interfaces: hme0 (moved to hme1) |
Note - In this special instance of failure detection, only static IP addresses that are specified in host name files are moved to a different physical interface within the same multipathing group.
This type of failure can be automatically repaired by a failback. The RCM DR Post-attach feature for IP Network Multipathing automates the DR attachment of a NIC. When a NIC is DR attached, the interface is plumbed and configured. If the interface was removed prior to a reboot, the IP multipathing Reboot-safe feature recovers the IP address. The IP address is transferred to the replaced NIC. The replaced NIC is added to the original IP multipathing interface group. See "How to Recover a Physical Interface That Was Not Present at System Boot".
Detecting Physical Interface Repairs
The in.mpathd daemon considers an interface repaired if the daemon receives responses to 10 consecutive probe packets, and the RUNNING flag is set on the interface.
When an interface fails, all addresses are moved to another functional interface in the group. Because in.mpathd needs an address for probing so that it can detect repairs, you must configure a test IP address that cannot move during the failover. Moreover, you should not allow a normal application to use this test address, because the failover of network access cannot occur for these addresses. "How to Configure a Multipathing Interface Group With Two Interfaces" describes the steps that you perform. If in.mpathd detects a repair, failback of all network access occurs to the repaired interface.
As noted in "Detecting Physical Interface Failures", automatic failback is supported for physical interfaces that are not present at system boot. See "How to Recover a Physical Interface That Was Not Present at System Boot".
Group Failures
A group failure is when all the network interface cards appear to fail at the same time. in.mpathd does not do any failovers for a group failure. Also, no failover occurs when all the targets fail at the same time. In this instance, in.mpathd flushes all of its current targets and discovers new targets (see "Detecting Physical Interface Failures").
Note - Group failures were previously known as link failures.
Administering Multipathing Groups With Multiple Physical Interfaces
This section describes how you enable IP Network Multipathing. To use the IP Network Multipathing feature, you should have more than one physical interface connected to the same IP link. For example, the same Ethernet switch or the same IP subnet, configured under the same multipathing group, works. If you have just one physical interface, refer to "Administering Multipathing Groups With a Single Physical Interface".
Multipathing groups are identified by non-null names. For example, math-link, bio-link, and chem-link make good names. The names typically represent where these groups are connected. When failure is detected in one of the network adapters in the multipathing group, all network access is failed over from the failed adapter to the working adapter in the group. The failover of network access includes IPv4 unicast, broadcast, and multicast traffic, as well as IPv6 unicast and multicast traffic. For IP network multipathing to function properly, the following conditions must exist for the network adapters that are part of the same multipathing group:
You must push and configure the same set of STREAMS modules on all network adapters in the multipathing group.
If you have plumbed IPv4 on one network adapter, then you must plumb IPv4 on all network adapters in the multipathing group.
If you have plumbed IPv6 on one network adapter, then you must plumb IPv6 on all network adapters in the multipathing group.
All Ethernet network adapters in the system should have unique MAC address in the case of ethernets. This is achieved by setting the local-mac-address? to TRUE in the openboot PROM for SPARC platforms. Nothing needs to be done for IA (x86) platforms.
All network adapters of the multipathing group must be connected to the same IP link.
The multipathing group should not contain dissimilar interfaces. The interfaces that are grouped together should be of the same interface type that is defined in /usr/include/net/if_types.h. For example, you cannot combine Ethernet with Token ring, and you cannot combine a Token bus with asynchronous transfer mode (ATM).
When you use IP network multipathing with ATMs, you must configure the ATM for LAN emulation (multipathing over classical IP instances is not currently supported).
Note - The fourth condition concerns all interfaces in the system, not just those belonging to the multipathing group.
For the adapters that do not come with factory-set unique MAC addresses, you can manually configure a MAC address for each adapter as a workaround. For example, use the ifconfig ether command in a startup script file.
Note - The MAC addresses configured manually cannot be maintained across system reboot. You are responsible for choosing unique MAC addresses. IP Network Multipathing might behave unpredictably if the MAC addresses of adapters are not unique.
Grouping Physical Interfaces
You use the ifconfig command to configure groups. This command uses a new group parameter that requires a group name and places both the IPv4 and IPv6 instances of the interface in that group. The group parameter has the following syntax:
ifconfig interface-name group group-name |
Note - Avoid using spaces in group names. The ifconfig status display does not show spaces. Consequently, if you have created two similar group names, but one of them contains a space, these group names look alike in the status display. However, they are different group names. This difference might be confusing.
Placing the IPv4 instance under a particular group automatically places the IPv6 instance under the same group. Also, you can place a second interface, connected to the same subnet, in the same group by using the same command. See "How to Configure a Multipathing Interface Group With Two Interfaces".
You can remove an interface from a multipathing group by using a null string with the group sub-command. See "How to Remove an Interface From a Group".
To place an interface in a new group when it is already part of some multipathing group, you do not need to remove it from any existing group. Placing the interface in a new group automatically removes it from any existing group. See "How to Move an Interface From an Existing Group to a Different Group".
You can have any number of network adapters that you can configure in the same multipathing group. You cannot use the group parameter with logical interfaces. For example, you can use the parameter with hme0, but not with hme0:1.
You must connect all the interfaces in the multipathing group to the same IP link, because when an interface fails, the failover operation moves all the IP addresses from the failed interface to an interface in the group that is functional. For routers to continue routing packets to the addresses that have been switched to the functional interface, the functional interface must be connected to the same IP link.
Configuring Test Addresses
You must configure all physical interfaces of a multipathing group with a test address. You need test addresses to detect failures and repairs. If a test address is not configured, it is not chosen for failover. Only in.mpathd uses test addresses. Normal applications should not use this address. This address does not fail over when the interface fails. In IPv4, you should configure the test address in such a way that normal applications do not use the test address. See "How to Configure a Multipathing Interface Group With Two Interfaces".
This section describes test address configuration concepts for the following Internet protocols:
IPv4 Test Addresses
The in.mpathd multipathing daemon requires a test IP address for detecting failures and repairs. You must use a routeable address for this IP address. That is, the subnet prefix of the address must be known to any routers present on the link. You use the ifconfig command's new -failover option to configure a test address. Use the following syntax to configure a test address:
# ifconfig interface-name addif ip-address <other-parameters> -failover up |
For <other-parameters>, use the parameters that are required by your configuration. See the ifconfig(1M) man page for descriptions. "How to Configure a Multipathing Interface Group With Two Interfaces" shows the steps you perform for an IPv4 test address.
For example, to add a new logical interface with an address of 19.16.85.21, the netmask and broadcast address set to the default value, and also configure the interface with a test address, type the following:
# ifconfig hme0 addif 19.16.85.21 netmask + broadcast + -failover up |
Note - You must mark an IPv4 test address as deprecated to prevent applications from using the test address. See "How to Configure a Multipathing Interface Group With Two Interfaces".
Use failover without the dash to turn on the failover attribute of the address.
Note - All test IP addresses in a multipathing group must use the same network prefix. That is, the test IP addresses must belong to a single IP subnet.