| SRDB ID | Synopsis | Date | ||
| 48120 | Sun Fire[TM] 15K: Management Networks (MAN) | 4 Nov 2002 |
| Status | Issued |
| Description |
What is a Management Network and how is it related to the Sun Fire[TM] 15K server (Starcat) platform?
SOLUTION SUMMARY:
The Sun Fire 15K has three ethernet Management (MAN) Networks. Each system controller
incorporates 20 RIO (eri) and two Cheerio (hme) based ethernet interfaces to comprise
the Sun Fire 15K MAN network.
This management network consists of three individual ethernet networks:
1) C network- Consisting of two network interfaces connected to the system
controller front panel providing the external community remote access to the system
controller. This network is often referred to as the Community Network.
2) I1 network- Consisting of 18 separate RIO-based Ethernet controllers between
the system controller and each I/O board on the platform. These 18 interfaces provide
a highly available, secure network console interface to facilitate communication
between a possible 18 domains and the system controller. This network is an internally
connected network. It is also referred to as the dman network.
3) I2 network- Consisting of two Ethernet interfaces to the other system
controller to provide a heartbeat and synchronizes system controller configuration
data between the main and spare system controller. This network is also referred to
as the scman network. It is an internally connected network.
--------------------------------------------------------------------------------------------------------
C Network
The world accesses the system controllers (sc0 and sc1) through the C Network (community
network). This means that the world will be attaching to either the hme0 interface or
the eri1 interface on the control board. This external interface set can be built on
top of Solaris IP Multi-Pathing (IPMP) to provide HA network access to the system
controllers.
The C Network interfaces on the system controller are the hme0 and the eri1 interfaces.
The hme0 interface is the top most RJ 45 connector on the system controller, while the
eri1 interface is the bottom most RJ 45 connector on the system controller.
The function of this network is as follows:
-Have one or more customer-provided network connection to the platform administrative
server (i.e.. the system controllers).
Here's an example of an ifconfig -a from a system controller (showing only the C network
interface):
# ifconfig -a
hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 172.20.66.148 netmask ffffff00 broadcast 172.20.66.255 ether 8:0:20:da:7e:f6
--------------------------------------------------------------------------------------------------------
I1 Network
The system controllers communicate directly to the platform's domains via the I1 Network
(dman network). The system controllers use eri2 through eri19 interfaces onboard on the
system controller to communicate with eri chips on each possible I/O board in the
platform (connections are internal, using circuitry). When configured into a domain,
the I/O board's eri interface on this network is called a dman0 interface. The interface
the system controller uses to connect to the domains dman0 interface is called scman0.
The system controller has the eri2 through eri19 chips imbedded in the system controller's
circuitry. Take a look at the system controller skematic from the Sun System Handbook and
you will see the eri basics (labeled as RIO chips; Note that two of the RIO chips are
partially obscured under the power supply daughter card). The system controller has an
individual chip for each possible domain which can be configured on the F15K/Starcat
platform (18 possible). Thus each of the eri chips on the system controller connects to
an individual eri chip on the I/O boards configured in their respective domains. If
multiple I/O boards are configured into a domain, the system controller's eri chip
communicates to the lowest numbered I/O board in the domain (typically). The eri chip
on the lowest configured I/O board is called the golden IOSRAM . You will see this "golden
IOSRAM" being configured in the last stages of an HPOST run when bringing up a domain. The
purpose behind the golden IOSRAM is to facilitate a network boot off of the MAN network
for that individual domain. What this means is that during the bringup stages, the eri
interface which is configured as the golden IOSRAM becomes the man-net device alias,
thus a network boot could be made by doing a "boot man-net", and thus the domain would be
booting off the dman network using the system controller as it's boot server.
The functions of this network are as follows:
-Log all domain console messages to the system controller(s).
-Log all domain messages to the system controller(s).
-Perform Dynamic Reconfiguration (DR) operations.
-Perform network boot/Solaris installation (optional).
-Perform time synchronization (optional).
Here's an example of an ifconfig -a from a system controller (showing only the I1 interface)
# ifconfig -a
scman0: flags=1008843<UP,BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 3
inet 10.1.1.1 netmask ffffffe0 broadcast 10.1.1.31 ether 8:0:20:da:7e:f6
Here's an example of an ifconfig -a from a domain (showing only the I1 interface):
# ifconfig -a
dman0: flags=1008863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 2
inet 10.1.1.2 netmask ffffffe0 broadcast 10.1.1.31 ether 0:0:be:a8:4a:40
--------------------------------------------------------------------------------------------------------
I2 Network
The system controllers communicate directly to each other via internally connected
circuitry called the I2 Network (scman network). Specifically, the system controllers
interfaces, hme1 and eri0, communicate directly to each other. The interface the system
controllers plumb up to communicate to each other is scman1.
The I2 Network is internally connected. Therefore the system controllers themselves
will have no RJ 45 connectors or external cables whatsoever related to the I2 network.
The hme1 and eri0 interfaces themselves are simply chips on the system controllers that
are connected to each other via internal circuitry on the expanders and centerplane that
the system controllers connect with.
The functions of this network are as follows:
-Perform inter-system controller communication (This is sometimes referred to as the
Heartbeat Network).
-The SMS failover mechanisms use this network as one means to determine the health of the
opposite system controller.
-This network is also used for SMS file propagation (datasync).
Here's an example of an ifconfig -a from a system controller (showing only the I2 interface):
# ifconfig -a
scman1: flags=1008842<BROADCAST,RUNNING,MULTICAST,PRIVATE,IPv4> mtu 1500 index 4
inet 10.2.1.1 netmask fffffffc broadcast 10.2.1.3 ether 8:0:20:da:7e:f6
--------------------------------------------------------------------------------------------------------
Management Network Daemon
The F15K/Starcat server uses a combination of hardware and software to provide
administrative services to the platform's network configuration. The management
network daemon (mand) supports all of the networks in the F15K/Starcat platform.
The primary operations this daemon serves are:
-Manages domain consoles and message logging services.
-Manages time synchronization.
-Manages Dynamic Reconfiguration (DR) processes.
-Manages network boot for the domains, as well as Solaris installation.
-Manages the system controller heartbeats.
How mand interacts with the platform functionality:
At system controller startup, the failover daemon (fomd) determines what role the system
controller will function in, whether it be spare mode or main mode. At the very beginning
of system controller startup, mand will be in spare mode. It does not switch over to main mode until fomd tells it to do so. The fomd daemon then starts mand appropriately.
The mand daemon (once in main mode) then checks the contents of the /etc/opt/SUNWSMS/config/MAN.cf
file (discussed later in this doc) to create a mapping between the "domain ID" and it's IP
address in the platform configuration database (pcd). Mand then obtains domain configuration
information from the pcd daemon in order to properly configure the scman driver.
Next, mand seeks domain change notifications, such as newly added system boards to a
particular domain, and as appropriate will then update the scman driver of the changes.
Also, mand tracks the active ethernet interface in the scman driver and if changes to that
interface occur (Such as a failure to one of the scman interfaces), then updates those
changes back to the pcd daemon.
When a domain is powered on (setkeyswitch -d <domainID> on) mand relays the system startup
MAN network information to that particular domain. The mand does not plumb up actual
interfaces on the domain, it simply passes on the network information (includes ethernet
and MAN network IP address information) to the dman driver running in the domain, and
the dman driver then plumbs up it's domain interfaces (This is where the golden IOSRAM
interface is used---obtaining the information passed on from the system controller via
mand). This process occurs during the domain's boot cycle and during the initial domain
OS install. This process is how the domain's dman obtains the main system controller's
MAC address.
So, to summarize, mand is responsible for updating all the responsible network drivers of
the platform's network configuration. Without mand running properly, the entire F15K/Starcat
platform's network is unlikely to continue working properly (certainly no new configuration
updates will occur at all). Also, because mand is responsible for assuring that the system
controller's remain synched, if it is having issues, it is possible that system clock can
become disrupted resulting in a platform wide disruption of service. So, mand is an
extremely critical daemon.
--------------------------------------------------------------------------------------------------------
Path "Policing" Mechanisms
C Network
The C Network is not "policed" per say via any direct inter-platform process or configuration.
If IPMP is configured, network testing on the C Network is done via the "test" network
interface. This test interface is an actual NIC (ex. qfe0) which has an IP address used
exclusively for testing whether the particular NIC is still active. The system uses the
logical interface (ex. qfe0:1) to transfer it's data to the C Network. If in the C network
tests on the test interface (Typically a ping test to a router or a multicast to the
network) a response does not come back, IPMP automatically will fail the logical IP over
to the failover IP, which is configured as the other interface(s) in the IPMP group
(assuming of course that failover was configured properly, and that the other interface
is also fine).
Outside of IPMP, the C Network is not "policed" any differently than a standard Solaris[TM]
network interface would be.
To see the "policing" in action, a snoop of the test interface (ex. qfe0) should be
executed if the configuration is IPMP. On the test interface, the only traffic is the
test ping or multicast.
If IPMP is not configured, the "policing" of the C Network is handled by Solaris. Solaris
uses tpe-link-test to perform network connection tests. Responses to this test come from
the pieces of equipment connected on the other side of the cable that is attached to the
particular C Network interface, such as a router, switch, hub, or other node.
I1 Network
The I1 Network is "policed" every 10 seconds. Specifically the dman0 interface on the
domain pings the system controller via the active domain NIC (golden IOSRAM) every 10
seconds. Also, dman0 checks the inbound packet count every 30 seconds. If the packet
count has increased, the connection is considered good. If not, a path switch is initiated
and the next available path is selected.
To see this "policing" in action, simply snoop the dman0 interface and watch out for
these 10 second and 30 second interface tests.
I2 Network
The I2 Network is also "policed" every 10 seconds. Specifically, scman1 on the main
system controller pings the spare system controller via it's active NIC every 10 seconds.
Also, every 30 seconds, scman1 on the main system controller checks it's inbound packet
count. If the packet count has increased, the connection is considered good. If not, the
active path is switched to the other NIC on the main system controller, provided it was
not previously marked as failed.
Assuming you have both system controllers up and running in the platform configuration,
you can see the "policing" in action simply by snooping the scman1 interface on one or
both of the system controllers.
NOTE: If at anytime a failure is detected with any of these NICs, mand is responsible
for updating the pcd (platform configuration database) and making sure all networks are
aware that a particular NIC is offline and that communication has switched over to a
different NIC if possible.
--------------------------------------------------------------------------------------------------------
Management Network (MAN) Configuration Files
/etc/opt/SUNWSMS/config/MAN.cf
-This is the MAN configuration file.
-Located on the system controllers.
-Read by the daemon mand when entering main system controller mode.
-The file contains all the IP addresses for the system controller, domain, and external
community interfaces.
-This file is created by smsconfig -m during the MAN Network configuration stage at install.
It should not be edited by hand.
Example:
# cat /etc/opt/SUNWSMS/config/MAN.cf
#
# ident "@(#)MAN.cf 1.5 01/05/23 SMI"
#
# Copyright (c) 2000-2001 by Sun Microsystems, Inc.
# All rights reserved.
#
# /etc/opt/SUNWSMS/config/MAN.cf
#
mc15k
I1 NM-I1 netmask-i1 255.255.255.224
I1 SC-I1 mc15k-sc-i1 10.1.1.1
I1 DA-I1 mc15k-a 10.1.1.2
I1 DB-I1 mc15k-b 10.1.1.3
I1 DC-I1 mc15k-c 10.1.1.4
I1 DD-I1 mc15k-d 10.1.1.5
I1 DE-I1 mc15k-e 10.1.1.6
I1 DF-I1 mc15k-f 10.1.1.7
I1 DG-I1 mc15k-g 10.1.1.8
I1 DH-I1 mc15k-h 10.1.1.9
I1 DI-I1 mc15k-i 10.1.1.10
I1 DJ-I1 mc15k-j 10.1.1.11
I1 DK-I1 mc15k-k 10.1.1.12
I1 DL-I1 mc15k-l 10.1.1.13
I1 DM-I1 mc15k-m 10.1.1.14
I1 DN-I1 mc15k-n 10.1.1.15
I1 DO-I1 mc15k-o 10.1.1.16
I1 DP-I1 mc15k-p 10.1.1.17
I1 DQ-I1 mc15k-q 10.1.1.18
I1 DR-I1 mc15k-r 10.1.1.19
I2 NM-I2 netmask-i2 255.255.255.252
I2 SC0-I2 mc15k-sc0-i2 10.2.1.1
I2 SC1-I2 mc15k-sc1-i2 10.2.1.2
/var/opt/SUNWSMS/data/<domain>/idprom.image
-This file contains the ethernet address of the domain.
-Located on the system controllers.
-This file is referenced by mand when creating the IOSRAM handoff structure.
-To view this file, you must run sysid -d <DomainID> on the system controller.
-It is also very important to note that these files are not easily regenerated (If lost,
at this point only CPRE can reproduce them), so back them up!
Example:
# sysid -d A
IDPROM in /var/opt/SUNWSMS/data/A/idprom.image for domain A
Format = 0x01
Machine Type = 0x82
Ethernet Address = 0:0:be:a8:4a:40
Manufacturing Date = Wed Jun 27 14:58:00 MDT 2001
Serial number (Host ID) = 0xa84a40 (11029056)
Checksum = 0xac
/var/opt/SUNWSMS/doors/mand
-This is the doors for mand.
-Located on the system controllers.
-This allows other processes to invoke mand methods. For example fomd instructing mand
to enter main mode. It is important to note that this is not a file, simply it is a file
structure which allows interprocess communication.
Example:
# file /var/opt/SUNWSMS/doors/mand
/var/opt/SUNWSMS/doors/mand: door to mand[18356]
/etc/hosts
-This is the standard Solaris hosts listing.
-Located on system controllers and domains in the platform.
-It contains the certain hosts specific to the MAN Networks.
Example:
# cat hosts
#
# Internet host table
#
127.0.0.1 localhost
172.20.66.148 mc15k-sc0 loghost
172.20.66.149 mc15k-sc1 loghost
172.20.66.19 route66
10.1.1.2 mc15k-a #smsconfig-entry#
10.1.1.3 mc15k-b #smsconfig-entry#
10.1.1.4 mc15k-c #smsconfig-entry#
10.1.1.5 mc15k-d #smsconfig-entry#
10.1.1.6 mc15k-e #smsconfig-entry#
10.1.1.7 mc15k-f #smsconfig-entry#
10.1.1.8 mc15k-g #smsconfig-entry#
10.1.1.9 mc15k-h #smsconfig-entry#
10.1.1.10 mc15k-i #smsconfig-entry#
10.1.1.11 mc15k-j #smsconfig-entry#
10.1.1.12 mc15k-k #smsconfig-entry#
10.1.1.13 mc15k-l #smsconfig-entry#
10.1.1.14 mc15k-m #smsconfig-entry#
10.1.1.15 mc15k-n #smsconfig-entry#
10.1.1.16 mc15k-o #smsconfig-entry#
10.1.1.17 mc15k-p #smsconfig-entry#
10.1.1.18 mc15k-q #smsconfig-entry#
10.1.1.19 mc15k-r #smsconfig-entry#
10.1.1.1 mc15k-sc-i1 #smsconfig-entry#
10.2.1.2 mc15k-sc1-i2 #smsconfig-entry#
10.2.1.1 mc15k-sc0-i2 #smsconfig-entry#
/etc/hostname.* /etc/hostname6.*
-This file shows the dman0, scman0, and scman1 varieties.
-Located on the system controllers (in the case of the scman varieties; dman interfaces
will be exclusive to the domains).
-They are referenced by drivers when the interfaces are plumbed.
Example:
# cat hostname.hme0
mc15k-sc0
# cat hostname.scman0
10.1.1.1 netmask + private up
# cat hostname.scman1
10.2.1.1 netmask + private up
# cat hostname6.hme0
#
/etc/ipnodes
-This is the standard Solaris hosts listing that contains hosts relevant to IPv6 hosts.
-Located on system controllers and domains where appropriate.
/etc/netmasks
-This is the standard Solaris netmasks settings file.
-Located on system controllers and domains.
-It may contain netmasks relevant to the MAN Networks.
Example:
# cat netmasks
#
# The netmasks file associates Internet Protocol (IP) address
# masks with IP network numbers.
#
# network-number netmask
#
# The term network-number refers to a number obtained from the Internet Network
# Information Center. Currently this number is restricted to being a class
# A, B, or C network number. In the future we should be able to support
# arbitrary network numbers per the Classless Internet Domain Routing
# guidelines.
#
# Both the network-number and the netmasks are specified in
# "decimal dot" notation, e.g.:
#
# 128.32.0.0 255.255.255.0
#
172.20.66.0 255.255.255.0
192.168.103.0 255.255.255.224
192.168.103.32 255.255.255.252
10.1.1.0 255.255.255.224
10.2.1.0 255.255.255.252
/etc/nsswitch.conf
-This is the standard Solaris name service configuration file.
-Located on system controllers and domains.
-It may require modification based on the site's name service. INTERNAL SUMMARY:
SUBMITTER: Joshua Freeman APPLIES TO: ATTACHMENTS: