- Problem Statement: 

        Locating the Kernel in a SF12K/SF15K Domain for DR Operations

- Symptoms:

        Locating non-pageable (or permanent) memory in the system
        is necessary to plan and predict which DR operations may require
        a quiesce of the OS. Permanent memory is indicative of the presence of
        kernel memory.
            

- Troubleshooting:

        From the System Controller, use the following command:

        sc% rcfgadm -d B -av | grep SB | grep permanent
        SB17::memory                   connected    configured   ok         base address 0x20000000000, 8388608 KBytes total, 997016 KBytes permanent

        From the Domain, use the following command:

        domain# cfgadm -av | grep SB | grep permanent
        SB17::memory                   connected    configured   ok         base address 0x20000000000, 8388608 KBytes total, 997016 KBytes permanent

        Any memory with the 'permanent' designation will require an OS
        quiesce if detached from a running domain. Note that permanent
        memory need not be confined to a single system board.

- Resolution:

        Output of above commands.

- Summary of part number and patch ID's 

        N/A
        
- References and bug IDs

        N/A

- Additional background information:

        At boot time, kernel memory is loaded into the highest address space
        in the domain. This is typically, but not necessarily always, the
        highest numbered system board in the domain. Prior DR detaches that
        involved OS quiesces alter the physical memory address mapping in
        the system.

        In the Sun Fire 12K/15K, the physical memory address space is divided
        into 128GB slices. Each memory slice is mapped to a system board. The
        current mapping of a system is available via redx:

        redx> pcd p
        mem_slice_map[17:0]= 17 16 15 14 13 12 11 10 09 08 07 06 05 04 03 02 01 00
                             ^^                                                 ^^
                            high                                               low

        Each column is a 128GB slice. Slice 17 is the highest memory address 
        range and slice 0 the lowest. The two digit value in each column is the
        system board to which that address slice is mapped. For example, memory 
        slice 15 is mapped to SB15. Assume a domain that contains SB15, SB16, 
        and SB17 and the platform memory slice map is as above. At boot time, 
        the kernel is loaded starting in SB17 since this system board is the 
        highest address range.

        If SB17 is later detached, since kernel memory is present on the board,
        a quiesce of the OS and a copy-rename operation is performed. A
        copy-rename operation takes the physical address range of one system
        board and remaps it to another. Suppose that SB15 receives kernel memory
        when SB17 is detached. Copy-rename assigns the address slice of SB17
        to SB15, and the slice map becomes:

        redx> pcd p
        mem_slice_map[17:0]= 15 16 17 14 13 12 11 10 09 08 07 06 05 04 03 02 01 00
                             ^^                                                 ^^
                            high                                               low

        The highest address range is now mapped to SB15. SB15's former address
        slice is mapped to SB17. Subsequent reboots/rePOSTs of this domain 
        result in the kernel being loaded starting in SB15 since this system
        board is now at the highest address range.

        For completeness, when DR must relocate the kernel memory, target selection
        is done based on weighting. DR categorizes all remaining system boards in 
        the system according to predefined preferences. The preferences in order are:

          1. System board with the same memory size 
          2. System board with a larger memory size
          3. System board with a smaller memory size
          4. System board with a smaller memory size, but offsetting pushes the
             end physical address beyond the maximum memory slice, reducing the
             overall available memory

        Once all boards have been categorized, the lowest numbered system board
        with the highest preference is selected as the new location of the kernel.

- Meta-Data/Problem categorization:

Product/Platform: SF12K/SF15K
Category:

- Keywords

kernel, DR, cage