A Sun Fire[TM] 12K/15K domain panics with a send mondo timeout.

You may see messages like the following occur:

Mar 24 19:28:32 skeleton SUNW,UltraSPARC-III+: [ID 563681 kern.notice] send mondo timeout [694314 NACK 0 BUSY] 
Mar 24 19:28:32 skeleton IDSR 0x100000  aids: 
Mar 24 19:28:32 skeleton SUNW,UltraSPARC-III+: [ID 823475 kern.notice]  42 
Mar 24 19:28:32 skeleton SUNW,UltraSPARC-III+: [ID 100000 kern.notice] 
Mar 24 19:28:32 skeleton unix: [ID 350512 kern.notice] panic: failed to stop cpu66 
Mar 24 19:28:32 skeleton unix: [ID 836849 kern.notice] 
Mar 24 19:28:32 skeleton ^Mpanic[cpu482]/thread=2a100017d20: 
Mar 24 19:28:32 skeleton unix: [ID 152620 kern.notice] send_mondo_set: timeout 
Mar 24 19:28:32 skeleton unix: [ID 100000 kern.notice] 
Mar 24 19:28:32 skeleton genunix: [ID 723222 kern.notice] 000002a100016f60 SUNW,UltraSPARC-III+:send_mondo_set+258 
2a100017490, a982a, 12, 2aaaaaaaaa, 13, 19bf7f1776)                              

See FIN I0765-1 for details.

Action:

See the FIN for details. There are three different cases requiring three different actions, and you must first decide which scenario you have encountered before resolving the issue. Rule out cases 1 and 2 first, before heading on to case 3.

Case 1: Bug 4432461

Case 2: J2SE v1.2.2 Non Standard JIT Compiler Optimization Option

Case 3: Bad Hardware

From Mark Woelfel:

This FIN states that one of the fixes for the 3800-6800 SunFire systems is to update the flashprom with the following patches.

112127-01 - Hardware/PROM: Sun Fire 3800/4800/4810/6800 Systems flashprom update.

111346-04 - Hardware/PROM: Sun Fire 3800/4800/4810/6800 Systems flashprom update.

This is not necessary on the Sun Fire 12K/15K; usually when you see these types of panics they indicate some type of a hardware error, with the exception of the 1 other case that are mentioned in the FIN:

If the system in question is using J2SE v1.2.2 (including product family updates, e.g., J2SE v1.2.2_10) with the non-standard/experimental JIT compiler optimization option:

-Xoptimize

send mondo panics may occur due to the code optimization strategy employed by the J2SE v1.2.2 unsupported "-Xoptimize" option.

This was verified by CPRE Marc Friedman.