[rancid] 7513 CBUS mash-up provoked by RANCID

[Ed Ravin]

This morning's RANCID run against our 7513 showed that all our
interfaces in slot 0 were no longer in the config, as if they'd
been pulled out.  The router logs show:

.Nov 14 10:13:23 EST: %DBUS-3-DBUSINTERR: Slot 0, Internal Error
.Nov 14 10:13:23 EST: %LB-5-CHAN_MEMBER_OUT: FastEthernet0/0/0 taken out of Port-channel1
.Nov 14 10:13:23 EST: %LB-5-CHAN_MEMBER_OUT: FastEthernet0/0/0 taken out of Port-channel1
.Nov 14 10:13:59 EST: %CBUS-3-CMDTIMEOUT: Cmd timed out, CCB 0xF800FFA0, slot 8, cmd code 2
 Nov 14 10:13:25 166.84.143.9/166.84.143.9 21384: .Nov 14 10:13:23 EST: %DBUS-3-DBUSINTERR: Slot 0, Internal Error 
-Traceback= 4032C744 404B1A5C 404B2330 404A962C 404B83B4 401A1E44 401A0A14 401A48DC 401A57A4 401A8290 4039BD64 404A0DEC 404AF9E0 404B0020 404A17FC
.Nov 14 10:13:59 EST: %CBUS-3-CCBCMDFAIL1: Controller 8, cmd (61 0x00000008) failed (0x8010)
.Nov 14 10:13:59 EST: %CBUS-3-CCBCMDFAIL1: Controller 8, cmd (36 0x00000060) failed (0x8010)
.Nov 14 10:13:59 EST: %CBUS-3-ADDRFILTR: Interface FastEthernet8/1/0, address filter write command failed, code 0x8010
-Traceback= 4032C744 404B7844 404B8044 404B83BC 401A1E44 401A0A14 401A48DC 401A57A4 401A8290 4039BD64 404A0DEC 404AF9E0 404B0020 404A17FC
.Nov 14 10:13:59 EST: %CBUS-3-CCBCMDFAIL1: Controller 8, cmd (36 0x0000FFFF) failed (0x8010)
.Nov 14 10:13:59 EST: %CBUS-3-CCBCMDFAIL1: Controller 8, cmd (36 0x00000060) failed (0x8010)
.Nov 14 10:13:59 EST: %CBUS-3-CCBCMDFAIL1: Controller 8, cmd (36 0x0000FFFF) failed (0x8010)
.Nov 14 10:13:59 EST: %CBUS-3-CCBCMDFAIL1: Controller 8, cmd (36 0x00000100) failed (0x8010)
.Nov 14 10:13:59 EST: %CBUS-3-CCBCMDFAIL1: Controller 8, cmd (36 0x00000100) failed (0x8010)
.Nov 14 10:13:59 EST: %CBUS-3-CCBCMDFAIL1: Controller 8, cmd (36 0x00000100) failed (0x8010)
.Nov 14 10:13:59 EST: %CBUS-3-CCBCMDFAIL1: Controller 8, cmd (36 0x00000100) failed (0x8010)
.Nov 14 10:13:59 EST: %CBUS-3-CCBCMDFAIL1: Controller 8, cmd (36 0x00000100) failed (0x8010)
.Nov 14 10:13:59 EST: %CBUS-3-CCBCMDFAIL1: Controller 8, cmd (36 0x00000100) failed (0x8010)
.Nov 14 10:13:59 EST: %LB-5-CHAN_MEMBER_IN: FastEthernet0/0/0 added as member-2 to Port-channel1
.Nov 14 10:13:59 EST: %LB-5-CHAN_MEMBER_OUT: FastEthernet8/1/0 taken out of Port-channel1
.Nov 14 10:13:59 EST: %LB-5-CHAN_MEMBER_OUT: FastEthernet8/1/0 taken out of Port-channel1
.Nov 14 10:13:59 EST: %SYS-3-CPUHOG: Task ran for 10828 msec (257/150), process = OIR Handler, PC = 404A158C.
-Traceback= 404A1594

And then a few moments later:

.Nov 14 10:14:59 EST: %DBUS-3-WEDGED: Line card in slot 8 is wedged
.Nov 14 10:15:37 EST: %LB-5-CHAN_MEMBER_IN: FastEthernet8/1/0 added as member-2 to Port-channel1
.Nov 14 10:15:37 EST: %SYS-3-CPUHOG: Task ran for 11732 msec (52/14), process = OIR Handler, PC = 404A158C.
-Traceback= 404A1594

And the router seems to have found its slot again.  I looked at the
router config, and the slot 0 devices are back in there.

As near as I can tell, one of RANCID's diagnostic commands provoked
the CBUS stall, and when RANCID subsequently read the config, pieces
of it were missing since the router was still trying to figure out
which hardware was working and which wasn't.

Has anyone else seen a 7500 router react this way to RANCID probes?