Clogin issue with Cisco 10K

Mack, Dave dmack at gnilink.net
Tue Oct 4 14:17:47 UTC 2005


Hello All!
	I am trying to troubleshoot an issue with clogin hanging with
Rancid. I have tried to troubleshoot it myself and have gotten to the
point where I have determined that  clogin is hanging on the output of
"show diag". The funny part is that it appears to be only one router
having this issue. I have tried running clogin -c "sh diag" on other
routers running the same exact code and they work fine. With this router
output hangs and I have to ctrl-c to kill clogin.


Here is an example of the input from a console session to the router:

POP-EDGE-RTR1#sh diag
Slot A:
        Primary PRE card
RP EEPROM contents:
        Hardware Revision        : 1.0
        Part Number              : 73-7041-03
        Board Revision           : C0
        Deviation Number         : 0-0
        Fab Version              : 02
        PCB Serial Number        : CAT07230QB1
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Part Number              : 800-17437-06
        CLEI Code                : CNRUGMVBAA
FP EEPROM contents:
        Hardware Revision        : 1.0
        Part Number              : 73-7042-06
        Board Revision           : A0
        Deviation Number         : 0-0
        Fab Version              : 05
        PCB Serial Number        : CAT07230Q6X
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
Operational Image Version, Slot A
        Cisco Internetwork Operating System Software 
IOS (tm) 10000 Software (C10K-K4P10-M), Experimental Version
12.0(20040113:203533) [REL-v120_26_sz_throttle.ios-weekly 100]
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Tue 13-Jan-04 15:35 by leccese


Slot B:
        Secondary PRE card
RP EEPROM Contents:
        Hardware Revision        : 1.0
        Part Number              : 73-7041-03
        Board Revision           : C0
        Deviation Number         : 0-0
        Fab Version              : 02
        PCB Serial Number        : CAT07230S3K
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Part Number              : 800-17437-06
        CLEI Code                : CNRUGMVBAA
FP EEPROM Contents:
        Hardware Revision        : 1.0
        Part Number              : 73-7042-06
        Board Revision           : A0
        Deviation Number         : 0-0
        Fab Version              : 05
        PCB Serial Number        : CAT07230RW3
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
Standby version: 
Cisco Internetwork Operating System Software 
IOS (tm) 10000 Software (C10K-K4P10-M), Experimental Version
12.0(20040113:203533) [REL-v120_26_sz_throttle.ios-weekly 100]
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Tue 13-Jan-04 15:35 by leccese


Slot/Subslot 1/0:
        1oc12atm-1 card, 1 port
        Card is full slot size
        Card is analyzed 
        Card detected 1y26w ago
        Card uptime 710 days, 16722 hours, 1002185 minutes, 60129563
seconds
        Card idle time 523 days, 11937 hours, 715828 minutes, 42949689
seconds
        Voltage status: 3.3V Nominal 2.5V Nominal
EEPROM contents, slot 1/0:
        Hardware Revision        : 1.0
        Part Number              : 800-06568-03
        Board Revision           : D0
        Deviation Number         : 0-0
        Fab Version              : 02
        PCB Serial Number        : CAT072504KB
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        CLEI Code                : BAA8LKDCAA
LCMON version, slot 1/0
        LCDOS (C10000 PowerQUICC-II Line Card MONitor Image : Release
branch:c10k_lc_bgp_reorg 20011120:145454)
        Built by richv at Tue Nov 20 17:04:22 2001.
        Reset reason 0x00000003 (external hard reset).
Operational Image version, slot 1/0
        LCDOS (C10000 1 Port OC12 ATM Line Card Image : DEVELOPMENT
BUILD REL-v120_26_s_lc_throttle.lcdos-weekly /vob/lcdos/obj-c10k-oc12atm
101) major version 1060699281.
        Built by leccese at Tue Aug 12 10:41:22 2003.
        ECC 1 bit errors since last cleared (dd hh mm ss) = 0 (0 days, 0
hours, 0 minutes, 0 seconds)
        ECC 1 bit errors while up (total) = 0


Slot/Subslot 3/0:
        1oc12pos-1 card, 1 port
        Card is full slot size
        Card is analyzed 
        Card detected 1y26w ago
        Card uptime 713 days, 16707 hours, 1002169 minutes, 60129583
seconds
        Card idle time 589 days, 13137 hours, 787424 minutes, 47244688
seconds
        Voltage status: 3.3V Nominal
EEPROM contents, slot 3/0:
        Hardware Revision        : 2.0
        Part Number              : 800-05546-03
        Board Revision           : F0
        Deviation Number         : 0-0
        Fab Version              : 03
        PCB Serial Number        : CAT072209MG
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
LCMON version, slot 3/0
        LCDOS (C10000 Line Card MONitor Image : Release 20000427:084233)
        Built by richv at Thu Apr 27 08:44:08 2000.
        Reset reason 0xC0000000 (external hard reset).
Operational Image version, slot 3/0
        LCDOS (C10000 1 Port OC12 POS Line Card Image : DEVELOPMENT
BUILD REL-v120_26_s_lc_throttle.lcdos-weekly /vob/lcdos/obj-c10k-oc12pos
101) major version 1060698969.
        Built by leccese at Tue Aug 12 10:36:10 2003.

Slot/Subslot 4/0:
        1oc12pos-1 card, 1 port
        Card is full slot size
        Card is analyzed 
        Card detected 1y26w ago
        Card uptime 713 days, 16707 hours, 1002171 minutes, 60129593
seconds
        Card idle time 588 days, 13128 hours, 787446 minutes, 47244643
seconds
        Voltage status: 3.3V Nominal
EEPROM contents, slot 4/0:
        Hardware Revision        : 2.0
        Part Number              : 800-05546-03
        Board Revision           : F0
        Deviation Number         : 0-0
        Fab Version              : 03
        PCB Serial Number        : CAT072209MS
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
LCMON version, slot 4/0
        LCDOS (C10000 Line Card MONitor Image : Release 20000427:084233)
        Built by richv at Thu Apr 27 08:44:08 2000.
        Reset reason 0xC0000000 (external hard reset).
Operational Image version, slot 4/0
        LCDOS (C10000 1 Port OC12 POS Line Card Image : DEVELOPMENT
BUILD REL-v120_26_s_lc_throttle.lcdos-weekly /vob/lcdos/obj-c10k-oc12pos
101) major version 1060698969.
        Built by leccese at Tue Aug 12 10:36:10 2003.

Slot/Subslot 5/0:
        6cht3-1 card, 6 ports
        Card is full slot size
        Card is analyzed 
        Card detected 1y26w ago
        Card uptime 644 days, 14333 hours, 859025 minutes, 51539618
seconds
        Card idle time 431 days, 9561 hours, 572664 minutes, 34359774
seconds
        Voltage status: 3.3V Nominal
EEPROM contents, slot 5/0:
        Hardware Revision        : 1.0
        Part Number              : 800-05547-05
        Board Revision           : A0
        Deviation Number         : 0-0
        Fab Version              : 03
        PCB Serial Number        : CAB0609MT72
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Board Revision           : AA
        Unknown Field (type 0036): 4B 
        Unknown Field (type 004C): 58 43 
        Hardware Revision        : 66.255
LCMON version, slot 5/0
        LCDOS (C10000 Line Card MONitor Image : Release 20000427:084233)
        Built by richv at Thu Apr 27 08:44:08 2000.
        Reset reason 0xC0000000 (external hard reset).
Operational Image version, slot 5/0
        LCDOS (C10000 6 Port Channelized T3 Line Card Image :
DEVELOPMENT BUILD REL-v120_26_s_lc_throttle.lcdos-weekly
/vob/lcdos/obj-c10k-ct3 101) major version 1060698849.
        Built by leccese at Tue Aug 12 10:34:11 2003.

Slot/Subslot 6/0:
        6cht3-1 card, 6 ports
        Card is full slot size
        Card is analyzed 
        Card detected 1y26w ago
        Card uptime 644 days, 14333 hours, 859003 minutes, 51539640
seconds
        Card idle time 431 days, 9551 hours, 572663 minutes, 34359740
seconds
        Voltage status: 3.3V Nominal
EEPROM contents, slot 6/0:
        Hardware Revision        : 1.0
        Part Number              : 800-05547-05
        Board Revision           : A0
        Deviation Number         : 0-0
        Fab Version              : 03
        PCB Serial Number        : CAB0609MT78
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Board Revision           : AA
        Unknown Field (type 0036): 4B 
        Unknown Field (type 004C): 58 43 
        Hardware Revision        : 66.255
LCMON version, slot 6/0
        LCDOS (C10000 Line Card MONitor Image : Release 20000427:084233)
        Built by richv at Thu Apr 27 08:44:08 2000.
        Reset reason 0xC0000000 (external hard reset).
Operational Image version, slot 6/0
        LCDOS (C10000 6 Port Channelized T3 Line Card Image :
DEVELOPMENT BUILD REL-v120_26_s_lc_throttle.lcdos-weekly
/vob/lcdos/obj-c10k-ct3 101) major version 1060698849.
        Built by leccese at Tue Aug 12 10:34:11 2003.

POP-EDGE-RTR1#

And here is what I get with clogin:

nms-lab3$ clogin -c "sh diag" pop-edge-rtr1
pop-edge-rtr1
spawn telnet pop-edge-rtr1
Trying 192.168.67.60...
telnet: Unable to connect to remote host: Connection refused
spawn ssh -c 3des -x -l rancid pop-edge-rtr1
rancid at pop-edge-rtr1's password: 

          !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
          ! NO CHANGES TO THIS CONFIG ARE TO BE MADE   !
          ! WITHOUT ENGINEERING OR NOC TIER 3 SUPPORT! !
          !                                            !
          !  ** DO NOT DISABLE CEF ON THIS ROUTER **   !
          !                                            !
          !        TACACS LOGS WILL BE AUDITED!        !
          !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

POP-EDGE-RTR1>enable
Password: 
POP-EDGE-RTR1#
POP-EDGE-RTR1#term length 0
POP-EDGE-RTR1#sh diag
Slot A:
        Primary PRE card
RP EEPROM contents:
        Hardware Revision        : 1.0
        Part Number              : 73-7041-03
        Board Revision           : C0
        Deviation Number         : 0-0
        Fab Version              : 02
        PCB Serial Number        : CAT07230QB1
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Part Number              : 800-17437-06
        CLEI Code                : CNRUGMVBAA
FP EEPROM contents:
        Hardware Revision        : 1.0
        Part Number              : 73-7042-06
        Board Revision           : A0
        Deviation Number         : 0-0
        Fab Version              : 05
        PCB Serial Number        : CAT07230Q6X
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
Operational Image Version, Slot A
        Cisco Internetwork Operating System Software 
IOS (tm) 10000 Software (C10K-K4P10-M), Experimental Version
12.0(20040113:203533) [REL-v120_26_sz_throttle.ios-weekly 100]
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Tue 13-Jan-04 15:35 by leccese


Slot B:
        Secondary PRE card
RP EEPROM Contents:
        Hardware Revision        : 1.0
        Part Number              : 73-7041-03
        Board Revision           : C0
        Deviation Number         : 0-0
        Fab Version              : 02
        PCB Serial Number        : CAT07230S3K
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Part Number              : 800-17437-06
        CLEI Code                : CNRUGMVBAA
FP EEPROM Contents:
        Hardware Revision        : 1.0
        Part Number              : 73-7042-06
        Board Revision           : A0
        Deviation Number         : 0-0
        Fab Version              : 05
        PCB Serial Number        : CAT07230RW3
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
Standby version: 
Cisco Internetwork Operating System Software 
IOS (tm) 10000 Software (C10K-K4P10-M), Experimental Version
12.0(20040113:203533) [REL-v120_26_sz_throttle.ios-weekly 100]
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Tue 13-Jan-04 15:35 by leccese


Slot/Subslot 1/0:
        1oc12atm-1 card, 1 port
        Card is full slot size
        Card is analyzed 
        Card detected 1y26w ago
        Card uptime 710 days, 16722 hours, 1002178 minutes, 60129567
seconds
        Card idle time 523 days, 11936 hours, 715883 minutes, 42949683
seconds
        Voltage status: 3.3V Nominal 2.5V Nominal
EEPROM contents, slot 1/0:
        Hardware Revision        : 1.0
        Part Number              : 800-06568-03
        Board Revision           : D0
        Deviation Number         : 0-0
        Fab Version              : 02
        PCB Serial Number        : CAT072504KB
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        CLEI Code                : BAA8LKDCAA
LCMON version, slot 1/0
        LCDOS (C10000 PowerQUICC-II Line Card MONitor Image : Release
branch:c10k_lc_bgp_reorg 20011120:145454)
        Built by richv at Tue Nov 20 17:04:22 2001.
        Reset reason 0x00000003 (external hard reset).
Operational Image version, slot 1/0
        LCDOS (C10000 1 Port OC12 ATM Line Card Image : DEVELOPMENT
BUILD REL-v120_26_s_lc_throttle.lcdos-weekly /vob/lcdos/obj-c10k-oc12atm
101) major version 1060699281.
        Built by leccese at Tue Aug 12 10:41:22 2003.
        ECC 1 bit errors since last cleared (dd hh mm ss) = 0 (0 days, 0
hours, 0 minutes, 0 seconds)
        ECC 1 bit errors while up (total) = 0
        

Slot/Subslot 3/0:
        1oc12pos-1 card, 1 port
        Card is full slot size
        Card is analyzed 
        Card detected 1y26w ago
        Card uptime 713 days, 16707 hours, 1002162 minutes, 60129584
seconds
        Card idle time 589 days, 13137 hours, 787419 minutes, 47244641
seconds
        Voltage status: 3.3V Nominal
EEPROM contents, slot 3/0:
        Hardware Revision        : 2.0
        Part Number              : 800-05546-03
        Board Revision           : F0
        Deviation Number         : 0-0
        Fab Version              : 03
        PCB Serial Number        : CAT072209MG
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
LCMON version, slot 3/0
        LCDOS (C10000 Line Card MONitor Image : Release 20000427:084233)
        Built by richv at Thu Apr 27 08:44:08 2000.
        Reset reason 0xC0000000 (external hard reset).
Operational Image version, slot 3/0
        LCDOS (C10000 1 Port OC12 POS Line Card Image : DEVELOPMENT
BUILD REL-v120_26_s_lc_throttle.lcdos-weekly /vob/lcdos/obj-c10k-oc12pos
101) major version 1060698969.
        Built by leccese at Tue Aug 12 10:36:10 2003.

Slot/Subslot 4/0:
        1oc12pos-1 card, 1 port
        Card is full slot size
        Card is analyzed 
        Card detected 1y26w ago
        Card uptime 713 days, 16707 hours, 1002164 minutes, 60129593
seconds
        Card idle time 588 days, 13128 hours, 787440 minutes, 47244657
seconds
        Voltage status: 3.3V Nominal
EEPROM contents, slot 4/0:
        Hardware Revision        : 2.0
        Part Number              : 800-05546-03
        Board Revision           : F0
        Deviation Number         : 0-0
        Fab Version              : 03
        PCB Serial Number        : CAT072209MS
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
LCMON version, slot 4/0
        LCDOS (C10000 Line Card MONitor Image : Release 20000427:084233)
        Built by richv at Thu Apr 27 08:44:08 2000.
        Reset reason 0xC0000000 (external hard reset).
Operational Image version, slot 4/0
        LCDOS (C10000 1 Port OC12 POS Line Card Image : DEVELOPMENT
BUILD REL-v120_26_s_lc_throttle.lcdos-weekly /vob/lcdos/obj-c10k-oc12pos
101) major version 1060698969.
        Built by leccese at Tue Aug 12 10:36:10 2003.

Slot/Subslot 5/0:
        6cht3-1 card, 6 ports
        Card is full slot size
        Card is analyzed 
        Card detected 1y26w ago
        Card uptime 644 days, 14333 hours, 859018 minutes, 51539657
seconds
        Card idle time 431 days, 9560 hours, 572720 minutes, 34359759
seconds
        Voltage status: 3.3V Nominal
EEPROM contents, slot 5/0:
        Hardware Revision        : 1.0
        Part Number              : 800-05547-05
        Board Revision           : A0
        Deviation Number         : 0-0
        Fab Version              : 03
        PCB Serial Number        : CAB0609MT72
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Board Revision           : AA
        Unknown Field (type 0036): 4B 
        Unknown Field (type 004C): 58 43 
        Hardware Revision        : 66.255
LCMON version, slot 5/0
        LCDOS (C10000 Line Card MONitor Image : Release 20000427:084233)
        Built by richv at Thu Apr 27 08:44:08 2000.
        Reset reason 0xC0000000 (external hard reset).
Operational Image version, slot 5/0
        LCDOS (C10000 6 Port Channelized T3 Line Card Image :
DEVELOPMENT BUILD REL-v120_26_s_lc_throttle.lcdos-weekly
/vob/lcdos/obj-c10k-ct3 101) major version 1060698849.
        Built by leccese at Tue Aug 12 10:34:11 2003.

Slot/Subslot 6/0:
        6cht3-1 card, 6 ports
        Card is full slot size
        Card is analyzed 
        Card detected 1y26w ago
        Card uptime 644 days, 14333 hours, 858997 minutes, 51539619
seconds
        Card idle time 431 days, 9550 hours, 572718 minutes, 34359783
seconds
        Voltage status: 3.3V Nominal
EEPROM contents, slot 6/0:
        Hardware Revision        : 1.0
        Part Number              : 800-05547-05
        Board Revision           : A0
        Deviation Number         : 0-0
        Fab Version              : 03
        PCB Serial Number        : CAB0609MT78
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Board Revision           : AA
        Unknown Field (type 0036): 4B 
        Unknown Field (type 004C): 58 43 
        Hardware Revision        : 66.255
LCMON version, slot 6/0
        LCDOS (C10000 Line Card MONitor Image : Release 20000427:084233)
        Built by richv at Thu Apr 27 08:44:08 2000.
        Reset reason 0xC0000000 (external hard reset).
Operational Image version, slot 6/0
^Cnms-lab3$ 

Does anyone have any ideas on how to troubleshoot this further?

Thanks in Advance!
Dave

___________________________________________________________
David A. Mack                           (703) 295-4398 (W)
CCIE #6963 JNCIS                        (703) 431-7617 (C)
email:  dmack at gnilink.net               (888) 631-0336 (P)
Text:   mackd at myairmail.com
___________________________________________________________
"For Those About to Route, We Salute You!"




More information about the Rancid-discuss mailing list