[rancid] Cisco IOS-XR Help - End of run not found

[Alan McKinnon]

On 13/06/2016 02:59, Asad wrote:
> Hi All,
>
> We are trying to add a few ASR9000 IOS-XR devices to Rancid 3.4.1. My
> issue seems similar
> to http://www.shrubbery.net/pipermail/rancid-discuss/2014-May/007597.html,
> I have un-commented NOPIPE=YES; export NOPIPE in rancid.conf and I have
> also tried to uncomment some of the show commands but it is still not
> working. To isolate any IOS-XR configuration issues I deployed a XRv
> with very basic configuration and I get same "end of run not found"
> error message.
>
> Any help would be appreciated.
>
> rancid at jp-script/opt/rancid#rancid -d -t cisco-xr lab-xrv-01
> loadtype: device type cisco-xr
> loadtype: found device type cisco-xr in /opt/rancid/etc/rancid.types.base
> executing clogin -t 90 -c"terminal exec prompt no-timestamp;admin show
> version;admin show install summary;admin show license udi;admin show
> license;admin show variables boot;admin show hw-module fpd location
> all;show redundancy;show install active;admin show diag;admin show
> inventory raw;show debug;show rpl maximum;admin show running;show
> running-config" lab-xrv-01
> PROMPT MATCH: RP/0/0/CPU0:lab-xrv-01#
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#terminal exec prompt no-timestamp
>      In RunCommand: RP/0/0/CPU0:lab-xrv-01#terminal exec prompt no-timestamp
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#admin show version
>      In ShowVersion: RP/0/0/CPU0:lab-xrv-01#admin show version
> TYPE = XRv Series
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#admin show install summary
>      In ShowInstallSummary: RP/0/0/CPU0:lab-xrv-01#admin show install
> summary
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#admin show license udi
>      In ShowLicense: RP/0/0/CPU0:lab-xrv-01#admin show license udi
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#admin show license
>      In ShowLicense: RP/0/0/CPU0:lab-xrv-01#admin show license
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#admin show variables boot
>      In ShowBootVar: RP/0/0/CPU0:lab-xrv-01#admin show variables boot
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#admin show hw-module fpd location all
>      In ShowRunning: RP/0/0/CPU0:lab-xrv-01#admin show hw-module fpd
> location all
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#show redundancy
>      In ShowRedundancy: RP/0/0/CPU0:lab-xrv-01#show redundancy
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#show install active
>      In ShowInstallActive: RP/0/0/CPU0:lab-xrv-01#show install active
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#admin show diag
>      In ShowDiag: RP/0/0/CPU0:lab-xrv-01#admin show diag
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#admin show inventory raw
>      In ShowInventory: RP/0/0/CPU0:lab-xrv-01#admin show inventory raw
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#show debug
>      In ShowDebug: RP/0/0/CPU0:lab-xrv-01#show debug
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#show rpl maximum
>      In ShowRPL: RP/0/0/CPU0:lab-xrv-01#show rpl maximum
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#admin show running
>      In ShowRunning: RP/0/0/CPU0:lab-xrv-01#admin show running
> HIT COMMAND:RP/0/0/CPU0:lab-xrv-01#show running-config
>      In WriteTerm: RP/0/0/CPU0:lab-xrv-01#show running-config
> lab-xrv-01: End of run not found
> lab-xrv-01: End of run not found
> end


This is an FAQ. The truth is that "End of run not found" is about as 
meaningful as "General exception error 1: Something went wrong." You 
need to get out the big debugging guns, or at least get your hands dirty.

In brief, rancid works like this: logs into a device, enables, and runs 
a sequence of commands[1][2] on the device. It captures all the output 
and starts to process it in perl, line by line.

The first major action is to find the string that defines the router's 
CLI prompt, then verify that the echoed command is the next expected one 
in the sequence printing "HIT COMMAND" for each one. If so, output is 
passed to a dedicated function for processing. And so on down the list 
of commands, each one checked for a prompt and the correct echo'ed command.

This can go wrong in many ways, mostly influenced by the phases of the 
moon and whatever Cisco felt like releasing in today's image. If a 
function does not end cleanly at the expected point, the wrong piece of 
running code processes everything thereafter and the rancid script do 
not exit cleanly, giving your error. If something unexpected happens in 
a multi-line chunk of output config, a loop may not exit cleanly giving 
the same result.[3]

In your case, the prompt was detected correctly and almost all router 
commands processed correctly. IIRC "show run" is the last one and it's 
not exiting cleanly per your output. It advise these steps first:

- look carefully at the temporary router output file right at the end 
where "show run" ends. Does the command exit correctly with the usual 
prompt followed by nothing just like all the other commands above it? 
That would indicate everything was processed correctly except the very 
last step.
- Something I've had to do a few times is find the main() loop in the 
perl code where it reads output line by line and insert a print 
statement to print the line just read to the screen then throw rancid at 
it. See what's the last line processed and let logic determine your next 
step.
- make sure you have the correct device type configured for that device.
I always found XR to be especially tricky - close enough to old-school 
IOS to look like it might work and different enough to need it's own 
codebase.


[1] In the rancid-2 series this was defined in @commandtable. The 3 
series may be different, but it must still exist and must be in an array 
somewhere.
[2] NOPIPE is not relevant here. It simply defines whether the router's 
output is saved to a disk file then read from there, or if it is piped 
directly into the next piece of running perl code. 15 years ago, server 
RAM may have made this important but nowadays not at all. The setting 
only serves to give you a permanent disk file or not.
[3] None of this is rancid's fault, it's trying to generate highly 
structured output from what too often seems like random free-form text.