[rancid] Re: Rancid output "Binary files differ"

[john heasley]

Wed, Nov 25, 2009 at 10:46:42PM +0000, ?rni Birgisson:
> Hi John, thanks for your reply.
> 
> > 
> > not likely.  do you have a fsys or memory problem on this host?  broken
> > expect/tcl?  some other unintentional change to rancid?
> 
> I don't believe there is any problem with the server itself, no errors in dmesg concerning
> hard disks or filesystems (mirrored disks), and 6GB free memory of total 8GB.
> Very little load on the server, and other processes running on it are working fine.
> 
> expect and tcl are newest versions from YUM,
> -- snip --
> Package 1:tcl-8.5.6-6.fc11.x86_64 already installed and latest version
> Package expect-5.43.0-17.fc11.x86_64 already installed and latest version
> -- snip --
> 
> And this behaviour is not constant, I have seen it from time to time with other routers, but
> most of the time it's this one. Still not always on this one either - usually the backups come
> through correct.

that'd lead me to believe its timing-based and most likely related to
expect reading portions of the input buffer, but thats a WAG.  could
be a number of things.

the easiest thing to try is get the lastest tcl and expect, apply the
expect patch from the rancid website and compile them yourself.

> - Arni
> 
> 
> On Nov 25, 2009, at 7:41 PM, john heasley wrote:
> 
> > Wed, Nov 25, 2009 at 11:39:20AM +0000, ?rni Birgisson:
> >> Hi guys,
> >> 
> >> I'm wondering if anybody has seen this problem before.
> >> This mail is going to be a bit long, starting with a simple description of the problem,
> >> and then my diagnosis of it.
> >> 
> >>> From time to time I'm getting a strange output from RANCID:
> >> 
> >> -----
> >> 
> >> Index: configs/router-x
> >> ===================================================================
> >> retrieving revision 1.88
> >> diff -U 4 -r1.88 router-x
> >> Binary files /tmp/cvsQSyJ8S and router-x differ
> >> 
> >> -----
> >> 
> >> As I noticed that the revision number was quite high I looked around and it seems that this
> >> does happen for more than one router - but happens very often to this one.
> >> Most of the network equipment has had about 30-40 revisions since RANCID was installed
> >> but this one has 88 revisions.
> >> 
> >> If I check a few versions of the config out of CVS I can see that indeed some of the files are weird.
> >> 
> >> -bash-4.0$ cvs diff -r1.86 -r1.89
> >> --normal diff output showing config changes--
> >> -bash-4.0$ cvs diff -r1.86 -r1.87
> >> --normal diff output showing config changes--
> >> -bash-4.0$ cvs diff -r1.86 -r1.88
> >> cvs diff: Diffing pe
> >> cvs diff: Diffing pe/configs
> >> Index: pe/configs/router-x
> >> ===================================================================
> >> RCS file: /var/rancid/CVS/pe/configs/router-x,v
> >> retrieving revision 1.86
> >> retrieving revision 1.88
> >> diff -r1.86 -r1.88
> >> Binary files /tmp/cvsCqpVfB and /tmp/cvsp09kv5 differ
> >> 
> >> Okay, looks like version 1.88 is the weird one.
> >> 
> >> Let's fetch 1.88 and 1.89
> >> -bash-4.0$ cvs co -r1.88 pe/configs/router-x
> >> U pe/configs/router-x
> >> -bash-4.0$ cp pe/configs/router-x v1.88
> >> -bash-4.0$ cvs co -r1.89 pe/configs/router-x
> >> U pe/configs/router-x
> >> -bash-4.0$ cp pe/configs/router-x v1.89
> >> 
> >> -bash-4.0$ ls -al v1.8*
> >> -rw-r--r-x 1 rancid rancid 70556 2009-11-25 05:28 v1.88
> >> -rw-r--r-x 1 rancid rancid 70556 2009-11-25 05:28 v1.89
> >> -bash-4.0$ diff v1.88 v1.89
> >> Binary files v1.88 and v1.89 differ
> >> -bash-4.0$ 
> >> 
> >> --- same size of the files, but diff won't compare them because they are (or one of them is) a binary file.
> >> 
> >> -bash-4.0$ strings v1.88 | wc -l
> >> 1387
> >> -bash-4.0$ strings v1.89 | wc -l
> >> 1542
> >> 
> >> Now when I have a look at the files it seems that the smaller file (v1.88 and "binary") starts with a lot of \0 bytes,
> >> it would seem one for each character that is missing - and then continues exactly like the other one.
> >> Or should I say the files are exactly the same - but it's as if the first (1542 - 1387) bytes of v1.88 are 0-byte instead
> >> of the character that should be there.
> >> 
> >> Diff --text on them shows that v1.89 includes all the lines from v1.88 and more and there has been no real
> >> change to the config.
> >> 
> >> I'm running rancid version 2.3.2, with a slight change to jrancid:
> >> The change to jrancid is to make it backup a "show configuration | display set" version of the config as well as the
> >> normal "show configuration" and is done by adding one function:
> >> 
> >> sub ShowConfigurationDisplaySet {
> >>    print STDERR "    In ShowConfigurationDisplaySet: $_" if ($debug);
> >> 
> >>    s/^[a-z]+@//;
> >>    ProcessHistory("","","","# $_");
> >>    while (<INPUT>) {
> >>        tr/\015//d;
> >>        last if (/^$prompt/);
> >>        next if (/^system (shutdown message from|going down )/i);
> >>        next if (/^\{(master|backup)}/);
> >> 
> >>        /Unrecognized command/ && return;
> >>        /command is not valid/ && return;
> >>        /^\s+\^/ && return;
> >>        /syntax error/ && return;
> >> 
> >>        ProcessHistory("","","","$_");
> >>    }
> >>    return(0);
> >> }
> >> 
> >> 
> >> and one item to @commandtable:
> >>        {'show configuration | display set'     => 'ShowConfigurationDisplaySet'},
> >> 
> >> It could very well be that the errors are introduced because of my addition,
> >> but I fail to see how.
> > 
> > not likely.  do you have a fsys or memory problem on this host?  broken
> > expect/tcl?  some other unintentional change to rancid?
> 
> 
> 
> Arni Birgisson	
> Network Engineer
> FATTOC Iceland
> Adalstraeti 6
> 101 Reykjavik
> Iceland