[rancid] Re: Rancid output "Binary files differ"

[Árni Birgisson]

Hi John, thanks for your reply.

> 
> not likely.  do you have a fsys or memory problem on this host?  broken
> expect/tcl?  some other unintentional change to rancid?

I don't believe there is any problem with the server itself, no errors in dmesg concerning
hard disks or filesystems (mirrored disks), and 6GB free memory of total 8GB.
Very little load on the server, and other processes running on it are working fine.

expect and tcl are newest versions from YUM,
-- snip --
Package 1:tcl-8.5.6-6.fc11.x86_64 already installed and latest version
Package expect-5.43.0-17.fc11.x86_64 already installed and latest version
-- snip --

And this behaviour is not constant, I have seen it from time to time with other routers, but
most of the time it's this one. Still not always on this one either - usually the backups come
through correct.

- Arni


On Nov 25, 2009, at 7:41 PM, john heasley wrote:

> Wed, Nov 25, 2009 at 11:39:20AM +0000, ?rni Birgisson:
>> Hi guys,
>> 
>> I'm wondering if anybody has seen this problem before.
>> This mail is going to be a bit long, starting with a simple description of the problem,
>> and then my diagnosis of it.
>> 
>>> From time to time I'm getting a strange output from RANCID:
>> 
>> -----
>> 
>> Index: configs/router-x
>> ===================================================================
>> retrieving revision 1.88
>> diff -U 4 -r1.88 router-x
>> Binary files /tmp/cvsQSyJ8S and router-x differ
>> 
>> -----
>> 
>> As I noticed that the revision number was quite high I looked around and it seems that this
>> does happen for more than one router - but happens very often to this one.
>> Most of the network equipment has had about 30-40 revisions since RANCID was installed
>> but this one has 88 revisions.
>> 
>> If I check a few versions of the config out of CVS I can see that indeed some of the files are weird.
>> 
>> -bash-4.0$ cvs diff -r1.86 -r1.89
>> --normal diff output showing config changes--
>> -bash-4.0$ cvs diff -r1.86 -r1.87
>> --normal diff output showing config changes--
>> -bash-4.0$ cvs diff -r1.86 -r1.88
>> cvs diff: Diffing pe
>> cvs diff: Diffing pe/configs
>> Index: pe/configs/router-x
>> ===================================================================
>> RCS file: /var/rancid/CVS/pe/configs/router-x,v
>> retrieving revision 1.86
>> retrieving revision 1.88
>> diff -r1.86 -r1.88
>> Binary files /tmp/cvsCqpVfB and /tmp/cvsp09kv5 differ
>> 
>> Okay, looks like version 1.88 is the weird one.
>> 
>> Let's fetch 1.88 and 1.89
>> -bash-4.0$ cvs co -r1.88 pe/configs/router-x
>> U pe/configs/router-x
>> -bash-4.0$ cp pe/configs/router-x v1.88
>> -bash-4.0$ cvs co -r1.89 pe/configs/router-x
>> U pe/configs/router-x
>> -bash-4.0$ cp pe/configs/router-x v1.89
>> 
>> -bash-4.0$ ls -al v1.8*
>> -rw-r--r-x 1 rancid rancid 70556 2009-11-25 05:28 v1.88
>> -rw-r--r-x 1 rancid rancid 70556 2009-11-25 05:28 v1.89
>> -bash-4.0$ diff v1.88 v1.89
>> Binary files v1.88 and v1.89 differ
>> -bash-4.0$ 
>> 
>> --- same size of the files, but diff won't compare them because they are (or one of them is) a binary file.
>> 
>> -bash-4.0$ strings v1.88 | wc -l
>> 1387
>> -bash-4.0$ strings v1.89 | wc -l
>> 1542
>> 
>> Now when I have a look at the files it seems that the smaller file (v1.88 and "binary") starts with a lot of \0 bytes,
>> it would seem one for each character that is missing - and then continues exactly like the other one.
>> Or should I say the files are exactly the same - but it's as if the first (1542 - 1387) bytes of v1.88 are 0-byte instead
>> of the character that should be there.
>> 
>> Diff --text on them shows that v1.89 includes all the lines from v1.88 and more and there has been no real
>> change to the config.
>> 
>> I'm running rancid version 2.3.2, with a slight change to jrancid:
>> The change to jrancid is to make it backup a "show configuration | display set" version of the config as well as the
>> normal "show configuration" and is done by adding one function:
>> 
>> sub ShowConfigurationDisplaySet {
>>    print STDERR "    In ShowConfigurationDisplaySet: $_" if ($debug);
>> 
>>    s/^[a-z]+@//;
>>    ProcessHistory("","","","# $_");
>>    while (<INPUT>) {
>>        tr/\015//d;
>>        last if (/^$prompt/);
>>        next if (/^system (shutdown message from|going down )/i);
>>        next if (/^\{(master|backup)}/);
>> 
>>        /Unrecognized command/ && return;
>>        /command is not valid/ && return;
>>        /^\s+\^/ && return;
>>        /syntax error/ && return;
>> 
>>        ProcessHistory("","","","$_");
>>    }
>>    return(0);
>> }
>> 
>> 
>> and one item to @commandtable:
>>        {'show configuration | display set'     => 'ShowConfigurationDisplaySet'},
>> 
>> It could very well be that the errors are introduced because of my addition,
>> but I fail to see how.
> 
> not likely.  do you have a fsys or memory problem on this host?  broken
> expect/tcl?  some other unintentional change to rancid?



Arni Birgisson	
Network Engineer
FATTOC Iceland
Adalstraeti 6
101 Reykjavik
Iceland