Very strange problem with 2.2.2 (and 2.3-rc1) on FreeBSD-5.x

[Jeff Aitken]

Hello all,

I'm seeing a very strange problem with rancid-2.2.2 (and 2.3-rc1).
I've been talking privately with some of the developers but this has
all of us stumped.  Part of the problem is that no one has managed to
duplicate the symptoms I'm seeing, so I'm I'm sending this to the
-discuss list in the hopes that someone out there has seen something
similar.  

I recently installed rancid-2.2.2 on a freshly-installed FreeBSD-5.1
system after installing tcl83 and expect (5.38.0) from /usr/ports.
After setting up a small group of (mostly) Cisco 26xx and 72xx
routers to be polled, on subsequent runs of rancid I saw oscillating
sets of diffs like these:

RUN1:
    [ on router1 ]                                                              
    - snmp-server contact <email>                                  
    + snmp-server contaact <email>                                

    [on router2]                                                                
    - tacacs-server host 644.124.X.Y                                            
    + tacacs-server host 64.124.X.Y                                             

    [on router3]                                                                
    - taacacs-server host <foo>                            
    + tacacs-server host <foo>                                                  

RUN2:
    [on router1]                                                                
    - snmp-server contaact <foo>                                                
    + snmp-server contact <foo>                                                 

    [on router2]                                                                
    - tacacs-server host 64.124.X.Y                                             
    + tacacs-server host 644.124.X.Y                                            

    [on router3]                                                                
    - tacacs-server host <foo>                                                  
    + taacacs-server host <foo>                                                 

Again, this is a stock FreeBSD-5.1-RELEASE system with a GENERIC 
kernel and very little in the way of additional software added... a
couple things like apache and MRTG are about it.

I was unable to duplicate this behavior by running:

    % clogin -c 'wr term' $router

no matter how many times I tried.  I also tried running 'rancid -d'
in case it had something to do with the sequence of commands run
by rancid and not just the 'wr term' itself, but did not see the
same problem.  I tried setting NOPIPE in rancid.conf but still saw
the duplicate characters.  I played with PAR_COUNT, to no avail.
Thinking that it was something particular to the version of tcl
and/or expect we're using, I tried several different combinations
of those (including the latest, tcl8.4.5 and expect 5.40.0) but got
the same results.

I then loaded 2.2.2 on a freshly-installed FreeBSD-5.2-RELEASE
system and got the same sort of behavior.  Again, I tried different
versions of everything (2.3-rc1, newer and older tcl, newer and
older expect) but none of those made a difference.  I also changed
the set of routers I was polling to one that includes Cisco GSRs
and Juniper M160s and saw the same sort of thing on both types of
router, ruling out anything vendor-specific.

After all of this, I loaded 2.3-rc1 on the system we currently use
to poll the routers (an ancient FreeBSD-3.4 system) and it works as
expected, with no duplicate characters.  At this point I don't know
what to think... it could be a 5.x issue, or a problem with newer
tcl+expect, or even rancid itself, althouth that seems unlikely.  

Thanks for any suggestions,

--Jeff