[rancid] Intermittent Rancid Failures

[heasley]

Wed, Jun 18, 2025 at 11:22:23PM +0000, Dan Mahoney (Gushi):
> Hey there all,
> 
> Something's driving me batty.
> 
> My ASR-1001-X is only able to be connected to intermittently.  Rancid (run
> as the rancid user) always works from the command line, but rancid-run fails
> for some reason.
> 
> When I watch rancid-run, I see several ssh processes start up, trying to
> shell to the router in question, but of course, the output of those aren't
> logged anywhwere?  Clogin works.  Running all the commands in rancid -d work
> (though of course there are many extra commands in there).

There should only be 1 ssh process per device, though it will try
rancid.conf:MAX_ROUNDS times.

Much of the output is filtered, but effort is made to log relevant
errors to rancid.conf:${LOGDIR}/<group>.<datestamp>

It is possible that the device is simply slow executing some commands.
This is not unusual for older devices or because of bugs such as
memory leaks.  Increasing the timeout can test this theory, either
increase the timeout for all devices of type cisco,
rancid.types.base: cisco;timeout;120
or specific devices,
~rancid/.cloginrc: add timeout <name glob> {<seconds>}

> But every time I call rancid-run groupname, I get the "routers have not been
> contacted in over 24 hours" email.  And only intermittently.  (It's been a
> little over 24 hours with no changes now).

Another thing to check, which would also be revealed in the
aforemention logs, is that the repository is not buggered in
some manner that control_rancid can not resolve.
su - rancid
cd <group>
<SCM> update or <SCM> status
and look for errors.

Those are the things that I would investigate or try first.

> Rancid is rancid3-3.13_3 under FreeBSD 13.4
> 
> I notice that some of the commands that one might need to run these things
> from the command line get put into /usr/local/libexec/rancid/ which isn't in
> the default path, but this only breaks command-line testing, not actual
> production runs.  (Maybe the rancid-run script sets the right path?)

The PATH for the environment is set in rancid.conf, which should include
that directory.  You too can source that file if your sell is
shell-comapatible.  You have seen the ssh processes, so this is not your
problem.

> I'll also note that these routers need some special things in .ssh/config
> for some older ciphers, etc, but that would be an all-or-nothign type
> problem, I'd think.

that should not matter.