[rancid] Improving Rancid's processing speed when having 1k+ devices

[Florin Vlad Olariu]

On 25 July 2019 at 18:16:48, Scott Granados
(scott.granados at gmail.com(mailto:scott.granados at gmail.com)) wrote:

> I would also recommend running multiple rancid servers maybe scatter them geographically so it’s not a single machine pulling all the weight. Break the work loads up among them.

Great advice which didn't cross my mind. Might have to resort to this
if I want ~ 1m poll times.

On 25 July 2019 at 17:55:31, john heasley (heas at shrubbery.net) wrote:

> - cvs is slow. use svn or git. svn is probably faster; but I have not
benchmarked the two for the functions that rancid uses.

I do use git already. Not sure git itself is to blame for the slowdown though.

> - make sure that the rancid user is not process rlimited to less than ~605
processes; or PAR_COUNT * 2 + 5 or so.

My `ulimit -u` gives "4096". I don't this this is a factor?

> - perl is a meory pig. if the host/vm has memory pressure, this would be
something to address.
> - retrieving device output does not require much cpu, but process does use
some - dont starve it

I have a Xeon 8-core box, and when running it with PAR_COUNT=400 it
runs to 50+ load, but only for a short period (the time it takes to
connect to devices) after ~ 2 minutes it goes back to normal, so I
don't think CPU is really the problem. Furthermore, I have 32G of ram,
and running `watch free -h` it does not look like rancid uses *that*
much memory, maybe ~ 5 G.

> - use rancid.conf:NOPIPE=YES; i think this is faster because perl is a pig.

Tried this, but no difference in time :(

> - if you only need configs, then reduce what is collected to just show version
and show running. or have one hourly group that collects that, and a daily
group that collects everything. less processing, and esp many fewer regexes.

I only need configs and the way rancid is configured already only
pools "show run" (or equivalent).

Seems only real solution might be to break down the amount of hosts
between different machines.

Thanks John.