[rancid] Improving Rancid's processing speed when having 1k+ devices

[john heasley]

Fri, Jul 26, 2019 at 02:34:49AM -0700, Florin Vlad Olariu:
> On 25 July 2019 at 18:16:48, Scott Granados
> (scott.granados at gmail.com(mailto:scott.granados at gmail.com)) wrote:
> 
> > I would also recommend running multiple rancid servers maybe scatter them geographically so it’s not a single machine pulling all the weight. Break the work loads up among them.
> 
> Great advice which didn't cross my mind. Might have to resort to this
> if I want ~ 1m poll times.

topologically close servers can help, but I would just run more processes
instead.  less mgmt overhead.

> > - make sure that the rancid user is not process rlimited to less than ~605
> processes; or PAR_COUNT * 2 + 5 or so.
> 
> My `ulimit -u` gives "4096". I don't this this is a factor?

unlikely.  make sure its not others; -n -d.  you'd see processes being
killed in the logs

...

Are your configs very large?  I have one group of 252 devices that are
scattered around the global totaling 1.2G of on-disk rancid output which
takes about 28m to collect with 16 processes.