[rancid] control_rancid slow start

[Robert Drake]

On 11/12/2014 8:23 PM, heasley wrote:
>> If I comment the following code out it runs in less than 3 seconds:
>>
>> # check for 'up' routers missing in RCS.  no idea how this happens to
>> some folks
>> for router in `cut -d\; -f1 ../routers.up` ; do
>>       if [ $RCSSYS = cvs ] ; then
>>       cvs status $router | grep -i 'status: unknown' > /dev/null 2>&1
>>       else
>>       svn status $router | grep '^?' > /dev/null 2>&1
>>       fi
>>       if [ $? -eq 0 ] ; then
>>       touch $router
>>       if [ $RCSSYS = cvs ] ; then
>>           cvs add -ko $router
>>       else
>>           svn add $router
>>       fi
>>       echo "$RCSSYS added missing router $router"
>>       fi
>> done
>>
>> Possible better option would be this (I think this will work with svn
>> but I don't have a tree to test it on):
>>
>>    cut -d: -f1 ../routers.up | xargs cvs status | grep -i 'status: unknown'
>>
>> Example test case:
>>
>>    (echo test ; cut -d: -f1 ../routers.up) | xargs cvs status | grep -i
>> 'status: unknown'
>> cvs status: nothing known about test
>> File: no file test              Status: Unknown
> that doesnt quite work for non-existent files.
The test case shows it does work for non-existent files.  "test" is a 
non-existent file.  Unless I'm mistaken about what you mean by that.  
Depending on where the file is non-existent, you can grep two different 
ways.

File exists on disk in the configs/ dir but it's not in CVS:

       cut -d\; -f1 ../routers.up | xargs cvs status 2>&1 | grep 'Status: Unknown' | awk '{print $2}' | xargs cvs add -ko

Router name exists is routers.up, file does not exist in configs/ dir or in CVS:

       cut -d\; -f1 ../routers.up | xargs cvs status 2>&1 | grep 'cvs status: nothing' | awk '{print $6}' | xargs cvs add -ko

If you need to do two operations, touch + cvs add then you've got a 
choice.  You could put the output of the pipe into a temp file, then run 
cat $TEMPFILE | xargs touch && cat $TEMPFILE | xargs cvs add -ko, or you 
could take advantage of magic xargs flags to do something like:

       cut -d\; -f1 ../routers.up | xargs cvs status 2>&1 | grep 'cvs status: nothing' | awk '{print $6}' | xargs -I % sh -c 'touch %; cvs add -ko %'

Rather than going this route I think I would have two pipelines. The 
first part dumps into a tempfile based on what RCS is running, and the 
second part (touch/add/commit) is also broken out with a second case 
statement that tells it what to do.

In almost every case in control_rancid for loops are going to be slower 
than a pipeline due to the nature of shell scripting.  In most cases it 
doesn't matter because the potential work is not that high, but in the 
cases where you might need to run a cvs command 1000 times, it's much 
better to run it once or twice on a long list of files.