[rancid] control_rancid slow start

[Charles T. Brooks]

Pretty much any time you call grep and awk in the same pipeline, there's a better, faster, easier way.

For example, this code:

grep 'cvs status: nothing' | awk '{print $6}' | xargs -I % sh -c 'touch %; cvs add -ko %'

looks like a complicated way to do this:

gawk '/cvs status: nothing/{system("touch " $6 ";cvs add -ko " $6)}'

...but I don't have or want CVS so unfortunately I can't test it.

Note you can do any number of operations in sequence (and conditionally dependent on each other, if you want) in that system() call without any need for temporary files. 

--Charlie

 Arnold Robbins:  AWK is a language similar to PERL, only considerably more elegant.
 Larry Wall:  Hey!

________________________________________
From: Rancid-discuss [rancid-discuss-bounces at shrubbery.net] on behalf of Robert Drake [rdrake at direcpath.com]
Sent: Tuesday, August 16, 2016 2:50 AM
To: heasley
Cc: rancid-discuss at shrubbery.net
Subject: Re: [rancid] control_rancid slow start

On 11/12/2014 8:23 PM, heasley wrote:
>> If I comment the following code out it runs in less than 3 seconds:
>>
>> # check for 'up' routers missing in RCS.  no idea how this happens to
>> some folks
>> for router in `cut -d\; -f1 ../routers.up` ; do
>>       if [ $RCSSYS = cvs ] ; then
>>       cvs status $router | grep -i 'status: unknown' > /dev/null 2>&1
>>       else
>>       svn status $router | grep '^?' > /dev/null 2>&1
>>       fi
>>       if [ $? -eq 0 ] ; then
>>       touch $router
>>       if [ $RCSSYS = cvs ] ; then
>>           cvs add -ko $router
>>       else
>>           svn add $router
>>       fi
>>       echo "$RCSSYS added missing router $router"
>>       fi
>> done
>>
>> Possible better option would be this (I think this will work with svn
>> but I don't have a tree to test it on):
>>
>>    cut -d: -f1 ../routers.up | xargs cvs status | grep -i 'status: unknown'
>>
>> Example test case:
>>
>>    (echo test ; cut -d: -f1 ../routers.up) | xargs cvs status | grep -i
>> 'status: unknown'
>> cvs status: nothing known about test
>> File: no file test              Status: Unknown
> that doesnt quite work for non-existent files.
The test case shows it does work for non-existent files.  "test" is a
non-existent file.  Unless I'm mistaken about what you mean by that.
Depending on where the file is non-existent, you can grep two different
ways.

File exists on disk in the configs/ dir but it's not in CVS:

       cut -d\; -f1 ../routers.up | xargs cvs status 2>&1 | grep 'Status: Unknown' | awk '{print $2}' | xargs cvs add -ko

Router name exists is routers.up, file does not exist in configs/ dir or in CVS:

       cut -d\; -f1 ../routers.up | xargs cvs status 2>&1 | grep 'cvs status: nothing' | awk '{print $6}' | xargs cvs add -ko

If you need to do two operations, touch + cvs add then you've got a
choice.  You could put the output of the pipe into a temp file, then run
cat $TEMPFILE | xargs touch && cat $TEMPFILE | xargs cvs add -ko, or you
could take advantage of magic xargs flags to do something like:

       cut -d\; -f1 ../routers.up | xargs cvs status 2>&1 | grep 'cvs status: nothing' | awk '{print $6}' | xargs -I % sh -c 'touch %; cvs add -ko %'

Rather than going this route I think I would have two pipelines. The
first part dumps into a tempfile based on what RCS is running, and the
second part (touch/add/commit) is also broken out with a second case
statement that tells it what to do.

In almost every case in control_rancid for loops are going to be slower
than a pipeline due to the nature of shell scripting.  In most cases it
doesn't matter because the potential work is not that high, but in the
cases where you might need to run a cvs command 1000 times, it's much
better to run it once or twice on a long list of files.

_______________________________________________
Rancid-discuss mailing list
Rancid-discuss at shrubbery.net
http://www.shrubbery.net/mailman/listinfo/rancid-discuss
       ------------------  CONFIDENTIALITY NOTICE  ---------------

  This message, including any attachments, is for the sole use of the
intended recipient(s) and may contain privileged confidential information
protected by law. Any unauthorized review, use, disclosure or distribution
of this message is prohibited. If you are not the intended recipient, please
contact the sender by reply e-mail and destroy all copies of this message.

       ------------------  CONFIDENTIALITY NOTICE  ---------------