From afort at choqolat.org Mon Sep 2 12:16:20 2002 From: afort at choqolat.org (Andrew Fort) Date: Mon, 2 Sep 2002 22:16:20 +1000 Subject: Ignoring regular differential updates Message-ID: <200209022216.20206.afort@choqolat.org> An architectural question has come up in a relatively large deployment of RANCID I'm working on.. brief: You operate an SP network, and you have software (RtConfig in our case) pushing out regular configuration changes to your routers, primarily consisting of prefix lists/ACL and route-map/policy-statement changes for your BGP routers. How do you cope with these regular changes operationally with RANCID? Architectural staff would like to avoid seeing the ACL/pfx.list changes that occur when customer registers a route with your IRR, but you'd still like to know when ops folk accidentally smoke some important part of the router config. The thoughts which come to mind for me are: 1. (messy) munge the rancid output for the filter changes when you do a nightly RANCID run after the RtConfig changes, grepping out ACLs and prefix lists. you still get a diff with a whole bunch of 'empty' diff section headers, however. 2. run RANCID 'quietly', immediately before and after each router's successful run. using 2.2.2, you could create a mail alias that goes to /dev/null and do do-diffs -r box -m nullmail for each box. 3. hack up your own version of do-diffs/control_rancid to perform 2. without mailing anyone/creating diffs in the first place. Other approaches? Also, how many devices do people have RANCID scaling to? I'm looking at a daily run of around 350 devices, with about 40 BGP routers that need to be treated using one of the methods above. Sofar it's scaling very well and the overall runtime (~ 1 hour) beats our existing config collection system by about 80% (snmp triggered tftp dumps). cheers -amf From rancid at jnull.rackspace.com Mon Sep 2 13:46:59 2002 From: rancid at jnull.rackspace.com (jnull) Date: Mon, 2 Sep 2002 08:46:59 -0500 Subject: Ignoring regular differential updates In-Reply-To: <200209022216.20206.afort@choqolat.org> References: <200209022216.20206.afort@choqolat.org> Message-ID: <20020902134659.GA9416@jnull.rackspace.com> Andrew Fort(afort at choqolat.org)@2002.09.02 22:16:20 +0000: > How do you cope with these regular changes operationally with RANCID? > Architectural staff would like to avoid seeing the ACL/pfx.list changes that > occur when customer registers a route with your IRR, but you'd still like to > know when ops folk accidentally smoke some important part of the router > config. I've the same issue, save it is with static route entries for secondary IPs. Also, I don't care about saved configs. I asked a similar question, but was basically told that the current release doesn't support it. > 1. (messy) munge the rancid output for the filter changes when you do a Yes, messy should be the first and last word for this method. > 2. run RANCID 'quietly', immediately before and after each router's successful I think there is too much risk here, defeating a prime benefit of RANCID. > 3. hack up your own version of do-diffs/control_rancid to perform 2. without I've got this on my tuit list. As soon as I'm done hacking on a DoS det. app. > Other approaches? One could stay off "do-diffs" and just use clogin with your own commands and some easy shell scripts to parse the data--basically a shortcut to 3. > Also, how many devices do people have RANCID scaling to? I'm looking at a Yours it about equal to the largest I've heard of for a full implementation. Personally, I only use it for a subset of our devices for monitoring changes, mainly due to the issues you described above. However, for changing snmp strings or local passwords I'll use it across the board. -- > -amf Let me know if you opt on number 3, possibly we could QA each others work or swap some ideas. My time schedule puts it a few weeks out yet. -- sig=$header From afort at choqolat.org Mon Sep 2 14:48:27 2002 From: afort at choqolat.org (Andrew Fort) Date: Tue, 3 Sep 2002 00:48:27 +1000 Subject: Ignoring regular differential updates In-Reply-To: <20020902134659.GA9416@jnull.rackspace.com> References: <200209022216.20206.afort@choqolat.org> <20020902134659.GA9416@jnull.rackspace.com> Message-ID: <200209030048.27706.afort@choqolat.org> On Mon, 2 Sep 2002 23:46, jnull wrote: > I've the same issue, save it is with static route entries for > secondary IPs. Also, I don't care about saved configs. I asked a similar > question, but was basically told that the current release doesn't support > it. Yep, I remember your earlier post. Like I'd suspect most others, we want to use the saved configs for long term problem tracking across the whole network. I imagine you're interested in diffs of specific devices, minus one or two regularly changing bits. We'd like to ignore all changes made at a particular time in the day, as we run our major filter update only once daily. I'm guessing some NSPs that use RANCID deal with the matter operationally by having some person review the diffs; expected ACL changes, warts and all, and then bring/forward out-of-spec stuff to the architects. I'd like a meat-free approach to this.... > > 2. run RANCID 'quietly', immediately before and after each router's > > successful > > I think there is too much risk here, defeating a prime benefit of > RANCID. I think this is only an issue because you don't care about saved configurations. My thoughts go something along the lines of: 1. run a "regular" do-diffs immediately before you run your nightly routing maintenance job (that builds router configs and spits em out). 2. run your maintenance, updating router configs. 3. run a "quiet" do-diffs, so that configs are still in CVS, but the usual aliases aren't mailed with the diff output which will consist of the maintenance changes (which we dont care to see, but we'd like a record of incase they fail). You may have rogue operatives attempting to sneak config changes under your nose during this quiet diff, but 1. you still have the diffs in your CVS tree, and 2. you've got bigger problems to deal with if this is happening :). > > 3. hack up your own version of do-diffs/control_rancid to perform 2. > > without > > I've got this on my tuit list. As soon as I'm done hacking on a DoS > det. app. OT: If you publish your work/findings, drop me a line, working with netflow/etc data on attack analysis was an a challenging and enjoyable part of my work in my previous life (running a colo/hosting farm similar to rackspace) and I'm interested in all efforts and research in this area. > However, > for changing snmp strings or local passwords I'll use it across the board. I've been using a combination of scripts to do this, including pancho, but find rancid *login useful for its cross platform capabilty and scripting flexibility. > Let me know if you opt on number 3, possibly we could QA each others work or > swap some ideas. > My time schedule puts it a few weeks out yet. Right now, I'm learning towards a silent diff after the routing update due to my requirements (i.e., I'm looking filtering alot of stuff out of the diff, rather than just a little). The difference between exclusive and inclusive route filtering, I suppose, and likely just as religious an argument :-) -amf From heas at shrubbery.net Tue Sep 3 05:51:01 2002 From: heas at shrubbery.net (john heasley) Date: Tue, 3 Sep 2002 05:51:01 +0000 Subject: Ignoring regular differential updates In-Reply-To: <200209022216.20206.afort@choqolat.org>; from afort@choqolat.org on Mon, Sep 02, 2002 at 10:16:20PM +1000 References: <200209022216.20206.afort@choqolat.org> Message-ID: <20020903055101.A1305@shrubbery.net> Mon, Sep 02, 2002 at 10:16:20PM +1000, Andrew Fort: > An architectural question has come up in a relatively large deployment of > RANCID I'm working on.. > > brief: You operate an SP network, and you have software (RtConfig in our case) > pushing out regular configuration changes to your routers, primarily > consisting of prefix lists/ACL and route-map/policy-statement changes for > your BGP routers. > > How do you cope with these regular changes operationally with RANCID? > Architectural staff would like to avoid seeing the ACL/pfx.list changes that > occur when customer registers a route with your IRR, but you'd still like to > know when ops folk accidentally smoke some important part of the router > config. > > The thoughts which come to mind for me are: > > 1. (messy) munge the rancid output for the filter changes when you do a > nightly RANCID run after the RtConfig changes, grepping out ACLs and prefix > lists. you still get a diff with a whole bunch of 'empty' diff section > headers, however. > > 2. run RANCID 'quietly', immediately before and after each router's successful > run. using 2.2.2, you could create a mail alias that goes to /dev/null and > do > > do-diffs -r box -m nullmail > > for each box. > > 3. hack up your own version of do-diffs/control_rancid to perform 2. without > mailing anyone/creating diffs in the first place. > > Other approaches? when someone bitched about nvram changes, i cooked up the following procmail recipe for them. perhaps you can adapt it for your needs. :0 H * ^Subject: .* config diffs * !^X-RLOOP: rancid #| egrep -v '\!Flash: nvram:' | formail -A "X-RLOOP: rancid" -s procmail | sed -e '/private-config/n' -e '/\!Flash: nvram:/d' | formail -A "X-RLOOP: rancid" -s procmail # /dev/null rancid msgs which have no diffs other than nvram: :0 HWi b * Subject: .* router config diffs$ | awk 'BEGIN{n=0;}{if(/...Flash: nvram:/)next; if(/^[-+]/){n++;}next;}END{exit n;}' i like things like these; it allows an individual see what they wish and preserves the whole of what rancid collects. > Also, how many devices do people have RANCID scaling to? I'm looking at a > daily run of around 350 devices, with about 40 BGP routers that need to be > treated using one of the methods above. Sofar it's scaling very well and the > overall runtime (~ 1 hour) beats our existing config collection system by > about 80% (snmp triggered tftp dumps). i've seen ~750 devices in < 1 hour, almost all bgp speakers. From afort at choqolat.org Tue Sep 3 08:55:10 2002 From: afort at choqolat.org (Andrew Fort) Date: Tue, 3 Sep 2002 18:55:10 +1000 Subject: Ignoring regular differential updates In-Reply-To: <20020903055101.A1305@shrubbery.net> References: <200209022216.20206.afort@choqolat.org> <20020903055101.A1305@shrubbery.net> Message-ID: <200209031855.10725.afort@choqolat.org> On Tue, 3 Sep 2002 15:51, john heasley wrote: > when someone bitched about nvram changes, i cooked up the following > procmail recipe for them. perhaps you can adapt it for your needs. > > i like things like these; it allows an individual see what they wish and > preserves the whole of what rancid collects. lovely, this type of approach fits our requirements quite nicely. thanks. -amf From pbelang1 at oss.cantel.rogers.com Fri Sep 6 23:13:57 2002 From: pbelang1 at oss.cantel.rogers.com (Pierre Belanger) Date: Fri, 06 Sep 2002 19:13:57 -0400 Subject: F5 support? Message-ID: <3D7936B5.49A9BC84@oss.cantel.rogers.com> Does RANCID support F5 Network stuff, like the BigIP? I don't have access to the F5 documentation right now... perhaps it's "compliant" with something "already" supported in RANCID? Thank you, Pierre B. From heas at shrubbery.net Fri Sep 6 23:29:46 2002 From: heas at shrubbery.net (john heasley) Date: Fri, 6 Sep 2002 23:29:46 +0000 Subject: F5 support? In-Reply-To: <3D7936B5.49A9BC84@oss.cantel.rogers.com>; from pbelang1@oss.cantel.rogers.com on Fri, Sep 06, 2002 at 07:13:57PM -0400 References: <3D7936B5.49A9BC84@oss.cantel.rogers.com> Message-ID: <20020906232946.A25121@shrubbery.net> Fri, Sep 06, 2002 at 07:13:57PM -0400, Pierre Belanger: > Does RANCID support F5 Network stuff, like the BigIP? no. > I don't have access to the F5 documentation right now... > perhaps it's "compliant" with something "already" supported > in RANCID? it may; have never seen one.