From me at falz.net Thu Nov 12 13:19:36 2020 From: me at falz.net (Chris Wopat) Date: Thu, 12 Nov 2020 07:19:36 -0600 Subject: [rancid] Cisco devices and calculating percent free Message-ID: I recently updated to v3.13, I do not recall exactly what version I was on but it was >1 year old, and could've been HEAD based on some previous debugging of other issues. Anyhow, we have many IOS devices that calculate percent free and it oscillates: - !Flash: bootflash: 1 GB total (19% free) + !Flash: bootflash: 1 GB total (18% free) Without manually patching ios.pm (or wherever this takes place) is there any type of config setting to just turn off showing any info about how much disk space is free? We get this info properly from snmp trending systems. Just trying to cut down on router diffs where there were no actual config changes. Cheers, Chris From randy at psg.com Thu Nov 12 18:17:17 2020 From: randy at psg.com (Randy Bush) Date: Thu, 12 Nov 2020 10:17:17 -0800 Subject: [rancid] Cisco devices and calculating percent free In-Reply-To: References: Message-ID: something analogous to --- rancid.types.base.ORIG 2020-03-19 01:11:40.000000000 +0000 +++ rancid.types.base 2020-03-21 19:41:36.522622000 +0000 @@ -221,7 +221,7 @@ cisco-nx;command;nxos::ShowEnvPower;show environment power cisco-nx;command;nxos::ShowBoot;show boot cisco-nx;command;nxos::DirSlotN;dir bootflash: -cisco-nx;command;nxos::DirSlotN;dir debug: +## cisco-nx;command;nxos::DirSlotN;dir debug: cisco-nx;command;nxos::DirSlotN;dir logflash: cisco-nx;command;nxos::DirSlotN;dir slot0: cisco-nx;command;nxos::DirSlotN;dir usb1: @@ -598,7 +598,7 @@ junos;module;junos junos;inloop;junos::inloop junos;command;junos::ShowChassisClocks;show chassis clocks -junos;command;junos::ShowChassisEnvironment;show chassis environment +## junos;command;junos::ShowChassisEnvironment;show chassis environment junos;command;junos::ShowChassisFirmware;show chassis firmware local junos;command;junos::ShowChassisFirmware;show chassis firmware From heas at shrubbery.net Thu Nov 12 18:32:45 2020 From: heas at shrubbery.net (john heasley) Date: Thu, 12 Nov 2020 18:32:45 +0000 Subject: [rancid] Cisco devices and calculating percent free In-Reply-To: References: Message-ID: <20201112183245.GA1501@shrubbery.net> Thu, Nov 12, 2020 at 07:19:36AM -0600, Chris Wopat: > I recently updated to v3.13, I do not recall exactly what version I > was on but it was >1 year old, and could've been HEAD based on some > previous debugging of other issues. > > Anyhow, we have many IOS devices that calculate percent free and it oscillates: > > - !Flash: bootflash: 1 GB total (19% free) > + !Flash: bootflash: 1 GB total (18% free) > > Without manually patching ios.pm (or wherever this takes place) is > there any type of config setting to just turn off showing any info > about how much disk space is free? We get this info properly from snmp > trending systems. > > Just trying to cut down on router diffs where there were no actual > config changes. Except that something to writing to your flash...maybe you expect that. anyway, private devices types without the dir commands could be created in rancid.types.conf(5). or, i have almost finished an option for a user-defined per-device type postscript filter. eg: +# example use of the postscript directive. This will replace control +# characters in the output with an ASCII representation. +arbor-ctrl;script;rancid -t arbor-ctrl +arbor-ctrl;login;anlogin +arbor-ctrl;module;arbor +arbor-ctrl;inloop;arbor::inloop +arbor-ctrl;postscript;/bin/cat -v +arbor-ctrl;command;arbor::ShowVersion;system version +arbor-ctrl;command;arbor::ShowVersion;system file show +arbor-ctrl;command;arbor::ShowHardware;system hardware +arbor-ctrl;command;arbor::ShowHardware;system show +arbor-ctrl;command;arbor::ShowFlash;system disk show +arbor-ctrl;command;arbor::ShowConfig;config show From me at falz.net Thu Nov 12 19:20:38 2020 From: me at falz.net (Chris Wopat) Date: Thu, 12 Nov 2020 13:20:38 -0600 Subject: [rancid] Cisco devices and calculating percent free In-Reply-To: References: Message-ID: On Thu, Nov 12, 2020 at 12:17 PM Randy Bush wrote: > > something analogous to > > --- rancid.types.base.ORIG 2020-03-19 01:11:40.000000000 +0000 > +++ rancid.types.base 2020-03-21 19:41:36.522622000 +0000 > @@ -221,7 +221,7 @@ > cisco-nx;command;nxos::ShowEnvPower;show environment power > cisco-nx;command;nxos::ShowBoot;show boot > cisco-nx;command;nxos::DirSlotN;dir bootflash: > -cisco-nx;command;nxos::DirSlotN;dir debug: > +## cisco-nx;command;nxos::DirSlotN;dir debug: > cisco-nx;command;nxos::DirSlotN;dir logflash: > cisco-nx;command;nxos::DirSlotN;dir slot0: > cisco-nx;command;nxos::DirSlotN;dir usb1: > @@ -598,7 +598,7 @@ > junos;module;junos > junos;inloop;junos::inloop > junos;command;junos::ShowChassisClocks;show chassis clocks > -junos;command;junos::ShowChassisEnvironment;show chassis environment > +## junos;command;junos::ShowChassisEnvironment;show chassis environment > junos;command;junos::ShowChassisFirmware;show chassis firmware local > junos;command;junos::ShowChassisFirmware;show chassis firmware I had not yet tested this out, but it looks like it may remove the full directory listing from whichever disc it's churning on. I should have noted that I do have value in knowing the contents of flash within RANCID, just not the amount of free space, which I get via SNMP. I'll test above or heasley's suggestion when it comes out to confirm. Hell maybe I'll even spend a few minutes checking the cause of the issue (ie what's happening to fluctuate disk utilization) Cheers, --Chris From weylin at bu.edu Fri Nov 13 00:55:46 2020 From: weylin at bu.edu (Piegorsch, Weylin William) Date: Fri, 13 Nov 2020 00:55:46 +0000 Subject: [rancid] Cisco devices and calculating percent free In-Reply-To: References: Message-ID: <5C9140AB-FE15-4171-B9D7-FBF0376E114A@bu.edu> If I recall, end users should alter rancid.types.conf, not .base - or better yet, add a custom device type to rancid.types.conf and leave the defaults alone. weylin ?-----Original Message----- From: Randy Bush Date: Thursday, November 12, 2020 at 1:17 PM To: Chris Wopat Cc: Subject: Re: [rancid] Cisco devices and calculating percent free something analogous to --- rancid.types.base.ORIG 2020-03-19 01:11:40.000000000 +0000 +++ rancid.types.base 2020-03-21 19:41:36.522622000 +0000 @@ -221,7 +221,7 @@ cisco-nx;command;nxos::ShowEnvPower;show environment power cisco-nx;command;nxos::ShowBoot;show boot cisco-nx;command;nxos::DirSlotN;dir bootflash: -cisco-nx;command;nxos::DirSlotN;dir debug: +## cisco-nx;command;nxos::DirSlotN;dir debug: cisco-nx;command;nxos::DirSlotN;dir logflash: cisco-nx;command;nxos::DirSlotN;dir slot0: cisco-nx;command;nxos::DirSlotN;dir usb1: @@ -598,7 +598,7 @@ junos;module;junos junos;inloop;junos::inloop junos;command;junos::ShowChassisClocks;show chassis clocks -junos;command;junos::ShowChassisEnvironment;show chassis environment +## junos;command;junos::ShowChassisEnvironment;show chassis environment junos;command;junos::ShowChassisFirmware;show chassis firmware local junos;command;junos::ShowChassisFirmware;show chassis firmware From affa_pan at hotmail.com Fri Nov 13 13:03:54 2020 From: affa_pan at hotmail.com (Pan Affa) Date: Fri, 13 Nov 2020 13:03:54 +0000 Subject: [rancid] Cisco devices and calculating percent free In-Reply-To: <20201112183245.GA1501@shrubbery.net> References: , <20201112183245.GA1501@shrubbery.net> Message-ID: I?m expecting the postscript filter because there more and more files in flash. I need per-device per-tag postscript filter. from Affa Pan > john heasley ? 2020?11?13? ??2:32 ??? > > ?Thu, Nov 12, 2020 at 07:19:36AM -0600, Chris Wopat: >> I recently updated to v3.13, I do not recall exactly what version I >> was on but it was >1 year old, and could've been HEAD based on some >> previous debugging of other issues. >> >> Anyhow, we have many IOS devices that calculate percent free and it oscillates: >> >> - !Flash: bootflash: 1 GB total (19% free) >> + !Flash: bootflash: 1 GB total (18% free) >> >> Without manually patching ios.pm (or wherever this takes place) is >> there any type of config setting to just turn off showing any info >> about how much disk space is free? We get this info properly from snmp >> trending systems. >> >> Just trying to cut down on router diffs where there were no actual >> config changes. > > Except that something to writing to your flash...maybe you expect that. > > anyway, private devices types without the dir commands could be > created in rancid.types.conf(5). > > or, i have almost finished an option for a user-defined per-device type > postscript filter. eg: > > +# example use of the postscript directive. This will replace control > +# characters in the output with an ASCII representation. > +arbor-ctrl;script;rancid -t arbor-ctrl > +arbor-ctrl;login;anlogin > +arbor-ctrl;module;arbor > +arbor-ctrl;inloop;arbor::inloop > +arbor-ctrl;postscript;/bin/cat -v > +arbor-ctrl;command;arbor::ShowVersion;system version > +arbor-ctrl;command;arbor::ShowVersion;system file show > +arbor-ctrl;command;arbor::ShowHardware;system hardware > +arbor-ctrl;command;arbor::ShowHardware;system show > +arbor-ctrl;command;arbor::ShowFlash;system disk show > +arbor-ctrl;command;arbor::ShowConfig;config show > > _______________________________________________ > Rancid-discuss mailing list > Rancid-discuss at www.shrubbery.net > https://www.shrubbery.net/mailman/listinfo/rancid-discuss From me at falz.net Fri Nov 13 14:31:25 2020 From: me at falz.net (Chris Wopat) Date: Fri, 13 Nov 2020 08:31:25 -0600 Subject: [rancid] Cisco devices and calculating percent free In-Reply-To: References: Message-ID: Looked into this a little further. The disk space churn is being caused by this device type - Cisco ASR920, with churn of whatever in bootflash:/tracelogs . IOS-XE seems to be doing its own thing in there as files tend to be <1 day old while no device changes are happening. The contents of this directory are properly ignored and aren't in diffs, but this slight change in disk space is causing the issue. If I do comment this from rancid.types.conf or rancid.types.base, as expected it removes the directory contents, which does contain valuable info I like to have (ie which IOS images are chilling there): #cisco;command;ios::DirSlotN;dir /all bootflash: So, the only way for me to currently workaround this is to make a change directly to rancid.pm, in "sub diskszsummary", such as, but not limited to: #return("$total total ($pcnt% free)"); return("$total total"); Certainly dirty but works. Curious if there's any thoughts on a 'clean' way to do this? I know there's an existing 'FILTER_OSC' flag that I believe is intended to reduce churn in general, my best guess for a 'clean' fix, if it makes sense to people, would be to check that here and only return "$pcnt% free" if that isn't set. My $.02 If people are opposed to such a thing, I'll likely add this to my 'manually patch these couple of things' list that already exists post-upgrades. Cheers --Chris From emille at abccommunications.com Fri Nov 13 16:27:02 2020 From: emille at abccommunications.com (Emille Blanc) Date: Fri, 13 Nov 2020 16:27:02 +0000 Subject: [rancid] Cisco devices and calculating percent free In-Reply-To: References: Message-ID: <4008c5607c9a4e3caed7526df4216392@EX2K13.ad.abccommunications.com> > ASR920, with churn of whatever in bootflash:/tracelogs I've been through the ringer on this one (about this time last year, in fact). If you dig even deeper, it seems to be attached to the cylon_mgr process, which presumably ties to the chipset. In our environment, busier boxes have greater churn (in the 100's of MBytes/day). Our patch around this (pre RANCID v3.10) was not to ignore the disk space, but rather impose an upper ceiling for our ASR920's. Eg; - !Flash: bootflash: 1339412480 bytes total (470 MB free) + !Flash: bootflash: 1339412480 bytes total (>300 MB free) This way, we still get insight into run-away processes that suddenly burn through the box' flash:, and it adequately stops the noise. Since upgrading 3.10, it has been less frequent, so we never re-patched. But can confirm, out of all our devices, it's only the ASR920's that exhibit this behavior in our environment. ASR901's, 1000's seem fine. Food for further thought. -----Original Message----- From: Rancid-discuss [mailto:rancid-discuss-bounces at www.shrubbery.net] On Behalf Of Chris Wopat Sent: Friday, November 13, 2020 6:31 AM To: Randy Bush Cc: rancid-discuss at www.shrubbery.net Subject: Re: [rancid] Cisco devices and calculating percent free Looked into this a little further. The disk space churn is being caused by this device type - Cisco ASR920, with churn of whatever in bootflash:/tracelogs . IOS-XE seems to be doing its own thing in there as files tend to be <1 day old while no device changes are happening. The contents of this directory are properly ignored and aren't in diffs, but this slight change in disk space is causing the issue. If I do comment this from rancid.types.conf or rancid.types.base, as expected it removes the directory contents, which does contain valuable info I like to have (ie which IOS images are chilling there): #cisco;command;ios::DirSlotN;dir /all bootflash: So, the only way for me to currently workaround this is to make a change directly to rancid.pm, in "sub diskszsummary", such as, but not limited to: #return("$total total ($pcnt% free)"); return("$total total"); Certainly dirty but works. Curious if there's any thoughts on a 'clean' way to do this? I know there's an existing 'FILTER_OSC' flag that I believe is intended to reduce churn in general, my best guess for a 'clean' fix, if it makes sense to people, would be to check that here and only return "$pcnt% free" if that isn't set. My $.02 If people are opposed to such a thing, I'll likely add this to my 'manually patch these couple of things' list that already exists post-upgrades. Cheers --Chris _______________________________________________ Rancid-discuss mailing list Rancid-discuss at www.shrubbery.net https://www.shrubbery.net/mailman/listinfo/rancid-discuss From me at falz.net Wed Nov 18 14:45:19 2020 From: me at falz.net (Chris Wopat) Date: Wed, 18 Nov 2020 08:45:19 -0600 Subject: [rancid] Cisco devices and calculating percent free In-Reply-To: References: Message-ID: On Fri, Nov 13, 2020 at 8:31 AM Chris Wopat wrote: > > So, the only way for me to currently workaround this is to make a > change directly to rancid.pm, in "sub diskszsummary", such as, but not > limited to: > > #return("$total total ($pcnt% free)"); > return("$total total"); > > Certainly dirty but works. Curious if there's any thoughts on a > 'clean' way to do this? I know there's an existing 'FILTER_OSC' flag > that I believe is intended to reduce churn in general, my best guess > for a 'clean' fix, if it makes sense to people, would be to check that > here and only return "$pcnt% free" if that isn't set. We've been running this way for a bit and it's solved our churn issue. I'll just manually patch this during upgrade for now. If a similar fix does get included in the FILTER_OSC variable, then all the better but no biggie. --Chris From heas at shrubbery.net Thu Nov 19 21:15:21 2020 From: heas at shrubbery.net (john heasley) Date: Thu, 19 Nov 2020 21:15:21 +0000 Subject: [rancid] Cisco devices and calculating percent free In-Reply-To: <4008c5607c9a4e3caed7526df4216392@EX2K13.ad.abccommunications.com> References: <4008c5607c9a4e3caed7526df4216392@EX2K13.ad.abccommunications.com> Message-ID: <20201119211521.GF27505@shrubbery.net> Fri, Nov 13, 2020 at 04:27:02PM +0000, Emille Blanc: > Our patch around this (pre RANCID v3.10) was not to ignore the disk space, but rather impose an upper ceiling for our ASR920's. > Eg; > - !Flash: bootflash: 1339412480 bytes total (470 MB free) > + !Flash: bootflash: 1339412480 bytes total (>300 MB free) > > This way, we still get insight into run-away processes that suddenly burn through the box' flash:, and it adequately stops the noise. > Since upgrading 3.10, it has been less frequent, so we never re-patched. > > But can confirm, out of all our devices, it's only the ASR920's that exhibit this behavior in our environment. > ASR901's, 1000's seem fine. Still seems like it should not do that; that it is something that mistakingly escaped DE. anyway, i'm open to your solution or more suggestions. The ios script could do this just for the asr900 series, but even then I am not sure that one size fits all. Prost From heas at shrubbery.net Thu Nov 19 21:19:49 2020 From: heas at shrubbery.net (john heasley) Date: Thu, 19 Nov 2020 21:19:49 +0000 Subject: [rancid] Cisco devices and calculating percent free In-Reply-To: References: <20201112183245.GA1501@shrubbery.net> Message-ID: <20201119211949.GG27505@shrubbery.net> Fri, Nov 13, 2020 at 01:03:54PM +0000, Pan Affa: > I?m expecting the postscript filter because there more and more files in flash. I need per-device per-tag postscript filter. > > from Affa Pan > > or, i have almost finished an option for a user-defined per-device type > > postscript filter. eg: You can try ftp://ftp.shrubbery.net/pub/rancid/alpha/rancid-3.99.99.tar.gz I've done minimal testing. This will become 4.0 and does require python >= 3.5, which is not exactly used yet and use that at your own risk. If you do not deviate from the rancid 3.x features + this postscript feature, there should be no ill effect. > > +# example use of the postscript directive. This will replace control > > +# characters in the output with an ASCII representation. > > +arbor-ctrl;script;rancid -t arbor-ctrl > > +arbor-ctrl;login;anlogin > > +arbor-ctrl;module;arbor > > +arbor-ctrl;inloop;arbor::inloop > > +arbor-ctrl;postscript;/bin/cat -v > > +arbor-ctrl;command;arbor::ShowVersion;system version > > +arbor-ctrl;command;arbor::ShowVersion;system file show > > +arbor-ctrl;command;arbor::ShowHardware;system hardware > > +arbor-ctrl;command;arbor::ShowHardware;system show > > +arbor-ctrl;command;arbor::ShowFlash;system disk show > > +arbor-ctrl;command;arbor::ShowConfig;config show