[Reconnoiter-users] understanging ping_icmp behaviour

Phil Pierotti phil.pierotti at gmail.com
Fri Oct 1 08:49:49 EDT 2010


Hi Theo,

Aha!

So for some reason I was thinking/assuming that "timeout" was *for each
ping* not for all the pings in this check.
Naturally I'd set it to something sane (ie 500ms), and *all* pings except
the first falls outside that.

Problem Solved!

Thanks,
Phil P

2010/10/1 Theo Schlossnagle <jesus at omniti.com>

> This sounds like a timeout issue.  Every check has to have a timeout
> (specified in milliseconds).
>
> The ping check (by default) has an internal "interval" of 2000ms.  which
> means when the check starts, a single ICMP packet is sent.  Then, 2000ms
> later, the next. A subsequent 2000ms, the third.
>
> So, if you have a count of 5, it will take _at least_ 8000ms to complete
> the check.  If your check timeout it lower than 2000ms you will almost
> certainly only see a single packet succeed in this check and the behavior
> you describe will present.
>
> In short, set the timeout up to something reasonable for the ping to
> complete running its test.
>
> 2010/10/1 Phil Pierotti <phil.pierotti at gmail.com>
>
>> I've thrown up an instance of reconnoiter, and configured it to ping the
>> routers and switches on my network.
>>
>> As far as I can tell from the reconnoiter graphs for any item to ping only
>> one ping "succeeds" and all the rest "fail".
>> With the default of 5 pings, available=20 (presumably that's "percent").
>> When I set it to 20 pings, available=5.
>>
>> I see this result *always*, and for *every* device on my network.
>>
>> The fact is that I do not have 80% packet loss anywhere in my network, let
>> alone *everywhere*, so I'm confused.
>>
>> Why is ping_icmp returning this result (feel free to tell me I've
>> misconfigured something, it's entirely likely).
>>
>> NB this instance of reconnoiter is running on a VM, but neither the VM not
>> the underlying server are busy.
>>
>> Here's the output with noitd in debug mode.
>>
>> # /usr/local/sbin/noitd -c /usr/local/etc/noit.conf -D
>> [2010-10-01 14:14:21.076597] Processed 1 includes
>> [2010-10-01 14:14:21.076898] Found 29 /noit/logs//log stanzas
>> [2010-10-01 14:14:21.077063] Found 1 outlets for log 'error'
>> [2010-10-01 14:14:21.084360] rlim { 1048576, 1048576 }
>> [2010-10-01 14:14:21.087976] Found 1 acl stanzas
>> [2010-10-01 14:14:21.089351] Module selfcheck successfully loaded.
>> [2010-10-01 14:14:21.090572] Module ping_icmp successfully loaded.
>> [2010-10-01 14:14:21.090991] Module dns successfully loaded.
>> [2010-10-01 14:14:21.174749] Module snmp successfully loaded.
>> [2010-10-01 14:14:21.175592] Module ssh2 successfully loaded.
>> [2010-10-01 14:14:21.177243] Module varnish successfully loaded.
>> [2010-10-01 14:14:21.179774] Module http successfully loaded.
>> [2010-10-01 14:14:21.181871] Module resmon successfully loaded.
>> [2010-10-01 14:14:21.183285] Module smtp successfully loaded.
>> [2010-10-01 14:14:21.186246] Module jmx successfully loaded.
>> [2010-10-01 14:14:21.193486] Found 4 /noit/listeners//listener stanzas
>> [2010-10-01 14:14:21.380382]
>> ping_icmp_send(0x8a76b48,202.62.143.25,2000,5)
>> [2010-10-01 14:14:21.380467] ping_icmp_real_send(202.62.143.25)
>> [2010-10-01 14:14:21.381640] ping_icmp: 202.62.143.25 1 is still
>> outstanding.
>> [2010-10-01 14:14:21.788361] ping_icmp_send(0x8a76b48,202.62.144.5,2000,5)
>> [2010-10-01 14:14:21.788433] ping_icmp_real_send(202.62.144.5)
>> [2010-10-01 14:14:21.789260] ping_icmp: 202.62.144.5 1 is still
>> outstanding.
>> [2010-10-01 14:14:21.880551] ping_icmp(202.62.143.25)
>> [cnt=5,avail=20,min=0.0011,max=0.0011,avg=0.0011]
>> [2010-10-01 14:14:22.288466] ping_icmp(202.62.144.5)
>> [cnt=5,avail=20,min=0.0008,max=0.0008,avg=0.0008]
>> [2010-10-01 14:14:22.772364] ping_icmp_send(0x8a76b48,202.62.144.8,2000,5)
>> [2010-10-01 14:14:22.772436] ping_icmp_real_send(202.62.144.8)
>> [2010-10-01 14:14:22.773227] ping_icmp: 202.62.144.8 1 is still
>> outstanding.
>> [2010-10-01 14:14:23.272461] ping_icmp(202.62.144.8)
>> [cnt=5,avail=20,min=0.0008,max=0.0008,avg=0.0008]
>> [2010-10-01 14:14:23.380453] ping check no longer active, bailing
>> [2010-10-01 14:14:23.788523] ping check no longer active, bailing
>> [2010-10-01 14:14:24.772421] ping check no longer active, bailing
>> [2010-10-01 14:14:24.988383] ping_icmp_send(0x8a76b48,202.62.144.6,2000,5)
>> [2010-10-01 14:14:24.988452] ping_icmp_real_send(202.62.144.6)
>> [2010-10-01 14:14:24.989231] ping_icmp: 202.62.144.6 1 is still
>> outstanding.
>> [2010-10-01 14:14:25.380453] ping check no longer active, bailing
>> [2010-10-01 14:14:25.488477] ping_icmp(202.62.144.6)
>> [cnt=5,avail=20,min=0.0008,max=0.0008,avg=0.0008]
>> [2010-10-01 14:14:25.788421] ping check no longer active, bailing
>> [2010-10-01 14:14:26.328614] ping_icmp_send(0x8a76b48,202.62.148.3,2000,5)
>> [2010-10-01 14:14:26.328679] ping_icmp_real_send(202.62.148.3)
>> [2010-10-01 14:14:26.347888] ping_icmp: 202.62.148.3 1 is still
>> outstanding.
>> [2010-10-01 14:14:26.772480] ping check no longer active, bailing
>> [2010-10-01 14:14:26.789316] ping_icmp_send(0x8a76b48,202.62.156.3,2000,5)
>> [2010-10-01 14:14:26.789382] ping_icmp_real_send(202.62.156.3)
>> [2010-10-01 14:14:26.802455] ping_icmp: 202.62.156.3 1 is still
>> outstanding.
>> [2010-10-01 14:14:26.829352] ping_icmp(202.62.148.3)
>> [cnt=5,avail=20,min=0.0192,max=0.0192,avg=0.0192]
>> [2010-10-01 14:14:26.988441] ping check no longer active, bailing
>> [2010-10-01 14:14:27.292398] ping_icmp(202.62.156.3)
>> [cnt=5,avail=20,min=0.0131,max=0.0131,avg=0.0131]
>> [2010-10-01 14:14:27.380454] ping check no longer active, bailing
>> [2010-10-01 14:14:27.788420] ping check no longer active, bailing
>> [2010-10-01 14:14:28.328665] ping check no longer active, bailing
>> [2010-10-01 14:14:28.772422] ping check no longer active, bailing
>> [2010-10-01 14:14:28.789365] ping check no longer active, bailing
>> [2010-10-01 14:14:28.988441] ping check no longer active, bailing
>> [2010-10-01 14:14:29.380455] ping check no longer active, bailing
>> [2010-10-01 14:14:29.788420] ping check no longer active, bailing
>> [2010-10-01 14:14:30.328664] ping check no longer active, bailing
>> [2010-10-01 14:14:30.772469] ping check no longer active, bailing
>> [2010-10-01 14:14:30.792332] ping check no longer active, bailing
>> [2010-10-01 14:14:30.988441] ping check no longer active, bailing
>> [2010-10-01 14:14:32.324403] ping_icmp_send(0x8a76b48,202.62.148.2,2000,5)
>> [2010-10-01 14:14:32.324475] ping_icmp_real_send(202.62.148.2)
>> [2010-10-01 14:14:32.328705] ping check no longer active, bailing
>> [2010-10-01 14:14:32.343216] ping_icmp: 202.62.148.2 1 is still
>> outstanding.
>> [2010-10-01 14:14:32.789366] ping check no longer active, bailing
>> [2010-10-01 14:14:32.824516] ping_icmp(202.62.148.2)
>> [cnt=5,avail=20,min=0.0187,max=0.0187,avg=0.0187]
>> [2010-10-01 14:14:32.988439] ping check no longer active, bailing
>> [2010-10-01 14:14:34.028486]
>> ping_icmp_send(0x8a76b48,202.62.144.14,2000,5)
>> [2010-10-01 14:14:34.028555] ping_icmp_real_send(202.62.144.14)
>> [2010-10-01 14:14:34.030254] ping_icmp: 202.62.144.14 1 is still
>> outstanding.
>> [2010-10-01 14:14:34.324453] ping check no longer active, bailing
>> [2010-10-01 14:14:34.328664] ping check no longer active, bailing
>> [2010-10-01 14:14:34.532247] ping_icmp(202.62.144.14)
>> [cnt=5,avail=20,min=0.0017,max=0.0017,avg=0.0017]
>> [2010-10-01 14:14:34.789365] ping check no longer active, bailing
>> [2010-10-01 14:14:36.028537] ping check no longer active, bailing
>> [2010-10-01 14:14:36.096472] ping_icmp_send(0x8a76b48,202.62.144.7,2000,5)
>> [2010-10-01 14:14:36.096559] ping_icmp_real_send(202.62.144.7)
>> [2010-10-01 14:14:36.098155] ping_icmp: 202.62.144.7 1 is still
>> outstanding.
>> [2010-10-01 14:14:36.324477] ping check no longer active, bailing
>> [2010-10-01 14:14:36.480487]
>> ping_icmp_send(0x8a76b48,202.62.144.11,2000,5)
>> [2010-10-01 14:14:36.480549] ping_icmp_real_send(202.62.144.11)
>> [2010-10-01 14:14:36.481413] ping_icmp: 202.62.144.11 1 is still
>> outstanding.
>> [2010-10-01 14:14:36.596580] ping_icmp(202.62.144.7)
>> [cnt=5,avail=20,min=0.0016,max=0.0016,avg=0.0016]
>> [2010-10-01 14:14:36.952468] ping_icmp_send(0x8a76b48,202.62.148.4,2000,5)
>> [2010-10-01 14:14:36.952545] ping_icmp_real_send(202.62.148.4)
>> [2010-10-01 14:14:36.970919] ping_icmp: 202.62.148.4 1 is still
>> outstanding.
>> [2010-10-01 14:14:36.980580] ping_icmp(202.62.144.11)
>> [cnt=5,avail=20,min=0.0008,max=0.0008,avg=0.0008]
>> [2010-10-01 14:14:37.452582] ping_icmp(202.62.148.4)
>> [cnt=5,avail=20,min=0.0184,max=0.0184,avg=0.0184]
>> [2010-10-01 14:14:37.476338]
>> ping_icmp_send(0x8a76b48,202.62.144.15,2000,5)
>> [2010-10-01 14:14:37.476412] ping_icmp_real_send(202.62.144.15)
>> [2010-10-01 14:14:37.477237] ping_icmp: 202.62.144.15 1 is still
>> outstanding.
>> [2010-10-01 14:14:37.976444] ping_icmp(202.62.144.15)
>> [cnt=5,avail=20,min=0.0008,max=0.0008,avg=0.0008]
>> [2010-10-01 14:14:38.028536] ping check no longer active, bailing
>> [2010-10-01 14:14:38.096533] ping check no longer active, bailing
>> [2010-10-01 14:14:38.324450] ping check no longer active, bailing
>> [2010-10-01 14:14:38.480534] ping check no longer active, bailing
>> [2010-10-01 14:14:38.952528] ping check no longer active, bailing
>> [2010-10-01 14:14:39.476392] ping check no longer active, bailing
>> [2010-10-01 14:14:39.720410] ping_icmp_send(0x8a76b48,202.62.148.1,2000,5)
>> [2010-10-01 14:14:39.720470] ping_icmp_real_send(202.62.148.1)
>> [2010-10-01 14:14:39.739387] ping_icmp: 202.62.148.1 1 is still
>> outstanding.
>> [2010-10-01 14:14:39.925275]
>> ping_icmp_send(0x8a76b48,202.62.144.62,2000,5)
>> [2010-10-01 14:14:39.925345] ping_icmp_real_send(202.62.144.62)
>> [2010-10-01 14:14:39.926579] ping_icmp: 202.62.144.62 1 is still
>> outstanding.
>> [2010-10-01 14:14:40.028536] ping check no longer active, bailing
>> [2010-10-01 14:14:40.096533] ping check no longer active, bailing
>> [2010-10-01 14:14:40.220497] ping_icmp(202.62.148.1)
>> [cnt=5,avail=20,min=0.0189,max=0.0189,avg=0.0189]
>> [2010-10-01 14:14:40.324449] ping check no longer active, bailing
>> [2010-10-01 14:14:40.425377] ping_icmp(202.62.144.62)
>> [cnt=5,avail=20,min=0.0012,max=0.0012,avg=0.0012]
>> [2010-10-01 14:14:40.480534] ping check no longer active, bailing
>> [2010-10-01 14:14:40.952528] ping check no longer active, bailing
>> [2010-10-01 14:14:41.232418] ping_icmp_send(0x8a76b48,202.62.148.9,2000,5)
>> [2010-10-01 14:14:41.232495] ping_icmp_real_send(202.62.148.9)
>> [2010-10-01 14:14:41.251028] ping_icmp: 202.62.148.9 1 is still
>> outstanding.
>> [2010-10-01 14:14:41.476394] ping check no longer active, bailing
>> [2010-10-01 14:14:41.560442] ping_icmp_send(0x8a76b48,202.62.156.1,2000,5)
>> [2010-10-01 14:14:41.560514] ping_icmp_real_send(202.62.156.1)
>> [2010-10-01 14:14:41.574505] ping_icmp: 202.62.156.1 1 is still
>> outstanding.
>> [2010-10-01 14:14:41.620400] ping_icmp_send(0x8a76b48,202.62.144.2,2000,5)
>> [2010-10-01 14:14:41.620466] ping_icmp_real_send(202.62.144.2)
>> [2010-10-01 14:14:41.621456] ping_icmp: 202.62.144.2 1 is still
>> outstanding.
>> [2010-10-01 14:14:41.720457] ping check no longer active, bailing
>> [2010-10-01 14:14:41.732524] ping_icmp(202.62.148.9)
>> [cnt=5,avail=20,min=0.0185,max=0.0185,avg=0.0185]
>> [2010-10-01 14:14:41.925325] ping check no longer active, bailing
>> [2010-10-01 14:14:42.028535] ping check no longer active, bailing
>> [2010-10-01 14:14:42.060552] ping_icmp(202.62.156.1)
>> [cnt=5,avail=20,min=0.0140,max=0.0140,avg=0.0140]
>> [2010-10-01 14:14:42.096533] ping check no longer active, bailing
>> [2010-10-01 14:14:42.120493] ping_icmp(202.62.144.2)
>> [cnt=5,avail=20,min=0.0010,max=0.0010,avg=0.0010]
>> [2010-10-01 14:14:42.244427]
>> ping_icmp_send(0x8a76b48,202.62.144.13,2000,5)
>> [2010-10-01 14:14:42.244504] ping_icmp_real_send(202.62.144.13)
>> [2010-10-01 14:14:42.245513] ping_icmp: 202.62.144.13 1 is still
>> outstanding.
>> [2010-10-01 14:14:42.480534] ping check no longer active, bailing
>> [2010-10-01 14:14:42.744524] ping_icmp(202.62.144.13)
>> [cnt=5,avail=20,min=0.0010,max=0.0010,avg=0.0010]
>> [2010-10-01 14:14:42.952527] ping check no longer active, bailing
>> [2010-10-01 14:14:43.232476] ping check no longer active, bailing
>> [2010-10-01 14:14:43.476416] ping check no longer active, bailing
>> [2010-10-01 14:14:43.560495] ping check no longer active, bailing
>> [2010-10-01 14:14:43.620445] ping check no longer active, bailing
>> [2010-10-01 14:14:43.720457] ping check no longer active, bailing
>> [2010-10-01 14:14:43.776437]
>> ping_icmp_send(0x8a76b48,202.62.144.17,2000,5)
>> [2010-10-01 14:14:43.776508] ping_icmp_real_send(202.62.144.17)
>> [2010-10-01 14:14:43.777719] ping_icmp: 202.62.144.17 1 is still
>> outstanding.
>> [2010-10-01 14:14:43.925324] ping check no longer active, bailing
>> [2010-10-01 14:14:44.096533] ping check no longer active, bailing
>> [2010-10-01 14:14:44.244487] ping check no longer active, bailing
>> [2010-10-01 14:14:44.276536] ping_icmp(202.62.144.17)
>> [cnt=5,avail=20,min=0.0012,max=0.0012,avg=0.0012]
>> [2010-10-01 14:14:44.320409] ping_icmp_send(0x8a76b48,202.62.144.3,2000,5)
>> [2010-10-01 14:14:44.320493] ping_icmp_real_send(202.62.144.3)
>> [2010-10-01 14:14:44.322446] ping_icmp: 202.62.144.3 1 is still
>> outstanding.
>> [2010-10-01 14:14:44.480535] ping check no longer active, bailing
>> [2010-10-01 14:14:44.822513] ping_icmp(202.62.144.3)
>> [cnt=5,avail=20,min=0.0019,max=0.0019,avg=0.0019]
>> [2010-10-01 14:14:44.952528] ping check no longer active, bailing
>> [2010-10-01 14:14:45.232477] ping check no longer active, bailing
>> [2010-10-01 14:14:45.476455] ping check no longer active, bailing
>> [2010-10-01 14:14:45.560496] ping check no longer active, bailing
>> [2010-10-01 14:14:45.620446] ping check no longer active, bailing
>> [2010-10-01 14:14:45.720457] ping check no longer active, bailing
>> [2010-10-01 14:14:45.776490] ping check no longer active, bailing
>> [2010-10-01 14:14:45.925323] ping check no longer active, bailing
>> [2010-10-01 14:14:46.116440] ping_icmp_send(0x8a76b48,202.62.144.4,2000,5)
>> [2010-10-01 14:14:46.116520] ping_icmp_real_send(202.62.144.4)
>> [2010-10-01 14:14:46.119236] ping_icmp: 202.62.144.4 1 is still
>> outstanding.
>> [2010-10-01 14:14:46.244487] ping check no longer active, bailing
>> [2010-10-01 14:14:46.320462] ping check no longer active, bailing
>> [2010-10-01 14:14:46.616561] ping_icmp(202.62.144.4)
>> [cnt=5,avail=20,min=0.0027,max=0.0027,avg=0.0027]
>> [2010-10-01 14:14:47.052374] ping_icmp_send(0x8a76b48,66.225.209.7,2000,5)
>> [2010-10-01 14:14:47.052442] ping_icmp_real_send(66.225.209.7)
>> [2010-10-01 14:14:47.232477] ping check no longer active, bailing
>> [2010-10-01 14:14:47.288886] ping_icmp: 66.225.209.7 1 is still
>> outstanding.
>> [2010-10-01 14:14:47.484438] ping_icmp_send(0x8a76b48,202.62.144.1,2000,5)
>> [2010-10-01 14:14:47.484519] ping_icmp_real_send(202.62.144.1)
>> [2010-10-01 14:14:47.485013] ping_icmp: 202.62.144.1 1 is still
>> outstanding.
>> [2010-10-01 14:14:47.560495] ping check no longer active, bailing
>> [2010-10-01 14:14:47.620447] ping check no longer active, bailing
>> [2010-10-01 14:14:47.720457] ping check no longer active, bailing
>> [2010-10-01 14:14:47.776492] ping check no longer active, bailing
>> [2010-10-01 14:14:47.925323] ping check no longer active, bailing
>> [2010-10-01 14:14:47.984549] ping_icmp(202.62.144.1)
>> [cnt=5,avail=20,min=0.0005,max=0.0005,avg=0.0005]
>> [2010-10-01 14:14:48.116504] ping check no longer active, bailing
>> [2010-10-01 14:14:48.244487] ping check no longer active, bailing
>> [2010-10-01 14:14:48.320464] ping check no longer active, bailing
>> [2010-10-01 14:14:49.052423] ping_icmp_real_send(66.225.209.7)
>> [2010-10-01 14:14:49.232477] ping check no longer active, bailing
>> [2010-10-01 14:14:49.288655] ping_icmp: 66.225.209.7 2 is still
>> outstanding.
>> [2010-10-01 14:14:49.484497] ping check no longer active, bailing
>> [2010-10-01 14:14:49.560496] ping check no longer active, bailing
>> [2010-10-01 14:14:49.620446] ping check no longer active, bailing
>> [2010-10-01 14:14:49.776492] ping check no longer active, bailing
>> [2010-10-01 14:14:50.116503] ping check no longer active, bailing
>> [2010-10-01 14:14:50.244488] ping check no longer active, bailing
>> [2010-10-01 14:14:50.320465] ping check no longer active, bailing
>> [2010-10-01 14:14:51.052424] ping_icmp_real_send(66.225.209.7)
>> [2010-10-01 14:14:51.288784] ping_icmp: 66.225.209.7 3 is still
>> outstanding.
>> [2010-10-01 14:14:51.484497] ping check no longer active, bailing
>> [2010-10-01 14:14:51.776490] ping check no longer active, bailing
>> [2010-10-01 14:14:52.116502] ping check no longer active, bailing
>> [2010-10-01 14:14:52.320462] ping check no longer active, bailing
>> [2010-10-01 14:14:53.052423] ping_icmp_real_send(66.225.209.7)
>> [2010-10-01 14:14:53.288581] ping_icmp: 66.225.209.7 4 is still
>> outstanding.
>> [2010-10-01 14:14:53.484498] ping check no longer active, bailing
>> [2010-10-01 14:14:54.116503] ping check no longer active, bailing
>> [2010-10-01 14:14:55.052422] ping_icmp_real_send(66.225.209.7)
>> [2010-10-01 14:14:55.288583] ping_icmp(66.225.209.7)
>> [cnt=5,avail=100,min=0.2361,max=0.2364,avg=0.2362]
>> [2010-10-01 14:14:55.484498] ping check no longer active, bailing
>> [2010-10-01 14:14:59.431416] ping_icmp bad size: 20+36
>> [2010-10-01 14:14:59.666411] ping_icmp bad size: 20+36
>>
>> Thanks,
>> Phil P
>>
>> _______________________________________________
>> Reconnoiter-users mailing list
>> Reconnoiter-users at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/reconnoiter-users
>>
>>
>
>
> --
>
> Theo Schlossnagle
>
> http://omniti.com/is/theo-schlossnagle
>
>
> _______________________________________________
> Reconnoiter-users mailing list
> Reconnoiter-users at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/reconnoiter-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.omniti.com/pipermail/reconnoiter-users/attachments/20101001/277d62b1/attachment-0001.html 


More information about the Reconnoiter-users mailing list