[Reconnoiter-users] understanging ping_icmp behaviour

Theo Schlossnagle jesus at omniti.com
Fri Oct 1 07:12:07 EDT 2010


This sounds like a timeout issue.  Every check has to have a timeout
(specified in milliseconds).

The ping check (by default) has an internal "interval" of 2000ms.  which
means when the check starts, a single ICMP packet is sent.  Then, 2000ms
later, the next. A subsequent 2000ms, the third.

So, if you have a count of 5, it will take _at least_ 8000ms to complete the
check.  If your check timeout it lower than 2000ms you will almost certainly
only see a single packet succeed in this check and the behavior you describe
will present.

In short, set the timeout up to something reasonable for the ping to
complete running its test.

2010/10/1 Phil Pierotti <phil.pierotti at gmail.com>

> I've thrown up an instance of reconnoiter, and configured it to ping the
> routers and switches on my network.
>
> As far as I can tell from the reconnoiter graphs for any item to ping only
> one ping "succeeds" and all the rest "fail".
> With the default of 5 pings, available=20 (presumably that's "percent").
> When I set it to 20 pings, available=5.
>
> I see this result *always*, and for *every* device on my network.
>
> The fact is that I do not have 80% packet loss anywhere in my network, let
> alone *everywhere*, so I'm confused.
>
> Why is ping_icmp returning this result (feel free to tell me I've
> misconfigured something, it's entirely likely).
>
> NB this instance of reconnoiter is running on a VM, but neither the VM not
> the underlying server are busy.
>
> Here's the output with noitd in debug mode.
>
> # /usr/local/sbin/noitd -c /usr/local/etc/noit.conf -D
> [2010-10-01 14:14:21.076597] Processed 1 includes
> [2010-10-01 14:14:21.076898] Found 29 /noit/logs//log stanzas
> [2010-10-01 14:14:21.077063] Found 1 outlets for log 'error'
> [2010-10-01 14:14:21.084360] rlim { 1048576, 1048576 }
> [2010-10-01 14:14:21.087976] Found 1 acl stanzas
> [2010-10-01 14:14:21.089351] Module selfcheck successfully loaded.
> [2010-10-01 14:14:21.090572] Module ping_icmp successfully loaded.
> [2010-10-01 14:14:21.090991] Module dns successfully loaded.
> [2010-10-01 14:14:21.174749] Module snmp successfully loaded.
> [2010-10-01 14:14:21.175592] Module ssh2 successfully loaded.
> [2010-10-01 14:14:21.177243] Module varnish successfully loaded.
> [2010-10-01 14:14:21.179774] Module http successfully loaded.
> [2010-10-01 14:14:21.181871] Module resmon successfully loaded.
> [2010-10-01 14:14:21.183285] Module smtp successfully loaded.
> [2010-10-01 14:14:21.186246] Module jmx successfully loaded.
> [2010-10-01 14:14:21.193486] Found 4 /noit/listeners//listener stanzas
> [2010-10-01 14:14:21.380382] ping_icmp_send(0x8a76b48,202.62.143.25,2000,5)
> [2010-10-01 14:14:21.380467] ping_icmp_real_send(202.62.143.25)
> [2010-10-01 14:14:21.381640] ping_icmp: 202.62.143.25 1 is still
> outstanding.
> [2010-10-01 14:14:21.788361] ping_icmp_send(0x8a76b48,202.62.144.5,2000,5)
> [2010-10-01 14:14:21.788433] ping_icmp_real_send(202.62.144.5)
> [2010-10-01 14:14:21.789260] ping_icmp: 202.62.144.5 1 is still
> outstanding.
> [2010-10-01 14:14:21.880551] ping_icmp(202.62.143.25)
> [cnt=5,avail=20,min=0.0011,max=0.0011,avg=0.0011]
> [2010-10-01 14:14:22.288466] ping_icmp(202.62.144.5)
> [cnt=5,avail=20,min=0.0008,max=0.0008,avg=0.0008]
> [2010-10-01 14:14:22.772364] ping_icmp_send(0x8a76b48,202.62.144.8,2000,5)
> [2010-10-01 14:14:22.772436] ping_icmp_real_send(202.62.144.8)
> [2010-10-01 14:14:22.773227] ping_icmp: 202.62.144.8 1 is still
> outstanding.
> [2010-10-01 14:14:23.272461] ping_icmp(202.62.144.8)
> [cnt=5,avail=20,min=0.0008,max=0.0008,avg=0.0008]
> [2010-10-01 14:14:23.380453] ping check no longer active, bailing
> [2010-10-01 14:14:23.788523] ping check no longer active, bailing
> [2010-10-01 14:14:24.772421] ping check no longer active, bailing
> [2010-10-01 14:14:24.988383] ping_icmp_send(0x8a76b48,202.62.144.6,2000,5)
> [2010-10-01 14:14:24.988452] ping_icmp_real_send(202.62.144.6)
> [2010-10-01 14:14:24.989231] ping_icmp: 202.62.144.6 1 is still
> outstanding.
> [2010-10-01 14:14:25.380453] ping check no longer active, bailing
> [2010-10-01 14:14:25.488477] ping_icmp(202.62.144.6)
> [cnt=5,avail=20,min=0.0008,max=0.0008,avg=0.0008]
> [2010-10-01 14:14:25.788421] ping check no longer active, bailing
> [2010-10-01 14:14:26.328614] ping_icmp_send(0x8a76b48,202.62.148.3,2000,5)
> [2010-10-01 14:14:26.328679] ping_icmp_real_send(202.62.148.3)
> [2010-10-01 14:14:26.347888] ping_icmp: 202.62.148.3 1 is still
> outstanding.
> [2010-10-01 14:14:26.772480] ping check no longer active, bailing
> [2010-10-01 14:14:26.789316] ping_icmp_send(0x8a76b48,202.62.156.3,2000,5)
> [2010-10-01 14:14:26.789382] ping_icmp_real_send(202.62.156.3)
> [2010-10-01 14:14:26.802455] ping_icmp: 202.62.156.3 1 is still
> outstanding.
> [2010-10-01 14:14:26.829352] ping_icmp(202.62.148.3)
> [cnt=5,avail=20,min=0.0192,max=0.0192,avg=0.0192]
> [2010-10-01 14:14:26.988441] ping check no longer active, bailing
> [2010-10-01 14:14:27.292398] ping_icmp(202.62.156.3)
> [cnt=5,avail=20,min=0.0131,max=0.0131,avg=0.0131]
> [2010-10-01 14:14:27.380454] ping check no longer active, bailing
> [2010-10-01 14:14:27.788420] ping check no longer active, bailing
> [2010-10-01 14:14:28.328665] ping check no longer active, bailing
> [2010-10-01 14:14:28.772422] ping check no longer active, bailing
> [2010-10-01 14:14:28.789365] ping check no longer active, bailing
> [2010-10-01 14:14:28.988441] ping check no longer active, bailing
> [2010-10-01 14:14:29.380455] ping check no longer active, bailing
> [2010-10-01 14:14:29.788420] ping check no longer active, bailing
> [2010-10-01 14:14:30.328664] ping check no longer active, bailing
> [2010-10-01 14:14:30.772469] ping check no longer active, bailing
> [2010-10-01 14:14:30.792332] ping check no longer active, bailing
> [2010-10-01 14:14:30.988441] ping check no longer active, bailing
> [2010-10-01 14:14:32.324403] ping_icmp_send(0x8a76b48,202.62.148.2,2000,5)
> [2010-10-01 14:14:32.324475] ping_icmp_real_send(202.62.148.2)
> [2010-10-01 14:14:32.328705] ping check no longer active, bailing
> [2010-10-01 14:14:32.343216] ping_icmp: 202.62.148.2 1 is still
> outstanding.
> [2010-10-01 14:14:32.789366] ping check no longer active, bailing
> [2010-10-01 14:14:32.824516] ping_icmp(202.62.148.2)
> [cnt=5,avail=20,min=0.0187,max=0.0187,avg=0.0187]
> [2010-10-01 14:14:32.988439] ping check no longer active, bailing
> [2010-10-01 14:14:34.028486] ping_icmp_send(0x8a76b48,202.62.144.14,2000,5)
> [2010-10-01 14:14:34.028555] ping_icmp_real_send(202.62.144.14)
> [2010-10-01 14:14:34.030254] ping_icmp: 202.62.144.14 1 is still
> outstanding.
> [2010-10-01 14:14:34.324453] ping check no longer active, bailing
> [2010-10-01 14:14:34.328664] ping check no longer active, bailing
> [2010-10-01 14:14:34.532247] ping_icmp(202.62.144.14)
> [cnt=5,avail=20,min=0.0017,max=0.0017,avg=0.0017]
> [2010-10-01 14:14:34.789365] ping check no longer active, bailing
> [2010-10-01 14:14:36.028537] ping check no longer active, bailing
> [2010-10-01 14:14:36.096472] ping_icmp_send(0x8a76b48,202.62.144.7,2000,5)
> [2010-10-01 14:14:36.096559] ping_icmp_real_send(202.62.144.7)
> [2010-10-01 14:14:36.098155] ping_icmp: 202.62.144.7 1 is still
> outstanding.
> [2010-10-01 14:14:36.324477] ping check no longer active, bailing
> [2010-10-01 14:14:36.480487] ping_icmp_send(0x8a76b48,202.62.144.11,2000,5)
> [2010-10-01 14:14:36.480549] ping_icmp_real_send(202.62.144.11)
> [2010-10-01 14:14:36.481413] ping_icmp: 202.62.144.11 1 is still
> outstanding.
> [2010-10-01 14:14:36.596580] ping_icmp(202.62.144.7)
> [cnt=5,avail=20,min=0.0016,max=0.0016,avg=0.0016]
> [2010-10-01 14:14:36.952468] ping_icmp_send(0x8a76b48,202.62.148.4,2000,5)
> [2010-10-01 14:14:36.952545] ping_icmp_real_send(202.62.148.4)
> [2010-10-01 14:14:36.970919] ping_icmp: 202.62.148.4 1 is still
> outstanding.
> [2010-10-01 14:14:36.980580] ping_icmp(202.62.144.11)
> [cnt=5,avail=20,min=0.0008,max=0.0008,avg=0.0008]
> [2010-10-01 14:14:37.452582] ping_icmp(202.62.148.4)
> [cnt=5,avail=20,min=0.0184,max=0.0184,avg=0.0184]
> [2010-10-01 14:14:37.476338] ping_icmp_send(0x8a76b48,202.62.144.15,2000,5)
> [2010-10-01 14:14:37.476412] ping_icmp_real_send(202.62.144.15)
> [2010-10-01 14:14:37.477237] ping_icmp: 202.62.144.15 1 is still
> outstanding.
> [2010-10-01 14:14:37.976444] ping_icmp(202.62.144.15)
> [cnt=5,avail=20,min=0.0008,max=0.0008,avg=0.0008]
> [2010-10-01 14:14:38.028536] ping check no longer active, bailing
> [2010-10-01 14:14:38.096533] ping check no longer active, bailing
> [2010-10-01 14:14:38.324450] ping check no longer active, bailing
> [2010-10-01 14:14:38.480534] ping check no longer active, bailing
> [2010-10-01 14:14:38.952528] ping check no longer active, bailing
> [2010-10-01 14:14:39.476392] ping check no longer active, bailing
> [2010-10-01 14:14:39.720410] ping_icmp_send(0x8a76b48,202.62.148.1,2000,5)
> [2010-10-01 14:14:39.720470] ping_icmp_real_send(202.62.148.1)
> [2010-10-01 14:14:39.739387] ping_icmp: 202.62.148.1 1 is still
> outstanding.
> [2010-10-01 14:14:39.925275] ping_icmp_send(0x8a76b48,202.62.144.62,2000,5)
> [2010-10-01 14:14:39.925345] ping_icmp_real_send(202.62.144.62)
> [2010-10-01 14:14:39.926579] ping_icmp: 202.62.144.62 1 is still
> outstanding.
> [2010-10-01 14:14:40.028536] ping check no longer active, bailing
> [2010-10-01 14:14:40.096533] ping check no longer active, bailing
> [2010-10-01 14:14:40.220497] ping_icmp(202.62.148.1)
> [cnt=5,avail=20,min=0.0189,max=0.0189,avg=0.0189]
> [2010-10-01 14:14:40.324449] ping check no longer active, bailing
> [2010-10-01 14:14:40.425377] ping_icmp(202.62.144.62)
> [cnt=5,avail=20,min=0.0012,max=0.0012,avg=0.0012]
> [2010-10-01 14:14:40.480534] ping check no longer active, bailing
> [2010-10-01 14:14:40.952528] ping check no longer active, bailing
> [2010-10-01 14:14:41.232418] ping_icmp_send(0x8a76b48,202.62.148.9,2000,5)
> [2010-10-01 14:14:41.232495] ping_icmp_real_send(202.62.148.9)
> [2010-10-01 14:14:41.251028] ping_icmp: 202.62.148.9 1 is still
> outstanding.
> [2010-10-01 14:14:41.476394] ping check no longer active, bailing
> [2010-10-01 14:14:41.560442] ping_icmp_send(0x8a76b48,202.62.156.1,2000,5)
> [2010-10-01 14:14:41.560514] ping_icmp_real_send(202.62.156.1)
> [2010-10-01 14:14:41.574505] ping_icmp: 202.62.156.1 1 is still
> outstanding.
> [2010-10-01 14:14:41.620400] ping_icmp_send(0x8a76b48,202.62.144.2,2000,5)
> [2010-10-01 14:14:41.620466] ping_icmp_real_send(202.62.144.2)
> [2010-10-01 14:14:41.621456] ping_icmp: 202.62.144.2 1 is still
> outstanding.
> [2010-10-01 14:14:41.720457] ping check no longer active, bailing
> [2010-10-01 14:14:41.732524] ping_icmp(202.62.148.9)
> [cnt=5,avail=20,min=0.0185,max=0.0185,avg=0.0185]
> [2010-10-01 14:14:41.925325] ping check no longer active, bailing
> [2010-10-01 14:14:42.028535] ping check no longer active, bailing
> [2010-10-01 14:14:42.060552] ping_icmp(202.62.156.1)
> [cnt=5,avail=20,min=0.0140,max=0.0140,avg=0.0140]
> [2010-10-01 14:14:42.096533] ping check no longer active, bailing
> [2010-10-01 14:14:42.120493] ping_icmp(202.62.144.2)
> [cnt=5,avail=20,min=0.0010,max=0.0010,avg=0.0010]
> [2010-10-01 14:14:42.244427] ping_icmp_send(0x8a76b48,202.62.144.13,2000,5)
> [2010-10-01 14:14:42.244504] ping_icmp_real_send(202.62.144.13)
> [2010-10-01 14:14:42.245513] ping_icmp: 202.62.144.13 1 is still
> outstanding.
> [2010-10-01 14:14:42.480534] ping check no longer active, bailing
> [2010-10-01 14:14:42.744524] ping_icmp(202.62.144.13)
> [cnt=5,avail=20,min=0.0010,max=0.0010,avg=0.0010]
> [2010-10-01 14:14:42.952527] ping check no longer active, bailing
> [2010-10-01 14:14:43.232476] ping check no longer active, bailing
> [2010-10-01 14:14:43.476416] ping check no longer active, bailing
> [2010-10-01 14:14:43.560495] ping check no longer active, bailing
> [2010-10-01 14:14:43.620445] ping check no longer active, bailing
> [2010-10-01 14:14:43.720457] ping check no longer active, bailing
> [2010-10-01 14:14:43.776437] ping_icmp_send(0x8a76b48,202.62.144.17,2000,5)
> [2010-10-01 14:14:43.776508] ping_icmp_real_send(202.62.144.17)
> [2010-10-01 14:14:43.777719] ping_icmp: 202.62.144.17 1 is still
> outstanding.
> [2010-10-01 14:14:43.925324] ping check no longer active, bailing
> [2010-10-01 14:14:44.096533] ping check no longer active, bailing
> [2010-10-01 14:14:44.244487] ping check no longer active, bailing
> [2010-10-01 14:14:44.276536] ping_icmp(202.62.144.17)
> [cnt=5,avail=20,min=0.0012,max=0.0012,avg=0.0012]
> [2010-10-01 14:14:44.320409] ping_icmp_send(0x8a76b48,202.62.144.3,2000,5)
> [2010-10-01 14:14:44.320493] ping_icmp_real_send(202.62.144.3)
> [2010-10-01 14:14:44.322446] ping_icmp: 202.62.144.3 1 is still
> outstanding.
> [2010-10-01 14:14:44.480535] ping check no longer active, bailing
> [2010-10-01 14:14:44.822513] ping_icmp(202.62.144.3)
> [cnt=5,avail=20,min=0.0019,max=0.0019,avg=0.0019]
> [2010-10-01 14:14:44.952528] ping check no longer active, bailing
> [2010-10-01 14:14:45.232477] ping check no longer active, bailing
> [2010-10-01 14:14:45.476455] ping check no longer active, bailing
> [2010-10-01 14:14:45.560496] ping check no longer active, bailing
> [2010-10-01 14:14:45.620446] ping check no longer active, bailing
> [2010-10-01 14:14:45.720457] ping check no longer active, bailing
> [2010-10-01 14:14:45.776490] ping check no longer active, bailing
> [2010-10-01 14:14:45.925323] ping check no longer active, bailing
> [2010-10-01 14:14:46.116440] ping_icmp_send(0x8a76b48,202.62.144.4,2000,5)
> [2010-10-01 14:14:46.116520] ping_icmp_real_send(202.62.144.4)
> [2010-10-01 14:14:46.119236] ping_icmp: 202.62.144.4 1 is still
> outstanding.
> [2010-10-01 14:14:46.244487] ping check no longer active, bailing
> [2010-10-01 14:14:46.320462] ping check no longer active, bailing
> [2010-10-01 14:14:46.616561] ping_icmp(202.62.144.4)
> [cnt=5,avail=20,min=0.0027,max=0.0027,avg=0.0027]
> [2010-10-01 14:14:47.052374] ping_icmp_send(0x8a76b48,66.225.209.7,2000,5)
> [2010-10-01 14:14:47.052442] ping_icmp_real_send(66.225.209.7)
> [2010-10-01 14:14:47.232477] ping check no longer active, bailing
> [2010-10-01 14:14:47.288886] ping_icmp: 66.225.209.7 1 is still
> outstanding.
> [2010-10-01 14:14:47.484438] ping_icmp_send(0x8a76b48,202.62.144.1,2000,5)
> [2010-10-01 14:14:47.484519] ping_icmp_real_send(202.62.144.1)
> [2010-10-01 14:14:47.485013] ping_icmp: 202.62.144.1 1 is still
> outstanding.
> [2010-10-01 14:14:47.560495] ping check no longer active, bailing
> [2010-10-01 14:14:47.620447] ping check no longer active, bailing
> [2010-10-01 14:14:47.720457] ping check no longer active, bailing
> [2010-10-01 14:14:47.776492] ping check no longer active, bailing
> [2010-10-01 14:14:47.925323] ping check no longer active, bailing
> [2010-10-01 14:14:47.984549] ping_icmp(202.62.144.1)
> [cnt=5,avail=20,min=0.0005,max=0.0005,avg=0.0005]
> [2010-10-01 14:14:48.116504] ping check no longer active, bailing
> [2010-10-01 14:14:48.244487] ping check no longer active, bailing
> [2010-10-01 14:14:48.320464] ping check no longer active, bailing
> [2010-10-01 14:14:49.052423] ping_icmp_real_send(66.225.209.7)
> [2010-10-01 14:14:49.232477] ping check no longer active, bailing
> [2010-10-01 14:14:49.288655] ping_icmp: 66.225.209.7 2 is still
> outstanding.
> [2010-10-01 14:14:49.484497] ping check no longer active, bailing
> [2010-10-01 14:14:49.560496] ping check no longer active, bailing
> [2010-10-01 14:14:49.620446] ping check no longer active, bailing
> [2010-10-01 14:14:49.776492] ping check no longer active, bailing
> [2010-10-01 14:14:50.116503] ping check no longer active, bailing
> [2010-10-01 14:14:50.244488] ping check no longer active, bailing
> [2010-10-01 14:14:50.320465] ping check no longer active, bailing
> [2010-10-01 14:14:51.052424] ping_icmp_real_send(66.225.209.7)
> [2010-10-01 14:14:51.288784] ping_icmp: 66.225.209.7 3 is still
> outstanding.
> [2010-10-01 14:14:51.484497] ping check no longer active, bailing
> [2010-10-01 14:14:51.776490] ping check no longer active, bailing
> [2010-10-01 14:14:52.116502] ping check no longer active, bailing
> [2010-10-01 14:14:52.320462] ping check no longer active, bailing
> [2010-10-01 14:14:53.052423] ping_icmp_real_send(66.225.209.7)
> [2010-10-01 14:14:53.288581] ping_icmp: 66.225.209.7 4 is still
> outstanding.
> [2010-10-01 14:14:53.484498] ping check no longer active, bailing
> [2010-10-01 14:14:54.116503] ping check no longer active, bailing
> [2010-10-01 14:14:55.052422] ping_icmp_real_send(66.225.209.7)
> [2010-10-01 14:14:55.288583] ping_icmp(66.225.209.7)
> [cnt=5,avail=100,min=0.2361,max=0.2364,avg=0.2362]
> [2010-10-01 14:14:55.484498] ping check no longer active, bailing
> [2010-10-01 14:14:59.431416] ping_icmp bad size: 20+36
> [2010-10-01 14:14:59.666411] ping_icmp bad size: 20+36
>
> Thanks,
> Phil P
>
> _______________________________________________
> Reconnoiter-users mailing list
> Reconnoiter-users at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/reconnoiter-users
>
>


-- 

Theo Schlossnagle

http://omniti.com/is/theo-schlossnagle
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.omniti.com/pipermail/reconnoiter-users/attachments/20101001/442c5b15/attachment-0001.html 


More information about the Reconnoiter-users mailing list