From b at mine.com Wed Jul 9 08:40:07 2014 From: b at mine.com (Brian Calhoun) Date: Wed, 9 Jul 2014 10:40:07 +0200 Subject: [Reconnoiter-users] How to get statsd functioning? Message-ID: <90B21EF3-5C59-46B3-8B8C-D00379782FC3@mine.com> Hi all, I?m having some trouble getting statsd working in a non-localhost setup. I have two machines in my reconnoiter setup: a machine running stratcond and a machine running noitd. I can successfully get test statistics such as ?ping to google dns? from the noit machine to the stratcon machine and display the out-of-the-box graphs. All good. If I have this in my noit.conf: That change allowed the debug message to say ?1 checks are matching?, but still no parent text (not sure if parent info is needed in this case). The statsd messages never made it to the stratcon, regardless. I really hope I?m misunderstanding something simple about how this is supposed to work. Any tips / clues? Thanks! Brian --------- Brian Calhoun b at mine.com -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: From jesus at omniti.com Wed Jul 9 12:29:38 2014 From: jesus at omniti.com (Theo Schlossnagle) Date: Wed, 9 Jul 2014 08:29:38 -0400 Subject: [Reconnoiter-users] How to get statsd functioning? In-Reply-To: <90B21EF3-5C59-46B3-8B8C-D00379782FC3@mine.com> References: <90B21EF3-5C59-46B3-8B8C-D00379782FC3@mine.com> Message-ID: The target is the not where it is listening, it specified what machine to listen for. If one check matched, the metric should flow through. According to the docs, if you want all statsd metrics (from anywhere) to flow to a single check you should add a "check" config option to the module load with a value of the uuid of the check to which metrics will be delegated. The module will then send metrics to any checks that match *and* the explicitly delegated check. Granted, we run most of this stuff multi-tenant and statsd is just horrid for that. Also, if you want to do really rapid reporting on a check using statsd and you put all your metrics on a single statsd check you have to increase the frequency of all metrics. Basically, there is huge value in logically partitioning your statsd metrics, but since statsd and graphite, etc. can't leverage that value it was never incorporated into the protocol design. There's a significant impedance mismatch. That said, we use statsd and it works for us. On Wed, Jul 9, 2014 at 4:40 AM, Brian Calhoun wrote: > Hi all, > > I?m having some trouble getting statsd working in a non-localhost setup. I > have two machines in my reconnoiter setup: a machine running stratcond and > a machine running noitd. I can successfully get test statistics such as > ?ping to google dns? from the noit machine to the stratcon machine and > display the out-of-the-box graphs. All good. > > If I have this in my noit.conf: > > target="127.0.0.1" period="60000" timeout="59999?/> > > Then as long as I?m logged into a noit machine?s shell I can successfully > send command-line messages such as this: > > echo ?bytes_out:1238|g? | nc -u -w0 127.0.0.1 8125 > > and the message will successfully bubble up to the stratcon?s web > interface. No problem. > > The issue comes in when I try to send a message from another machine to > the noit machine. > > Perhaps I?m misunderstanding the purpose of the <> bit > in the noit.conf, but my understanding is that that?s meant to specify the > place where statsd is listening. > > If I try to send the message ?bytes_out:1238|g? to the noit machine over > udp port 8125, I can see the data hitting the port by using "nc -luv 8125? > while the noitd is down, just to test that the messages are hitting the > correct port. > > I?ve tried running the noitd in debug mode, and it tells me ?0 checks are > matching? when a statsd message comes in. So then I thought to change the > IP in the target clause to the IP of the _source_ machine like this: > > target=?192.168.xx.yy" period="60000" timeout="59999"/> > > That change allowed the debug message to say ?1 checks are matching?, but > still no parent text (not sure if parent info is needed in this case). The > statsd messages never made it to the stratcon, regardless. > > I really hope I?m misunderstanding something simple about how this is > supposed to work. > > Any tips / clues? > > Thanks! > > Brian > > > > --------- > Brian Calhoun > b at mine.com > > > > > _______________________________________________ > Reconnoiter-users mailing list > Reconnoiter-users at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/reconnoiter-users > > -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle -------------- next part -------------- An HTML attachment was scrubbed... URL: From b at mine.com Wed Jul 9 14:44:12 2014 From: b at mine.com (Brian Calhoun) Date: Wed, 9 Jul 2014 16:44:12 +0200 Subject: [Reconnoiter-users] How to get statsd functioning? In-Reply-To: References: <90B21EF3-5C59-46B3-8B8C-D00379782FC3@mine.com> Message-ID: <547CDF0A-9672-4921-9BE1-059A71A1B932@mine.com> Ah, ok. Yes, I saw that doc about the ?check? config option but didn?t really understand it until now. Thanks very much for the explanation and how you?re using statsd now. It?s good to hear real-world stories. Best, Brian On 09 Jul 2014, at 14:29, Theo Schlossnagle wrote: > The target is the not where it is listening, it specified what machine to listen for. > > If one check matched, the metric should flow through. > > According to the docs, if you want all statsd metrics (from anywhere) to flow to a single check you should add a "check" config option to the module load with a value of the uuid of the check to which metrics will be delegated. The module will then send metrics to any checks that match *and* the explicitly delegated check. > > Granted, we run most of this stuff multi-tenant and statsd is just horrid for that. Also, if you want to do really rapid reporting on a check using statsd and you put all your metrics on a single statsd check you have to increase the frequency of all metrics. > > Basically, there is huge value in logically partitioning your statsd metrics, but since statsd and graphite, etc. can't leverage that value it was never incorporated into the protocol design. There's a significant impedance mismatch. That said, we use statsd and it works for us. > > > On Wed, Jul 9, 2014 at 4:40 AM, Brian Calhoun wrote: > Hi all, > > I?m having some trouble getting statsd working in a non-localhost setup. I have two machines in my reconnoiter setup: a machine running stratcond and a machine running noitd. I can successfully get test statistics such as ?ping to google dns? from the noit machine to the stratcon machine and display the out-of-the-box graphs. All good. > > If I have this in my noit.conf: > > > I?ve tried running the noitd in debug mode, and it tells me ?0 checks are matching? when a statsd message comes in. So then I thought to change the IP in the target clause to the IP of the _source_ machine like this: > > > > That change allowed the debug message to say ?1 checks are matching?, but still no parent text (not sure if parent info is needed in this case). The statsd messages never made it to the stratcon, regardless. > > I really hope I?m misunderstanding something simple about how this is supposed to work. > > Any tips / clues? > > Thanks! > > Brian > > > > --------- > Brian Calhoun > b at mine.com > > > > > _______________________________________________ > Reconnoiter-users mailing list > Reconnoiter-users at lists.omniti.com > http://lists.omniti.com/mailman/listinfo/reconnoiter-users > > > > > -- > Theo Schlossnagle > > http://omniti.com/is/theo-schlossnagle > --------- Brian Calhoun b at mine.com -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: Message signed with OpenPGP using GPGMail URL: