[Reconnoiter-users] Problems with live view

viq viq at viq.ath.cx
Fri Mar 12 06:41:17 EST 2010


On Thu, Mar 11, 2010 at 10:57:23AM -0500, Theo Schlossnagle wrote:
> Okay... more debugging:
> 
> Startup stratcond, connect over console and capture the output of:
> 
> show eventer debug sockets
> 
> Start the streaming and (while it is working) run that console command and capture the output again...
> Wait until it breaks and and run that command and capture the output a third time.
> 
> Send me all three outputs.
 
Before:

stratcon# show eventer debug sockets
  [0x2877cae0] fd: 4 [r-e-] -> noit_listener_acceptor(0x2874e930)
  [0x2877cb40] fd: 5 [r-e-] -> noit_listener_acceptor(0x2874e940)
  [0x2877cc20] fd: 6 [r-e-] -> noit_listener_acceptor(0x2874e950)
  [0x31ee76c0] fd: 9 [r-e-] -> stratcon_jlog_recv_handler(0x287393c0)
  [0x287a03a0] fd: 10 [r-e-] -> noit_console(0x2879f670)

During:

stratcon# show eventer debug sockets
  [0x2877cae0] fd: 4 [r-e-] -> noit_listener_acceptor(0x2874e930)
  [0x2877cb40] fd: 5 [r-e-] -> noit_listener_acceptor(0x2874e940)
  [0x2877cc20] fd: 6 [r-e-] -> noit_listener_acceptor(0x2874e950)
  [0x31ee7a40] fd: 9 [r-e-] -> stratcon_jlog_recv_handler(0x287393c0)
  [0x287a03a0] fd: 10 [r-e-] -> noit_console(0x2879f670)
  [0x2ab450c0] fd: 13 [--e-] -> stratcon_realtime_http(0x2879f7f0)
  [0x31ee7de0] fd: 15 [r-e-] -> 0x8068e50(0x28739600)


Broken:

stratcon# show eventer debug sockets
  [0x2877cae0] fd: 4 [r-e-] -> noit_listener_acceptor(0x2874e930)
  [0x2877cb40] fd: 5 [r-e-] -> noit_listener_acceptor(0x2874e940)
  [0x2877cc20] fd: 6 [r-e-] -> noit_listener_acceptor(0x2874e950)
  [0x31ee7de0] fd: 9 [r-e-] -> stratcon_jlog_recv_handler(0x287393c0)
  [0x287a03a0] fd: 10 [r-e-] -> noit_console(0x2879f670)
  [0x2ab450c0] fd: 13 [--e-] -> stratcon_realtime_http(0x2879f7f0)

Slider moved, updatin again:

stratcon# show eventer debug sockets
  [0x2877cae0] fd: 4 [r-e-] -> noit_listener_acceptor(0x2874e930)
  [0x2877cb40] fd: 5 [r-e-] -> noit_listener_acceptor(0x2874e940)
  [0x2877cc20] fd: 6 [r-e-] -> noit_listener_acceptor(0x2874e950)
  [0x3200d080] fd: 9 [r-e-] -> stratcon_jlog_recv_handler(0x287393c0)
  [0x287a03a0] fd: 10 [r-e-] -> noit_console(0x2879f670)
  [0x2ab450c0] fd: 13 [--e-] -> stratcon_realtime_http(0x2879f7f0)
  [0x287a0440] fd: 15 [--e-] -> stratcon_realtime_http(0x2879f940)
  [0x287a0400] fd: 17 [--e-] -> stratcon_realtime_http(0x2879ff40)
  [0x32022f80] fd: 18 [r-e-] -> 0x8068e50(0x28739900)

Broken again:

stratcon# show eventer debug sockets
  [0x2877cae0] fd: 4 [r-e-] -> noit_listener_acceptor(0x2874e930)
  [0x2877cb40] fd: 5 [r-e-] -> noit_listener_acceptor(0x2874e940)
  [0x2877cc20] fd: 6 [r-e-] -> noit_listener_acceptor(0x2874e950)
  [0x3200d280] fd: 9 [r-e-] -> stratcon_jlog_recv_handler(0x287393c0)
  [0x287a03a0] fd: 10 [r-e-] -> noit_console(0x2879f670)
  [0x2ab450c0] fd: 13 [--e-] -> stratcon_realtime_http(0x2879f7f0)
  [0x287a0440] fd: 15 [--e-] -> stratcon_realtime_http(0x2879f940)
  [0x287a0400] fd: 17 [--e-] -> stratcon_realtime_http(0x2879ff40)

Moving slider again:

stratcon# show eventer debug sockets
  [0x2877cae0] fd: 4 [r-e-] -> noit_listener_acceptor(0x2874e930)
  [0x2877cb40] fd: 5 [r-e-] -> noit_listener_acceptor(0x2874e940)
  [0x2877cc20] fd: 6 [r-e-] -> noit_listener_acceptor(0x2874e950)
  [0x32022980] fd: 9 [r-e-] -> stratcon_jlog_recv_handler(0x287393c0)
  [0x287a03a0] fd: 10 [r-e-] -> noit_console(0x2879f670)
  [0x2ab450c0] fd: 13 [--e-] -> stratcon_realtime_http(0x2879f7f0)
  [0x287a0460] fd: 15 [--e-] -> stratcon_realtime_http(0x2879fb20)
  [0x3200d400] fd: 16 [r-e-] -> 0x8068e50(0x28739600)
  [0x287a0400] fd: 17 [--e-] -> stratcon_realtime_http(0x2879ff40)

Broken again:

stratcon# show eventer debug sockets
  [0x2877cae0] fd: 4 [r-e-] -> noit_listener_acceptor(0x2874e930)
  [0x2877cb40] fd: 5 [r-e-] -> noit_listener_acceptor(0x2874e940)
  [0x2877cc20] fd: 6 [r-e-] -> noit_listener_acceptor(0x2874e950)
  [0x3200d540] fd: 9 [r-e-] -> stratcon_jlog_recv_handler(0x287393c0)
  [0x287a03a0] fd: 10 [r-e-] -> noit_console(0x2879f670)
  [0x2ab450c0] fd: 13 [--e-] -> stratcon_realtime_http(0x2879f7f0)
  [0x287a0460] fd: 15 [--e-] -> stratcon_realtime_http(0x2879fb20)
  [0x287a0400] fd: 17 [--e-] -> stratcon_realtime_http(0x2879ff40)


This is a while after stopping:

stratcon# show eventer debug sockets
  [0x2877cae0] fd: 4 [r-e-] -> noit_listener_acceptor(0x2874e930)
  [0x2877cb40] fd: 5 [r-e-] -> noit_listener_acceptor(0x2874e940)
  [0x2877cc20] fd: 6 [r-e-] -> noit_listener_acceptor(0x2874e950)
  [0x32020140] fd: 9 [r-e-] -> stratcon_jlog_recv_handler(0x287393c0)
  [0x287a03a0] fd: 10 [r-e-] -> noit_console(0x2879f670)
  [0x2ab450c0] fd: 13 [--e-] -> stratcon_realtime_http(0x2879f7f0)
  [0x287a0460] fd: 15 [--e-] -> stratcon_realtime_http(0x2879fb20)
  [0x31cce780] fd: 16 [--e-] -> stratcon_realtime_http(0x2879fc10)
  [0x287a0400] fd: 17 [--e-] -> stratcon_realtime_http(0x2879ff40)


Sorry for the length of it, but I guess it's better to give too much
information that too little ;)

> On Mar 11, 2010, at 8:01 AM, viq wrote:
> 
> > On Wed, Mar 10, 2010 at 10:24:43AM -0500, Theo Schlossnagle wrote:
> >> The real-time streaming shouldn't touch the database at all.  The data flow path (after the a database lookup to find on which noitd the check lives) is all browser <-> stratcond <-> noitd.
> >> 
> >> So, it would appear that stratcond is spiraling out of control.  Likely an issue with an event loop gone wrong.  Unfortunately, also a bit challenging to debug.  I'd turn up debugging all they way and look at the stratcon logs.
> > 
> > Ha, wonders of bzip, I have a 300MB log file compressed to 295kB ;) I
> > can send it if required, but I observed two things:
> > 1) After the graph stops updating, it resumes when I move the update
> > period slider around, but still at 2 second intervals.
> > 2) Even after I stop the live view in the browser it appears stratcon
> > still attempts to send it, I keep seeing in terminal stuff like
> > kqueue: fire on 12/1 to stratcon_realtime_http(0x8067940)
> > -> noit_http_session_drive(12) [1]
> >   -> dispatch(12)
> >   <- dispatch(12) = 4
> > <- noit_http_session_drive(12) [4]
> > 
> > 
> >> On Mar 10, 2010, at 8:10 AM, viq wrote:
> >> 
> >>> Noticed that a while ago, now finally sat down to observe this and
> >>> report.
> >>> Live view works, and draws the graphs as data comes in... for about
> >>> 1/5th of the width of the graph, then just stops. This is happening both
> >>> with Firefox and Chrome. After that stratcond
> >>> together with postgres (and probably ZFS - database on same machine, on
> >>> FreeBSD, database files stored on ZFS) take up most of available CPU:
> >>> CPU: 46.1% user,  0.0% nice, 53.2% system,  0.7% interrupt,  0.0% idle
> >>> A while after killing stratcond (all is running under daemontools so it's
> >>> immediately restarted) system is mostly idle:
> >>> CPU:  9.0% user,  0.0% nice,  6.0% system,  0.4% interrupt, 84.6% idle
> >>> 
> >>> I didn't manage to convince daemontools yet to log anything useful, so I
> >>> could try running stratcond and noitd in terminal if that would help, if
> >>> any more information is needed.
> >>> -- 
> >>> viq
> >>> _______________________________________________
> >>> Reconnoiter-users mailing list
> >>> Reconnoiter-users at lists.omniti.com
> >>> http://lists.omniti.com/mailman/listinfo/reconnoiter-users
> >> 
> >> --
> >> Theo Schlossnagle
> >> http://omniti.com/is/theo-schlossnagle
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> _______________________________________________
> >> Reconnoiter-users mailing list
> >> Reconnoiter-users at lists.omniti.com
> >> http://lists.omniti.com/mailman/listinfo/reconnoiter-users
> > 
> > -- 
> > viq
> > _______________________________________________
> > Reconnoiter-users mailing list
> > Reconnoiter-users at lists.omniti.com
> > http://lists.omniti.com/mailman/listinfo/reconnoiter-users
> 
> --
> Theo Schlossnagle
> http://omniti.com/is/theo-schlossnagle
> 
> 
> 
> 
> 

-- 
viq
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.omniti.com/pipermail/reconnoiter-users/attachments/20100312/50ad4410/attachment.bin 


More information about the Reconnoiter-users mailing list