[Reconnoiter-devel] [patch] libcurl based http module

Theo Schlossnagle jesus at omniti.com
Fri Sep 11 08:14:09 EDT 2009

On Sep 10, 2009, at 4:54 AM, Paul Querna wrote:

> Hi,
> We have been having some problems with the serf based HTTP client used
> by reconnoiter.
> It crashes, it leaks memory and sockets.
> Even myself being a Serf developer, it is faster to replace this
> component with libcurl, than it is to fix the issues in serf (as nice
> as it would be for serf to be fixed!)
> And so here it is, a libcurl based http module:
> <http://people.apache.org/~pquerna/noit-curl-4eva.patch>
>> From my testing so far, it keeps memory usage flat, and I haven't
> been able to crash it yet hitting a couple thousand random http
> servers. (And SSHD instances running on HTTPS port dont
> crash either.... heh)
> The only change outside of build changes and http.c itself, was the
> removing of a free() inside Eventer.  This use case seems to stem from
> serf not correctly using the eventer API, and since curl actually
> always removes its sockets correctly, this was causing me a double
> free in some cases.  I think (and memory leak checkers agree with me
> so far that I've seen), this can safely be removed.

That free inside the eventer seems needed.  The long long story in  
there wasn't explaining the free, it was explaining the master_fds[e- 
 >fd].e = NULL;

The call back functions (both time and activity) should not have to  
free their own events.  If they return 0, it means that they no longer  
need the event and libeventer will perform the free for them.  This  
change seems to break that usage (which permeates everything).  I've  
no idea why a memory checker doesn't flag that.  This exact code  
exists in each of the other schedulers, so we'll need to get a deep  
grasp of this change before allowing it to metastasize.

> Thoughts on anything that needs to be done to include this in trunk?

Aside from the eventer change -- it looks pretty good.

Theo Schlossnagle
p: +1.443.325.1357 x201   f: +1.410.872.4911

More information about the Reconnoiter-devel mailing list