Trying to connect to a suspended ElasticSearch server using Rubberband hangs forever and never returns
I'm trying to debug a very weird issue that is happening on our environments.
I have a Rails app running on ree-1.8.7-2011.12 that is exhibiting an strange behavior when the ElasticSearch server it is connected to is suspended for some reason (to suspend it, start the ES server in foreground then CTRL-Z it), the Ruby process hangs forever and doesn't timeout or anything like that. It doesn't happen if the ES server is suspended before the rails app connects for the first time, it only happens when the Rails app runs some queries and then the ES server decides to stop answering requests.
Since I'm not that good with native code, i don't even know where to start trying to figure this out (and yeah, unfortunately we can't upgrade our Ruby for many reasons right now). Going in the activity monitor and getting a sample of the process I get this thread dump:
So it seems that it's locked at this curl_wait_for_resolv call.
I'm using the latest ElasticSearch and the Rubberband ES adapter for Ruby. This is reproductible on Mac OS and Linux.
Any hints on how to debug this or correctly timeout the server?
And the issue was at Rubberband, I have filed a pull request for it.