FAQ

The Solr leader should stop sending requests to the stopped replica once
that replica's live node is removed from ZK (after session expiry).
Fwiw, here's the Zookeeper log entry for a graceful shutdown of the Solr
replica:

2014-12-12 15:04:21,304 [myid:2] - INFO [ProcessThread(sid:2
cport:8181)::PrepRequestProcessor@476] - Processed session termination for
sessionid: 0x34a1701a1df0037

And here's the Zookeeper log entry for a non-graceful shutdown via EC2 stop
or terminate of the replica:

2014-12-12 14:19:22,000 [myid:2] - INFO [SessionTracker:ZooKeeperServer@325]
- Expiring session 0x14a1700c19c003f, timeout of 15000ms exceeded
2014-12-12 14:19:22,001 [myid:2] - INFO [ProcessThread(sid:2
cport:8181)::PrepRequestProcessor@476] - Processed session termination for
sessionid: 0x14a1700c19c003f

There was no hang in the graceful shutdown.
I'm running ZK version 3.4.5 and Solr 4.6.1

Peter



On Fri, Dec 12, 2014 at 3:39 PM, Shalin Shekhar Mangar wrote:

Okay, that should solve the hung threads on the leader.

When you stop the jetty service then it is a graceful shutdown where
existing requests finish before the searcher thread pool is shutdown
completely. A EC2 terminate probably just kills the processes and leader
threads just wait due to a lack of read/connection timeouts.

The Solr leader should stop sending requests to the stopped replica once
that replica's live node is removed from ZK (after session expiry). I think
most of these issues are because of the lack of timeouts. Just add them and
if there are more problems, we can discuss more.
On Fri, Dec 12, 2014 at 8:14 PM, Peter Keegan wrote:

No, I wasn't aware of these. I will give that a try. If I stop the Solr
jetty service manually, things recover fine, but the hang occurs when I
'stop' or 'terminate' the EC2 instance. The Zookeeper leader reports a
15-sec timeout from the stopped node, and expires the session, but the Solr
leader never gets notified. This seems like a bug in ZK.

Thanks,
Peter


On Fri, Dec 12, 2014 at 2:43 PM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:
Do you have distribUpdateConnTimeout and distribUpdateSoTimeout set to
reasonable values in your solr.xml? These are the timeouts used for
inter-shard update requests.

On Fri, Dec 12, 2014 at 2:20 PM, Peter Keegan <peterlkeegan@gmail.com>
wrote:
We are running SolrCloud in AWS and using their auto scaling groups
to
spin
up new Solr replicas when CPU utilization exceeds a threshold for a period
of time. All is well until the replicas are terminated when CPU
utilization
falls below another threshold. What happens is that index updates
sent
to
the Solr leader hang forever in both the Solr leader and the SolrJ
client
app. Searches work fine. Here are 2 thread stack traces from the
Solr
leader and 2 from the client app:

1) Solr-leader thread doing a distributed commit:

Thread 23527: (state = IN_NATIVE)
- java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
byte[],
int, int, int) @bci=0 (Compiled frame; information may be imprecise)
- java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150
(Compiled frame)
- java.net.SocketInputStream.read(byte[], int, int) @bci=11,
line=121
(Compiled frame)
- org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() @bci=71,
line=166 (Compiled frame)
- org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
line=90
(Compiled frame)
-
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
@bci=137, line=281 (Compiled frame)
-
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
@bci=16, line=92 (Compiled frame)
-
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
@bci=2, line=61 (Compiled frame)
- org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=254
(Compiled frame)
-
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
@bci=8, line=289 (Compiled frame)
-
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
@bci=1, line=252 (Compiled frame)
-
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
@bci=6, line=191 (Compiled frame)
-
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
org.apache.http.HttpClientConnection,
org.apache.http.protocol.HttpContext)
@bci=62, line=300 (Compiled frame)
-
org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
org.apache.http.HttpClientConnection,
org.apache.http.protocol.HttpContext)
@bci=60, line=127 (Compiled frame)
-
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
org.apache.http.protocol.HttpContext) @bci=198, line=715 (Compiled
frame)
-
org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
@bci=574, line=520 (Compiled frame)
-
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
@bci=344, line=906 (Compiled frame)
-
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
frame)
-
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
@bci=6, line=784 (Compiled frame)
-
org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
org.apache.solr.client.solrj.ResponseParser) @bci=1175, line=395
(Interpreted frame)
-
org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
@bci=17, line=199 (Interpreted frame)
-
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
@bci=101, line=293 (Compiled frame)
-
org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Req)
@bci=127, line=226 (Interpreted frame)
-
org.apache.solr.update.SolrCmdDistributor.distribCommit(org.apache.solr.update.CommitUpdateCommand,
java.util.List, org.apache.solr.common.params.ModifiableSolrParams)
@bci=112, line=195 (Interpreted frame)
-
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
@bci=174, line=1250 (Interpreted frame)
-
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(org.apache.solr.update.CommitUpdateCommand)
@bci=61, line=157 (Interpreted frame)
-
org.apache.solr.handler.RequestHandlerUtils.handleCommit(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.update.processor.UpdateRequestProcessor,
org.apache.solr.common.params.SolrParams, boolean) @bci=100, line=69
(Interpreted frame)
-
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=60, line=68
(Compiled
frame)
-
org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
(Compiled
frame)
-
org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
(Compiled
frame)
-
org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
org.apache.solr.request.SolrRequestHandler,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
(Compiled
frame)
-
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
@bci=1380, line=417 (Compiled frame)
-
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5, line=201
(Compiled frame)
-
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
-
org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled frame)
-
org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled frame)
-
org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled frame)
-
org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled frame)
-
org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
frame)
- org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled frame)
-
org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled frame)
-
org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
frame)
-
org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled frame)
-
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled frame)
-
org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled frame)
-
org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled frame)
-
org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
@bci=146, line=368 (Compiled frame)
- org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
@bci=323, line=489 (Compiled frame)
- org.eclipse.jetty.server.BlockingHttpConnection.handleRequest() @bci=1,
line=53 (Interpreted frame)
-
org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
@bci=13, line=953 (Interpreted frame)
-
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
@bci=5, line=1014 (Interpreted frame)
- org.eclipse.jetty.http.HttpParser.parseNext() @bci=3948, line=921
(Compiled frame)
- org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
line=240
(Compiled frame)
- org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
line=72 (Compiled frame)
-
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
@bci=100, line=264 (Compiled frame)
-
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
@bci=1, line=608 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47, line=543
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)

2) Solr-leader thread doing a delete by query:

Thread 19580: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
information
may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object)
@bci=14,
line=186 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=834 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
int) @bci=67, line=867 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int)
@bci=17, line=1197 (Compiled frame)
- java.util.concurrent.locks.ReentrantLock$NonfairSync.lock()
@bci=21,
line=214 (Compiled frame)
- java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290
(Compiled frame)
-
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished()
@bci=81, line=391 (Compiled frame)
- org.apache.solr.update.StreamingSolrServers.blockUntilFinished()
@bci=35, line=98 (Compiled frame)
- org.apache.solr.update.SolrCmdDistributor.finish() @bci=8, line=62
(Compiled frame)
-
org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(org.apache.solr.update.DeleteUpdateCommand)
@bci=1088, line=1088 (Compiled frame)
-
org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
@bci=14, line=825 (Compiled frame)
-
org.apache.solr.update.processor.LogUpdateProcessor.processDelete(org.apache.solr.update.DeleteUpdateCommand)
@bci=61, line=121 (Compiled frame)
-
org.apache.solr.handler.loader.JavabinLoader.delete(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.client.solrj.request.UpdateRequest,
org.apache.solr.update.processor.UpdateRequestProcessor) @bci=208, line=150
(Compiled frame)
-
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse, java.io.InputStream,
org.apache.solr.update.processor.UpdateRequestProcessor) @bci=68, line=111
(Compiled frame)
-
org.apache.solr.handler.loader.JavabinLoader.load(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse,
org.apache.solr.common.util.ContentStream,
org.apache.solr.update.processor.UpdateRequestProcessor) @bci=18,
line=58
(Compiled frame)
-
org.apache.solr.handler.UpdateRequestHandler$1.load(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse,
org.apache.solr.common.util.ContentStream,
org.apache.solr.update.processor.UpdateRequestProcessor) @bci=165, line=92
(Compiled frame)
-
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=129, line=74
(Compiled
frame)
-
org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=43, line=135
(Compiled
frame)
-
org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859
(Compiled
frame)
-
org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
org.apache.solr.request.SolrRequestHandler,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=31, line=721
(Compiled
frame)
-
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean)
@bci=1380, line=417 (Compiled frame)
-
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5, line=201
(Compiled frame)
-
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=100, line=1419 (Compiled frame)
-
org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Compiled frame)
-
org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled frame)
-
org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled frame)
-
org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Compiled frame)
-
org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Compiled
frame)
- org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Compiled frame)
-
org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Compiled frame)
-
org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Compiled
frame)
-
org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled frame)
-
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled frame)
-
org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled frame)
-
org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
org.eclipse.jetty.server.Request,
javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled frame)
-
org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
@bci=146, line=368 (Compiled frame)
- org.eclipse.jetty.server.AbstractHttpConnection.handleRequest()
@bci=323, line=489 (Compiled frame)
- org.eclipse.jetty.http.HttpParser.parseNext() @bci=3597, line=866
(Compiled frame)
- org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51,
line=240
(Compiled frame)
- org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51,
line=72 (Compiled frame)
-
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
@bci=100, line=264 (Compiled frame)
-
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
@bci=1, line=608 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47, line=543
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)

3) SolrJ client thread doing commit:

Thread 1872: (state = IN_NATIVE)
- java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
byte[],
int, int, int) @bci=0 (Compiled frame; information may be imprecise)
- java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150
(Compiled frame)
- java.net.SocketInputStream.read(byte[], int, int) @bci=11,
line=121
(Compiled frame)
- org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() @bci=71,
line=166 (Compiled frame)
- org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
line=90
(Compiled frame)
-
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
@bci=137, line=281 (Compiled frame)
-
org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
@bci=5, line=115 (Compiled frame)
-
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
@bci=16, line=92 (Compiled frame)
-
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
@bci=2, line=62 (Compiled frame)
- org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=254
(Compiled frame)
-
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
@bci=8, line=289 (Compiled frame)
-
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
@bci=1, line=252 (Compiled frame)
-
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
@bci=6, line=191 (Compiled frame)
-
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
org.apache.http.HttpClientConnection,
org.apache.http.protocol.HttpContext)
@bci=73, line=301 (Compiled frame)
-
org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
org.apache.http.HttpClientConnection,
org.apache.http.protocol.HttpContext)
@bci=60, line=127 (Compiled frame)
-
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
frame)
-
org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
@bci=597, line=522 (Compiled frame)
-
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
@bci=344, line=906 (Compiled frame)
-
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
frame)
-
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
@bci=6, line=784 (Compiled frame)
-
org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396 (Compiled
frame)
-
org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
@bci=17, line=199 (Compiled frame)
-
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
@bci=132, line=285 (Compiled frame)
-
org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
@bci=838, line=640 (Compiled frame)
-
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
@bci=17, line=117 (Interpreted frame)
- org.apache.solr.client.solrj.SolrServer.commit(boolean, boolean)
@bci=16, line=168 (Interpreted frame)
- org.apache.solr.client.solrj.SolrServer.commit() @bci=3, line=146
(Interpreted frame)

4) SolrJ client thread doing delete by query:

Thread 1873: (state = IN_NATIVE)
- java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
byte[],
int, int, int) @bci=0 (Compiled frame; information may be imprecise)
- java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150
(Compiled frame)
- java.net.SocketInputStream.read(byte[], int, int) @bci=11,
line=121
(Compiled frame)
- org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer() @bci=71,
line=166 (Compiled frame)
- org.apache.http.impl.io.SocketInputBuffer.fillBuffer() @bci=1,
line=90
(Compiled frame)
-
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
@bci=137, line=281 (Compiled frame)
-
org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(org.apache.http.util.CharArrayBuffer)
@bci=5, line=115 (Compiled frame)
-
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
@bci=16, line=92 (Compiled frame)
-
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(org.apache.http.io.SessionInputBuffer)
@bci=2, line=62 (Compiled frame)
- org.apache.http.impl.io.AbstractMessageParser.parse() @bci=38, line=254
(Compiled frame)
-
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader()
@bci=8, line=289 (Compiled frame)
-
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader()
@bci=1, line=252 (Compiled frame)
-
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader()
@bci=6, line=191 (Compiled frame)
-
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(org.apache.http.HttpRequest,
org.apache.http.HttpClientConnection,
org.apache.http.protocol.HttpContext)
@bci=73, line=301 (Compiled frame)
-
org.apache.http.protocol.HttpRequestExecutor.execute(org.apache.http.HttpRequest,
org.apache.http.HttpClientConnection,
org.apache.http.protocol.HttpContext)
@bci=60, line=127 (Compiled frame)
-
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(org.apache.http.impl.client.RoutedRequest,
org.apache.http.protocol.HttpContext) @bci=198, line=717 (Compiled
frame)
-
org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost,
org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
@bci=597, line=522 (Compiled frame)
-
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.HttpHost,
org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext)
@bci=344, line=906 (Compiled frame)
-
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest,
org.apache.http.protocol.HttpContext) @bci=21, line=805 (Compiled
frame)
-
org.apache.http.impl.client.AbstractHttpClient.execute(org.apache.http.client.methods.HttpUriRequest)
@bci=6, line=784 (Compiled frame)
-
org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest,
org.apache.solr.client.solrj.ResponseParser) @bci=1184, line=396 (Compiled
frame)
-
org.apache.solr.client.solrj.impl.HttpSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
@bci=17, line=199 (Compiled frame)
-
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(org.apache.solr.client.solrj.impl.LBHttpSolrServer$Req)
@bci=132, line=285 (Compiled frame)
-
org.apache.solr.client.solrj.impl.CloudSolrServer.request(org.apache.solr.client.solrj.SolrRequest)
@bci=838, line=640 (Compiled frame)
-
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(org.apache.solr.client.solrj.SolrServer)
@bci=17, line=117 (Compiled frame)
-
org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String,
int) @bci=21, line=285 (Compiled frame)
-
org.apache.solr.client.solrj.SolrServer.deleteByQuery(java.lang.String)
@bci=3, line=271 (Compiled frame)

It looks as if the Solr-leader threads are waiting on a response from
the
terminated replicas, but I would have expected an IOException to
occur.
Any
ideas on what may be happening?

Thanks,
Peter


--
Regards,
Shalin Shekhar Mangar.


--
Regards,
Shalin Shekhar Mangar.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 8 of 15 | next ›
Discussion Overview
groupsolr-user @
categorieslucene
postedDec 12, '14 at 2:21p
activeDec 16, '14 at 12:15p
posts15
users3
websitelucene.apache.org...

People

Translate

site design / logo © 2021 Grokbase