FAQ
Hi there,

is there anything that would cause camel to ignore messages in a route? I'm getting persistent messages simply disappearing from a transacted route. I've verified the messages arrive at the activemq broker (tcpdump) and also that on one occasion, 5 out of 6 of the messages just disappeared. They didn't make it into the route as the processor didn't see them and the consumer never saw them either. They just vanished. Restarting the broker seems to ease the situation until the next disappearing messages and it gets worse and worse.

thanks,

Alistair

--
mov eax,1
mov ebx,0
int 80h

Search Discussions

  • Mick Knutson at Sep 28, 2011 at 10:22 am
    Do you have a deadLetterChannel setup in case of message exceptions?

    ---
    Thank You…

    Mick Knutson, President

    BASE Logic, Inc.
    Enterprise Architecture, Design, Mentoring & Agile Consulting
    p. (855) BASE-LOGIC: (227-3564-42)
    p. (478) BASE-LOGIC (227-3564-42)
    f. (855) BASE-LOGIC: (227-3564-42)

    Website: http://www.baselogic.com
    Blog: http://www.baselogic.com/blog/
    Linked IN: http://linkedin.com/in/mickknutson
    Twitter: http://twitter.com/mickknutson
    ---


    On Wed, Sep 28, 2011 at 6:16 AM, Alistair Young wrote:

    Hi there,

    is there anything that would cause camel to ignore messages in a route? I'm
    getting persistent messages simply disappearing from a transacted route.
    I've verified the messages arrive at the activemq broker (tcpdump) and also
    that on one occasion, 5 out of 6 of the messages just disappeared. They
    didn't make it into the route as the processor didn't see them and the
    consumer never saw them either. They just vanished. Restarting the broker
    seems to ease the situation until the next disappearing messages and it gets
    worse and worse.

    thanks,

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



  • Alistair Young at Sep 28, 2011 at 10:35 am
    thanks for that. Have set one up now and will watch it.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 11:21, Mick Knutson wrote:

    Do you have a deadLetterChannel setup in case of message exceptions?

    ---
    Thank You…

    Mick Knutson, President

    BASE Logic, Inc.
    Enterprise Architecture, Design, Mentoring & Agile Consulting
    p. (855) BASE-LOGIC: (227-3564-42)
    p. (478) BASE-LOGIC (227-3564-42)
    f. (855) BASE-LOGIC: (227-3564-42)

    Website: http://www.baselogic.com
    Blog: http://www.baselogic.com/blog/
    Linked IN: http://linkedin.com/in/mickknutson
    Twitter: http://twitter.com/mickknutson
    ---


    On Wed, Sep 28, 2011 at 6:16 AM, Alistair Young wrote:

    Hi there,

    is there anything that would cause camel to ignore messages in a route? I'm
    getting persistent messages simply disappearing from a transacted route.
    I've verified the messages arrive at the activemq broker (tcpdump) and also
    that on one occasion, 5 out of 6 of the messages just disappeared. They
    didn't make it into the route as the processor didn't see them and the
    consumer never saw them either. They just vanished. Restarting the broker
    seems to ease the situation until the next disappearing messages and it gets
    worse and worse.

    thanks,

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



  • Alistair Young at Sep 28, 2011 at 10:42 am
    just noticed this:

    Default Error Handler:
    "whenever an Exchange could not be processed the error is propagated back to the client"

    the client sends to topic://A and camel routes to topic://B. If the routing fails will the Default Error Handler tell the client, even though the client doesn't know about topic://B ?

    the client does not receive any errors for the disappearing messages.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 11:21, Mick Knutson wrote:

    Do you have a deadLetterChannel setup in case of message exceptions?

    ---
    Thank You…

    Mick Knutson, President

    BASE Logic, Inc.
    Enterprise Architecture, Design, Mentoring & Agile Consulting
    p. (855) BASE-LOGIC: (227-3564-42)
    p. (478) BASE-LOGIC (227-3564-42)
    f. (855) BASE-LOGIC: (227-3564-42)

    Website: http://www.baselogic.com
    Blog: http://www.baselogic.com/blog/
    Linked IN: http://linkedin.com/in/mickknutson
    Twitter: http://twitter.com/mickknutson
    ---


    On Wed, Sep 28, 2011 at 6:16 AM, Alistair Young wrote:

    Hi there,

    is there anything that would cause camel to ignore messages in a route? I'm
    getting persistent messages simply disappearing from a transacted route.
    I've verified the messages arrive at the activemq broker (tcpdump) and also
    that on one occasion, 5 out of 6 of the messages just disappeared. They
    didn't make it into the route as the processor didn't see them and the
    consumer never saw them either. They just vanished. Restarting the broker
    seems to ease the situation until the next disappearing messages and it gets
    worse and worse.

    thanks,

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



  • Alistair Young at Sep 28, 2011 at 11:01 am
    I now have a dead letter channel which is empty after losing 9 out of 10 messages. I also added a logging handler which logged nothing. Verified the messages arrived at the broker, then they just vanished.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 11:42, Alistair Young wrote:

    just noticed this:

    Default Error Handler:
    "whenever an Exchange could not be processed the error is propagated back to the client"

    the client sends to topic://A and camel routes to topic://B. If the routing fails will the Default Error Handler tell the client, even though the client doesn't know about topic://B ?

    the client does not receive any errors for the disappearing messages.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 11:21, Mick Knutson wrote:

    Do you have a deadLetterChannel setup in case of message exceptions?

    ---
    Thank You…

    Mick Knutson, President

    BASE Logic, Inc.
    Enterprise Architecture, Design, Mentoring & Agile Consulting
    p. (855) BASE-LOGIC: (227-3564-42)
    p. (478) BASE-LOGIC (227-3564-42)
    f. (855) BASE-LOGIC: (227-3564-42)

    Website: http://www.baselogic.com
    Blog: http://www.baselogic.com/blog/
    Linked IN: http://linkedin.com/in/mickknutson
    Twitter: http://twitter.com/mickknutson
    ---


    On Wed, Sep 28, 2011 at 6:16 AM, Alistair Young wrote:

    Hi there,

    is there anything that would cause camel to ignore messages in a route? I'm
    getting persistent messages simply disappearing from a transacted route.
    I've verified the messages arrive at the activemq broker (tcpdump) and also
    that on one occasion, 5 out of 6 of the messages just disappeared. They
    didn't make it into the route as the processor didn't see them and the
    consumer never saw them either. They just vanished. Restarting the broker
    seems to ease the situation until the next disappearing messages and it gets
    worse and worse.

    thanks,

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



  • Marco Westermann at Sep 28, 2011 at 11:09 am
    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10 messages. I also added a logging handler which logged nothing. Verified the messages arrived at the broker, then they just vanished.

    Alistair
  • Alistair Young at Sep 28, 2011 at 11:24 am
    which is the best trace method to use? trace="true", or camelTracer and traceFormatter beans? and where does the log end up? I've tried them all but no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10 messages. I also added a logging handler which logged nothing. Verified the messages arrived at the broker, then they just vanished.

    Alistair
  • Taariq Levack at Sep 29, 2011 at 5:22 am
    Hi

    Where the logs go, if it's logged at all, still depends on your logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq

    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young wrote:

    which is the best trace method to use? trace="true", or camelTracer and
    traceFormatter beans? and where does the log end up? I've tried them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10
    messages. I also added a logging handler which logged nothing. Verified the
    messages arrived at the broker, then they just vanished.
    Alistair
  • Claus Ibsen at Sep 29, 2011 at 7:54 am
    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as possible.

    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq

    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young wrote:

    which is the best trace method to use? trace="true", or camelTracer and
    traceFormatter beans? and where does the log end up? I've tried them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10
    messages. I also added a logging handler which logged nothing. Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Sep 29, 2011 at 8:10 am

    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
    <property name="brokerURL" value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as possible.

    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq

    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young wrote:

    which is the best trace method to use? trace="true", or camelTracer and
    traceFormatter beans? and where does the log end up? I've tried them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10
    messages. I also added a logging handler which logged nothing. Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Claus Ibsen at Sep 29, 2011 at 8:21 am
    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
    <property name="brokerURL" value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as possible.

    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq

    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young wrote:

    which is the best trace method to use? trace="true", or camelTracer and
    traceFormatter beans? and where does the log end up? I've tried them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10
    messages. I also added a logging handler which logged nothing. Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Sep 29, 2011 at 8:37 am
    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
    <property name="brokerURL" value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as possible.

    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq

    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young wrote:

    which is the best trace method to use? trace="true", or camelTracer and
    traceFormatter beans? and where does the log end up? I've tried them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10
    messages. I also added a logging handler which logged nothing. Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Claus Ibsen at Sep 29, 2011 at 9:17 am
    Yes <transacted> should be before the <process>.

    We may add a validation check in the future | Camel 3.0 for that so
    you cannot mis configure it.


    On Thu, Sep 29, 2011 at 10:36 AM, Alistair Young
    wrote:
    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
    <property name="brokerURL" value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as possible.

    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq

    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young wrote:

    which is the best trace method to use? trace="true", or camelTracer and
    traceFormatter beans? and where does the log end up? I've tried them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10
    messages. I also added a logging handler which logged nothing. Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Sep 29, 2011 at 9:18 am
    just noticed a batch of identical 5 messages, three were missing and another single message vanished. tracer logged nothing. No errors, dead letter queue empty.

    One thing that happens is another machine polls the stats topic in activemq every 2mins. Would that cause a problem? It asks for stats on the matrix topic, which is part of the transacted route.

    29 September 2011 10:05:07 - Adding destination : Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id: ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address: vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo {commandId = 1, responseRequired = true, connectionId = ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId = ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null, password = *****, brokerPath = null, brokerMasterConnector = false, manageable = true, clientMaster = true, faultTolerant = false}

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
    <property name="brokerURL" value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as possible.

    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq

    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young wrote:

    which is the best trace method to use? trace="true", or camelTracer and
    traceFormatter beans? and where does the log end up? I've tried them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10
    messages. I also added a logging handler which logged nothing. Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Claus Ibsen at Sep 29, 2011 at 9:24 am
    Hi

    For topics you ought to set an unique clientId and subscription name
    the consumer side.
    http://camel.apache.org/jms
    http://activemq.apache.org/how-do-durable-queues-and-topics-work.html



    On Thu, Sep 29, 2011 at 11:18 AM, Alistair Young
    wrote:
    just noticed a batch of identical 5 messages, three were missing and another single message vanished. tracer logged nothing. No errors, dead letter queue empty.

    One thing that happens is another machine polls the stats topic in activemq every 2mins. Would that cause a problem? It asks for stats on the matrix topic, which is part of the transacted route.

    29 September 2011 10:05:07 - Adding destination : Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id: ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address: vm://matrixBroker#28592
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo {commandId = 1, responseRequired = true, connectionId = ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId = ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null, password = *****, brokerPath = null, brokerMasterConnector = false, manageable = true, clientMaster = true, faultTolerant = false}

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
    <property name="brokerURL" value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as possible.

    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq

    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young wrote:

    which is the best trace method to use? trace="true", or camelTracer and
    traceFormatter beans? and where does the log end up? I've tried them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10
    messages. I also added a logging handler which logged nothing. Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Sep 29, 2011 at 9:24 am
    just saw your info about transacted being before from - will change that and monitor again.

    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing and another single message vanished. tracer logged nothing. No errors, dead letter queue empty.

    One thing that happens is another machine polls the stats topic in activemq every 2mins. Would that cause a problem? It asks for stats on the matrix topic, which is part of the transacted route.

    29 September 2011 10:05:07 - Adding destination : Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id: ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address: vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo {commandId = 1, responseRequired = true, connectionId = ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId = ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null, password = *****, brokerPath = null, brokerMasterConnector = false, manageable = true, clientMaster = true, faultTolerant = false}

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
    <property name="brokerURL" value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as possible.

    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq

    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young wrote:

    which is the best trace method to use? trace="true", or camelTracer and
    traceFormatter beans? and where does the log end up? I've tried them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10
    messages. I also added a logging handler which logged nothing. Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Sep 29, 2011 at 10:52 am
    dunno - nothing works. Random messages are just vanishing once they reach the broker. No trace, no logs, no dead letter queue. Just vanishing. I've removed <transacted /> and <process> but it doesn't help. The producer is a few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT, presumably the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will change that and monitor again.

    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing and another single message vanished. tracer logged nothing. No errors, dead letter queue empty.

    One thing that happens is another machine polls the stats topic in activemq every 2mins. Would that cause a problem? It asks for stats on the matrix topic, which is part of the transacted route.

    29 September 2011 10:05:07 - Adding destination : Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name [null]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id: ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address: vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo {commandId = 1, responseRequired = true, connectionId = ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId = ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null, password = *****, brokerPath = null, brokerMasterConnector = false, manageable = true, clientMaster = true, faultTolerant = false}

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
    <property name="brokerURL" value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as possible.

    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq

    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young wrote:

    which is the best trace method to use? trace="true", or camelTracer and
    traceFormatter beans? and where does the log end up? I've tried them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10
    messages. I also added a logging handler which logged nothing. Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Tim at Sep 29, 2011 at 11:17 am
    Sorry you might have tried this since I haven't been following this thread.
    But can you check your jmx console.
    In particular check 2 things.. the route to see if the number of exchanges
    match what you think and how if any exchanges failed.
    Also check the JMX console on activemq for the queue or topic in question
    and see how many were enqueued vs dispatched.
    Check your deadletter queue from there too

    On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
    wrote:
    dunno - nothing works. Random messages are just vanishing once they reach
    the broker. No trace, no logs, no dead letter queue. Just vanishing. I've
    removed <transacted /> and <process> but it doesn't help. The producer is a
    few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT, presumably
    the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will change that
    and monitor again.
    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing and
    another single message vanished. tracer logged nothing. No errors, dead
    letter queue empty.
    One thing that happens is another machine polls the stats topic in
    activemq every 2mins. Would that cause a problem? It asks for stats on the
    matrix topic, which is part of the transacted route.
    29 September 2011 10:05:07 - Adding destination :
    Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name [null]:
    PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id:
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
    vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
    {commandId = 1, responseRequired = true, connectionId =
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
    ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null,
    password = *****, brokerPath = null, brokerMasterConnector = false,
    manageable = true, clientMaster = true, faultTolerant = false}
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is
    logging so hopefully I can see any errors.
    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory"
    class="org.apache.activemq.ActiveMQConnectionFactory"
    depends-on="matrixBrokerID">
    <property name="brokerURL"
    value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager"
    class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq"
    class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager"
    ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler"
    class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy"
    ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig"
    class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as
    possible.
    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your
    logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq


    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    which is the best trace method to use? trace="true", or
    camelTracer and
    traceFormatter beans? and where does the log end up? I've tried
    them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your
    route.
    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9
    out of 10
    messages. I also added a logging handler which logged nothing.
    Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Sep 29, 2011 at 2:14 pm
    route goes from topicA -> topicB, transacted.
    topicA inflightCount = 96 and increases on each batch of incoming messages
    topicB dispatchCount = enqueueCount

    wondering if the missing messages are connected to topicA inflightCount. I noticed there are two consumers for topicB. The main consumer gets its messages fine. Wonder if the second consumer is a durable topic consumer and therefore activemq is persisting its messages but it hasn't connected in a very long time. Would that cause the topic to get too big? i.e. messages go into the topic until the limit is reached. Main consumer pulls messages off and messages are able to go onto topicB again. Before consumer pulls and after limit reached, messages can't get from topicA -> topicB, hence the topicA inflightCount not zero?

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 12:17, Tim wrote:

    Sorry you might have tried this since I haven't been following this thread.
    But can you check your jmx console.
    In particular check 2 things.. the route to see if the number of exchanges
    match what you think and how if any exchanges failed.
    Also check the JMX console on activemq for the queue or topic in question
    and see how many were enqueued vs dispatched.
    Check your deadletter queue from there too

    On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
    wrote:
    dunno - nothing works. Random messages are just vanishing once they reach
    the broker. No trace, no logs, no dead letter queue. Just vanishing. I've
    removed <transacted /> and <process> but it doesn't help. The producer is a
    few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT, presumably
    the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will change that
    and monitor again.
    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing and
    another single message vanished. tracer logged nothing. No errors, dead
    letter queue empty.
    One thing that happens is another machine polls the stats topic in
    activemq every 2mins. Would that cause a problem? It asks for stats on the
    matrix topic, which is part of the transacted route.
    29 September 2011 10:05:07 - Adding destination :
    Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name [null]:
    PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id:
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
    vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
    {commandId = 1, responseRequired = true, connectionId =
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
    ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null,
    password = *****, brokerPath = null, brokerMasterConnector = false,
    manageable = true, clientMaster = true, faultTolerant = false}
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is
    logging so hopefully I can see any errors.
    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory"
    class="org.apache.activemq.ActiveMQConnectionFactory"
    depends-on="matrixBrokerID">
    <property name="brokerURL"
    value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager"
    class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq"
    class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager"
    ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler"
    class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy"
    ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig"
    class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as
    possible.
    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?


    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <taariql@gmail.com>
    wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your
    logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq


    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    which is the best trace method to use? trace="true", or
    camelTracer and
    traceFormatter beans? and where does the log end up? I've tried
    them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your
    route.
    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9
    out of 10
    messages. I also added a logging handler which logged nothing.
    Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Sep 29, 2011 at 2:42 pm
    nup - cleaned out the data dir and restarted the broker. First message in vanished. Wasn't persisted. So something is fundamentally broken.

    topicA inflightCount = dispatchCount = enqueueCount = 1
    topicB is completely empty

    so the message wasn't persisted, wasn't processed, wasn't routed and just vanished from the broker.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:13, Alistair Young wrote:

    route goes from topicA -> topicB, transacted.
    topicA inflightCount = 96 and increases on each batch of incoming messages
    topicB dispatchCount = enqueueCount

    wondering if the missing messages are connected to topicA inflightCount. I noticed there are two consumers for topicB. The main consumer gets its messages fine. Wonder if the second consumer is a durable topic consumer and therefore activemq is persisting its messages but it hasn't connected in a very long time. Would that cause the topic to get too big? i.e. messages go into the topic until the limit is reached. Main consumer pulls messages off and messages are able to go onto topicB again. Before consumer pulls and after limit reached, messages can't get from topicA -> topicB, hence the topicA inflightCount not zero?

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 12:17, Tim wrote:

    Sorry you might have tried this since I haven't been following this thread.
    But can you check your jmx console.
    In particular check 2 things.. the route to see if the number of exchanges
    match what you think and how if any exchanges failed.
    Also check the JMX console on activemq for the queue or topic in question
    and see how many were enqueued vs dispatched.
    Check your deadletter queue from there too

    On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
    wrote:
    dunno - nothing works. Random messages are just vanishing once they reach
    the broker. No trace, no logs, no dead letter queue. Just vanishing. I've
    removed <transacted /> and <process> but it doesn't help. The producer is a
    few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT, presumably
    the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will change that
    and monitor again.
    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing and
    another single message vanished. tracer logged nothing. No errors, dead
    letter queue empty.
    One thing that happens is another machine polls the stats topic in
    activemq every 2mins. Would that cause a problem? It asks for stats on the
    matrix topic, which is part of the transacted route.
    29 September 2011 10:05:07 - Adding destination :
    Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name [null]:
    PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id:
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
    vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
    {commandId = 1, responseRequired = true, connectionId =
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
    ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null,
    password = *****, brokerPath = null, brokerMasterConnector = false,
    manageable = true, clientMaster = true, faultTolerant = false}
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is
    logging so hopefully I can see any errors.
    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory"
    class="org.apache.activemq.ActiveMQConnectionFactory"
    depends-on="matrixBrokerID">
    <property name="brokerURL"
    value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager"
    class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq"
    class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager"
    ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler"
    class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy"
    ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig"
    class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as
    possible.
    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?


    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <taariql@gmail.com>
    wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your
    logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq


    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    which is the best trace method to use? trace="true", or
    camelTracer and
    traceFormatter beans? and where does the log end up? I've tried
    them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your
    route.
    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9
    out of 10
    messages. I also added a logging handler which logged nothing.
    Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Sep 29, 2011 at 3:06 pm
    I think this way madness lies.

    17 sent to topicA, dispatchCount = 15, dequeueCount = 12
    topicB enqueueCount = 12

    so 17 came in, 12 made it through, of the 5 that went missing it claims to have sent 3 to topicB but they never arrived and the last 2 just simply vanished completely.

    What on earth?

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:41, Alistair Young wrote:

    nup - cleaned out the data dir and restarted the broker. First message in vanished. Wasn't persisted. So something is fundamentally broken.

    topicA inflightCount = dispatchCount = enqueueCount = 1
    topicB is completely empty

    so the message wasn't persisted, wasn't processed, wasn't routed and just vanished from the broker.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:13, Alistair Young wrote:

    route goes from topicA -> topicB, transacted.
    topicA inflightCount = 96 and increases on each batch of incoming messages
    topicB dispatchCount = enqueueCount

    wondering if the missing messages are connected to topicA inflightCount. I noticed there are two consumers for topicB. The main consumer gets its messages fine. Wonder if the second consumer is a durable topic consumer and therefore activemq is persisting its messages but it hasn't connected in a very long time. Would that cause the topic to get too big? i.e. messages go into the topic until the limit is reached. Main consumer pulls messages off and messages are able to go onto topicB again. Before consumer pulls and after limit reached, messages can't get from topicA -> topicB, hence the topicA inflightCount not zero?

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 12:17, Tim wrote:

    Sorry you might have tried this since I haven't been following this thread.
    But can you check your jmx console.
    In particular check 2 things.. the route to see if the number of exchanges
    match what you think and how if any exchanges failed.
    Also check the JMX console on activemq for the queue or topic in question
    and see how many were enqueued vs dispatched.
    Check your deadletter queue from there too

    On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
    wrote:
    dunno - nothing works. Random messages are just vanishing once they reach
    the broker. No trace, no logs, no dead letter queue. Just vanishing. I've
    removed <transacted /> and <process> but it doesn't help. The producer is a
    few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT, presumably
    the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will change that
    and monitor again.
    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing and
    another single message vanished. tracer logged nothing. No errors, dead
    letter queue empty.
    One thing that happens is another machine polls the stats topic in
    activemq every 2mins. Would that cause a problem? It asks for stats on the
    matrix topic, which is part of the transacted route.
    29 September 2011 10:05:07 - Adding destination :
    Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name [null]:
    PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport: vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id:
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
    vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
    {commandId = 1, responseRequired = true, connectionId =
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
    ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName = null,
    password = *****, brokerPath = null, brokerMasterConnector = false,
    manageable = true, clientMaster = true, faultTolerant = false}
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is
    logging so hopefully I can see any errors.
    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory"
    class="org.apache.activemq.ActiveMQConnectionFactory"
    depends-on="matrixBrokerID">
    <property name="brokerURL"
    value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager"
    class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq"
    class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager"
    ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler"
    class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy"
    ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig"
    class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as
    possible.
    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?


    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <taariql@gmail.com>
    wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your
    logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq


    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    which is the best trace method to use? trace="true", or
    camelTracer and
    traceFormatter beans? and where does the log end up? I've tried
    them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your
    route.
    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9
    out of 10
    messages. I also added a logging handler which logged nothing.
    Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Tim at Sep 29, 2011 at 4:32 pm
    Well with only 17 you definitely aren't hitting any prefetch limits or
    anything like that.
    Are you using a connection pool?
    On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young wrote:

    I think this way madness lies.

    17 sent to topicA, dispatchCount = 15, dequeueCount = 12
    topicB enqueueCount = 12

    so 17 came in, 12 made it through, of the 5 that went missing it claims to
    have sent 3 to topicB but they never arrived and the last 2 just simply
    vanished completely.

    What on earth?

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:41, Alistair Young wrote:

    nup - cleaned out the data dir and restarted the broker. First message in
    vanished. Wasn't persisted. So something is fundamentally broken.
    topicA inflightCount = dispatchCount = enqueueCount = 1
    topicB is completely empty

    so the message wasn't persisted, wasn't processed, wasn't routed and just
    vanished from the broker.
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:13, Alistair Young wrote:

    route goes from topicA -> topicB, transacted.
    topicA inflightCount = 96 and increases on each batch of incoming
    messages
    topicB dispatchCount = enqueueCount

    wondering if the missing messages are connected to topicA inflightCount.
    I noticed there are two consumers for topicB. The main consumer gets its
    messages fine. Wonder if the second consumer is a durable topic consumer and
    therefore activemq is persisting its messages but it hasn't connected in a
    very long time. Would that cause the topic to get too big? i.e. messages go
    into the topic until the limit is reached. Main consumer pulls messages off
    and messages are able to go onto topicB again. Before consumer pulls and
    after limit reached, messages can't get from topicA -> topicB, hence the
    topicA inflightCount not zero?
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 12:17, Tim wrote:

    Sorry you might have tried this since I haven't been following this
    thread.
    But can you check your jmx console.
    In particular check 2 things.. the route to see if the number of
    exchanges
    match what you think and how if any exchanges failed.
    Also check the JMX console on activemq for the queue or topic in
    question
    and see how many were enqueued vs dispatched.
    Check your deadletter queue from there too

    On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
    wrote:
    dunno - nothing works. Random messages are just vanishing once they
    reach
    the broker. No trace, no logs, no dead letter queue. Just vanishing.
    I've
    removed <transacted /> and <process> but it doesn't help. The producer
    is a
    few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
    presumably
    the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will change
    that
    and monitor again.
    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing and
    another single message vanished. tracer logged nothing. No errors,
    dead
    letter queue empty.
    One thing that happens is another machine polls the stats topic in
    activemq every 2mins. Would that cause a problem? It asks for stats on
    the
    matrix topic, which is part of the transacted route.
    29 September 2011 10:05:07 - Adding destination :
    Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name
    [null]:
    PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id:
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
    vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
    {commandId = 1, responseRequired = true, connectionId =
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
    ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName =
    null,
    password = *****, brokerPath = null, brokerMasterConnector = false,
    manageable = true, clientMaster = true, faultTolerant = false}
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is
    logging so hopefully I can see any errors.
    <route id="matrix"
    errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory"
    class="org.apache.activemq.ActiveMQConnectionFactory"
    depends-on="matrixBrokerID">
    <property name="brokerURL"
    value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager"
    class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq"
    class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager"
    ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler"
    class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy"
    ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig"
    class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as
    possible.
    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel
    context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
    taariql@gmail.com>
    wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your
    logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging
    FAQ[2]
    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq


    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    which is the best trace method to use? trace="true", or
    camelTracer and
    traceFormatter beans? and where does the log end up? I've
    tried
    them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your
    route.
    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing
    9
    out of 10
    messages. I also added a logging handler which logged nothing.
    Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Sep 29, 2011 at 5:11 pm
    no connection pool. What was disturbing was the first message sent to the broker after a restart and clean out of the data dir, disappeared. There's a similar route on the broker that works fine. The only difference is the producer for the wonky route is on windows and is up to 1min ahead of the broker's clock. Would have thought the timestamplugin would take care of that though. Can see it changing the timestamp in the logs.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 17:31, Tim wrote:

    Well with only 17 you definitely aren't hitting any prefetch limits or
    anything like that.
    Are you using a connection pool?
    On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young wrote:

    I think this way madness lies.

    17 sent to topicA, dispatchCount = 15, dequeueCount = 12
    topicB enqueueCount = 12

    so 17 came in, 12 made it through, of the 5 that went missing it claims to
    have sent 3 to topicB but they never arrived and the last 2 just simply
    vanished completely.

    What on earth?

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:41, Alistair Young wrote:

    nup - cleaned out the data dir and restarted the broker. First message in
    vanished. Wasn't persisted. So something is fundamentally broken.
    topicA inflightCount = dispatchCount = enqueueCount = 1
    topicB is completely empty

    so the message wasn't persisted, wasn't processed, wasn't routed and just
    vanished from the broker.
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:13, Alistair Young wrote:

    route goes from topicA -> topicB, transacted.
    topicA inflightCount = 96 and increases on each batch of incoming
    messages
    topicB dispatchCount = enqueueCount

    wondering if the missing messages are connected to topicA inflightCount.
    I noticed there are two consumers for topicB. The main consumer gets its
    messages fine. Wonder if the second consumer is a durable topic consumer and
    therefore activemq is persisting its messages but it hasn't connected in a
    very long time. Would that cause the topic to get too big? i.e. messages go
    into the topic until the limit is reached. Main consumer pulls messages off
    and messages are able to go onto topicB again. Before consumer pulls and
    after limit reached, messages can't get from topicA -> topicB, hence the
    topicA inflightCount not zero?
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 12:17, Tim wrote:

    Sorry you might have tried this since I haven't been following this
    thread.
    But can you check your jmx console.
    In particular check 2 things.. the route to see if the number of
    exchanges
    match what you think and how if any exchanges failed.
    Also check the JMX console on activemq for the queue or topic in
    question
    and see how many were enqueued vs dispatched.
    Check your deadletter queue from there too

    On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
    wrote:
    dunno - nothing works. Random messages are just vanishing once they
    reach
    the broker. No trace, no logs, no dead letter queue. Just vanishing.
    I've
    removed <transacted /> and <process> but it doesn't help. The producer
    is a
    few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
    presumably
    the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will change
    that
    and monitor again.
    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing and
    another single message vanished. tracer logged nothing. No errors,
    dead
    letter queue empty.
    One thing that happens is another machine polls the stats topic in
    activemq every 2mins. Would that cause a problem? It asks for stats on
    the
    matrix topic, which is part of the transacted route.
    29 September 2011 10:05:07 - Adding destination :
    Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name
    [null]:
    PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id:
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
    vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
    {commandId = 1, responseRequired = true, connectionId =
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
    ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName =
    null,
    password = *****, brokerPath = null, brokerMasterConnector = false,
    manageable = true, clientMaster = true, faultTolerant = false}
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is
    logging so hopefully I can see any errors.
    <route id="matrix"
    errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory"
    class="org.apache.activemq.ActiveMQConnectionFactory"
    depends-on="matrixBrokerID">
    <property name="brokerURL"
    value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager"
    class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq"
    class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager"
    ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler"
    class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy"
    ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig"
    class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as
    possible.
    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel
    context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
    taariql@gmail.com>
    wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your
    logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging
    FAQ[2]
    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq


    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    which is the best trace method to use? trace="true", or
    camelTracer and
    traceFormatter beans? and where does the log end up? I've
    tried
    them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your
    route.
    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing
    9
    out of 10
    messages. I also added a logging handler which logged nothing.
    Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Sep 30, 2011 at 8:13 am
    getting somewhere.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <transacted />
    <process ref="matrixProcessor" />
    <to uri="activemq:topic:edirectory"/>
    </route>

    - producer clock is 2mins ahead of broker clock, timestampplugin enabled on broker
    - no delay between messages at the producer = 62 out of 100 get through the route, consistently
    - 3sec delay at producer, around 88 - 97 get through. That's as good as it gets
    - 5sec delay at producer, 93 made it through the route
    - remove the route and send direct to topic in activemq, no delay, no message loss, consistently

    so the messages are being lost in the route. No matter what the delay in sending messages, some are always lost and vanish. There are no errors, nothing in any dead letter queue. They simply vanish. They don't even make it as far as the <process ref="matrixProcessor" />.

    sending from a ruby producer that blats them out far quicker than the java producer is even worse. Only 1 - 3 ever get through the route. Removing the route and sending to the activemq topic instead, all messages get through no matter how fast they come.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 29 Sep 2011, at 18:10, Alistair Young wrote:

    no connection pool. What was disturbing was the first message sent to the broker after a restart and clean out of the data dir, disappeared. There's a similar route on the broker that works fine. The only difference is the producer for the wonky route is on windows and is up to 1min ahead of the broker's clock. Would have thought the timestamplugin would take care of that though. Can see it changing the timestamp in the logs.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 17:31, Tim wrote:

    Well with only 17 you definitely aren't hitting any prefetch limits or
    anything like that.
    Are you using a connection pool?
    On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young wrote:

    I think this way madness lies.

    17 sent to topicA, dispatchCount = 15, dequeueCount = 12
    topicB enqueueCount = 12

    so 17 came in, 12 made it through, of the 5 that went missing it claims to
    have sent 3 to topicB but they never arrived and the last 2 just simply
    vanished completely.

    What on earth?

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:41, Alistair Young wrote:

    nup - cleaned out the data dir and restarted the broker. First message in
    vanished. Wasn't persisted. So something is fundamentally broken.
    topicA inflightCount = dispatchCount = enqueueCount = 1
    topicB is completely empty

    so the message wasn't persisted, wasn't processed, wasn't routed and just
    vanished from the broker.
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:13, Alistair Young wrote:

    route goes from topicA -> topicB, transacted.
    topicA inflightCount = 96 and increases on each batch of incoming
    messages
    topicB dispatchCount = enqueueCount

    wondering if the missing messages are connected to topicA inflightCount.
    I noticed there are two consumers for topicB. The main consumer gets its
    messages fine. Wonder if the second consumer is a durable topic consumer and
    therefore activemq is persisting its messages but it hasn't connected in a
    very long time. Would that cause the topic to get too big? i.e. messages go
    into the topic until the limit is reached. Main consumer pulls messages off
    and messages are able to go onto topicB again. Before consumer pulls and
    after limit reached, messages can't get from topicA -> topicB, hence the
    topicA inflightCount not zero?
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 12:17, Tim wrote:

    Sorry you might have tried this since I haven't been following this
    thread.
    But can you check your jmx console.
    In particular check 2 things.. the route to see if the number of
    exchanges
    match what you think and how if any exchanges failed.
    Also check the JMX console on activemq for the queue or topic in
    question
    and see how many were enqueued vs dispatched.
    Check your deadletter queue from there too

    On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
    wrote:
    dunno - nothing works. Random messages are just vanishing once they
    reach
    the broker. No trace, no logs, no dead letter queue. Just vanishing.
    I've
    removed <transacted /> and <process> but it doesn't help. The producer
    is a
    few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
    presumably
    the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will change
    that
    and monitor again.
    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing and
    another single message vanished. tracer logged nothing. No errors,
    dead
    letter queue empty.
    One thing that happens is another machine polls the stats topic in
    activemq every 2mins. Would that cause a problem? It asks for stats on
    the
    matrix topic, which is part of the transacted route.
    29 September 2011 10:05:07 - Adding destination :
    Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name
    [null]:
    PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id:
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
    vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
    {commandId = 1, responseRequired = true, connectionId =
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
    ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName =
    null,
    password = *****, brokerPath = null, brokerMasterConnector = false,
    manageable = true, clientMaster = true, faultTolerant = false}
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is
    logging so hopefully I can see any errors.
    <route id="matrix"
    errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory"
    class="org.apache.activemq.ActiveMQConnectionFactory"
    depends-on="matrixBrokerID">
    <property name="brokerURL"
    value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager"
    class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq"
    class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager"
    ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler"
    class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy"
    ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig"
    class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as
    possible.
    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel
    context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
    taariql@gmail.com>
    wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your
    logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging
    FAQ[2]
    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq


    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    which is the best trace method to use? trace="true", or
    camelTracer and
    traceFormatter beans? and where does the log end up? I've
    tried
    them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your
    route.
    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing
    9
    out of 10
    messages. I also added a logging handler which logged nothing.
    Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Sep 30, 2011 at 9:28 am
    I can now reproduce this every time. Sending 100 messages in very quick succession to a camel route. JMX:

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <to uri="activemq:topic:edirectory"/>
    </route>

    EnqueueCount = 100
    DispatchCount = 100
    InFlightCount = 100
    DequeueCount = 1

    only 1 message ever gets through the route. That's 99 percent message loss, every time. This happens on two servers with the same config.

    This happens with the producer on the same machine as the broker as well as a remote broker.

    Is there anything I should be looking at to see where the other 99 messages are? JMX says they're in flight but all dead letter queues are empty and there are no errors anywhere.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 30 Sep 2011, at 09:12, Alistair Young wrote:

    getting somewhere.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <transacted />
    <process ref="matrixProcessor" />
    <to uri="activemq:topic:edirectory"/>
    </route>

    - producer clock is 2mins ahead of broker clock, timestampplugin enabled on broker
    - no delay between messages at the producer = 62 out of 100 get through the route, consistently
    - 3sec delay at producer, around 88 - 97 get through. That's as good as it gets
    - 5sec delay at producer, 93 made it through the route
    - remove the route and send direct to topic in activemq, no delay, no message loss, consistently

    so the messages are being lost in the route. No matter what the delay in sending messages, some are always lost and vanish. There are no errors, nothing in any dead letter queue. They simply vanish. They don't even make it as far as the <process ref="matrixProcessor" />.

    sending from a ruby producer that blats them out far quicker than the java producer is even worse. Only 1 - 3 ever get through the route. Removing the route and sending to the activemq topic instead, all messages get through no matter how fast they come.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 29 Sep 2011, at 18:10, Alistair Young wrote:

    no connection pool. What was disturbing was the first message sent to the broker after a restart and clean out of the data dir, disappeared. There's a similar route on the broker that works fine. The only difference is the producer for the wonky route is on windows and is up to 1min ahead of the broker's clock. Would have thought the timestamplugin would take care of that though. Can see it changing the timestamp in the logs.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 17:31, Tim wrote:

    Well with only 17 you definitely aren't hitting any prefetch limits or
    anything like that.
    Are you using a connection pool?
    On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young wrote:

    I think this way madness lies.

    17 sent to topicA, dispatchCount = 15, dequeueCount = 12
    topicB enqueueCount = 12

    so 17 came in, 12 made it through, of the 5 that went missing it claims to
    have sent 3 to topicB but they never arrived and the last 2 just simply
    vanished completely.

    What on earth?

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:41, Alistair Young wrote:

    nup - cleaned out the data dir and restarted the broker. First message in
    vanished. Wasn't persisted. So something is fundamentally broken.
    topicA inflightCount = dispatchCount = enqueueCount = 1
    topicB is completely empty

    so the message wasn't persisted, wasn't processed, wasn't routed and just
    vanished from the broker.
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:13, Alistair Young wrote:

    route goes from topicA -> topicB, transacted.
    topicA inflightCount = 96 and increases on each batch of incoming
    messages
    topicB dispatchCount = enqueueCount

    wondering if the missing messages are connected to topicA inflightCount.
    I noticed there are two consumers for topicB. The main consumer gets its
    messages fine. Wonder if the second consumer is a durable topic consumer and
    therefore activemq is persisting its messages but it hasn't connected in a
    very long time. Would that cause the topic to get too big? i.e. messages go
    into the topic until the limit is reached. Main consumer pulls messages off
    and messages are able to go onto topicB again. Before consumer pulls and
    after limit reached, messages can't get from topicA -> topicB, hence the
    topicA inflightCount not zero?
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 12:17, Tim wrote:

    Sorry you might have tried this since I haven't been following this
    thread.
    But can you check your jmx console.
    In particular check 2 things.. the route to see if the number of
    exchanges
    match what you think and how if any exchanges failed.
    Also check the JMX console on activemq for the queue or topic in
    question
    and see how many were enqueued vs dispatched.
    Check your deadletter queue from there too

    On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
    wrote:
    dunno - nothing works. Random messages are just vanishing once they
    reach
    the broker. No trace, no logs, no dead letter queue. Just vanishing.
    I've
    removed <transacted /> and <process> but it doesn't help. The producer
    is a
    few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
    presumably
    the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will change
    that
    and monitor again.
    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing and
    another single message vanished. tracer logged nothing. No errors,
    dead
    letter queue empty.
    One thing that happens is another machine polls the stats topic in
    activemq every 2mins. Would that cause a problem? It asks for stats on
    the
    matrix topic, which is part of the transacted route.
    29 September 2011 10:05:07 - Adding destination :
    Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name
    [null]:
    PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id:
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
    vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
    {commandId = 1, responseRequired = true, connectionId =
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId =
    ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName =
    null,
    password = *****, brokerPath = null, brokerMasterConnector = false,
    manageable = true, clientMaster = true, faultTolerant = false}
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is
    logging so hopefully I can see any errors.
    <route id="matrix"
    errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory"
    class="org.apache.activemq.ActiveMQConnectionFactory"
    depends-on="matrixBrokerID">
    <property name="brokerURL"
    value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager"
    class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq"
    class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager"
    ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler"
    class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy"
    ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig"
    class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as
    possible.
    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel
    context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
    taariql@gmail.com>
    wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your
    logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging
    FAQ[2]
    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq


    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    which is the best trace method to use? trace="true", or
    camelTracer and
    traceFormatter beans? and where does the log end up? I've
    tried
    them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your
    route.
    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing
    9
    out of 10
    messages. I also added a logging handler which logged nothing.
    Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Tim at Sep 30, 2011 at 12:23 pm
    Alistair. I just tried the same with an embedded activemq instance and it
    works great.
    Maybe this has to do with broker settings? I can setup a tiny test project
    for you if you want to try that?

    On Fri, Sep 30, 2011 at 11:27 AM, Alistair Young
    wrote:
    I can now reproduce this every time. Sending 100 messages in very quick
    succession to a camel route. JMX:

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <to uri="activemq:topic:edirectory"/>
    </route>

    EnqueueCount = 100
    DispatchCount = 100
    InFlightCount = 100
    DequeueCount = 1

    only 1 message ever gets through the route. That's 99 percent message loss,
    every time. This happens on two servers with the same config.

    This happens with the producer on the same machine as the broker as well as
    a remote broker.

    Is there anything I should be looking at to see where the other 99 messages
    are? JMX says they're in flight but all dead letter queues are empty and
    there are no errors anywhere.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 30 Sep 2011, at 09:12, Alistair Young wrote:

    getting somewhere.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <transacted />
    <process ref="matrixProcessor" />
    <to uri="activemq:topic:edirectory"/>
    </route>

    - producer clock is 2mins ahead of broker clock, timestampplugin enabled on broker
    - no delay between messages at the producer = 62 out of 100 get through
    the route, consistently
    - 3sec delay at producer, around 88 - 97 get through. That's as good as it gets
    - 5sec delay at producer, 93 made it through the route
    - remove the route and send direct to topic in activemq, no delay, no
    message loss, consistently
    so the messages are being lost in the route. No matter what the delay in
    sending messages, some are always lost and vanish. There are no errors,
    nothing in any dead letter queue. They simply vanish. They don't even make
    it as far as the <process ref="matrixProcessor" />.
    sending from a ruby producer that blats them out far quicker than the
    java producer is even worse. Only 1 - 3 ever get through the route. Removing
    the route and sending to the activemq topic instead, all messages get
    through no matter how fast they come.
    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 29 Sep 2011, at 18:10, Alistair Young wrote:

    no connection pool. What was disturbing was the first message sent to
    the broker after a restart and clean out of the data dir, disappeared.
    There's a similar route on the broker that works fine. The only difference
    is the producer for the wonky route is on windows and is up to 1min ahead of
    the broker's clock. Would have thought the timestamplugin would take care of
    that though. Can see it changing the timestamp in the logs.
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 17:31, Tim wrote:

    Well with only 17 you definitely aren't hitting any prefetch limits or
    anything like that.
    Are you using a connection pool?

    On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    I think this way madness lies.

    17 sent to topicA, dispatchCount = 15, dequeueCount = 12
    topicB enqueueCount = 12

    so 17 came in, 12 made it through, of the 5 that went missing it
    claims to
    have sent 3 to topicB but they never arrived and the last 2 just
    simply
    vanished completely.

    What on earth?

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:41, Alistair Young wrote:

    nup - cleaned out the data dir and restarted the broker. First
    message in
    vanished. Wasn't persisted. So something is fundamentally broken.
    topicA inflightCount = dispatchCount = enqueueCount = 1
    topicB is completely empty

    so the message wasn't persisted, wasn't processed, wasn't routed and
    just
    vanished from the broker.
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:13, Alistair Young wrote:

    route goes from topicA -> topicB, transacted.
    topicA inflightCount = 96 and increases on each batch of incoming
    messages
    topicB dispatchCount = enqueueCount

    wondering if the missing messages are connected to topicA
    inflightCount.
    I noticed there are two consumers for topicB. The main consumer gets
    its
    messages fine. Wonder if the second consumer is a durable topic
    consumer and
    therefore activemq is persisting its messages but it hasn't connected
    in a
    very long time. Would that cause the topic to get too big? i.e.
    messages go
    into the topic until the limit is reached. Main consumer pulls
    messages off
    and messages are able to go onto topicB again. Before consumer pulls
    and
    after limit reached, messages can't get from topicA -> topicB, hence
    the
    topicA inflightCount not zero?
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 12:17, Tim wrote:

    Sorry you might have tried this since I haven't been following this
    thread.
    But can you check your jmx console.
    In particular check 2 things.. the route to see if the number of
    exchanges
    match what you think and how if any exchanges failed.
    Also check the JMX console on activemq for the queue or topic in
    question
    and see how many were enqueued vs dispatched.
    Check your deadletter queue from there too

    On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
    wrote:
    dunno - nothing works. Random messages are just vanishing once
    they
    reach
    the broker. No trace, no logs, no dead letter queue. Just
    vanishing.
    I've
    removed <transacted /> and <process> but it doesn't help. The
    producer
    is a
    few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
    presumably
    the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will
    change
    that
    and monitor again.
    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing
    and
    another single message vanished. tracer logged nothing. No errors,
    dead
    letter queue empty.
    One thing that happens is another machine polls the stats topic
    in
    activemq every 2mins. Would that cause a problem? It asks for
    stats on
    the
    matrix topic, which is part of the transacted route.
    29 September 2011 10:05:07 - Adding destination :
    Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name
    [null]:
    PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id:
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
    vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
    {commandId = 1, responseRequired = true, connectionId =
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId
    =
    ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName
    =
    null,
    password = *****, brokerPath = null, brokerMasterConnector =
    false,
    manageable = true, clientMaster = true, faultTolerant = false}
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer
    is
    logging so hopefully I can see any errors.
    <route id="matrix"
    errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory"
    class="org.apache.activemq.ActiveMQConnectionFactory"
    depends-on="matrixBrokerID">
    <property name="brokerURL"
    value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager"
    class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq"
    class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager"
    ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler"
    class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy"
    ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig"
    class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as
    possible.
    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel
    context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
    taariql@gmail.com>
    wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on
    your
    logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel
    logging
    FAQ[2]
    [1]
    http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq


    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    which is the best trace method to use? trace="true", or
    camelTracer and
    traceFormatter beans? and where does the log end up? I've
    tried
    them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in
    your
    route.
    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after
    losing
    9
    out of 10
    messages. I also added a logging handler which logged
    nothing.
    Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Sep 30, 2011 at 12:36 pm
    thanks Tim, that would be very helpful of you. I was about to dive into the camel source to see why the route wouldn't accept anything but if you have time to do a small test project I would be very grateful.

    I'm running camel 2.8.1, activemq 5.5.0, spring 3.0.5 under tomcat. I've attached my pom.xml and camel-config.xml. mvn clean install then deploy to tomcat.

    many thanks,

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h
  • Alistair Young at Oct 2, 2011 at 8:43 am
    eventually found the problem.

    if transactions are enabled nothing works. Only the first message gets through the route, the rest disappear. The problems start by adding transacted and transactionManager to the ActiveMQComponent as per:

    http://camel.apache.org/transactional-client.html

    so I'm not sure how to go about fixing it. Would you have any pointers please?

    <bean id="poolConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
    <property name="maxConnections" value="8"/>
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
    <property name="brokerURL" value="vm://matrixBroker?create=false" />
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="poolConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" >
    <property name="connectionFactory" ref="poolConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 30 Sep 2011, at 13:35, Alistair Young wrote:

    thanks Tim, that would be very helpful of you. I was about to dive into the camel source to see why the route wouldn't accept anything but if you have time to do a small test project I would be very grateful.

    I'm running camel 2.8.1, activemq 5.5.0, spring 3.0.5 under tomcat. I've attached my pom.xml and camel-config.xml. mvn clean install then deploy to tomcat.

    many thanks,

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h
    <camel-config.xml>
    <pom.xml>

    On 30 Sep 2011, at 13:23, Tim wrote:

    Alistair. I just tried the same with an embedded activemq instance and it
    works great.
    Maybe this has to do with broker settings? I can setup a tiny test project
    for you if you want to try that?

    On Fri, Sep 30, 2011 at 11:27 AM, Alistair Young
    wrote:
    I can now reproduce this every time. Sending 100 messages in very quick
    succession to a camel route. JMX:

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <to uri="activemq:topic:edirectory"/>
    </route>

    EnqueueCount = 100
    DispatchCount = 100
    InFlightCount = 100
    DequeueCount = 1

    only 1 message ever gets through the route. That's 99 percent message loss,
    every time. This happens on two servers with the same config.

    This happens with the producer on the same machine as the broker as well as
    a remote broker.

    Is there anything I should be looking at to see where the other 99 messages
    are? JMX says they're in flight but all dead letter queues are empty and
    there are no errors anywhere.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 30 Sep 2011, at 09:12, Alistair Young wrote:

    getting somewhere.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <transacted />
    <process ref="matrixProcessor" />
    <to uri="activemq:topic:edirectory"/>
    </route>

    - producer clock is 2mins ahead of broker clock, timestampplugin enabled on broker
    - no delay between messages at the producer = 62 out of 100 get through
    the route, consistently
    - 3sec delay at producer, around 88 - 97 get through. That's as good as it gets
    - 5sec delay at producer, 93 made it through the route
    - remove the route and send direct to topic in activemq, no delay, no
    message loss, consistently
    so the messages are being lost in the route. No matter what the delay in
    sending messages, some are always lost and vanish. There are no errors,
    nothing in any dead letter queue. They simply vanish. They don't even make
    it as far as the <process ref="matrixProcessor" />.
    sending from a ruby producer that blats them out far quicker than the
    java producer is even worse. Only 1 - 3 ever get through the route. Removing
    the route and sending to the activemq topic instead, all messages get
    through no matter how fast they come.
    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 29 Sep 2011, at 18:10, Alistair Young wrote:

    no connection pool. What was disturbing was the first message sent to
    the broker after a restart and clean out of the data dir, disappeared.
    There's a similar route on the broker that works fine. The only difference
    is the producer for the wonky route is on windows and is up to 1min ahead of
    the broker's clock. Would have thought the timestamplugin would take care of
    that though. Can see it changing the timestamp in the logs.
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 17:31, Tim wrote:

    Well with only 17 you definitely aren't hitting any prefetch limits or
    anything like that.
    Are you using a connection pool?

    On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    I think this way madness lies.

    17 sent to topicA, dispatchCount = 15, dequeueCount = 12
    topicB enqueueCount = 12

    so 17 came in, 12 made it through, of the 5 that went missing it
    claims to
    have sent 3 to topicB but they never arrived and the last 2 just
    simply
    vanished completely.

    What on earth?

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:41, Alistair Young wrote:

    nup - cleaned out the data dir and restarted the broker. First
    message in
    vanished. Wasn't persisted. So something is fundamentally broken.
    topicA inflightCount = dispatchCount = enqueueCount = 1
    topicB is completely empty

    so the message wasn't persisted, wasn't processed, wasn't routed and
    just
    vanished from the broker.
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:13, Alistair Young wrote:

    route goes from topicA -> topicB, transacted.
    topicA inflightCount = 96 and increases on each batch of incoming
    messages
    topicB dispatchCount = enqueueCount

    wondering if the missing messages are connected to topicA
    inflightCount.
    I noticed there are two consumers for topicB. The main consumer gets
    its
    messages fine. Wonder if the second consumer is a durable topic
    consumer and
    therefore activemq is persisting its messages but it hasn't connected
    in a
    very long time. Would that cause the topic to get too big? i.e.
    messages go
    into the topic until the limit is reached. Main consumer pulls
    messages off
    and messages are able to go onto topicB again. Before consumer pulls
    and
    after limit reached, messages can't get from topicA -> topicB, hence
    the
    topicA inflightCount not zero?
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 12:17, Tim wrote:

    Sorry you might have tried this since I haven't been following this
    thread.
    But can you check your jmx console.
    In particular check 2 things.. the route to see if the number of
    exchanges
    match what you think and how if any exchanges failed.
    Also check the JMX console on activemq for the queue or topic in
    question
    and see how many were enqueued vs dispatched.
    Check your deadletter queue from there too

    On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
    wrote:
    dunno - nothing works. Random messages are just vanishing once
    they
    reach
    the broker. No trace, no logs, no dead letter queue. Just
    vanishing.
    I've
    removed <transacted /> and <process> but it doesn't help. The
    producer
    is a
    few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
    presumably
    the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will
    change
    that
    and monitor again.
    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing
    and
    another single message vanished. tracer logged nothing. No errors,
    dead
    letter queue empty.
    One thing that happens is another machine polls the stats topic
    in
    activemq every 2mins. Would that cause a problem? It asks for
    stats on
    the
    matrix topic, which is part of the transacted route.
    29 September 2011 10:05:07 - Adding destination :
    Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name
    [null]:
    PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id:
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
    vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
    {commandId = 1, responseRequired = true, connectionId =
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId
    =
    ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName
    =
    null,
    password = *****, brokerPath = null, brokerMasterConnector =
    false,
    manageable = true, clientMaster = true, faultTolerant = false}
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer
    is
    logging so hopefully I can see any errors.
    <route id="matrix"
    errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory"
    class="org.apache.activemq.ActiveMQConnectionFactory"
    depends-on="matrixBrokerID">
    <property name="brokerURL"
    value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager"
    class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq"
    class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager"
    ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler"
    class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy"
    ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig"
    class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as
    possible.
    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel
    context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
    taariql@gmail.com>
    wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on
    your
    logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel
    logging
    FAQ[2]
    [1]
    http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq


    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    which is the best trace method to use? trace="true", or
    camelTracer and
    traceFormatter beans? and where does the log end up? I've
    tried
    them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in
    your
    route.
    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after
    losing
    9
    out of 10
    messages. I also added a logging handler which logged
    nothing.
    Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Oct 4, 2011 at 7:44 am
    why would adding this:

    <property name="transacted" value="true"/>

    to this:

    org.apache.activemq.camel.component.ActiveMQComponent

    cause the broker to stop working? There are no transacted routes. The faster the messages come in, the more disappear. I have a very simple config that is guaranteed to lose 99/100 messages if the broker is transacted.

    Is there anything special the client has to do? Surely not though, as there are no transacted routes.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 2 Oct 2011, at 09:43, Alistair Young wrote:

    eventually found the problem.

    if transactions are enabled nothing works. Only the first message gets through the route, the rest disappear. The problems start by adding transacted and transactionManager to the ActiveMQComponent as per:

    http://camel.apache.org/transactional-client.html

    so I'm not sure how to go about fixing it. Would you have any pointers please?

    <bean id="poolConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
    <property name="maxConnections" value="8"/>
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
    <property name="brokerURL" value="vm://matrixBroker?create=false" />
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="poolConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" >
    <property name="connectionFactory" ref="poolConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 30 Sep 2011, at 13:35, Alistair Young wrote:

    thanks Tim, that would be very helpful of you. I was about to dive into the camel source to see why the route wouldn't accept anything but if you have time to do a small test project I would be very grateful.

    I'm running camel 2.8.1, activemq 5.5.0, spring 3.0.5 under tomcat. I've attached my pom.xml and camel-config.xml. mvn clean install then deploy to tomcat.

    many thanks,

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h
    <camel-config.xml>
    <pom.xml>

    On 30 Sep 2011, at 13:23, Tim wrote:

    Alistair. I just tried the same with an embedded activemq instance and it
    works great.
    Maybe this has to do with broker settings? I can setup a tiny test project
    for you if you want to try that?

    On Fri, Sep 30, 2011 at 11:27 AM, Alistair Young
    wrote:
    I can now reproduce this every time. Sending 100 messages in very quick
    succession to a camel route. JMX:

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <to uri="activemq:topic:edirectory"/>
    </route>

    EnqueueCount = 100
    DispatchCount = 100
    InFlightCount = 100
    DequeueCount = 1

    only 1 message ever gets through the route. That's 99 percent message loss,
    every time. This happens on two servers with the same config.

    This happens with the producer on the same machine as the broker as well as
    a remote broker.

    Is there anything I should be looking at to see where the other 99 messages
    are? JMX says they're in flight but all dead letter queues are empty and
    there are no errors anywhere.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 30 Sep 2011, at 09:12, Alistair Young wrote:

    getting somewhere.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <transacted />
    <process ref="matrixProcessor" />
    <to uri="activemq:topic:edirectory"/>
    </route>

    - producer clock is 2mins ahead of broker clock, timestampplugin enabled on broker
    - no delay between messages at the producer = 62 out of 100 get through
    the route, consistently
    - 3sec delay at producer, around 88 - 97 get through. That's as good as it gets
    - 5sec delay at producer, 93 made it through the route
    - remove the route and send direct to topic in activemq, no delay, no
    message loss, consistently
    so the messages are being lost in the route. No matter what the delay in
    sending messages, some are always lost and vanish. There are no errors,
    nothing in any dead letter queue. They simply vanish. They don't even make
    it as far as the <process ref="matrixProcessor" />.
    sending from a ruby producer that blats them out far quicker than the
    java producer is even worse. Only 1 - 3 ever get through the route. Removing
    the route and sending to the activemq topic instead, all messages get
    through no matter how fast they come.
    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 29 Sep 2011, at 18:10, Alistair Young wrote:

    no connection pool. What was disturbing was the first message sent to
    the broker after a restart and clean out of the data dir, disappeared.
    There's a similar route on the broker that works fine. The only difference
    is the producer for the wonky route is on windows and is up to 1min ahead of
    the broker's clock. Would have thought the timestamplugin would take care of
    that though. Can see it changing the timestamp in the logs.
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 17:31, Tim wrote:

    Well with only 17 you definitely aren't hitting any prefetch limits or
    anything like that.
    Are you using a connection pool?

    On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    I think this way madness lies.

    17 sent to topicA, dispatchCount = 15, dequeueCount = 12
    topicB enqueueCount = 12

    so 17 came in, 12 made it through, of the 5 that went missing it
    claims to
    have sent 3 to topicB but they never arrived and the last 2 just
    simply
    vanished completely.

    What on earth?

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:41, Alistair Young wrote:

    nup - cleaned out the data dir and restarted the broker. First
    message in
    vanished. Wasn't persisted. So something is fundamentally broken.
    topicA inflightCount = dispatchCount = enqueueCount = 1
    topicB is completely empty

    so the message wasn't persisted, wasn't processed, wasn't routed and
    just
    vanished from the broker.
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:13, Alistair Young wrote:

    route goes from topicA -> topicB, transacted.
    topicA inflightCount = 96 and increases on each batch of incoming
    messages
    topicB dispatchCount = enqueueCount

    wondering if the missing messages are connected to topicA
    inflightCount.
    I noticed there are two consumers for topicB. The main consumer gets
    its
    messages fine. Wonder if the second consumer is a durable topic
    consumer and
    therefore activemq is persisting its messages but it hasn't connected
    in a
    very long time. Would that cause the topic to get too big? i.e.
    messages go
    into the topic until the limit is reached. Main consumer pulls
    messages off
    and messages are able to go onto topicB again. Before consumer pulls
    and
    after limit reached, messages can't get from topicA -> topicB, hence
    the
    topicA inflightCount not zero?
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 12:17, Tim wrote:

    Sorry you might have tried this since I haven't been following this
    thread.
    But can you check your jmx console.
    In particular check 2 things.. the route to see if the number of
    exchanges
    match what you think and how if any exchanges failed.
    Also check the JMX console on activemq for the queue or topic in
    question
    and see how many were enqueued vs dispatched.
    Check your deadletter queue from there too

    On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
    wrote:
    dunno - nothing works. Random messages are just vanishing once
    they
    reach
    the broker. No trace, no logs, no dead letter queue. Just
    vanishing.
    I've
    removed <transacted /> and <process> but it doesn't help. The
    producer
    is a
    few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
    presumably
    the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will
    change
    that
    and monitor again.
    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing
    and
    another single message vanished. tracer logged nothing. No errors,
    dead
    letter queue empty.
    One thing that happens is another machine polls the stats topic
    in
    activemq every 2mins. Would that cause a problem? It asks for
    stats on
    the
    matrix topic, which is part of the transacted route.
    29 September 2011 10:05:07 - Adding destination :
    Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name
    [null]:
    PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id:
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
    vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
    {commandId = 1, responseRequired = true, connectionId =
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId
    =
    ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName
    =
    null,
    password = *****, brokerPath = null, brokerMasterConnector =
    false,
    manageable = true, clientMaster = true, faultTolerant = false}
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer
    is
    logging so hopefully I can see any errors.
    <route id="matrix"
    errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory"
    class="org.apache.activemq.ActiveMQConnectionFactory"
    depends-on="matrixBrokerID">
    <property name="brokerURL"
    value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager"
    class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq"
    class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager"
    ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler"
    class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy"
    ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig"
    class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as
    possible.
    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel
    context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
    taariql@gmail.com>
    wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on
    your
    logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel
    logging
    FAQ[2]
    [1]
    http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq


    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    which is the best trace method to use? trace="true", or
    camelTracer and
    traceFormatter beans? and where does the log end up? I've
    tried
    them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in
    your
    route.
    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after
    losing
    9
    out of 10
    messages. I also added a logging handler which logged
    nothing.
    Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Alistair Young at Oct 4, 2011 at 3:01 pm
    I think camel + tomcat is a non starter. The same config works perfectly in activemq 5.5.0 standalone with camel. Used embedded in a webapp in tomcat 6, only the first message makes it through the route. The rest vanish.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 4 Oct 2011, at 08:43, Alistair Young wrote:

    why would adding this:

    <property name="transacted" value="true"/>

    to this:

    org.apache.activemq.camel.component.ActiveMQComponent

    cause the broker to stop working? There are no transacted routes. The faster the messages come in, the more disappear. I have a very simple config that is guaranteed to lose 99/100 messages if the broker is transacted.

    Is there anything special the client has to do? Surely not though, as there are no transacted routes.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 2 Oct 2011, at 09:43, Alistair Young wrote:

    eventually found the problem.

    if transactions are enabled nothing works. Only the first message gets through the route, the rest disappear. The problems start by adding transacted and transactionManager to the ActiveMQComponent as per:

    http://camel.apache.org/transactional-client.html

    so I'm not sure how to go about fixing it. Would you have any pointers please?

    <bean id="poolConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory">
    <property name="maxConnections" value="8"/>
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
    <property name="brokerURL" value="vm://matrixBroker?create=false" />
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="poolConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent" >
    <property name="connectionFactory" ref="poolConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 30 Sep 2011, at 13:35, Alistair Young wrote:

    thanks Tim, that would be very helpful of you. I was about to dive into the camel source to see why the route wouldn't accept anything but if you have time to do a small test project I would be very grateful.

    I'm running camel 2.8.1, activemq 5.5.0, spring 3.0.5 under tomcat. I've attached my pom.xml and camel-config.xml. mvn clean install then deploy to tomcat.

    many thanks,

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h
    <camel-config.xml>
    <pom.xml>

    On 30 Sep 2011, at 13:23, Tim wrote:

    Alistair. I just tried the same with an embedded activemq instance and it
    works great.
    Maybe this has to do with broker settings? I can setup a tiny test project
    for you if you want to try that?

    On Fri, Sep 30, 2011 at 11:27 AM, Alistair Young
    wrote:
    I can now reproduce this every time. Sending 100 messages in very quick
    succession to a camel route. JMX:

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <to uri="activemq:topic:edirectory"/>
    </route>

    EnqueueCount = 100
    DispatchCount = 100
    InFlightCount = 100
    DequeueCount = 1

    only 1 message ever gets through the route. That's 99 percent message loss,
    every time. This happens on two servers with the same config.

    This happens with the producer on the same machine as the broker as well as
    a remote broker.

    Is there anything I should be looking at to see where the other 99 messages
    are? JMX says they're in flight but all dead letter queues are empty and
    there are no errors anywhere.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 30 Sep 2011, at 09:12, Alistair Young wrote:

    getting somewhere.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <transacted />
    <process ref="matrixProcessor" />
    <to uri="activemq:topic:edirectory"/>
    </route>

    - producer clock is 2mins ahead of broker clock, timestampplugin enabled on broker
    - no delay between messages at the producer = 62 out of 100 get through
    the route, consistently
    - 3sec delay at producer, around 88 - 97 get through. That's as good as it gets
    - 5sec delay at producer, 93 made it through the route
    - remove the route and send direct to topic in activemq, no delay, no
    message loss, consistently
    so the messages are being lost in the route. No matter what the delay in
    sending messages, some are always lost and vanish. There are no errors,
    nothing in any dead letter queue. They simply vanish. They don't even make
    it as far as the <process ref="matrixProcessor" />.
    sending from a ruby producer that blats them out far quicker than the
    java producer is even worse. Only 1 - 3 ever get through the route. Removing
    the route and sending to the activemq topic instead, all messages get
    through no matter how fast they come.
    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 29 Sep 2011, at 18:10, Alistair Young wrote:

    no connection pool. What was disturbing was the first message sent to
    the broker after a restart and clean out of the data dir, disappeared.
    There's a similar route on the broker that works fine. The only difference
    is the producer for the wonky route is on windows and is up to 1min ahead of
    the broker's clock. Would have thought the timestamplugin would take care of
    that though. Can see it changing the timestamp in the logs.
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 17:31, Tim wrote:

    Well with only 17 you definitely aren't hitting any prefetch limits or
    anything like that.
    Are you using a connection pool?

    On Thu, Sep 29, 2011 at 5:05 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    I think this way madness lies.

    17 sent to topicA, dispatchCount = 15, dequeueCount = 12
    topicB enqueueCount = 12

    so 17 came in, 12 made it through, of the 5 that went missing it
    claims to
    have sent 3 to topicB but they never arrived and the last 2 just
    simply
    vanished completely.

    What on earth?

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:41, Alistair Young wrote:

    nup - cleaned out the data dir and restarted the broker. First
    message in
    vanished. Wasn't persisted. So something is fundamentally broken.
    topicA inflightCount = dispatchCount = enqueueCount = 1
    topicB is completely empty

    so the message wasn't persisted, wasn't processed, wasn't routed and
    just
    vanished from the broker.
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 15:13, Alistair Young wrote:

    route goes from topicA -> topicB, transacted.
    topicA inflightCount = 96 and increases on each batch of incoming
    messages
    topicB dispatchCount = enqueueCount

    wondering if the missing messages are connected to topicA
    inflightCount.
    I noticed there are two consumers for topicB. The main consumer gets
    its
    messages fine. Wonder if the second consumer is a durable topic
    consumer and
    therefore activemq is persisting its messages but it hasn't connected
    in a
    very long time. Would that cause the topic to get too big? i.e.
    messages go
    into the topic until the limit is reached. Main consumer pulls
    messages off
    and messages are able to go onto topicB again. Before consumer pulls
    and
    after limit reached, messages can't get from topicA -> topicB, hence
    the
    topicA inflightCount not zero?
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 12:17, Tim wrote:

    Sorry you might have tried this since I haven't been following this
    thread.
    But can you check your jmx console.
    In particular check 2 things.. the route to see if the number of
    exchanges
    match what you think and how if any exchanges failed.
    Also check the JMX console on activemq for the queue or topic in
    question
    and see how many were enqueued vs dispatched.
    Check your deadletter queue from there too

    On Thu, Sep 29, 2011 at 12:52 PM, Alistair Young
    wrote:
    dunno - nothing works. Random messages are just vanishing once
    they
    reach
    the broker. No trace, no logs, no dead letter queue. Just
    vanishing.
    I've
    removed <transacted /> and <process> but it doesn't help. The
    producer
    is a
    few secs behind the broker:

    sent : 11:25:26
    arrived : 11:24:57
    timstamp on message : 1317291897071 = 29 Sep 2011 10:24:57 GMT,
    presumably
    the timestampplugin doing this
    message vanishes

    but all messages display this clock behaviour and not all vanish.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:24, Alistair Young wrote:

    just saw your info about transacted being before from - will
    change
    that
    and monitor again.
    thanks,

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 10:18, Alistair Young wrote:

    just noticed a batch of identical 5 messages, three were missing
    and
    another single message vanished. tracer logged nothing. No errors,
    dead
    letter queue empty.
    One thing that happens is another machine polls the stats topic
    in
    activemq every 2mins. Would that cause a problem? It asks for
    stats on
    the
    matrix topic, which is part of the transacted route.
    29 September 2011 10:05:07 - Adding destination :
    Topic:ActiveMQ.Advisory.Connection
    29 September 2011 10:05:07 - Creating new transaction with name
    [null]:
    PROPAGATION_REQUIRED,ISOLATION_DEFAULT
    29 September 2011 10:05:07 - Stopping connection:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Stopped transport:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Connection Stopped:
    vm://matrixBroker#285916
    29 September 2011 10:05:07 - Setting up new connection id:
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, address:
    vm://matrixBroker#285920
    29 September 2011 10:05:07 - Adding Connection : ConnectionInfo
    {commandId = 1, responseRequired = true, connectionId =
    ID:prodprovisioning-matrix-41707-1317215126074-4:142961, clientId
    =
    ID:prodprovisioning-matrix-41707-1317215126074-5:142961, userName
    =
    null,
    password = *****, brokerPath = null, brokerMasterConnector =
    false,
    manageable = true, clientMaster = true, faultTolerant = false}
    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:36, Alistair Young wrote:

    <transacted/> Should be after <from>
    it is after from - do you mean it should be before?
    <route id="eDirSuccessBroadcast">
    <from uri="activemq:topic:edirectoryprocessed"/>
    <transacted />
    <process ref="groupwiseProcessor" />
    <to uri="activemq:topic:blackboard"/>
    </route>

    thanks for the dead letter tips, will apply them.

    Alistair

    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 09:20, Claus Ibsen wrote:

    Hi

    <transacted/> Should be after <from>


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer
    is
    logging so hopefully I can see any errors.
    <route id="matrix"
    errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory"
    class="org.apache.activemq.ActiveMQConnectionFactory"
    depends-on="matrixBrokerID">
    <property name="brokerURL"
    value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager"
    class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq"
    class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory"
    ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager"
    ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler"
    class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy"
    ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig"
    class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as
    possible.
    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel
    context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack <
    taariql@gmail.com>
    wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on
    your
    logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel
    logging
    FAQ[2]
    [1]
    http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq


    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young <
    alistair.young@uhi.ac.uk>wrote:
    which is the best trace method to use? trace="true", or
    camelTracer and
    traceFormatter beans? and where does the log end up? I've
    tried
    them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in
    your
    route.
    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after
    losing
    9
    out of 10
    messages. I also added a logging handler which logged
    nothing.
    Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/
  • Claus Ibsen at Sep 29, 2011 at 8:23 am
    And its better to use the native redelivery / dead letter queue,
    features of the AMQ message broker, than from Camel.
    http://activemq.apache.org/message-redelivery-and-dlq-handling.html
    http://activemq.apache.org/redelivery-policy.html

    So remove the matrixDeadLetterErrorHandler and use the DLQ from the AMQ instead.


    On Thu, Sep 29, 2011 at 10:09 AM, Alistair Young
    wrote:
    Do you use message expiry? no
    timestamp plugin
    using that

    activemq 5.5.0
    camel 2.8.0
    spring 3.0.5

    noticed sl4j errors on startup, fixed that and now the tracer is logging so hopefully I can see any errors.

    <route id="matrix" errorHandlerRef="matrixDeadLetterErrorHandler">
    <from uri="activemq:topic:matrix"/>
    <process ref="matrixProcessor" />
    <transacted />
    <to uri="activemq:topic:edirectory"/>
    </route>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="matrixBrokerID">
    <property name="brokerURL" value="vm://matrixBroker?create=false"/>
    </bean>

    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    </bean>

    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
    <property name="connectionFactory" ref="jmsConnectionFactory"/>
    <property name="transacted" value="true"/>
    <property name="transactionManager" ref="jmsTransactionManager"/>
    </bean>

    <bean id="matrixDeadLetterErrorHandler" class="org.apache.camel.builder.DeadLetterChannelBuilder">
    <property name="deadLetterUri" value="jms:queue:dead"/>
    <property name="redeliveryPolicy" ref="matrixRedeliveryPolicyConfig"/>
    </bean>

    <bean id="matrixRedeliveryPolicyConfig" class="org.apache.camel.processor.RedeliveryPolicy">
    <property name="maximumRedeliveries" value="10"/>
    <property name="redeliveryDelay" value="250"/>
    </bean>

    thanks,

    Alistair


    --------------
    mov eax,1
    mov ebx,0
    int 80
    On 29 Sep 2011, at 08:53, Claus Ibsen wrote:

    Hi

    Do you use message expiry?
    Make sure clocks between server/clients is synced as much as possible.

    There is a timestamp plugin
    http://activemq.apache.org/timestampplugin.html

    And do you use queue or topic.
    What version of AMQ and Camel are you using?
    And how have you configured the AMQ broker, and the Camel context?

    On Thu, Sep 29, 2011 at 7:21 AM, Taariq Levack wrote:
    Hi

    Where the logs go, if it's logged at all, still depends on your logger and
    how you configured it.

    Here are links to how to enable logging[1] and camel logging FAQ[2]

    [1]http://camel.apache.org/how-do-i-enable-debug-logging.html
    [2]http://camel.apache.org/logging-questions.html

    Taariq

    On Wed, Sep 28, 2011 at 1:23 PM, Alistair Young wrote:

    which is the best trace method to use? trace="true", or camelTracer and
    traceFormatter beans? and where does the log end up? I've tried them all but
    no log appears.

    Alistair

    --
    mov eax,1
    mov ebx,0
    int 80h



    On 28 Sep 2011, at 12:08, Marco Westermann wrote:

    Hi,

    I suggest enable tracing to see exactly what happens in your route.

    regards, Marco

    Am 28.09.2011 13:01, schrieb Alistair Young:
    I now have a dead letter channel which is empty after losing 9 out of 10
    messages. I also added a logging handler which logged nothing. Verified the
    messages arrived at the broker, then they just vanished.
    Alistair


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/


    --
    Claus Ibsen
    -----------------
    FuseSource
    Email: cibsen@fusesource.com
    Web: http://fusesource.com
    Twitter: davsclaus, fusenews
    Blog: http://davsclaus.blogspot.com/
    Author of Camel in Action: http://www.manning.com/ibsen/

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupusers @
categoriescamel
postedSep 28, '11 at 10:17a
activeOct 4, '11 at 3:01p
posts31
users6
websitecamel.apache.org

People

Translate

site design / logo © 2021 Grokbase