I've configured tomcat on 2 EC2 instances behind an ELB set for sticky
sessions.
I first experimented with JDBC storage of sessions and it worked OK.
Then I went for memcached. The problem I'm seeing is that every
request is generating a brand new session ID. This is causing the
ELB to switch over to the other host toggling back and forth between
hosts with fresh sessions every request.

I've configured context.xml like this :


<Manager
className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:localhost:11211,n2:10.195.218.100:11211"
failoverNodes="n1"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFactory"
/>


With the coresponding other tomcat similarly. Each EC2 node runs 1
tomcat + 1 memcached and the memcachedNodes point the opposite server.

I am not seeing errors in the log files. Requests are coming through
fine and appear to be being stuck into memcached, except if I watch
the JSESSIONID (with chrome developer tools) it changes on every
request.

Any suggestions ?

Thanks
-david lee

Search Discussions

  • Martin Grotzke at Oct 11, 2011 at 12:18 pm
    Hi David,

    does the ELB stickyness work when you remove the msm configuration?
    If it does the it's really likely that there's an issue

    When you try to get some debug logging from msm you can follow

    http://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration#Configure_logging
    You can post the msm log output from catalina.out (for startup and
    requests) here so that I can have a look at it.

    Just in the case you don't know: msm can also do non-sticky sessions,
    just set sticky="false" in the config.

    Cheers,
    Martin


    On 10/11/2011 02:06 PM, DALDEI wrote:
    I've configured tomcat on 2 EC2 instances behind an ELB set for sticky
    sessions.
    I first experimented with JDBC storage of sessions and it worked OK.
    Then I went for memcached. The problem I'm seeing is that every
    request is generating a brand new session ID. This is causing the
    ELB to switch over to the other host toggling back and forth between
    hosts with fresh sessions every request.

    I've configured context.xml like this :


    <Manager
    className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="n1:localhost:11211,n2:10.195.218.100:11211"
    failoverNodes="n1"
    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

    transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFactory"
    />


    With the coresponding other tomcat similarly. Each EC2 node runs 1
    tomcat + 1 memcached and the memcachedNodes point the opposite server.

    I am not seeing errors in the log files. Requests are coming through
    fine and appear to be being stuck into memcached, except if I watch
    the JSESSIONID (with chrome developer tools) it changes on every
    request.

    Any suggestions ?

    Thanks
    -david lee
  • DALDEI at Oct 11, 2011 at 12:38 pm
    Yes ELB does stickyness correctly if I comment out the manager
    completely, or if I use the JDBC tomcat Store manager.
    But when I add the memcached manager stickyness stops working and
    every request generates a new JSESSIONID cookie with different values
    There's a LOT of debugging ... how much do you want ?
    Here's a snippet ... let me know if I'm at all close ...











    FINE: Session 30DB4BBB41175F758D7288944638C4C9-n2 not found in
    memcached.
    Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.SessionTrackerValve
    invoke
    FINE: >>>>>> Request starting: /odd_admin/Login.jsp ==================
    Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.MemcachedSessionService
    createSession
    FINE: createSession invoked: null
    Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.SessionIdFormat
    createSessionId
    FINE: Creating new session id with orig id
    '09720EC692210B9AB9EF05908961BC01' and memcached id 'n2'.
    Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.MemcachedSessionService
    createSession
    FINE: Created new session with id 09720EC692210B9AB9EF05908961BC01-n2
    Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.BackupSessionService
    backupSession
    FINE: Starting for session id 09720EC692210B9AB9EF05908961BC01-n2
    Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.BackupSessionTask call
    FINE: Starting for session id 09720EC692210B9AB9EF05908961BC01-n2
    Oct 11, 2011 5:01:04 AM
    de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
    FINE: Ignoring attribute 'ODD_NLS_RESOURCE' as it does not implement
    Serializable
    Oct 11, 2011 5:01:04 AM
    de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
    FINE: storing attribute 'characterEncoding' with value 'utf-8'
    Oct 11, 2011 5:01:04 AM
    de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
    FINE: storing attribute 'UserName' with value 'admin'
    Oct 11, 2011 5:01:04 AM
    de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
    FINE: storing attribute 'UserID' with value '1'
    Oct 11, 2011 5:01:04 AM
    de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
    FINE: storing attribute 'sLoginErr' with value 'sendRedirect'
    Oct 11, 2011 5:01:04 AM
    de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
    FINE: storing attribute 'UserRights' with value '3'
    Oct 11, 2011 5:01:04 AM
    de.javakaffee.web.msm.JavaSerializationTranscoder writeAttributes
    FINE: storing attribute 'ODD_LOCALE' with value 'en_US'
    Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.BackupSessionTask
    doBackupSession
    FINE: Trying to store session in memcached:
    09720EC692210B9AB9EF05908961BC01-n2
    Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.SessionTrackerValve
    logDebugRequestSessionCookie
    FINE: Have request session cookie: domain=null, maxAge=-1, path=null,
    value=30DB4BBB41175F758D7288944638C4C9-n2, version=0, secure=false
    Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.SessionTrackerValve
    logDebugResponseCookie
    FINE: Request finished, with Set-Cookie header:
    JSESSIONID=09720EC692210B9AB9EF05908961BC01-n2; Path=/odd_admin;
    Secure
    Oct 11, 2011 5:01:04 AM de.javakaffee.web.msm.SessionTrackerValve
    invoke
    FINE: <<<<<< Request finished: /odd_admin/Login.jsp ==================













    On Oct 11, 8:18 am, Martin Grotzke wrote:
    Hi David,

    does the ELB stickyness work when you remove the msm configuration?
    If it does the it's really likely that there's an issue

    When you try to get some debug logging from msm you can follow

    http://code.google.com/p/memcached-session-manager/wiki/SetupAndConfi...
    You can post the msm log output from catalina.out (for startup and
    requests) here so that I can have a look at it.

    Just in the case you don't know: msm can also do non-sticky sessions,
    just set sticky="false" in the config.

    Cheers,
    Martin

    On 10/11/2011 02:06 PM, DALDEI wrote:






    I've configured tomcat on 2 EC2 instances behind an ELB set for sticky
    sessions.
    I first experimented with JDBC storage of sessions and it worked OK.
    Then I went for memcached.    The problem I'm seeing is that every
    request is generating a brand new session ID.  This is causing the
    ELB to switch over to the other host toggling back and forth between
    hosts with fresh sessions every request.
    I've configured context.xml like this :
    <Manager
    className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="n1:localhost:11211,n2:10.195.218.100:11211"
    failoverNodes="n1"
    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
    transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFa ctory"
    />
    With the coresponding other tomcat similarly.  Each EC2 node runs 1
    tomcat + 1 memcached and the memcachedNodes point the opposite server.
    I am not seeing errors in the log files.  Requests are coming through
    fine and appear to be being stuck into memcached, except if I watch
    the JSESSIONID (with chrome developer tools) it changes on every
    request.
    Any suggestions ?
    Thanks
    -david lee


    signature.asc
    < 1KViewDownload
  • Martin Grotzke at Oct 11, 2011 at 12:57 pm
    It seems as if this was the output of the first request, was it? Or was
    this a request that already had a session?
    Please also the log output of two requests for the same session (logs
    from different tomcats if they're both involved).

    Can you please also post log output when msm starts?

    Can you please configure sessionBackupAsync="false" and try + send logs
    again?

    Cheers,
    Martin
  • DALDEI at Oct 11, 2011 at 1:13 pm
    I couldnt see a way to attach files here so I put them on FTP.
    If you want me to copy them I will but they are 100k each.
    This is the catalina.out from a fresh run of 2 tomcat servers while I
    hit the load balencer trying to login at odd_admin/Login.jsp

    ftp://lee.calldei.com/pub/catalina.out.1
    ftp://lee.calldei.com/pub/catalina.out.2


    Here's my config
    tomcat1:

    <Manager
    className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="n1:localhost:11211,n2:10.195.218.100:11211"
    failoverNodes="n1"
    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

    transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFactory"
    sessionBackupAsync="false"
    />


    tomcat2
    <Manager
    className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="n1:localhost:11211,n2:10.116.79.43:11211"
    failoverNodes="n1"
    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

    transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFactory"
    sessionBackupAsync="false"
    />

    Let me know if there's anything else I can try , and thanks for the
    help !

    -David



    On Oct 11, 8:57 am, Martin Grotzke wrote:
    It seems as if this was the output of the first request, was it? Or was
    this a request that already had a session?
    Please also the log output of two requests for the same session (logs
    from different tomcats if they're both involved).

    Can you please also post log output when msm starts?

    Can you please configure sessionBackupAsync="false" and try + send logs
    again?

    Cheers,
    Martin

    signature.asc
    < 1KViewDownload
  • Martin Grotzke at Oct 11, 2011 at 2:04 pm
    There's an issue with your memcachedNodes config, as n1 and n2 point to
    different memcached servers. memcachedNodes should be the same for both
    tomcats, just with different failoverNodes.

    E.g.
    tomcat1:
    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
    failoverNodes="n1"
    tomcat2:
    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
    failoverNodes="n2"


    There also seems to be an issue with the stickyness configuration. The
    first request was served by tomcat2, from catalina.out.2:
    Oct 11, 2011 6:04:19 AM de.javakaffee.web.msm.SessionTrackerValve
    logDebugResponseCookie
    FINE: Request finished, with Set-Cookie header:
    JSESSIONID=D81FCF8A0B8F939B7C359C8AF8481D49-n2; Path=/odd_admin; Secure

    The next request was served by tomcat1, from catalina.out.1:
    Oct 11, 2011 6:04:26 AM de.javakaffee.web.msm.MemcachedSessionService
    loadFromMemcached
    FINE: Loading session from memcached: D81FCF8A0B8F939B7C359C8AF8481D49-n2

    For this request on tomcat1 a new session was created:
    Oct 11, 2011 6:04:26 AM de.javakaffee.web.msm.SessionTrackerValve
    logDebugResponseCookie
    FINE: Request finished, with Set-Cookie header:
    JSESSIONID=B943B81E6153A6B015FA65363689C6D8-n2; Path=/odd_admin; Secure


    There's also one thing that looks somehow strange: the session cookie
    that was sent to tomcat1 does not have the Path set (which was set as
    you can see from the logged Set-Cookie header from tomcat2) and it was
    not flagged as secure:
    Oct 11, 2011 6:04:26 AM de.javakaffee.web.msm.SessionTrackerValve
    logDebugRequestSessionCookie
    FINE: Have request session cookie: domain=null, maxAge=-1, path=null,
    value=D81FCF8A0B8F939B7C359C8AF8481D49-n2, version=0, secure=false

    Not sure if this is a problem, but perhaps it indicates an issue related
    to SSL/Non-SSL (see e.g.
    https://forums.aws.amazon.com/message.jspa?messageID=201320)


    Another thing that I noticed: you should exclude heartbeat from msm
    tracking by adding it to the requestUriIgnorePattern (s.th. like
    requestUriIgnorePattern="/heartbeat|.*\.(ico|png|gif|jpg|css|js)$").


    Cheers,
    Martin
  • Martin Grotzke at Oct 11, 2011 at 2:11 pm
    Btw, another observation was that you have lots of webapps deployed.
    Shall all be run with msm? If not you should configure msm only for the
    relevant webapps (in META-INF/context.xml or
    conf/[enginename]/[hostname]/[context].xml, see also
    http://tomcat.apache.org/tomcat-6.0-doc/config/context.html#Introduction).

    Cheers,
    Martin
  • DALDEI at Oct 11, 2011 at 3:08 pm
    Thanks for the help ! I was about to give up when I finally got it to
    work.
    The final fix was I was mucking around with the ELB settings trying
    out both app level ("JSESSIONID") sticky cookies and ELB generated
    cookies.
    A complexity is I'm using the SSL terminated on ELB ... I had port 80
    and 443 configured to different sticky settings.
    I set them both to JSESSIONID and then NOTHING worked ... even without
    any special Manager ELB wasnt being sticky !
    So I switched it back to ELB managed cookies then stickyness started
    working again. Then I tried both JDBC and memcached and both are
    working great.

    Need to do some more tests now but the concept is working.

    Question: is there and advantage in this setup of using memcached
    instead of Tomcat clustering (I know I cant use multicast on EC2 but I
    found a way to force the IP for the clustering). What advantage does
    using memcached have over the builtin tomcat clustering ?

    Thanks again for the help !

    Going to try to get off the need for sticky sessions but I still have
    some state information for back-to-back requests (a temp file being
    made ...) that needs stickyness. If the server really fails its OK
    for the user to retry - just not to have to login again.


    On Oct 11, 10:11 am, Martin Grotzke wrote:
    Btw, another observation was that you have lots of webapps deployed.
    Shall all be run with msm? If not you should configure msm only for the
    relevant webapps (in META-INF/context.xml or
    conf/[enginename]/[hostname]/[context].xml, see alsohttp://tomcat.apache.org/tomcat-6.0-doc/config/context.html#Introduction).

    Cheers,
    Martin
  • DALDEI at Oct 11, 2011 at 3:37 pm
    One more question I'm a little confused about.
    I have a 2 server configuration like in the docs for testing.

    Server A: Tomcat 1 , Memcached 1
    Server B : Tomcat 2 , Memcached 2

    The Manager is configured to know both memcached's but "ignore"
    itself. Like you suggested:

    <Manager
    className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="n1:localhost:11211,n2:10.195.218.100:11211"
    failoverNodes="n1"
    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
    transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFa
    ctory"
    sessionBackupAsync="false"
    />
    tomcat2
    <Manager
    className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="n1:localhost:11211,n2:10.116.79.43:11211"
    failoverNodes="n1"
    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
    transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFa
    ctory"
    sessionBackupAsync="false"
    />

    =================





    I tried shutting down the active tomcat (sticky session) and it
    properly failed over to the other tomcat.
    Great. Then I started it up again and waited for it to stabilize.

    Then I shutdown both tomcat and memcached on the same system. This
    would simulate a host going down.
    Failover occured but not session information ! I had to login
    again ...

    My guess is that the state was

    Session -> Active on Tomcat1 , stored on Memcached 2

    When I kill the host (tomcat1 , memcached1 ) the load balencer hits
    Tomcat2 ... But tomcat2 was told to ignore Memcached 2 so it looks in
    Memcached1 and cant find the session !

    How do I arrange for this kind of failover to work ?

    I think I'm missing something fundimental.

    Thanks for any suggestions.

    -David
  • Martin Grotzke at Oct 11, 2011 at 4:38 pm

    On 10/11/2011 05:16 PM, DALDEI wrote:
    One more question I'm a little confused about.
    I have a 2 server configuration like in the docs for testing.

    Server A: Tomcat 1 , Memcached 1
    Server B : Tomcat 2 , Memcached 2

    The Manager is configured to know both memcached's but "ignore"
    itself. Like you suggested:

    <Manager
    className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="n1:localhost:11211,n2:10.195.218.100:11211"
    failoverNodes="n1"
    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
    transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFa
    ctory"
    sessionBackupAsync="false"
    />
    tomcat2
    <Manager
    className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="n1:localhost:11211,n2:10.116.79.43:11211"
    failoverNodes="n1"
    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
    transcoderFactoryClass="de.javakaffee.web.msm.JavaSerializationTranscoderFa
    ctory"
    sessionBackupAsync="false"
    />
    The failoverNodes of tomcat2 should be n2, not n1.

    I tried shutting down the active tomcat (sticky session) and it
    properly failed over to the other tomcat.
    Great. Then I started it up again and waited for it to stabilize.

    Then I shutdown both tomcat and memcached on the same system. This
    would simulate a host going down.
    Right, the previous restart of tomcat1 was not really necessary for
    simulating a server crash...


    Failover occured but not session information ! I had to login
    again ...

    My guess is that the state was

    Session -> Active on Tomcat1 , stored on Memcached 2

    When I kill the host (tomcat1 , memcached1 ) the load balencer hits
    Tomcat2 ... But tomcat2 was told to ignore Memcached 2 so it looks in
    Memcached1 and cant find the session !
    msm only moves the session to another memcached if a memcached goes down
    that should store a session.

    The memcached nodeId is stored in the sessionId, so you can check for
    each request in which memcached the session is stored.

    How do I arrange for this kind of failover to work ?
    Actually it should work. You can send logs so that I can have a look at
    what happened.

    Cheers,
    Martin
  • Martin Grotzke at Oct 11, 2011 at 4:31 pm

    On 10/11/2011 05:08 PM, DALDEI wrote:
    Thanks for the help ! I was about to give up when I finally got it to
    work. Great!
    So I switched it back to ELB managed cookies then stickyness started
    working again. Then I tried both JDBC and memcached and both are
    working great.
    Out of couriosity: how does ELB-managed differ from app-managed? Also in
    ELB-managed mode I'd expect the app to set the JSESSIONID cookie when
    it's needed, as ELB does not know when to set the cookie (and which value).
    Question: is there and advantage in this setup of using memcached
    instead of Tomcat clustering (I know I cant use multicast on EC2 but I
    found a way to force the IP for the clustering). What advantage does
    using memcached have over the builtin tomcat clustering ?
    The advantage is better scalability. Tomcat session replication is an
    all-to-all replication and thus is not very scalable. The backup session
    replication is/was not recommended for production (now this is changed
    to "Downside of the BackupManager: not quite as battle tested as the
    delta manager", see [1]).

    Cheers,
    Martin

    [1] http://tomcat.apache.org/tomcat-6.0-doc/cluster-howto.html
  • Martin Grotzke at Oct 11, 2011 at 4:49 pm
    As I forgot: another advantage of msm is pluggable serialization, you're not
    restricted to java serialization (e.g. kryo is faster and you can provide
    custom serializers which is especially useful for 3rd party classes stored
    in the session).

    Cheers,
    Martin

    Am 11.10.2011 17:08 schrieb "DALDEI" <dlee@calldei.com>:
    Thanks for the help ! I was about to give up when I finally got it to
    work.
    The final fix was I was mucking around with the ELB settings trying
    out both app level ("JSESSIONID") sticky cookies and ELB generated
    cookies.
    A complexity is I'm using the SSL terminated on ELB ... I had port 80
    and 443 configured to different sticky settings.
    I set them both to JSESSIONID and then NOTHING worked ... even without
    any special Manager ELB wasnt being sticky !
    So I switched it back to ELB managed cookies then stickyness started
    working again. Then I tried both JDBC and memcached and both are
    working great.

    Need to do some more tests now but the concept is working.

    Question: is there and advantage in this setup of using memcached
    instead of Tomcat clustering (I know I cant use multicast on EC2 but I
    found a way to force the IP for the clustering). What advantage does
    using memcached have over the builtin tomcat clustering ?

    Thanks again for the help !

    Going to try to get off the need for sticky sessions but I still have
    some state information for back-to-back requests (a temp file being
    made ...) that needs stickyness. If the server really fails its OK
    for the user to retry - just not to have to login again.


    On Oct 11, 10:11 am, Martin Grotzke wrote:
    Btw, another observation was that you have lots of webapps deployed.
    Shall all be run with msm? If not you should configure msm only for the
    relevant webapps (in META-INF/context.xml or
    conf/[enginename]/[hostname]/[context].xml, see alsohttp://
    tomcat.apache.org/tomcat-6.0-doc/config/context.html#Introduction).
    Cheers,
    Martin
  • DALDEI at Oct 11, 2011 at 5:02 pm
    Thanks for all the tips. Custom serialization might be really
    valuable as I'm using for some apps a codebase that does not implement
    Serilalizable !!!!

    Here's links to the 2 logfiles (catalina.out)
    Failure case

    Start both
    Login to 1 - works great
    Kill 1 - wait 30 sec
    Click around - Failover works great !!!
    restart #1 ... kill #2 - wait 30 sec
    click ... requires login (session data must have been lost)


    ftp://lee.calldei.com/pub/catalina.out.1a
    ftp://lee.calldei.com/pub/catalina.out.2a


    As for what ELB differs between ELB sticky cookies and App specified
    cookies ?
    Here's the link ... but I cant get the App specified cookie
    (JSESSIONID) to actually work.
    http://stackoverflow.com/questions/5093309/amazon-load-balancer-sticky-sessions-configuration-for-jsessionid-in-url

    -David
    (again thanks for spending the time on this !)

    On Oct 11, 12:49 pm, Martin Grotzke wrote:
    As I forgot: another advantage of msm is pluggable serialization, you're not
    restricted to java serialization (e.g. kryo is faster and you can provide
    custom serializers which is especially useful for 3rd party classes stored
    in the session).

    Cheers,
    Martin

    Am 11.10.2011 17:08 schrieb "DALDEI" <d...@calldei.com>:






    Thanks for the help ! I was about to give up when I finally got it to
    work.
    The final fix was I was mucking around with the ELB settings trying
    out both app level ("JSESSIONID") sticky cookies and ELB generated
    cookies.
    A complexity is I'm using the SSL terminated on ELB ... I had port 80
    and 443 configured to different sticky settings.
    I set them both to JSESSIONID and then NOTHING worked ... even without
    any special Manager ELB wasnt being sticky !
    So I switched it back to ELB managed cookies then stickyness started
    working again.   Then I tried both JDBC and memcached and both are
    working great.
    Need to do some more tests now but the concept is working.
    Question:  is there and advantage in this setup of using memcached
    instead of Tomcat clustering (I know I cant use multicast on EC2 but I
    found a way to force the IP for the clustering).   What advantage does
    using memcached have over the builtin tomcat clustering ?
    Thanks again for the help !
    Going to try to get off the need for sticky sessions but I still have
    some state information for back-to-back requests (a temp file being
    made ...) that needs stickyness.   If the server really fails its OK
    for the user to retry - just not to have to login again.
    On Oct 11, 10:11 am, Martin Grotzke <martin.grot...@googlemail.com>
    wrote:
    Btw, another observation was that you have lots of webapps deployed.
    Shall all be run with msm? If not you should configure msm only for the
    relevant webapps (in META-INF/context.xml or
    conf/[enginename]/[hostname]/[context].xml, see alsohttp://
    tomcat.apache.org/tomcat-6.0-doc/config/context.html#Introduction).
    Cheers,
    Martin
  • Martin Grotzke at Oct 11, 2011 at 6:54 pm

    On 10/11/2011 07:02 PM, DALDEI wrote:
    Here's links to the 2 logfiles (catalina.out)
    Failure case

    Start both
    Login to 1 - works great
    Kill 1 - wait 30 sec
    Click around - Failover works great !!!
    restart #1 ... kill #2 - wait 30 sec
    click ... requires login (session data must have been lost)
    Can you please be more specific if/when you're killing tomcat, memcached
    or both?

    Again, as written previously in a previous mail: It seems your
    failoverNodes are wrong. The failoverNode should point to the memcached
    running on the same machine. This also explains why your session was
    lost on host crash.

    It should be:
    tomcat1:
    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
    failoverNodes="n1"
    tomcat2:
    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
    failoverNodes="n2"

    ftp://lee.calldei.com/pub/catalina.out.1a
    ftp://lee.calldei.com/pub/catalina.out.2a
    I'm getting "550 Failed to change directory.".

    As for what ELB differs between ELB sticky cookies and App specified
    cookies ?
    Here's the link ... but I cant get the App specified cookie
    (JSESSIONID) to actually work.
    http://stackoverflow.com/questions/5093309/amazon-load-balancer-sticky-sessions-configuration-for-jsessionid-in-url
    Thanx, I'll read it later.

    Cheers,
    Martin
  • DALDEI at Oct 11, 2011 at 11:31 pm
    Hi sorry for the bad links forgot to do the chmod.
    They should work now

    1) when I say "kill tomcat and memcached" I mean on 'server n1' I do
    service tomcat stop
    service memcached stop

    2) as for the config, I have it fixed as you initially mentioned (and
    have for the last 3 posts :).
    The failoverNode refers to the same machine as the current machine
    'aka localhost'.
    For example
    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
    failoverNodes="n1"
    This is on the machine with IP=10.116.79.43

    and
    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
    failoverNodes="n2"
    This is on the machine with
    IP=10.195.218.100




    Thanks !
    -David
  • Martin Grotzke at Oct 12, 2011 at 12:09 am
    Hi David,

    to which tomcat belongs catalina.out.1a? It says

    configured nodes definition
    n1:10.116.79.43:11211,n2:10.195.218.100:11211, failover nodes n2

    and therefore should belong to tomcat2. Is this right?

    Cheers,
    Martin

    On 10/12/2011 01:31 AM, DALDEI wrote:
    Hi sorry for the bad links forgot to do the chmod.
    They should work now

    1) when I say "kill tomcat and memcached" I mean on 'server n1' I do
    service tomcat stop
    service memcached stop

    2) as for the config, I have it fixed as you initially mentioned (and
    have for the last 3 posts :).
    The failoverNode refers to the same machine as the current machine
    'aka localhost'.
    For example
    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
    failoverNodes="n1"
    This is on the machine with IP=10.116.79.43

    and
    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
    failoverNodes="n2"
    This is on the machine with
    IP=10.195.218.100




    Thanks !
    -David
    --
    Dennis Brakhane, Martin Grotzke und Ole Langbehn GbR
    Breitenfelder Str. 13c, 20251 Hamburg
  • DALDEI at Oct 12, 2011 at 10:09 am
    Yes that is right.

    On Oct 11, 8:09 pm, Martin Grotzke wrote:
    Hi David,

    to which tomcat belongs catalina.out.1a? It says

    configured nodes definition
    n1:10.116.79.43:11211,n2:10.195.218.100:11211, failover nodes n2

    and therefore should belong to tomcat2. Is this right?

    Cheers,
    Martin

    On 10/12/2011 01:31 AM, DALDEI wrote:








    Hi sorry for the bad links forgot to do the chmod.
    They should work now
    1) when I say "kill tomcat and memcached" I mean on 'server n1' I do
    service tomcat stop
    service memcached stop
    2) as for the config, I have it fixed as you initially mentioned (and
    have for the last 3 posts :).
    The failoverNode refers to the same machine as the current machine
    'aka localhost'.
    For example
    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
    failoverNodes="n1"
    This is on the machine with IP=10.116.79.43
    and
    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
    failoverNodes="n2"
    This is on the machine with
    IP=10.195.218.100
    Thanks !
    -David
    --
    Dennis Brakhane, Martin Grotzke und Ole Langbehn GbR
    Breitenfelder Str. 13c, 20251 Hamburg
  • Martin Grotzke at Oct 12, 2011 at 7:35 pm
    Now I had a look at the logs, this is what happened:

    1) tomcat2: created session C57F53867C23BECF9EE0F566EB4594CB-n1 (n1 ->
    stored in memcached1)
    2) tomcat2: shutdown
    3) tomcat1: take over C57F53867C23BECF9EE0F566EB4594CB-n1
    4) tomcat1, memcached1: shutdown,
    5) tomcat2: start, cannot connect to memcached1, says: "The node n1 is
    not available, therefore C57F53867C23BECF9EE0F566EB4594CB-n1 cannot be
    loaded from this memcached."

    So you have created the (improbable?) situation that both the tomcat
    holding a session and the memcached that keeps the backup go both down
    at the same moment.

    Even if you had a persistent store this would not have helped you in
    this situation, if it were running on machine1. For this situation a
    membase cluster with servers on machine1 and machine2 and replication
    enabled would be the solution.

    Cheers,
    Martin


    On 10/12/2011 12:09 PM, DALDEI wrote:
    Yes that is right.

    On Oct 11, 8:09 pm, Martin Grotzke wrote:
    Hi David,

    to which tomcat belongs catalina.out.1a? It says

    configured nodes definition
    n1:10.116.79.43:11211,n2:10.195.218.100:11211, failover nodes n2

    and therefore should belong to tomcat2. Is this right?

    Cheers,
    Martin

    On 10/12/2011 01:31 AM, DALDEI wrote:








    Hi sorry for the bad links forgot to do the chmod.
    They should work now
    1) when I say "kill tomcat and memcached" I mean on 'server n1' I do
    service tomcat stop
    service memcached stop
    2) as for the config, I have it fixed as you initially mentioned (and
    have for the last 3 posts :).
    The failoverNode refers to the same machine as the current machine
    'aka localhost'.
    For example
    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
    failoverNodes="n1"
    This is on the machine with IP=10.116.79.43
    and
    memcachedNodes="n1:10.116.79.43:11211,n2:10.195.218.100:11211"
    failoverNodes="n2"
    This is on the machine with
    IP=10.195.218.100
    Thanks !
    -David
  • DALDEI at Oct 13, 2011 at 10:57 am
    Thank you, I appreciate the time you took analizing this and I
    agree ... I am trying to stress the limits in order to understand how
    it works.
    ( you never really know how well something works until you break
    it ! )

    Question: "replication enabled would be the solution. "

    Does msm support replication ? I cant find any reference to it.

    Thanks.

    And thanks again for spending so much time helping me out, I really
    appreciate it (and so will the company when I get it working
    properly !)

    -David
  • Martin Grotzke at Oct 13, 2011 at 11:13 am
    With "replication enabled" I referred to membase. It allows to configure
    replication per bucket.

    Cheers,
    Martin

    Am 13.10.2011 12:56 schrieb "DALDEI" <dlee@calldei.com>:
    Thank you, I appreciate the time you took analizing this and I
    agree ... I am trying to stress the limits in order to understand how
    it works.
    ( you never really know how well something works until you break
    it ! )

    Question: "replication enabled would be the solution. "

    Does msm support replication ? I cant find any reference to it.

    Thanks.

    And thanks again for spending so much time helping me out, I really
    appreciate it (and so will the company when I get it working
    properly !)

    -David

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupmemcached-session-manager @
categoriesmemcached
postedOct 11, '11 at 12:07p
activeOct 13, '11 at 11:13a
posts20
users2
websitememcached.org

2 users in discussion

Martin Grotzke: 11 posts DALDEI: 9 posts

People

Translate

site design / logo © 2022 Grokbase