FAQ
We have a load balanced application where the proxy does *not* force any
type of sticky session with each backend. The app was running in
production as a single instance (pre load-balanced) and we moved it
behind the proxy to be able to add backends and make it load balanced. I
had the app in test for several weeks as a load-balanced setup using
memcached to store sessions across the 2 backends and I am pretty
confident that the auth areas that relied on session storage were
working fine.



As of last night the app throws errors (in UAT) about every other
request when inside an auth area with the 2 backends enabled. I can't
get the actual message as this time, but it basically was a "Can't use
12345 as a HASHREF..." type of error where '12345' is the ID of the user
I was authenticated as. If I disable one of the backends, clear my
cookies and re-login the entire session works fine. As soon as I
re-enable the other backend I start getting the errors again and it
doesn't matter which backend I use first, they both work independent of
one another. Obviously, the first backend that stores the session works
and the other doesn't, but what I don't know is why.



As I stated above, I'm pretty confident that all the auth areas of the
site were functioning in the load-balanced setup before we went to PROD.
So, I looked at what changed and the only thing that changed related to
session storage was that I added a 2nd memcached backend instance to the
config file. I wondered if this might have been part of the problem, but
I have since dropped back to only one memcached instance, restarted it
and the errors still persist.



Here is the relevant config section:



session:

flash_to_stash: 1

expires: 3600

memcached_new_args:

data:

- 10.xxx.xxx.xxx:11211

namespace: tangeroutlet_tangerweb_session_uat



In an effort to troubleshoot this I have also tried to implement
memcached storage using C::P::Session::Store::Cache and
C::P::Cache::Memcached, but I haven't gotten this to work yet.



I'm hoping that answers to the following questions will guide me to a
solution:



1. What is the acceptable way to use memcached for session storage? Is
it using C::P::Session::Store::Memcached or is it
C::P::Session::Store::Cache in conjunction with C::P::Cache::Memcached?



2. Is memcached even a "good" way to store sessions? DBIC seems to be a
popular option, but I figured memcached's considerable speed would make
it the preferred choice.



3. When using memcached to store sessions, is it OK to have more than
one backend? At first I didn't think anything about adding another
backend, but then I thought that the 2 memcached instances wouldn't have
the same set of data which would cause problems with sessions, unless of
course the session writes get distributed to all the backends. I've
looked into the source of C::P::Session::Store::Memcached,
Cache::Memcached::Managed and Cache::Memcached and cannot tell if the
writes are spread across all backends. Does anyone know the answer to
this?



4. Is it just crazy to run a load balanced setup without some type of
sticky session setup on the proxy? If so, any implementations of this
using Apache 2.x mod_proxy(_balancer) as the frontend would be greatly
appreciated.



At this point I think I'd be willing to take a performance hit and store
sessions in DBIC if it's going to work better in a distributed
environment.



TIA for any input.



v/r



-matt pitts

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.scsys.co.uk/pipermail/catalyst/attachments/20080220/2564fe56/attachment.htm

Search Discussions

  • Lars Balker Rasmussen at Feb 20, 2008 at 6:35 pm

    On Wed, Feb 20, 2008 at 01:08:08PM -0500, Matt Pitts wrote:
    1. What is the acceptable way to use memcached for session storage? Is
    it using C::P::Session::Store::Memcached or is it
    C::P::Session::Store::Cache in conjunction with C::P::Cache::Memcached?
    Well, either, really, they do much the same thing. I wrote CPS::Store::Cache
    to avoid the silliness of having two connections to memcached per backend,
    using two different Cache:: modules.

    So, if you have the cache working, it should be very easy to set up
    sessions.
    2. Is memcached even a "good" way to store sessions? DBIC seems to be a
    popular option, but I figured memcached's considerable speed would make
    it the preferred choice.
    How important are sessions to you? If dropping a session once in a bluemoon
    is acceptable, memcached is perfectly fine. In fact, the memcached process
    is about as stable as the rest of the machine it runs on, which should
    hopefully be very stable.
    3. When using memcached to store sessions, is it OK to have more than
    one backend?
    YES! Definitely. Silly not to.
    At first I didn't think anything about adding another
    backend, but then I thought that the 2 memcached instances wouldn't have
    the same set of data which would cause problems with sessions, unless of
    course the session writes get distributed to all the backends. I've
    looked into the source of C::P::Session::Store::Memcached,
    Cache::Memcached::Managed and Cache::Memcached and cannot tell if the
    writes are spread across all backends. Does anyone know the answer to
    this?
    As long as you specify the servers in the same order on all memcached-
    clients, they will hash the keys the exact same way, and a set() on
    server B will cause a get() on server A to just work(tm).
    4. Is it just crazy to run a load balanced setup without some type of
    sticky session setup on the proxy? If so, any implementations of this
    using Apache 2.x mod_proxy(_balancer) as the frontend would be greatly
    appreciated.
    Sticky sessions are annoying.
    --
    Lars Balker Rasmussen Consult::Perl
  • Matt Pitts at Feb 20, 2008 at 6:59 pm

    -----Original Message-----
    From: Lars Balker Rasmussen
    Sent: Wednesday, February 20, 2008 1:35 PM
    To: The elegant MVC web framework
    Subject: Re: [Catalyst] Distributed session storage problems/questions
    On Wed, Feb 20, 2008 at 01:08:08PM -0500, Matt Pitts wrote:
    1. What is the acceptable way to use memcached for session storage? Is
    it using C::P::Session::Store::Memcached or is it
    C::P::Session::Store::Cache in conjunction with
    C::P::Cache::Memcached?

    Well, either, really, they do much the same thing. I wrote
    CPS::Store::Cache
    to avoid the silliness of having two connections to memcached per
    backend,
    using two different Cache:: modules.

    So, if you have the cache working, it should be very easy to set up
    sessions.
    2. Is memcached even a "good" way to store sessions? DBIC seems to
    be
    a
    popular option, but I figured memcached's considerable speed would make
    it the preferred choice.
    How important are sessions to you? If dropping a session once in a
    bluemoon
    is acceptable, memcached is perfectly fine. In fact, the memcached
    process
    is about as stable as the rest of the machine it runs on, which should
    hopefully be very stable.
    3. When using memcached to store sessions, is it OK to have more
    than
    one backend?
    YES! Definitely. Silly not to.
    At first I didn't think anything about adding another
    backend, but then I thought that the 2 memcached instances wouldn't have
    the same set of data which would cause problems with sessions,
    unless
    of
    course the session writes get distributed to all the backends. I've
    looked into the source of C::P::Session::Store::Memcached,
    Cache::Memcached::Managed and Cache::Memcached and cannot tell if
    the
    writes are spread across all backends. Does anyone know the answer
    to
    this?
    As long as you specify the servers in the same order on all memcached-
    clients, they will hash the keys the exact same way, and a set() on
    server B will cause a get() on server A to just work(tm).
    4. Is it just crazy to run a load balanced setup without some type
    of
    sticky session setup on the proxy? If so, any implementations of
    this
    using Apache 2.x mod_proxy(_balancer) as the frontend would be greatly
    appreciated.
    Sticky sessions are annoying.
    --
    Lars Balker Rasmussen
    Consult::Perl

    _______________________________________________
    List: Catalyst@lists.scsys.co.uk
    Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
    Searchable archive: http://www.mail-
    archive.com/catalyst@lists.scsys.co.uk/
    Dev site: http://dev.catalyst.perl.org/
    This is very, very helpful information. Much thanks.

    v/r

    -matt pitts
  • Carl Johnstone at Feb 21, 2008 at 3:11 pm
    >>>>>
    4. Is it just crazy to run a load balanced setup without some type of
    sticky session setup on the proxy? If so, any implementations of this
    using Apache 2.x mod_proxy(_balancer) as the frontend would be greatly
    appreciated.
    <<<<<

    You should only use "sticky sessions" as a performance enhancer (more likely
    the back-end will have the right stuff for that user in cache) rather than a
    requirement for your app. Once it becomes a requirement your app is no
    longer able to cope with a server failure.

    Carl
  • Jay K at Feb 21, 2008 at 6:34 pm
    Hello Matt,

    This looks like a version mismatch of the
    Catalyst::Authentication::Store::DBIx::Class module on the two machines.

    Recently I added functionality to the module which allow users to
    avoid the DB hit when restoring a user from the session, part of this
    change entailed changing the data stored in the session for a user.

    The old module simply stored the user's ID in the session. The new
    way stores all the user information in the session, and then attempts
    to restore it either using the id field or, if caching is turned on,
    all the fields stored.

    It sounds like one of your machines is using the older version of the
    module and one is using the newer version of the module.

    Bring the modules into sync and the problem should go away. Though I
    expect that at least half of your sessions will need to be reset
    because you probably have 2 different types of data in there now from
    authentication on the two machines.

    Jay

    On Feb 20, 2008, at 11:08 AM, Matt Pitts wrote:

    We have a load balanced application where the proxy does *not* force
    any type of sticky session with each backend. The app was running in
    production as a single instance (pre load-balanced) and we moved it
    behind the proxy to be able to add backends and make it load
    balanced. I had the app in test for several weeks as a load-balanced
    setup using memcached to store sessions across the 2 backends and I
    am pretty confident that the auth areas that relied on session
    storage were working fine.

    As of last night the app throws errors (in UAT) about every other
    request when inside an auth area with the 2 backends enabled. I
    can?t get the actual message as this time, but it basically was a
    ?Can?t use 12345 as a HASHREF?? type of error where ?12345? is the
    ID of the user I was authenticated as. If I disable one of the
    backends, clear my cookies and re-login the entire session works
    fine. As soon as I re-enable the other backend I start getting the
    errors again and it doesn?t matter which backend I use first, they
    both work independent of one another. Obviously, the first backend
    that stores the session works and the other doesn?t, but what I
    don?t know is why.

    As I stated above, I?m pretty confident that all the auth areas of
    the site were functioning in the load-balanced setup before we went
    to PROD. So, I looked at what changed and the only thing that
    changed related to session storage was that I added a 2nd memcached
    backend instance to the config file. I wondered if this might have
    been part of the problem, but I have since dropped back to only one
    memcached instance, restarted it and the errors still persist.

    Here is the relevant config section:

    session:
    flash_to_stash: 1
    expires: 3600
    memcached_new_args:
    data:
    - 10.xxx.xxx.xxx:11211
    namespace: tangeroutlet_tangerweb_session_uat

    In an effort to troubleshoot this I have also tried to implement
    memcached storage using C::P::Session::Store::Cache and
    C::P::Cache::Memcached, but I haven?t gotten this to work yet.

    I?m hoping that answers to the following questions will guide me to
    a solution:

    1. What is the acceptable way to use memcached for session storage?
    Is it using C::P::Session::Store::Memcached or is it
    C::P::Session::Store::Cache in conjunction with
    C::P::Cache::Memcached?

    2. Is memcached even a ?good? way to store sessions? DBIC seems to
    be a popular option, but I figured memcached?s considerable speed
    would make it the preferred choice.

    3. When using memcached to store sessions, is it OK to have more
    than one backend? At first I didn?t think anything about adding
    another backend, but then I thought that the 2 memcached instances
    wouldn?t have the same set of data which would cause problems with
    sessions, unless of course the session writes get distributed to all
    the backends. I?ve looked into the source of
    C::P::Session::Store::Memcached, Cache::Memcached::Managed and
    Cache::Memcached and cannot tell if the writes are spread across all
    backends. Does anyone know the answer to this?

    4. Is it just crazy to run a load balanced setup without some type
    of sticky session setup on the proxy? If so, any implementations of
    this using Apache 2.x mod_proxy(_balancer) as the frontend would be
    greatly appreciated.

    At this point I think I?d be willing to take a performance hit and
    store sessions in DBIC if it?s going to work better in a distributed
    environment.

    TIA for any input.

    v/r

    -matt pitts
    _______________________________________________
    List: Catalyst@lists.scsys.co.uk
    Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
    Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
    Dev site: http://dev.catalyst.perl.org/
    ---
    "May we not return to those scoundrels of old, the illustrious
    founders of superstition and fanaticism, who first took the knife from
    the altar to make victims of those who refused to be their disciples."
    - Voltaire
  • Matt Pitts at Feb 21, 2008 at 7:18 pm

    -----Original Message-----
    From: Jay K
    Sent: Thursday, February 21, 2008 1:35 PM
    To: The elegant MVC web framework
    Subject: Re: [Catalyst] Distributed session storage problems/questions

    Hello Matt,

    This looks like a version mismatch of the
    Catalyst::Authentication::Store::DBIx::Class module on the two
    machines.

    Recently I added functionality to the module which allow users to
    avoid the DB hit when restoring a user from the session, part of this
    change entailed changing the data stored in the session for a user.

    The old module simply stored the user's ID in the session. The new
    way stores all the user information in the session, and then attempts
    to restore it either using the id field or, if caching is turned on,
    all the fields stored.

    It sounds like one of your machines is using the older version of the
    module and one is using the newer version of the module.

    Bring the modules into sync and the problem should go away. Though I
    expect that at least half of your sessions will need to be reset
    because you probably have 2 different types of data in there now from
    authentication on the two machines.

    Jay
    Jay,

    Confirmed!

    One was running:
    Catalyst::Authentication::Store::DBIx::Class 0.104

    And the other:
    Catalyst::Plugin::Authentication::Store::DBIx::Class 0.10

    This is the whole reason I'm moving to PAR-based deployments.

    One instance of the app was on the existing app server and the other had
    been deployed from a PAR that was built on a fresh system on which I had
    installed all the Cat/Perl libraries. After my post to the list I "took
    a step back" reverted back may changes and deployed 2 instances from the
    same PAR and the problem vanished. I haven't yet re-enabled my 2nd
    memcached instance but, based on your post and that of Lars, I'm now
    confident that's not the issue.

    Even though the problem is solved it's great to know what the underlying
    problem was.

    Thanks for the post!

    v/r

    -matt pitts
  • Ashley Pond V at Feb 21, 2008 at 7:33 pm

    On Feb 21, 2008, at 11:18 AM, Matt Pitts wrote:
    One was running:
    Catalyst::Authentication::Store::DBIx::Class 0.104

    And the other:
    Catalyst::Plugin::Authentication::Store::DBIx::Class 0.10

    This is the whole reason I'm moving to PAR-based deployments.
    Another good way to keep this sort of thing from biting besides
    turning it into a PAR is to keep a current Makefile which exactly
    reflects the changes/upgrades. Then when you deploy, you can run
    it and the tests against it and quickly see if your modules are
    out of sync or exactly where your environments differ.

    Parts of Catalyst and DBIC are still moving a bit quickly and
    without being specific about versions you will get breakage.
    [My last one turned out to be out of sync plugin stuff where
    the dev side used the new/better config key pattern of
    "Plugin::..." but it broke in production where things were
    a single version older and only supported the older key style.]

    -Ashley

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcatalyst @
categoriescatalyst, perl
postedFeb 20, '08 at 6:08p
activeFeb 21, '08 at 7:33p
posts7
users5
websitecatalystframework.org
irc#catalyst

People

Translate

site design / logo © 2021 Grokbase