FAQ
Hello!

I'm experiencing a problem with a Catalyst 5.65 installation under
Ubuntu Linux 5.10. All the tables on the MySQL database are marked
UTF-8, and the collation is utf8_general (as I always set it).

I'm using DBIx::Class, and I loaded the Unicode plugin in Catalyst.
However, when I read something from the SQL database, the string gets
converted to latin1, so the accents (widely used in the Italian
language) are not readable anymore - well, unless I switch the pages to
latin1, but I'd like to use UTF-8.

Any hints?

Thanks,
Michele.

--
Michele Beltrame
http://www.varlogarthas.net/
ICQ# 76660101
Informativa privacy: http://www.italpro.net/em.html

Search Discussions

  • Matt S Trout at Mar 2, 2006 at 3:50 pm

    On Thu, Mar 02, 2006 at 04:35:37PM +0100, Michele Beltrame wrote:
    Hello!

    I'm experiencing a problem with a Catalyst 5.65 installation under
    Ubuntu Linux 5.10. All the tables on the MySQL database are marked
    UTF-8, and the collation is utf8_general (as I always set it).

    I'm using DBIx::Class, and I loaded the Unicode plugin in Catalyst.
    However, when I read something from the SQL database, the string gets
    converted to latin1, so the accents (widely used in the Italian
    language) are not readable anymore - well, unless I switch the pages to
    latin1, but I'd like to use UTF-8.
    A prototype of a DBIx::Class::UTF8Columns component was posted to the
    dbix-class list today.

    http://lists.rawmode.org/pipermail/dbix-class/2006-March/000837.html

    --
    Matt S Trout Offering custom development, consultancy and support
    Technical Director contracts for Catalyst, DBIx::Class and BAST. Contact
    Shadowcat Systems Ltd. mst (at) shadowcatsystems.co.uk for more information

    + Help us build a better perl ORM: http://dbix-class.shadowcatsystems.co.uk/ +
  • Daniel McBrearty at Mar 2, 2006 at 3:49 pm
    may not be related to your problem, but make sure you are issuing the "set
    names utf8" command to mysql at teh start of the session. This has bit my
    ass and caused me much headaches in the past ...


    http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html

    if you don't do this, mysql may take it upon itself to convert streams to
    some non utf8 form, even though you think you have told it really thoroughly
    that it shouldn't.


    On 3/2/06, Matt S Trout wrote:
    On Thu, Mar 02, 2006 at 04:35:37PM +0100, Michele Beltrame wrote:
    Hello!

    I'm experiencing a problem with a Catalyst 5.65 installation under
    Ubuntu Linux 5.10. All the tables on the MySQL database are marked
    UTF-8, and the collation is utf8_general (as I always set it).

    I'm using DBIx::Class, and I loaded the Unicode plugin in Catalyst.
    However, when I read something from the SQL database, the string gets
    converted to latin1, so the accents (widely used in the Italian
    language) are not readable anymore - well, unless I switch the pages to
    latin1, but I'd like to use UTF-8.
    A prototype of a DBIx::Class::UTF8Columns component was posted to the
    dbix-class list today.

    http://lists.rawmode.org/pipermail/dbix-class/2006-March/000837.html

    --
    Matt S Trout Offering custom development, consultancy and
    support
    Technical Director contracts for Catalyst, DBIx::Class and BAST.
    Contact
    Shadowcat Systems Ltd. mst (at) shadowcatsystems.co.uk for more
    information

    + Help us build a better perl ORM:
    http://dbix-class.shadowcatsystems.co.uk/ +

    _______________________________________________
    Catalyst mailing list
    Catalyst at lists.rawmode.org
    http://lists.rawmode.org/mailman/listinfo/catalyst


    --
    Daniel McBrearty
    email : danielmcbrearty at gmail.com
    www.engoi.com : the multi - language vocab trainer
    BTW : 0873928131
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rawmode.org/pipermail/catalyst/attachments/20060302/cf69d770/attachment.htm
  • Matt S Trout at Mar 2, 2006 at 4:07 pm

    On Thu, Mar 02, 2006 at 04:49:13PM +0100, Daniel McBrearty wrote:
    may not be related to your problem, but make sure you are issuing the "set
    names utf8" command to mysql at teh start of the session. This has bit my
    ass and caused me much headaches in the past ...


    http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html

    if you don't do this, mysql may take it upon itself to convert streams to
    some non utf8 form, even though you think you have told it really thoroughly
    that it shouldn't.
    In your DBIC schema you can add

    __PACKAGE__->storage->on_connect_do([ "set names utf8" ]);

    to ensure this gets run immediately after the $dbh is (re-)connected.

    --
    Matt S Trout Offering custom development, consultancy and support
    Technical Director contracts for Catalyst, DBIx::Class and BAST. Contact
    Shadowcat Systems Ltd. mst (at) shadowcatsystems.co.uk for more information

    + Help us build a better perl ORM: http://dbix-class.shadowcatsystems.co.uk/ +
  • Michele Beltrame at Mar 2, 2006 at 4:32 pm
    Hello!
    In your DBIC schema you can add

    __PACKAGE__->storage->on_connect_do([ "set names utf8" ]);

    to ensure this gets run immediately after the $dbh is (re-)connected.
    All the examples I found around regarding Catalyst were using the "old"
    object-based calls, and not the new schema-based ones.

    Is there a way to perform this operation the old way? Or, better yet, is
    there some documentation or good example on how to use the schema-based
    thing under Catalyst?

    Thanks, Michele.

    --
    Michele Beltrame
    http://www.varlogarthas.net/
    ICQ# 76660101
    Informativa privacy: http://www.italpro.net/em.html
  • Geoff Richards at Mar 2, 2006 at 4:35 pm

    On Thu, Mar 02, 2006 at 04:07:54PM +0000, Matt S Trout wrote:
    On Thu, Mar 02, 2006 at 04:49:13PM +0100, Daniel McBrearty wrote:
    may not be related to your problem, but make sure you are issuing the "set
    names utf8" command to mysql at teh start of the session. This has bit my
    ass and caused me much headaches in the past ...


    http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html

    if you don't do this, mysql may take it upon itself to convert streams to
    some non utf8 form, even though you think you have told it really thoroughly
    that it shouldn't.
    In your DBIC schema you can add

    __PACKAGE__->storage->on_connect_do([ "set names utf8" ]);

    to ensure this gets run immediately after the $dbh is (re-)connected.
    Note that DBD::mysql will not set the 'UTF8' flag on the strings, so
    you have to be careful what you do with them or they'll get screwed up.

    http://rt.cpan.org/Public/Bug/Display.html?id829

    --

    --- Geoff Richards -------------><-------------- http://ungwe.org/ ---
    "I tried to fling my shadow at the moon,
    The while my blood leapt with a wordless song." -- Theodore Roethke
  • Michele Beltrame at Mar 3, 2006 at 10:10 am
    Hello Matt!
    __PACKAGE__->storage->on_connect_do([ "set names utf8" ]);

    to ensure this gets run immediately after the $dbh is (re-)connected.
    At last I was able to migrate one of my Catalyst application to
    DBIC::Schema. However, putting this query in the schema class does no
    good for the UTF-8 thing.

    How can I be sure it is executed? I tried replacing it with something
    invalid such as "set nameddswgs utf8" (which returns an error in
    mysqlclient), but I get no error in Catalyst. That's why I ask this
    question. ;-)

    Thanks, Michele.

    --
    Michele Beltrame
    http://www.varlogarthas.net/
    ICQ# 76660101
    Informativa privacy: http://www.italpro.net/em.html
  • Paul Makepeace at Mar 3, 2006 at 10:14 am

    Je 2006-03-03 10:10:49 +0000, Michele Beltrame skribis:
    Hello Matt!
    __PACKAGE__->storage->on_connect_do([ "set names utf8" ]);

    to ensure this gets run immediately after the $dbh is (re-)connected.
    At last I was able to migrate one of my Catalyst application to
    DBIC::Schema. However, putting this query in the schema class does no
    good for the UTF-8 thing.
    As an anecdotal datapoint, I didn't find this to be of use; it seemed to
    garble the output. At the same time, I didn't look too hard at it.

    Matt - did you suggest this off the back of my experimentation on IRC
    or have you actually had it work?
    How can I be sure it is executed? I tried replacing it with something
    Put
    DBI->trace(1);

    early on, before you perform a query.

    P

    --
    Paul Makepeace .............................. http://paulm.com/inchoate/

    "What is edmontonians Map? One leg is both the same."
    -- http://paulm.com/toys/surrealism/
  • Matt S Trout at Mar 3, 2006 at 12:27 pm

    On Fri, Mar 03, 2006 at 10:14:54AM +0000, Paul Makepeace wrote:
    Je 2006-03-03 10:10:49 +0000, Michele Beltrame skribis:
    Hello Matt!
    __PACKAGE__->storage->on_connect_do([ "set names utf8" ]);

    to ensure this gets run immediately after the $dbh is (re-)connected.
    At last I was able to migrate one of my Catalyst application to
    DBIC::Schema. However, putting this query in the schema class does no
    good for the UTF-8 thing.
    As an anecdotal datapoint, I didn't find this to be of use; it seemed to
    garble the output. At the same time, I didn't look too hard at it.
    I know on_connect_do works. I didn't suggest the 'set names utf8' bit this
    time round and I'm not familiar with the problem myself.
    Matt - did you suggest this off the back of my experimentation on IRC
    or have you actually had it work?
    Somebody else suggested it; I'm more hopeful about the UTF8Columns component
    that Daisuke (typester) is working on.

    --
    Matt S Trout Offering custom development, consultancy and support
    Technical Director contracts for Catalyst, DBIx::Class and BAST. Contact
    Shadowcat Systems Ltd. mst (at) shadowcatsystems.co.uk for more information

    + Help us build a better perl ORM: http://dbix-class.shadowcatsystems.co.uk/ +
  • Daniel McBrearty at Mar 3, 2006 at 3:12 pm
    I suggested the "set names utf8" thing. That's because I have personal
    experience of mysql doing just what the OP described, and this turned out to
    be the fix.

    I have no idea if or how catalyst makes this call ... but I just know that
    SOMETHING has to ...

    On 3/3/06, Matt S Trout wrote:
    On Fri, Mar 03, 2006 at 10:14:54AM +0000, Paul Makepeace wrote:
    Je 2006-03-03 10:10:49 +0000, Michele Beltrame skribis:
    Hello Matt!
    __PACKAGE__->storage->on_connect_do([ "set names utf8" ]);

    to ensure this gets run immediately after the $dbh is
    (re-)connected.
    At last I was able to migrate one of my Catalyst application to
    DBIC::Schema. However, putting this query in the schema class does no
    good for the UTF-8 thing.
    As an anecdotal datapoint, I didn't find this to be of use; it seemed to
    garble the output. At the same time, I didn't look too hard at it.
    I know on_connect_do works. I didn't suggest the 'set names utf8' bit this
    time round and I'm not familiar with the problem myself.
    Matt - did you suggest this off the back of my experimentation on IRC
    or have you actually had it work?
    Somebody else suggested it; I'm more hopeful about the UTF8Columns
    component
    that Daisuke (typester) is working on.

    --
    Matt S Trout Offering custom development, consultancy and
    support
    Technical Director contracts for Catalyst, DBIx::Class and BAST.
    Contact
    Shadowcat Systems Ltd. mst (at) shadowcatsystems.co.uk for more
    information

    + Help us build a better perl ORM:
    http://dbix-class.shadowcatsystems.co.uk/ +

    _______________________________________________
    Catalyst mailing list
    Catalyst at lists.rawmode.org
    http://lists.rawmode.org/mailman/listinfo/catalyst


    --
    Daniel McBrearty
    email : danielmcbrearty at gmail.com
    www.engoi.com : the multi - language vocab trainer
    BTW : 0873928131
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: http://lists.rawmode.org/pipermail/catalyst/attachments/20060303/13fc69d4/attachment.htm
  • Matt S Trout at Mar 3, 2006 at 3:38 pm

    On Fri, Mar 03, 2006 at 04:12:51PM +0100, Daniel McBrearty wrote:
    I suggested the "set names utf8" thing. That's because I have personal
    experience of mysql doing just what the OP described, and this turned out to
    be the fix.

    I have no idea if or how catalyst makes this call ... but I just know that
    SOMETHING has to ...
    Catalyst is a web framework. It has no idea what you're talking about.

    DBIx::Class is an ORM. The config change I suggest should make it run that
    SQL statement - but I've seen previous reports that 'set names utf8' isn't
    sufficient, hence also mentioning the UTF8Columns component (which is now
    in DBIx-Class-current).

    --
    Matt S Trout Offering custom development, consultancy and support
    Technical Director contracts for Catalyst, DBIx::Class and BAST. Contact
    Shadowcat Systems Ltd. mst (at) shadowcatsystems.co.uk for more information

    + Help us build a better perl ORM: http://dbix-class.shadowcatsystems.co.uk/ +
  • Michele Beltrame at Mar 3, 2006 at 4:50 pm
    Hi!
    DBIx::Class is an ORM. The config change I suggest should make it run that
    SQL statement - but I've seen previous reports that 'set names utf8' isn't
    sufficient, hence also mentioning the UTF8Columns component (which is now
    in DBIx-Class-current).
    I tried it and it did no good. Also this full-featured version did no good:

    __PACKAGE__->storage->on_connect_do([
    'set names utf8',
    'set_character_set_client=utf8',
    'set_character_set_results=utf8',
    'set_character_set_connection=utf8'
    ]);

    Michele.

    --
    Michele Beltrame
    http://www.varlogarthas.net/
    ICQ# 76660101
    Informativa privacy: http://www.italpro.net/em.html
  • Matt S Trout at Mar 3, 2006 at 5:34 pm

    On Fri, Mar 03, 2006 at 05:50:50PM +0100, Michele Beltrame wrote:
    Hi!
    DBIx::Class is an ORM. The config change I suggest should make it run that
    SQL statement - but I've seen previous reports that 'set names utf8' isn't
    sufficient, hence also mentioning the UTF8Columns component (which is now
    in DBIx-Class-current).
    I tried it and it did no good. Also this full-featured version did no good:

    __PACKAGE__->storage->on_connect_do([
    'set names utf8',
    'set_character_set_client=utf8',
    'set_character_set_results=utf8',
    'set_character_set_connection=utf8'
    ]);
    Did DBI->trace show that being run?

    --
    Matt S Trout Offering custom development, consultancy and support
    Technical Director contracts for Catalyst, DBIx::Class and BAST. Contact
    Shadowcat Systems Ltd. mst (at) shadowcatsystems.co.uk for more information

    + Help us build a better perl ORM: http://dbix-class.shadowcatsystems.co.uk/ +
  • Michele Beltrame at Mar 6, 2006 at 8:41 am
    Hi!
    __PACKAGE__->storage->on_connect_do([
    'set names utf8',
    'set_character_set_client=utf8',
    'set_character_set_results=utf8',
    'set_character_set_connection=utf8'
    ]);
    Did DBI->trace show that being run?
    I just tried with DBI->trace and these queries are not executed. The
    question is, am I putting the statement in the right place? I've put it in:

    MyApp/lib/SchemaLoader/Dbs.pm

    which inherits from SchemaLoader, which in turn inherits from Schema.

    Thanks, Michele.

    --
    Michele Beltrame
    http://www.varlogarthas.net/
    ICQ# 76660101
    Informativa privacy: http://www.italpro.net/em.html
  • Bernhard Graf at Mar 3, 2006 at 10:31 am

    Michele Beltrame wrote:

    Hello Matt!
    __PACKAGE__->storage->on_connect_do([ "set names utf8" ]);

    to ensure this gets run immediately after the $dbh is
    (re-)connected.
    At last I was able to migrate one of my Catalyst application to
    DBIC::Schema. However, putting this query in the schema class does no
    good for the UTF-8 thing.

    How can I be sure it is executed? I tried replacing it with something
    invalid such as "set nameddswgs utf8" (which returns an error in
    mysqlclient), but I get no error in Catalyst. That's why I ask this
    question. ;-)

    Thanks, Michele.
    Look at
    http://lists.rawmode.org/pipermail/dbix-class/2006-March/000847.html
    and the rest of the thread.

    HTH
    --
    Bernhard Graf
  • Islue at Mar 3, 2006 at 4:44 pm
    Hello, Michele

    I am interesting in whether your data is correct. Generally, if your
    data is stored in utf8 correctly, "set names utf8" is meaningful.
    So do you mind telling the detail of how you write/read your data?
    Hello Matt!

    At last I was able to migrate one of my Catalyst application to
    DBIC::Schema. However, putting this query in the schema class does no
    good for the UTF-8 thing.

    How can I be sure it is executed? I tried replacing it with something
    invalid such as "set nameddswgs utf8" (which returns an error in
    mysqlclient), but I get no error in Catalyst. That's why I ask this
    question. ;-)

    Thanks, Michele.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcatalyst @
categoriescatalyst, perl
postedMar 2, '06 at 3:35p
activeMar 6, '06 at 8:41a
posts16
users7
websitecatalystframework.org
irc#catalyst

People

Translate

site design / logo © 2022 Grokbase