FAQ
Hello @all,

now it's my time to make my Catalyst app Unicode aware and I (like some
others) ran into a Unicode problem.

I read the article from the calender 2006 and read the unicode chapter in
Daminas "Advanced Perl Programming" hence I don't manage to get it working.

What I have:
- Catalyst, TT, DBIx::Class app that needs to deal with german umlauts.
- Plugin Unicode::Encoding loaded.
- Postgres database with UTF8 enabled (ENCODING = 'UTF8').
- Browser is FF2, it says encoding is UTF8, mode is standard compliant
(translated from what it says in german).
- Plugin I18N loaded, with a german po file.
- Useing the inwork HTML-Widget replacement FormFu with I18N and a german
language module in lexicon style.
- No german umlauts are html-escaped (that's why we make unicode, right?)

Now my problems:
- text with german umlauts from tt templates is shown as expected
- text with german umlauts from po file is shown as expected
- text with german umlauts from lexicon style pm files is currupted, it is
shown as expected, when Unicode::Encoding is disabled, so this might due to a
double encoding - but why?
(lexicon = perlmode loading = encoding by default, po file = no encoding at
first place ?, I realy don't know.)
- when submitting form input Catalyst dies with an error:
[info] *** Request 207 (0.139/s) [14828] [Tue Mar 20 16:29:05 2007] ***
[error] Caught exception in engine "utf8 "\xE4" does not map to Unicode
at /usr/local/share/perl/5.8.8/Catalyst/Plugin/Unicode/Encoding.pm line 73."
(The formdata is submitted through ajax/xmlhttprequest, is that a possible
source for that problem?)

I would be very thankfull for every hint on how to hunt these things down.

By the way I usually send some debug messages with 'warn' to the console, as
catalysts logging is not available in other modules (is it somehow?). Is that
unicode save, or do I see crap on the console?

Greets,
Mario Minati

Search Discussions

  • Carl Franks at Mar 20, 2007 at 3:52 pm

    On 20/03/07, Mario Minati wrote:

    - text with german umlauts from lexicon style pm files is currupted, it is
    Does your .pm file have "use utf8;" ?
    It's necessary for when you have UTF-8 in perl source files.

    Cheers,
    Carl
  • Mario Minati at Mar 20, 2007 at 4:06 pm

    Am Dienstag 20 M?rz 2007 16:52 schrieb Carl Franks:
    On 20/03/07, Mario Minati wrote:
    - text with german umlauts from lexicon style pm files is currupted, it
    is
    Does your .pm file have "use utf8;" ?
    It's necessary for when you have UTF-8 in perl source files.
    Waow that did the trick for my first problem.
    I thought it would only be necessary if you have var names with umlauts. Now I
    know better ;-)

    thank you,
    Mario

    PS:
    The formfu validation works fine and I think I'm gone start with
    the 'repeatable' stuff in this week. would you prefer a different name for
    it.
  • Dave Rolsky at Mar 20, 2007 at 4:16 pm

    On Tue, 20 Mar 2007, Mario Minati wrote:

    Am Dienstag 20 März 2007 16:52 schrieb Carl Franks:
    On 20/03/07, Mario Minati wrote:
    - text with german umlauts from lexicon style pm files is currupted, it
    is
    Does your .pm file have "use utf8;" ?
    It's necessary for when you have UTF-8 in perl source files.
    Waow that did the trick for my first problem.
    I thought it would only be necessary if you have var names with umlauts. Now I
    know better ;-)
    A good rule with Unicode is "Unicode anywhere, then Unicode everywhere".
    It's not worth trying to sometimes encode things and sometimes not. Just
    encode all incoming data as utf8, all the time. Submit everything as UTF8,
    all the time.


    -dave

    /*===================================================
    VegGuide.Org www.BookIRead.com
    Your guide to all that's veg. My book blog
    ===================================================*/
  • Mario Minati at Mar 20, 2007 at 5:45 pm

    Am Dienstag 20 M?rz 2007 17:16 schrieb Dave Rolsky:
    On Tue, 20 Mar 2007, Mario Minati wrote:
    Am Dienstag 20 M?rz 2007 16:52 schrieb Carl Franks:
    On 20/03/07, Mario Minati wrote:
    - text with german umlauts from lexicon style pm files is currupted, it
    is
    Does your .pm file have "use utf8;" ?
    It's necessary for when you have UTF-8 in perl source files.
    Waow that did the trick for my first problem.
    I thought it would only be necessary if you have var names with umlauts.
    Now I know better ;-)
    A good rule with Unicode is "Unicode anywhere, then Unicode everywhere".
    It's not worth trying to sometimes encode things and sometimes not. Just
    encode all incoming data as utf8, all the time. Submit everything as UTF8,
    all the time.
    I'm trying to achieve that ;-)

    Greets,
    Mario
  • Renaud Drousies at Mar 20, 2007 at 4:07 pm

    (The formdata is submitted through ajax/xmlhttprequest, is that a possible
    source for that problem?)
    That was for me.

    Which request headers are you sending when doing the ajax call?
    If you are not doing so yet, try using "application/x-www-form-urlencoded;
    charset=UTF-8".


    Regards,
    Renaud Drousies
  • Mario Minati at Mar 20, 2007 at 5:44 pm

    Am Dienstag 20 M?rz 2007 17:07 schrieb Renaud Drousies:
    (The formdata is submitted through ajax/xmlhttprequest, is that a
    possible source for that problem?)
    That was for me.

    Which request headers are you sending when doing the ajax call?
    If you are not doing so yet, try using "application/x-www-form-urlencoded;
    charset=UTF-8".
    Firebug says:

    -- request header --
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    Content-Type: application/x-www-form-urlencoded

    -- response header --
    Content-Type: text/html; charset=utf-8
    X-Catalyst: 5.7006

    For me that looks ok.
    Or do I need a charset=utf-8 in the _header_?

    Greets,
    Mario Minati
  • Renaud Drousies at Mar 20, 2007 at 6:12 pm

    Am Dienstag 20 M?rz 2007 17:07 schrieb Renaud Drousies:
    (The formdata is submitted through ajax/xmlhttprequest, is that a
    possible source for that problem?)
    That was for me.

    Which request headers are you sending when doing the ajax call?
    If you are not doing so yet, try using
    "application/x-www-form-urlencoded;
    charset=UTF-8".
    Firebug says:

    -- request header --
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    Content-Type: application/x-www-form-urlencoded

    -- response header --
    Content-Type: text/html; charset=utf-8
    X-Catalyst: 5.7006

    For me that looks ok.
    Or do I need a charset=utf-8 in the _header_?
    I'd still explicitely set the request character set. If you don't do so,
    it will be handled with the default character set of your server. (correct
    me if i'm wrong)

    Regards,
    Renaud
  • Mario Minati at Mar 20, 2007 at 7:03 pm

    Am Dienstag 20 M?rz 2007 19:12 schrieb Renaud Drousies:
    Am Dienstag 20 M?rz 2007 17:07 schrieb Renaud Drousies:
    (The formdata is submitted through ajax/xmlhttprequest, is that a
    possible source for that problem?)
    That was for me.

    Which request headers are you sending when doing the ajax call?
    If you are not doing so yet, try using
    "application/x-www-form-urlencoded;
    charset=UTF-8".
    Firebug says:

    -- request header --
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    Content-Type: application/x-www-form-urlencoded

    -- response header --
    Content-Type: text/html; charset=utf-8
    X-Catalyst: 5.7006

    For me that looks ok.
    Or do I need a charset=utf-8 in the _header_?
    I'd still explicitely set the request character set. If you don't do so,
    it will be handled with the default character set of your server. (correct
    me if i'm wrong)
    I played a little with the FF2 settings and changed the coding to utf-8 and
    guess what, suddently it work as it should be.

    So I think you might be right I have to force charset even in the request
    header, but I don't know yet how to achieve that in Dojo, but I'll find out.

    I'll post my results, stay tuned.

    Greets,
    Mario
  • Mario Minati at Mar 20, 2007 at 9:34 pm

    Am Dienstag 20 M?rz 2007 20:03 schrieben Sie:
    Am Dienstag 20 M?rz 2007 19:12 schrieb Renaud Drousies:
    Am Dienstag 20 M?rz 2007 17:07 schrieb Renaud Drousies:
    (The formdata is submitted through ajax/xmlhttprequest, is that a
    possible source for that problem?)
    That was for me.

    Which request headers are you sending when doing the ajax call?
    If you are not doing so yet, try using
    "application/x-www-form-urlencoded;
    charset=UTF-8".
    Firebug says:

    -- request header --
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    Content-Type: application/x-www-form-urlencoded

    -- response header --
    Content-Type: text/html; charset=utf-8
    X-Catalyst: 5.7006

    For me that looks ok.
    Or do I need a charset=utf-8 in the _header_?
    I'd still explicitely set the request character set. If you don't do so,
    it will be handled with the default character set of your server.
    (correct me if i'm wrong)
    I played a little with the FF2 settings and changed the coding to utf-8 and
    guess what, suddently it work as it should be.

    So I think you might be right I have to force charset even in the request
    header, but I don't know yet how to achieve that in Dojo, but I'll find
    out.

    I'll post my results, stay tuned.
    Finally I figured out howto set the header with dojo to request a specific
    charset. For those who are interested:

    var x = new dojo.io.FormBind({
    headers: { 'accept-charset': 'utf-8'},
    // reference the form
    formNode: dojo.byId('[% form.id %]'),
    preventCache: true,
    load: function(load, data, e) {
    // what to do when the form finishes
    // for example, populate a DIV:
    dojo.byId("[% form.id %]").disable=false;
    _container_.setContent(data);
    },
    error: function(type, err) {
    dojo.debug("Error during FormBind:" + err.message);
    }
    });

    After defineing the accept-charset header the form setting still worked when
    changing the FF2 setting to an other encoding.

    :D

    Greets,
    Mario Minati
  • Bill Moseley at Mar 20, 2007 at 4:35 pm

    On Tue, Mar 20, 2007 at 04:40:57PM +0100, Mario Minati wrote:
    now it's my time to make my Catalyst app Unicode aware and I (like some
    others) ran into a Unicode problem.

    - Postgres database with UTF8 enabled (ENCODING = 'UTF8').
    By the way:

    Are you setting DBD::Pg's pg_enable_utf8?



    --
    Bill Moseley
    moseley@hank.org
  • Mario Minati at Mar 20, 2007 at 9:45 pm

    Am Dienstag 20 M?rz 2007 17:35 schrieb Bill Moseley:
    On Tue, Mar 20, 2007 at 04:40:57PM +0100, Mario Minati wrote:
    now it's my time to make my Catalyst app Unicode aware and I (like some
    others) ran into a Unicode problem.

    - Postgres database with UTF8 enabled (ENCODING = 'UTF8').
    By the way:

    Are you setting DBD::Pg's pg_enable_utf8?
    That was also missing.
    Thank you.

    For those who are interested:

    package glue::Model::glueDB;

    use strict;
    use base 'Catalyst::Model::DBIC::Schema';

    __PACKAGE__->config(
    schema_class => 'glueDB',
    connect_info => [
    'dbi:Pg:dbname=glue',
    'xxx',
    'xxx',
    {AutoCommit=>1,
    pg_enable_utf8 => 1},
    ],
    );

    1;

    Greets,
    Mario

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcatalyst @
categoriescatalyst, perl
postedMar 20, '07 at 3:40p
activeMar 20, '07 at 9:45p
posts12
users5
websitecatalystframework.org
irc#catalyst

People

Translate

site design / logo © 2022 Grokbase