FAQ
Hi,

I have installed and ran successfully the AutoCRUD plugin, I set up a
mysql database tables to use UT8 charset, the charset in the ajax
requests is utf-8, everything seems correct, except the data in the
grids are double encoded, that means ?? instead of ?.

I am pretty sure that the data in the database are correct, and the
json data are also correctly displayed (the raw data received contains
the wrong characters). The double encoding seems then to appear in the
View of AutoCRUD, that is just a Catalyst::JSON::View simply used
without any customization.

I have hacked around this AutoCRUD JSON view, and end up with a solution :
- overload the encode_json method in the view to call directly
JSON::XS without the utf8 call, ie a simple
JSON::XS->new->encode($data);
- and overload the process method in order to remove the $json =
Encode::encode($encoding, $json);

In short : don't touch my data, they are already in utf8, just send
them to the browser. It works, but I don't know why...

It is probably a bad solution, as I can't imagine that
Catalyst::JSON::View is wrong, but I wonder what is the correct way to
do it.
Couldn't it be a problem in DBIx::Class that does not correctly handle
ut8 columns ?

--
Julien Gilles.

Search Discussions

  • Oliver Gorwits at Oct 7, 2011 at 10:34 am

    I have hacked around this AutoCRUD JSON view, and end up with a solution

    It is probably a bad solution, as I can't imagine that
    Catalyst::JSON::View is wrong, but I wonder what is the correct way to
    do it.
    In terms of AutoCRUD, I can't offer a lot more, only to add I'm very
    pleased that you did get something working :-)

    Those with good memory may know that over the years there have been a
    couple of similar issues with AutoCRUD. My deep thanks to those on IRC and
    this list for helping!

    I've definitely heard of success stories where AutoCRUD happily displays
    UTF8. So could this be a storage issue as Julien suggests? Table or
    database connection configuration? Is this a question better directed at
    the DBIx::Class mail list?

    regards,
    oliver.
  • Jul Gil at Oct 7, 2011 at 3:22 pm
    As I sayed in one another thread, I have developped a simple Catalyst
    application to test this utf-8 bug. You can download it at
    http://gilles.tk/TestUTF8-0.01.tar.gz

    Only simple catalyst.pl calls, a simple sqlite3 database, using
    default debian environnement.
    I have added a line in the sqlite database with "??", and you can see
    the result...

    Thanks for testing to confirm (or not) if you have the same behaviour.

    --
    Julien Gilles.
  • Oliver Gorwits at Oct 9, 2011 at 2:26 pm

    On 07/10/2011 16:22, jul.gil@gmail.com wrote:
    Only simple catalyst.pl calls, a simple sqlite3 database, using
    default debian environnement.
    I have added a line in the sqlite database with "??", and you can see
    the result...

    Thanks for testing to confirm (or not) if you have the same behaviour.
    I do - double encoded here as well, many thanks for the example case.

    regards,
    oliver.
  • Francisco Obispo at Oct 7, 2011 at 5:16 pm
    JSON will try to encode in UTF-8 format and if the data is already in UTF-8, most likely it will be double encoded.

    This could be fixed in two ways:

    1) when loading your UTF-8 data, convert it to perl's internal encoding with:

    use Encoding qw(decode_utf8);

    my $perl_encoded=decode_utf8($utf_encoded_string);

    and then use JSON->enconde to encode the data


    or:

    2) instruct JSON to avoid converting a string if it's already
    in UTF-8 by using is_utf8() from the Encode module.


    You might want to look at the documentation of DBD::Mysql (I haven't used it in a while), and consult at the section regarding encodings., it seems like there might be flags that need to be raised to do the right encoding/decoding on serialization.

    Cheers,

    Francisco




    On Oct 7, 2011, at 2:21 AM, jul.gil@gmail.com wrote:

    Hi,

    I have installed and ran successfully the AutoCRUD plugin, I set up a
    mysql database tables to use UT8 charset, the charset in the ajax
    requests is utf-8, everything seems correct, except the data in the
    grids are double encoded, that means ?? instead of ?.

    I am pretty sure that the data in the database are correct, and the
    json data are also correctly displayed (the raw data received contains
    the wrong characters). The double encoding seems then to appear in the
    View of AutoCRUD, that is just a Catalyst::JSON::View simply used
    without any customization.

    I have hacked around this AutoCRUD JSON view, and end up with a solution :
    - overload the encode_json method in the view to call directly
    JSON::XS without the utf8 call, ie a simple
    JSON::XS->new->encode($data);
    - and overload the process method in order to remove the $json =
    Encode::encode($encoding, $json);

    In short : don't touch my data, they are already in utf8, just send
    them to the browser. It works, but I don't know why...

    It is probably a bad solution, as I can't imagine that
    Catalyst::JSON::View is wrong, but I wonder what is the correct way to
    do it.
    Couldn't it be a problem in DBIx::Class that does not correctly handle
    ut8 columns ?

    --
    Julien Gilles.

    _______________________________________________
    List: Catalyst@lists.scsys.co.uk
    Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
    Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
    Dev site: http://dev.catalyst.perl.org/
    Francisco Obispo
    email: fobispo@isc.org
    Phone: +1 650 423 1374 || INOC-DBA *3557* NOC
    Key fingerprint = 532F 84EB 06B4 3806 D5FA 09C6 463E 614E B38D B1BE
  • Oliver Gorwits at Oct 9, 2011 at 3:45 pm

    On 07/10/2011 10:21, jul.gil@gmail.com wrote:
    I have installed and ran successfully the AutoCRUD plugin, I set up a
    mysql database tables to use UT8 charset, the charset in the ajax
    requests is utf-8, everything seems correct, except the data in the
    grids are double encoded, that means ?? instead of ?.
    After some investigation and poking around in documentation I found that
    the following is essential to using Unicode in Catalyst apps:

    1) tell your database connection to use unicode (this will be a flag in
    the connect options, alongside your username, password, etc).

    2) load the Catalyst::Plugin::Unicode::Encoding plugin.

    These ensure that data is flagged correctly when moving in both
    directions between the database and the user interface.

    Acknowledgment must go the author(s) of the following pages:

    http://wiki.catalystframework.org/wiki/tutorialsandhowtos/using_unicode
    https://metacpan.org/module/DBIx::Class::Manual::Cookbook#Using-Unicode

    The wiki page tells you the flag to use for each of the popular database
    engines (I only tested with SQLite).

    I hope this helps you Julien, and also the list archives!

    regards,
    oliver.
  • Jul Gil at Oct 10, 2011 at 3:27 pm

    2011/10/9 Oliver Gorwits <oliver@cpan.org>:
    On 07/10/2011 10:21, jul.gil@gmail.com wrote:

    I have installed and ran successfully the AutoCRUD plugin, I set up a
    mysql database tables to use UT8 charset, the charset in the ajax
    requests is utf-8, everything seems correct, except the data in the
    grids are double encoded, that means ?? instead of ?.
    After some investigation and poking around in documentation I found that the
    following is essential to using Unicode in Catalyst apps:

    1) tell your database connection to use unicode (this will be a flag in the
    connect options, alongside your username, password, etc).

    2) load the Catalyst::Plugin::Unicode::Encoding plugin.

    These ensure that data is flagged correctly when moving in both directions
    between the database and the user interface.

    Acknowledgment must go the author(s) of the following pages:

    http://wiki.catalystframework.org/wiki/tutorialsandhowtos/using_unicode
    https://metacpan.org/module/DBIx::Class::Manual::Cookbook#Using-Unicode

    The wiki page tells you the flag to use for each of the popular database
    engines (I only tested with SQLite).

    I hope this helps you Julien, and also the list archives!
    Works perfectly well. Thanks!
    I knew the bug was between the keyboard and the chair...

    --
    Julien Gilles.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcatalyst @
categoriescatalyst, perl
postedOct 7, '11 at 9:21a
activeOct 10, '11 at 3:27p
posts7
users3
websitecatalystframework.org
irc#catalyst

People

Translate

site design / logo © 2022 Grokbase