[Catalyst] Form validation in insert() and update()?

It gets hard. The validation state depends on the controller more than you
would think. My old projects pushed the validation from my Model, the
problem is that while some data may be technically valid for the database
tables, depending on the controller's use and intent it may not be valid
for that state. I have since started to tie the Validation in the
Controller files and it seems to be saving me a lot of bouncing while
editing.

Some people like townhouses, some people like to architect their own house.
Catalyst is as simple or complex as you make it. We have normality. I
repeat, we have normality.
Anything you still can't cope with is therefore your own problem.





Any of the above.





You are not in the airport -- there are no people movers here.


I think the agreement is use whatever you are most productive with -- try a
few of them out. Most exist because the authors had touble with the way
another one worked and decided her way was better, the outcome is more
choice; fewer chains.

Yeah, that's always been my worry too. I could see placing the
controller-related constraints in the controller and pushing the
rest into the model though. Seems like handling all the errors across
different DBIx::Class instances and formatting them in a sane way to
the user would be difficult though (I haven't played with Rails enough
to know how exactly they handle that).

Anyone have validation logic in the model and are happy with it?
I'm not against that flexibility at all. I'm just wanting to know if
there's a generally agreed upon set of core modules to use with the
framework.

Clearly Catalyst has an opinion on how to handle the request
with Catalyst::Request. I'm sure you could override that somehow and
use your own, but it's helpful that there's a default request
interface that new developers can get up to speed quickly with.

I don't see why that can't apply to form creation and validation too.
Sure, if you don't like it, use something else, but for someone new
to Catalyst, having a Catalyst Way is helpful (unless we don't know
what a good way is).

Maurice

The current consensus for building/validating forms seems to be around
CGI::FormBuilder via the Catalyst::Controller::FormBuilder base
controller. Personally I have some reservations about using FormBuilder
for validation, though am persisting with it for now. Someone recently
threatened to do a Data::FormValidator plugin for Catalyst. Would be
most welcome, but sadly is beyond my capabilities to do such a thing at
present.

There are two kinds of validation here. One is model-level validation, and
yes, it's in my model code. My model throws exceptions, which I trap in
the controller and "mess with" to make it work for the web UI.

The controller might also do some validation, but all it's doing is
validating things specific to the controller's environment (in this case,
the web UI).

Putting your model validation in the controller is a horrible violation of
the layering that makes MVC work. What do you do when you want to
insert/update/delete outside of the web UI?

The controller is the bridge between a specific environment (web, CLI,
REST API) and the model. That means _most_ of your code for an app should
probably be in the model. The controller is basically a simple translation
layer between the stuff the client provides and the model API.


-dave

/*===================================================
VegGuide.Org www.BookIRead.com
Your guide to all that's veg. My book blog
===================================================*/

That's makes total sense to me.

Could you give an example of how you munge the exceptions into error
messages for the user?

For example, say we have a User model with an email field. And we want
to ensure the e-mail format is somewhat sane. So you have something in
your DBIx::Class User model that checks the e-mail value whenever an
attempt is made to modify it. If it's invalid, you raise an exception
of what sort? An object exception or just a string?

And then you need to map that back to a user message. The html form
could be asking for something very specific -- say, your mother's e-mail
address. So you need to convert the generic "e-mail invalid" error into
something like "Your mother's e-mail address is invalid". Something of
that sort?

Thanks.

eval
{
$user->update( %bunch_of_stuff );
};

if ( my $e = Exception::Class->caught( 'My::App::Exception::DataValidation') )
{
# $e->errors contains multiple data validation error messages
# stuff them in the session
# save the user's form submission in the session
# redirect back to form
}
elsif ( my $e = $@ )
{
die $e;
}

Then on the display side, I check the session for error messages and saved
form arguments, and do something useful with them.
I've never had to do that. If I did, I'd probably make my exception give
back more info, like a field name _plus_ an error message. Presumably, I
have a map from form field name to model object field name, so I can
reverse that mapping if needed.

The real point here is that you have all the information you need in your
controller to map from environment to model, so you also have the
information to do the reverse mapping as needed.

Obviously, you'll quickly find common patterns that can be abstracted into
base classes or traits or
whatever-you-like-for-handling-this-sort-of-abstraction.


-dave

/*===================================================
VegGuide.Org www.BookIRead.com
Your guide to all that's veg. My book blog
===================================================*/

That's excellent. Thank you. I'm going to throw together an experimental
page using this idiom.

Maurice

Although there's no real problem I know of with using exceptions this
way, I've gotten away from using them for expected user input
mistakes. There's no need for the flow control they provide in this
kind of scenario, since you're going to keep going long enough to
collect all the input mistakes anyway. It makes more sense to me to
just use a separate validate call and a normal data structure for this
and save exceptions for times when you actually need to bail out of
something.

Right now, I just check the input with Data::Form::Validator and
convert the results it returns into error message names, which are
then mapped to strings for display. The mapping allows the messages
to be overridden for a specific form (this code uses
Class::PhraseBook) so that they can be context-specific.

- Perrin

And where do you handle the validation? Only in the controller or in
both the model and controller?

Could you give a short example of taking an actual field from the
request parameters, validating it, and updating a row with it?

Thanks,

Maurice

Fail Early. Fail Often.

Some will say redundancy sucks. I agree, except for where validation is
concerned. If you're writing data from the web into a model, check the
data at the page level, and at the model level.

If you're writing to a model from a command line utility...check data at
the command line level, and the model level...

The trick is to define those check in a way that can be used in both
places....

The controller should validate.

The model should also validate.

The model should provide a way for the controller to get the constraints.

The controller should check the constraints and return errors to the user,
and if there are errors NOT TRY AND UPDATE THE MODEL.

The model's validation should die screaming if it doesn't pass - if bad data
gets that far, either the model's changed under you or there's a bug in
your code.

Okay, thanks very much for this. So in terms of the model constraints,
you will validate everything twice. Once at the controller layer (where
it leverages info from the model), and once in the model itself.

So you can interrogate the model to find out, for example, that the
email column can't be longer than 100 characters and should match the
pattern /@/ (just to keep it simple here).

Do you expose something like a Model->valid_email() method that can be
used both within the model and from the controller?

Maurice

Off Topic. Making a note for myself and DBIC::Validation

100 can be validated because we have %colinfo {size => 100}
Patterns could be covered in validation_profiles at the source
level...but maybe this would be nice as well:

{
type => 'VARCHAR',
size => 100,
validation => qr/\@{1}/
#validation => sub{}
}


/me urns for better validaiton profiles created from DBIC %colinfo

-=Chris

Back on topic...
And of course, there are times where you're not doing simple CRUD...and
the form/validation profile doesn't match the destination field for
field...like a form that covers the data for a parent/child table... or
a join...

I've been known to keep the UI form and it's validation completely
seperate fomr the actual model/model validation....

Damned if you do... damned if you don't. There is no one correct answer.

So true ;-|

In terms of form validation, what do you guys think of this
interface? It uses perl to handle conditional logic/dependencies
instead of using a spec language like Data::FormValidator.

# pass a hash ref or object with param() interface to ne
my $form = MLA::Form->new($params);

# Validate the data. This terse syntax uses Getopt::Long-style
# option specifiers.

$form->check('first_name=s'); # required field first_name
$form->check('your_email=email'); # required e-mail

# After doing checks, see if we hit any errors.
if ($form->error) {
print "Error: $_\n" foreach $form->error;
} else { # No errors, validation passed
# data() returns a hash ref with the validated data
my $data = $form->data;
}

Could also check for specific fields failing by

if ($form->error(qw/ first_name last_name /)) {
...
}

By default, it will convert the field name to a label
name (e.g., first_name => First Name) and use that for the error
messages. So, for example, in the first check, if the first_name
isn't supplied, it would produce an error message like
"First name is required".

You can override that though:

$form->check('first_name=s', label => "Mother's first name");

Usually you'd subclass the form class to implement the
data types that are specific to your domain.

Other examples of validation checks:

$form->check('title:s'); # optional title
# list of colors; if validation passes, $data->{fav_colors} will
# contain an array ref. Each individual color is checked.
$form->check('fav_colors:color@')

The verbose interface:

$form->check(
field => 'first_name',
type => 'string',
max_length => 100,
label => 'Your first name',
);

Any unknown arguments are passed into the validation type routine so
you can parameterize your types easily.

Anyone have a problem with this approach? What are the drawbacks?

I originally turned to it because I was hacking Data::FormValidator to
have more complex dependency logic (I think it's added some stuff since
then) and thought you can't beat code for that.

Thanks,

Maurice

Might look at Rose::HTML::Objects and Form::Processor form similar
approaches.

This gets at precisely the issue I've been wrestling with lately.
Validation constraints at all levels (db, model, controller, form, etc.)
seem to sort themselves into two general categories: field-specific
questions ("Is it required/unique?", "Does it match a given pattern?",
etc.) and broader domain-level interdependencies ("Require at least one
of the following...", "If A is 'foo' then B must match /^bar|baz|qux$/",
etc.) This leads me to want to define those constraints in different
places.

For example, I'd love to be able to define per-column validation
patterns/subs at the DBIC level as Chris describes above, and then
utilize that meta info to validate input at higher levels (model,
controller, form) as MST described in another post on this thread. But
I'd also need a way to create a profile for, let's say, form field
validation interdependencies (which can be independent of
domain/model/db-level interdependencies, as previously noted). But how
do I implement a concise way of accomplishing all these things?

Or is the fact that I have so many things to worry about an indicator
that this isn't going to be concise?

If Chris were to implement a per-column profile option for
DBIx::Class::Validation, I'd be inclined to look for a way to use
Data::FormValidator at several levels. Would it make sense to have a
method that would compile everything from %colinfo and
validation_profile() into a single profile that could then be utilized
(and modified/extended) at the model/controller level?

And should this branch of the thread be moved to the DBIC list?

Jason

I was thinking something similar. We could have an
optional DBIC::Validation::Auto that would create a
base profile based on the columnn metadata, like
required, unique, must be FK, etc. Then your custom
profile would be automatically merged in. Would be
useful to help CYA at the physical model level.
However you still need to do checking at your business
model level and possible at the form level, depending
on how you are doing forms. Since I've been writing
my most recent business logic stuff with Moose, I've
been trying to use Moose's built in type checking for
a lot of that. Still have a lot to learn with that.

For the top most layer of validation, associated with
a particular form, I don't think this belongs in the
controller, I think it's more a view thing. Lately
I've been doing it as macros in template toolkit, for
the most top level, final validation. This is a
combination of Ajax style callbacks and client-side
Javascript. Maybe that's ugly. Not sure yet, I need
to work with a bit more. Jquery has some nice plugins
to help with this.

--john

__________________________________________________

I'm thinking Catalyst::Model should provide some sort of interface for
this.

From what I understand, Catalyst::Model::DBIC::Schema is a wrapper class
that provides the "model interface" for DBIx::Class objects. Having that
interface is important so the framework as a whole has a consistent API
to the models.

But Catalyst isn't tightly coupled to DBIx::Class. I can design the
model however I want, with whatever data store I like, and as long as it
exposes that common model interface, I can plug it into Catalyst.

In the common case, I want models with attributes, accessors/mutators,
constraints, and the ability to load/save instances.

If there was a consistent interface to the model that provided those
functions then a lot of the standard data validation and CRUD stuff
becomes dead single.

Any generic model that wanted to be pluggable into Catalyst could
provide that "attribute" interface (other models would still work,
just not with anything that assumed the attribute interface was
present), either directly or as a wrapper class.

We'd still want the validation module to be independent of Catalyst,
of course... but having the ability to automatically
interrogate and manage a model like seems very powerful.

Maurice

This doesn't belong at the Catalyst core level since Catalyst is expressly
-not- opinionated about how your model is written.
Which we've done with great success with Reaction and the
Reaction::InterfaceModel system.
What myself and the other Reaction hackers are working towards is

(a) a generalised collection interface
(b) a reflection system that's powerful enough to provide typed mutations
(c) an abstract representation of transactional mutations

But it's a -bloody- hard job, and getting the design right is proving an
exercise in repeated iteration.

What I'm hoping is we can get it all figured out in Reaction and then
abstract out what works into some sort of standard API - which I don't
think will need to depend on anything at all except for the Moose metaprotocol
interface (which would handle the 'attribute interface' part of what you're
describing). Once we have that, a Catalyst::Model::${whatever_we_call_it} can
be written to tie any collection of classes that implements this API into
a Catalyst app.

Yes, that sounds good. I see there's nothing on CPAN yet. Where can
I find more on Reaction?

Maurice

http://code2.0beta.co.uk/reaction/svn

We're holding off on a CPAN release on the grounds it's still relatively
experimental and at this stage we only want users who're willing to roll with
the fact if a piece of design turns out to be completely useless it'll get
obsoleted and replaced with something that isn't.

Discussion is happening on irc.perl.org#reaction primarily.

my $tc = Model->meta->get_attribute('email')->type_constraint;

my $error = $tc->validate($value); # undef for 'value ok'

Here lies the joy of Moose everywhere :)

Whew. Okay, Moose. New way to make objects. Wasn't familiar with that.

So you base your entire app on Moose. It's not a dependency for
either DBIx::Class or Catalyst... so when you say everywhere,
you mean everything in your particular app inherits from Moose?

Who implements the type_constraint object? I just grepped my
perl sources and didn't see it anywhere.

Thanks,

Maurice

Controller only in my case. There could be extra validation in the
ORM, and of course in the database.
I don't work on the field level, but the form level. The form knows
the collection of fields make up the form, and knows how to validate
each field. It also knows how to update the database.

I do it like this:

package MyApplication::Controller::User;
use strict;
use MyApplication::Form::User;

sub edit : Local {
my ( $self, $c, $id ) = @_;

# Create the form object
my $form = MyApplication::Form::User->new( $id );


# Update or create the user record if form posted and form validates
$form->update_from_from( $c->request->parameters )
if $c->form_posted;


# Pass form object to template so can display fields
# and any error messages.
$c->stash->{form} = $form;

}


Where MyApplication::Form defines the fields and any custom validation
requirements. That same form module can be used outside of Catalyst.

Actually, since the form is named after the action and the above code
is the often the same for different forms, I have a shortened
version I use under Catalyst:

sub edit : Local {
my ( $self, $c, $id ) = @_;

$c->update_from_form( $id );
}

Where $c->update_from_form requires the form module and does
everything else the first example does. I commonly do a redirect
if $c->update_from_form returns true.

In the form processing code. This system has a CMS where users get to
generate forms and decide which fields will be on specific forms, so
the required fields depend on the actual form being used. It could be
separated out better, MVC-wise, but in practice that would be wasted
effort for this system.

There are additional constraints in the database to prevent
accidentally putting in broken data, and a few in perl (with
Params::Validate in the model objects) where they would be difficult
to code in the database.
The actual system has a lot of complexity coming from things that
aren't directly related to your question, so it's hard to show a
literal code snip. The form handling classes are structured so they
follow this basic recipe when a form is submitted:

- Get a Data::FormValidator profile from the form class.
- If the input passes validation:
- Run the fullfill_request() method for this class.
- Show the response page.
- If the input fails:
- Translate the Data::FormValidator results object into error codes
- Hand the codes to our message class which looks up the local error
text and adds it to the template data.
- Re-display the form with the messages at the top and the problem
fields highlighted.

This is handled by a base class so in most cases you only need to
write a form profile and a fulfill_request() method to add a new type
of form.

No attempt is made to handle errors that come from the database or
model objects called from fulfill_request(). Errors at that point are
real errors, i.e. unexpected and indicating a bug, not a user input
mistake. They result in a friendly error screen and a detailed log
message.

- Perrin

You can see an example of this with Rose::DB::Object,
Rose::HTML::Objects and the db (sqlite in this case) at:

http://search.cpan.org/src/KARMAN/Catalyst-Controller-Rose-0.02/t/examples/CatRose/

Basically, the form object validates and spits back friendly errors
messages to the user. If for some reason the data makes it to the ORM
(RDBO) level, the data is also validated there and 500 error returned to
the user. Finally, the db itself has contraints on it that should
validate in consistent ways with the form and the ORM.

This pattern seems to be similar to what mst, Perrin and Bill are also
describing.

Surely you mean the stash, not the session?

Regards,

No, I mean the session. I always keep the URIs for my form display and
form submission separate, and I do a redirect after form submission.

In the Catalyst case, I generally take advantage of the flash feature,
since it's perfect for this sort of stuff.


-dave

/*===================================================
VegGuide.Org www.BookIRead.com
Your guide to all that's veg. My book blog
===================================================*/

I used to do that, but I've found that what I thought were style reasons
mostly turned out to be insufficiently flexible tooling reasons and I'm
finding one URI works pretty well under Cat.

Would you mind elaborating as to why you're keeping separate URIs? (rather
than one that behaves differently on GET vs. POST)
That's cuz we stole it lock, stock and barrel from rails which implemented
it for exactly that.

I'm not sure I've ever used ->flash myself :)

I've always though that the form is a different "thing" from the the thing
that handles its submission, and to combine them doesn't make sense.

There's also something about the whole "if-else" logic that ends up
handling this that becomes rather gross. Of course, if you can make the
framework handle this by considering the method and dispatch based on
that, it becomes less gross.

OTOH, if you're thinking in REST terms, the form is not the same thing as
the entity or collection of entities to which you'd POST or PUT.

Also, consider the case where the form submission is successful. If the
action is a create or update, you want to redirect to the URI for the
thing being created or updated. You may _also_ want to include some form
"successfully updated" message, so this pattern works well for this.
That's not very RESTful, though, which is making me think about this some
more for my latest project.


-dave

/*===================================================
VegGuide.Org www.BookIRead.com
Your guide to all that's veg. My book blog
===================================================*/

Why not stash the message in the query string for the redirect
target? Or a hash key.

303 See Other
Location: /article/why-webarch-matters?msg=created

Then have infrastructure for an `msg` param to cause a message
box to be shown, with the content ?Your article was successfully
created? as found under the `created` key of your message
phrasebook (which also gives you a convenient hook for i18n).

It looks a bit crude, but it?s actually a rather good solution in
REST terms. (Eg. intermediaries won?t cache that resource as the
same you can retrieve from /article/why-webarch-matters where the
message box is missing; etc.)

Regards,

Except then the end-user can bookmark that success URL, which
isn't ideal. I've used that approach before and it worked okay,
but I think the flash message makes more sense for this.

Maurice

A couple problems. The biggest is that besides messages, I also pass along
the submitted form data, so the form can be displayed with what the user
submitted. That won't fit in a query string in many cases.

I also like the fact that reloading the page gets rid of the messages that
were being displayed, but that's really a small point compared to the the
form data issue.


-dave

/*===================================================
VegGuide.Org www.BookIRead.com
Your guide to all that's veg. My book blog
===================================================*/

if the post is handled by /order/new I usually redirect to /order/15 with the
flash message. I think there's plenty of room in the URL for the
altered/created object's primary key

Yes, that's what I do too.

I was talking about the case where the form submission _fails_, and you
want to send the client back to the form, show them 1+ error messages, and
re-populate the form with the data they submitted. That may not fit in a
query string in many cases.


-dave

/*===================================================
VegGuide.Org www.BookIRead.com
Your guide to all that's veg. My book blog
===================================================*/

I tend to redirect on 'OK' and stay put on 'Apply'.

A failure doesn't redirect.

An 'Apply' of a -create- redirects to an edit form.

Not -exactly- restful but seems to work well.

That state should really be exposed in the URI. (Then your blank
form page is potentially cachable because the state of that
resource never changes, f.ex.)

So if the blank form is `/order/form`, you?d stash the form data
away somewhere under ID `7z32a` (f.ex) and redirect them to
`/order/form/7z32a` (or `/order/form?populate=7z32a`). The data
could be in the session, as long as it?s keyed off of an ID that
shows up in the URI and the ID is unique across all
users/sessions. Stick the ID in a hidden field in the form so you
can garbage the data immediately if the submit goes through.
(Else you?ll have to expire the data after a while.)

This way, you don?t have a page whose content changes based on
implicit state on the server that isn?t contained in the client?s
request ? the REST way.

F.ex., the user can then have two windows open showing the same
form, type different things into each of them, submit them both
nearly simultaneously, and get failure for both, and still not
suffer a race condition where one form overwrites the other?s
prepopulation data.


* Matt S Trout [2007-05-21 02:00]:
Hmm, that?s perfectly RESTful.


Regards,

I do like this, in theory. It's basically equivalent to puttint the
session key in the URI, though, which means it has some security issues.

If I were to do this, I'd probably associate each mini-session (for a form
or whatnot) with the user_id, and I'd still be checking the user_id
against the cookie.

I think that would be sufficiently secure.

I know that user cookies are not considered RESTful, but this is one
problem I have not been able to work around. If web browsers could be made
to handle HTTP auth with more flexibility (use forms, allow logout, etc)
then I think this solution could 100% RESTful.


-dave

/*===================================================
VegGuide.Org www.BookIRead.com
Your guide to all that's veg. My book blog
===================================================*/

Yeah, you would have to make sure that whoever is requesting the
prepopulated form is permitted to do so. But that?s easy ? as I
said, just continue to put the data in the session as you would
have, just make sure to add a layer of indirection through a
hash. Eg.

# previously
$c->session->{foo_form} = { name => $name, email => $email };

# better:
my $form_state_id = get_appwide_unique_id();
$c->session->{foo_form}->{$form_state_id}
= { name => $name, email => $email };

Then when the user asks for `/oder/form/7z32a`, you check if
there?s a `$c->session->{foo_form}->{7z32a}` key. If not, spit
out a 404.

That?s it ? it?s no harder than what you already do. All you need
is to calculate an ID and to add the routing for `.../form/{id}`
URIs.
Actually, if you use the cookie purely as an authentication
token, your app is still RESTful.

Regards,

I tend to have much less validation in the "model" (if that's the ORM
layer to the database) and instead use the database to enforce the model
constraints, where possible.

User input (such as a web form) on the other hand, has to have extra
validation -- the database can't know the context of the update, of
course. That's where the form validation tool comes in for me. It
knows how to move the data from the database to an external
representation and how to go the other direction, validating along the
way. It's a tool the controller can use for populating and updating a
web form, for example.
Which is why I like the "form" validation tools to not be specific to
the web/HTML side of things. The HTML side of the forms are easy, anyway,
and often require hand-customizing. That way the same "forms" can be
used for more than just the web environment.

The database is also part of the model. I wasn't suggesting that
validation has to happen in Model class code versus the database.


-dave

/*===================================================
VegGuide.Org www.BookIRead.com
Your guide to all that's veg. My book blog
===================================================*/

So you're saying you place as many constraints on the actual db tables
as possible (e.g., NOT NULL, check constraints, unique indexes, etc).

But then you rely on the form data validation to catch all the normal
user-supplied form errors, right?

You don't rely on the db to actually raise exceptions and propagate
those errors back to the user in some way, right? A db exception would
indicate that the form validation missed something and you'd just croak
in that case. Is that right?

Thanks.

Yes, I define form modules that validate the field types and do any
other context based validation and cross field validation. So when
the form validates the data should go into the database without any
complaints.

Right. The database throwing an exception is, well an exception
that is not expected and I log the message and send the user a 500 and
a message that they have been a very, very bad user.