Uncategorized





Introducing furious – a proof-of-concept RRM for F#

Cats: Uncategorized

Tue - 13 Jul 2010 - 09:29 PM

One of the first things that newcomers to F# look for is an ORM system, be it a native one, or an adaptation of something already present. Since there are plenty of systems in the .net ecosystem, it is possible to get one of them to work with F#. But as we start to do that, the more seasoned f#'er will notice that something is amiss. One of the core tenets of F# is immutability, and classic ORMs rely on mutability by operating on class properties. While F# does support classes with mutable properties, the preferred unit of work is the record. This also follows the representation of the database better - at any one point in time you're either holding an uncommited record, or a representation of database state; immutability there prevents any confusion. These considerations, and discussions with some of the F# folks out there (Aaron Erickson being a major influencer of this idea), I decided to put something together meant for F# specifically.

Meet furious. Furious is a RRM - a Record Relation Mapping system. It is a database-independent object query dsl for records, that currently has a (barebones) mysql implementation. Here's how it works. Suppose you have a record graph like so:

type person = {
    personId: string
    firstname: string
    lastname: string
    homeAddress: address
    workAddress: address
    altAddresses: address seq
}
and address = {
    addressId: string
    street1: string
    zip: string
}

Now, if i wanted the list of all people that live in the 60614 zipcode, i could write this:

let (neighbor: person seq) =
    db.Yield <@ Seq.filter (fun p -> p.homeAddress.zip = "60614") @>

That's it. Neat, huh? Now, if you wanted something like a count of all such neighbors, you could do this:

let (neighbors: int seq) =
    db.Compute <@ fun people -> Seq.length <| Seq.filter (fun p -> p.homeAddress.zip = "60614") people @>

The main idea behind furious is to avoid inventing a new dsl, and use what's already there - the concepts defined over the Sequence space. Furious provides two methods - Yield (expr: Expr<(seq<'a>->seq<'b>)>), and Compute (expr: Expr<(seq<'a>->'b)>). The idea is that one is for mapping one sequence into another, and the other is for computing a value from a sequence (such as our Seq.length example). The interesting thing is that this approach is enough to cover a majority of all orm/rrm usecases. While there are statements that can't (easily) be expressed this way, they reach a complexity level that should really be done via explicit sql, rather than this type of generation.

Furious is currently a proof-of-concept project that will evolve into a full library. It currently lacks a lot of major features, and has only been tested for a handful of scenarios. That said, I welcome input and contribution, as this is a complex project that needs more than one mind.

The code is located here: http://github.com/kolosy/furious

5 Comments »


Building an image mosaic in python

Cats: Uncategorized

Wed - 10 Mar 2010 - 08:45 AM

I've always gotten a kick out of those giant mosaic posters - you know, the "picture made of pictures" stuff. Turns out, they're pretty straightforward to make.

Here's a quick stab at one (target and source both taken from public news sources):

source image
tiled - original

The basic algorithm is this:

assuming you have a target image, t and a set of thumbnails, s0..sN, all of a fixed square with a side s

  1. take all thumbnails, compute their average color* and put that in a map, keyed by thumb filename
  2. take t, and subdivide it into square tiles with a side s
  3. for every tile, compute the average color, and iterate over the map up above, looking for the closest average color**
  4. for simplicity's sake, rather than drawing a new image, i just generate a little html table with pointers to the images. makes things a bit simpler

* average color is figured out by using the python image library's getcolors() method. it returns a list of (count, color) tuples, telling you the full range of rgb values that occur, along with the number of times they occur. then, take an a weighted average of the individual r,g,b values, and that's your average color

** closest average color is almost as simple as it sounds. with two rgb values, just do (abs(r-r1) + abs(g-g1) + abs(b-b1))/3. the reason you want the absolute value of the difference is this - let's say we have colors 50,100,150 and 150,100,50. if you compute it without the absolute values, the fact that the 50 and the 150 are in different channels won't matter - they'll cancel each other out: (50-150)+(100-100)+(150-50) == -100 + 0 + 100 == 0. the absolute value lets you register that difference = abs(50-150)+abs(100-100)+abs(150-50) == 100 + 0 + 100 == 200.

here's the end result:

tiled - single

not bad, but not great either. the problem is that if a tile is half black and half white, the average color will end up being the equivalent of grey. so let's try a modification.

  1. for all thumbnails: take each thumbnail, divide it into four quadrants, and compute the average color value of each of the quadrants. store the list of averages into a map, as before.
  2. take t, and subdivide it into square tiles with a side s
  3. for every tile, divide it into four quadrants, like the thumbnails, compute the average color, and iterate over the map up above, looking for the closest average color. this time, closest average color is computed as the average of the color distances of each quadrant (color distance is the result of the computation shown above)

aand... voila
tiled - quad

that's better. now, you could ostensibly do the same thing with further subdivision (16 sections instead of 4), but that will probably get more and more limited results, unless you have a massive thumbnail library. these mosaics were built with a library of ~1000 thumbnails.

now, all this said, there's lots of room for improvement. the current search algorithm is simply unscalable, especially if you want to go past a 4-quadrant tile. my next stab at it will be some variation of a binary search (though the fact that it's 3 values instead of 1 combined with me not wanting to give any one color priority complicates things a bit).

source code here

No Comments »


Not everyone will steal your idea

Cats: Uncategorized

Fri - 05 Mar 2010 - 02:23 PM

Don't get me wrong, there are plenty of people that do, but there's even more that couldn't give two (insert appropriately grotesque object) about it. I'll give you an example. About 3 weeks ago, I gave notice. I'm leaving a job I liked quite a bit that paid well and a position of some power at that. Certainly, everyone's first question from there on was "where you goin'? whatcha doin' next?", and I would always answer "I'm doing my own thing, trying out a startup".

9 times out of 10, their reaction would be a mix of benign jealousy and admiration, followed quickly by rapid questions about the startup and a series of "good luck"s and "you'll do great"s. It's not that they aren't entrepreneurial (to a certain extent), or that they don't have ideas of their own. It's more that they have kids and wives and dogs and mortgages and a 401k and ... and ... and ... stuff. And all of us in the startup community have at least some subset of all those, but we make a choice. We choose to not get enough sleep and work after hours, or we choose to cut into the nest egg (or the piggy bank) and quit and try it out and see. None of us know ahead of time, all of us hope that we're the next (insert inspirational startup founder). But all of us think that what we're doing is a natural progression; after all - everyone around us is doing it, right?

But who's everyone? Everyone is the community of driven risk-takers that post to hacker news, that read joel on software, that know about gigaom and all the other bloggers in our sector of the blogosphere (g-d i hate that word). And like in so many other communities, we assume that the vast majority of people not there are just like us too, because - come on, just continue the model outward, right? If 90% of the people here are in startups or trying to get in, shouldn't the rest of the world follow suit?

The startup community, and its flagship blogs and forums have rallied around them those few in the world that are willing to take that plunge with no bottom in sight. And it has gathered so many of us that we think that everyone we meet thinks the same way, wants the same things and is willing to take the same risks.

They are not. Take a moment, and reflect on the decision you are about to make. If you are reading this, chances are this decision will cost you dearly. And chances are, it's the right decision, and one that few will make.

No Comments »


Introducing LoveSeat – a light-weight editor for CouchDB

Cats: Uncategorized

Tue - 09 Feb 2010 - 11:41 PM

Edit:by popular demand, I went back and made sure that LoveSeat works under mono. a few minor tweaks later, we're good to go. See below for pics.

In the course of my work on friendsell, i got fed up with the rather limited view function editing abilities of futon, and rolled my own. The main goals were getting out of utf-8 cr-lf hell, avoid having to escape quote strings, and avoid having to manually tweak json docs. It's far from perfect (or done), but it did cover the main use-cases.

LoveSeat lets you connect to a cdb database, browse the databases and design documents within it, CRUD design docs and call views.

Screenshot-LoveSeat

loveseat-1

When running views, results are represented in a simplistic (but for the time being adequate) tree view

loveseat-2

The current set of context menus is aware of views, map and reduce function and lucene indices
loveseat-3

Finally, there are a few maintenance tasks supported - extract and importing all design documents in a database as flat javascript files, and cloning the design docs in one database to another
loveseat-5

I"ll be expanding this app in the foreseeable future, first and foremost focusing on mono compatibility, and then moving on to some core features that might help.

Loveseat is available on GitHub

15 Comments »


want brevity in .net mvc code? switch to f#.

Cats: Uncategorized
Tags: , ,

Fri - 18 Dec 2009 - 03:56 PM

It's been a while (seems like my writing is slightly manic - either all or nothing), but i actually have something to write about. I finally got a chance to clean up the FSharpExtensions project for bistro (you can find the new release here). While the underlying concepts are the same, a recent stint with Django (which is a strong influence in the Bistro framework) reminded me of how little work you have to do in python to get going with a controller, and that bistro and the fs extensions still have some catching up to do on the brevity front.

So here's what's new (from the download page)

Return values can now contain expressions, using the 'named' aliasing function: given function foo(), "foo() |> named 'bar'" will place the result of calling 'foo' onto the request context under the name 'bar'

Translation: before you had to

1
2
3
4
[<Bind("get /hello/world/{parm}"); ReflectedDefinition>]
let reallySimpleController (ctx: ictx) parm =
    let foo = parm + "2"
    foo

but now, you can

1
2
3
[<Bind("get /hello/world/{parm}"); ReflectedDefinition>]
let reallySimpleController (ctx: ictx) parm =
    parm + "2" |> named "foo"

Session values do not need aliasing - the statement 'SessionValue foo' will place 'foo' onto the session context directly

Which means this will work without the 'named' function shown above

1
2
3
[<Bind("get /hello/world/{parm}"); ReflectedDefinition>]
let reallySimpleController (ctx: ictx) parm =
    SessionValue parm

The session and request context namespaces have now been merged. This means that it is no longer necessary to qualify function input parameters with the session discriminator, as data will be pulled from wherever it is available. Return values should still use the SessionValue discriminator to specify that something should go onto the session

This one is a bit trickier and has some ramifications (good and bad).

1
2
3
4
5
6
7
8
[<Bind("get /hello/world/{parm}"); ReflectedDefinition>]
let reallySimpleController (ctx: ictx) parm =
    let something = some_complicated_function parm
    SessionValue something

[<Bind("get /hello/world/{parm}"); ReflectedDefinition>]
let slightlyMoreComplicatedController (ctx: ictx) something =
    do_something_else something |> named "something_else"

You can see that I didn't need to use the old-style syntax of having to specify the type of something in my second controller (it would have been (something: string session) before). The drawback of this is that you now can have a collision between the two namespaces, which is bad. There are two considerations here - one, the order of processing is deterministic. Request context always wins, as bistro apps tend to be light on the session. Two, my work with NDjango (which has a similar limitation) has shown me how little I run into that, so it's a limitation i'm willing to live with, seeing as how it drops the need for funky types and wrapping/unwrapping.

A new syntax for form fields is now available. If a controller function takes a record type as a parameter, and that record type is marked with the 'FormData' attribute, the fields of that record are populated from the form data.

This one is pretty cool. Say I have a form (with ndjango syntax):

1
2
3
4
5
6
<form name='foo' method='post' action='/do/something'>
    First Name <input name="fName" value="{{fName}}" />
    Middle Name <input name="mName" value="{{mName}}" />
    Last Name <input name="lName" value="{{lName}}" />
    Weight Name <input name="weight" value="{{weight}}" />
</form>

and a corresponding controller. The old way of handling this form would have been

1
2
3
4
5
[<Bind("post /do/something"); ReflectedDefinition>]
let reallySimpleController (ctx: ictx) (fName: string form) (mName: string form) (lName: string form) (weight: weight form) =
    if String.IsNullOrEmpty fName.Value then report_error "first name is required"
    ...
    fName.Value |> named 'fName', mName.Value |> named 'mName', lName.Value |> named 'lName', weight.Value |> named 'weight'

the new way, however, is a bit neater:

1
2
3
4
5
6
7
8
9
10
[<FormData>]
type somethingForm = {
    fName: string; lName: string; mName: string; weight: int;
}

[<Bind("post /do/something"); ReflectedDefinition>]
let reallySimpleController (ctx: ictx) (data: somethingForm) =
    if String.IsNullOrEmpty somethingForm.fName then report_error "first name is required"
    ...
    ()

All you do is declare a record that defines the fields and data types you care about, and work with that record. What's more, is that by default (this is an overridable parameter to the FormData attribute) the fields are automatically passed through to the request context without your intervention, so you don't have to repeat the field list twice for no good reason.

Neat, huh? All of these changes has taken it to the point where you almost never need type declarations, and the compiler can typically infer the necessary data types.

On the über-geeky side of this, the most complex part of all of this was actually recognizing and parsing out calls to the 'named' function. It took the inference code from this:

1
2
3
4
5
6
7
8
9
10
11
12
   /// attempts to locate the return type of the given expression tree
    let rec try_get_return_sig = function
    | Lambda (_,ex) -> try_get_return_sig ex
    | Let (_,_,cont) -> try_get_return_sig cont
    | IfThenElse (_,t,e) -> choose try_get_return_sig t e
    | NewTuple e -> Some <| List.map (fun (ex: Expr) -> ex.ToString(), ex.Type) e
    | Sequential (f,s) -> choose try_get_return_sig f s
    | TryFinally (tr,fn) -> choose try_get_return_sig tr fn
    | TryWith (tr,_,wt,_,fn) -> choose (choose try_get_return_sig fn) wt tr
    | Var var as ex -> Some [(ex.ToString(), ex.Type)]
    | Value (o, tp) -> if tp = typeof<Unit> then Some [] else None
    | _ -> None

to this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
   let rec get_raw_value = function
    | Value (o, tp) -> o.ToString()
    | Var (e) -> e.Name
    | NewUnionCase (info, lst) when List.exists ((=) info.Name) valid_unions -> get_raw_value <| List.head lst
    | _ -> ""

    let rec extract_named cont =
        match cont with
        | Lambda (p,ex) ->
            match ex with
            | Call (_, mi, exlist) ->
                match mi.Name with
                | "named" -> Some mi.ReturnType
                | _ -> None
            | _ -> extract_named ex
        | ShapeLambda (var2, expr) -> extract_named expr
        | ShapeCombination (obj, expr_lst) -> List.tryPick extract_named expr_lst
        | _ -> None

    let (|CallNamed|_|) = function
    | Lambda (var,ex) ->
        match var.Name, ex with
        | "name", Lambda (var2, ex2) ->
            match var2.Name, ex2 with
            | "expr", Call (_, mi2, exlist) ->
                match mi2.Name, exlist with
                | "named", h::t when h.Type = typeof<String> -> Some (unwrap_type mi2.ReturnType)
                | _ -> None
            | _ -> None
        | _ -> None
    | _ -> None
   
    let (|LambdaNamed|_|) = function
    | "name", Lambda (var2, ex2) ->
        match var2.Name, ex2 with
        | "expr", Call (_, mi2, exlist) ->
            match mi2.Name, exlist with
            | "named", h::t when h.Type = typeof<String> -> Some (get_raw_value h, unwrap_type mi2.ReturnType)
            | _ -> None
        | _ -> None
    | _ -> None

    /// attempts to locate the return type of the given expression tree
    let rec try_get_return_sig = function
    | Call (ex,mi,exlist) ->
        let run_list = List.tryPick (fun e -> match e with | Var (_) | NewUnionCase (_,_) -> None | _ -> try_get_return_sig e)
        match mi.Name with
        | "named" ->
            match exlist with
            | h::t when h.Type = typeof<String> -> Some [get_raw_value h, unwrap_type mi.ReturnType]
            | _ -> run_list exlist
        | "op_PipeLeft" ->
            match exlist with
            | f::s::t ->
                match f with
                | CallNamed t -> Some [get_raw_value s, t]
                | _ -> run_list exlist
            | _ -> run_list exlist
        | "op_PipeRight" ->
            match exlist with
            | f::s::t ->
                match s with
                | CallNamed t -> Some [get_raw_value f, t]
                | _ -> run_list exlist
            | _ -> run_list exlist
        | _ -> run_list exlist
    | Lambda (var,ex) ->
        match var.Name, ex with
        | LambdaNamed (name,tp) -> Some [name, tp]
        | _ -> try_get_return_sig ex
    | Let (var,value,cont) ->
        match var.Name with
        | "name" ->
            match extract_named cont with
            | Some func_type -> Some [get_raw_value value, unwrap_type func_type]
            | None -> try_get_return_sig cont
        | _ -> try_get_return_sig cont
    | IfThenElse (_,t,e) -> choose try_get_return_sig t e
    | NewTuple e ->
        Some (
            List.fold (fun s (ex: Expr) ->
                            match try_get_return_sig ex with
                            | Some l -> s @ l
                            | None -> s) [] e)
    | Sequential (f,s) -> choose try_get_return_sig f s
    | TryFinally (tr,fn) -> choose try_get_return_sig tr fn
    | TryWith (tr,_,wt,_,fn) -> choose (choose try_get_return_sig fn) wt tr
    | Var var as ex -> Some [(get_raw_value ex, unwrap_type ex.Type)]
    | Value (o, tp) -> if tp = typeof<Unit> then Some [] else None
    | NewUnionCase (info, lst) as v when List.exists ((=) info.Name) valid_unions -> Some [get_raw_value v, v.Type]
    | _ -> None

I'm sure there's room for improvement, but a lot of work comes in recognizing all the different ways the quotation tree can be built. The unit tests show all the different invocations. Let's just say that part took some digging...

2 Comments »


linq2couch (look, ma, no puns!)

Cats: Uncategorized

Wed - 23 Sep 2009 - 11:39 AM

Both my work and side projects lately have taken me in for a closer look at http://couchdb.apache.org/. I'll assume that if you're reading this, you've run into it in the past, and will forgo an explanation of what couchdb is.

One of the ndjango contributors had mentioned in passing that it would be nice to have a .net library that would wrap couchdb in an IQueriable, which got me thinking - what would a LINQ implementation for couch look like? Obviously, a full on "orm" implementation would be impractical - the whole point of couch is that querying (views) is done using a limited set of operations on predefined views. The couchdb wiki even goes so far as to say that development of views in couchdb should be treated like schema in a conventional database: do it at the start of the project, don't change it later. Changing a view once the database already has several million documents in it becomes an expensive endeavor. That said, there is still merit in a simplified, type-checked querying approach.

I started off using a great open source .net couch wrapper - Divan, and forked it (you can find mine here, but Göran has already pulled my changes into the official repository), and then used the LINQ provider code from linq in action as a template started off.

Divan already has a full implementation of the couch view api, so this became a simple matter of mapping a linq expression tree onto that api. Using the ExpressionVisitor concept from linq in action, I ended up decomposing the expression tree like this:

 
/// Recursivley processes the expression tree
protected virtual void VisitExpression(Expression expression)
{
    switch (expression.NodeType)
    {
        case ExpressionType.AndAlso:
            hasAnd = true;
            VisitBinary((BinaryExpression)expression);
            break;
        case ExpressionType.OrElse:
            hasOr = true;
            VisitBinary((BinaryExpression)expression);
            break;
        case ExpressionType.GreaterThan:
        case ExpressionType.LessThan:
            throw new NotSupportedException(expression.NodeType + " is not a supported expression");
        case ExpressionType.Equal:
            if (hasAnd)
                throw new NotSupportedException("'and' operations cannot be performed on key sets. All key sets are 'or' operations.");
            if (startKeySet || endKeySet)
                throw new NotSupportedException("Key and range operations cannot be combined in a single query");
            CallIfPresent((BinaryExpression)expression, (val) => keys.Add(val));
            break;
        case ExpressionType.LessThanOrEqual:
            if (endKeySet || hasOr)
                throw new NotSupportedException("Range queries over multiple ranges are not supported");
            if (keys.Count > 0)
                throw new NotSupportedException("Key and range operations cannot be combined in a single query");
            CallIfPresent((BinaryExpression)expression, (val) => Query.EndKey(val));
            endKeySet = true;
            break;
        case ExpressionType.GreaterThanOrEqual:
            if (startKeySet || hasOr)
                throw new NotSupportedException("Range queries over multiple ranges are not supported");
            if (keys.Count > 0)
                throw new NotSupportedException("Key and range operations cannot be combined in a single query");
            CallIfPresent((BinaryExpression)expression, (val) => Query.StartKey(val));
            startKeySet = true;
            break;
        case ExpressionType.Lambda:
            VisitExpression(((LambdaExpression)expression).Body);
            break;
        default:
            if (expression is MethodCallExpression)
                VisitMethodCall((MethodCallExpression)expression);
            break;
    }
}
 

A few things to point out here:

  • Range operations get translated to key ranges in couchdb parlance. That means the expression "where a.b >= 5" will be translated a "startKey" parameter. This also means that a straight GreaterThan and LessThan won't be supported, because the startKey/endKey parameters in couch are inclusive. Also, notice the check for whether an equality operation has been recorded. If it has, we won't allow a range operation, as you can't combine a set of keys with a key range
  • Equality operations get grouped together, and then translated to either a GET with a key="" or a POST with a key range. Again, note that you can't mix equality operators and range operators
  • Boolean operators are also restricted by first usage. You can use ORs to build a key list, and you can use ANDs to build a range (a greater than b and c less than d). ANDs and ORs can't be mixed, just like equality and ranges

So what we end up with is the ability to specify a couchdb view api call using a LINQ syntax. This certainly won't allow you make arbitrary calls or generate views, but it does make your c# apps that interact with couchdb a bit more homogeneous:

Before

 
var hps = new int[] {176, 177};
var twoMoreCars =
    query
        .Keys(hps)
        .GetResult()
        .ValueDocuments<Car>();
 
foreach (var c in twoMoreCars)
    Console.WriteLine(c.Make + " " + c.Model + " with " + c.HorsePowers + "HPs");
 

After

 
var hps = new int[] {176, 177};
var twoMoreCars = from c in linqCars where hps.Contains(c.HorsePowers) select c.Make + " " + c.Model + " with " + c.HorsePowers + "HPs";
foreach (var twoCar in twoMoreCars)
    Console.WriteLine(twoCar);
 

A small digression
While putting this together, I ran into an interesting question. How do you apply the select clause to the result set? The simple answer is to build a facade enumerator for the IQueriable to return. the problem is what to actually return? We need to somehow go from the expression tree within the select clause to a value that can be returned. What I did in linq2couch was to extract out the Select expression as part of my parsing:

 
/// Processes a "MethodCall" node
private void VisitMethodCall(MethodCallExpression expression)
{
    ...
    else if ((expression.Method.DeclaringType == typeof(Queryable)) &&
        (expression.Method.Name == "Select"))
    {
        SelectExpression = expression;
        VisitExpression(expression.Arguments[0]);
    }
    ...
}
 

Then, generate a delegate out of it:

 
public TransformingEnumerator(IEnumerator e, MethodCallExpression transformer)
{
    this.e = e;
 
    var t = (UnaryExpression)transformer.Arguments[1];
    this.transformer = ((LambdaExpression)t.Operand).Compile();
}
 

And then apply it as objects are pulled out

 
public TReturn Current { get { return (TReturn)transformer.DynamicInvoke(e.Current); } }
 

The trick is that the the Func that the select wraps is Func<T1, T2>, where the inbound parameter is of whatever type the queriable is made to be, and the return value is whatever the expression evaluates to be, so the whole thing lines up quite nicely.

2 Comments »


putting the m in bistro mvc

Cats: Uncategorized

Mon - 17 Aug 2009 - 02:04 PM

the last few months of my time have been spent on polishing up bistro. We have come a long way, and have been very happy with what we've seen. The system is now in full use at my day job, and (for the most part) everyone is happy. Of course, status quo boring, and there are always new request that come up. I started talking about the first of these in my previous post about validation. Once implemented, that set us down an interesting direction, as we realized that while bistro never wanted to have a formally defined "model", the reality is that a model still exists, and applications need to interact with it.

The core concept for "model" definition in bistro becomes one of mapping and inference. The idea is that your controller tier, ideally, is a thin wrapper, whose sole purpose in life is to ferry data back and forth between your view and your model, applying some transformations along the way. so, let's say i have this model, controller and view:

 
    // our model
    class Person
    {
        public string firstName, lastName;
    }
 
    // our controller
    [Bind("/entityTest")]
    public class EntityController : AbstractController
    {
        [FormField, Request]
        public string
            firstName,
            lastName;
 
        [Request]
        public Person person;
 
        [Request]
        string validationMessage;
 
        public override void DoProcessRequest(IExecutionContext context)
        {
            if (String.IsNullOrEmpty(firstName) || String.IsNullOrEmpty(lastName))
            {
                validationMessage = "first and last name are required";
                return;
            }
 
            person = new Person() { firstName = this.firstName, lastName = this.lastName };
        }
    }
 
<-- and the view -->
{% if validationMessage %}
    {{ validationMessage }}
{% endif %]
<form method="post">
    <p>First name <input name="firstName" value="{{firstName}}" /></p>
    <p>Last name <input name="lastName" value="{{lastName}}" /></p>
    <p><input type="submit" value="Submit" /></p>
</form>

This is all fairly straightforward. I have an entity of type Person, with fields first and last name, i have a controller that gets those fields from a form, tries to populate them, and if there are validation issues, doesn't do anything, except complain about validation.

So let's start with that - validation. I already talked about a theoretical approach to the problem. Well, in that time the theoretical has become the practical, and here's how we'd do the validation:

First, we implement the new IValidatable interface. It just says that an object (note, not just a controller) can be validated by the bistro validation mechanism, and that the object needs to be able to accept the results of that validation:

 
    // our controller
    [Bind("/entityTest")]
    public class EntityController : AbstractController, IValidatable {
        public List<IValidationResult> Messages { get; set; }
        public bool IsValid { get; set; }
    ...
    }
 

Next, we build a class that will hold the validation rules:

 
    public class SimpleValidator: Validator<EntityController>
    {
        protected override void Define()
        {
            Define(
                Value(c => c.firstName).IsRequired("First Name is required"))
            .And(
                Value(c => c.lastName).IsRequired("First Name is required"));
        }
    }
 

Fairly straightfoward (the Bistro.Extensions.Validation.Common namespace defines a series of extension methods that provide a fairly complete range of validation rules, plus you can define your own). Finally we have to tell the controller who is going to perform the validation, and modify our template to expose the validation rules:

 
    // our controller
    [Bind("/entityTest")]
    [ValidateWith(typeof(SimpleValidator))]
    public class EntityController : AbstractController, IValidatable {
    ...
    }
 
<-- and the view -->
{% if Messages %}
    <ul>
    {% for m in Messages %}
        <li>{{m.Message}}</li>
    {% endfor %}
    </ul>
{% endif %]
<form method="post">
    <p>First name <input name="firstName" value="{{firstName}}" /></p>
    <p>Last name <input name="lastName" value="{{lastName}}" /></p>
    <p><input type="submit" value="Submit" /></p>
</form>

With all that in place, we can now modify the DoProcess method do not worry about the validation rules themselves:

 
    public class EntityController : AbstractController
    {
        ...
 
        public override void DoProcessRequest(IExecutionContext context)
        {
            if (!IsValid)
                return;
 
            person = new Person() { firstName = this.firstName, lastName = this.lastName };
        }
    }
 

Cool. Now we've been able to define our validation rules externally to the controller, and simplified the work the controller has to do. An added benefit of this (that I won't cover much here) is the validation rules are now discoverable externally. The new BistroIntegration branch comes with a set of ndjango tags that read the validation rules, and expose them directly in the html as jquery validators, mapping bistro validators over (pretty cool stuff, really - expect a separate blog post and a dimecast on the matter in the next few weeks).

With that out of the way, we're still left with two unpleasant aspects. First of all, we still have a bunch of code dealing with populating the Person entity with data. In a more complex scenario, we'll also have to deal with populating the controller from the entity, in addition to the other way around (an add/edit screen is a perfect example). Second of all, our rules about what is or isn't required for the entity, are defined on the controller, and if i have another controller that deals with it for whatever reason, i'll have to duplicate those, or risk inconsistent data. The solution to both is simple: make bistro aware of the existance of some poco whose fields the controller is linked to:

 
    class SimpleMapper: EntityMapper<EntityController, Person>
    {
        public SimpleMapper()
        {
            Map(x => x.firstName).To(y => y.firstName)
                .Map(x => x.lastName).To(y => y.lastName);
        }
    }
 

and then tell bistro about it

 
    // our controller
    [Bind("/entityTest")]
    [MapsWith(typeof(SimpleMapper))]
    public class EntityController : AbstractController, IValidatable, IMappable {
        public IEntityMapper Mapper { get; set;}
        public override void DoProcessRequest(IExecutionContext context)
        {
            if (!IsValid)
                return;
 
            person = new Person();
            Mapper.Map(this, entity);
        }
        ...
    }
 

*Cue tv announcer voice* buuut wait, there's more! Why go through manually lining up fields that have the same name and type? Let's do this instead:

 
    class SimpleMapper: EntityMapper<EntityController, Person>
    {
        public SimpleMapper() { Infer(); }
    }
 

And if we need a bit more control, we can do this:

 
    class SimpleMapper: EntityMapper<EntityController, Person>
    {
        public SimpleMapper()
        {
            Infer()
                .Except(x => x.extra)
                .Map(x => x.thirdField).To(y => y.baz);
        }
    }
 

The Infer method will automatically build a mapping for fields that have the same name and assignable types, removing the need to spell everything out. Now that we have our mapping, our final trick is possible. If bistro knows how to move data from EntityController.firstName to Person.firstName, and vice-versa, doesn't it stand to reason that we could do the same thing to validation rules? So the final trick is this: define the validation rules on the entity, rather than the controller:

 
    public class SimpleValidator: Validator<Person>
    {
        protected override void Define()
        {
            Define(
                Value(c => c.firstName).IsRequired("First Name is required"))
            .And(
                Value(c => c.lastName).IsRequired("First Name is required"));
        }
    }
 
    [ValidateWith(typeof(SimpleEntityValidator))]
    public class Person: IValidatable
    {
        ...
 
        public List<IValidationResult> Messages { get; set; }
        public bool IsValid { get; set; }
    }
 

and then use the field mapping to also map over validation rules from the entity to the controller:

 
    class EntityControllerValidator: Validator<EntityController>
    {
        protected override void Define() { ByMapping(); }
    }
 
    [Bind("/entityTest")]
    [ValidateWith(typeof(EntityControllerValidator))]
    [MapsWith(typeof(SimpleMapper))]
    public class EntityController : AbstractController, IMappable, IValidatable
    ...
 

which is exactly what "ByMapping" does.

Whew. That's a lot of stuff. Let's step back for a second and see what we ended up with. We now have a type Person, a controller EntityController, and a mapping between them, that tells bistro how to move data from one to the other, without imposing restrictions on the form on the model class. Additionally, we've defined a series of validation rules on the entity, that we can then also expose to the controller, and subsequently to the UI. Cool, huh?

There are of course a few usability things that we'll be ironing out over the next few weeks (like there are a few more classes that you end up having to define than i like, so that may be replaced with attributes for more common scenarios), but the core concept remains the same. Model mapping and validation are now core concepts of bistro, and an interesting solution to a classic problem.

And, as always, we want your feedback! So if something makes sense, or something doesn't, either let me know here, or shoot the bistro-dev list an email.

No Comments »


not an f# post

Cats: Uncategorized

Thu - 09 Jul 2009 - 10:46 AM

Yes, ladies and gentlemen, i have a day job, and sometimes i must put down the f# koolade and do real work. well... maybe not "real" work per se, but... Anyway, to that end...

I've spent some time scratching my head about the best way to integrate a validation mechanism into bistro. Now... there's not an explicit concept of a model in bistro - bistro is just the controller tier, so that means that there isn't a clear-cut point where you attach validation rules. What's worse is that data coming in from a request may, and more than likely will be split among different controllers for processing. Add on top of that the fact that some validation rules are applicable to multiple scenarios (client side, server side, at the controller level at the entity level), and you have a cluster of wonderful proportions.

I'm starting to think that rather than defining the concept of a generic "model", it's better to define a core of complex data types and associated validations. That way, you can assemble different pieces on the fly and still have a central validation definition.

More specifically - introduce the concept of something "validatable":

 
public interface IValidatable {
	IValidator Validator { get; }
}
 
public interface IValidator {
	bool IsValid(out List<string> messages);
 
	string Name { get; }
	...
}
 

Fairly straightforward. Now, the trick is using this... and not just server side. One of the main things we want to be able to do, is have the rules defined in one place, and made available to both the client and the server. That's where the Name property comes in. By being able to reference a rule (or set of rules), I can get a little fancy client-side. Let's keep with django. I can introduce a tag for making these rules available client side -

 
<form>
{% validate "memberAdd" %}
<input name="firstName" value="{{ firstName }}">
	...
{% endvalidate %}
</form>
 

And have that rule emit jquery validation instructions, as an example. All of this is fairly straightforward. There's still the issue of forward lookups - the controller generating the current form may not necessarily be the controller receiving the form, but that should all be manageable.

The trick comes when you start talking about usability. How do I manage the hundreds of different rules you're going to have here? How do I enumerate them all, how do I manage naming conflicts, how do I combine rules (notice the prominent absence of a list of Validators on the IValidatable interface)?

The answer (in my mind, at least) is composition. Let's assume a typical builder pattern approach:

 
	IValidator Validator {
		get {
			Builder<MyController>()
				.As("memberAdd")
				.For(c => c.firstName).RequireValue()
				.For(c => c.lastName).RequireValue()
				.WithRulesFrom(Member);
		}
	}
 

Now we have a set of rules, that require the first and last name fields, which will be identified by "memberAdd.firstName" and "memberAdd.lastName", and will also expose rules from the "Member" entity, whatever those may be.

Client side, we get access to this rule set through the validate tag, and use matching form element names to make everything line up. You can start playing games with defaults and overriden names and whatnot, but this way I have a mechanism of defining a set of validation rules in a single place, and being able to consume them in multiple other places.

No Comments »


ndjango included in mvccontrib wiki

Cats: Uncategorized

Mon - 15 Jun 2009 - 03:23 PM

So the good folks over at mvccontrib have included NDjango in their documentation as a view engine for asp.mvc. After some discussion with Jeff Palermo and Jeremy Skinner, it was decided that rather than including the ASPMVCIntegration project in the MVCContrib libraries, they'll just put up a reference on their documentation page.

I'm really excited about this, as hopefully this will create more visibility for both NDjango and a mainstream use of F#. We'll see where this goes.

This, combined with some good feedback (so far, at least ;) ) from the alt.net mailing list on bistro, there's a chance that we're on to something here...

No Comments »


bistro f# extensions go beta! and move to codeplex…

Cats: Uncategorized

Wed - 03 Jun 2009 - 10:44 PM

After some more work on the f# extensions project, we've decided that it's stable enough to start showing to people. To that end, we've moved it to a project on codeplex (here), and put up a beta release of it. It's built with the may ctp (1.9.6.16), so you'll need that installed, but the sample project (which is actually the subject of the post below) is included in the release, as a nice way to provide everything you need to get started, and give you an example app.

happy hunting!

oh, and for the time being (at least), the documentation will still stay on the bistro wiki.

2 Comments »
Meta
Posts | Comments | RDF | Atom | Valid XHTML | CSS | Log in