How Much Should Our ColdFusion Applications Actually Know About ORM?

That is how I think of it. I would not put any of the ORM specific functionality into my controller, just as I do not do it with Transfer. Some might disagree, but that is how I do it.

Well, Bob's idea about a base persistent object is VERY intriguing, since it basically takes over the service layer,espcially once he finishes the more advanced version with dependency injection and validation.

http://www.silverwareconsulting.com/index.cfm/2009/9/7/Simple-Base-Persistent-ORM-Object-for-CF9-Now-Available

I'd ditto Jason. To me, this is almost a non-question if you have a good MVC framework in place. Both the controller and view have no idea where the data is coming from - as it should be.

I'd have everything behind a service API. The controller would ask, let's say, GroupsService, for data. The GroupsService will take care of talking to the DAO or straight to the ORM. Point is - the "how" is hidden behind the service layer.

@Ben,

I would likely eliminate the DAO object completely and use EntityXXX() in my service layer instead of talking to the DAO object. Unless the DAO had additional functionality that required it's continued use.

@Ray or @Jason... could you elaborate by what you mean by not putting any ORM in your controller? (I am sure some will have no idea what you mean... and my desire is just to 'make sure' what you mean.) Are you saying you abstract the data interaction through another object (service in this case) to keep the particular data interface from being an issue if it ever needed to be swapped out, updated or the like?

@Ben,

These are all good questions. You've got me thinking for sure. To answer your last questions, I think of my Service layer as the API to my model, Its interface does not change. Its method signatures and returned objects should be the same. What happens inside of them or at any level below that is subject to change.

It is not important to me that the code inside of my service object does not change. In fact, I think it can change all I want, as long at the API for it does not. So If I switch to a DB that does not support hibernate, no problem, I will need to rewrite most of my model, including the service layer, but my controller will not care.

@Ben,

I am intrigued by your questions on encapsulation. I had not really considered that question in this whole thought process.

I guess, just like all of the built-in functions of CFML, I do not really thing of EntitySave() EntityLoad() is methods within an object and more than I think of StructKeyExist() or QueryNew() as methods of a utility object that needs to be passed in.

Maybe I am way off on that line of thinking, but that is my initial reaction.

@John,

Yes, in case I have not already answered that question with my other responses. That is essentially what I am saying.

I have not messed with the CF9 goodies really, but in the context of the Four main patterns in our Model ( Gateway, DAO, Service and Bean) I see the ORM integrating nicely into the Gateway, DAO, and Bean.

Not that it HAS to.. but we can make use of the query functionality in our Gateways and DAO's which would be nice! And of course, we would define the entity it self in the Bean.

With that said, I think that is why it is so important to use a Service layer to wrap around all these components in order to make a ORM/Framework/whatever agnostic API for that entity.

So getStudent() method does not care if it is being called from a DAO that uses the CF9 ORM, Transfer, cfquery, whatever...

Then on top of that, if you are using a Framework, maybe ColdBox? Your event handler will only be invoking methods on your Service Layer, therefore the bulk of your application will remained the same besides maybe rewriting some portions of your Model.

@Ben,

I am not sure that my model would need to change that much. For example, in this pseudocode, here is how my service would change if I switched from Hibernate to Transfer and then switched to a standard DAO.

function getUser(id):User {
return EntityLoad('user', id);
}

function getUser(id):User {
return variables.Transfer.get('user', id);
}

function getUser(id):User {
return variables.userDAO.get(id);
}

My service objects tend to be pretty thin. I'm not saying this is the right way. But to me, this is not really that complex a change if back end swapping needs to be done.

@Ben, Well, like I said, that is my first reaction. I don't know if it is right. But I cannot think of a better way to handle it. If I could pass in a reference to hibernate into my service and interface with it like I can Transfer, then that would be great. But to move all of the ORM functionality into a DAO that does nothing more than call EntityXXX() seems like an unnecessary level of abstraction.

@Ben,

I see a little clearer what you are saying now. And for an existing DAO, maybe that would be a good choice. I've not been involved in that much refactoring (I have done more new development), so there may be unseen pitfalls. But I guess I don't see a reason that it would not work just fine.

@Ben,

If you wanted to, you could create a generic HiberanteDAO that encapsulated ColdFusion's Entity***() methods and inject it into all the services that required the necessary CRUD operations. However, I don't see a huge benefit in adding a DAO layer just for the sake of adding another layer of abstraction. Sooner or later you gotta get a little pragmatic. The key in my opinion is having a stable service layer; what happens behind that is almost moot to an extent.

@Ben,

I think the level of abstraction will really vary on the project. "Using the right tool for the right job" really comes into play here I think.

I mean a silly CMS or a Blog? Why not throw EnityXYZ() into a Service Layer?

But if we developing a really intense system with hundreds of components, I think a deep level of abstraction with the whole (bean, dao, gateway, service) would be beneficial. But to answer that question, I think in this case the EntityXZY() should only deal with the layers that deal with data (DAO and Gateway). After all, we are using it to map attributes to a database... So lets keep it in the layers that deal with that type of stuff.

@Ben

I think the level of abstraction you need truly varies based on the project. If you are working on a project for an "internal" client - e.g. your company, you can make an internal decision to support database 'A'. In this case, there is no need to abstract things to the DAO level, all you would be doing is delegating the Entity***() calls. On the other hand, if you are releasing a product to the public, or to clients, having the ability to configure which implementation of your DAO to use (dependency injection, a factory, etc.) can be useful. In this case having a DAO interface, and multiple DAO implementations could prove be critical. As an example, consider needing to support both CF9 and railo, you could have a CF9DAO using Entity***(), CF8DAO and RailoDAO using CFQuery.

I feel that the main idea is baking in the proper level of encapsulation is a matter of looking at the project and seeing where things may *realistically* need to change.

"So, I guess, the even bigger question becomes - if someone has a decent MVC architecture in place, it seems that adding ORM would change the application only very slightly."

I totally agree with this statement Ben. In general there needs to be separation of concerns, what happens when you want to switch to Reactor, unless you've duly separated out your data access then it'll be a tough change.

@Ben,

If you are going to go with ORM, then you will not be able to escape from ORM. It is a magic window into a universe of entities, but it is not a magically disappearing magic window. Anywhere that you deal with entities, you will need to use the magic window into the universe of entities, the ORM.

ORM interaction is necessarily intermingled in dealing with entities. It goes wherever your entities go.

Within a single small, cohesive app, there is not much reason to create separate service layers abstracting away the ORM. Within a larger app, you may want to move all of the business logic / entity logic into services, and keep only controller logic in the controller.

I don't think of the ORM as just like the database - difficult to use, inflexible, unportable. The ORM is not something you need to abstract away. The ORM is easy to use, flexible, and portable.

Again, the point of the ORM is not to be a data-access layer for you, although that comes as a side-effect. The point is so that you can deal only in persistent objects which you get from the magic window, in an object-oriented fashion, rather than in a record-oriented fashion. Once you have the ORM in place, there is no reason to abstract it away - you are now dealing in objects, which is exactly what you want.

Justice

@Ben,

To answer more specifically: your application should know all about the ORM and all about your objects (whether the controllers know about the ORM or the services know about the ORM depends on the complexity of the controllers and business logic). But the application should know zero, nada, zilch about the database.

The application should deal with objects, and objects alone, and leave the persistence of the objects up to Hibernate.

Database-oriented developers tend to take some time to adjust this new perspective: the point of ORM is that you no longer use a database.

Justice

@Ben

Sort of. I would substitute CFQuery to pull the result set, and then use that to populate an object or an array of objects to return. This way, from an "OO" perspective, the application is working with objects and does not care where they came from.

If you have a well abstracted service layer, and you only need to support a single DB, i've found that developing a DAO layer to be overkill for most projects.

However, if you need to support multiple DB vendors, it makes much more sense to use interfaces and multiple DAOs and let the customer decide which implementation to use through configuration (e.g. MySqlDao, MsSqlDao, etc.). The results from the getById(id) method should always return an object, regardless of which DB vendor you used to store it.

Does that make my intent clearer?

I'm late to the party and are a lot of interesting stuff has already been said. Here are some of my thoughts:

I do not agree with Justice that "your application should know all about the ORM and all about your objects". Yes it should know about your objects, but no, it doesn't need to know about the ORM. I'm not saying that it absolutely shouldn't know about the ORM, but that it doesn't need to, and ideally it shouldn't. The ORM is really just "middleware" that sits between the objects, which the application must know about, and the database, which the application shouldn't know about. In theory one should be able to replace the ORM with something else that does a similar job (e.g., replace CF9 ORM with Transfer for sites running on earlier versions of CF), and that's not possible if your application knows all about the ORM.

That is one of the reasons that I personally think it's a good idea to encapsulate the ORM in a lower-level object, such as a gateway or DAO. Sure, you can put your calls to EntityXXX in your services, but that makes it more difficult to switch out CF9 ORM for something else. If you encapsulate your ORM access in a gateway or DAO (which are one and the same thing in my terminology), which you then compose into either your service, or the objects themselves (as I'm doing with my Base Persistent Object), then you can change your persistence mechanism without having to change any other part of your model. And that's one of the reasons for encapsulation in the first place.

Having said that, I agree entirely with the points that have been made that it might be more pragmatic to simply put the calls to EntityXXX in the service. In some cases you don't need the extra layer of abstraction, so I wouldn't say it's a hard and fast rule (to encapsulate ORM access), but it's definitely my preference.

@Bob

Well said, and I completely agree! To elaborate, the reason I said that in smaller projects, a discreet DAO layer may not be needed is I've found that many times the code in the service layer looks something like: function getUser() { getDao().getUser() }. So you end up just writing a ton of code that only serves to keep up the proper abstractions.

@Bob,

The ORM, like language, like the RoutingEngine, like the ControllerBase, like the TemplateRenderer, cannot be abstracted away (these examples are from arbitrary languages/frameworks). None of these is middleware - middleware is the stuff that hooks into these things. But each of these is a critical choice that has to be made at some point, and cannot easily be reversed.

Yes, in building software, you make decisions that cannot easily be reversed. Yesterday, the decision was between SQL Server, Oracle, Informix, and DB2. Today, the decision is between ColdFusion ORM, Transfer, and Reactor (each of which has transparent support for multiple RDBMSes). Tomorrow, who knows what the foundational infrastructure for building applications will be? In any event, you will still have to make some critical decisions, whether yesterday, today, or tomorrow.

Attempting to abstract away your critical infrastructure building blocks will simply lead to a world of pain, where now you need dozens of XxxDao and XxxGateway components. If you had simply accepted that CF/Hibernate (or Transfer, or Reactor) is going to be a critical piece of infrastructure for your app - which it is, regardless of how hard you try to hide it - the rest of your application will be so much simpler. You are only going to end up with an API with fewer features, less configurability, worse performance, and poorer caching, and which is more brittle and more error-prone, than the CF/Hibernate API. Not to say you suck - but to say that Hibernate is phenomenal technology and trying to cover it up in the sand will simply make your life harder.

You cannot simply swap ORMs as and when you like. It doesn't work - in practice as well as in theory. They each work too differently, have far too different APIs, have far too different levels of complexity, etc. You need a concrete foundation for a house, and you need to put it in place before building the rest of the house, and you can only change it with great expense. But nobody complains about that, or thinks we should be building houses with hot-swappable foundations.

When all you need is a simple function, accessible by browsing to "POST http://example.com/orders/334 HTTP/1.1", which finds an entity and updates some of its properties - it's so simple to do:

transaction {
var o = EntityLoad("Order", Url["id"], true);
o.setStatus = Form["order[Status]"];
o.setEstDeliveryDate = Form["order[EstDeliveryDate]"];
}

The code is already DB-agnostic. It is already clean (ignore the Url and Form scopes for now). It is already SQL-less and object-oriented (insofar as setting properties can be called object-oriented).

If that's all you need in your controller, why are you jumping through hoops with OrderService and OrderDao and OrderGateway? (Between DAO and Gateway, one is DB-specific, the other is DB-agnostic, I think; in any case, the terminology's inventor has already apologized for introducing these concepts in a fit of uninformed glee.)

Justice

A really good discussion thus far. Some good points made by all and lots of architectural head scratching ... always a plus in my book!

As I'm currently designing an app that will straddle CF8 and CF9 and will need to be able to take advantage of ORM once we upgrade our clients the following hold true (for me at least) :

DAO (and Gateway Objects by extension) are all based on a defined interface and a base class

This base class is then extended by specific DAO classes - each present the same interface and the internals change based on an ORM setting in the app configuration bean

So ORM enabled versions of the App use the daoOrm class and normal versions user daoBase (or which ever version is relevant) and are instructed as to which version they should use by the return of the config bean.

The data input for both sub classes is identical (as defined by the interface &| the methods of the base class) so, in theory the business layer (controller or whatever) doesn't need to give a monkey's about whether the app is using ORM or not. Encapsulation is maintained, interfaces stay consistent and the world is a happy shiny place once more.

I'll reiterate this is just the solution I've come up with and may not be best practice or agree with other interpretations of the problem.

Rob

@Justice,

I can only speak from experience, and perhaps it's a result of Transfer being very similar to Hibernate in terms of how it's utilized, but I have found that I can quite easily swap out Transfer and replace it with Hibernate with very little affect on my application, including most of the model.

@Justice

I could not disagree more!

"The ORM, like language, like the RoutingEngine, like the ControllerBase, like the TemplateRenderer, cannot be abstracted away (these examples are from arbitrary languages/frameworks)"

Each of the things you have identified *are* abstractions! TemplateRenderer abstracts which template you end up rendering, RoutingEngine abstracts away routing logic from your app, etc. You are comparing apples and oranges.

"... You are only going to end up with an API with fewer features, less configurability, worse performance, and poorer caching, and which is more brittle and more error-prone, than the CF/Hibernate API..."

How does abstracting data persistence have ANY relevance on harming configurability, caching, etc? IMO it is the direct opposite of your position. I agree that using the API (*within* your DAO) will allow developers to leverage the battle tested code of hibernate. Having your data persistence code localized to a DAO will allow for more flexibility, not less.

For a concrete example, what if you need to introduce a key/value store (such as memcachd, S3, etc)? If you have data persistence code encapsulated in a DAO, you can easily modify it to include this extra functionality, but if your application has EntityXXX() scattered all over the place, you are in for an extremely painful and risky refactor.

@Ben,

Maybe I'm missing something, but I'm not sure what you mean about the transaction issue. I don't see whether you encapsulate ORM access or not having anything to do with how you choose to control transactions.

In a service, for example, you could still do something like:

transaction {
var o = get("Order", id);
o.populate(Form);
update(o);
}

Where get() calls a composed DAO/Gateway to do the EntityLoad() and update() calls the DAO/Gateway to do the EntitySave(). Or you could do something like:

transaction {
var o = get("Order", id);
o.populate(Form);
o.save();
}

Where the DAO/Gateway is actually composed into the Order object, which allows you to call save() on it.

In either case the transaction works just as it would if you were coding your EntityLoad/EntitySave calls in the service.

@Rich,

All of your controllers need to inherit some base controller class (or take mixins from some mixin class) so that your app controllers can become aware of framework-provided services. You get to choose among several base controller classes, and the varying and disparate services that come with the base controller classes. That's a choice of which infrastructure to use. Once you make the choice, it's difficult to switch.

Of course, if for some masochistic reason you want the ability not to depend on any given framework, you can certainly implement multiple base-controller-adapter layers, one for each framework you want to support as a base controller class.

Here's the tricky part. If all you use are the most rudimentary features common to all base controller classes, then this is easy. But if you need advanced features which some support in one way, others support in vastly different ways, and still others do not support, you have a problem.

If you can actually switch with ease between cfquery, Transfer, and CF/Hibernate, then that indicates to me you are only using some of the rudimentary features of ORM and that you are not taking full advantage of the different advanced features that each ORM supports.

- Do you configure default fetching strategies per persistent class as well as per persistent relation, and then in your entity queries use different fetching strategies based on performance profiling?
- Do you use the second-level cache? Do you use optimistic concurrency?
- Do you minimize database roundtrips by using batch updates on flush and configuring the ORM to flush only where strictly necessary?
- Do you use transitive persistence (persisting all objects that are reachable from the object passed in to be persisted)?
- Does the ORM automatically watch all objects it knows about for changes, and automatically flush down to the database all changes it finds (without anyone telling it to do so, or tell it to save changes to the database on a change-by-change basis)?
- Do you configure filters in your mappings?
- Do you use multi-column primary keys? Do you use custom primary key classes?
- Do you have custom datatypes that need to be persisted (configured through the ORM extension points), or do you only use the standard supported datatypes?

Obviously, if all you use is get-entity-by-pkey and get-associated-entities, and don't care about fine-tuning performance, and have a very simplistic data model, and take care to tell the ORM about all changes you make, then you easily switch from ORM to ORM, and you can easily write an ORM-adapter layer to facilitate that.

But if you use the advanced features of a given ORM, it would be far more difficult to make that kind of arbitrary switch.

As for your concrete example, that is a fairly complex case and there are a ton of sticky issues there. But I don't think they are solved by sticking all of the code that deals with loading and managing entities into a separate DAO layer will help that much.

Justice

@Justice

In your bulleted list, you are getting very detailed about the configuration parameters of an ORM. While I agree that these parameters need to configured and tuned appropriately, I think they all fall behind the data persistence layer. For any given ORM framework that you implement, you will need to have very custom configuration, but that concern should not leak into your application.

Regarding your controller example - that is again mixing apples and oranges. Your controller should be separate from your service/dao layer, and yes your controller may need be tied to your MVC framework of choice. If you needed to also support a Flex interface for example, you should be able to easily reuse your service/dao/domain, which has no ties to your controller. The whole point of encapsulation is to ease the pain of these types of changes to your system. This is not just a CF best practice, or one of java, it's universal to programming.

@Rich,

It is rather difficult to pretend that your application does not depend on infrastructure. As long as you don't have to swap infrastructure just to install the thing on a different machine, to connect to a different database, to get significantly different performance characteristics, etc., I see no reason to hide the infrastructure you depend on behind an adapter layer. I see no reason to limit myself to the lowest common denominator.

Take a look at how Rails does controllers - the same controller (inheriting the Rails controller base) can render HTML, XML, JSON, JSONP, YAML, or any format at all with the flick of a switch. This feature makes developing web apps in Rails so easy - but if you depend on Rails' advanced features, then you can't simply switch frameworks on a whim.

It may certainly be the case that you will need a service layer to support various types of scenarios (controllers responding with Web pages, Flex, etc.). In that case - certainly put all business logic common to all scenarios in the service layer. But then the service layer should have free use and full knowledge of the chosen ORM infrastructure, and the front layer will simply dispatch to the appropriate service methods.

Only two of the items in my bulleted list are tied down to configuration. The rest of them cannot be contained within a data-access layer; they are how the top-level code or the service-layer code deals with multiple data-access layer calls. Again, they inform how you write your application code, not simply the ORM config you set up. You cannot hide this stuff behind a data-access layer - or if you tried, it would turn out to be a very leaky abstraction at best.

Justice