HashKeyCopier - An AngularJS Utility Class For Merging Cached And Live Data
A while back, I blogged about how AngularJS uses an expando property, $$hashKey, in order to associate a $scope-based object with a current node in the Document Object Model (DOM). The existence of this expando property is critical because it prevents AngularJS from accidentally and needlessly re-creating DOM elements (and re-linking Directives) every time your objects change. In my previous post, I described a way to work within this context by copying data from new objects to existing objects; today, however, I wanted to look at a slightly different approach that simply copies over the $$hashKey expando properties from the existing objects into the new objects.
View the HashKeyCopier project on GitHub.
JavaScript operations are super fast. DOM (Document Object Model) operations are, by comparison, much slower. This is why both AngularJS and the browser's own internal rendering engine try to "chunk" updates to the DOM. But, when it comes to AngularJS, performance is not the only consideration - we also have to think about Directives. If a DOM node is re-created, all of the directives attached to that entire descendant DOM tree will be $destroy'd and then re-linked. Not only does this add a significant number of operations, it can also create a palpable performance lag and trigger unexpected events.
Luckily, this concern can be hidden from us (the developers) most of the time. But, if we have locally cached data that needs to be overwritten by live data, this concern becomes very real. In such a situation, it would behoove us to make the live-data merge as performant as possible. To help with this, I have created an AngularJS utility class - HashKeyCopier. This class exposes one "static" method that will recurse through two objects, copying the $$hashKey entries form the source object over to the destination object:
HashKeyCopier.copyHashKeys( source, destination [, uniqueIdentifiers ] ) :: destination
The "source" object is the reference currently being rendered by AngularJS. The "destination" object is the new, live data being retrieved from the server. The copy operation will perform a depth-first walk of the source object, looking for $$haskKeys; it will then perform a depth-first walk of the destination object, looking for logically-equivalent objects into which it will copy the known $$hashKey values.
By default, the HashKeyCopier determines object equivalence by looking for "id" properties. However, you can tell it to use any set of unique identifiers when iterating over the data trees:
HashKeyCopier.copyHashKeys( cached, live, [ "id", "uuid", "offset" ] );
It will search for each unique identifier in the same order in which they were provided. So, using the above example, the copy operation would first look for the key, "id". If it didn't find it, it would look for the key, "uuid." If it didn't find it, it would look for the key, "offset." If none of the keys can be found, the copy operation will simply move on with its depth-first exploration.
To see this in action, take a look at the following code. Here, we have an AngularJS app that renders two identical lists, logging the number of times a given Directive is linked. The only difference between the two lists is the way in which the collections are defined: in the first, the data is strictly overwritten; in the second, the data is overwritten after the $$hashKey values have been copied.
<!doctype html>
<html ng-app="ExampleApp" ng-controller="AppController">
<head>
<meta charset="utf-8" />
<meta name="author" content="Ben Nadel, ben@bennadel.com" />
<title>
HashKeyCopier - An AngularJS Utility Class
</title>
</head>
<body>
<h1>
HashKeyCopier - An AngularJS Utility Class
</h1>
<p>
<a ng-click="updateData()">Update Cached Data With Live Data</a>
</p>
<!--
In this first list, we're going to "update" the target data without copying
over the hash-keys from the cached data to the "live" data.
-->
<h2>
No Keys Copied ( DOM Operations: {{ domOperations.withoutCopyCount }} )
</h2>
<ul>
<li
ng-repeat="friend in friendsWithoutCopy"
bn-log-dom-operation="withoutCopyCount">
<strong>{{ friend.name }}</strong>
<!-- Include nicknames ONLY if they are available. -->
<span ng-switch="!! friend.nicknames">
<em ng-switch-when="true">
( aka.
<span
ng-repeat="nickname in friend.nicknames"
bn-log-dom-operation="withoutCopyCount">
{{ nickname.name }}
<span ng-show="! $last">,</span>
</span>
)
</em>
</span>
( Favorite Number: {{ friend.favoriteNumber }} )
</li>
</ul>
<!--
In this second list, we're going to use the hashKeyCopier to copy over the
proprietary $$hashKey values from the cached data to the "live" data. Notice
that the number of "dom operations" is much less.
-->
<h2>
Hash Keys Copied ( DOM Operations: {{ domOperations.withCopyCount }} )
</h2>
<ul>
<li
ng-repeat="friend in friendsWithCopy"
bn-log-dom-operation="withCopyCount">
<strong>{{ friend.name }}</strong>
<!-- Include nicknames ONLY if they are available. -->
<span ng-switch="!! friend.nicknames">
<em ng-switch-when="true">
( aka.
<span
ng-repeat="nickname in friend.nicknames"
bn-log-dom-operation="withCopyCount">
{{ nickname.name }}
<span ng-show="! $last">,</span>
</span>
)
</em>
</span>
( Favorite Number: {{ friend.favoriteNumber }} )
</li>
</ul>
<!-- Include the core framework libraries. -->
<script type="text/javascript" src="vendor/jquery/jquery-2.0.0.min.js"></script>
<script type="text/javascript" src="vendor/angular-1.0.4/angular.min.js"></script>
<!-- Include our HashKeyCopier module (adds our utility class to the dependency injection). -->
<script type="text/javascript" src="../lib/hash-key-copier.js"></script>
<!-- Define our demo application. -->
<script type="text/javascript">
// I am the AngularJS module for this example. The HashKeyCopier module is a
// dependency so that the "HashKeyCopier" class is added as an injectable into
// the current application's dependency injection container.
var app = angular.module( "ExampleApp", [ "hashKeyCopier" ] );
// I am the main controller for the example.
app.controller(
"AppController",
function( $scope, HashKeyCopier ) {
// -- Controller Methods. ------------------- //
// I mock the retrieval of data from the server. I simply return data
// and make not assumptions about how this data will be used.
function getFriends() {
// Used to create a "fake" random number. NOTE: We are creating the random
// number to demonstrate that data bindings will be updated even if the
// DOM node does not get re-created.
var baseNumber = ( ( new Date() ).getTime() % 123 );
// When creating the friends data, use complex, nested structure to make
// sure that the hashKey copy operation goes deep.
var friends = [
{
id: 1,
name: "Tricia",
nicknames: [
{
id: 1,
name: "Trish"
},
{
id: 2,
name: "Tricialicious"
}
],
favoriteNumber: ( baseNumber * 1 )
},
{
id: 2,
name: "Joanna",
nicknames: [
{
id: 11,
name: "Jo"
}
],
favoriteNumber: ( baseNumber * 2 )
},
{
id: 3,
name: "Sarah",
favoriteNumber: ( baseNumber * 3 )
}
];
return( friends );
}
// -- Scope Methods. ------------------------ //
// I update the collections being used in the demo. Notice that one of
// collections is done using straight reference; the other collection is
// updated using the hash-key-copy optimization.
$scope.updateData = function() {
// Standard collection copy.
$scope.friendsWithoutCopy = getFriends();
// Collection update performed using hash-key-copy optimization. In this
// case, I am using the existing collection as the "source" and the newly
// gotten friends collection as the "destination" into which the AngularJS
// "$$hashKey" values will be copied.
$scope.friendsWithCopy = HashKeyCopier.copyHashKeys( $scope.friendsWithCopy, getFriends() );
};
// -- Scope Properties. --------------------- //
// Initialize both collections with the same collection.
$scope.friendsWithoutCopy = getFriends();
$scope.friendsWithCopy = getFriends();
// Initialize the DOM operation counts. This works hand-in-hand with our
// bnLogDomOperation directive.
$scope.domOperations = {
withoutCopyCount: 0,
withCopyCount: 0
};
}
);
// I simply log the number of times this directive has been linked to a DOM element
// by the AngularJS framework. We're looking at this as a sort of pseudo count for
// the number of DOM operations that AngularJS has to perform.
app.directive(
"bnLogDomOperation",
function() {
// I bind the DOM element to the directive behaviors.
function link( $scope, element, attributes ) {
var countKey = attributes.bnLogDomOperation;
$scope.domOperations[ countKey ]++;
}
// Return the directive configuration.
return({
link: link,
restrict: "A"
});
}
);
</script>
</body>
</html>
The most critical part of the demo is the updateData() method where the collections are overwritten with "live" data:
$scope.updateData = function() {
$scope.friendsWithoutCopy = getFriends();
$scope.friendsWithCopy = HashKeyCopier.copyHashKeys( $scope.friendsWithCopy, getFriends() );
};
As you can see, the second collection uses the HashKeyCopier utility class to copy over the $$hashKeys. As a result, the second lists requires significantly fewer DOM operation in order to render.
Most of the time, you won't have to worry about such things. However, if your application uses a lot of caching, DOM node generation and Directive linking can lead to noticeable performance consequences (and unexpected outcomes). In such cases, you may benefit from reaching under the hood in order to leverage the proprietary, internal mechanisms being used by AngularJS.
Want to use code from this post? Check out the license.
Reader Comments
Great post Ben. You did a good job explaining this and I really like the simplicity of implementing the hashKeyCopier module. It takes away the pain of having to compare hash keys and merging the data manually. i will be using this!
@Josh,
Thank you good sir! Glad you found it useful. If nothing else, I find explorations like this provide a greater understanding of how AngularJS is actually rendering things. Since AngularJS seems so magical, the extra bit of understanding can really make a big difference.
The one thing I realize now, however, is that my code example is so wide :( I took it out of the example page in the GitHub repository, which is much wider than the ~70 chars I like to use in my blog posts.
Damn you horizontal scroll bars!!!!
@Ben - Much of Angular still remains a mystery to me, so posts like this are very helpful and gets me thinking. I wonder why Angular doesn't have this type of optimization built in for collection updating. Are there times when you wouldn't want to do this hash comparison? I'm also curious what kind of overhead there is in processing and matching hashKeys? Is it minimal enough to always assume it's best to use this method?
Don't sweat the code line lengths. Most of the longer lines are comments anyway.
@Josh,
There are probably too many assumptions that have to be made for AngularJS to build this kind of thing into the framework. Not sure. I tried to make it flexible by allowing the developer to define the keys being used as "unique identifiers."
As far as performance goes, I think the framing thought is DOM rendering vs. JavaScript operation. If you don't copy the hashKeys, the DOM nodes *will be* recreated, the directives *will be* relinked.
So, is there a cost to iterating over the data structures and looking for keys... Yes. But, I think that cost probably far outweighs the cost of recreating / relinking DOM nodes and directives.
Plus, remember that in my example, the Directive isn't doing much of anything - just incrementing a counter. Imagine if your Directive was binding event handlers? Then, your link and $destroy proccessing would be doing much more work.
Excellent article Ben.
Do you plan on continuing to post AngularJs related material?
Being new to this framework, I love exactly this kind of post! Thanks!
@Mike,
Oh heck yeah! I definitely intend to keep posting AngularJS experiments and exploration. Glad you're liking it!
Ben,
AngularJS 1.1.4 has expanded the repeat expression to include a "track by" expression - http://code.angularjs.org/1.1.4/docs/api/ng.directive:ngRepeat.
I've updated your example to use the track by expression in place of the HashKeyCopier if you care to check it out - http://plnkr.co/edit/r4Kpoe58VkO6vJjKm5S6?p=preview.
Thank you for sharing your exploration of the code world with all of us, I look forward to / enjoy reading your posts. Keep up the good work sir.
Kevin James
@Kevin,
Very cool! And I see that they added animation for add/remove from a collection. That (animation) is something I've been struggling to think about lately.
It looks like 1.1.4 is still classified as "unstable"; but, the improvements to ngRepeat make it super exciting! I hope they release it soon. Has there been any word on the timing?
Thanks for the insight.
@Kevin, Much thanks to you sir! This is awesome! "track by" will be a nice feature to have. @Ben - The animations will be nice too!
@Ben,
I experiment with both the stable / unstable releases, my bad for not making note of this. The three things I'm looking forward to are Scope#$watchCollection (which has been incorporated into ngRepeat), ngAnimate, and the ability to publish a controller instance to a scope. It's easy enough to do
within the controller but
seems like it'd be readily apparent if you were just digging through the views. All good things in my opinion. I'm not sure about a release date but I'd put money on early to mid June given the release history.
Kevin James
@Kevin,
I just found a YouTube video about the animation stuff: http://www.youtube.com/watch?v=cF_JsA9KsDM Taking a look right now.
@Kevin,
Just watched the ngAnimate video on YouTube. Looks like some really cool stuff. Having that kind of functionality built into the core is going to make a huge difference. Custom animation using directives has been a difficult task (and something that I shy away from in all but the most generic use-cases).
The exposure of the Controller is not something I quite understand (ie. the "Controller as ctrl"). Would the idea be to use the controller to expose methods rather than the $scope... so the $scope is just data and the controller is the "API"?
Such a useful utility, thanks for sharing Ben.
I'm now ~3 weeks into learning Angular thanks entirely to you. I was split between Backbone and Angular but when I saw you chose Angular and that you were blogging about it you made my decision simple.
BTW - I'm feeling a lot like your My Experience With AngularJS post... I just hope I grasp the language as you have (in about a years time).
@Brian,
That's awesome - thank you so much for saying that :) I've really been loving AngularJS, despite the learning curve. And, honestly, part of the learning curve is also just learning how to build large JavaScript apps (at least for me).
We'll rock this beast together!
@Ben,
the ngAnimate video on YouTube was excellent, good find! I fumbled around with some animation ideas a while back but came out of it feeling like I got an ng-beatdown.. so yeah, I'm glad it's making its way into the core hah.
Regarding Controller exposure I believe you are correct. If the methods are exposed via. the Controller then the Controller acts as a service to the scope for its respective portion of the view, more or less, and allows us to keep a cleaner $scope. I've seen this pattern in a couple NG projects and find it intriguing. The latest commits show they're planning on incorporating this functionality into $routeProvider so there must be something to it. Speaking of which have you seen ui-router and if so what do you think of it? https://github.com/angular-ui/ui-router
It's been intense trying to adapt to such a different way of developing JavaScript apps, I feel like I've had to learn / unlearn a whole slew of things. Your adoption of the framework* is super encouraging, too, so keep the posts coming for all our sakes :)
@Kevin,
Yeah, I played around with a little animation stuff too - and it hurt my head. One thing that I found really difficult to handle was:
1. I don't want to animate the FIRST set of items in a collection (ie. the initial collection) - just the newly added and the newly removed.
2. ngRepeat renders on an async-watch, so it always renders AFTER it's parent directive's link function.
This drove me crazy - I had to really create specialized directives that worried about timing and had to coordinate with some common "parent" directive that set flags.
And it was irksome. Having it built into ngRepeat / ngShow / etc is going to be sweet. But even from the video, looks like they still don't differentiate between the initial load and subsequent changes.
@Kevin,
As far as exposing the Controller, I think that is how they did it originally (from what I saw in some other videos). I think $scope *used* to point to the controller instance.
As for ui-router, I've seen it a bit, but have not been part of the conversation in a long time. It seems to be very active in its development. I have a similar demo with nested views:
www.bennadel.com/blog/2441-Nested-Views-Routing-And-Deep-Linking-With-AngularJS.htm
... one of these days, I'd LOVE to re-build mine using ui-router, to see how it compares / contrasts. Just been short on time lately.
Ben - thanks so much for posting these Angular articles and findings, they've been a huge help towards learning one of the more 'complex' JavaScript frameworks out there (IMO). I have been using Angular on several projects lately and absolutely love it, and your blog has been a go-to resource for me.
I was originally a UI designer, but later got into coding, so I love how Angular uses so much html to 'create' the app (far more than other frameworks I've tried), and the ability to use Directives to invent new tags/functionality is so slick.
Keep up the great work!
@Chris,
I'm super excited to hear that my posts are helpful. I am also loving AngularJS; but, it definitely has some caveats and some odd behaviors and some things that just don't seem to "work" at first, until you really understand how some of the features interact. Hopefully I'll keep getting good stuff out there. And, there's some really cool stuff coming out in AngularJS soon.
Thanks a lot for the very useful utility.
Nice company on the picture also. Congrats.
Thanks Ben and Kevin -- the "track by" option in angular 1.1.5 is working great to preserve keys.
Arrrg, I can't wait for the next release! I'm still on earlier stable branches :)
Hi Ben,
- thanks for these great posts. We are many relatively new to Angular (aren't we all?) and your posts really help a lot.
I'm also on the journey of transforming into "webapps" way of thinking, and Angular is really exciting.
Hi Ben,
thanks a lot for this great article. I also read this article of yours: www.bennadel.com/blog/2432-Applying-A-Cached-Response-To-An-AngularJS-Resource.htm. Now I am wondering, if and how those 2 approaches should be combined. What are your thoughts here?
Regards,
schacki
@Lars,
Thanks my man! AngularJS is pretty exciting, agreed. It's really starting to change the way I think about building JavaScript apps. Now, I'm trying to use AngularJS as my foundation for GitHub examples too.
@Schacki,
That's a really good question. When I first started getting into AngularJS, I started using $resource, because that is what everyone else said to do and it's what all the demos are using.
$resource is really interesting because it returns "empty" objects/arrays while the resource is being retrieved from the server. The point is to make the object reference easy-to-use such that it doesn't have to be changed over time -- it simply "hydrates."
However, as I have gotten into AngularJS, I find that I almost NEVER used this feature of $resource. My workflow is typically something like this (pseudo-code):
Since I never show a resource before the UI / data has finished loading (or at least I never show the data rendering before the data is loaded), the whole reference/hydrate thing is completely overkill.
All to say, that when I talked about injecting cache data into the $resource, it was to uphold this feature, even though I never use it.
Sorry, I know I'm off on a bit of tangent; to answer your question, this HashKeyCopier and that cached data work hand-in-hand, and in fact, that is exactly how I use it. My code has stuff like this:
Here, the HashKeyCopier merges any references held by any existing data into the "live" data returned by the resource. This way, if there is cache data, it used, if not, it's ignored.
Hi Ben,
thank you for the post ! It gave me a big hand.
It seems the hashkeyCopier module try to find the same id or other unique identifier between source and destination objects, and then copy the $$hashkey from the source to destination.
Here, I have another scenario: I would like to print 200 logs and make a pagination for it. Each page will display 20 logs so the total page is 10. In scope, there is a show array which size is 20 for showing logs. Every time I change to another page, the show array will be assigned to another 20 logs. Since a log contains lots of key/value data, the performance became very bad when I click the button to go different page.
Finally I found it is a good idea to take $$hashkey's advantage. I copied the show array's $$hashkey to the assigned logs and then assign the logs (with hashkey) to show array. Thus the performance become great.
Don't know if this will cause any side effect.. If it does, please tell me. Anyway, until know it works good.
Just want to provide an idea for non unique identifier use scenario.
Thanks.
For anyone that found this post useful, I'd Recommend this post!!
www.bennadel.com/blog/2707-implementing-ngrepeat-track-by-using-a-directive-in-angularjs-1-0-8.htm