Creating A Bidirectional Infinite Scroll Page With jQuery And ColdFusion
A few days ago, I explored the concept of creating an "infinite scroll" effect using jQuery (with AJAX) and ColdFusion. The idea behind infinite scrolling is that as the user scrolls down the page, more content is dynamically appended to the content area such that the user can keep scrolling on and on forever (or at least until they run out of data or their browser crashes). In the comments to that post, George Bridgeman, Gareth Arch, and myself discussed the pros and cons of this kind of approach.
With an infinite scroll effect, there is arguably more demand placed on the content sever since requests for more content are made as quickly as the user can scroll (which is quite quickly). Of course, a similar demand could be placed on the server if the user kept clicking the "Next" pagination button in fast succession. But, server issues aside, does this effect provide a good user experience? Strictly from a personal standpoint, yes, absolutely - I've seen this effect used many times and I always appreciate it.
But going beyond the emotional perception, it was raised in the discussion that continually adding content to the page could quickly overload the browser's DOM and cause a crash (or at least some sluggishness). To this point, Gareth brought up the idea that if we could dynamically alter the content at both the bottom and the top of the document, we could keep the infinite scroll effect without creating an obese document object model. I liked this idea a lot, and started playing around with the following "bidirectional infinite scrolling" effect.
NOTE: Sorry the scrolling doesn't show up very smoothly in this video (but trust me, it's smooth, not jumpy).
Now that you've seen the "demo" video, here is a closer examination of the code. Because there is a lot of code, I skipped a bunch of it and just concentrated on what I felt was the most exciting part: dealing with window offset as content is both added to and removed from the page.
At the abstract level, this is not much more complicated than a downward-only infinite scroll; we're still just thinking about a view frame and the offset of the content. The only abstract difference this time is that we are thinking about the top of the content as well as the bottom:
At the abstract level, it's just more math. At the practical level, however, things get a lot more exciting (read as: complex). In the downward only version, when we added content to the bottom of the page, the scroll bar changed, but the view frame offset did not; as such, it was not an issue. This time, however, we are adding and removing content from the top of the document. Both of these actions (add and remove) have direct and obvious effects on the view frame offset. As such, when we alter the top of the content, we have to take care to maintain the user's "perceived" position within the document.
In order to do this - keep the user in the "same" spot as the content is mutated - we need to adjust the view frame offset manually. To do that, however, we sort of need to know the height of the content being added or taken away. And, to do that, it is sort of required that the content being added or taken away has an easily calculated height. And, to ensure that, we sort of need the new content to not "flow" into the old content (ie. we need a clean horizontal cut between content chunks).
As you can see, at the practical level, a bidirectional, infinite scroll requires a few more content constraints than the downward-only infinite scroll. If we can embrace these new constraints, however, building the effect isn't too much of a nightmare. In the following code, I am using jQuery to create an AJAX-powered, bidirectional infinite scroll effect that gets its content from a ColdFusion server. The ColdFusion code will shown at the end.
<!DOCTYPE HTML>
<html>
<head>
<title>Bidirectional Infinite Scroll With jQuery And AJAX</title>
<style type="text/css">
div.list-chunk {
padding-bottom: 1px ;
}
div.list-item {
border: 4px solid #E0E0E0 ;
margin: 0px 0px 13px 0px ;
padding: 10px 10px 10px 10px ;
}
div.list-item a.thumbnail {
float: left ;
height: 150px ;
margin: 0px 15px 10px 0px ;
width: 100px ;
}
div.list-item a.thumbnail img {
border: 1px solid #999999 ;
display: block ;
}
div.list-item h4 {
font-size: 18px ;
margin: 0px 0px 12px 0px ;
}
div.list-item div.offset {
border-top: 1px solid #CCCCCC ;
clear: both ;
font-size: 11px ;
padding-top: 4px ;
}
</style>
<script type="text/javascript" src="../jquery-1.4a2.js"></script>
<script type="text/javascript">
// I get more list items and either prepend them or append
// them to the list depending on the target area.
function getMoreListItems(
container,
targetArea,
onComplete
){
// Check to see if there is any existing AJAX call
// for the list data items. If there is, we want to
// return out of this method - no reason to overload
// the server with extraneous requests (more so than
// an infinite scroll effect already does!!).
if (container.data( "xhr" )){
// Let the active AJAX request complete.
return;
}
// Get the min and max offsets of the current
// container.
var minOffset = (container.data( "minOffset" ) || 0);
var maxOffset = (container.data( "maxOffset" ) || 0);
// The count of list items to load per AJAX request.
// We are calling it a "chunk" size because each
// list chunk will be stored in its own sub-container
// to make DOM manipulation easier.
var chunkSize = 3;
// Check our target area to see what our next offset
// for loading should be.
if (targetArea == "top"){
// We are prepending list items.
var nextOffset = (minOffset - 1 - chunkSize);
} else {
// We are appending list items.
var nextOffset = (maxOffset + 1);
}
// Launch AJAX request for next set of results and
// store the resultant XHR request with the container.
container.data(
"xhr",
$.ajax({
type: "get",
url: "./bidirectional.cfm",
data: {
offset: nextOffset,
count: chunkSize
},
dataType: "json",
success: function( response ){
// Apply the response to the container
// for the given target area.
applyListItems( container, targetArea, response );
},
complete: function(){
// Remove the stored AJAX request. This
// will allow subsequent AJAX requests
// to execute.
container.removeData( "xhr" );
// Call the onComplete callback.
onComplete();
}
})
);
}
// I apply the given AJAX response to the container.
function applyListItems( container, targetArea, items ){
// Get a reference to our HTML template for a new
// list item.
var template = $( "#list-item-template" );
// Create an array to hold our HTML buffer - this will
// be faster than creating individual DOM elements and
// appending them piece-wise.
var htmlBuffer = [];
// Loop over the array to create each list element.
$.each(
items,
function( index, item ){
// Modify the template and append the result
// to the HTML buffer.
htmlBuffer.push(
template.html().replace(
new RegExp( "\\$\\{(src|offset)\\}", "g" ),
function( $0, $1 ){
// Return property.
return( item[ $1.toUpperCase() ] );
}
)
);
}
);
// Create a list chunk which will hold our data.
var chunk = $( "<div class='list-chunk'></div>" );
// Append the list item html buffer to the chunk.
chunk.append( htmlBuffer.join( "" ) );
// Create the min and max offset of the chunk.
chunk.data( "minOffset", items[ 0 ].OFFSET );
chunk.data( "maxOffset", items[ items.length - 1 ].OFFSET );
// Check to see which target area we are adding the
// list items to (top vs. bottom).
if (targetArea == "top"){
// Get the current window scroll before we update
// the list contente.
var viewTop = $( window ).scrollTop();
// Prepend list items.
container.prepend( chunk );
// Now that the chunk has been added to the page,
// it should have a height that can be calculated.
var chunkHeight = chunk.height();
// Re-adjust the scroll of the window to make sure
// the user doesn't suddenly jump to a crazy place.
$( window ).scrollTop( viewTop + chunkHeight );
// Now that we moved the list up, let's remove
// the last chunk from the list.
container.find( "> div.list-chunk:last" ).remove();
} else {
// Append list items.
container.append( chunk );
// Check to see if we have more chunks than we
// want (an arbitrary number, but enough to make
// sure we can comfortable fill the page).
if (container.children().size() > 3){
// We want to remove the first chunk in the
// list to free up some browser memory.
// Get the current window scroll before we
// remove a chunk.
var viewTop = $( window ).scrollTop();
// Get the chunk that we are going to remove.
var oldChunk = container.children( ":first" );
// Get the height of the chunk we are about
// to remove.
var oldChunkHeight = oldChunk.height();
// Remove the hunk.
oldChunk.remove();
// Now, we need to ajust the scroll offset
// of the window so the user is not jumped
// around to a crazy place.
$( window ).scrollTop( viewTop - oldChunkHeight );
}
}
// Now that we have updated the chunks in the
// container, let's update the min / max offsets of
// the container (which will be used on subsequent
// AJAX requests).
container.data(
"minOffset",
container.children( ":first" ).data( "minOffset" )
);
container.data(
"maxOffset",
container.children( ":last" ).data( "maxOffset" )
);
}
// I check to see if more list items are needed based on
// the scroll offset of the window and the position of
// the container. I return a complex result that not only
// determines IF more list items are needed, but on what
// end of the list.
//
// NOTE: These calculate are based ONLY on the offset of
// the list container in the context of the view frame.
// This does not take anything else into account (more
// business logic might be required to see if loading
// truly needs to take place).
function isMoreListItemsNeeded( container ){
// Create a default return. This return value contains
// requirements for both the top and bottom of the
// content list.
var result = {
top: false,
bottom: false
};
// Get the view frame for the window - this is the
// top and bottom coordinates of the visible slice of
// the document.
var viewTop = $( window ).scrollTop();
var viewBottom = (viewTop + $( window ).height());
// Get the offset of the top of the list container.
var containerTop = container.offset().top;
// Get the offset of the bottom of the list container.
var containerBottom = Math.floor(
containerTop + container.height()
);
// I am the scroll buffers for the top and the bottom;
// this is the amount of pre-top and pre-bottom space
// we want to take into account before we start
// loading the next items.
//
// NOTE: The top buffer is a bit bigger only to make
// the transition feel a bit *safer*.
var topScrollBuffer = 500;
var bottomScrollBuffer = 200;
// Check to see if the container top is close enough
// (with buffer) to the top scroll of the view frame
// to trigger loading more items (at the top).
if ((containerTop + topScrollBuffer) >= viewTop){
// Flag requirement at top.
result.top = true;
}
// Check to see if the container bottom is close
// enought (with buffer) to the scroll of the view
// frame to trigger loading more items (at the
// bottom).
if ((containerBottom - bottomScrollBuffer) <= viewBottom){
// Flag requirement at bottom.
result.bottom = true;
}
// Return the requirments for the loading.
return( result );
}
// I check to see if more list items are needed, and, if
// they are, I load them.
function checkListItemContents( container ){
// Check to see if more items need to be loaded at
// the top or the bottom (based purely on position).
// Returns: { top: boolean, bottom: boolean }.
var isMoreLoading = isMoreListItemsNeeded( container );
// Define an onComplete method for the AJAX load that
// will call this method again to make sure there is
// always enough data loaded on the page.
var onComplete = function(){
checkListItemContents( container );
};
// Check to see if more list items are needed at the
// top. If so, we will check to offsets to see if the
// load needs to take place.
//
// NOTE: Position is only *part* of how we determine
// if additional content is needed at the top.
if (
isMoreLoading.top &&
container.data( "minOffset" ) &&
(container.data( "minOffset" ) > 1)
){
// Load and prepend more list items.
getMoreListItems(
container,
"top",
onComplete
);
// Check to see if more list items are needed at the
// bottom. For this, all we are going to rely on is
// the offset of the container (since we can load
// ad-infinitum in the bottom direction).
} else if (isMoreLoading.bottom){
// Load and append more list items.
getMoreListItems(
container,
"bottom",
onComplete
);
}
}
// -------------------------------------------------- //
// -------------------------------------------------- //
// When the DOM is ready, initialize document.
jQuery(function( $ ){
// Get a reference to the list container.
var container = $( "#container" );
// Bind the scroll and resize events to the window.
// Whenever the user scrolls or resizes the window,
// we will need to check to see if more list items
// need to be loaded.
$( window ).bind(
"scroll resize",
function( event ){
// Hand the control-flow off to the method
// that worries about the list content.
checkListItemContents( container );
}
);
// Now that the page is loaded, trigger the "Get"
// method to populate the list with data.
checkListItemContents( container );
});
</script>
</head>
<body>
<h1>
Bidirectional Infinite Scroll With jQuery And AJAX
</h1>
<div id="container">
<!-- Content will be loaded here dynamically. -->
</div>
<!--
This is the tempalte that will be used when adding the
list items to the DOM. It contains two different variable
place-holders:
${src} : The source of the image.
${offset} : The offset of the record.
-->
<script id="list-item-template" type="application/template">
<div class="list-item">
<a href="./${src}" target="_blank" class="thumbnail">
<img src="./thumbs/${src}" width="100" height="150" />
</a>
<div class="details">
<h4>
${src}
</h4>
<p>
Pauline Nordin is a fitness figure professional
athlete, AST sports science spokesperson,
Fitness journalist, fitness model, TV
personality, Trainer and Nutrition coach,
Swedish Bodybuilding champion three years in a
row, Cover model Ironman magazine, Cover model
Body Magazine, fitness profile etc etc.
</p>
</div>
<div class="offset">
Offset: ${offset}
</div>
</div>
</script>
</body>
</html>
I won't go into too much explanation at this point since I have both video and heavily documented code. The one thing I will say, however is that in the above code, I differentiate the "desire" to change the content from the "decision" to change the content. By that, I mean that I have one function, isMoreListItemsNeeded(), which looks purely at the content offset relative to the view frame when flagging an area (top or bottom) as needing more content; I then have a different function, checkListItemContents(), which takes those "mutation flags" and decides whether or not to put them into action (factoring the min and max record offset into the logic). This separation made the problem easier for me to think about.
If you are interested, here is the ColdFusion code:
bidirectional.cfm
<!--- Param the offset variable. --->
<cfparam name="url.offset" type="numeric" default="1" />
<cfparam name="url.count" type="numeric" default="30" />
<!--- Create the return array. --->
<cfset items = [] />
<!---
Loop over the count to populate the return array with
test data. In this case, test data is 1 of 5 randomly
selected thumbnail src values.
NOTE: We are including the offset of each record to
make the Javascript a bit easier (no offset calculations
will be required).
--->
<cfloop
index="index"
from="#url.offset#"
to="#(url.offset + url.count)#"
step="1">
<!--- Create a random item. --->
<cfset item = {
offset = index,
src = "girl#randRange( 1, 5 )#.jpg"
} />
<!--- Append a random image name. --->
<cfset arrayAppend( items, item ) />
</cfloop>
<!--- Serialize the array. --->
<cfset serializedItems = SerializeJSON( items ) />
<!--- Convert it to binary for streaming to client. --->
<cfset binaryItems = toBinary( toBase64( serializedItems ) ) />
<!--- Set the content length. --->
<cfheader
name="content-length"
value="#arrayLen( binaryItems )#"
/>
<!--- Stream binary content back as JSON. --->
<cfcontent
type="application/x-json"
variable="#binaryItems#"
/>
The additional constraints placed on the content - specifically the requirements of clean horizontal lines - definitely limits the kind of scenarios in which a bidirectional infinite scroll effect can be used. I'll put my thinking cap on to see if I can come up with a clever way to use this in conjunction with floating content.
Want to use code from this post? Check out the license.
Reader Comments
@Ben,
Nice solution to the problem!
They do something similar with the DataGrid component in Flex. They draw the grid (adding a few extra rows at the bottom for padding), then recycle the display elements when the user scrolls through the rows of data and update the data that's contained within each row. This way it's just data that's returned from the server (as a query, or array of objects, etc), and the view is modified with the data, and as you show, it lessens the burden on the client side.
Cool! :)
That's an interesting technique, and an interesting way of achieving it (not something I've seen before) but is it really necessary to use this technique? - I guess you're trading the size of the page for more scripting overhead, but does it improve the experience that much?
@Gareth,
Sounds cool - I don't have much experience with Flex.
@James,
I happen to think that it makes a very nice experience. I don't know if the "bidirectional" part is necessary; but, I know that on pages where I am searching for something, the ability to just keep scrolling is VERY nice. It certainly cuts down on having to think about *anything* else other than the search.
I like this approach. It reminds me of those Firefox and Chrome extensions that auto-paginate. The next 'chunk' of content is appended to the bottom of the page. They don't remove in the other direction, though.
As an aside, I always have wished that the Google search results page would do this. I don't like that I can only pull back 10-100 results before I actively have to click to see the next page. If I am scrolling past the first set of results it makes sense to return the next set of results automatically....
@Kristopher,
Yeah, I think for searching, this kind of approach creates a really nice experience. I too have wished that Google did this, especially for Google Images.
In fact, the more I think about it, the more I think this is perfect for any "visual" search where one can quickly scan images and keep scrolling.
For text-only search results, I am not sure that this is applicable as it's not really data that lends well to the quick-scan.
Awesome, this is fantastic stuff.
Question: I'm trying to recreate the server stuff in PHP, what is the data that needs to get sent back, exactly?
I've created my php script and it returns the following JSON:
[{"scr":"https:\/\/s3.amazonaws.com\/gbblr_2\/100\/IMG_1403 - original.jpg","offset":"2594"},{"scr":"https:\/\/s3.amazonaws.com\/gbblr_2\/100\/IMG_1402 - original.jpg","offset":"2593"},{"scr":"https:\/\/s3.amazonaws.com\/gbblr_2\/100\/IMG_1401 - original.jpg","offset":"2592"},{"scr":"https:\/\/s3.amazonaws.com\/gbblr_2\/100\/IMG_1400 - original.jpg","offset":"2591"},
...
In the console, I see it's calling the same JSON call again and again:
GET ./getcontent.php?offset=1&count=3
200 OK
14ms
GET ./getcontent.php?offset=1&count=3
200 OK
16ms
GET ./getcontent.php?offset=1&count=3
I would expect it to change the offset and the count on each call.. should I expect that? If so, then I'm probably returning the wrong data?
Thanks for any pointers you could give, if you don't have time, no problem :)
I've changed the JSON to return this:
[{"src":"https:\/\/s3.amazonaws.com\/gbblr_2\/100\/IMG_1400 - original.jpg","offset":"5"},{"src":"https:\/\/s3.amazonaws.com\/gbblr_2\/100\/IMG_1399 - original.jpg","offset":6},{"src":"https:\/\/s3.amazonaws.com\/gbblr_2\/100\/IMG_1398 - original.jpg","offset":7}]
But in the code, both "src" and "offset" are shown as "undefined", instead of their values.
Still trying to find the right format for the json :)
This might be useful for others: I got this working, by making sure that in the JSON the words "src" and "offset" are all-caps: SRC and OFFSET. So just make sure the json uses all-caps for those two words.
Also, there seems to be some bug (or some mistake I'm making) using this in Chrome, with images, that causes the scroll-loading code to go crazy and load infinitely. It works fine in Safari and Firefox though.
@Peter,
Sorry for not helping you debug this; my inbox is a huge beast to battle from time to time. The uppercase keys are a byproduct of working with ColdFusion. In ColdFusion, all object keys are converted to uppercase when serialized to JSON (unless created with array-notation). PHP has a more common behavior, which is why you probably were running into problems.
In any case, I'm glad you got it sorted!
Why infinte scroll except to dodge advertisements? I bet the ad money guys are not liking this a bit at the giant searches.
@Richard,
Ha ha, yeah, the ad stuff is probably getting hurt. But, the infinite scroll, in general, is a really nice user experience. YOu can see that things like Twitter and Facebook are really jumping on this band wagon.
Total scroll on search results is available now. If the data base has information, scroll up and down at will after search reults loading on Norele.com.
Ben, very nice solution.
Where can i see this in action?
I just googled infinite scroll while using my online text book, and wishing I had an infinite side scroll, that I could easily scroll back to previous pages instead of having to click and wait for the browser to load the pages, then do the same to return to my original page. I just needed to tell someone.
@Jake,
Not trying to spam you but.... Scrolling search results are possible with my website, http://www.norele.com All the results are on one page, limited to a lot less than search results that include synonyms and close spelling. It even has sub-catagories of exact match, all the words, then the rest per each search querry.If you don't find your querry, try a different search string. Simple.
This is super awesome. Has anyone seen this get implicated in the wild? I'm not sure what Richard Hance is talking about but it doesn't seem to be related?
Thanks for sharing! This post helped me a lot with setting up the architecture of a jQuery plugin I'm writing that does something similar. Having the short videos and well commented code snippets is a great presentation method. Thanks again!
Hi,
I have tried the above code but it is not working. In fact I am not able to make it work. Could you please provide me one working example of it?
Best Regards,
Kalpesh
Hi,
do you have a DEMO?
I didn't understand the response model.
Thanks for that good post - by examining your code, i found the bug in my own home-grown solution (developed from scratch, but "somewhat" stable :-)
Regards
Imprssive solution. I injected the code in my project and felt its a must have for a great search experience.
I need to know what changes i need to make to display "End of the search" message after few search results.