Creating A PouchDB Plugin For Bulk Document Updates
Yesterday, I looked at how to update multiple documents in a PouchDB database. Unlike in SQL, where we can run arbitrary UPDATE statement, in a document database like PouchDB, we have to fetch the full documents, update them in memory, and then push them back into the database. This isn't necessarily complicated; but, it is certainly tedious. And, it strikes me as exactly the kind of tedium that could be encapsulated inside a PouchDB plugin.
Run this demo in my JavaScript Demos project on GitHub.
Much like jQuery plugins, PouchDB plugins are just functions that are added to the PouchDB prototype. If you look at the PouchDB source code, you can see that the static .plugin() method does nothing more than iterate over your plugin definitions and inject them into the prototype:
// Extracted from PouchDB source code.
Object.keys( obj ).forEach(
function ( id ) {
PouchDB.prototype[ id ] = obj[ id ];
}
);
This means that when a new PouchDB instance is created, your plugin is available as an instance method. And therefore, when your plugin method is invoked, it is executed with "this" bound to the given PouchDB instance. Basically, this is how JavaScript works - it isn't specific to PouchDB.
Now, keep in mind that PouchDB provides a Promise-based API, which is currently polyfilled with the Lie library. This means that if your plugin makes use of core PouchDB methods like .allDocs(), it can tap into that Promise workflow. But, it's important to remember that your plugins are nothing more than instance methods on PouchDB. This means that your plugins are not inherently wrapped in Promises. So, any top-level errors that your plugin might throw (either explicitly or implicitly due to invalid inputs) may not be wrapped up in a Promise fulfillment.
CAUTION: If your plugin method is coincidentally being invoked inside of an existing Promise chain, your errors will be wrapped up in a Promise fulfillment. But, this is strictly coincidental!
To bridge this Promise gap, the PouchDB Plugin seed project uses the higher-order function, toPromise(), in order to proxy the invocation of your plugin method, ensuring that top-level errors precipitate rejected Promises. But, from what I can see, using the toPromise() method requires a build-step that can bundle files using the CommonJS module format. This means that if you are just writing simple stand-alone plugins, you're out of luck.
This strikes me as a bit odd. I can understand why other utility classes would be hidden or require a build step; but, it would seem helpful to have some sort of static method on PouchDB for creating Promises. Especially since PouchDB polyfills the Promise library; but, does so without actually altering the global object. Meaning, it only polyfills the Promise library internally to the PouchDB library. This means that, as a PouchDB plugin author, you don't have access to a normalized Promise library; at least, not without a built step.
I bring all this up as a prologue to point out that my PouchDB plugin demo is actually "incomplete". Meaning, my plugin invocation is not being proxied and will, therefore, not wrap top-level errors in Promise rejections. But, I believe that everything else about my PouchDB plugin is standard.
Pontification aside, my bulk update plugin is fairly simple. In encapsulates the following logic:
- Perform a bulk fetch using either .allDocs() or .query().
- Iterate over each fetched document and apply the provided operator() function.
- Perform a bulk put using .bulkDocs().
Since the .query() method requires the View name, I'm routing to either .allDocs() or .query() based on the invocation signature of the plugin at runtime:
- .updateMany( options, operator ) : Fetches using .allDocs().
- .updateMany( viewName, options, operator ) : Fetches using .query().
Since both the .allDocs() method and the .query() method return similar result structures, I can handle the update and put operations uniformly once the bulk fetch has returned:
// I provide an API for updating many documents (encapsulating the fetch and subsequent
// .bulkDocs() call). This method will use either the .allDocs() method or the .query()
// method for fetching, depending on the invocation signature:
// --
// .updateMany( options, operator ) ==> Uses .allDocs()
// .updateMany( viewName, options, operator ) ==> Uses .query()
// --
// In each case, the "options" object is passed to the underlying fetch method. Each
// document in the resultant collection is then passed to the given operator function -
// operator( doc ) - to perform the update transformation.
PouchDB.plugin({
updateMany: function( /* [ viewName, ] options, operator */ ) {
var pouch = this;
// CAUTION: Top-level errors MAY NOT be caught in a Promise.
// .allDocs() invocation signature: ( options, operator ).
if ( arguments.length === 2 ) {
var options = arguments[ 0 ];
var operator = arguments[ 1 ];
var promise = pouch.allDocs( ensureIncludeDocs( options ) );
// .query() invocation signature: ( viewName, options, operator ).
} else {
var viewName = arguments[ 0 ];
var options = arguments[ 1 ];
var operator = arguments[ 2 ];
var promise = pouch.query( viewName, ensureIncludeDocs( options ) );
}
// Even though the results are potentially coming back from two different search
// methods - .allDocs() or .query() - the result structure from both methods is
// the same. As such, we can count on the following keys to exist in the results:
// --
// * offset
// * total_rows
// * rows : [{ doc }]
// --
promise = promise.then(
function( results ) {
var docsToUpdate = results.rows.map(
function iterator( row, index, rows ) {
return( operator( row.doc, index, rows ) || row.doc );
}
);
return( pouch.bulkDocs( docsToUpdate ) );
}
);
return( promise );
// -- Utility methods for my PouchDB plugin. Thar be hoistin'! -- //
// I ensure that the given search options has the "include_docs" set to true.
// Since we are working on updating documents, it is important that we actually
// fetch the docs being updated. Returns options.
function ensureIncludeDocs( options ) {
options.include_docs = true;
return( options );
}
}
});
CAUTION: Top-level errors will not be properly translated into rejected Promises in my version.
To see this in action, I updated yesterdays Fruit demo. Only today, we're performing two updates - one that uses the .allDocs() method internally and one that uses the .query() method internally. To do so, I have to create a secondary index / View that indexes the fruit documents by type. I then output the before and after collections for comparison:
<!doctype html>
<html>
<head>
<meta charset="utf-8" />
<title>
Creating A PouchDB Plugin For Bulk Document Updates
</title>
</head>
<body>
<h1>
Creating A PouchDB Plugin For Bulk Document Updates
</h1>
<p>
<em>Look at console — things being logged, yo!</em>
</p>
<script type="text/javascript" src="../../vendor/pouchdb/6.0.7/pouchdb-6.0.7.min.js"></script>
<script type="text/javascript">
// I provide an API for updating many documents (encapsulating the fetch and
// subsequent .bulkDocs() call). This method will use either the .allDocs() method
// or the .query() method for fetching, depending on the invocation signature:
// --
// .updateMany( options, operator ) ==> Uses .allDocs()
// .updateMany( viewName, options, operator ) ==> Uses .query()
// --
// In each case, the "options" object is passed to the underlying fetch method.
// Each document in the resultant collection is then passed to the given operator
// function - operator( doc ) - to perform the update transformation.
PouchDB.plugin({
updateMany: function( /* [ viewName, ] options, operator */ ) {
var pouch = this;
// CAUTION: Top-level errors MAY NOT be caught in a Promise.
// .allDocs() invocation signature: ( options, operator ).
if ( arguments.length === 2 ) {
var options = arguments[ 0 ];
var operator = arguments[ 1 ];
var promise = pouch.allDocs( ensureIncludeDocs( options ) );
// .query() invocation signature: ( viewName, options, operator ).
} else {
var viewName = arguments[ 0 ];
var options = arguments[ 1 ];
var operator = arguments[ 2 ];
var promise = pouch.query( viewName, ensureIncludeDocs( options ) );
}
// Even though the results are potentially coming back from two different
// search methods - .allDocs() or .query() - the result structure from
// both methods is the same. As such, we can count on the following keys
// to exist in the results:
// --
// * offset
// * total_rows
// * rows : [{ doc }]
// --
promise = promise.then(
function( results ) {
var docsToUpdate = results.rows.map(
function iterator( row, index, rows ) {
return( operator( row.doc, index, rows ) || row.doc );
}
);
return( pouch.bulkDocs( docsToUpdate ) );
}
);
return( promise );
// -- Utility methods for my PouchDB plugin. Thar be hoistin'! -- //
// I ensure that the given search options has the "include_docs" set to
// true. Since we are working on updating documents, it is important
// that we actually fetch the docs being updated. Returns options.
function ensureIncludeDocs( options ) {
options.include_docs = true;
return( options );
}
}
});
// --------------------------------------------------------------------------- //
// --------------------------------------------------------------------------- //
getPristineDatabase( window, "db" ).then(
function() {
// To experiment with the bulk update PLUGIN, we need to have documents
// on which to experiment. Let's create some food products with names
// and prices that we'll update with the bulk update plugin.
var promise = db
.bulkDocs([
{
_id: "apple:fuji",
name: "Fuji",
price: 1.05
},
{
_id: "apple:applecrisp",
name: "Apple Crisp",
price: 1.33
},
{
_id: "pear:bosc",
name: "Bosc",
price: 1.95
},
{
_id: "apple:goldendelicious",
name: "Golden Delicious",
price: 1.27
},
{
_id: "pear:bartlett",
name: "Bartlett",
price: 1.02
}
])
.then(
function() {
// Since the bulk update plugin can also fetch documents
// using a VIEW, let's create a view that indexes the
// fruit products by type (ie, key prefix).
var promise = db.put({
_id: "_design/by-type",
views: {
"by-type": {
map: function( doc ) {
// Emit the key prefix, example "apple".
emit( doc._id.split( ":", 1 )[ 0 ] );
}.toString()
}
}
});
// NOTE: View are not populated proactively - they are
// populated at query time. As such, one might ordinarily
// kick off a query to force indexing. However, to keep this
// demo simple, we won't care about pre-heating the index.
return( promise );
}
)
;
return( promise );
}
)
.then(
function() {
// Now that we've inserted the documents, let's fetch all the Apples
// and output them so we can see the pre-update values.
var promise = db
.allDocs({
startkey: "apple:",
endkey: "apple:\uffff",
include_docs: true
})
.then( renderResultsToConsole )
;
return( promise );
}
)
.then(
function() {
// Now, let's update the Apples, applying a 10% price increase. Since we
// invoking the .updateMany() plugin method with TWO ARGUMENTS, it will
// use the .allDocs() bulk fetch method under the hood.
var promise = db.updateMany(
{
startkey: "apple:",
endkey: "apple:\uffff"
},
function operator( doc ) {
// Apply the 10% price increase.
doc.price = +( doc.price * 1.1 ).toFixed( 2 )
}
);
return( promise );
}
)
.then(
function() {
// Now, let's update the Apples, upper-casing the name. Since we are
// invoking the .updateMany() plugin method with THREE ARGUMENTS, it will
// use the .query() bulk fetch method under the hood. In this case, the
// first argument is the View / secondary-index name.
var promise = db.updateMany(
"by-type",
{
key: "apple"
},
function operator( doc ) {
doc.name = doc.name.toUpperCase();
}
);
return( promise );
}
)
.then(
function() {
// Now that we've updated the Apples twice (once using .allDocs() and
// once using .query() under the hood), let's re-fetch the Apples to see
// how the values have changed.
var promise = db
.allDocs({
startkey: "apple:",
endkey: "apple:\uffff",
include_docs: true
})
.then( renderResultsToConsole )
;
return( promise );
}
)
.catch(
function( error ) {
console.warn( "An error occurred:" );
console.error( error );
}
);
// --------------------------------------------------------------------------- //
// --------------------------------------------------------------------------- //
// I ensure that a new database is created and stored in the given scope.
function getPristineDatabase( scope, handle ) {
var dbName = "javascript-demos-pouchdb-playground";
var promise = new PouchDB( dbName )
.destroy()
.then(
function() {
// Store new, pristine database in to the given scope.
return( scope[ handle ] = new PouchDB( dbName ) );
}
)
;
return( promise );
}
// I use the console.table() method to log the documents in the given results
// collection to the console.
function renderResultsToConsole( results ) {
var docs = results.rows.map(
function( row ) {
return( row.doc )
}
);
console.table( docs );
}
</script>
</body>
</html>
As you can see, in the first .updateMany() call, we're using two arguments which will invoke the .allDocs() method internally. Then, in the second .updateMany() call, we're using three arguments which will invoke the .query() method internally, using the first argument as the View name. And, when we run the above code, we get the following output:
As you can see, both invocations of the .updateMany() PouchDB plugin modified the Apple-related documents.
PouchDB plugins are just like jQuery plugins in that they are nothing more than functions injected into the PouchDB prototype. And, just like jQuery plugins, the internal mechanics of the plugin have to be aware of the greater context. In the case of PouchDB, which exposes a Promise-based workflow, each plugin that provides an asynchronous action should perform its work by returning a Promise. This way, the plugin can be used seamlessly alongside the other core PouchDB API methods.
Want to use code from this post? Check out the license.
Reader Comments
@All,
After reading through Nolan Lawson's "upsert" PouchDB plugin, I wanted to try and incorporate retry logic into my "updateMany" plugin:
www.bennadel.com/blog/3200-retrying-bulk-updates-in-pouchdb-using-a-recursive-promise-chain.htm
... took me 3 days to figure out :D But, I think I came up with an interesting solution that included, among other things, recursive Promises.