A Cloudant “Tip”: Separate Design Documents for Each View

This is not an official “best practice” or anything, just a habit I fell into while working with Cloudant and large data sets.

To very quickly review, there are no “queries” as such when working with a Cloudant database.  Instead one can create secondary or search indexes or “views” against a database.  Both are functions written in Javascript that are executed against every document in a database and the results stored.  This makes getting results from large data sets blazing fast, as all possible results are pre-calculated, but creating the index can be very slow if there is lots of data or the index function is complex.

Each view is part of a design document, and each design document can contain any number of views. As we can see from the documentation:

View indexes are rebuilt when a design document is updated. An update to any one of the views causes all the views in the document to be rebuilt.

That second sentence is the important bit. xbrl.mybluemix.net as of the time of this post has just under 70 million documents in its main database.  Building a simple secondary index against this data usually takes about 72 hours.  If you have an application depending on such a view it will either show very incomplete data, or no data at all, during this time.

There are other articles out there that say that each design document conceptually represents an “application” and each view a part of that application.  That sounds nice, but if you’ve got hundreds of gigabytes or terabytes of data it means that if you want to change one part of your application, or even just add to it, you’re going to bring the whole thing down for days.  Even if it’s just dev and not prod, that’s really annoying.

So whether or not it’s an official “best practice” or not, I’ve fallen into the habit of having as few views per design document as possible, usually one or two.  It may not be as pretty, but if I want to change one part of an application without taking down all the other parts it’s really nice.

Advertisements
A Cloudant “Tip”: Separate Design Documents for Each View

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s