jsormdb--an Embedded JavaScript Database

With the rise of Web 2.0 and Rich Internet Applications, the need for a solid JavaScript-based data management paradigm has become much more acute. Using jsormdb, you can develop your applications with the same data-driven paradigm you use on server-side applications.

The (almost) standardization and acceptance of JavaScript as a browser-side development language has enabled the rapid growth of dynamic Web applications. These applications often look and feel as snappy as native desktop applications. Add the excellent open-source client-side development frameworks, like jQuery and ExtJS, along with the communicative power of AJAX, and you have an incredibly powerful and largely open browser-side development platform.

It has gotten so powerful, in fact, that applications are living entirely within the browser, with the server acting as a simple file store. One excellent example is password managers, like Clipperz and jkPassword. These work entirely in the browser to provide the user interface and encryption. The only services provided by the server are serving the static HTML and JavaScript files, and persisting the already-browser-side-encrypted file.

As the client side grows, and self-contained applications like these password managers increase, the need for basic services that are normally available in a development environment increase. In a normal development environment, the ability to manage data reliably is one of the first and key issues tackled. Sometimes, it leads to full-blown client-server database systems, like Oracle or MySQL. In other instances, it leads to embedded databases—for example, Apache Derby, which are critical for managing data within a single instance of an application.

Unfortunately, our browser development platform has lacked any such data management system. As it turns out, the availability of such a system is even more critical in the browser environment than in a server environment. For complete browser applications, the entire data set and transaction semantics are local and cannot depend on a server. For more traditional server-driven applications, the need is even more acute. Whereas a server can rely on redundancy, high bandwidth and low latency to its data store, the browser has none of these attributes. It becomes critically important for the browser application developer to be able to perform the majority of the application's data activities locally, sending the result to the server with the lowest frequency and bandwidth possible.

Enter a new class of data stores, embedded JavaScript databases. This article introduces jsormdb, an advanced JavaScript database engine. jsormdb includes most of the features you, as a developer, would expect in an embedded database system, as well as many that make sense only within the context of a browser environment. I describe jsormdb and its basic usage, including creating a database, loading data, modifying and querying the data, and transactions. jsormdb also supports event signaling and, critical for a browser-side data store, persisting changes or the entire data set to the server, but those are the subject of another article.

Why a Database?

As anyone who has done even a moderate amount of development knows, data storage structures are, in many ways, the fundamental building block of any application. There are several good reasons for this:

  1. Performance: choice of data structure can have a significant impact on application performance. For example, using a linked list versus a serial array is slower to access elements in the middle of the list, but much faster at list reorder and element insertion/addition/removal.

  2. Usability: a standard data structure makes it much easier for someone else (or you) to understand what your application does and how.

  3. Separation of concerns: by separating data structure and storage from business logic, you are able to work with each separately.

A good database structure addresses those concerns, while providing the following features:

  1. Queries: you will want to query your data to discover which elements of the data—for example the first, 35th and 60th records—match certain arbitrary criteria. Of course, big RDBMS systems excel at this, and even have their own language for doing so, SQL. Of course you're not about to embed Oracle or MySQL type systems into a JavaScript environment, just to support local temporary queries.

  2. Indexing: if you go to an arbitrary array or table of data and ask it to return all the records where the third field is equal to 2, you (or your data store engine) will need to go to each and every record and check whether the third field is equal to 2. This is called a full table scan, and it is terribly inefficient. For fields that are checked frequently, you would much prefer a more efficient method, one that does not require checking every single record linearly. This is known as indexing.

  3. Transactions: in a simple world, events are single-stage and one-way. For example, login is a simple process: you enter your credentials and are either in or out. In the real world, however, events usually are multistage. For example, in order to transfer $100 from your checking account to your savings account, you need to deduct $100 from your checking account and add $100 to your savings account. Ideally, both steps will succeed. If both fail, it is not great, but you can live with it. However, if only one of the two succeeds, either you (if it is the deduction) or your bank (if it is the addition) will be very unhappy.

  4. Events: sometimes, you want the database to tell you whether something of significance has occurred. For example, you may want to know that a certain table has been updated or perhaps that an account balance has dropped below a certain threshold. Many databases support events, often known as triggers, that cause reactions.

Because I am discussing the browser environment, one additional feature is important: persistence. A browser environment is a transient one; the moment users close their windows, all of the data is lost. Because the browser application relies on the server to provide permanence, you want a way to handle both loading your local data store from, as well as persisting our local changes back to, that server seamlessly.

jsormdb provides a solution to all of these problems. When configured correctly, the presentation and business logic within your application can treat jsormdb as the entire data store, leaving jsormdb to handle persistence to/from the server. Without jsormdb, your application looks like Figure 1.

Figure 1. Web Application without a Database

With jsormdb, you can work with a much cleaner design (Figure 2).

Figure 2. Web Application with a Database