Self-Diagnostic APIs: Software Quality's Next Frontier

With embedded software adding intelligence to so many everyday objects, it seems remarkable that the tools used to create these programs aren't smarter when it comes to catching highly destructive bugs. In assigning blame for such errors, one culprit lies in the application programming interfaces (APIs) provided by software publishers. Developers have long chosen libraries of pre-built software for communication, data management, messaging and other purposes rather than creating this functionality from scratch.

But while middleware libraries offer benefits including convenience, portability and productivity, the manner in which they are constructed and used leads to bugs. This stems from the fact that software functions in APIs are nearly always data structure ignorant—they handle data without knowing its type. This severely limits the compiler's and middleware runtime's abilities to perform any validation, greatly increasing the likelihood of programming mistakes slipping through QA.

The potential for a new kind of API that helps to catch and fix such bugs is built into C and C++. API vendors can deliver a programming interface that is data-aware and self-diagnostic by taking advantage of the function argument type-checking ability of every ANSI C/C++ compiler. This article explores the idea by looking at the API of McObject's eXtremeDB, an in-memory database available on Linux—but the idea of a self-diagnostic API applies as well to other middleware categories.

The concept requires us to abandon the old idea that an API must be a static library of functions that is applied in every situation. Instead, the programming interface is generated for each project or implementation of the middleware and therefore is aware of that project's data types.

APIs for database software development kits (SDKs) fall into two categories: interfaces for SQL and navigational interfaces. With navigational interfaces, developers interact with the contents of a database one record at a time. SQL, in contrast, is a set-oriented programming interface. With this API, the user submits SQL statements to the database in order to select, filter, sort and join together rows (records) from many tables. The query results in a set of tuples (rows). The API then is used to fetch the results, either one row at a time or in batches. There is no industry or de-facto standard interface for navigational APIs, so database vendors offering navigational interfaces provide proprietary APIs.

Yet pre-defined database APIs, whether SQL or navigational, carry a significant downside: for an interface library to be able to manage data of any database definition, it must have a programming interface that ignores the type of all data. In other words, the database programming interface must treat the data as un-typed, or opaque.

To accomplish this, databases use void pointers to pass data between the database library and the application program. A void pointer is a C/C++ language variable that legally can point to any type of data. With no type, neither the C/C++ compiler nor the database runtime can perform any validation on them. This opens the possibility of passing a pointer to the wrong type of data, with consequences ranging from nonsense data in the database to a corrupted, unusable database to a crashed program.

Let's look at three API examples, one each from Berkeley DB, SQL/ODBC and eXtremeDB. First, let's define a simple database in SQL:

create table make (
   make_id   integer,
   make_name char(20)

   primary key make_id;
)

create table model (
   make_id    integer foreign key references make,
   model_name char(20)
)

Here's how the same database would be defined in eXtremeDB:

declare database cars;

class make
{
   unsigned<4> make_id;
   char<20>    make_name;

   hash <make_id> by_make_id[10000];
};

class model
{
   unsigned<4> make_id; // foreign key of class make
   char<20>    model_name;

   tree <make_id> by_make_id;
};

Berkeley DB does not have a data definition language. Instead, a host program stores name/value pairs, and it is up to the development team to express the organization and interrelationships of the data through source code comments and system documentation.

Next, we write code to populate these databases, first with SQL and ODBC:

insert_make( long make_id, char *make_name )
{
   unsigned char *sql = 
      "insert into make values(?,?)";

   // housekeeping omitted

   SQLBindParameter( handle, // statement handle
      1,      // parameter number
      SQL_PARAM_INPUT, // InputOutputType
      SQL_C_LONG, // ValueType
      SQL_INTEGER, // ParameterType
      0, // ColumnSize ignored for SQL_INTEGER
      0, // DecimalDigits ignored for SQL_INTEGER
      &make_id, // ParameterValuePtr
      sizeof(make_id), // BufferLength
      sizeof(make_id) // ignored
   );
   SQLBindParameter( handle, // statement handle
      2,      // parameter number
      SQL_PARAM_INPUT, // InputOutputType
      SQL_C_CHAR, // ValueType
      SQL_VARCHAR, // ParameterType
      strlen(make_name), // ColumnSize 
      0, // DecimalDigits ignored for SQL_VARCHAR
      make_name, // ParameterValuePtr
      strlen(make_id), // BufferLength
      strlen(make_id) 
   );
   SQLExecute( handle );
}