Whitepaper: International IDEA's Unified Database

The customer

The International Institute for Democracy and Electoral Assistance (International IDEA) is an intergovernmental organization that supports democracy worldwide. Their mission is to support sustainable democratic change by providing comparative knowledge, assisting in democratic reform, and influencing policies and politics. Read more

Motivation

The situation with IDEA databases was problematic for both users and staff. For visitors to IDEA’s site, the databases each appeared as completely distinct entities with different navigation and appearance (maps, tables, menus etc.) with only minimum connection between the databases. Furthermore, there was no clear indication of how up-to-date each database is, and no clear workflow on how visitors can contribute to improve the quality of each database.

Better staff interface

For staff, each database had a different interface to browse and edit data. The editing interfaces had little or no help or documentation, and no error checking code. This had led to interns entering incorrect data or data in the wrong format (i.e. typing decimals with the comma) which in turn led to errors in the publicly visible database interface. Now there is a single location for IDEA data. Staff can browse any field whether it previously resided in the VT, ESD, EDR or any other IDEA database. This makes it a valuable tool for in-house research and comparison.

Systematic data definitions

By completely reviewing the databases at IDEA and building a central database structure to house all the results, we could build a single consistent interface to both web visitors and staff. During the creation of this platform, all data fields were specified in more detail (i.e. given a title, an example, notes to staff about the correct format, and formatting instructions for how each field should be displayed on the web). Most of this meta-data was ascertained from the existing formatting of the different databases, but teams were involved to ensure accuracy, and used the opportunity to update some of the glossary of terms.

Consistent data lookups

Part of the work was the creation of a set of functions for querying the database in different ways. You can think of these as query “templates”. For example, there may be a template for querying and displaying the average value for a certain field (say Turnout/VAP) for a set of Countries (which is a parameter to the query). This same template would be used to display data for all the different data themes. The implication is that looking up data from the VT, ESD or EDR webpage would function the same and be displayed in the same way.

Consistent naming

Previously, most of IDEA’s databases already shared a country database table, which included the standard 2-letter country codes and the “IDEA” names of countries. So, changing the name of a country (from “Taiwan, province of China” to “Taiwan”) could already be done once and affect the whole site. However, it was preferable to have both an official short name (e.g. “Congo DRC”) and an official long name (e.g. “Congo, the Democratic Republic of”) that we can use as the space allows (e.g. in a map we decided to use short names because of lack of space)

Unfortunately, there was no simple solution for regional breakdowns. Some databases did not provide a breakdown by region at all, and those that did differed to the rest of the IDEA website. Creating standard regions in the unified structure, allowed us to keep regional names and breakdowns consistent.

Cross-thematic data lookup

While the unified database could be used to bring consistency to the current database web pages, it could also be used to provide whole new options for web visitors to explore IDEA’s data. Since all the data is in a shared format and location, it became possible to query combinations of data from what were previously separate databases. For example, a query can list the electoral system, the voting age population, and whether a country has remote voting all in a single table on screen.

Shared improvements

Another benefit of having a single library for all the databases, is that when we add a new feature, all the thematic interfaces can immediately make use of the new feature with a bare minimum of changes. For example, if we create an new RSS feature to track new related database entries, it would just be a matter of adding a new link to each thematic page for that feature to come into effect.