Administration

In addition to the project code itself, WOUDC Data Registry depends on several databases which must be set up before the project can run.

The core backend, referred to as the Data Registry, is an archive of file metadata and links using a relational database, currently PostgreSQL. The Data Registry is the trusted and long-term source of file metadata, history and versioning, and file location.

The secondary backend, known as the Search Index, is derived from the Data Registry and is currently kept in ElasticSearch. The Search Index is denormalized and will contain additional metrics data for quick access.

The whole file contents, as opposed to metadata only, are stored in the WOUDC Web Accessible Folder (WAF) and are linked via URLs in both the Data Registry and Search Index.

The Search Index is intended for use by web applications and for searchability, while the Data Registry is intended for internal operations and to prop and compare against the Search Index in case of error.

Creating Backends

make ENV=/path/to/environment/config.env createdb

Create the Data Registry database instance

woudc-data-registry admin registry setup

Create Data Registry tables and schema

woudc-data-registry admin registry setup

Create Search Index mappings

Deleting Backends

woudc-data-registry admin registry teardown

Delete all Data Registry tables and schema

woudc-data-registry admin search setup

Delete all Search Index indexes and mappings

make ENV=/path/to/environment/config.env dropdb

Delete the Data Registry database instance

Populating Backends

When necessary, the entire history of data records can be recreated by processing all files in the WAF (backfilling). Not all metadata can be extracted from these files, and so there are alternate methods to recreate metadata from scratch.

If the Data Registry is empty, its metadata is recovered from a series of files. The WOUDC Data Registry code comes with two files in the data/init directory:

• ships.csv

• countries.json

The .csv and .json files contain core metadata for the Data Registry. Contact WOUDC to provide the remaining required metadata files.

After ensuring all metadata files are together in one directory, run:

woudc-data-registry admin init -d <initialization> <flags>

Searches the directory path <initialization> for .csv and .json files of metadata, and loads them into the Data Registry tables. If the –init-search-index flag is provided, loads them to the Search Index as well.

If the Data Registry is filled but not the Search Engine, the latter can be populated by using the sync command (see Publishing).