C++ API

class FDB

A handle to a general FDB.

FDB and its methods are threadsafe. However the caller needs to be aware that flush acts on all archive calls, including arcived messages from other threads. I.e. a call to flush will persist all archived messages regardles from which thread the message has been archived. In case the caller wants a finer control it is advised to instanciate one FDB object per thread to ensure only messages are flushed that have been archived on the same FDB object.

Subclassed by fdb_handle_t

Public Functions

FDB(const Config &config = Config().expandConfig())

~FDB()

FDB(const FDB&) = delete

FDB &operator=(const FDB&) = delete

FDB(FDB&&)

FDB &operator=(FDB&&)

void archive(eckit::message::Message msg)

Archive a eckit::message::Message.

Due to the message being self describing no key needs to be supplied. Any callback set with registerArchiveCallback will be invoked.

Parameters:

handle – eckit::message::Message to data to archive

void archive(eckit::DataHandle &handle)

Archives a stream of one or more messages.

Reads messages from the eckit::DatAaHandle and calls archive() on the corresponding messages. Any callback set with registerArchiveCallback will be invoked on each message.

Parameters:

handle – eckit::DataHandle reference data to archive

void archive(const void *data, size_t length)

Archive binary data to a FDB.

Internally creates a DataHandle and calls archive(). Any callback set with registerArchiveCallback will be invoked on each message.

Parameters:

• data – Pointer to the binary data to archive

• length – Size of the data to archive with the given

void archive(const metkit::mars::MarsRequest &request, eckit::DataHandle &handle)

Archives data from Datahandle and ensures all keys exactly match the provided MarsRequest.

Any callback set with registerArchiveCallback will be invoked on each message.

Parameters:

• request – a mars request

• handle – a data handle pointing to the data

Throws:

• eckit::UserError – if there are more keys in the MarsRequest then in the messages.

• eckit::UserError – if message key not present in MarsRequest.

void archive(const Key &key, const void *data, size_t length)

Archive a binary blob into FDB.

Any callback set with registerArchiveCallback will be invoked.

Note

No constistency checks are applied. The caller needs to ensure the provided key matches metadata present in data.

Parameters:

• key – Key used for indexing and archiving the data

• data – Pointer to the binary blob to archive

• length – Size in bytes of the binary blob to archive

void reindex(const Key &key, const FieldLocation &location)

Generate an new index entry for an existing field location.

Can be used to reindex existing data into a new catalogue (see fdb-reindex tool).

Parameters:

• key – Key used to index the data.

• location – Location of existing data in an FDB store.

void flush()

Flush all buffers and closes all data handles into a consistent DB state

Note

always safe to call

eckit::DataHandle *read(const eckit::URI &uri)

Read binary data from an URI.

Parameters:

uri – eckit uri to the data source

Returns:

DataHandle for reading the requested data from

eckit::DataHandle *read(const std::vector<eckit::URI> &uris, bool inStorageOrder = false)

Read binary data from an list of URI.

Parameters:

• vector – of uris eckit uris to the data source

• inStorageOrder – if set data will be returned in the order it is stored. If unset data will be returned in the order it was requested.

Returns:

DataHandle for reading the requested data

eckit::DataHandle *read(ListIterator &it, bool inStorageOrder = false)

Read binary from a ListIterator.

Parameters:

• uris – a list iterator which resembles a set of fields which should be read

• inStorageOrder – if set data will be returned in the order it is stored. If unset data will be returned in the order it was requested.

Returns:

DataHandle for reading the requested data from

eckit::DataHandle *retrieve(const metkit::mars::MarsRequest &request)

Retrieve data which is specified by a MARS request.

Parameters:

request – MarsRequest which describes the data which should be retrieved

Returns:

DataHandle for reading the requested data from

ListIterator inspect(const metkit::mars::MarsRequest &request)

ListIterator list(const FDBToolRequest &request, bool deduplicate = false, int level = 3)

List data present at the archive and which can be retrieved.

Parameters:

• request – FDBToolRequest stating which data should be queried

• deduplicate – bool whether the returned iterator should ignore duplicates

• length – Size of the data to archive with the given key

Returns:

ListIterator for iterating over the set of found items

DumpIterator dump(const FDBToolRequest &request, bool simple = false)

Dump the structural content of the FDB

In particular, in the TOC formulation, enumerate the different entries in the Table of Contents (including INIT and CLEAR entries). The dump will include information identifying the data files that are referenced, and the “Axes” which describe the maximum possible extent of the data that is contained in the database.

Parameters:

• request –

• simple –

Returns:

DumpIterator for iterating over the set of found items

StatusIterator status(const FDBToolRequest &request)

WipeIterator wipe(const FDBToolRequest &request, bool doit = false, bool porcelain = false, bool unsafeWipeAll = false)

Wipe data from the database.

Deletes FDB databases and the data therein contained. Uses the passed request to identify the database to delete. This is equivalent to a UNIX rm command. This tool deletes either whole databases, or whole indexes within databases

Parameters:

• request – FDBToolRequest stating which data should be queried

• doit – flag for committing to the wipe (default is dry-run)

• porcelain – flag print only a list of files to be deleted / that are deleted

• unsafeWipeAll – flag for omitting all security checks and force a wipe

Returns:

WipeIterator for iterating over the set of wiped items

MoveIterator move(const FDBToolRequest &request, const eckit::URI &dest)

Move content of one FDB database.

This locks the source database, make it possible to create a second database in another root, duplicates all data. Source data are not automatically removed.

Parameters:

• request – a fdb tool request for the data which should be move

• dest – destination uri to which the data should be moved

Returns:

MoveIterator for iterating over the set of found items

PurgeIterator purge(const FDBToolRequest &request, bool doit = false, bool porcelain = false)

Remove duplicate data from the database.

Purge duplicate entries from the database and remove the associated data if the data is owned and not adopted. Data in the FDB5 is immutable. It is masked, but not removed, when overwritten with new data using the same key. Masked data can no longer be accessed. Indexes and data files that only contains masked data may be removed. If an index refers to data that is not owned by the FDB (in particular data which has been adopted from an existing FDB5), this data will not be removed.

Parameters:

• request – a fdb tool request for the data which should be purged

• doit – bool if true the purge is triggered, otherwise a dry-run is executed

• porcelain – bool for printing only those files which are deleted

Returns:

PurgeIterator for iterating over the set of found items

StatsIterator stats(const FDBToolRequest &request)

Prints information about FDB databases, aggregating the information over all the databases visited into a final summary.

Parameters:

request – FDB tool request for which the stats should be shown

Returns:

StatsIterator for iterating over the set of found items

ControlIterator control(const FDBToolRequest &request, ControlAction action, ControlIdentifiers identifiers)

Parameters:

• request – FDB tool request

• action – control action

• identifiers – identifiers

Returns:

ControlIterator for iterating over the set of found items

IndexAxis axes(const FDBToolRequest &request, int level = 3)

Parameters:

• request – FDB tool request

• level – maximum level the axis visitor should respect

Returns:

IndexAxis

AxesIterator axesIterator(const FDBToolRequest &request, int level = 3)

Parameters:

• request – FDB tool request

• level – maximum level the axis visitor should respect

Returns:

AxisIterator

bool enabled(const ControlIdentifier &controlIdentifier) const

Check whether a specific control identifier is enabled

Parameters:

controlIdentifier – a given control identifier

Returns:

bool true or false, depending on the internal status of the FDB

bool dirty() const

Return whether a flush of the FDB is needed

Returns:

true if an archive has happened and a flush is needed

void registerArchiveCallback(ArchiveCallback callback)

Register an archive callback.

Parameters:

callback – an archive callback which should be triggered during archive

void registerFlushCallback(FlushCallback callback)

Register a flush callback.

Parameters:

callback – an flush callback which should be triggered during flushing

const std::string id() const

Returns the coniguration serialised into a string.

Returns:

config as string.

FDBStats stats() const

const std::string &name() const

Type of FDB, local or remote

Returns:

name of FDB type used, ‘local’ or ‘remote’

const Config &config() const

Read-only access to this FDBs configuration.

Returns:

configuration