Open Data Format

What a Data Format Describes

A data format defines how information is represented and structured.

A very simple format could, for example, specify:

name;email;customer_number

and then contain data:

Anna Example;anna@example.org;4711

For another program to process this data reliably, however, it must know more than the positions of individual delimiters.

For example:

Which character encoding is used?
Is the first line always a header?
May fields be empty?
How are delimiters within a value represented?
What does customer_number mean?
Is the number unique?
Which version of the format applies?

A data format therefore includes more than the visible representation of the data.
Rules and meaning are also part of it.

Open Means Documented and Independently Implementable

With an open data format, the information necessary for an implementation is sufficiently publicly documented.
As a result, more than one software implementation can in principle implement the format.

Simplified:

public specification
        ↓
Implementation A
Implementation B
Implementation C
        ↓
common data format

The applications do not have to come from the same vendor.
They also do not have to use the same source code.
What matters is that they can refer to a sufficiently clearly described structure and meaning of the data.

Open Does Not Mean Text-Based

A common shortcut is:

Can I read it with a text editor
        =
open format

That is not true.
A proprietary format can consist entirely of readable text and still be insufficiently documented.
Conversely, an open format can contain binary components.
The technical encoding alone therefore does not determine whether a format is open.
What matters is whether its structure and meaning can be understood and implemented independently of a specific application.

JSON Is Not Automatically an Open Data Format

JSON has a publicly documented syntax.

This tells a program, for example, how this document can be read structurally:

{
  "customer": 4711,
  "status": 3
}

But this still does not tell us:

What does customer mean?
Is 4711 an internal ID or customer number?
What does status = 3 mean?
Which other values are permitted?
Which fields are mandatory?
How does the schema change between versions?

Using an open serialization format therefore does not automatically make a specific data model open or interoperable.
JSON can provide the outer syntax.
For the actual data format, the meaning of the structures used within it must additionally be described.

The Same Applies to XML

XML defines rules for structuring documents.

From:

<customer>
  <status>3</status>
</customer>

it does not follow what status means in domain terms.
The openness of a syntax and the openness of a data model built on it are therefore two different layers.
A documented schema can help.
For genuine exchangeability, however, the semantics must also be sufficiently clear.

CSV Is a Good Warning Sign

At first glance, CSV appears to be the simplest conceivable open format:

4711;Anna;active

But even here questions arise:

Comma or semicolon?
UTF-8 or another encoding?
Is there a header?
How are line breaks in fields handled?
What does column 3 mean?
Which values are permitted there?

The readability of the file does not answer these questions.
A simple format can be excellently suited to data exchange.
Its specific conventions must, however, be known.

Open Format and Open Specification

A specification describes the format.

It can define, for example:

• structure
• elements
• attributes
• value ranges
• data types
• relationships
• encoding
• mandatory fields
• extensions
• versioning behavior
• semantics

The more complete and unambiguous this description is, the better independent implementations can process the same format.
A published specification alone, however, does not guarantee a good specification.
If important properties remain unclear, different programs can produce different results despite using the same documentation.

OpenDocument as an Example

The OpenDocument Format, ODF for short, is a standardized document format for texts, spreadsheets, presentations, and graphical documents, for example.
The specification describes, among other things, the structure, package format, and schema of the documents.
It is not tied to a single office application.
As a result, different programs can implement ODF documents.
This does not mean, however, that every program supports all capabilities of the standard identically.
A complex document can therefore be displayed differently in different applications despite the open standard.

An Open Format Does Not Guarantee Interoperability

Suppose a standard permits the properties:

A
B
C
D
E

Program 1 supports:

A B C D E

Program 2 supports:

A B C

Both can implement the same open format.
When exchanging a document with property E, information can nevertheless be lost.

Therefore:

open format
        ≠
guaranteed complete interoperability

An open format creates an important technical prerequisite for Interoperability.
Whether specific implementations are actually compatible must be examined separately.

Profiles and Subsets

Complex standards often have many optional features.
For a specific use case, a defined subset can therefore be useful.

For example:

Our exchange format uses:

Version 2
Fields A through G
UTF-8
no proprietary extensions
date format YYYY-MM-DD

Such a restriction can make practical data exchange more reliable than the general statement:

We support Standard X.

The precise description of the profile actually used is therefore also part of Interoperability.

Extensions Can Restrict Openness in Practice

An open format can have extension mechanisms.

An application could use them to store its own functions:

standard data
+
vendor extension X

Another application may understand only:

standard data

If the format permits such extensions, the document can still conform to the open format.
Part of its meaning can nevertheless depend on a single implementation.
As a result, an open format can in practice once again contain a vendor dependency.

The decisive question is therefore not only:

“Which format do we use?”

But also:

“Which parts of this format do we actually use?”

Data Export and Open Data Format

A system can offer an export:

Export successful.

This says little about how useful the exported data is outside the original system.

A usable export should make it possible to examine, for example:

• which data is included
• which data is missing
• how relationships are represented
• which format is used
• which version applies
• how fields are interpreted
• how attachments are handled
• which metadata is included

The technical existence of an export is therefore not automatically a realistic exit strategy.

Open Formats and Digital Sovereignty

Open data formats can support Digital Sovereignty.
If data can be understood and processed independently of a single application, additional options for action emerge.

For example, one can:

write custom tools
use alternative applications
check data
migrate data
archive data long-term
build automated processing

This reduces certain forms of Vendor Lock-in.
It does not, however, automatically eliminate all other dependencies.

An Open Format Can Still Be Difficult to Use

Openness and simplicity are different properties.
A format can be fully documented and still be extremely complex.

An independent implementation may, for example, require:

several thousand pages of specification
numerous optional features
complex dependency rules
multiple supplementary standards

Formally, independent implementation is then possible.
In practice, this possibility may be limited for small organizations.
Here too, therefore, theoretical openness alone is not what matters.
What matters is the actual ability to act that results from it.

Open Does Not Mean Immutable

Open formats also evolve.

There can be, for example:

Format 1.0
Format 1.1
Format 2.0

This raises new questions:

Can Version 2 read files from Version 1?
Can Version 1 process data from Version 2?
Which features were changed?
How are unknown elements handled?

Versioning is therefore part of a durable data format.
A specification should make sufficiently clear how different versions relate to one another.

Long-Term Archiving

For data that must be retained over the long term, the documentation of a format is particularly important.
The original application may no longer exist after decades.
Hardware and operating systems may have disappeared.
If the format is sufficiently documented, there is at least the possibility of developing new software to process the data.
This makes open formats interesting for archiving strategies.
It does not, however, automatically guarantee that all required information will actually be preserved permanently.

Open Source and Open Data Format

Open Source and open data formats can complement each other, but they describe different things.

Open Source
    ↓
How may software be used, examined,
modified, and redistributed?

Open Data Format
    ↓
How is data structured and described
so that different implementations
can process it?

A proprietary application can use an open data format.
Conversely, an Open Source application can use its own poorly documented data format.
One does not automatically follow from the other.

Self-Hosting and Open Data Formats

Open formats are also relevant for self-hosted systems.
Operating a system oneself can provide direct access to the stored data.
That does not automatically mean, however, that the data can easily be transferred to another system.
An application can, for example, store data in a complex internal database structure.
As long as only the application itself reliably understands this structure, migration remains difficult.
Self-Hosting provides access.
An open and documented data format can additionally provide understandability and exchangeability.

Why Open Data Formats Have Practical Effects

• Data becomes more independent of individual applications: Other implementations can in principle process it.
• Migrations become easier: Structure and meaning do not have to be reconstructed entirely through reverse engineering.
• Custom tools become possible: Data can be analyzed and processed independently.
• Archiving is supported: Later software can continue to refer to a documented specification.
• Vendor Lock-in can be reduced: Data is less tightly bound to a single implementation.
• Interoperability is supported: Different systems receive a common technical foundation.
• Verifiability increases: Structure and content can be examined against documented rules.
• Openness remains verifiable: Extensions, profiles, and subsets actually used must still be considered.

Typical Example

An organization wants to replace a system it has been using.

System A provides an export as:

backup.dat

The file contains all data.
The format, however, is undocumented.
Only System A can read it.
Technically, a complete export exists.
In practice, the organization therefore has only limited options for action.
System B exports the same information in a documented format.

The description includes:

field definitions
data types
relationships
character encoding
version number
schema

Several programs can process the data.
In addition, the organization has a small custom tool that checks the export for completeness.
This does not guarantee that every future migration will work without problems.
But the data is considerably less tightly bound to a single implementation.

Distinction

• Open Data Format describes a sufficiently publicly documented data format that enables independent implementations.
• Open Standard refers to a technical standard whose specification is publicly accessible and whose development and use take place under defined open conditions. Which additional requirements apply to an open standard depends on the definition used. An open data format can be specified by an open standard.
• Serialization Format describes the technical representation of data, for example JSON or XML. An open serialization syntax does not automatically make the specific data model built on it open.
• Interoperability describes the ability of different systems to exchange information and use it meaningfully. Open data formats can support this, but do not guarantee it. -Open Source concerns software and its licensing or the usage rights associated with its source code. It is not identical to an open data format. -Digital Sovereignty describes the ability to retain real options for decision and action in the presence of digital dependencies. Open data formats can expand these options.

Further Reading

Related Concepts

→ Digital Sovereignty
→ Interface
→ System Integration
→ Legacy System
→ Legacy Modernization
→ Self-Hosting
→ Interoperability