Pattern: Metadata Element

How can messages be enriched with additional information so that receivers can interpret the message content correctly, without having to hardcode assumptions about the data semantics?


The final version of this pattern is featured in our book Patterns for API Design: Simplifying Integration with Loosely Coupled Message Exchanges.

Pattern: Metadata Element

a.k.a. Data about/on Data, Semantic Annotation, Embedded API Microdescription, Metadata Representation

Context

The request and response message representations of an API operation have been defined, possibly using one or more of the basic structural patterns (Atomic Parameter, Atomic Parameter List, Parameter Tree and Parameter Forest). To be able to process these representations accurately and efficiently, message receivers require more information about their meaning and content than just name and type.

Problem

How can messages be enriched with additional information so that receivers can interpret the message content correctly, without having to hardcode assumptions about the data semantics?

Forces

The above design problem is discussed in terms of the following conflicting forces:

  • Interoperability
  • Coupling
  • Ease of use versus runtime efficiency (message size)

Pattern forces are explained in depth in the book.

Solution

Introduce one or more Metadata Elements to explain and enhance the other representation elements that appear in request and response messages. Populate the values of the Metadata Elements thoroughly and consistently; process them as to steer interoperable, efficient message consumption and processing.

Sketch

A solution sketch for this pattern from pre-book times is:

Interlude: Background Information on Metadata

Metadata and metadata modeling are mature and well established concepts in many fields in computer science, e.g., databases and programming languages (e.g., under terms such as runtime type information, reflection, and introspection); in the real world, libraries apply it extensively (to give just one example). For a general introduction to types of metadata and eligible standards, refer to:

Example

The following example shows examples of all three metadata types, provenance (regionCode), control (nextPageToken, prevPageToken), and aggregated metadata (totalResults, resultsPerPage), all organized as Atomic Parameters. The pageInfo forms a simple Parameter Tree that bundles two of these.

GET https://www.googleapis.com/youtube/v3/search?
    HSR&part=snippet&maxResults=2&pageToken=CAIQAA&key=<KEY>

{
 "kind": "youtube#searchListResponse",
 "etag": etag,
 "nextPageToken": "CBQQAA",
 "prevPageToken": "CAoQAQ",
 "regionCode": "CH",
 "pageInfo": {
    "totalResults": 1662,
    "resultsPerPage": 10
  },
 "items": [
     search Resource
  ]
}

The HTTP concept of etag, defined in RFC 7232, qualified as a usage of both control and provenance metadata. It is explained and elaborated upon in the Conditional Request pattern.

Are you missing implementation hints? Our papers publications provide them (for selected patterns).

Consequences

The resolution of pattern forces and other consequences are discussed in our book.

Known Uses

A Swiss software vendor specializing on the insurance industry introduces Aggregated Metadata such as query counts and sums in its internal REST API Design Guidelines. Sorting and filtering of collection records is supported via operators that travel as HTTP parameters that contain Control Metadata Elements.

Many social networks apply this pattern in their response messages, e.g., to amend search results. For instance, the Facebook Graph API uses a dedicated metadata parameter in its response messages. The Twitter REST API and the YouTube Data API are two more examples of public Web APIs that use metadata.

Another known use can be found in the Force.com API. It uses a Salesforce Object Query Language (SOQL) that defines the query parameters; in the responses, the individual result records are annotated with control metadata (“done” boolean) and aggregated metadata (“totalSize” integer).  

Some of the transport-level protocol headers, for instance those in HTTP or TCP/IP, also qualify as Metadata Elements (e.g., size information, usage counters such as Rate Limit, access tokens, message identifiers).

A Swiss e-government standard called GBDBS (“Grundbuchdatenbezugsschnittstelle”) returns metadata to the response data, e.g., the date of when the data was last updated. GBDBS is used by Terravis Lübke and Lessen (2016), for instance, to access Swiss land registries.

The APIs.json website provides “a simple format for publishing API meta data on the Web” in Version 0.15 of its specification. JSON API has the notion of “meta objects”.

More Information

Related Patterns

Data Element and Metadata Element are two of four specializations of the abstract concept of a representation element; all Metadata Elements are Data Elements but not vice versa. Id Elements may qualify to be metadata; other Id Elements may be accompanied by Metadata Elements (same for Link Element). Metadata often comes in the syntactic form of one or more Atomic Parameters, but several related instances of the pattern can also be transported in the form of Parameter Trees.

Like all representation elements, Metadata Elements should be defined the API contract and explained in the API Description.

The Pagination pattern relies on metadata to inform the client about the current, previous, and next result pages and the total amount of pages/results etc. HATEOAS Link Elements (typed link relations) contain metadata as well.

A context object which interceptors can add their information to is presented in several pattern languages, including (Voelter, Kircher, and Zdun (2004)). Our Context Representation pattern suggests to define an API-wide, technology-independent standard location and structure for metadata in general and control metadata in particular.

The Format Indicator and Message Expiration information, both introduced in Enterprise Integration Patterns (EIP) by Hohpe and Woolf (2003), as well as control/provenance information Message Id and Message Date in messaging systems and APIs such as Java JMS use metadata. Other EIPs, for instance Correlation Identifier and Routing Slip, can be seen as specializations of this pattern. A Correlation Identifier is control metadata primarily, but also has the properties of provenance metadata (because it identifies a previous request message). The same holds for Return Address. Message Filters, Message Selectors and Aggregators often operate on (control and provenance) metadata.

Other Sources

Metadata is covered in the information management literature. Two examples are Introduction to Metadata, 3rd edition (Baca, Murtha, ed., Getty Publications 2016) and A Gentle Introduction to Metadata by Jeff Good.

The Zalando RESTful API and Event Scheme Guidelines point out the importance of OpenAPI metadata.

In the context of distributed hypermedia systems and Web APIs, metadata is mentioned and characterized in the formal definition of REST as an architectural style; a blog post by Steve Klabnik covers metadata in resource representations.

See “Interlude: Background Information on Metadata” above for additional pointers.

References

Evans, Eric. 2003. Domain-Driven Design: Tacking Complexity in the Heart of Software. Addison-Wesley.
Fowler, Martin. 2002. Patterns of Enterprise Application Architecture. Addison-Wesley.
Hohpe, Gregor, and Bobby Woolf. 2003. Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley.
Lübke, Daniel, and Tammo van Lessen. 2016. “Modeling Test Cases in BPMN for Behavior-Driven Development.” IEEE Software 33 (5): 15–21. https://doi.org/10.1109/MS.2016.117.
Voelter, Markus, Michael Kircher, and Uwe Zdun. 2004. Remoting Patterns - Foundations of Enterprise, Internet, and Realtime Distributed Object Middleware. Wiley.