Pattern Sections
Pattern: Retrieval
Operation 
a.k.a. Read-Only Operation, State Lookup Operation, Query, Data Extractor
Context
An API endpoint has been established; functional and non-functional requirements for it have been specified. However, the operations of these resources do not cover all required integration capabilities yet; the API client(s) also demand read only access to large amounts of structured, possibly aggregated data. This data can be expected to be structured differently than in the underlying domain model; for instance, it might pertain to a particular time interval or subdomain element (like a product category or customer profile group). The information need arises either ad hoc or regularly, e.g., at the end of/for a certain time interval (such as week, month, quarter, or year).
Problem
How can information available from a remote party (the API provider, that is) be retrieved to satisfy an information need of an end user or to allow further client-side processing?
Related sub-problems are:
- How can data model differences be overcome and data be aggregated and combined with information from other sources?
- How can clients influence the scope and the selection criteria for the retrieval results?
- How can the time frame for reports be specified?1
Forces
The following top-level forces have to be resolved when exposing data in API operations:
- Veracity, variety, velocity, volume, four of the five Vs in big data.
- Workload management
- Networking efficiency vs. data parsimony (message sizes)
Pattern forces are explained in depth in the book.
Solution
Add a read-only operation ro: (in,S) -> out to an API
endpoint, which often is an
Information
Holder Resource, to request a result report that contains a
machine-readable representation of the requested information. Add
search, filter, and formatting capabilities to the operation
signature.
Sketch
A simple graphical sketch (from pre-book times) that visualizes the read-only nature of this operation type is:
Examples
In an online shopping example, an analytic Retrieval Operation “show all orders customer ABC has placed in the last 12 months”.
In the Lakeside Mutual example, we can define two operations to find
customers as illustrated in Figure 2. CRM stands for Customer
Relationship Management; the allData parameter is a simple
Wish
List, allowing to return either an
Embedded
Entity or a
Linked
Information Holder.
A code-level example is:
curl http://localhost:8080/claims?limit=10\&offset=0
@GET
public ClaimsDTO listClaims(
@DefaultValue("3")
@QueryParam("limit") Integer limit,
@DefaultValue("0")
@QueryParam("offset") Integer offset,
@QueryParam("orderBy") String orderBy
) {
List<ClaimDTO> result = [...]
return new ClaimsDTO(limit, offset,
claims.getSize(), orderBy, result);
}Steps 5 and 6 of the demo “Domain-Driven Service Design with Context Mapper and MDSL” features a an instance of this pattern and introduces the MDSL decorator for it:
"RETRIEVAL_OPERATION" Set<@PaperItemDTO>
lookupPapersFromAuthor (String who);Are you missing implementation hints? Our papers publications provide them (for selected patterns).
Consequences
The resolution of pattern forces and other consequences are discussed in our book.
Known Uses
Known uses of this pattern are very common in enterprise information systems and public Web APIs:
- eBay has a traffic report API operation
- The
Cargo Repositoryin the Cargo Aggregate of the Domain-Driven Design Sample Application implements two basic find operationsCargo find(TrackingId trackingId)andList<Cargo> findAll(). - The Open Weather Map API illustrates this pattern in its many lookup options (expressed as parameters). For instance, historic weather data per city is provided by History Bulk. Note that only the responses are batched/bulked in a file; the requests are single/individual ones.
- The Slack Web API has a
files.listmethod. While not truly RESTful (but HTTP-based), this call shows a typical parameter and query response message structure. Another known use can be found in the Force.com API. It works with a Salesforce Object Query Language (SOQL) to define the query parameters. - The Swiss eGov initiative offers a central company lookup WSDL/SOAP
service called Zefix
with operations such as
searchByName.
GraphQL queries (but not its mutations) and restSQL can be seen as advanced, particularly flexible instances of the pattern (designed to avoid over-fetching).
The pattern is often applied in enterprise settings as well (just to show how common the pattern is:
- Some of the 1000+ services in Brandner et al. (2004) qualify as known uses of this pattern, for instance, “Kundengesamtübersicht” (i.e., a wild carded search for customers and an overview of their activities and products used). See this article for a business-oriented architecture overview.
- The same holds for a subset of the business and application services in the order management SOA presented in Zimmermann et al. (2005). An example is the customer master data lookup performed during phone number and address validation.
- A large Swiss banking software provider provides its clients an API to fetch aggregated data in the context of domain-specific use cases, e.g., a 360-degree view on a customer.
Terravis’ parcel information retrieval API Berli, Lübke, and Möckli (2014) provides examples of more sophisticated Retrieval Operations that do not directly translate to database queries. Because land registry data in Switzerland is federated, the parcel information retrieval API is a facade that hides the location of the data from the client. Internally, a request is resolved and routed to the responsible land registry, which in turn offers standardized APIs for land register data retrieval. The main operation retrieves a list of parcels by an electronic parcel ID (eGRID, German: “Elektronische Grundstücks-ID”). IDs can be resolved by querying by municipality, owner names, and so on. Requests have size limitations, i.e., only ten parcels may be queried at once in order to manage load on land registry systems. Operations may return historical data if this is desired. Depending on the number of managed objects, queries can be paginated (as described in our Pagination pattern) or the amount of data can be set to different levels (e.g., full, partial, full history) with a Wish List.
More Information
Related Patterns
The endpoint pattern Processing Resource and all types of Information Holder Resources may expose Retrieval Operations. The Pagination pattern is often applied in Retrieval Operations.
The sibling patterns are State Transition Operation, State Creation Operation, and Computation Function. An State Creation Operation pushes data from the client to the API provider, whereas a Retrieval Operation pulls data; both Computation Function and State Transition Operation can support unidirectional data flows and bidirectional ones.2
Other Sources
The is a large body of literature on database design and information integration, including data warehouses Kimball and Ross (2002). Chapter 8 in the RESTful Web Services Cookbook by Allamaraju (2010) discusses queries (in the context of HTTP APIs).
“Implementing Domain-Driven Design” Vernon (2013) talks about Query Models in Chapter 4 (in the section on CQRS). Endpoints that only expose Retrieval Operations form the Query Model in CQRS.
References
Note that the scheduling of periodic query execution (a form of batch processing) is out of scope here.↩︎
We call state-preserving processing roles functions (as they just get some work done on behalf of a client) and state changing ones operations (as they become active because the client hands in some data, which is then processed and stored).↩︎