Tutorial 1: Quality Patterns

This tutorial teaches you some of the patterns in the Quality category of MAP and introduces you to our pattern structure so that you can explore other ones on your own.

Self Study 0 Self Study 1

Time estimate: You should be able to complete this intermediate tutorial in about 45 minutes.

Learning Objectives

Having completed this tutorial, you should be able to:

  • Explain how the Quality category is organized.
  • Maneuver through individual patterns section by section, or based on your information need.
  • Select and apply patterns that help to choose request and response message sizes.
  • Select and apply quality-related patterns for API management and governance.

Note: We recommend to have a look at Tutorial 0 before going through this one.

Step 0: Baseline

In the previous tutorial, you learned about the Pagination pattern that lets you request data owned by an Information Holder Resource in small chunks (or pages, hence the name). This is helpful if clients want to retrieve and process data elements incrementally, but not so much if they only need parts of each of the data elements.

Our running example for this tutorial will be the policies Master Data Holder endpoint from our fictitious Lakeside Mutual insurance business sample application. This endpoint allows API clients to request insurance policies (paginated by limit and offset) via HTTP GET. See this excerpt from its OpenAPI specification:

"/policies": {
    "get": {
    "tags": [
        "policy-information-holder"
    ],
    "summary": "Get all policies, newest first.",
    "operationId": "getPoliciesUsingGET_1",
    "produces": [
        "*/*"
    ],
    "parameters": [
        {
            "name": "limit",
            ...
        },
        {
            "name": "offset",
            ...
        }
    ],
    "responses": {
        "200": {
            "description": "OK",
            "schema": {
                "$ref": "#/definitions/PaginatedPolicyResponseDto"
            }
        }
    }
},

In the OpenAPI specification a Data Transfer Object (DTO) called PaginatedPolicyResponseDto is referenced, containing an array of PolicyDtos. They are defined as follows:

"PaginatedPolicyResponseDto": {
    "type": "object",
    "properties": {
        "limit": {
            "type": "integer",
            "format": "int32"
        },
        "offset": {
            "type": "integer",
            "format": "int32"
        },
        "policies": {
            "type": "array",
            "items": {
            "$ref": "#/definitions/PolicyDto"
            }
        },
        "size": {
            "type": "integer",
            "format": "int32"
        }
    },
    "title": "PaginatedPolicyResponseDto"
}
"PolicyDto": {
    "type": "object",
    "properties": {
        "creationDate": {
            "type": "string",
            "format": "date-time"
        },
        "customer": {
            "type": "object"
        },
        "deductible": {
            "$ref": "#/definitions/MoneyAmountDto"
        },
        "insurancePremium": {
            "$ref": "#/definitions/MoneyAmountDto"
        },
        "insuringAgreement": {
            "$ref": "#/definitions/InsuringAgreementDto"
        },
        "policyId": {
            "type": "string"
        },
        "policyLimit": {
            "$ref": "#/definitions/MoneyAmountDto"
        },
        "policyPeriod": {
            "$ref": "#/definitions/PolicyPeriodDto"
        },
        "policyType": {
            "type": "string"
        }
    },
    "title": "PolicyDto"
},

Looking at the PolicyDto data type definition, we can see that it refers to many other DTOs, which in turn might refer to further DTOs. Chances are high that clients only need parts of that information.

The overview of the Quality Patterns category lists a number of patterns that can be applied to improve this situation (for an in-depth introduction, see our EuroPLoP 2018 Interface Quality Patterns paper).

Task Overview

In this tutorial, you will learn about patterns that help you to:

  1. Reduce the message size.
  2. Reduce the number of requests.
  3. Provide basic access control and document the terms and conditions for API usage.

Step 1: Data Transfer Parsimony

The Data Transfer Parsimony patterns offer solutions to let API clients communicate exactly what data they need to reduce the amount of unnecessarily transferred data.

Task 1.1: Familiarize yourself with the Wish List Pattern

Please have a look at the problem and solution of the Wish List pattern.

  1. Can the pattern help us avoid unnecessary data transfer?
  2. Which other sections is the pattern structured into?
  3. Does the pattern only aid in reducing the amount of data transferred, or are there variants that address a different quality aspect (hint: consult Chapter 7 of our book or the EuroPLoP 2018 paper version of the pattern?

Hint: Apply the divide-and-conquer strategy to comprehend, and take one step at a time when browsing the pattern language and individual patterns. Some pattern readers read the context, problem, and “bold face” solution summary first, and then dive into individual sections (such as forces, consequences, or “how it works” solution details and known uses).

Solution(s) (click to show)

  1. Yes, by letting the client wish what data elements it wants, the endpoint can respond with exactly the right amount of data.
  2. The sections are: Context, Problem, Forces, Solution, Consequences, Known Uses, More Information. See section ” Our pattern template” on the page “Introduction to Microservices and APIs” page or Section 3 in “Introduction to Microservice API Patterns (MAP)” for explanations.
  3. The pattern not only helps to optimize the amount of data that is transferred: the expansion variant offers a way for clients to request more data in the initial request, thus reducing the amount of requests needed.

Task 1.2: Apply the Wish List Pattern

An analysis of API clients has shown that most do not need the information about the customer of a requested policy. Let us assume that the API uses Semantic Versioning. For the next major API version, the customer should not be included by default, but clients can request it to be included.

What does it mean for the /policies endpoint to apply the Wish List pattern? Which changes are necessary? If you’re comfortable with the OpenAPI Specification, you can formulate your answer as an update to the specification or describe it in plain text.

Solution(s) (click to show)

The endpoint must provide a way for clients to indicate which result parameters should be expanded. For example, this could be done using a simple Atomic Parameter List:

"/policies": {
    "parameters": [
        {
            "name": "expand",
            "in": "query",
            "description": "a comma-separated list of the fields 
                that should be expanded in the response",
            "required": false,
            "type": "string"
        }
    ],

So in our next version of the API, by default only the customer’s ID will be returned:

curl http://localhost:8090/policies

{
  "policies" : [ {
    "policyId" : "fvo5pkqerr",
    "customer" : "rgpp0wkpec",
    ...
    "_expandable" : [ "customer" ]
  } ]
}

But if the client wishes, the customer can also be expanded:

curl http://localhost:8090/policies?expand=customer
{
  "policies" : [ {
    "policyId" : "fvo5pkqerr",
    "customer" : {
      "customerId" : "rgpp0wkpec",
      "firstname" : "Max",
      "lastname" : "Mustermann",
      ...
    },
    ...
    "_expandable" : [ "customer" ]
  } ]
}

See Lakeside Mutual’s PolicyInformationHolder::getPolicy method for the implementation details.

Task 1.3: Compare Wish List and Wish Template Patterns

You might have noticed that Wish List has an alternative pattern, called Wish Template. Have a look at the solution section of this pattern.

  1. How do the two patterns compare? What is similar? What is different?
  2. Does Wish Template remind you of a currently trending technology (hint: look at its known uses section if you need help)?

Solution(s) (click to show)

  1. Both patterns address the same problem, to avoid over-fetching of data. Wish List uses a flat enumeration rather than a mock object; both patterns deal with instances of Parameter Tree in response messages, but the Wish Template becomes part of a Parameter Tree that appears in the request message and is populated by the endpoint.

  2. GraphQL can be seen as an “über-solution” to the problem solved by Wish Template (i.e., avoid over-fetching). Among other things, GraphQL defines the structure of responses in a schema, and clients use instances of this schema to express their data selection and navigation wishes. This requires rather advanced server-side processing.

Step 2: Reference Management

Our Reference Management patterns are concerned with the granularity of request and response messages.

Task 2.1: Have a look at the Reference Management Patterns

The original designer of the /policies endpoint decided to embed the entire content of all policy-related entities in the GET response (for instance, customers and their attributes).

  1. Can you guess what the reasoning behind that decision was?
  2. When you read through the names of the patterns in the Reference Management sub-category only, which pattern has most likely been applied to this endpoint?
  3. Can you think of other solutions?

Solution(s) (click to show)

  1. Proactively delivering all related entities preempts the client of having to perform follow-up requests. This can simplify the development of the client and increases performance, but only if the client needs that data. As we saw in the previous step, this is not always the case.
  2. The designer applied the Embedded Entity pattern. Here is its problem statement: “How can one avoid sending multiple messages when their receivers require insights about multiple related information elements?”
  3. If a single, general-purpose API struggles to accommodate different kinds of clients, applying the Backend for Frontend pattern documented by Sam Newman might help. This in effect creates specialized API backend “facades” that address individual client’s needs.

Task 2.2: Change API contract from Embedded Entity to Linked Information Holder

The following excerpt of a policy response exemplifies the issue that requesting a policy includes all the customer details. This not only increases the response message size considerably, the API implementation most likely will also need to query the customer entity from the database. This is all in vain if the data is not subsequently used.

curl http://localhost:8090/policies/fvo5pkqerr

{
  "policyId" : "fvo5pkqerr",
  "customer" : {
    "customerId" : "rgpp0wkpec",
    "firstname" : "Max",
    "lastname" : "Mustermann",
    "birthday" : "1989-12-31T23:00:00.000+0000",
    "streetAddress" : "Oberseestrasse 10",
    "postalCode" : "8640",
    "city" : "Rapperswil",
    "email" : "admin@example.com",
    "phoneNumber" : "055 222 4111",
    "moveHistory" : [ 
      ...
    ]
  },
  "creationDate" : "2020-05-02T12:07:41.544+0000",
  "policyPeriod" : {
    "startDate" : "2018-02-04T23:00:00.000+0000",
    "endDate" : "2018-02-09T23:00:00.000+0000"
  },
  "policyType" : "Health Insurance",
  "deductible" : {
    "amount" : 1500.00,
    "currency" : "CHF"
  }
}

Instead of an embedding the whole entity, the API could just return a Linked Information Holder. Read the Problem, Forces and Solution of Linked Information Holder and answer the following questions:

  1. After applying this pattern, how can the client know that there are related entities?
  2. Take the response above and transform it so that the customer entity is no longer embedded but a Linked Information Holder. Which pattern from another category can help with (re-)structuring the response (don’t worry about implementation details such as endpoint URLs)?

Solution(s) (click to show)

  1. A hyperlink to the related entity is included in the response, which indicated where to direct the next HTTP GET to.
  2. All the detailed information about the customer should be removed, but the id could still be included. A Link Element has been added to show where the full customer details can be obtained:
{
  "policyId" : "fvo5pkqerr",
  "customer" : {
    "id" : "rgpp0wkpec",
  },
  
  ...,

  "_links": {
    "customer": "http://localhost:8090/customers/rgpp0wkpec"
  }
}

This is just a minimal example, the link-relation could/should also include additional metadata, like the content-type of the response (Allamaraju (2010)):

  "_links": [{
    "rel": "customer",
    "href": "http://localhost:8090/customers/rgpp0wkpec",
    "type": "application/vnd+lakeside.customer+json",
  }]

Task 2.3: Q&A

Now that you have read about the Embedded Entity pattern and applied the Linked Information Holder reference management pattern, you should be able to answer the following questions about Embedded Entity and Linked Information Holder:

  • Which of the two patterns causes less messages to be sent?
  • Which of the two patterns downsizes messages?
  • Can the patterns be combined?

Solution(s) (click to show)

The answers are:

  • Embedded Entitycauses less messages to be sent because related entities are already included in the response to the initial request.
  • Using Linked Information Holders reduces message sizes because only the id is included and not the whole representation of the entity.
  • Yes, if some related entities are needed by most clients and others just by a few, the two reference management approaches can also be combined. In fact, such hybrid approach is quite common in practice.

See the online article “What is the Right Service Granularity in APIs?” for more detailed coverage of the two patterns.

Step 3: Quality Management and Governance

The Lakeside Mutual business is expanding, and they have decided to open up their platform and APIs for selected third parties.

Task 3.1: Review Quality Management and Governance overview

Take a look at the problem/solution pairs of the patterns listed under Quality Management and Governance:

  1. Which Quality Management and Governance patterns should you consider for the first sprint?
  2. The API should only be accessible to selected clients. Is there any pattern that could help?

Solution(s) (click to show)

  1. As a provider, you want to communicate the quality-of-service characteristics of the API, which you can do with a Service Level Agreement. You could also specify a Pricing Plan to meter API service consumption. Want to know what other patterns should be considered during the inspection of a new API? Take a look at the patterns by phase filter.
  2. The API should be secured by an API Key, which approved clients can obtain. By including their individual API Key in each request, the API provider can identify each client. Note that the API Key pattern appears under “Special Purpose Representations” in the Structure category.

Task 3.2: Introduce basic access control

  1. Outline the steps you, as the API provider, need to follow when introducing API Keys for clients.
  2. Are API Keys a good fit for all kinds of clients, or are there better alternatives?

Solution(s) (click to show)

  1. Consider the following steps when applying the API Key pattern:
    • Research if your framework, or a third-party library, already offers support for working with API keys. If not, then you will most likely find tutorials and blog posts describing how to integrate API Keys with your framework.
    • Decide on how the API Keys should be transported, e.g. in a header, as part of a URL query string, or in the request body.
    • Secure your endpoints and write automated integration or end-to-end tests to make sure that your endpoints are only accessible with a valid key.
    • Provide a means for clients to manage, i.e., create and delete, API Keys.
    • See the How it Works section of the pattern for more hints.
  2. API Keys are a useful first step to identify non-human clients. An API Key can also be combined with an additional secret key to ensure the integrity of requests. The secret key is shared between the client and the server but never transmitted in API requests. The client uses this key to create a signature hash of the request and sends the hash along with the API Key. The provider can identify the client with the provided API Key, calculate the same signature hash using the shared secret key and compare the two. This ensures that the request was not tampered with. To authenticate human users, better alternatives such as JWT, OAuth or OpenID exist.

Task 3.3: Document terms & conditions in an SLA

Don’t worry, you will not be tasked with writing terms and conditions. If you have made it this far in our tutorial, you’re probably an engineer or otherwise technically minded person, not a lawyer. While you need someone to set out the general terms and conditions that your clients will need to observe when using your API, one aspect where technical expertise will be needed is the Service Level Agreement with its Service Level Objectives (SLOs).

Please read the Problem and Solution sections of the pattern and answer (or even better, discuss with a colleague) the following questions:

  1. What do you see as the main benefit of an SLA?
  2. For which of the following scenarios might an SLA be useful?
    1. A Public API used in Frontend Integration, for example a payment provider.
    2. A Solution-Internal API for Backend Integration, for example a component in a platform that integrates mortgage processing.
  3. Which section(s) of our patterns help you answer these kinds of questions?

Solution(s) (click to show)

  1. There’s no single answer to this question, as the SLA bridges technical and contractual concerns; different people with different roles will come up with a wide range of possible answers. One might argue that in practice, SLAs are worthless because the penalties are too small and the burden of proof too high. On the other hand, providers (hopefully) won’t risk making promises that they can’t hold, so an SLA gives an indication of what kind of quality to expect.
    1. Yes, the SLA will help prospective clients decide whether the provider’s offering matches with its business needs.
    2. If both API client and provider are part of the same organization, an SLA might not be needed. Defining SLOs can still make sense as part of a Site Reliability Engineering operation: “The SLO violation rate can be compared against the error budget, with the gap used as an input to the process that decides when to roll out new releases.”
  3. The Forces section shows what makes a pattern’s problem hard (but worthwhile) to tackle; the Consequences section discusses how the pattern solution can help to achieve that.

End Result

Congratulations! You now know how the Quality patterns category and our patterns are structured, how messages sizes can be rightsized with patterns such as Wish List, Wish Template, and Linked Information Holder. And you also have learned about API Keys and Service Level Agreements.

A different sample implementation of the patterns featured in this tutorial is available in the public Lakeside Mutual repo.

Tutorial 2 features all pattern categories and progresses from service/endpoint identification to role and responsibility design, structure and quality design through to API evolution.

Self-Study Exercise and Repetition Questions

Find a similar, self-study tutorial and some repetition questions here.

References

Allamaraju, Subbu. 2010. RESTful Web Services Cookbook. O’Reilly.

Back to homepage