Problem Details for HTTP APIs

The positive experience of RFC 7807, whose journey began in 2016, is concluded (deprecation) but also confirmed with a new official proposition: the RFC 9457.
The changes made are small but it is a suitable opportunity to analyze the evolution of this topic.

Context

The RFC 7807, and now the RFC 9457 which perfects it, was born after a period in which the Internet Companies have made it clear that the integration of the business, the eventual transformation of the company services into a platform and a part of the organizational flexibility can pass from using the API.

The lack of references and the need to be time-to-market has meant that these same entities pollute the panorama of approaches and implementations in a set of non-interoperable solutions directing to some effects:

invalidating, in part, the effectiveness of the API
creating the need to define alternative tools
defining the need for standards to have a more effective and evolving product, consultancy, and research market over time

The motivation leading to these RFCs is the impossibility of expressing errors, both semantically and in terms of contents, using only HTTP Status Codes.
This premise does not mean that:

HTTP status codes are bad
HTTP status codes are not useful
HTTP status codes need to be replaced

Why are we talking about HTTP API?
HTTP API means the possibility of communications machine to machine, implemented upon the HTTP protocol thanks to modular elements defined by specific RFCs. So we are not talking about REST but about a more open superset in favor of large-scale developments.

This RFC therefore offers the benefit of allowing information to be conveyed for both humans and machines: this paves the way for the creation of better related tools.
It is not, however, the goal of the RFC to provide an application debugging tool.
All this is summarized in the specification document of which important sentences are as follows:

...This specifcation's aim is to define common error formats for applications that need one so that they aren't required to define their own or, worse, tempted to redefine the semantics of existing HTTP status codes. Even if an application chooses not to use it to convey errors, reviewing its design can help guide the design decisions faced when conveying errors in an existing format...

...Problem details are not a debugging tool for the underlying implementation; rather, they are a way to expose greater detail about the HTTP interface itself...

...truly generic problems -- i.e., conditions that might apply to any resource on the Web -- are usually better expressed as plain status codes...

...an application might have a more appropriate way to carry an error in a format that it already defines. Problem details are intended to avoid the necessity of establishing new "fault" or "error" document formats, not to replace existing domain-specific formats...

Communication aspects

In the context of HTTP, the first element to consider is the proper representation of the type of information transported:
this also means having a reference to the REST approach and Hypermedia.
The specification adopts two specific media types as representation indicators, also considering the most used standards at the moment, for data transport, namely JSON and XML;
resulting in two registered media types:

application/problem+json
application/problem+xml The RFC does not offer particular recommendations for managing other types of media, even if it proposes the concept of "encapsulating information" in other representations, where possible. This is, for example, the case of HTML5 which has more possibilities for hosting metadata not directly related to rendering web pages (scripts, as for the example that follows, or microformats or microdata). Here's an example:

<script type="application/problem+json">
{ 
 "type": "https://example.com/probs/out-of-credit",
 "title": "You do not have enough credit.",
 "detail": "Your current balance is 30, but that costs 50.",
 "instance": "/account/12345/msgs/abc",
 "balance": 30,
 "accounts": ["/accounts/12345",
 "/account/67890"]
}
</script>

The RFC confirms the use of the proactive content negotiation approach.
The communication includes the main HTTP headers valid for content negotiation such as Content-Type, Accept but also Accept-Language, aiming to provide non-binding hints on the language for the texts present in the HTTP responses.

What happened to the header problem? A curiosity

One draft of the new specification, initially called RFC 7807bis, introduced the Problem HTTP Header: this would allow a Problem to be shown according to the specification bypassing a response body.

HTTP/1.1 200 OK
Content-Type: application/json
Problem: type="https://example.net/problems/almost-out", title="you're almost out of credit", credit_left=20

This idea has not been approved, even because it could have introduced protocol inconsistency, e.g., what concerning the media type? What concerning the representation: is it an error or an answer?
An alternative to the Problem Header can be the embedding approach or the consideration that the specification does not force migration if a previous solution is already present or the possibility of creating customized problem types.

The response body

The specification requires structured content to convey information about errors;
it's worth noting that the non-understanding information, expressed in the body, must not block the activity of a generic HTTP software receiving data, permitting the latter to skip the information unknown.
Different attributes are offered:

We can then have specification-compliant dialogs like in the following example:

POST /purchase HTTP/1.1
Host: store.example.com
Content-Type: application/json
Accept: application/json, application/problem+json
{
 "items": 123456,
 "quantity": 2
}

HTTP/1.1 403 Forbidden
Content-Type: application/problem+json
Content-Language: en
{
 "type": "https://example.com/probs/out-of-credit",
 "title": "You do not have enough credit.",
 "detail": "Your current balance is 30, but that costs 50.",
 "instance": "/account/12345/msgs/abc",
 "balance": 30,
 "accounts": ["/accounts/12345",
 "/account/67890"]
}

An interesting aspect of the specification is the definition of a default type in case you simply want to report the status code information but in the format provided by the RFC:

{
 "type": "about:blank ",
 "status": [ number related to the specific HTTP status code ],
 "title": [ string message related to the specific HTTP status code ],
}

Note that the "type" attribute, if not specified, acquires the value about:blank with all that follows.
The response body of this RFC may have extensions discussed later.

The type attribute

It is a mandatory attribute, expressed as a URI that identifies the type of problem.
The value of this attribute can be:

about:blank
URI Locator
URI Tag

If the attribute value is a URI Locator, then it is strongly recommended that dereferencing it results in human-readable documentation for the type of problem.
Clients should not dereference the locator automatically but only as needed to provide documentation to developers.
If the attribute value is a Tag URI, e.g., tag:example@example.org,2021-09-17:OutOfLuck, it is not dereferenceable and uniquely represents the different types of problems.
Using a URI to have a unique ID is not recommended especially from the point of view of integration and use of tools for which providing further information at the Developer Experience level is a fundamental objective.
The locator must use absolute rather than relative URIs to avoid confusion and malfunctions

The status attribute

It is an optional attribute that has an informative nature.
When created it must report the same status code defined in the HTTP response and its purpose is to allow Consumers to be aware of the original status code possibly modified by an HTTP intermediary.
Being a duplicate in the HTTP response code value, this attribute can be a point of attack if used without awareness.

The title attribute

It is a mandatory attribute that has an informative nature for all those users who do not know or cannot know the semantics of the problem expressed by the URI of the type attribute.
The value of this attribute is strongly recommended to remain constant between one occurrence and another of the problem, except for possible localization needs.

The detail attribute

It is an optional attribute containing a human-readable explanation of the specific occurrence of the problem.
It’s highly recommended that the value of this attribute does not contain technical information for debugging purposes which instead can be introduced in extension attributes: this attribute should therefore not be subject to particular manipulations.
This is also a point to remember for security purposes to reduce the amount of information in transit that aids API attacks.

The instance attribute

It is an optional attribute containing a URI that identifies the specific occurrence of the problem.
The value of this attribute can be:

URI Locator
URI Tag

When the URI is a dereferenceable locator, the problem details object can be retrieved from it.
It may also return information about the problem occurrence in other formats through the use of proactive content negotiation.
When the URI is a Tag (non-dereferenceable), it serves as a unique identifier for the problem occurrence which may be meaningful to the server but is opaque to the client.
As for the type attribute, absolute rather than relative addresses are to be preferred.

Extension members

It is possible to customize the body of a response with other attributes that offer more details about the problem that has occurred.
The specification does not contain particular rules to limit these extensions, except for adherence to the media type and relative naming of reference, but it highlights how these can be entirely avoided by a client:
unrecognized extensions must be ignored and therefore not block management of the answer.

Handling multiple problems

What does multiple problems mean? Who are they referring to? Can they happen?

The answers come from analyzing some coordinates:

To what (which is the subject) the errors found refer
How to represent a set of errors
Which errors are priority or not

RFC 9457 clarifies and responds with simple and immediately adoptable concepts:

A response of type Problem Details refers to a specific type, declared in the same, for which there may be multiple problems that must refer to the same resource (type) of the issued request. The presence of JSON among the main formats allows support for JSON Pointers to highlight the parts of the Json document that are affected by errors.
A set of errors defined, in the reference media type, thanks to the concept of body extensions (Extension Members): those additional attributes of the response body, are possible in the respect of the specification itself

POST /details HTTP/1.1
Host: account.example.com
Accept: application/json
{
 "age": 42.3,
 "profile": { "color": "yellow" }
}

HTTP/1.1 422 Unprocessable Content
Content-Type: application/problem+json
Content-Language: en
{
 "type": "https://example.net/validation-error",
 "title": "Your request is invalid.",
 "errors": [
   { "detail": "must be a positive integer", "pointer": "#/age" },
   { "detail": "must be 'green', 'red' or 'blue'", "pointer": "#/profile/color" }
  ]
}

The creation of a set of errors that do not concern the same type becomes strongly discouraged; this implies that of the possible errors, relating to different types, only the one considered a priority (relevant or urgent) is reported consistently with the above. It should also be remembered that if a format already exists for specific application needs, it is not required to convert to the RFC format.

The proposed vision, in the presence of multiple problems, may seem limiting but it is a balancing point between protocol cleanliness, representation effectiveness, and tool support.

In the deprecated RFC 7807, the use of the HTTP 207 status code was proposed, which allows you to report multiple statuses and errors in the same body. This solution, technically convenient, can lead to ambiguity and also reopens the issue of multi-request and multi-response which are not concepts natively provided for in the HTTP protocol.

The importance of the type

The specification highlights the idea of type related to a specific problem thus allowing two essential evolutions:

The ability to define custom types
The possibility of recording types of Problems in favor of their reuse where appropriate

The concept of type is significant and requires careful planning and the need to follow rules.
Such perspective leads to a balanced use of the specification and the possibility of improving and enriching the automation, and therefore supporting with products, of API design.

Defining custom problems involves documenting:

A specific URI that is relative to the type, this URI is preferable to be a locator and therefore dereferenceable in the documentation that indicates how to approach the problem
A title that briefly and appropriately describes the problem
The indication of the HTTP status code with which the type of problem is supposed to be related.

A custom type can have extensions (which must follow naming rules) and can use the Retry-After response header (if appropriate). The reference to Web Linking is interesting as a possibility to be used in extensions to refer to other resources and address the problem in question.

The registration of a new problem, in the registry provided for this specification at the address https://iana.org/assignments/http-problem-types, involves the use of a reference template, a possibly usable prefix URI, and the submission of a request to a group of experts for the validation of the proposition.

Conclusion

This new specification defines better support to obtain the order, conceptual cleanliness, and appeal in the significant market of APIs. RFC 9457 is conceptually simple but offers the elements to operate on almost all the use cases that can be experienced. It’s one of more components, the HTTP API RFCs, that should not be seen in isolation but moved in the continuous trade-off of own development/integration needs and efforts to be open and flexible in a market full of initiatives and possible new tools.

Blog

Problem Details for HTTP APIs - RFC 7807 is dead, long live RFC 9457

Stefano Fago

Context