HTTP Status Codes For Invalid Data: 400 vs. 422

Great catch. I'll have to add that one to my list of things to watch out for in all of the remote services I've been working on lately.

My pleasure! When I first read a spec definition for 422, it still didn't feel like a perfect fit, but it does seem better fit than 400. Cheers.

I agree, 400 shouldn't be used here, as the http request is syntactically valid and conform to the protocol specifications.

I have problems with a 422, too - in the outlined scenario, the user (and not the http client, likely a browser) is causing the error, hence the error should not be reported back to the http client at all - the user is the only one that can provide a valid alternative here, so i'd opt for a 200 and a nice, lengthy description of what went wrong (and how it can be fixed) in the payload.

If you write a custom client application or any other form of machine-machine interface, it may have a more distinct meaning (other than "something went wrong"), but nobody stops you from using 499 or any other custom 4xx code anyway.

I stumbled across this a while back. Maybe from your site!

The (ColdFusion 9) code will dump a list of HTTP status codes.

<cfdump var="#getPageContext().getResponse()#" />

Nah, 400 is correct in this case. "Syntax" doesn't just mean "well-formed XML" (or JSON or post data or whatever), it means that it's syntactically correct in the context of the application.

@tps12,

A "400 Bad Request" status code indicates that the http request could not be understood - this is pretty much only related to wel-formedness of the request header and rarely (if ever) used when the post data is malformed, and should be used only when the server is unable to interprete the request.

This is not the case here.

As a POST body can contain arbitrary data, a problem with these data can only arise when the provided mime types does not match the body encoding (i.e. "application/x-www-form-urlencoded" when sending an XML or JSON body), or the encoding is invalid (bad XML structure, for instance).

These are all encoding (syntax) problems.

If there is no problem interpreting the body data, but a problem with the contents of these data (like Ben outline in the opening post), a 400 status code is not justified.

A "406 Not Acceptable" or "409 Conflict" would be more appropriate here, if you really want to indicate the problem in the response header, but a problem that occurs in a higher layer should not be indicated in the same layer as the http request - otherwise you are just trying to redefine http/enhance it with non-standard features.

@wizzszz,

I don't read the spec that way: if an XML request entity is not well-formed then respond 400 and say "couldn't parse XML" in the response body, and if it has a "foo" element in place of a "bar" then respond 400 and say "found foo in place of bar" in the response. I understand the nuanced situation 422 is supposed to address, I just don't think it's necessary.

406 and 409 are completely wrong in this context.

@tps12,

i think you missed my point - the hyper text transfer protocol (http) does not define anything related to the payload (http body data) - all it does is hinting how the body data can be interpreted (by providing a mime type).

If a problem arises from the payload (does not matter if syntactical or semantical), it is not a problem directly related to the http transport mechanism, and should not be indicated with http means.

The description of 422 matches better, but it suffers from the same problem - plus, it is a WebDAV only extension, which you'd basically abuse for your own flavour of http.

We have standards for a reason, one of the is interoperability - if you change http to "my custom transfer protocol" by adding a different or enhanced meaning to well-defined status codes, you totally defeat the purpose of having standards.

And last but not least: It is a well-established technique to separate data transport from data processing - in case the meaning of "different layers" i mentioned in reference to the OSI layer model before was not clear.

@Ben,

but wasn't that exactly my point to not mix up HTTP transport and business-logic?

Granted, it is convenient to report a problem in the http status code already (and save the hassle of parsing the http body), but...

... http status codes all have something in common - they report problems (or the lack thereof) with the connection to the server, the request itself or the requested resource, and they have no relation to what happens in the business-logic layer (and should not have any relation to them, because that's the basic idea behind layers)

That's why there is no appropriate http status code for "something went wrong in the business logic layer", and frankly, if we start to add such http result codes, we will have a gazzillion new error codes in no time.

It's not like it's forbidden to use whatever status code you want to report custom conditions - it's just very bad form imho.
And it is usually a good indicator that an application suffers from severe misplannings.

Just imagine this:
Your business logic only calls fetchData() with the URL and (optional body data), and returns an already parsed JSON object or an integer status code on error.

The fetchData() subroutine handles all the http stuff, i.e. redirections are resolved automatically and re-requested, and everything that cannot be handled results in an error report (and leaves it to the business logic to decide what to do, i.e. try a different server)

This way, the business logic doesn't "ooze" into the networking stuff and vice versa.

Both can be tested independently (you can do as many fetchData() calls for testing as you like) and the business logic can be tested by making fetchData() return whatever you need to test.

And the best part is that you can re-use your thoroughly tested network logic in other projects, as it is completely independent from any business logic.

This is the way to create rock solid code, the only way imho.

If someone knows a better way, please, let me know - i'm always happy to learn something more efficient.

"Weeks of coding can save you hours of planning."

I'm sorry, forgot to answer your question in my last post, so here you go:

The hyper text transfer protocol (HTTP) was originally designed by Roy Fielding, Tim Berners-Lee and others at CERN. It is meanwhile maintained by the IETF, just as any other RFC.

The "standard" for HTTP/1.1 (RFC 2616) can be found here:
http://tools.ietf.org/html/rfc2616

@wizzszz,

I guess I would quibble with "http status codes...have no relation to what happens in the business-logic layer."

E.g., imagine an application where a given resource can only be DELETEd by certain users (admins, the resource owner, or whatever) as identified in the Authorization header. The determination for who can DELETE what is completely in the realm of business logic, but if a different user makes the request then a 403 response seems completely appropriate according to the spec.

@tps12,

deleting resources is beyond the scope of http, and that's not the only reason why a 403 would be inappropriate here - many http clients will not even retry a http request when they receive a 403 status code.

http only defines access in a yes/no way (basic, digest and nowadays oauth), so if you have access to a resource (read-only is enough), you can't reply wih a 403 Forbidden.

If you add more granularity (read: yes, delete:no), it all happens in the business logic, and has to be reported as part of the business logic (in the http body data).

When it is not a http problem, why indicate a http problem?

You appear to make much effort to find a status code with a definition that matches the raised issue, but you forget that in your example, the only thing that is "Forbidden (403)" is the delete operation - and there is no "Operation denied" status code.

Someone using your service/API will curse you for days when it is not clear why a resource that he was able to read just seconds ago is now "Forbidden" all of a sudden...

@wizzszz,

How is DELETE beyond the scope of HTTP? It's in the spec!

@tps12,

sorry, didn't realize that you were talking about the DELETE request method.

If you are using DELETE, a 403 would be of course appropriate, although a more specific status code would be better - i.e. WebDAV is using a 207 Multi-Status here (see example here: http://tools.ietf.org/html/rfc2518#page-36 )
, but WebDAV is a completely different matter anyway:
Designing a read/write logic on top of a read-only protocol kind of always results in ugly hacks.

The DELETE command is in the http itself (header), not in the http payload (body), and when you issue a DELETE it will be clear what a 403 in return means.

That's entirely different to what happens in the http payload and should stay separated.

Not really sure why you think this must be part of the business logic though - the server handles it without calling your script/isapi.dll/whatever.

Please have a look at how i.e. apache handles WebDAV access rights, it's all done in the server configuration.

If you write your own server/client application, which nobody else will ever use, then, by all means, go for it and (ab)use whatever you like.
But if you write code like that for a public API, stick to what other people (read: fellow coders) know and expect (even if that's not always 100% standard).
If i'd receive a 4xx error in return when calling your API, the last thing i'd consider is a problem within the body data.

@wizzszz,

Fair enough. By the same token, if I were to attempt to update a resource with a PUT request when calling *your* API, and got a 200 status in response, then the last thing I would expect is that the resource might not in fact have been updated.

Always interesting to hear other perspectives on stuff like this, though. Thanks!

@tps12, @ben,

Why would i respond to a PUT operation with a 200 when an error occurs?

That's a plain vanilla http operation, hence the use of http status code is not only justified, it is mandatory.
(i.e. 201 Created or something like that)

There are, besides WebDAV, no applications using PUT or DELETE "in the wild", so this is a merely academical consideration. The specs list a lot of stuff that was meant to become part of HTTP/1.2 or even 2.0 (like 402 Payment Required).

-

And Ben, i got caught on the "philosophical" in your comment - applying uniform standards is not exactly important to you?

The only reason why computers can interoperate are standards - if you add custom stuff or give standardized components a different meaning, you "pollute" the standard, and subsequently, it will lose it's meaning in that particular context.
If everyone does that, what do you think happens to the standard?

Don't do that, if you really hate parsing that much, how about adding a custom header field instead to indicate success/failure of the operation?
But you should return the error in the body nonetheless.

@wizzszz,

"Why would i respond to a PUT operation with a 200 when an error occurs?"

So what do you use then? A client tries to PUT a resource and you determine that for whatever reason it's invalid in the context of your application and reject the request. The status code you use is...?

@tps12,

Frankly, i wouldn't use PUT at all, there are other means to upload a file (i.e. mimic the behaviour of a html FORM elements and send a multipart/form-data encoded body)

But, let's say i would need it for a project (not trying to dodge your question here), i'd implement WebDAV (as far as i need it) and return a status code in compliance with the WebDAV specs.

This way i am using either RFC2616 (HTTP/1.1) or RFC2518 (WebDAV) and not some homegrown, incompatible modifications to standards.

It maybe my non-native english or my passion for nested appositions that it has not become clear yet where i draw the line - hence here a rule of thumb:

If the error is a result of a http operation, report the error with an appropriate http status code.
If it's WebDAV operation (DELETE, PUT,...) use the http status codes including those added for WebDAV.
If the error occurs in the business logic, report the error in the body and use 200 Ok

There are a few exceptions though, i.e. the business logic should be able to trigger a 500 response when something really goes foobar and replying with a proper response in the body is not possible.

@wizzszz,

Cool, I think we are just coming at things from totally different perspectives. If your API is limited to GETs and form data, then I can see that you wouldn't be able to rely on status codes as much as is possible in a more RESTful design.

FWIW, PUT is very much part of HTTP/1.1 and in no way homegrown or limited to WebDAV (see http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.6).

@tps12,

Doesn't have much to do with how complex an API is - if it's a basic function, why use advanced status codes?
You'll just forfeit the chance that simple/older clients could still use parts of your API, even if they can't handle the advanced parts. (read: maintain backwards compatibility)

PUT is in the specs, i know, but not everything in the specs makes much sense (most of it must have been specified to reserve it for future use) - with status 402 being a prime example.
And 418 is the result of an aprils fool joke...

I know, i was the one coming up with the specs, but "commonly used like that" outweights the specs by far.

I have never seen any good use of PUT and DELETE outside of WebDAV, and not even WebDAV is much used...
(Microsofts fault mainly, but let's better not go there)

@Ben,

own API, own client = do whatever you like.
A "488 Ben says: Invalid parameter type" would work fine, too, not sure why you make an effort to find an appropriate existing status code when you don't stick to standards anyway... ;)
Every 4xx status code should work fine with any http(s) library out there.

However, i, for one, would still not do that (nor hire someone who does):
Imagine you win the lottery and move to Hawaii, and your customers are stuck with heavily customized code nobody can maintain/enhance.
Same goes for own projects, you might be too busy to code everything yourself one day and hire someone who runs into the same problem.

And i wouldn't worry too much about efficiency here, parsing the small JSON snippet in your example above won't make any notable difference - unless your API has to handle several 1000 request per minute. It's an error that shouldn't occur often anyway (outside of development testing) if you write the client code, too... ;)

@Ben,

you still shouldn't "bend" the standards for that.

You have posted a tricky example, because request.done can't call request.fail here directly, but if don't use a lambda style function for fail here and instead pass a static function, you can.

var request = getSomeServerData().
 
var errorHandler = function( response )
{
	/* report error to user / log it */
}
 
// 200 response.
request.done(
	function( response )
	{
 
		var R = jQuery.parseJSON( response );
 
		// Check for app error
		if ( R['error'] != 0 ) {
			errorHandler(response);
			return;
		}
 
		// No error, do whatever your request.done used to do before
		....
	}
);
 
// 4xx / 5xx response.
request.fail( errorHandler );

It's only a few more additional lines, plus, the static errorHandler function is way more efficient than creating a new lambda-style function every time you make an ajax request...
(Actually, deferred object are real resource hogs, as you create all the functions anew for every new promise object, but i digress...)

@Ben,

In those old posts, I actually DO return a 200 OK response for all "successful" HTTP requests, and then encapsulate the business-logic-failure within the structure response. Then, I use a response preprocessor to route the response to the error-handler IF the business logic indicates a failure.

If you already did it this way before, i wonder why you don't stick to that concept and try to dabble with http response codes now...

It feels a bit like you're trying to evolve, to refine the way you do things, and take a step back without realizing it.

I've never really thought of the Promise as being a resource hog. I mostly assume that as long as we don't create circular references in the closures to DOM elements, then memory [leaks] is not too much of a concern.

The promise object is filled with callback handlers, which are created every time anew when you use anonymous (lambda-style) functions.
Better to simply pass static functions as handlers wherever you can.
(Which applies to every function taking handlers as parameter - unless you need the full functionality of closures, you should always stick to passing static functions)

But i mentioned that only because you were so worried about efficiency, it's not like it'a a big deal when the functions passed are somewhat small.

@Ben,

I've never used AngularJS, and very likely never will.

What i don't really understand is why you need to use AngularJS to issue a simple AJAX call - AngularJS will call the jQuery ajax method without doubt anyway.

And what i don't really believe is that AngularJS kinda leaves you with no other (practicable) cchoice than abusing http status codes.

-

It seems we've taken a wrong turn somewhere, my example addressed your example, but that does not mean it is limited to Promise objects.

You can always call the error handler from your success handler when the parsed body data indicate an error in the business layer - that's a quick, simple and elegant solution that allows you to stick to the http standards.

However, you already made a decision for the current project, so i can only hope that you remember my objections here when you start a new project.

@wizzszz,

It's not just AngularJS: Backbone, Spine, and presumably pretty much every other frontend framework that's around these days is geared towards talking to REST interfaces using the full assortment of methods and status codes defined in the HTTP spec.

That's one reason it took me a minute to understand where you were coming from: eschewing PUT/DELETE and error statuses is a fairly eccentric approach, though of course you can make anything work.

@tps12,

every xmlhttprequest is "geared towards" the full set of request verbs/status codes.

If your particular frontend library is using well-documented (read:standardized) methods for other means, it is basically only mimicking http (and WebDAV).

SIP mimics http, too, but nobody would say that SIP is using http standards.

-

It's not like REST is a standard - basically REST is a set of DOs and DON'Ts only - some of them make sense, some don't, but i digress...

If your REST implementation resorts to a wild mixture of http and WebDAV it is nonetheless a "homegrown" protocol and does not follow standards... and i already said before that if you use your own "flavour" of http/WebDAV/custom extensions, you can invent status codes as you please - every ajax library/implementation can handle unknown error codes, as long as they are 3 digits and the first digit follows the standards.
(plus, you can still follow REST paradigms with standard http mechanisms)

But doing so renders this thread basically pointless, because in your own custom http/WebDAV/whatever mix it doesn't even make sense when you try to find a matching status code - simply invent your own.
(... like i said a number of times already, how about "456 CompuGlobalHyperMegaNet API indicates a big no-no: You have provided bad data" - doesn't that satisfy all the needs?)

-

Sorry if this is somehow my mistake because i didn't realize that we were talking about AngularJS only - the topic doesn't mention it, nor does the opening post.

@wizzszz,

"If your REST implementation resorts to a wild mixture of http and WebDAV it is nonetheless a "homegrown" protocol and does not follow standards..."

Totally agree, that's my original reason for not liking 422 in this case.

Our only difference is between "return 400 and an error description in the body" vs. "return 200 and an error description in the body."

Responding "200 OK" to a request that failed is clearly contrary to the plain language of the HTTP spec.

@tps12,

but 400 is defined as follows (taken from RFC2616):

400 Bad Request
The request could not be understood by the server due to malformed syntax.

This applies to "malformed syntax" of the header only, because the standard does not define anything related to the body data.

The request per se succeded (you've sent back data, mind you!), the body actually contains valid data, so i opted for a 200.

It's simply doesn't make much sense to send a body with a 400 response - all info available on the error is that it couldn't be understood anyway - displaying that to the user, as RFC2616 suggests is not useful either, because the user can hardly do anything about something the client app f**ked up.

-

My interpretation of RFC2616 is this (and that's how i've implemented it in the past):

400 is merely a reply to a junk request, a port number that has been mistakenly used for a different protocol or out-of-sync (meaning the server tries to i.e. interprete parts of the body data as request), and what's more important, the server should close the tcp connection directly after sending the response (header!), so the client won't i.e. send an out-of-sync request over that tcp connection again.

@wizzszz,

According to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4 all 4xx responses should describe the error in the response body.

@tps12,

I know, but it doesn't make sense to simply repeat 'Bad Request' in the http body.
All of the scenarios that i've listed above indicate that the client would not interprete the response correctly/at all anyway.

And it makes even less sense to do that when your body contains a JSON string, that is not really suited to be displayed to the end-user.

I.e. no browser displays the 401 Unauthorized to the user, they use a pop-up window for user credentials, so it's not like this basic rule applies to every 4xx code in the same way.

-

But that's only a side scenario...
You ignored my main point though, the "malformed syntax" part covers only the http header and not the body.

And imho, only the server should return a 400, not your scripts - if the request reaches your script, it must be syntactically valid, otherwise the server would have replied already.

@wizzszz,

Right, you don't repeat "Bad Request," you spell out the reason it was rejected. E.g., "the specified 'foo' must be between 7 and 12."

IMHO 5xx errors are the exclusive domain of the web server, and 4xx are fine to return from the application.

Interestingly, just noticed the spec calls out 404 as "commonly used" when no other error status applies: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5

But also note that the HTTPbis working group's draft describes 400 a little more broadly than the HTTP/1.1 spec: http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-22#page-57

I don't think we're ever going to agree on this. But I maintain that returning what is explicitly defined as a "success" response to a request that was not fulfilled is, at the very least, surprising.

@tps12,

I see where you're coming from, but yes, we're very likely not going to agree on this... at least not anytime soon... :P

I, for one, would be surprised to have a 400 response with body data in the expected format (instead of a placeholder, i.e. human readable text/html instead of the expected JSON data)

That would mean i mean i could end up with non-parseable junk - after all, the server could have replied with a 400, too, and it very likely won't use JSON in a (to me) meaningful format to do so - what are you going to do there, check Content-type prior to parsing?

A 200 response clearly contains something i can parse without having to worry that the server sent some placeholder text.

The server won't send a 200 on it's own when addressing a script, and that's one of my main reasons to prefer it over any other 4xx/5xx code!

Thanks for the suggestion, I've been wanting to come up with a standardized way to handle exceptions for a long time. I used the same thing you did, a flag at the top level of the response with a 200, but that felt really wrong.... We're going to use 422 (or if people really don't like using a WebDav extension, we'll fallback to 400). In that case, the body of the response is going to contain a list of errors with our application specific error code, maybe some exception specific data and a message to display to the user.

@wizzszz

Most public APIs use 4xx to indicate a business logic error, YouTube, Amazon, Twitter etc, so whether or not this fits the rigid definition of the standard, I would much rather go with what these large and technically savvy companies use/

Anyway, by your definition of HTTP status codes, 403 does not make any sense. 403 clearly states that the request was completely valid but the server is refusing to respond to it, for instance due to business rules...

@yolk,

the http request IS completely valid, because http does neither define nor perform checks of the http payload (that's solely a matter of the business logic).

And it's the business logic that refuses to complete the requested action.

If you re-read my posts, you will see that i have not been evangelizing the use of a 403 response code (that was someone else) - but suggested a 200 with an error code in the response body, nicely keeping the transport logic separated from the business logic.

Whether others use this or not is not authoritative, but, if you insist on indicating a business logic problem in a http status code, (unnecessarily intertwining business and transport logic), why not use a 4xx status code that doesn't have an accurately defined meaning yet?

There is a reason why i.e. IP layer, TCP layer and HTTP layer are NOT intertwined, and every coder should stick to that paradigm - and code custom logic ON TOP of the http layer (and not mix your custom stuff into a well-defined internet standard) - if everyone mixes his custom stuff into the http protocol, it will all end up in a horrible mess.

wizzszz,

1. HTTP is not a transport protocol it's an Application Layer TRANSFER protocol. Your comments suggest that using an extended error code is equivalent to hacking the IP protocol header which is absurd. intermediaries are designed to ignore codes they do not support (it would be impractical otherwise) and I bet they actually do meaningful work for only a few, and that's if your non-SSL users ever actually pass through one - THE SYSTEM WILL NOT BREAK DOWN (lol)

2. The HTTP error codes are indeed targeted to be interpreted by Stacks (first) but 99% percent of the time they get propagated to the user simply dressed up differently so the separation of concerns you argue for is pretty much semantics.

3. STANDARDS: Yes I agree, if you use 422 you create your "own http" (lol)
so what!? You seem like advocating for adhering to standards (a good thing) but at the same you totally ignore an equally important practice - ENGINEERING TRADE OFF.

The HTTP standard (great one) was written 15 years ago, under tight time constraints, and couldn't have anticipated all use-cases, some areas can be interpreted in different ways but that still OK - IT WORKS! because BASICS WORK.

What not ok is that you are prepared to introduce complexity in the form of custom entity formats, unnecessary confusion and incompatibility with standard logging/debugging tooling just to avoid a deviating a bit.

Furthermore, on a conceptual level, I see no problem in having HTTP return a generic Application Error code. HTTP does not live in vacuum and if you noticed most of its core operations are defined in terms of (application) business processes (posting to a bulletin board etc...)

4. WHERE IS THE DAMAGE? I'll tell you, the damage is when your building something strictly according to (your interpretation) of the standard, inventing your own business protocol envelopes, only to eventually put yourself (and others) in great pain when attempting to integrate with some other system that was built in a more pragmatic way - that's the problem.

Think proportional, the standards will eventually get tuned to align with what's on the ground.

-JooJoo

@JooJoo,

i really have a hard time to take feedback riddled with lols for serious.

Nonetheless i'll try to reply:
Engineering trade off is just a different term for "diluting the standards" here.

Nobody will ever stop you from creating an own (http-like) protocol with silly error codes and such - why does every reply completely ignore that?

Rape http as much as you like, add new request verbs, 8xx status codes, whatever you please (or need to get your stuff done) but don't go telling everyone it is still http.

Let me, once again, rephrase it, because it still appears to be a problem:
Basically, you have two options.
a) stick to the standard
b) create your own protocol (based on http)

If you option a) is your choice, there is no need to discuss this any further, because http does not define a status code for "bad payload data".

If you go for option b), use whatever status codes you like. Still no need to discuss which one is better, because it is YOUR protocol and does not claim to be compatible with any standard (even though it might be very similar to http) - you might want to have a look at SIP.

Standards are important. Imagine 1,000,000 coders and everyone is adding at discretion... which means there IS NO STANDARD anymore.
And with the standard you lose compatibility and interoperability.

Same goes for the layer model - there is a reason why we have a strict separation of protocol data and payload (next higher layer).
If you don't follow these conventions you clearly haven't understood the reason why they are separated.

To sum it up:
Don't mess with the standards.
If you need more than the standards lets you do, DERIVE.
Best practice is to stick to the payload with your custom needs entirely.

@Wizzszz,

Tester uses fiddler
Errors visible - not masked by 200
No need to develop custom parser
Less bugs
Less Money!!

If you can't see the value (in this particular context) and you truly believe that by using the WebDAV 422 Status Code Extension to HTTP/1.1 for distinguishing application level errors is "diluting the standards", "Rape http as much as you like" then please accept my apology for misinterpreting the discussion.

-JooJoo

@JooJoo,

"Tester uses fiddler" - that is your one big killer argument to tip the scales your way? *sigh*

And why should anyone need a custom parser?? Are you being serious??

If everybody may change standards at will, there will be (insert arbitrarily big number) of different "flavours" of that standard in a wink and that means there is no standard anymore.
Applies to every standard there is.

Why do you have such a hard time to accept something that is so painfully obvious?

Please, ask yourself what the purpose of having standards in the first place might be. A surprising insight is right in the line for your there...