If transactions over REST are unachievable, how can REST ever be really useful?

Question

When looking into REST one of the first things probably anybody will notice is there isn't any transaction semantics defined, some say this is implicitly against what REST is, whilst others say any attempt to do so would result in 'tainting' the REST systems.

But lets say for arguments sake that REST did become a popular 'api' choice, and every site in the universe started to expose restful entry points.

How exactly are these usable without transaction behavior (and I'm talking non compensating)? because it seems to me one of the benefits of REST is that it breaks down the components of data, this you would think opens them up to having smart clients compose data (and add and adjust such composed data) from multiple services. But if I cannot do my changes to this composition of data atomically and in isolation then using REST becomes useless.

As time goes on and the need for serious data exposition arrives, we're gonna want something that is: Simple (REST wins there), and supports transactional behavior so we can manipulate this data reliably.

Now, I've read a specific argument a couple of times in my research, and its related to how we're supposed to think about transactions in REST, and the example given is the shopping cart, where you implicitly have isolation because the cart is YOURS.

However I disagree with this argument, firstly, the isolation a shopping cart has is merely convenient, this isn't a transaction isolation.. what happens if I'm simultaneously doing an operation against my cart whilst some part of my application is reading data from it? I wouldn't expect the reading part of my application to see data that is 'still in transaction'.

Not to mention the fact that not all data changes have an implicit transaction model, transactions over multiple services definitely don't.

It seems to me that transactions need to happen, and need to happen in a way that enables the actual REST calls to be ignorant of the fact (adding to the rest payload being a big no no, but adding headers being OK).

I've read a few suggestions about how a transaction model can be created over REST, and some of the specifications being written seem to be very recent.

Are there any real thoughts about this? shouldn't there be something 'more' than REST so that REST's simplistic nature can be harnessed against solid data manipulation ('acid' transactions).

If not are we expected to really UP the ante, and tell service developers that if they want to interact in a pure data world they need to support something arguably monolithic like soap? or even worse try and build their own custom transaction support into something like REST, making each service non-standard and breaking the entire power of REST?

Thanks in advance for any thoughts.

---

Edit, added brief scenario:

Imagine a client form that handle creation of an Album, for convenience on that Album, rather than asking the user to give the uri for the artist resource, they can pick from a list of artists (most likely GET'd from the artists catalog).

For sake of usability the client can write the artist name in manually so they can create an artist 'inline'.. in the posting scenario, the client code understands this, and before sending the request to create the album, it firstly attempts to determine if the artist already exists, if so, gets the uri for that artist, otherwise creates the artist and gets the artists uri.

The client code then continues on to create the album, this is the smarter than usual client, it isn't sat right on top of REST and 'dumbly' posting, but instead has some interaction that handles the purer REST logic.

However in this scenario it would be nice to guarantee that the artist isn't created unless the album is, given the artist is created first.

This isn't as 'critical' as a transaction would imply, but it defines a set of work the client code would prefer to be happening as one operation (after all, it IS making this look like a single operation to the user).

The only guidance I've seen for this scenario is to have the client do compensating action in the event the album creation fails, and specifically call to delete the artist. But this seems problematic, because the client is assuming the artist was isolated, as unlikely as it may be, what happens if another client already 'saw' that artist, and assigns to it?

These are my concerns regarding doing data changes, and whilst there are certainly other gaps (who says the artist couldn't be just deleted at a later date anyway), those actions are NOT transparent (ie, the actions aren't something automated by the client, but something a user has specifically requested).

I hope that helps illuminate the topic some.

Solution 1

I am going to assume that when you talk about transactions you are talking about a distributed Two Phase Commit protocol.

If I understand correctly you are trying to understand how we could ever use REST to perform operations that span multiple systems if REST cannot support transactions across distinct REST requests. The problem is you are making a potentially flawed assumption that we should be using transactions to achieve consistency. What price are we paying for using them, and what alternatives exist?

Pat Helland who used to work for Amazon and is now at Microsoft, wrote a paper Life beyond Distributed Transactions. In the paper the Author makes the following statement:

Unfortunately, programmers striving to solve business goals like eCommerce, supply-chain-management, financial, and health-care applications increasingly need to think about scaling without distributed transactions. They do this because attempts to use distributed transactions are too fragile and perform poorly.

His paper explores alternative solutions to distributed transactions that do scale and perform well.

Maybe REST will be successful because it does not support transactions. Here is a quote from Roy Fielding, the guy who invented the term REST

If you find yourself in need of a distributed transaction protocol, then how can you possibly say that your architecture is based on REST? I simply cannot see how you can get from one situation (of using RESTful application state on the client and hypermedia to determine all state transitions) to the next situation of needing distributed agreement of transaction semantics wherein the client has to tell the server how to manage its own resources.

...for now I consider "rest transaction" to be an oxymoron.

This is from a message on the REST-discuss list from June 9th, 2009. I can't provide a link because Yahoo groups is useless.

Solution 2

If you want transactions in a ReST application, at the ReST API function, it's usually because you still have your techie webservice guy googles on.

Let's look at it another way.

Soap googles on: Open a transaction, create a customer record, create an order record, commit transaction.

ReST googles on: Ask server what to do. Server says "create customer resource", so POST /customers. Server says, "now create an order if you want", client creates the order by following the form.

In ReST, the application protocol is expressed in terms of resources being created and manipulated, not in terms of data that has transaction boundaries.

If you want to have a long-running transaction that spans all those operations, it's the server that decides to initiate it, not the client.

You can still implement long running transactions on the server side. If you attempt to want transactions from the client side, then you assume the client already knows all the operations it's going ot execute and the transaction boundaries that exist between those operations. If that's what your expectations are of the client, you've already given up the late binding, hypermedia-driven nature of a rest architecture.

So indeed, if you're not doing ReST and are trying to fit in RPC over http, you'll have a problem with not having transactions.

Solution 3

I think most actions that normally require transactions can be reworked to occur without them.

For example, the classic bank transfer. Suppose I want to move $100 from account A to B:

Begin Transaction
  /Debit  A, $100  
  /Credit B, $100
Commit Transaction

This could be reworked as:

 /Transfer  A, B, $100  

In this way, the server might do this in two steps, but the action from the client is a single, atomic operation that makes logical sense.

I'm sure there are lots of examples where it is more convenient to do an all or nothing set of operations (and I'm curious what people can come up with to address them), but I usually rework things in this way.

Solution 4

A REST operation can start a transaction, perform multiple database or other transactional operations, then commit or rollback - all within the scope of a transaction.

What REST cannot do is to be other than the root of a transaction. You can't start a transaction, then perform two REST operations and a database operation, then commit them all together. That's just like the situation of ASMX web services in .NET. They can be the root of a transaction, but that's all. They were successful for years, until WCF was introduced, supporting WS-Transactions. Even today and using WCF, most web service operations do not need to be transactional in the sense you're asking about.

Solution 5

This is an interesting topic. As you mentioned, SOAP has this sort of functionality already, but it took many years before SOAP matured to the point where people would consider doing real security and transactions with it. Before that, it was CORBA.

The various high-grade extensions to SOAP, including security and transactions, took a lot of work by a lot of people to get them right. This isn't going to happen to REST overnight, and it may never happen at all.

I feel that much of REST's currently popularity is something of a backlash against poorly-designed and over-complicated SOAP implementations, which is a shame, because SOAP doesn't have to be like that. As with anything else, it needs good design to make it work nicely.

View more solutions

Comments

[SerialSeb]

It's not "Kind of Rest". It's not restful. URIs are documented, no links are used between documents. It's better than say Twitter or Flickr, but it's far from being a hypermedia application, hence it cannot be restful.

[Rob]

Giggles at googles cause it's goggles.

[meandmycode]

I think the majority of people understand that within the 'operation' that rest is acting it would be transactional like, in the sense the changes to that entity should appear atomic and shouldn't leave the data in a broken state. This seems somewhat obvious and perhaps even outside of the scope of REST, I would expect a single operation to be successful or not. My question is specifically about getting transactions between operations and what peoples feedback about A) thoughts about doing this, B) attempts to make this happen they've seen.

[meandmycode]

Personally I don't mind soap, I think the interchange is completely for the machines so to what degree those messages are 'verbose' or not doesn't really bother me, and I think as an architecture it works fine. However, it does raise the bar on how sophisticated a client has to be, which is where REST comes in, of course a fully thought out transaction model for REST increases the bar, but I don't think anywhere near that of SOAP.

[meandmycode]

The problem with your example is it entirely server orientated, the client is what is important, in a world where the client wants to orientate changes to a domain and wants them to happen all or nothing the argument of 'you are thinking about it wrong' falls over. I definitely get the example, but I think it is perhaps too purist about what REST clients are about, REST as a simple way to communicate data changes is predominately what I think is important.

[meandmycode]

It's interesting, and its one of the examples of REST I've previously seen, but this imo taints the model of rest, the links become transaction aware, and this causes the linkage and therefore resources to become transaction aware, and this is definitely not the route I think transactional REST should take.

[meandmycode]

I like the argument about simplicity but I think in terms of a bigger picture for how REST could be used this would require clients to be very disciplined and almost pre-guarantee their operation should work (bar catastrophic failures, such as servers going down).

[Darrel Miller]

Any reason why you did not name the service? We are always looking for good REST apis to use as examples.

[Darrel Miller]

If you redefine the term REST to mean what is convenient to you, then you are not going to get very helpful answers. Believe it or not, REST has a very well defined meaning and within that context REST clients can only do what the server says they can do via hypermedia. In RESTful systems the server has way more control over the conversation than in RPC based systems. The client simply follows one of the options presented to it.

[meandmycode]

I'm not trying to redefine REST, but in terms of how real physics ui clients interact with them, I'm getting a better feeling for the reason why transactions in such a controlled fashion are not ideal, but really a unit of work is somewhat important.

[John Saunders]

@Aaronaught: he's talking about distributed transactions, where several database operations, web service calls, or other transactional operations, are all combined into a single transaction using three-phase commit.

[meandmycode]

Hi Darrel, thanks for posting a reply, I'm certainly not out to try and infer my own theory on how REST should work, this merely started with 'OK, so I use REST to do these scenarios.. what happens when I want to do this?' and these questions arise, transactions might not be the exact semantics I'm trying to describe, but I hope my scenario concerns are better illustrated in the edits I made to the original question. Thanks for the feedback.

[Aaronaught]

@John Saunders: That's not what it looks like to me. He seems to be talking about allowing clients to set transactional boundaries in order to wrap sequential web service calls in an atomic transaction; my answer was explaining why this makes no sense in a REST architecture or even a modern SOAP architecture. I even mentioned distributed transactions and why I don't think they apply to this question.

[John Saunders]

@Aaronaught: look at his third paragraph where he talks about clients being able to compose data from multiple sources. Also, consider that the multiple web service calls, in general, need not be on the same server or using the same service, and may, in general, be composed with other transactional operations. But not with REST.

[Rune]

@Darrel Miller: e-conomic.com

[Aaronaught]

@John Saunders: But then read his example, where he talks about adding an artist and then an album. That doesn't need to be a distributed transaction. Anyway, I've never heard of trying to compose a single resource across multiple services with REST; that seems to go against the very concept of REST.

[John Saunders]

@Aaronaught: the general question would be - if you had a set of SOAP services, database transactions, etc., could you replace one of the SOAP services with a REST service? The answer is "no". The question then becomes, "does this make REST useless?"

[John Saunders]

@Aaronaught: BTW, I don't think it makes REST useless. I think it makes it useless as part of a system of distributed transactions. That gives us one reason why REST shouldn't take over the world, at least not until there's no longer any use for distributed transactions across service boundaries.

[Aaronaught]

@John Saunders: I agree completely, SOAP is not interchangeable with REST. REST imposes certain architectural constraints, one of which is that an atomic operation must be expressed as a single resource and managed by one service at a time. I also think that most (obviously not all) public-facing APIs can be expressed this way. REST won't take over the world; it's just a different (and useful) method of designing web services.

[Darrel Miller]

I am unable to find any REST api information. The only api documentation that I can relates to using a client library that talks to the server using SOAP. Did I miss something?

[John Saunders]

And of the systems which require transactions, most will not require distributed transactions.

[John Saunders]

@Jacob: this doesn't work unless all parts of the transaction are being performed by the same service. What if part 1 is service 1, part 2 is service 2, part 3 is service 1 again, and part 4 is in the local database? Your technique can't solve that problem.

[meandmycode]

Says who? if you are writing an open api, how do you know who will consume your service? the topic probably isn't best described, I'm sure REST will always have its place, but when it comes to composition of services that weren't specifically intended to 'transact' together, REST is seemingly out of the question.. so is that it? is that ah well, that's the limitation, now go write a soap service like a good boy, or is there actually some work to try and layer some some of transact behaviour over REST? lots of headers can change the results of a REST request, so why not an (ie) transact header?

[John Saunders]

@meandmycode: just counting heads. It's not my fault if there aren't as many heads in the room as you'd like. How many SOAP web services are required to be part of a distributed transaction? I've yet to see my first one.

[meandmycode]

@John, I'm not sure what you mean at the start, but in terms of transactions, I wrote a scenario (which is common) in the initial post, I noted points about how without a transaction, the set of operations has to deal with rollbacks manually, and how these actions are still flawed. Having the option to say, well those things don't bother me is one thing, but some systems having such 'untidy' ends IS problematic, and performance overhead of a transaction is absolutely acceptable vs. 'corrupting' data.

[Jacob]

Sure it can. The cooperating services just need to communicate the URL, and the coordinating server needs to set up the rules for authorizing access to the transaction.

[John Saunders]

@meandmycode: what we seem to be arguing about is my suggestion that the set of systems that require distributed transactions is a subset of the set of systems that require transactions.

[SerialSeb]

Again, in ReST the server is responsible for declaring unit of work boundaries, not the client. If you want the client to know about such things, as opposed to cross teh boundaries by following where the server lead it, then you're better off putting more knowledge in the client than ReST allows. You'll have stronger coupling, you wont be ReST, but whatever it is you're trying to do you'll be able to do.

[SerialSeb]

We've already seen that transactions are not somehting useful for the client to know about, and as ReST is about the client-server interaction, there is no such thing as transactional ReST. I'm not sure I understand your comment on links becoming transaction aware? Links are used by a client to operate a state change to the whole application. If a transaction boundary or UoW boundary is crossed by the clietn following the link, it is done without the client knowledge.

[Admin]

Really, when you stop and think, serialseb is absolutely right. We get our terms mixed up when talking about web apps b/c we call client what is really just the service boundary of the server. Web pages are messages to a client (typically the browser) from the server.

[Admin]

@meandmycode I think what you are trying to describe is a workflow. Have a look at How to Get a Cup of Coffee: infoq.com/articles/webber-rest-workflow

[John Saunders]

@serialseb: why do you restrict yourself to clients which cannot understand transactions? What's so special about transactions that you cannot imagine a client platform understanding them?

[John Saunders]

that works fine if the services cooperate. But what if they're from different vendors who have never heard of each other?

[SerialSeb]

Because HTTP is the application protocol in ReST, and HTTP does not formulate transactions. Implementing explicit transaction support for the client to use (as opposed to trasaction boundaries being inherent to the state transitions) means you're reintroducing coupling where it didn't exist before. There's lots of litterature on ReST and transactions already that I won't repeat here.

[LCJ]

Could you please answer stackoverflow.com/questions/9483286/… ?

[Rajiv]

What happens when the transfer is complete at server but just before the response could be written on the wire, internet connection breaks? Client will assume that the transfer errored out. In this event, the customer is going to try the transfer again.

[Michael Haren]

I would suggest using optimistic concurrency in that case. E.g. when you send your transaction, include some value that indicates the state of the data as you know it. The server can check this "last updated" value when it does its atomic operation. If they match: continue. If not: error. When the client get this concurrency error it can refresh everything and see that the transaction already happened. Note that this problem is not exclusive to REST-- regular RDBMSs have it, too. Related: en.wikipedia.org/wiki/Two_Generals'_Problem

[inf3rno]

@JohnSaunders Just curiosity, how do you implement transactions by services "from different vendors who have never heard of each other" with other techniques?

[inf3rno]

@Rajiv This would be clearly a REST client issue, and has nothing to do with the service...

[inf3rno]

I can tell you, you will never achieve that, because you send your commands over the network, and you can always lost network connection... Therefore you will never implement any transaction solution over a network in which you cannot perfectly control the connections... Btw I don't think REST needs such a feature...

[John Saunders]

@inf3rno: sorry, but you're mistaken. Take a look at WS-ReliableMessaging, which takes network failures into account.

[inf3rno]

@JohnSaunders Yes, it is possible to check whether a message is received, but that is irrelevant. You cannot do anything to solve issues caused by persistent connection losses. So it is possible to have a situation when one of the services got a commit request and another a rollback request, and you cannot contact with them. If the client handles the transaction, it can even forget the error by a restart... So it's far from a robust system, and I don't think you can implement any ACID property on your transactions, but good luck!

[John Saunders]

@inf3rno: I guess you didn't read the Wikipedia article that I linked to. This has been sorted for well over a decade with non-SOAP solutions like HP's Reliable Transaction Router. Which product I was one of the developers of, so kindly stop telling me it can't be done. This is software that has been running banks, stock exchanges and railroads, so you seem to be disagreeing with reality.

[inf3rno]

It's hard to accept :-P Maybe I just don't understand the algorithms behind these solutions, can you tell me more about them? (links are enough)

[jhegedus]

Isn't this RPC instead of REST?

[AaronLS]

@jhegedus Everytime I've seen this debate, the solution proposed is to encapsulate the unit of work in a single rest resource, which is an RPC method with a really strange name. They take a verb phrase and turn it into a noun phrase. The resource semantics become strange, and what you have is an RPC style method with a resource style name. Same suggestion in other answers: "You can implement ... transactions on the server side." My opinion: strive for REST, leverage optimistic concurrency to minimize need for transactions, but occasionally you need a couple RPC style operations on an API.

[jhegedus]

Yes, this kinda makes sense, so the idea is to try to do everything as much CRUD/REST as possible, so basically part of what REST actually means is to keep try the API as simple as possible, i.e. CRUD + transactions limited to single entities/rows (using optimistic "locking"). OTH, Transfer in this example can be seen as an event. This resonates nicely with the idea of event sourcing.