Azure DocumentDb Storage Limits - what exactly do they mean?

Question

Azure DocumentDb seems to have some strange storage limits:

--------------------------------------------------------------------
| Entity                           | Default quota (Standard Offer)|
-------------------------------------------------------------------|
| Document storage per collection  | 250 GB*                       |
-------------------------------------------------------------------|
| Throughput per collection,       | 250,000 RU/s*                 |
| measured in Request Units per    |                               |
| second per collection            |                               |
-------------------------------------------------------------------|
| Request document size            | 512 KB*                       |
-------------------------------------------------------------------|
| Response document size           | 1MB                           |
-------------------------------------------------------------------|

Request document size - Does that mean that the size of the json payload that gets sent for storage in the documentdb can't be bigger than 512KB without a request to support?

Also, if a request to support is made for a larger request document size, what are the limits on that? Can I ask for 1MB? 2MB?

Response document size - Does that mean that the size of the json response can't exceed 1MB? That hardly seems useful for anything but the simplest entities.

Solution

Update: As of December 2016, DocumentDB has increased the default document size quota to 2mb and page size quota to 4mb

DocumentDB team here. It looks like the wording on our "limits" page can be improved... In the meantime, allow me to try to clarify:

Document Size Quota (default: 2MB per document)

The default document size quota is 2MB per document. In other words, you are able to store <=2MB worth of JSON in each and every record - without having to send in a ticket. Reference: https://docs.microsoft.com/en-us/azure/cosmos-db/documentdb-resources#documents

We are actively working on some long term improvements to increase this our default quota broadly. In the meantime, you can submit a support request to enable a preview of higher document size quotas on your DocumentDB account today.

Also worth noting - most documents (generally speaking) that are >2MB involve large unbounded arrays - in which it would better to abstract out and model each array element as a separate document.

Response Size Quota per Page of Results (default: 4MB per page)

To be clear - DocumentDB allows query results of any arbitrary size (e.g. 1 KB, 1 GB, 1 TB, etc).

For large query results - DocumentDB will paginate results, and each page will be limited to the response size quota (by default, 4MB per page).

Why paginated query results is a really cool feature:

Have you ever run a query in another data store (other than DocumentDB), and it seems to take forever... If a query takes an hour to complete - how can you tell if it will take minutes vs hours vs is progress is being made at all?

DocumentDB resolves this by splitting query results in to a set of pages that you can iterate through. The results are paginated on a number of criteria:

• Max Response Size per Page: 4 mb
• Max Time Limit per Page: 5 sec
• If you exhaust provisioned throughput w/ a sizable buffer (depends on how much throughput you've provisioned)

This means you are able to stream and take advantage of query results right away, as well as control the rate you resume execution of queries.

The following snippet illustrates how to retrieve query results 1 page at a time using the C# .NET SDK:

var query = client.CreateDocumentQuery<Family>(collectionUri, "SELECT * FROM Families", options).AsDocumentQuery();
while (query.HasMoreResults)
{
    foreach (Family family in await query.ExecuteNextAsync())
    {
        families.Add(family);
    }
}

The following snippet illustrates how to conventionally have the client SDK iterate through paginates results and materialize the entire query result on your behalf:

var families= client.CreateDocumentQuery<Family>(collectionUri, "SELECT * FROM Families", options).toList();

Comments

[richard]

Thanks for the detailed explanation. Does this mean that the max document request and response is 1MB even with support intervention? Any plans for larger document sizes?

[Andrew Liu]

We can increase req/resp size even higher depending on the details for specific scenarios. And yes, we are working to support much larger document sizes broadly (by default) :)

[LowlyDBA - John M]

@AndrewLiu I had a crazy time finding the current document size limit...is this posted anywhere that I can keep an eye on and refer back to?

[Chris Doyle]

@AndrewLiu With the release/rename to CosmosDB, has any of the above changed?

[Bola]

@AndrewLiu we also need to increase default limit (at least temporary till we refactor the way data is saved). Does this increase documentDb cost as well?