Google Drive: maximum number of files in a directory

Question

Is there any maximum number of files that can reside in a Google Drive folder? Are there performance hits when a lot of files (for instance, a million of them) stay in the same folder?

From what I understand (mostly on reading how the API works), Google Drive has no real concept of "folder". Folders are just represented by a specific kind of file and folder belonging is just described within the files' metadata, but files by themselves are just a long unstructured list of blobs with metadata. This would suggest that having a big number of files in the same directory should not be a big problem.

But I would like to have more expert opinions on the matter.

(of course folders with a lot of files are going to hurt if a synchronize with my disk; but I am just going to query them with the API)

EDIT I am not going to use the web UI. The types of queries that I am going to perform are to post a file in this giant folder and retrieve a file given its name. Basically this means I am using this folder as a hash table. So I guess the actual question is: if a make a query like

'big_folder_id' in parents and title = 'some_key'

(assuming that there is just one file named some_key in the folder), is the performance impact associated with the fact that in the folder identified by big_folder_id there are a lot of files going to be bearable?

Solution 1

Performance hit will be on the UI side. Scrolling to the bottom of a long files list will take a very long time. Also, in a folder's web view (i.e. if you share it with 'anyone with a link can view' permission) only the first 500 files will be displayed, with no way to see the rest of the files.

From API access perspective - it depends on what you are doing with the API. For example, if you try to get a list of files in a folder with a lot of files in it, you will likely run into script execution timeout (6 minutes max).

Solution 2

I think Google recently started limiting this. They now have a 500k item limit per folder (root folder is exempt from this limit): https://developers.google.com/drive/api/v3/handle-errors#resolve_a_403_error_number_of_items_in_folder

I designed my system thinking there's no limit, and my logs indicate they started enforcing my account at 2020-06-15T17:13:37.020232715Z. At the time I had reached 3 232 458 files in a single folder. Limit is at 500k, so this is further evidence that this quota was retroactively added and enforcement was started without warning which brought my system down.

More proof is that this error code (numChildrenInNonRootLimitExceeded) started existing in this document somewhere between 2020-04-12 and 2020-06-11:

https://web.archive.org/web/20200412153122/https://developers.google.com/drive/api/v3/handle-errors => not present

https://web.archive.org/web/20200611105741/https://developers.google.com/drive/api/v3/handle-errors => present

Also, web search for that error code turn up very few links. The only non-Google result I find is dated 2020-06-11: https://scrapbox.io/ci7lus/Error:_The_limit_for_this_folder's_number_of_children_(files_and_folders)_has_been_exceeded.#5eeb086bae0f140000d5c509

Comments

[Giovanni Mascellani]

I never did systematic benchmarking (mine is just a hobby project). However I did not observe any performance degradation (for queries of the type above, always returning one or zero results) when passing from a few files to something around 250k (for around 10 TB total storage).

[Saeed Neamati]

Your hobby project contains 10 TB of data? What a hobby my friend ;)

[Giovanni Mascellani]

Thanks for the answer. I edited the question to make it more specific. If you can add the details, I will accept the answer.