Storing Image Data for offline web application (client-side storage database)

Question

I have an offline web application using appcaching. I need to provide it about 10MB - 20MB of data that it will save (client-side) consisting mainly of PNG image files. The operation is as follows:

1. Web application downloads and installs in appcache (uses manifest)
2. Web app requests from server PNG data files (how? - see alternatives below)
3. Occasionally web app resyncs with server, and does small partial updates/deletes/additions to PNG database
4. FYI: Server is a JSON REST server, that can place files in wwwroot for pickup

Here is my current analysis of client-based "databases" that handle binary blob storage

SEE UPDATE at Bottom

• AppCache (via manifest add all the PNG and then update on demand)

  • CON: any change of a PNG database item will mean complete download of all items in manifest (Really bad news!)

• WebStorage

  • CON: Designed for JSON storage
  • CON: can only store blobs via base64 encoding (probably fatal flaw due to cost of de-encoding)
  • CON: Hard limit of 5MB for webStorage http://htmlui.com/blog/2011-08-23-5-obscure-facts-about-html5-localstorage.html

• PhoneGap & SQLLite

  • CON: Sponsor will reject it as a native app requiring certification

• ZIP file

  • Server creates a zip file, places it in wwwroot, and notifies client
  • user has to manually unzip (At least that is how I see it) and save to client file system
  • Web app uses FileSystem API to reference files
  • CON: ZIP might be too large (zip64?), long time to create
  • CON: Not sure if FileSystem API can always read out of the sandbox (I think so)

• USB or SD card (back to the stone age....)

  • The user will be local to the server before going offline
  • So we could have him insert a SD card, let the server fill it with PNG files
  • Then the user will plug it into the laptop, tablet
  • Web app will use FileSystem API to read the files
  • CON: Not sure if FileSystem API can always read out of the sandbox (I think so)

• WebSQL

  • CON: w3c has abandoned it (pretty bad)
  • I might consider a Javascript wrapper that uses IndexedDB and WebSQL as a fall-back

• FileSystem API

  • Chrome supports read/write of blobs
  • CON: not clear about IE and FireFox (IE10, has non-standard msSave)
  • caniuse.com reports IOS and Android support (but again, is this just r/w of JSON, or does it include the full blob API for writing?
  • CON: FireFox folks dislike FileSystem API & not clear if they are supporting saving blobs: https://hacks.mozilla.org/2012/07/why-no-filesystem-api-in-firefox/
  • PRO: Much faster than IndexedDB for blobs according to jsperf http://jsperf.com/indexeddb-vs-localstorage/15 (page 2)

• IndexedDB

  • Good support in IE10, FireFox (save, read blobs)
  • Good speed and easier management than a file system (deletes, updates)
  • PRO: see speed tests: http://jsperf.com/indexeddb-vs-localstorage/15
  • See this article on storing and display of images in IndexedDB: https://hacks.mozilla.org/2012/02/storing-images-and-files-in-indexeddb/
  • CON: I confirmed that Chrome does not yet support blob writing (current bug, but not clear when it will be fixed)
  • UPDATE: A June 2014 blogpost suggests Chrome now supports blobs in IndexedDB
  • UPDATE: This caniuse/indexeddb confirms: "Chrome 36 and below did not support Blob objects as indexedDB values."; suggesting >Chrome36 supports Blob objects.

• LawnChair JavaScript wrapper http://brian.io/lawnchair/

  • PRO: very clean wrapper for IndexedDB, WebSQL or whatever database you have (think polyfill)
  • CON: cannot store binary blobs, only data:uri (base64 encoding) (probably fatal flaw due to cost of de-encoding)

• IndexedDB JQUERY polyFill https://github.com/axemclion/jquery-indexeddb

  • Parashuram has writtent a nice JQUERY wrapper for the raw IndexedDB interface
  • PRO: greatly simplifies using IndexedDB, I was hoping to add a shim/polyfill for Chrome FileSystemAPI
  • CON: It should handle blobs, but I was unable to get it to work

• idb.filesystem.js http://ericbidelman.tumblr.com/post/21649963613/idb-filesystem-js-bringing-the-html5-filesystem-api

  • Eric Bidelman @ Google has written a well tested PolyFill the FileSystem API that uses Indexed DB as a fall back
  • PRO: FileSystem API is well suited for storing blobs
  • PRO: works great on FireFox and Chrome
    • PRO: great for synchronizing with cloud based CouchDB
  • CON: no clear why, but it is not working on IE10

• PouchDB JavaScript Library http://pouchdb.com/

  • great for syncing a CouchDB with a local DB (uses either WebSQL or IndexedDB (not my problem though)
  • CON: NO CONS, PouchDB now supports binary blobs for all recent browsers (IE, Chrome, Firefox, Chrome on mobile, etc.) as well as many older browsers. That was not the case when I first did this post.

NOTE: to see a data:uri encoding of PNG I created an example at: http://jsbin.com/ivefak/1/edit

Desired/Usefull/Uneeded Features

• No native (EXE, PhoneGap, ObjectiveC, etc) app on client (pure web application)
• Only needs to run on latest Chrome, FireFox, IE10 for laptops
• Strongly want same solution for Android Tablet (IOS would be nice too) but only need one browser to work (FF, Chrome, etc.)
• Fast initial DB population
• REQUIREMENT: Very fast retrieval of images by web application from storage (DB, file)
• Not meant for consumers. We can restrict browsers, and ask user to do special setup & tasks, but let's minimize that

IndexedDB Implementations

• There is an excellent article on how IE,FF,and Chrome internally implement this at: http://www.aaron-powell.com/web/indexeddb-storage
• In short:
  • IE uses the same database format as Exchange and Active Directory for IndexedDB
  • Firefox is using SQLite so are kind of implementing a NoSQL database in to SQL database
  • Chrome (and WebKit) are using a Key/ Value store which has heritage in BigTable

My Current Results

• I chose to use an IndexedDB approach (and polyfill with FileSystemAPI for Chrome until they ship blob support)
• For fetching the tiles, I had a dilemna since the JQUERY folks are kvetching about adding this to AJAX
• I went with XHR2-Lib by Phil Parsons, which is very much like JQUERY .ajax() https://github.com/p-m-p/xhr2-lib
• Performance for 100MB downloads (IE10 4s, Chrome 6s, FireFox 7s).
• I could not get any of the IndexedDB wrappers to work for blobs (lawnchair, PouchDB, jquery-indexeddb, etc.)
• I rolled my own wrapper, and performance is (IE10 2s, Chrome 3s, FireFox 10s)
• With FF, I assume we are seeing the performance issue of using a relational DB (sqllite) for a non-sql storage
• NOTE, Chrome has outstanding debug tools (developer tab, resources) for inspecting the state of the IndexedDB.

FINAL Results posted below as answer

Update

PouchDB now supports binary blobs for all recent browsers (IE, Chrome, Firefox, Chrome on mobile, etc.) as well as many older browsers. That was not the case when I first did this post.

Solution 1

Results Offline blob cache for PNG slippy maps

Testing

• 171 PNG files (total of 3.2MB)
• Platforms tested: Chrome v24, FireFox 18, IE 10
• Should also work with Chrome & FF for Android

Fetch from web server

• using XHR2 (supported on almost all browsers) for blob download from web server
• I went with XHR2-Lib by Phil Parsons, which is very much like JQUERY .ajax()
  • https://github.com/p-m-p/xhr2-lib

Storage

• IndexedDB for IE and FireFox
• Chrome: Polyfill (blob stored using FileSystem API, reference kept in IndexedDB) polyfill
• A Must read article on "How the browsers store IndexedDB data"
  • http://www.aaron-powell.com/web/indexeddb-storage
• Note: FireFox uses SQLlite for the NOSQL IndexedDB. That might be the reason for the slow performance. (blobs stored separately)
• Note: Microsoft IE uses the extensible storage engine:
  • http://en.wikipedia.org/wiki/Extensible_Storage_Engine
• Note: Chrome uses LevelDB http://code.google.com/p/leveldb/

Display

• I am using Leaflet http://leafletjs.com/ to show the map tiles
• I used the functional tile layer plugin by Ishmael Smyrnow for fetching the tile layer from the DB
  • https://github.com/ismyrnow/Leaflet.functionaltilelayer
• I compared the DB-based tiles layer with a purely local (localhost://) storage
• There is no noticeable difference in performance! between using IndexedDB and local files!

Results

• Chrome: Fetch (6.551s), Store (8.247s), Total Elapsed Time: (13.714s)
• FireFox: Fetch (0.422s), Store (31.519s), Total Elapsed Time: (32.836s)
• IE 10: Fetch (0.668s), Store: (0.896s), Total Elapsed Time: (3.758s)

Solution 2

For your requirements I suggest that developing a new polyfill based on two others: FileSystem API to IndexedDB and IndexedDB to WebSQL — is the best option.

The former one will enable support for storing blobs in Chrome (FileSystem API) and Firefox (IndexedDB), while the latter should provide the support for Android and iOS (WebSQL). What is needed is just making these polyfills work together, and I suppose it's not hard.

NB: Since I couldn't find any information on the web about this, you should test if storing blobs using the WebSQL polyfill will work on iOS and Android. It looks like it should work though:

var sql = ["CREATE TABLE", idbModules.util.quote(storeName), "(key BLOB", createOptions.autoIncrement ? ", inc INTEGER PRIMARY KEY AUTOINCREMENT" : "PRIMARY KEY", ", value BLOB)"].join(" ")

Source

Solution 3

I have map caching examples(open example, discover regions and zooms, switch offline and discovered regions will availaible).

There are map.js - map layer for offline tiles, storage.js - storage implementation based on IndexedDb and WebSQL (but this just test implementation with poor performance).

• For site files (html, css, js and etc.) I prefer use application cache.
• For storage I prefer use Indexed DB (support blob), Web SQL (only base64), FileWriter (support blob, but only chrome). Frankly storage is big issue for this. You need the fastest key value solution that will mix them all. I think is good decision to use exist solution.
• For fetching I used canvas with CORS. But I thinking about WebWorkers and XHR2 and this can be better instead canvas because canvas have some troubles with CORS in different browsers and other (for example this title was stored bad in opera).

Additional information about sizes for 2 billion city (Minsk):

• Zoom - 9, tiles - 2, size - 52 kb, with previous - 52 kb;
• Zoom - 10, tiles - 3, size - 72 kb, with previous - 124 kb;
• Zoom - 11, tiles - 7, size - 204 kb, with previous - 328 kb;
• Zoom - 12, tiles - 17, size - 348 kb, with previous - 676 kb;
• Zoom - 13, tiles - 48, size - 820 kb, with previous - 1.5 mb;
• Zoom - 14, tiles - 158, size - 2.2 mb, with previous - 3.7 mb;
• Zoom - 15, tiles - 586, size - 5.5 mb, with previous - 9.3 mb;
• Zoom - 16, tiles - 2264, size - 15 mb, with previous - 24.3 mb;

Solution 4

A few years back (not exactly the stone age), I was using a signed java applet that would query its server for syncing/updating requirements, download appropriate files from the server and save them on the user's filesystem (not a database). That solution might work for you, although you will need someone to write the applet and sign it. For database solutions, such an applet can use the jdbc available for most databases using localhost on a suitable port (e.g., 3306 for MySQL). I believe the applet tag is deprecated in Html5 but it still works. No experience on Android tablets, so can't comment on that part.

View more solutions

Comments

[mpm]

webstorage doesnt support json but strings , so you can base64 encode your imagez and serve them back as dataurls.

[Dr.YSG]

Ok, but probably not optimal (or within quota) for 20MB of imagery, that actually are slippy map tiles, that need to be fetched & displayed rapidly by a LEAFLET map application as you zoom and pan.

[Bogdan Kulynych]

The research you've done is quite helpful.

[mpm]

my point is you dont need to deal with binary blobs if you are using png images.

[Dr.YSG]

You are right, would you mind if I update the document to reflect your input?

[Kyaw Tun]

Also check out my library bitbucket.org/ytkyaw/ydn-db/wiki/Home

[Dr.YSG]

I updated the post with new information I found out about lawnchair, IDB polyfill, and pouch, as well as the fact that the Chrome folks are definitely going to add blob support to Chrome. I am now doing some performance testing of FileSystemAPI versus IDB.

[Peppelorum]

It seems that Lawnchair indeed supports Blob, at least in Firefox. It doesn't work in Chrome due to that it's not implemented yet (issue for this code.google.com/p/chromium/issues/detail?id=108012)

[Dr.YSG]

Are you sure these are not just Base64, I still have an open issue in the github about blob support: github.com/brianleroux/lawnchair/issues/141

[Dr.YSG]

Right now I am using PouchDB, which has a shim for Chrome.

[Dr.YSG]

I started programming in FORTRAN in 1968 using a card-punch. So stone-age solutions are not new to me.

[Dr.YSG]

I am leaning towards your suggestion, but I am waiting to hear from others. I don't have handy android, but it would be good to create a jsBin or jsFiddle and see what works on the Android.

[Kyaw Tun]

These two blob are different. Sqlite blob is arraybuffer in javascript, while js blob has no equivalent in sqlite. Blob cannot convert to arraybuffer, although it can be structurally cloned.

[Dr.YSG]

I assume those are JPG tiles in slippy EGPS3857 format right? since I am using leaflet and doing raster ovelays I had to go with PNG. also check out my demo of using PouchDB (which uses IDB underneath). stackoverflow.com/questions/16721312/…

[Dr.YSG]

Oh yes, you are caching on the fly, but do you know where I can go to get a pre-built OSM map (world wide) down to zoom 10 or 11 or 12? We would keep this on our offline server.

[tbicr]

No, used PNG with default projection (EGPS:3857) but no matter JPEG or PNG because it used by img tag or canvas. With my example you can just preload tiles if you know tiles keys (storage.add('x_y_z', 'data:image/png;base64,...') for each stored tiles), but you always can get them if know just bounds (polygon) and zooms.

[Dr.YSG]

I want to make sure we don't have a language problem. Do you any place that we can get a world-wide OSM set of slippy tiles (PNG or JPG) to zoom level 10?

[tbicr]

You can get tiles form tile.osm.org (mapnik renderer). For example http://tile.openstreetmap.org/10/590/329.png (zoom/x/y.png). This tiles have Access-Control-Allow-Origin: * header so you can get them by ajax or get data uri (base64) by canvas. You already can download tiles with your manifest.json {id: 0-0-0}, but you must sure that have right zoom, x, y sequence.

[Karussell]

Nice effort you both have done for offline maps. Now we are thinking and trying your and similar techniques for offline routing. See here: stackoverflow.com/q/23296295/194609