How to decode/decipher Mozilla Firefox proprietary .jsonlz4 format? (sessionstore-backups/recovery.jsonlz4)

Question

I'm trying to get a handle on Mozilla Firefox's proprietary file format .jsonlz4, used, for example, for sessionstore-backups/recovery.jsonlz4, but to no avail.

How do I get back my data, specifically, some long text I've typed in some textareas of a crashed session? It's my data!

Solution 1

As per https://www.reddit.com/r/firefox/comments/2ps6wg/jsonlz4_bookmark_backups/, the following appears to work most reliably:

• in about:config, toggle the devtools.chrome.enabled setting from the default of false to a value of true

• open Scratchpad from within Firefox:

  • either with fn+Shift+F4 on a MacBook,
  • or Shift+F4,
  • or through the menu bar through Tools → Web Developer → Scratchpad

• in the menu bar within Scratchpad of Firefox, change Environment from Content to Browser (omitting this step would subsequently result in errors like Exception: ReferenceError: OS is not defined at the next step)

• use code like the following within the Scratchpad of Firefox:

  var file = "/Users/…/sessionstore-backups/recovery.baklz4";
  //OS.File.read(file, { compression: "lz4" }).then(bytes => 
  //  OS.File.writeAtomic(file + ".uncompressed", bytes));
  
  OS.File.read(file, { compression: "lz4" }).then(bytes => {
    OS.File.writeAtomic(file + ".uncompressed.stringify",
      JSON.stringify(JSON.parse(new TextDecoder().decode(bytes)), null, 1))
  });
  

  The final parameter to JSON.stringify handles how many spaces would be used at each line; putting 0 causes the whole thing to be printed on a single line, putting 1 splits the lines properly (putting 2 would create too much useless whitespace and increases the size of the file for little benefit)

• click the Run button

• run fgrep :textarea /Users/…/sessionstore-backups/recovery.baklz4.uncompressed.stringify from within the Terminal app

Solution 2

Unfortunately, due to a non-standard header, standard tools won't work. There's an open proposal to change that. Apparently the Mozilla header was devised before a standard lz4 frame format existed; it does wrap a standard lz4 block.

That said, the same bug report includes a few alternative methods. I'll list them briefly:

• Use the dejsonlz4 tool, which includes binary builds for Windows and should be easy to build on *nix
  • lz4json is a similar tool, but relies on an external liblz4 and is somewhat easier to build on *nix but harder on Windows (outside WSL)
• Use this fairly simple Python script: https://gist.github.com/Tblue/62ff47bef7f894e92ed5 (requires the lz4 package via pip or your package manager) -- the script appears to be python3 but is trivially adaptable to python2
• There is a webextension available that should be able to open these. NB: while source is available, I have not verified it, and the permissions it requests are a bit concerning (especially the response to concerns)
• In theory, you should be able to strip the first 8 bytes (e.g. with dd if=original.jsonlz4 of=stripped.lz4 bs=8 skip=1) and that should leave you with a valid lz4 block. Note that this is distinct from a lz4 frame. While most programming languages have libraries that can easily decode a block, finding a prebuilt tool to do so is more difficult, e.g. the liblz4-tool package only accepts the frame format.

Solution 3

I found the following methods to be working, testing on Ubuntu 20.04:

Method 1: Using the mozlz4 binary from GitHub:

Download the linux binary for mozlz4 from https://github.com/jusw85/mozlz4. Then run the following:

chmod u+x mozlz4-linux

./mozlz4-linux -x filename.jsonlz4

Method 2: Using the lz4json package from Ubuntu repos:

Ubuntu 20.04 repos have a package named lz4json. I haven't checked whether it is present on previous Ubuntu versions.

To install and use that, run

sudo apt install lz4 lz4json

lz4jsoncat ~/.mozilla/firefox/*default*/sessionstore-backups/recovery.jsonlz4

The above output will show a minified json. To make it readable, you can use the 'jq' json parser:

sudo apt install jq

# then pipe the output of the previous command through jq to make it readable:
lz4jsoncat ~/.mozilla/firefox/*default*/sessionstore-backups/recovery.jsonlz4 | jq

If you just want to see the list of URLs and the page titles, you can use this:

lz4jsoncat ~/.mozilla/firefox/*default*/sessionstore-backups/recovery.jsonlz4 \
  | jq '.["windows"] | .[0] | .["tabs"] | .[] | .["entries"] | .[0] | .url,.title' \
  | grep -v 'New Tab' | grep -v 'about:newtab' | sed 's/"http/\n"http/g'

Solution 4

I was able to extract the URLs from the {profile-dir}/sessionstore-backups/recovery.jsonlz4 file using the following free online tool designed expressly for this purpose:

https://www.jeffersonscher.com/ffu/scrounger.html

The same site offers a similar tool for decrypting jsonlz4 files from the {profile-dir}/bookmarkbackups directory.

Solution 5

On UNIX® and UNIX-like systems, like Mac OS X with MacPorts, FreeBSD, OpenBSD or NetBSD with pkgsrc, the following https://github.com/cnst/lz4json fork of lz4json could also be used to compile cleanly out of the box, e.g., on Mac OS X w/ MacPorts:

sudo port install lz4
git clone https://github.com/cnst/lz4json.git
cd lz4json
make
./lz4jsoncat ~/Library/Application\ Support/Firefox/Profiles/CHANGE\
THIS.default/sessionstore-backups/recovery.jsonlz4 \
| python -m json.tool | fgrep :textarea | more

View more solutions

Comments

[Ruslan]

I wouldn't call the format proprietary. Granted, it's custom, not used anywhere outside Mozilla projects, but since the whole of Firefox—including the relevant (de)compression code—is free and open source, this format shouldn't be called proprietary. (P.S. I'm not talking of the branding, which is licensed differently.)

[cnst]

@Ruslan, but it is in fact proprietary — just because it's OSS doesn't make it non-proprietary, as there are zero standard tools to look into the content of these files, whereas all other files, even Java's JAR format, can easily be managed with 100% standard non-proprietary tools that are available in ports/packages of every decent UNIX system. OTOH, it is completely non-trivial to actually get back your own data from these .jsonlz4 files.

[neverMind9]

JsonLZ4 was a bad idea.

[grantwparks]

I'd agree with "it's custom, not proprietary". Irrespective of it not being widely used or supported. Proprietary means owns exclusive rights to.

[lowtechsun]

Also just found this, works flawless.

[neverMind9]

Why was LZ4 necessary in first place? LZ4 is an absolutely moronic idea.

[cnst]

BTW, here's the lz4json that compiles cleanly on UNIX®, Mac OS X, MacOS, FreeBSD, OpenBSD and NetBSD — github.com/cnst/lz4json.

[mivk]

This doesn't seem to work in FF 72 on Linux. Scratchpad has been removed, but the Console now supports multi-line mode. However, the OS.File.read(... line gives : "ReferenceError: OS is not defined".

[cnst]

@mivk that error is mentioned above — happens due to wrong environment; is there no way to chance the environment?

[mivk]

I have not found this "Environment" setting or anything similar in FF 72.0.1.

[Awaaaaarghhh]

Python script mozlz4a.py: Could not compress/decompress file 'bookmarks.jsonlz4': module 'lz4' has no attribute 'decompress (Python 3.8.3, conda 4.9.0)

[jrw32982]

Good explanation. It would be nice to have some kind of reference as to how to add back the missing lz4 frame (after stripping the leading 8 bytes) so as to be able to use the commonly available lz4 utility to do the decompression. Unfortunately, it seems this must be somewhat difficult to do or someone would already have done it. Something like { echo 0x184D2204 | xxd -r ; tail -c+9 previous.jsonlz4 ; } | lz4 -dc (which doesn't work).

[tvdo]

@jrw32982 At a minimum you'd also need a frame descriptor. But you'll run into another problem: the standard lz4 frame format has rather limited block sizes (4 MB max uncompressed size in the current version). Seems it's intended to support one frame with many smaller blocks, rather than one big block as the Mozilla format uses.

[Till Kolditz]

Hi this still works, but instead of the Scratchpad, you need to use the Web Developer Browser Console: Menu --> Web Developer --> Browser Console Then hit "Ctrl + B" or click the icon to enable the multi-line editor and then the script above still works. Verified with firefox 87 on Ubuntu 21.04. See also developer.mozilla.org/en-US/docs/Tools/Deprecated_tools

[Ben C]

It works on newer versions as @TillKolditz says, if you open the Browser Console directly from the menu or keyboard shortcut. But it doesn't work if you open Tools --> Browser Tools --> Web Developer Tools from the menu (or Ctrl+Shift+I) and click the Console tab at the top: then it gives the error "Uncaught ReferenceError: OS is not defined".

[facehugger]

Crashes when trying load huge (10000kB+) files

[Martian2020]

I used repos, it works, have not checked completeness (that all tabs are there), but output of that long pipe looks promising: "https://www.quora.com/How-does-multi-threading-in-a-uni-pro‌​cessor-work" "(1) How does multi-threading in a uni-processor work? - Quora"

[Martian2020]

Would be great to expand the answer adding how to add links back to Firefox.