How to inspect a Tensorflow .tfrecord file?

Question

I have a .tfrecord but I don't know how it is structured. How can I inspect the schema to understand what the .tfrecord file contains?

All Stackoverflow answers or documentation seem to assume I know the structure of the file.

reader = tf.TFRecordReader()
file = tf.train.string_input_producer("record.tfrecord")
_, serialized_record = reader.read(file)

...HOW TO INSPECT serialized_record...

Solution 1

Found it!

import tensorflow as tf

for example in tf.python_io.tf_record_iterator("data/foobar.tfrecord"):
    print(tf.train.Example.FromString(example))

You can also add:

from google.protobuf.json_format import MessageToJson
...
jsonMessage = MessageToJson(tf.train.Example.FromString(example))

Solution 2

Above solutions didn't work for me so for TF 2.0 use this:

import tensorflow as tf 
raw_dataset = tf.data.TFRecordDataset("path-to-file")

for raw_record in raw_dataset.take(1):
    example = tf.train.Example()
    example.ParseFromString(raw_record.numpy())
    print(example)

https://www.tensorflow.org/tutorials/load_data/tfrecord#reading_a_tfrecord_file_2

Solution 3

Improvement of the accepted solution :

import tensorflow as tf
import json

dataset = tf.data.TFRecordDataset("mydata.tfrecord")
for d in dataset:
    ex = tf.train.Example()
    ex.ParseFromString(d.numpy())
    m = json.loads(MessageToJson(ex))
    print(m['features']['feature'].keys())

In my case, I was running on TF2, and a single example was too big to fit on my screen, so I needed to use a dictionary to inspect the keys (the accepted solution return a full string).

Solution 4

If your .tftrecord contains SequenceExample, the accepted answer won't show you everything. You can use:

import tensorflow as tf

for example in tf.python_io.tf_record_iterator("data/foobar.tfrecord"):
    result = tf.train.SequenceExample.FromString(example)
    break
print(result)

This will show you the content of the first example.

Then you can also inspect individual Features using their keys:

result.context.feature["foo_key"]

And for FeatureLists:

result.feature_lists.feature_list["bar_key"]

Solution 5

If it's an option to install another Python package, tfrecord_lite is very convenient.

Example:

In [1]: import tensorflow as tf
   ...: from tfrecord_lite import decode_example
   ...:
   ...: it = tf.python_io.tf_record_iterator('nsynth-test.tfrecord')
   ...: decode_example(next(it))
   ...:
Out[1]:
{'audio': array([ 3.8138387e-06, -3.8721851e-06,  3.9331076e-06, ...,
        -3.6526076e-06,  3.7041993e-06, -3.7578957e-06], dtype=float32),
 'instrument': array([417], dtype=int64),
 'instrument_family': array([0], dtype=int64),
 'instrument_family_str': [b'bass'],
 'instrument_source': array([2], dtype=int64),
 'instrument_source_str': [b'synthetic'],
 'instrument_str': [b'bass_synthetic_033'],
 'note': array([149013], dtype=int64),
 'note_str': [b'bass_synthetic_033-100-100'],
 'pitch': array([100], dtype=int64),
 'qualities': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int64),
 'sample_rate': array([16000], dtype=int64),
 'velocity': array([100], dtype=int64)}

You can install it by pip install tfrecord_lite.

View more solutions

Comments

[PatriceG]

It seems that this solution doesn't show all the content of the file.

[Bob van Luijt]

Is that so? I didn’t have that issue

[Mahmoud Akl]

If I'm not mistaken, this loops through the entire TFRecord file to give you the contents of one single example. Is there a more efficient way to just read one example?

[Robert Lugg]

TFRecord files must be read sequentially from the start per documentation. I'm sure there is a way to read them randomly but maybe no supported standard.

[SantoshGupta7]

Answer should be changed to this one

[wvxvw]

broken link11111

[Vinson Ciawandy]

Is the MessageToJson comes from google protobuf?

[HyeonPhil Youn]

This is the one

[PascalIv]

I get: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 206: invalid start byte