What's in my ZFS ARC and L2ARC caches?

Question

I have a simple ZFS setup at home, four disks mirrored, 8GB ram and 120GB Intel X25-M SSD for L2ARC. I ran zpool add poolname cache cXtXdX then generated load and eagerly tracked the warming process by running zpool iostat -v mypool. A few hours later, the L2ARC is full, but what exactly has been cached?

Can I peek in and see what's in my ARC/L2ARC? I understand that certain blocks would map to multiple filesystems (due to snapshots, cloning or dedup) but I'd still like to sample the statistics like these:

• Type of block cached: metadata. iSCSI, data. etc.
• Age: How long has each block been in the L2ARC
• Which files: directories that have some of their contents cached
• Filesystem breakdown: GB cached per filesystem
• Location of non-volatile copies of blocks (which drive/vdev)

I'd imagine zdb and/or dtrace would be the tools de jour, but don't really know where to start.

Solution 1

Here's a script that's been updated to reflect the data in L2ARC.

See: http://blog.harschsystems.com/2010/09/08/arcstat-pl-updated-for-l2arc-statistics/

Source available here: https://github.com/mharsch/arcstat

Solution 2

I would probably argue that you should care not about what's in the cache, but whether the most effective things have been cached. Ie, is the ARC cache being used to its maximum efficiency?

I run a small-scale (9TB) ZFS implementation at work and I find Ben Rockwood's tool arc_summary.pl, and Sun's arcstat.pl, both detailed in this cuddletech blog post, to be immensely useful.

I know it doesn't answer your question directly about what exactly is in the ARC cache, but these should give you some solid data about how your cache is being used, and whether you're in need of more memory/SSD space.

Comments

[user9517]

There is a excellent blog entry by Brendan Gregg that answers some of your points.

[ewwhite]

Check the script linked in my answer.

[Marco]

The blog entry by Brendan Gregg has moved: ZFS L2ARC

[Marcel Waldvogel]

None of the links work anymore. However, they seem to be available at github.com/benr/solaris_tools/blob/master/arc_summary.pl and github.com/mharsch/arcstat/blob/master/arcstat.pl, respectively.