Can we run Linux in something faster than RAM?

Question

This is perhaps a silly question, and may be the result of a misunderstanding. I'm studying CPU's right now, and memory in particular. I was just reading about how much faster SRAM is than DRAM but more expensive. SRAM is very expensive: I shopped for a bit and found a battery powered SRAM card with 16 MB for around $400.

Recently a friend mentioned he has been running Puppy Linux in RAM, and that it is fast. I noticed, though, that tiny core Linux can be even smaller... as small as 8 MB! This got me thinking: can we run Linux in SRAM? Is that question even well-formed?

Googling this question proved ineffective, but it raised yet more questions. Could one run Linux in L3 Cache? Intel Core i7 can have an L3 Cache big enough to fit the 8MB... but am I making a categorical error? What is the difference between this and 'embedded' Linux?

That's the question: can we run Linux in SRAM or L3 Cache? Is there anything faster? How fast can we Linux!?

Solution 1

Linux, or any other OS does not know how the RAM works. As long as the memory controller is properly configured (e.g. refresh rates set for non-SRAM) then the OS does not care is it runs on plain dynamic memory (plain RAM), fast page mode RAM (FP RAM, from the C64-ish times), Extended data out mode RAM (EDO) , synchronious RAM (SDRAM), any of the double data rate SDRAMS (DDR 1/2/3) whatever.

All of those support reading and writing from random places. All will work.

Now cache is a bit different. You do not have to write to it for the contents to change. That will get in the way. Still, it is somewhat usable. I know that coreboot uses the cache as a sort of memory during boot, before the memory controller is properly configured. (For the details, check out the videos from the coreboot talks during FOSDEM 2011).

So in theory yes, you could use it.

BUT: For practical tasks a system with 1 GB 'regular' 'medium speed' memory will perform a lot better than with only a few MB super fast memory. Which means you have three choices:

1. Build things the normal 'cheap' way. If you need more speed add a few dozen extra computers (all with 'slow' memory)
2. Or build a single computer with a dozen times the price and significantly less then a dozen times the performance.

Except in very rare cases the last is not sensible.

Solution 2

Yes, you can, and this is in fact how it's already done, automatically. The most frequently used parts of RAM are copied in cache. If your total RAM usage is smaller than your cache size (as you suppose), the existing caching mechanism will have copied everything in RAM.

The only time when the cache would then be copied back to normal RAM is when the PC goes to S3 sleep mode. This is necessary because the caches are powered down in S3 mode.

Solution 3

Many CPUs allow the Cache to be used as RAM. For example, most newer x86 CPUs can configure certain regions as writeback with no-fill on reads via MTRRs. This can be used to designate a region of the address space as - effectively - cache-as-ram.

Whether this would be beneficial is another question - it would lock the kernel into RAM, but at the same time would reduce the effective size of the cache. There might also be side effects (such as having to disable caching for the rest of the system) that would make this far slower.

Solution 4

In x86 there's a thing called CAR (Cache as RAM) which allows you to write "bare-metal" code such as bootloaders or BIOS routines in a high-level language like C instead of assembly. Many other architectures may have the same feature

So it's possible for some OS to run entirely in cache. Imagine having a Ryzen™ Threadripper™ 3990X with total 292 MB of cache. That's more than enough to run even some modern tiny Linux. I guess you'll need significant changes to the Linux kernel to make it work, but it's definitely possible

For more information read

• A Framework for Using Processor Cache as RAM (CAR)
• CAR: Using Cache as RAM in LinuxBIOS
• Can a CPU function with nothing more than a power supply and a ROM, using only the internal cache as RAM?
• Cache-as-Ram (no fill mode) Executable Code
• What use is the INVD instruction?
• How Does BIOS initialize DRAM?

Solution 5

"can we run linux in L3 Cache?"

No, this is not possible because the cache memory is not directly/linearly addressed.
Due to the way the cache memory is designed, the CPU Program Counter (IP) registry cannot point to a location in the cache memory.

A CPU cache have it's own "associativity" and this associativity define the way the "normal" memory is "mapped" to the cache memory. This feature of the cache memory is one of the reason the cache memories are so fast.

View more solutions

Comments

[LawrenceC]

Many CPU's support a "cache-as-RAM" mode through the CPU's model specific registers (MSRs). Also note that SRAM consumes more power than DRAM and that is also a design factor. If the CPU's cache was big enough or the kernel small enough you could enable this cache-as-RAM mode and keep it executing entirely in SRAM on the CPU. You would have a limited amount of RAM to run programs, etc. though. because AFAIK cache-as-RAM and normal mode will not work simultaneously. I could be wrong about that though. Even if it did, most of a CPU's speed these days is due to use the L2, L3 cache.

[Alvin Wong]

@Hennes is it that Linux only cares about (mapped) memory addresses?

[user]

SDRAM is Synchronous D(ynamic) RAM, whereas SRAM is Static RAM. I don't know which one you meant to refer to in the first paragraph and I don't have the rep to make "trivial" edits, but maybe you could fix that? Other than that, good answer.

[ganesh]

Not all can/will be copied. For Intel/x86 cache structure: If I have 256KiB cache and 1024KiB cache I can read address 0. It will be stored in the cache at location 0. I can then read address 1, and it will be stored in the cache at location 1. However if I read the address from (256Kib+1) that will also be stored at address 1 in the cache. The cache uses an extra (tag) SRAM to indicate which of the two is stored. This means that reading from multiples of the caches size will not work well. (Note that this would be a rare thing and can usually be ignored).

[ganesh]

I do not mind clarifying, but I am I not sure what you want to have clarified. Can you add that in a comment and I will edit it.

[Ziggy]

This is insightful! Why would I clumsily stuff what I think is important into the L3 Cache when I could let an army of geniuses determine the optimal thing to do, and program a CPU to do that optimal thing. Right?

[Ziggy]

When I first read this comment I saw, "Linux, or any other OS dies not knowing how the RAM works." Your breakdown is a good one: I think I had no illusions that this would be "better". I just wondered whether it could be done.

[ganesh]

Indeed. I only learned that this was possible after the answer. From the coreboot team where the firmware boots before the memory controller was configured. Using something as insane as threadrippers would almost make this practically doable without home building. (Still insanely expensive, but doable and probably fun).