How to Increase Ubuntu's Virtual Memory and/or Swap for Matlab?
Solution 1
OK, quite a list you have there. Let me respond inline
- How to Apply the Better Error-handling of Error trapping here? See my script for the example in the source. Thread How to Do Error-trapping and Swapoff if Error/Warning?.
I don't like the concept of this script at all. That you have an external harddrive that you're trying to use as swap is just a bad idea. If you really intend to do this on a regular basis then resize your partitions to put a proper swap partition in, add a swap file, or just buy a bigger internal disk.
- How to put warnings if the matrix size exceeds the swap size?
Just do do the math. If you know the size of the matrix before the program begins then compute the size in MiB and compare it to the available swap.
- How to have a progress bar in computing your huge matrix in Matlab?
matlab has an API right? I don't think this is the right forum for that question. Even if you had an API, you'll be blocking on IO via swap so it'll just be a jerky progress bar that doesn't actually reflect reality.
- How to kill busy progress and/or swapon -s/swapoff in iteration (2)?
You don't. just because you're done with the computation doesn't mean the operating system is done with the resources you allocated. When it's done writing out to swap, it'll free up. You've consumed so much memory that lots of applications can't get the memory they need so they're using swap too. Just leave it on and let the operating system do it's thing. Before you perform your next run clear the caches.
echo 3 > /proc/sys/vm/drop_caches
There's probably more to it than that, I'm not a Linux VM expert. It would be worthwhile to investigate how the SLAB/SLUB allocator works and how to tune it for your large memory requirements. You may be able to MLOCK matlab into memory. That forces the OS to reserve memory for you, or it just doesn't start, you also have to unlock it when you're done. I can do this with the C API just fine but I not sure how you would do that outside of a process I can't recompile, that would require some research.
Finally, this is the sort of stuff EC2 was made for. It looks like 16G is what you need, a m4.4xlarge has 64G ram @ $0.958 per Hour. That's less than a cup coffee. Script your install of matlab using a juju charm or similar and turn the whole thing into a computation as a service.
Is 16G 16 GB?
- Yes, normally when we leave off the suffix we mean base2 numbers in units of bytes. If you want to concise you would write 16GiB.
"I need matrices that are > 100 GB. I do not know if you can do it with EC2."
- EC2 has machines with up to 2TB main memory. See for yourself. https://aws.amazon.com/ec2/instance-types/
Should you clear your caches also by echo 3 > /proc/sys/vm/drop_caches
?
- Yes, it doesn't hurt to always do that. See Documentation/sysctl/vm.txt in the linux kernel.
How can you MLOCK Matlab into memory?
-
man mlock
. Though I goofed when I quoted that. This call assures that you can allocate all the memory you want and keep in from being swapped out, it'll never use virtual memory. That's not what you want.
I think you can bind C API to Matlab. - - Do you have any idea of turning off swap if there is any failure in the processes?
- I'm going to be frank here, the concept of micromanaging swap files in the manner you propose is ridiculous. The operating system's job is to manage resources and hand them out in a fair & consistent manner. Once you give it more resources, it's going to use them as it sees fit. You don't get to tell it when you're done and yank resources out from underneath it, the OS tells you when it's done.
When I ask the OS for an memory address space, sometimes it doesn't always succeed, that doesn't mean I can't try again. That matlab can't figure out to call malloc twice is matlab's problem.
So, to affect the change you want, if that 100G of space is really at a premium then you need to figure out how to tell the operating system to trim it's memory footprint (by clearing caches for starters) so the memory manager doesn't feel the need to use the additional swap space that it was provided. Then and only then can you ask the memory manager to release the swap file.
It's easy to grow things like memory and disks, it's a lot harder to shrink them. Shrinking forces a re-balancing of every user who has resources allocated in that space. If I instead said "I have a 100TB storage array but now I only need 60TB, why is it when I remove 40TB of disk that the array stops working?" Well, the answer would be obvious right?
So here are your options as I see it.
investigate the matlab C API to see if you can get better control over how memory is allocated for these massive working sets.
refactor your computation to compute what you have now using sub-matrices or some other sparse data representation.
write your own program in C/C++ using the plethora of linear algebra libraries out there to perform the computation and use
malloc
ormmap
anonymous to allocate the address space you need.
Solution 2
You cannot dedicate swap for a software. What you can do is create a shell script that creates a swap, start MATLAB, and delete the swap when MATLAB exits.
Here's an example script which creates a swap of 10Mb in the /tmp directory, mounts it, starts R (I don't have matlab), wait R exits, umount the swap file and delete it.
Please note that: - you will have a warning because the swap file is not owned by root. That's because the system will use if for any software, maybe not ran by you, and you can read on this file... I let you fix it. - if you [ctrl]-[c] the script, or logoff, or etc., the swap will remain mounted. I let you fix it as well.
#!/usr/bin/env bash
SWAP_FILE=/tmp/my_swap_file
SIZE_MB=10
TO_RUN="R"
dd if=/dev/zero of=${SWAP_FILE} bs=1M count=${SIZE_MB}
mkswap ${SWAP_FILE}
chmod 0600 ${SWAP_FILE}
sudo swapon -v ${SWAP_FILE}
echo "Swap enabled. Press enter to continue"; read
${TO_RUN}
echo "I will remove the swap. Press enter to continue"; read
sudo swapoff -v ${SWAP_FILE}
rm -vf ${SWAP_FILE}
Solution 3
Use zswap
if you have swap portions on HDD/SSD.
Tho module zram
is without swap portions on HDD/SSD, so Hakala's answer is not applicable.
See the thread zram vs zswap vs zcache Ultimate guide: when to use which one for explanations.
Setup zswap
as described in the thread How to Activate Zswap Successfully for Matlab Computation in Ubuntu 16.04?
-
Replace the corresponding line with the following line in
/etc/default/grub
# https://wiki.archlinux.org/index.php/Zswap GRUB_CMDLINE_LINUX_DEFAULT="quiet splash zswap.enabled=1 zswap.max_pool_percent=25 zswap.compressor=lzo"
Run
sudo update-grub
.
Solution 4
I would at least test how well compressed RAM (zram kernel module, available since kernel version 3.14) performs.
Following archlinux wiki instructions
modprobe zram
echo lz4 > /sys/block/zram0/comp_algorithm
echo 4G > /sys/block/zram0/disksize
mkswap --label zram0 /dev/zram0
swapon --priority 100 /dev/zram0
My guess is that compressed RAM should be faster than disk I/O.
To keep the change also after the restart, place boot time commands in /etc/rc.local
and run sudo systemctl enable rc-local.service
.
Comments
-
Léo Léopold Hertz 준영 over 1 year
Situation: fix
Out of Memory
errors in Ubuntu's Matlab
Goal: allocate some virtual memory and/or Swap on external HDD/SSD; the read/write decrease from 20 GBps to 0.1 GBps, which is ok!
Terminology: swap and virtual memory hereNo, swap and virtual memory are completely different. For example, memory mapping a 1GB file uses an additional 1GB of virtual memory but has no change in the use of swap. Swap is a form of backing store. Many uses of virtual memory don't have anything to do with backing store. (And there have been systems with virtual memory and no swap as well as systems with swap but no virtual memory.)
Consumer hardware is limited so I need to use more virtual memory and/or swap with my external HDDs. Matlab says about the swap memory (TODO excluding virtual memory?)
Linux Systems — Change your swap space by using the
mkswap
andswapon
commands.Characteristics of the system
-
You can see how much you have it by
swapon -s
Filename Type Size Used Priority /dev/sda3 partition 8326140 0 -1
-
Matlab's configurations
% https://stackoverflow.com/a/35971040/54964 com.mathworks.services.Prefs.setIntegerPref('JavaMemHeapMax', 2048); % MB % TODO cannot find ways how to put Matlab use /dev/sda3
So you see that my Matlab is not using it. I receive
Out of Memory
errors with big matrices in Matlab. I have been unsuccessful inreshape
'ing matrices to vectors and writing parallel code. So I want to use virtual memory because I need to get work done; the rate does not matter.
Pseudocode
- Shell script that creates a swap, start MATLAB, and delete the swap when MATLAB exits. (MichaelHooreman)
- Enable swap on external HDD. How to use
sudo swapon -a
here? - Start Matlab.
- Put Matlab use the swap.
- Delete swap when Matlab exits.
Temporary Swap, Running Client and Closing/Removing Swap in Micheal's script
Situation: cannot control errors in setting up the environment (1), running Matlab (2) and closing the environment (3)
Script#!/usr/bin/env bash # https://stackoverflow.com/a/69808/54964 set -e # TODO How to do swapoff if any error? SWAP_FILE="/media/masi/SamiSwapVirtual/.swap_file_20.7.2016" SIZE_MB=16000 TO_RUN="matlab" dd if="/dev/zero" of=${SWAP_FILE} bs="1M" count=${SIZE_MB} status="progress" mkswap ${SWAP_FILE} chmod 0600 ${SWAP_FILE} sudo chown 0.0 ${SWAP_FILE} # https://unix.stackexchange.com/a/297153/16920 sudo swapon -v ${SWAP_FILE} echo "Swap enabled. Press enter to continue"; read ${TO_RUN} echo "I will remove the swap. Press enter to continue"; read sudo swapoff -v ${SWAP_FILE} rm -vf ${SWAP_FILE}
Iteration 1 with Transcend 25M3 1 TB with little use berofe where filesystem
ext4
-
Logs after starting the script
sh start_matlab_with_swap.sh 16000+0 records in 16000+0 records out 16777216000 bytes (17 GB, 16 GiB) copied, 134.489 s, 125 MB/s Setting up swapspace version 1, size = 15.6 GiB (16777211904 bytes) no label, UUID=48c2835b-4499-4534-aa49-0648e15bd5d9 [sudo] password for masi: swapon /media/masi/SamiWeek/tmp/swap_file_18.7.2016 swapon: /media/masi/SamiWeek/tmp/swap_file_18.7.2016: insecure file owner 1000, 0 (root) suggested. swapon: /media/masi/SamiWeek/tmp/swap_file_18.7.2016: found swap signature: version 1d, page-size 4, same byte order swapon: /media/masi/SamiWeek/tmp/swap_file_18.7.2016: pagesize=4096, swapsize=16777216000, devsize=16777216000 Swap enabled. Press enter to continue start_matlab_with_swap.sh: 11: read: arg count
-
Running the client
Command history gone in the first start of Matlab in the swap memory (ticket #02075943), with the error There was a problem reading your command history - -. Just restart your Matlab and and the problem is solved if you have default settings. Command
prefdir
gives/home/masi/.matlab/R2016a
which is the default location (/home/{username}/.matlab/R2016a
. File/home/masi/.matlab/R2016a/matlab.prf
exists after the restart, here.... [other errors] ...
-
Closing Matlab and typing password again in Terminal
[sudo] password for masi: swapoff /media/masi/SamiWeek/tmp/swap_file_18.7.2016 [ bugs here! ]
Open: How to Apply the Better Error-handling of Error trapping here? See my script for the example in the source. Thread How to Do Error-trapping and Swapoff if Error/Warning?
Permanent Swap = Separate Swap Setup from Running Client
Setting up Swap
# https://unix.stackexchange.com/q/297767/16920 masi@masi:~$ sudo fallocate -l 20G /mnt/.swapfile masi@masi:~$ sudo mkswap /mnt/.swapfile Setting up swapspace version 1, size = 20 GiB (21474832384 bytes) no label, UUID=45df9e48-1760-47e8-84d7-7a14f56bbd72 masi@masi:~$ sudo swapon /mnt/.swapfile swapon: /mnt/.swapfile: insecure permissions 0644, 0600 suggested. masi@masi:~$ sudo chmod 600 /mnt/.swapfile masi@masi:~$ free -m total used free shared buff/cache available Mem: 7925 1494 175 196 6255 5892 Swap: 28610 0 28610
Put the following in the end of
/etc/fstab
for the permanent change# https://unix.stackexchange.com/a/298212/16920 # https://unix.stackexchange.com/a/298543/16920 # If swap is on SSD, trim blocks each time at startup. #/mnt/.swapfile none swap defaults,discard 0 0 # If swap on External HDD, just use sw. /media/masi/SamiWeek/.swapfile none swap sw 0 0
System: Linux Ubuntu 16.04 64 bit
Linux kernel: 4.6
Linux kernel options: wl
Matlab: 2016a
Official Matlab docs: Resolve "Out of Memory" Errors
External HDD: Transcend 1 TB StoreJet 25M3 review, Transcend 2 TB StoreJet 25M3
External HDD filesystem: ext4
External HDD buffer: 8 MB
Related threads: How to increase MATLAB memory limitation in ubuntu? (how to use mkswap, swapon for MATLAB?), How to Reduce Physical memory increase in Matlab?, How to Resolve out of memory error in Matlab?, How to Fix Out of Memory error in Matlab for 10800x10800 matrix?, How do I increase memory limit (contiguous as well as overall) in Matlab r2012b?, How to Increase Array Block and Resolve Out of Memory Error in Matlab 2009b?, How to Resolve this Out of Memory Issue for a Small Variable in Matlab?, 'Out of memory' in Matlab. A slow but a permanent solution?-
Admin almost 8 yearsSorry this isn't answering the question. But are you sure you want this? Doing computations from swap takes unbelievably long. Ram reads at at least 20 GB per second, your external HDD will be reading at less than 0.1 GBps
-
Admin almost 8 years@Anake Yes, I know. It is not a problem. I have huge matrices which can be computed with 32/64 GB but not with my current 8 GB ultrabook. I still need to do tho computation on holidays.
-
Admin almost 8 yearsAnother non-useful comment sorry. Could you leave a computer on at home so that you could SSH over and run them at home/uni?
-
Admin almost 8 years@Anake Not possible, sorry, at the moment. Also, Matlab here so need local computation. Also, do not have sufficient keys to do it at the moment.
-
Admin almost 8 yearsNo. See the pseudo code.
-
-
Michael Hooreman almost 8 yearsWell, it is not really dedicated to MATLAB, but for any situation needing temporarily more RAM. Please note that swapon/swapoff needs root right (so: sudo)
-
Michael Hooreman almost 8 yearsYes, you have. That's how to become root.
-
Sergiy Kolodyazhnyy almost 8 yearsGood answer. I personally would approach it this was, and keep my own add-swap.sh script for thise lovely moments when memory gets eaten up
-
Admin almost 8 yearsWell, swap is already slow, but I can't imagine having swap on USB drive. It will become slow like hell IMHO
-
Admin almost 8 yearsI never have created a script, but will give it a go.
-
ppetraki almost 8 yearsYes, but it might not be enough and you may have to wait and retry before it succeeds, if it ever does.
-
ppetraki almost 8 yearsheh, the rest of your comment didn't show up in my smartphone. you can't sudo echo like that, instead do.
echo 3 | sudo tee /proc/sys/vm/drop_caches
-
Léo Léopold Hertz 준영 almost 8 yearsI extended the answer to the thread unix.stackexchange.com/q/297752/16920 for the effect of zram on the memory size and paging rate. I cannot find any effect on the memory size, probably mostly on the paging. However, systemd/... warning have increased with runit. It would be great to know about the error rate of zram.
-
J.J. Hakala almost 8 yearsIn case of zram0, there are some statistics available in /sys/block/zram0 in files orig_data_size and compr_data_size. I don't have much data swapped there, but for what is there the ratio is about 0.35. With that ratio 11 GB of data might fit in to 4 GB of RAM. That could mean 7 GB of data not swapped to disk.
-
J.J. Hakala almost 8 yearsYou can place boot time commands in /etc/rc.local and then
sudo systemctl enable rc-local.service
-
Léo Léopold Hertz 준영 almost 8 yearsI found out that
zram
is not applicable for swap on HDD/SSD sozram
is not applicable here, please see the thread askubuntu.com/a/472227/25388 -
Léo Léopold Hertz 준영 almost 8 yearsI awarded the bounty here because it made me thing the correct way. I cannot however accept the answer because the error-management is very difficult and incomplete. It is best to setup the swap separately from running Matlab, please see the answer here unix.stackexchange.com/a/298543/16920
-
Léo Léopold Hertz 준영 almost 8 yearsI accept this answer because it has the right sense in its answer. To run Client in the same script as setting up the environment is insanely error-prone. More discussion about it here unix.stackexchange.com/a/298543/16920 - - It would be great if you could clean up your body little.