how to save CPU logs or GPU usage values?

Question

folks!

I want to check my CPU logs continuously. I used top and htop commands. But I just check them during simulations.

Now I want to run a real ROBOT for approx 2 hours (my laptop would be on ROBOT while moving).

So How can I save all the records in a log file so that I will access it later?

2ndly I also checked GPU (AMD graphics card) performance. But how can I make a log file?

Solution 1

First decide which CPU stats you want to log

You can choose different statistics to log:

• CPU speed (frequency in Mhz or Ghz)
• CPU utilization percentage
• CPU temperature
• CPU average load factor
• Further these stats can be segmented for each CPU, ie #1 to #8 for quad-core hyper-threaded CPU.

For simplicity sake, I'll base this answer using average load factor similar to the answer in: How to log CPU load?

Create a loop for two hours logging every second

You'll need a bash script to loop 7,200 seconds (2 hours) which would look like this:

#!/bin/bash

for ((i=0; i<7200; i++))
do
    uptime >> /home/user/cpuload.log
    sleep 1
done

Parse the data in a spreadsheet

To look at your output use the command:

$ cat cpuload.log
 20:04:06 up 2 days, 14 min,  1 user,  load average: 1.39, 1.12, 0.95

The load average is reporting three variables--last minute, last five minutes and last fifteen minutes. For simplicity sake only consider the last minute load average which is reported every second in our loop.

You can import the file cpuload.log into a spreadsheet and graph the data points over the two hours or simply scroll over the data. I use Libre Office Calc but all modern spreadsheets can import the file.

Brief points about load average

In the example above the one minute load average is 1.39. This appears dangerously high because anything over .70 deserves investigation and 1.00 means there is a bottle-neck and processes aren't being served and have to wait.

However in your spreadsheet you'll have to divide the load average by the number of CPUs you have. To quickly find this out use the command:

$ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq
2074968
2133093
2094750
1863843
1728562
1855875
1849125
1778156

This shows there are 8 CPUs (it's a quad-core hyper-threaded Intel i-7 3630QM laptop CPU running 1200 Mhz to 3400 Mhz). In this snapshot CPU#1 (called CPU0 internally) is running at 2,0749.68 Mhz and CPU#8 is running at 1,7781.56 Mhz. But I digress, the important thing is to count how many CPUs there are which is 8.

So divide the load average 1.39 by 8 and the TRUE load average is 0.17 which is very respectable. Once again any value over 0.70 deserves investigation and when it hits 1.00 your system is stalling. You can read further here

---

Using top command to see top 10 processes

To use the top command to see the 10 most resource intensive processes use this command instead of the uptime command:

top -n 1 -b | head -n 17 | tail -n 10 >> /home/user/top10.log

The file /home/user/top10.log will look something like this (repeated every second for two hours):

$ top -n 1 -b | head -n 17 | tail -n 10
 6170 rick      20   0 1437432 537000 126060 S  62.5  6.7   8:50.24 chrome
 2466 rick      20   0 1210040 140568  61864 S   6.2  1.8  22:16.88 compiz
 4111 rick      20   0  742396 248724 185820 S   6.2  3.1  36:26.68 chrome
 6280 rick      20   0   41800   3668   3124 R   6.2  0.0   0:00.01 top
10096 root      20   0       0      0      0 S   6.2  0.0   0:00.47 kworker/0:2
    1 root      20   0  120064   6244   4000 S   0.0  0.1   0:02.23 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.05 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:01.31 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+
    7 root      20   0       0      0      0 S   0.0  0.0   1:39.28 rcu_sched

NOTE: replace user with your actual user name.

---

Using top command to get us, sy, id and si CPU values

Similar to the first section, create a bash script to loop 7,200 seconds:

#!/bin/bash

# NAME: ~/bin/cpu-top-summary
# DATE: June 13, 2017
# DESC: Call `top` command every second to obtain CPU(s) stats for
#       us, sy, ni, id, wa, hi, si, and st. Log to /tmp/top-cpu-summary.log
#       with time stamp in hh:mm:ss 24 hour format.
# PARM: $1 number of seconds to run, ie 2 hours = 7200

now="$(date +'%d/%m/%Y')"
printf "top CPU(s) summary for %s\n" "$now" > /tmp/top-cpu-summary.log
for ((i=0; i<$1; i++))
do
    TimeStamp=`date +"%H:%M:%S"`
    printf "$TimeStamp - " >> /tmp/top-cpu-summary.log
    top -n 1 -b | head -n 3 | tail -n 1 >> /tmp/top-cpu-summary.log
    sleep 1
done

When you call the bash script using top-cpu-summary 10 you can see the output for 10 seconds using:

$ cat /tmp/top*
top CPU(s) summary for 13/06/2017
19:17:34 - %Cpu(s): 25.0 us,  9.2 sy,  0.0 ni, 65.4 id,  0.2 wa,  0.0 hi,  0.2 si,  0.0 st
19:17:35 - %Cpu(s): 25.0 us,  9.2 sy,  0.0 ni, 65.4 id,  0.2 wa,  0.0 hi,  0.2 si,  0.0 st
19:17:36 - %Cpu(s): 25.0 us,  9.2 sy,  0.0 ni, 65.4 id,  0.2 wa,  0.0 hi,  0.2 si,  0.0 st
19:17:37 - %Cpu(s): 25.0 us,  9.2 sy,  0.0 ni, 65.4 id,  0.2 wa,  0.0 hi,  0.2 si,  0.0 st
19:17:38 - %Cpu(s): 25.0 us,  9.2 sy,  0.0 ni, 65.4 id,  0.2 wa,  0.0 hi,  0.2 si,  0.0 st
19:17:39 - %Cpu(s): 25.0 us,  9.2 sy,  0.0 ni, 65.4 id,  0.2 wa,  0.0 hi,  0.2 si,  0.0 st
19:17:41 - %Cpu(s): 25.0 us,  9.2 sy,  0.0 ni, 65.4 id,  0.2 wa,  0.0 hi,  0.2 si,  0.0 st
19:17:42 - %Cpu(s): 24.9 us,  9.2 sy,  0.0 ni, 65.4 id,  0.2 wa,  0.0 hi,  0.2 si,  0.0 st
19:17:43 - %Cpu(s): 24.9 us,  9.2 sy,  0.0 ni, 65.4 id,  0.2 wa,  0.0 hi,  0.2 si,  0.0 st
19:17:44 - %Cpu(s): 24.9 us,  9.2 sy,  0.0 ni, 65.4 id,  0.2 wa,  0.0 hi,  0.2 si,  0.0 st

Solution 2

Rather than writing scripts yourself, you could use collectl and perhaps combine that with graphite and grafana to visualise the results.

From the collectl website:

There are a number of times in which you find yourself needing performance data. These can include benchmarking, monitoring a system's general heath or trying to determine what your system was doing at some time in the past. Depending on what you're doing, you often end up using different tools, each designed to for that specific situation.

Unlike most monitoring tools that either focus on a small set of statistics, format their output in only one way, run either interatively or as a daemon but not both, collectl tries to do it all. You can choose to monitor any of a broad set of subsystems which currently include buddyinfo, cpu, disk, inodes, infiniband, lustre, memory, network, nfs, processes, quadrics, slabs, sockets and tcp.

Collectl output can also be saved in a rolling set of logs for later playback or displayed interactively in a variety of formats. If all that isn't enough there are plugins that allow you to report data in alternate formats or even send them over a socket to remote tools such as ganglia or graphite. You can even create files in space-separated format for plotting with external packages like gnuplot.

Comments

[Farzan M. Ñóòrî]

Hy. I need to get us,sy,id and si data. along these I need to see Gzserver and GZclient as i am using gazebo (Top 10 most CPU processes). but i am unable to get these things from collectl. can you guide me about some plugin which shows these stuff?

[Farzan M. Ñóòrî]

Hy. I need to get us,sy,id and si CPU data. along these I need to see Gzserver and GZclient as i am using gazebo (Top 10 most CPU processes). you link was really help full.. even I got to save data of top command using top -b > ~/cpu.txt but it shows data in rubbish form. how can I make it straight.

[Farzan M. Ñóòrî]

2ndly i am unable to import this file using top command. for ((i=0; i<7200; i++)) do uptime >> /home/user/cpuload.log sleep 1 done I need the statistics which shows on the top of top command i.e us, id , sy and si

[Farzan M. Ñóòrî]

also I used aticonfig --odgc --odgt for showing the percentage of my GPU usage. How can i save this thing to0?

[Farzan M. Ñóòrî]

also I used aticonfig --odgc --odgt for showing the percentage of my GPU usage. How can i save this thing to0?

[WinEunuuchs2Unix]

@FarzanM.Ñóòrî I've updated the answer with a new section using the top command to show top 10 processes. I need to research the fields us, sy, id and si tonight to see how they can be selected.

[WinEunuuchs2Unix]

@FarzanM.Ñóòrî Sorry I don't have access to aticonfig --odgc --odgt for testing. Someone else using an ATI chipset will have to answer that part of your question.

[Farzan M. Ñóòrî]

Thankyou very much @WinEunuuchs2Unix sure I will wait for your answer regarding us,sy,id and si.

[Farzan M. Ñóòrî]

I had another curie. This data is showing very complex. Is it any way to sort same data in similar column ? Like if I want to check consumption of specific software (lets say Rviz or gazebo), so how can i log only these softwares. so it would be easy for me to analyze while data checking or to make a graph

[Farzan M. Ñóòrî]

also please have a look upon this command @WinEunuuchs2Unix top -b > ~/cpu.txt this command showing the statistics about us, sy, id and si. But again the question is same.. I dont want extra information.. how to pick only 3rd line from this..

[WinEunuuchs2Unix]

@FarzanM.Ñóòrî The top -b > ~/cpu.txt command you refer to was added this morning before work (13 hours ago). It's in the answer on the line top -n 1 -b | head -n 17 | tail -n 10. Please run this in your terminal and paste the output into your question with comments on how you would like the output massaged to your needs.

[WinEunuuchs2Unix]

@FarzanM.Ñóòrî Sorry for delay have been working lots lately. I think the new third section I added is close to what you wanted. Let me know...

[Farzan M. Ñóòrî]

Thankyou very much :) I got my answers :) :)

[WinEunuuchs2Unix]

You're most welcome. Thank you for exposing me to areas I haven't looked at before. Good luck with your ROBOT :)