High Apache busy workers and high load average

Question

I've been researching this for a bit and I've seen answers that have gotten me closer to solving this, but nothing that actually gets me all the way there.

I have the following JSON file:

{
  "applications": {
    "app1": [
      {
        "id": "1",
        "filename": "install_latest_app.pkg",
        "version": "1.2.3",
        "date": "20190903"
      },
      {
        "id": "2",
        "filename": "install_latest_app.pkg",
        "version": "1.2.4",
        "date": "20190904"
      },
      {
        "id": "3",
        "filename": "install_latest_app.pkg",
        "version": "1.2.5",
        "date": "20190905"
      }
    ]
  }
}

I am trying to use jq in a Bash script to dynamically add entries to this file under the specific "app". In my example there's only one "app" defined (app1) but in a real world scenario there would be more.

#!/bin/bash

title="app1"
filename="install_latest_app.pkg"
version="1.2.6"
date="20190906"
id="6"

jq --arg title "$title" \
--arg filename "$filename" \
--arg version "$version" \
--arg date "$date" \
--arg id "$id" \
'.applications.$title += [{"id": $id, "filename": $filename, "version": $version, "date": $date}]' data.json >> data.json

when trying to run this, I get a compile error.

jq: error: syntax error, unexpected '$', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
.applications.$title += [{"id": $id, "filename": $filename, "version": $version, "date": $date}]

Same when running it in ways like .applications.[$title] or (.applications.[$title]) (all based on other discussions).

If I do something like .applications | .[$title] |= ... I get a lot closer as it actually returns the JSON with the correct entries, however it doesn't add it to the current "app1", instead adding a new entry for "app1". I would be fine overwriting the entire file every time with the added entry, but I am struggling to get it to print the entire JSON + the new entry. Any help or ideas would be appreciated!

Solution

It looks like you've got a script somewhere that is causing the load.

Start by going through your apache error log and looking for max_execution times or timeouts. Move on to the access logs and look for scripts that are being accessed which may be causing the hanging.

Comments

[Mxx]

Consider using iotop tool to see if there's some unaccounted disk activity going on. Also consider using some monitoring tool that can show cpu steal time. If your host is severely over-provisioned, that might indicate why load is high but no processes uses it.

[Ben Everard]

The server just peaked again... I quickly showed sudo iotop and apache was writing a lot to the server, I'm not sure where I can find logs to say what what being written and at what time... can you advise?

[Mxx]

check in /var/log/apache/ to see if you have large log files and perhaps if they are being rotated right now.

[Ben Everard]

Here is a screenshot of the apache log dir, sorted by size --> cl.ly/image/3Z0r2c1w0c2R. Here I caught apache reading lots from the disk --> cl.ly/image/3b3I3s2y3y2K.

[Charles Duffy]

jq --arg title "$title" --arg filename "$filename" --arg version "$version" --arg date "$date" --arg id "$id" '.applications[$title] += [{"id": $id, "filename": $filename, "version": $version, "date": $date}]' works fine, so the claim that you get the "same when running it ways like[...]" needs to be demonstrated.

[Charles Duffy]

...or, rather: You were close with applications.[$title], but had an extra . that isn't valid syntax; make it .applications[$title] and you're good.

[Ben Everard]

Hi there, I used grep to find anything hitting the max_execution, but nothing came back. I did find, however, lots of strange entries in the access log. See "Update 1" in my question. Thanks.

[Daniel Widrick]

Go ahead and google that message but essentially: you can ignore it. It should not be the issue unless something is horribly misconfigured.

[Ben Everard]

Yeah a quick Google suggested it was just a request from the server to itself... just seemed funny it happened at the same time.

[Barmar]

Probably not coincidental that this is similar to how you do it in JavaScript.

[Charles Duffy]

...or Python, or Ruby, or most other recent languages that using square brackets for interacting with maps.

[Barmar]

My point was that it's like JS's object.literalProp vs object[variableProp]. Python doesn't have that distinction. He was making the same mistake that many JS beginners do by trying to write object.variable