List top level manually installed packages without their dependencies

Question

There are many ways to show packages installed manually using apt, such as:

apt-mark showmanual

But sometimes that output is too much. For example if the user manually installed package foo:

apt-get install foo

...and foo depended on bar and baz, then apt-mark showmanual would output:

bar
baz
foo

How can we list only the top level manually installed packages (i.e. foo) without their dependencies (i.e. not baz, nor bar)?

---

The following code seems to work, but GNU parallel calling apt-rdepends a few hundred times is too slow, (three hours with a 4 core CPU):

apt-mark showmanual | 
tee /tmp/foo | 
parallel "apt-rdepends -f Depends,PreDepends,Suggests,Recommends {} |
          tail +2" 2> /dev/null | 
tr -s ' ' '\n' | 
grep -v '[():]' | 
sort -Vu | 
grep -wv -f - /tmp/foo

Solution 1

This could be done using the Python apt API. The packages you see in apt-mark showmanual are exactly the ones in apt.cache.Cache() for which is_installed is true and is_auto_installed is false. But, it's easier to process the dependencies:

#! /usr/bin/env python3

from apt import cache

manual = set(pkg for pkg in cache.Cache() if pkg.is_installed and not pkg.is_auto_installed)
depends = set(dep_pkg.name for pkg in manual for dep in pkg.installed.get_dependencies('PreDepends', 'Depends', 'Recommends') for dep_pkg in dep)

print('\n'.join(pkg.name for pkg in manual if pkg.name not in depends))

Even this lists some packages which I would not expect to see there (init, grep?!).

Solution 2

You can find all manually installed packages without their 1st level of dependencies as follows:

apt-mark showmanual | sort > manually-installed.txt

apt show $(apt-mark showmanual) 2>/dev/null | 
grep -e ^Depends -e ^Pre-Depends > deps1.txt

cat deps1.txt | 
sed 's/^Depends: //; s/^Pre-Depends: //; 
     s/(.*)//g; s/:any//g' > deps2.txt

cat deps2.txt | tr -d ',|' | tr ' ' '\n' | grep -v ^$ |
sort -u > all-dep-packages.txt

grep -v -F -f all-dep-packages.txt manually-installed.txt

You can also use the following one-liner magic:

apt-mark showmanual | sort | grep -v -F -f <(apt show $(apt-mark showmanual) 2> /dev/null | grep -e ^Depends -e ^Pre-Depends | sed 's/^Depends: //; s/^Pre-Depends: //; s/(.*)//g; s/:any//g' | tr -d ',|' | tr ' ' '\n' | grep -v ^$ | sort -u)

Solution 3

The following shell script searches for the parents of all installed dependencies.

function get_installed_packages() {
    apt list --installed | sed 's#/.*##'
}

function get_installed_packages_with_deps() {
    dpkg-query --show --showformat '${Package} ${Depends} \
        ${Pre-Depends}\n' $(get_installed_packages) | 
    sed 's/ ([^(]*)//g; s/:any\|,//g'
}

function get_package_relations() {
    awk '{print $1 " " $1; for(i = 2; i <= NF; i++) print $1 " " $i;}'
}

function add_marker() {
    echo "~ ~"
}

function resolve_parents() {
    tsort | sed -n '1,/~/ p' | head -n -1
}

(get_installed_packages_with_deps | get_package_relations; add_marker) | 
resolve_parents

I used tsort in this script. I assume that when adding a marker at the end without dependencies the marker will also be the last entry without dependencies in my result. So I can differenciate between the last package without dependencies and the first package with depenencies.

I noticed one problem with this solution:
There are cycles in the dependency graph. Those entries are ignored by tsort.

Solution 4

See this answer on Askubuntu, which uses a comparison of apt-mark showmanual with the content of /var/log/installer/initial-status.gz :

comm -23 <(apt-mark showmanual | sort -u) <(gzip -dc /var/log/installer/initial-status.gz | sed -n 's/^Package: //p' | sort -u)

That gave me exactly the packages I had added manually since the fresh install of Ubuntu.

View more solutions

Comments

[agc]

Hmm. The answers, and the OP code, are all so different, and return somewhat different data, that I'm getting a bit fuzzy on which method's data is the most correct. Perhaps a survey answer is needed, starting from a minimal test system, and adding programs a few at a time to see how and when the output vary.

[agc]

On my system that code outputs a superset of my 3-hour code, but shows no surprises like init and grep, (maybe your apt data is corrupt?), also it shows too many libraries. OTOH, my 3-hour code misses a few items that should be there, items which the above python code prints. Possibly the missing items weren't installed with apt.

[muru]

@agc that's probably because I didn't recurse. I'll try a recursive option after the weekend. Even with recursion, though, I'd expect this to be way faster than calling apt-rdepends repeatedly

[agc]

The above python code is 3600 times faster (i.e. it took 3 seconds) than my code (3 hours). Looking forward to testing the recursive version...

[agc]

Much faster. This outputs what's mostly a superset of the OP code, but it also misses a few, such as the dasher package. On my system the OP code piped through sort -V outputs 475 lines, muru's code outputs 914 lines, (including dasher), and this answer's code outputs 995 lines.

[sealor]

Yes, my script does not consider the complete dependency tree. You could try to adapt it for more hierarchy levels.

[Michael Stoll]

There should be a additional -x on the final grep. That will match full lines only. Your version will strip out "apt-transport-s3" if there's a dependency on "apt".