Why are there deb files in /var/cache/apt/archives/?

Question

Why are there deb files in /var/cache/apt/archives/?

In what cases were the deb files downloaded into that directory?

Solution

Background Information

There are two things in Ubuntu (and in Debian, on which Ubuntu is based) that are commonly called "the package manager": dpkg, and APT.

dpkg is a program that installs and removes packages and maintains a database of what packages are installed, at what versions. When it installs packages, it does so from .deb archive files. These archives are the way packages are distributed.

dpkg does more than copy (or delete) files and update its database--it will also run installation scripts and removal scripts. It may be considered pretty sophisticated. But what it will not do is find and download packages.

APT finds and downloads packages, and runs dpkg appropriately to install, upgrade, and remove them. APT is not really any single piece of software, though there is an apt package that provides its essential parts.

APT includes a library (libapt-pkg) that provides core functionality, accessed by various tools. Some such tools, like apt-get, are also provided by the apt package. Other tools are installed separately. APT also provides various configuration files that control how its core library, and programs that use it, work.

More than that, APT is a system for storing and retrieving data about what software sources provide what packages in what versions, and for acting on those data to automatically perform package management tasks like installing and upgrading packages. This includes keeping track of what packages are needed for other packages to work, and automatically picking and installing appropriate packages to satisfy these dependencies.

For example, when I tell the apt-get utility to install the chocolate-doom package, it figures out what other packages have to be installed for it to work, and will automatically retrieve and install them too:

$ sudo apt-get install chocolate-doom
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  doom-wad-shareware libmikmod3 libsdl-mixer1.2 libsdl-net1.2 libsdl1.2debian
Recommended packages:
  doom-engine
The following NEW packages will be installed:
  chocolate-doom doom-wad-shareware libmikmod3 libsdl-mixer1.2 libsdl-net1.2
  libsdl1.2debian
0 upgraded, 6 newly installed, 0 to remove and 1 not upgraded.
Need to get 0 B/2,846 kB of archives.
After this operation, 8,626 kB of additional disk space will be used.

Graphical tools like the Software Center, Software Updater / Update Manager, and Synaptic perform functions similar to apt-get. They also use the APT system--through a combination of observing its rules and utilizing functionality provided by core libraries--to change what packages are installed, to give the user what s/he (usually) wants, in such a way that things (usually) continue to work.

How and When the /var/cache/apt/archives Folder Is Used

When apt-get, aptitude, Software Center, Synaptic, Software Updater / Update Manager, or any other package management utility downloads packages (and the other packages they depend on) from configured software sources and installs them, the downloaded packages (i.e., the .deb archive files from which they are installed) are stored in /var/cache/apt/archives.

If you have a .deb file on disk and you install it with dpkg, that does not cause a copy of it to be made in /var/cache/apt/archives. But when a program that uses or implements the APT system of automatic dependency resolution and package retrieval downloads packages, that's where it stores them.

While .deb files for automatically downloaded packages are stored in and installed from /var/cache/apt/archives, they are not actually downloaded directly to that location. Instead, incomplete (i.e., in progress) .deb file downloads exist in /var/cache/apt/archives/partial and are then moved to /var/cache/apt/archives when finished.

.deb package files remain cached in /var/cache/apt/archives even after successful installation in case they're needed for future use. But you can safely remove .deb files that are old enough they're unlikely to be needed again with sudo apt-get autoclean, or clear out the cached .deb files entirely with sudo apt-get clean. See man apt-get for more information.

To illustrate the function of sudo apt-get autoclean and similar commands, as well as to determine if you wish to run them, you can simulate their actions by running then with the -s flag. It's best to pass this flag before the autoclean (or whatever other) action verb. It's not typically necessary to run a utility as root with sudo when using -s, because no actual changes are being made.

For example, on my system:

$ apt-get -s autoclean
NOTE: This is only a simulation!
      apt-get needs root privileges for real execution.
      Keep also in mind that locking is deactivated,
      so don't depend on the relevance to the real current situation!
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Del libubsan0 4.9.1-15ubuntu1 [83.7 kB]
Del cups-server-common 1.7.5-3 [519 kB]
Del python2.7 2.7.8-9 [205 kB]
Del g++-4.9 4.9.1-15ubuntu1 [17.4 MB]
Del libstdc++-4.9-dev 4.9.1-15ubuntu1 [1,121 kB]
    ....

The listed packages would not be uninstalled; instead, just cached .deb files would be deleted. (In fact neither happened when I ran that command, as it was merely a simulation.)

Other Downloaded Files

Software installed via dpkg (via APT or otherwise) is not actually run from the .deb file that provides it. As noted in the Background Information section, .deb files are extracted. This is why they can be removed and the installed package continues functioning.

Files throughout the system that are installed from .deb packages aren't downloaded to their current locations--rather, they are extracted to them from the .deb files that provide them (or perhaps specially created during installation or afterwards, as is the case for some configuration files).

But .deb files aren't the only files downloaded by package management utilities. Often other files are downloaded that provide information about where packages exist.

For example, when I run sudo apt-get update, the output includes lines like:

Get:13 http://us.archive.ubuntu.com utopic/main i386 Packages [1,341 kB]

This means a file like http://us.archive.ubuntu.com/ubuntu/dists/utopic/main/binary-amd64/Packages.bz2 was downloaded and examined to help update the APT database with information on what packages are available (and in what versions) from that particular software source. You can find Packages files, and other such files, in the /var/lib/apt/list directory. The copies stored there are given distinguishing names like us.archive.ubuntu.com_ubuntu_dists_utopic_main_binary-amd64_Packages, as there may be many such files.

You asked about apt-cache specifically. It queries the package database, but it does not make any changes to the system or download any files.

Downloaded Files Specific to the Software Center

When the Ubuntu Software Center downloads data for its own use, specific to it and not relating to general APT (or dpkg) functionality, it stores them in /var/cache/software-center.

For example, on my system (and on most systems), that folder contains a xapian subfolder:

ek@Ilex:/var/cache/software-center/xapian$ ls
flintlock       postlist.baseB  record.baseB    spelling.baseB  termlist.baseB
iamchert        postlist.DB     record.DB       spelling.DB     termlist.DB
postlist.baseA  record.baseA    spelling.baseA  termlist.baseA

The Software Center uses Xapian to index information for quick searching.

Downloaded Files for Other Related Utilities

Some additional utilities that work with the APT system download and store files as part of their operation. apt-file is not installed by default in Ubuntu, but may be installed from the apt-file package.

apt-file stores information on what files are provided by what packages (even for packages that aren't locally installed). It obtains this information about the contents of packages by downloading compressed Contents files:

$ sudo apt-file update
Downloading complete file http://us.archive.ubuntu.com/ubuntu/dists/utopic/Contents-amd64.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 29.6M  100 29.6M    0     0   856k      0  0:00:35  0:00:35 --:--:--  832k

When apt-file update is run as root (as above), these files are downloaded to the /var/cache/apt/apt-file directory. To discover this, I ran:

$ sudo updatedb
$ locate Contents-amd64
/var/cache/apt/apt-file/us.archive.ubuntu.com_ubuntu_dists_utopic-proposed_Contents-amd64.gz
/var/cache/apt/apt-file/us.archive.ubuntu.com_ubuntu_dists_utopic_Contents-amd64.gz

Note that the information provided in this post gives a number of examples of data files dowloaded and stored by package management utilities on Ubuntu, but should not be considered an exhaustive list of instances where such files are used and stored.

Comments

[Tim]

Thanks. Do apt-cache or/and apt-file also download many things? If yes, where are they?

[Tim]

What does "But when a program that uses or implements the APT system of automatic dependency resolution and package retrieval downloads packages, that's where it stores them." mean?

[Eliah Kagan]

@Tim I've expanded my answer considerably, in a way that I hope properly answers those questions (and more). As always your feedback is most welcome--especially if anything is unclear.

[Tim]

Thanks "apt-cache queries the package database", is the database online or on the local computer? Does apt-cache need access to the internet?

[Eliah Kagan]

@Tim It queries your local package database. apt-cache doesn't actually check your remote repositories. The local database is updated by other programs, for example when you run sudo apt-get update or automatically by the Software Center and the Software Update / Update Manager. The results of running apt-cache are of course affected by changes made to the local package database, but apt-cache won't itself make those changes.

[Tim]

To get free space, are the removable files only the deb files in /var/cache/apt/archives?

[Eliah Kagan]

@Tim It's not generally safe to remove most of the other files I've mentioned in my answer, but they're quite small, so you wouldn't benefit much if your goal is to free up space. In contrast, cached debs in /var/cache/apt/archives (and occasionally unfinished downloads in /var/cache/apt/archives/partial) sometimes take up quite a bit of space. You can safely delete them all with sudo apt-get clean. This may or may not be sufficient to free up the space you need, of course. du -sh /var/cache/apt/archives will show you how much space is currently taken up by cached deb files.

[Orian]

I truly wish there were a method for bookmarking helpful answers for future use. I had to dig for a while to find an answer that made sense (and helped with the problem). Since this is a regular and unavoidable problem with linux, I am pretty sure I will need to read this again in the future.

[Eliah Kagan]

@Orian (I'm glad this answer helped!) You can click the star on the question. It's just below the question's score and vote arrows. Then the question appears in your profile's favorites tab. Only questions can be favorited. In the case of this answer, it's currently the only one on the question. Others might be posted--though you would probably want to look at those too, since sometimes new answers have information about new developments in a topic. You can use your browser's bookmarks, which work even for answers. Use the share link under the answer to open the page at a specific answer.

[WinEunuuchs2Unix]

+1 What a wonderful answer! Small points that could be expanded on when you run: $ sudo du /var/cache/apt/archives -h you get 4.0K /var/cache/apt/archives/partial and then 383M /var/cache/apt/archives. The partial directory isn't mentioned in your answer? 383MB seems large for a small Ubuntu installation like mine?

[gerardw]

deb package files remain cached in /var/cache/apt/archives even after successful installation in case they're needed for future use What's an example of a future use that uses the files?