Files list file for package 'package' is missing final newline

Question

To resolve this issue I have run these commands

sudo mv /var/lib/dpkg/info/<package name>.* /var/www
sudo apt-get install <package name>

But when I try to install any new packages or upgrade using the below command it gives errors for other packages.

What could have caused this problem and how can I resolve it?

$ sudo apt-get dist-upgrade
dpkg: warning: files list file for package 'libexpat1:amd64' missing;
 assuming package has no files currently installed
dpkg: warning: files list file for package 'base-passwd' missing;
 assuming package has no files currently installed
dpkg: warning: files list file for package 'bash' missing;
 assuming package has no files currently installed
dpkg: warning: files list file for package 'debconf' missing;
 assuming package has no files currently installed
dpkg: warning: files list file for package 'dash' missing;
 assuming package has no files currently installed   
dpkg: warning: files list file for package 'coreutils' missing;
 assuming package has no files currently installed
dpkg: warning: files list file for package 'python3-software-properties' missing;
 assuming package has no files currently installed
dpkg: warning: files list file for package 'software-properties-common' missing;
 assuming package has no files currently installed
dpkg: unrecoverable fatal error, aborting: 
 files list file for package 'libcap-ng0:amd64' is missing final newline
E: Sub-process /usr/bin/dpkg returned an error code (2)

$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  lxd lxd-client open-iscsi openssl tmux tzdata unattended-upgrades
7 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/9806 kB of archives.
After this operation, 604 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Preconfiguring packages ...
dpkg: warning: files list file for package 'libexpat1:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'base-passwd' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'bash' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'debconf' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'dash' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'coreutils' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'base-files' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'diffutils' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'python3-software-properties' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'ncurses-bin' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'software-properties-common' missing; assuming package has no files currently installed
dpkg: unrecoverable fatal error, aborting:
 files list file for package 'libcap-ng0:amd64' is missing final newline
E: Sub-process /usr/bin/dpkg returned an error code (2)

$ sudo apt-get install --reinstall base-passwd
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 7 not upgraded.
Need to get 0 B/47.2 kB of archives.
After this operation, 0 B of additional disk space will be used.
Preconfiguring packages ...
dpkg: warning: files list file for package 'libexpat1:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'base-passwd' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'bash' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'debconf' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'dash' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'coreutils' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'base-files' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'diffutils' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'python3-software-properties' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'ncurses-bin' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'software-properties-common' missing; assuming package has no files currently installed
dpkg: unrecoverable fatal error, aborting:
 files list file for package 'libcap-ng0:amd64' is missing final newline
E: Sub-process /usr/bin/dpkg returned an error code (2)

$ sudo chmod -R a+r /var/lib/dpkg/info
$ cd /var/lib/dpkg/info

/var/lib/dpkg/info$ sudo mv libcap-ng0\:amd64.
libcap-ng0:amd64.list      libcap-ng0:amd64.md5sums   libcap-ng0:amd64.shlibs    libcap-ng0:amd64.triggers

/var/lib/dpkg/info$ sudo mv libcap-ng0\:amd64.
libcap-ng0:amd64.list      libcap-ng0:amd64.md5sums   libcap-ng0:amd64.shlibs    libcap-ng0:amd64.triggers

/var/lib/dpkg/info$ sudo mv libcap-ng0\:amd64.* /var/www/html/

/var/lib/dpkg/info$ sudo apt-get install --reinstall libcap-ng0:amd64
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 7 not upgraded.
Need to get 11.1 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://ap-south-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 libcap-ng0 amd64 0.7.7-3.1 [11.1 kB]
Fetched 11.1 kB in 0s (39.2 kB/s)
dpkg: warning: files list file for package 'libexpat1:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'base-passwd' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'bash' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'debconf' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'dash' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'coreutils' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'base-files' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'diffutils' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'python3-software-properties' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libcap-ng0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'ncurses-bin' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'software-properties-common' missing; assuming package has no files currently installed
dpkg: unrecoverable fatal error, aborting:
 files list file for package 'libcom-err2:amd64' is missing final newline
E: Sub-process /usr/bin/dpkg returned an error code (2)

I have removed libcap-ng0:amd64 files from /var/lib/dpkg/info folder and later tried to reinstall it but now it has provided error for other package and warning for libcap-ng0.
It has occurred for rest packages in same way

Solution 1

Here are step by step instructions to fix this problem, which appears to be file corruption in the .list files related to these pacakges:

1. Verify that the .list files are full of garbage:

cat /var/lib/dpkg/info/<name-of-package>.list

2. Remove the corrupt .list file:

sudo rm /var/lib/dpkg/info/<name-of-package>.list

3. Run the reinstall for that package (will show a warning, but should reinstall):

sudo apt install <name-of-package> --reinstall

If what happened to me happens to you, each time you run this it will run into another corrupted .list file and you will have to step through many before your system is fixed. If you have many, then step three is optional -- just run one sudo apt upgrade at the end and it will reinstall everything you need.

Solution 2

From: apt-get dpkg error: files list file ... missing final newline - LinuxQuestions.org

Well, with a big help from #debian at freenode, I found the solution (thanks justin!!):

the file /var/lib/dpkg/info/smbfs.list was corrupted, it was all binary crap. Dunno how this could happen, but it was. So justin advised me to delete the file (actually, I moved it) - and, voila, apt warned me that the files list could not be found for smbfs, but at least I could apt-get remove smbfs --purge and install it again (apt-get install smbfs) - that took the biscuit!

Solution 3

If you're getting 50-80 damaged files on multiple machines like me, the manual method will be an endless task, so here's my take on the issue (Tested on Ubuntu 20.04). This is basically an automated version of yourcomputergenius' answer:

cd /var/lib/dpkg/info/; \
for i in *; do \
  if (file $i|grep -P '(?<!__DAMAGED:)[\s]data$'); then \
    sudo mv -v "$i" "$i__DAMAGED"; \
  fi; \
done; \
ls *__DAMAGED | cut -d'.' -f1 - | uniq | xargs -I'{}' sudo apt-get install {} --reinstall

What it does:

• checks all files in the directory
• renames all files with binary content by appending __DAMAGED to the filename
• then:
  • sorts out all damaged files
  • extracts the package names of the filenames
  • drops duplicate package names
  • then feeds the remaining list to apt to reinstall the affected packages

I hope it helps anybody who encounters this problem!

Comments

[cloud.009]

Try using 'sudo apt-get -f install'

[insoftservice]

Its not duplicate as i have already mentioned that i have tried the solution but its now occurring to other packages and has gone to infinite loop

[insoftservice]

yes,But it did not worked for me.

[insoftservice]

Thx for your answer but i am getting same o/p. dpkg: warning: files list file for package 'software-properties-common' missing; assuming package has no files currently installed dpkg: unrecoverable fatal error, aborting: files list file for package 'libcap-ng0:amd64' is missing final newline E: Sub-process /usr/bin/dpkg returned an error code (2)

[Zanna]

did you move all the package list files? The command(s) you actually ran is not clear from your question. Please run sudo apt update and give the complete verbatim output formatted as code. I would expect that command to possibly fix your problem actually, but if not it may give some clue

[insoftservice]

Yes i had removed all package list file and more i remove file more i get warnings and issue of newline for other packages. sudo apt update Get:1 security.ubuntu.com...... InRelease [83.2 kB] Get:5 ap-south-1.ec2.archive.ubuntu.com/ubuntu InRelease [74.6 kB] Fetched 247 kB in 1s (249 kB/s) Reading package lists... Done Building dependency tree Reading state information... Done 7 packages can be upgraded. Run 'apt list --upgradable' to see them.

[insoftservice]

After executing sudo apt list --upgradable Listing... Done lxd/bionic-updates 3.0.3-0ubuntu1~18.04.1 amd64 [upgradable from: 3.0.2-0ubuntu1~18.04.1] lxd-client/bionic-updates 3.0.3-0ubuntu1~18.04.1 amd64 [upgradable from: 3.0.2-0ubuntu1~18.04.1] open-iscsi/bionic-updates 2.0.874-5ubuntu2.6 amd64 [upgradable from: 2.0.874-] openssl/bionic 1.1.1a-2+ubuntu18.04.1+deb.sury.org+1 amd64 tmux/bionic-updates 2.6-3ubuntu0.1 amd64 [upgradable from: 2.6-3] tzdata/bionic-updates,bionic-security 2018i-0ubuntu0.18.04 all unattended-upgrades/bionic-updates 1.1ubuntu1.18.04.8 all

[insoftservice]

After executing reinstall for anyone packages mentioned in upgrade . sudo apt-get install --reinstall lxd Reading package lists... Done After this operation, 587 kB of additional disk space will be used. dpkg: warning: files list file for package 'software-properties-common' missing; assuming package has no files currently installed dpkg: unrecoverable fatal error, aborting: files list file for package 'libcap-ng0:amd64' is missing final newline E: Sub-process /usr/bin/dpkg returned an error code (2) Same issue occurs again. just pasted few lines due to space restriction

[Zanna]

Please edit your question to add information - command output is really hard to read in the comments

[insoftservice]

Sorry i can understand. But o/p is same for all commands like sudo apt-get install --reinstall packagename. or upgrade or update it gives same o/p given on question

[insoftservice]

I am not getting text option probably as its mark duplicate. I know it has to be marked under code , but no option for the same. I dont have issue of IP now as it has got replaced with other :)

[insoftservice]

Thx for your help . paste.ubuntu.com/p/mjR98PybwK

[insoftservice]

Hope question has been updated as per standard ? Is there any solution for this issue ?

[DragonflyRobotics]

That helped a lot. Thanks!

[yourcomputergenius]

You're welcome!

[Gergely Lukacsy]

I've encountered with the same issue and I find it extremely concerning. What's the cause of this phenomenon, could it be some sort of security breach?

[yourcomputergenius]

@GergelyLukacsy my top theory for the machine that it happened to me on was actual hard drive corruption... It was on a spinning disk, but not too old...

[yourcomputergenius]

Very cool, I’m going to check it out! Thanks!!

[Saren Tasciyan]

It fails to detect package names with a dot "." in the name. Otherwise, it has been working well so far.

[Gergely Lukacsy]

@SarenTasciyan Thanks, I edited my answer accordingly. Cheers!