The last du (containing listsystemd and xserver-xorg-core) has screwed my system. A journalctl -b -p err show lots of errors mostly relating to deleted services (mistimed?). Also my original /home partition appears not to be mounted yet the system boots (after a lo..ng time) to a sddm login screen and thence after another lo..mg time to the standard KDE wallpaper but minus the panel.
There is a bug #798097 on xserver-xorg-core which may affect login but I could still login. The hd seems to me to cpreforming correctly so I don't think it's a hardware failure. I'm tending towards a software bug causing a system wide failure.
Strange this also happened to me when the last upgrade to libsystemd occurred a few days ago. Could it be that systemd is now too important and too integrated into Debian that it makes using sid no longer a practical proposition?
Has anyone else experienced this and perhaps have a solution?
For anyone interested:
investigating why home is not mounting at boot
df -h
Filesystem Size Used Avail Use% Mounted on
udev 10M 0 10M 0% /dev
tmpfs 1.6G 1.2M 1.6G 1% /run
/dev/sda1 19G 7.1G 11G 40% /
tmpfs 4.0G 184K 4.0G 1% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 4.0G 0 4.0G 0% /sys/fs/cgroup
tmpfs 799M 0 799M 0% /run/user/119
tmpfs 799M 0 799M 0% /run/user/1000
note:no /home partition (/dev/sda2)
fstab ie OK
UUID=9bd3729b-c15c-48a8-8719-f83c6b97c387 / ext4 defaults,relatime,errors=remount-ro 0 1
UUID=e9aa0c2d-6161-422f-875e-a1a6194491c7 /home ext4 auto,users,rw,exec,relatime 0 0
UUID=40199a5b-aeeb-45eb-9f74-0911df8e7c22 none swap sw 0 0
mounting /dev/sda2 enables me gain access to the original /home. and it's files.
journalctl -b -p err gives
- Logs begin at Sat 2015-09-26 10:03:25 BST, end at Thu 2015-10-08 22:32:35 BST. --
Oct 08 21:10:57 westfield2 systemd[1]: nfs-common.service: Job nfs-common.service/start deleted to break ordering cycle$
Oct 08 21:10:57 westfield2 systemd[1]: rpcbind.service: Job rpcbind.service/start deleted to break ordering cycle start$
Oct 08 21:10:59 westfield2 systemd[1]: rpcbind.service: Job rpcbind.service/start deleted to break ordering cycle start$
Oct 08 21:11:04 westfield2 kernel: k10temp 0000:00:18.3: unreliable CPU thermal sensor; monitoring disabled
Oct 08 21:11:04 westfield2 systemd[1]: nfs-common.service: Job nfs-common.service/start deleted to break ordering cycle$
Oct 08 21:11:05 westfield2 systemd[1]: rpcbind.target: Job rpcbind.target/start deleted to break ordering cycle startin$
Oct 08 21:11:05 westfield2 systemd[1]: rpcbind.service: Job rpcbind.service/start deleted to break ordering cycle start$
Oct 08 21:11:05 westfield2 systemd[1]: nfs-common.service: Job nfs-common.service/start deleted to break ordering cycle$
Oct 08 21:11:05 westfield2 systemd[1]: rpcbind.target: Job rpcbind.target/start deleted to break ordering cycle startin$
Oct 08 21:11:05 westfield2 systemd[1]: rpcbind.service: Job rpcbind.service/start deleted to break ordering cycle start$
Oct 08 21:11:16 westfield2 hp[1128]: io/hpmud/pp.c 627: unable to read device-id ret=-1
Oct 08 22:11:17 westfield2 hpfax[1129]: [1129]: error: Failed to create /var/spool/cups/tmp/.hplip
Oct 08 22:11:19 westfield2 systemd[1229]: Failed unmounting /sys/kernel/debug.
Oct 08 22:11:19 westfield2 systemd[1229]: Failed unmounting /dev/mqueue.
Oct 08 22:11:19 westfield2 systemd[1229]: Failed unmounting /run/user/119.
Oct 08 22:11:19 westfield2 systemd[1229]: Failed unmounting /dev/hugepages.
Oct 08 22:11:30 westfield2 colord-sane[1100]: io/hpmud/pp.c 627: unable to read device-id ret=-1
Oct 08 22:11:44 westfield2 sddm-helper[1172]: pam_systemd(sddm-greeter:session): Failed to create session: Connection t$
Oct 08 22:11:56 westfield2 systemd[3953]: Failed unmounting /dev/hugepages.
Oct 08 22:11:56 westfield2 systemd[3953]: Failed unmounting /sys/kernel/debug.
Oct 08 22:11:56 westfield2 systemd[3953]: Failed unmounting /run/user/119.
Oct 08 22:11:56 westfield2 systemd[3953]: Failed unmounting /dev/mqueue.
Oct 08 22:11:56 westfield2 systemd[3953]: Failed unmounting /run/user/1000.
Oct 08 22:12:21 westfield2 sddm-helper[3952]: pam_systemd(sddm:session): Failed to create session: Connection timed out
Oct 08 22:12:37 westfield2 pulseaudio[5626]: [pulseaudio] pid.c: Daemon already running.
Oct 08 22:14:19 westfield2 systemd[1]: Failed to start User Manager for UID 119.
Oct 08 22:14:56 westfield2 systemd[1]: Failed to start User Manager for UID 1000.
Oct 08 22:20:23 westfield2 systemd[30270]: Failed unmounting /run/user/0.
Oct 08 22:20:23 westfield2 systemd[30270]: Failed unmounting /.
Oct 08 22:20:53 westfield2 systemd[1]: Failed to start Clean up any mess left by 0dns-up.
Oct 08 22:21:12 westfield2 systemd[1075]: Failed unmounting /run/user/0.
Oct 08 22:21:12 westfield2 systemd[1075]: Failed unmounting /.
Oct 08 22:21:44 westfield2 systemd[1]: Failed to start Clean up any mess left by 0dns-up.
Oct 08 22:21:45 westfield2 systemd[2493]: Failed unmounting /run/user/119.
Oct 08 22:21:45 westfield2 systemd[2493]: Failed unmounting /dev/hugepages.
Oct 08 22:21:45 westfield2 systemd[2493]: Failed unmounting /dev/mqueue.
Oct 08 22:21:45 westfield2 systemd[2493]: Failed unmounting /sys/kernel/debug.
Oct 08 22:22:10 westfield2 sddm-helper[2490]: pam_systemd(sddm-greeter:session): Failed to create session: Connection t$
Oct 08 22:22:18 westfield2 systemd[4921]: Failed unmounting /sys/kernel/debug.
Oct 08 22:22:18 westfield2 systemd[4921]: Failed unmounting /dev/mqueue.
Oct 08 22:22:18 westfield2 systemd[4921]: Failed unmounting /run/user/1000.
Oct 08 22:22:18 westfield2 systemd[4921]: Failed unmounting /dev/hugepages.
Oct 08 22:22:18 westfield2 systemd[4921]: Failed unmounting /run/user/119.
Oct 08 22:22:43 westfield2 sddm-helper[4920]: pam_systemd(sddm:session): Failed to create session: Connection timed out
Oct 08 22:22:49 westfield2 pulseaudio[6468]: [pulseaudio] pid.c: Daemon already running.
Oct 08 22:22:49 westfield2 pulseaudio[6474]: [pulseaudio] pid.c: Daemon already running.
Oct 08 22:24:45 westfield2 systemd[1]: Failed to start User Manager for UID 119.
Oct 08 22:25:19 westfield2 systemd[1]: Failed to start User Manager for UID 1000.
Far too many error for my small brain to get around!
Any ideas anyone?
[/code]
CPU~Quad core AMD Phenom 9650 (-MCP-) speed/max~1150/2300 MHz Kernel~4.2.3-towo.1-siduction-amd64 x86_64 Up~40 min Mem~1569.1/7989.8MB HDD~250.1GB(64.9% used) Procs~194 Client~Shell inxi~2.2.28
[/code]
Edit: forgot to add the output of BLKID
/sbin/blkid
/dev/sda1: UUID="9bd3729b-c15c-48a8-8719-f83c6b97c387" TYPE="ext4" PARTUUID="0007a04f-01"
/dev/sda2: LABEL="Home" UUID="e9aa0c2d-6161-422f-875e-a1a6194491c7" TYPE="ext4" PARTUUID="0007a04f-02"
/dev/sda3: UUID="40199a5b-aeeb-45eb-9f74-0911df8e7c22" TYPE="swap" PARTUUID="0007a04f-03"
I don't have time right now to document everything, but my data files are on a second drive/partition that is not being mounted automatically. Nothing changed other than the my doing the DU. UUID is still good in fstab (with blkid and ls -l /dev/disk/by-uuid/). I can mount the partition by clicking in Dolphin, Manually, and it will mount with mount -a.
I would try removing nfs-common package if you don't plan to use/read NFS volumes.
This is the closest Debian bug I could find: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=788191
Removing nfs-common did not change anything. Everything was fine first thing this morning I did an apt-get update and du - there were no new packages, no changes. (Eastern Time, US), it is now almost 8PM.
My data folders live on sdc3 (a conventional drive), they are linked to /home on sda2 (a SSD). Everything works, other than my desktop background and dropbox, those files are on the unmounted sdc3.
I did a du by going to tty1, becoming root and issuing systemctl isolate multi-user.target. Following the du which finished uneventfully I rebooted
QuoteStart-Date: 2015-10-08 12:42:19
Commandline: apt-get dist-upgrade
Upgrade: man-db:amd64 (2.7.3-1, 2.7.4-1), librsvg2-2:amd64 (2.40.10-1, 2.40.11-1), os-prober:amd64 (1.67, 1.68), libexiv2-14:amd64 (0.25-2, 0.25-2.1), xserver-xorg-input-all:amd64 (7.7+9, 7.7+12), libpam-systemd:amd64 (226-4, 227-1), xserver-xorg-core:amd64 (1.17.2-1.1, 1.17.2-3), udev:amd64 (226-4, 227-1), xserver-common:amd64 (1.17.2-1.1, 1.17.2-3), libspice-server1:amd64 (0.12.5-1.2, 0.12.5-1.3), libudev1:amd64 (226-4, 227-1), libmm-glib0:amd64 (1.4.10-1, 1.4.12-1), python-six:amd64 (1.9.0-5, 1.10.0-1), librsvg2-common:amd64 (2.40.10-1, 2.40.11-1), systemd-sysv:amd64 (226-4, 227-1), libcairomm-1.0-1v5:amd64 (1.10.0-1.2, 1.12.0-1), modemmanager:amd64 (1.4.10-1, 1.4.12-1), systemd:amd64 (226-4, 227-1), libsigc++-2.0-0v5:amd64 (2.6.1-1, 2.6.1-2), xserver-xorg:amd64 (7.7+9, 7.7+12), libnss-myhostname:amd64 (226-4, 227-1), libsystemd0:amd64 (226-4, 227-1), x11-common:amd64 (7.7+9, 7.7+12), python3-six:amd64 (1.9.0-5, 1.10.0-1), libpq5:amd64 (9.4.4-2, 9.4.5-1), exfat-utils:amd64 (1.2.1-1, 1.2.1-2)
End-Date: 2015-10-08 12:42:49
Upon reboot, the screen stayed black for an unusual amount of time before the sddm screen appeared. It still seems to be taking a long time, but I have never timed it. I get an error from Dropbox, since it can't find it's data.
cat /etc/fstab
# /etc/fstab: static file system information.
#
# <file system> <mount point> <type> <options> <dump> <pass>
#Entry for /dev/sda1 :
UUID=f5236926-cfcd-4df4-9439-f26a29f1d6c9 / ext4 noatime,commit=120,errors=remount-ro 0 1
#Entry for /dev/sda2 :
UUID=c84955dd-4a65-4bbf-aabb-0166f8af7b26 /home ext4 defaults,relatime,errors=remount-ro 0 2
#Entry for /dev/sda3 :
UUID=4dabe4a5-4b70-4d93-a4ca-97ae840b1204 none swap sw 0 0
#Entry for /dev/sdb1 :
UUID=44228B67228B5D34 /disks/disk2part1 ntfs-3g defaults,auto,users,locale=en_US.utf8 0 0
#Entry for /dev/sdb2 :
UUID=EE96907496903ED1 /disks/disk2part2 ntfs-3g defaults,auto,users,locale=en_US.utf8 0 0
#Entry for /dev/sdc2 :
UUID=030cf640-699f-423f-98ea-f89e06ad9278 /disks/disk3part2 ext4 auto,users,rw,exec,relatime 0 0
#Entry for /dev/sdc3 :
UUID=0b8ed16b-f34f-4e98-aebd-5912ed524de4 /disks/disk3part3 ext4 auto,users,rw,exec,relatime 0 0
The UUID's have not changed.
# blkid /dev/sdc3
/dev/sdc3: LABEL="data" UUID="0b8ed16b-f34f-4e98-aebd-5912ed524de4" TYPE="ext4" PARTUUID="f5fda669-03"
# ls -l /dev/disk/by-uuid
total 0
lrwxrwxrwx 1 root root 10 Oct 8 19:41 0b8ed16b-f34f-4e98-aebd-5912ed524de4 -> ../../sdc3
As I stated earlier, I can open Dolphin, click the Data(sdc3) folder and it will mount after I provide the root password.
If I issue mount /dev/sdc3 it will mount, and if I issue mount -a all the sdc partitions mount.
At that point I can open Dropbox, change the background, etc. Everything seems to work fine unless I reboot.
I just did a du, boot up is fine, but I found that boot up did take longer, and it took a bit longer for lightdm to start up.
The one thing I did notice was that systemd-analyze is showing me that it took 3mins to boot up. Now, usually its between 3-7 seconds. systemd-analyze blame show the two big culprits are;
terroreek@darthvader ~ ❯❯❯ systemd-analyze blame ⏎
3min 312ms user@105.service
30.232s systemd-rfkill.service
when I look at the user@105.service status (systemctl status user@105.service) I am seeing the following.
● user@105.service - User Manager for UID 105
Loaded: loaded (/lib/systemd/system/user@.service; static; vendor preset: enabled)
Active: failed (Result: timeout) since Fri 2015-10-09 00:14:57 EDT; 4min 27s ago
Process: 3551 ExecStart=/lib/systemd/systemd --user (code=killed, signal=KILL)
Main PID: 3551 (code=killed, signal=KILL)
Oct 09 00:13:27 darthvader systemd[3551]: Stopped target Default.
Oct 09 00:13:27 darthvader systemd[3551]: Stopped target Basic System.
Oct 09 00:13:27 darthvader systemd[3551]: Reached target Shutdown.
Oct 09 00:13:27 darthvader systemd[3551]: Stopped target Timers.
Oct 09 00:13:27 darthvader systemd[3551]: Stopped target Sockets.
Oct 09 00:13:27 darthvader systemd[3551]: Stopped target Paths.
Oct 09 00:14:57 darthvader systemd[1]: user@105.service: State 'stop-final-sigterm' timed out. Killing.
Oct 09 00:14:57 darthvader systemd[1]: Failed to start User Manager for UID 105.
Oct 09 00:14:57 darthvader systemd[1]: user@105.service: Unit entered failed state.
Oct 09 00:14:57 darthvader systemd[1]: user@105.service: Failed with result 'timeout'.
Also seeing the following for the systemd-rfkill.service
● systemd-rfkill.service - Load/Save RF Kill Switch Status
Loaded: loaded (/lib/systemd/system/systemd-rfkill.service; static; vendor preset: enabled)
Active: failed (Result: timeout) since Fri 2015-10-09 00:12:24 EDT; 12min ago
Docs: man:systemd-rfkill.service(
Process: 753 ExecStart=/lib/systemd/systemd-rfkill (code=killed, signal=TERM)
Main PID: 753 (code=killed, signal=TERM)
Oct 09 00:11:54 darthvader systemd[1]: Starting Load/Save RF Kill Switch Status...
Oct 09 00:12:24 darthvader systemd[1]: systemd-rfkill.service: Start operation timed out. Terminating.
Oct 09 00:12:24 darthvader systemd[1]: Failed to start Load/Save RF Kill Switch Status.
Oct 09 00:12:24 darthvader systemd[1]: systemd-rfkill.service: Unit entered failed state.
Oct 09 00:12:24 darthvader systemd[1]: systemd-rfkill.service: Failed with result 'timeout'.
I am thinking its the systemd update. -edit- sometimes boot up completely fails and I just get
systemd[1]: Failed to subscribe to NameOwnerChanged signal: Connection Timed out
over and over again.
I believe this bug is the relevant bug; https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=801354
Do you folks have rpcbind package installed? (I don't and I'm not experiencing the issue). It may just be a symptom, but there are error messages related to rpcbind in the OP's output and in the bug report linked in the last post. I would try removing it if not needed.
Quote from: seasons on 2015/10/09, 07:20:54
Do you folks have rpcbind package installed? (I don't and I'm not experiencing the issue). It may just be a symptom, but there are error messages related to rpcbind in the OP's output and in the bug report linked in the last post. I would try removing it if not needed.
I removed rpcbind and it didn't help.
so until we find the root cause for this we blame systemd? :P
why dont you try downgrading it? maybe xserver-xorg-core too
I ran into the same problems after the systemd upgrade and downgrading systemd from 227 back to 226-4 did the trick.
I got the 226-4 versions of libpam-systemd, libsystemd0, libsystemd0:i386 and systemd from debian snapshot http://snapshot.debian.org/
Hope it helps!
cs
Quote from: cs on 2015/10/09, 08:43:11
I ran into the same problems after the systemd upgrade and downgrading systemd from 227 back to 226-4 did the trick.
I got the 226-4 versions of libpam-systemd, libsystemd0, libsystemd0:i386 and systemd from debian snapshot http://snapshot.debian.org/ (http://snapshot.debian.org/)
Hope it helps!
cs
nice..so putting it on hold for now
you can also find it in testing repos
systemd:
Installed: 226-4
Candidate: 227-1
Version table:
227-1 0
500 http://debian.netcologne.de/debian/ unstable/main amd64 Packages
*** 226-4 0
500 http://debian.netcologne.de/debian/ testing/main amd64 Packages
100 /var/lib/dpkg/status
Hi all,
I can also confirm the problems with the new version 227-1 of systemd. It is either /home is not mounted or the root filesystem mounted read-only when the system is booted.
Are fixed until the problems can you systemd put on hold with the following command:
# apt-mark hold systemd
Probably related, perhaps some of you who are affected could add logs or information:
Debian Bug #801354 (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=801354) - libpam-systemd, boot and services fail with "Connection timed out"
Debian Bug #801361 (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=801361) - systemd, attempts to unmount filesystems at end of boot
Good to know I'm not the only one.
I'm into hospital this morning so hopefully I'll be able to try downgrading systemd some time this evening( providing the don't keep me in).
der_bud: Will do bearing in mind the above.
Hello der_bud,
the debian team is working on a patched version that will adress the two issues:
systemd (227-2) UNRELEASED; urgency=medium
* Revert "sd_pid_notify_with_fds: fix computing msg_controllen", it causes
connection errors from various services on boot. (Closes: #801354)
* debian/tests/boot-smoke: Check for failed unmounts. This reproduces
#801361 (but not in a minimal VM, just in a desktop one).
The package isn't uploaded yet.
greetings
musca
I'm guessing these are the relevant entries from journalctl -b -p err
ct 09 07:32:23 p690 systemd[3039]: Failed unmounting /sys/fs/fuse/connections.
Oct 09 07:32:23 p690 systemd[3039]: Failed unmounting /dev/mqueue.
Oct 09 07:32:23 p690 systemd[3039]: Failed unmounting /dev/hugepages.
Oct 09 07:32:23 p690 systemd[3039]: Failed unmounting /sys/kernel/debug.
Oct 09 07:32:23 p690 systemd[3039]: Failed unmounting /run/user/118.
Oct 09 07:32:23 p690 systemd[3039]: Failed unmounting /home.
Hello,
now also the second issue got adressed:
* Revert "core: add a "Requires=" dependency between units and the
slices they are located in". This causes user systemd instances to try and
unmount system mounts (and succeed if you login as root). (Closes: #801361)
systemd, systemd-sysv, libsystemd0, libnss-myhostname, libpam-systemd, udev, libudev1 have been uploaded in version 227-2 and packages will be available from the usual mirrors after the next sync (about 15:30 UTC, i think).
@ghstryder: yes.
greetings
musca
Thanks musca!
After a couple of minor changes, all is well.
Quote162 upgraded, 0 newly installed, 0 to remove and 0 not upgraded
Lightdm and Siduction are booting up correctly but I am still seeing that systemd-rfkill.service is timing out.
-edit- looks like systemd-rfkill.service is tied to my bluetooth adapter.
I upgraded systemd to 227.2 this morning and everything seems OK. My /home is now mounted and I can get to the desktop.
I'm still getting the following errors:
Oct 10 09:46:32 westfield2 systemd[1]: rpcbind.service: Job rpcbind.service/start deleted to break ord$
Oct 10 09:46:34 westfield2 systemd[1]: rpcbind.service: Job rpcbind.service/start deleted to break ord$
Oct 10 09:46:40 westfield2 kernel: k10temp 0000:00:18.3: unreliable CPU thermal sensor; monitoring dis$
Oct 10 09:47:00 westfield2 colord-sane[1090]: io/hpmud/pp.c 627: unable to read device-id ret=-1
Ignoring the last two errors (the sensor one is well known and is as old as the moon and the colord-sane one is probably because the scanner isn't connectted) I'm still getting the rcpbind.sefvice errors even though:
systemctl status rcpbind.service
● rcpbind.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
The major breakdown seem to me to be fixed .
That said I think we should consider whether systemd is so too deeply integrated into our systems. For me having a systemd bug cause such a catastrophic breakdown and there be now way I can fix it except to wait for an update to systemd is unexceptable, Perhaps this a matter for opening another topic?
Anyway if no one objects I'll close this topic as solved tomorrow morning (11th Oct).
You have to add in the favour of the two Debian Maintainers of systemd (Michael Bibl & Martin Pitt) that they react very fast. Maybe they do release new versions too fast as well. But that is Debian Unstable.
greetz
devil
Quote from: clubex on 2015/10/10, 11:26:17
I'm still getting the rcpbind.service errors
https://ask.fedoraproject.org/en/question/49946/how-to-delete-a-systemd-service/?answer=49954#post-id-49954