system failed to start due to systemd upgrade!

[dibl]

...
Kann es sein, dass der Eintrag in der fstab sich mit der systemd Unit ins Gehege kommt?

EN: Is it possible that the tmpfs entry in fstab is in conflict with the systemd unit tmp.mount?

Apparently so.

However, here is a good running system that was fully upgraded as of last Sunday (26 MAY) and as you see, it has both the tmpfs entry in fstab, and the tmp.mount unit, with no problem booting to X.

Code Select Expand

System:
  Host: dibl-MOW Kernel: 6.8.10-1-siduction-amd64 arch: x86_64 bits: 64
  Desktop: KDE Plasma v: 5.27.11 Distro: siduction 22.1.2 Masters_of_War -
    kde - (202303151559)
Machine:
  Type: Desktop Mobo: ASUSTeK model: ROG STRIX X299-E GAMING v: Rev 1.xx
    serial: <superuser required> UEFI: American Megatrends v: 1401
    date: 05/21/2018
CPU:
  Info: quad core Intel Core i7-7740X [MT MCP] speed (MHz): avg: 800
    min/max: 800/4500
Graphics:
  Device-1: NVIDIA GP106 [GeForce GTX 1060 6GB] driver: nouveau v: kernel
  Display: x11 server: X.Org v: 21.1.11 with: Xwayland v: 24.1.0 driver: X:
    loaded: modesetting unloaded: fbdev,vesa dri: nouveau gpu: nouveau
    resolution: 1: 1920x1200~60Hz 2: 1920x1080~60Hz
  API: OpenGL v: 4.5 compat-v: 4.3 vendor: mesa v: 24.0.8-1 renderer: NV136
Network:
  Device-1: Intel Ethernet I219-V driver: e1000e
Drives:
  Local Storage: total: 4.1 TiB used: 748.76 GiB (17.8%)
Info:
  Memory: total: 32 GiB available: 31.27 GiB used: 1.95 GiB (6.2%)
  Processes: 276 Uptime: 3h 25m Shell: Bash inxi: 3.3.34

don@dibl-MOW:~$ sudo apt policy systemd
systemd:
  Installed: 256~rc3-2
  Candidate: 256~rc3-5
  Version table:
     256~rc3-5 500
        500 https://deb.debian.org/debian unstable/main amd64 Packages
*** 256~rc3-2 100
        100 /var/lib/dpkg/status

don@dibl-MOW:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=F765-9806                            /boot/efi      vfat    umask=0077                                     0 2
UUID=f1bd917e-47c9-43dd-bed8-b9ccdc518968 /boot          ext2    defaults,noatime                               0 2
UUID=9cdc98b4-ebed-478d-9d14-0ebfb4613543 /              ext4    defaults,noatime,errors=remount-ro,commit=120  0 1
UUID=b7eb0948-83b6-4a2a-8724-12e258a0c5fb /home              ext4        defaults,relatime,errors=remount-ro,commit=120 0 2
UUID=44ee6177-c0e3-47e7-aee7-6c4a74c63c4d /mnt/DATA          btrfs       noatime,compress=lzo,space_cache=v2            0 2
tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777                     0 0

don@dibl-MOW:~$ cat /usr/lib/systemd/system/tmp.mount
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Temporary Directory /tmp
Documentation=https://systemd.io/TEMPORARY_DIRECTORIES
Documentation=man:file-hierarchy(7)
Documentation=https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
ConditionPathIsSymbolicLink=!/tmp
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=swap.target

[Mount]
What=tmpfs
Where=/tmp
Type=tmpfs
Options=mode=1777,strictatime,nosuid,nodev,size=50%%,nr_inodes=1m

[Install]
WantedBy=local-fs.target

I have 3 more systems configured like this one, a desktop and 2 laptops, and they all run error-free.

So, this "conflict" is a new feature of the updated systemd, I guess.

[devil]

The changelog for systemd 256~rc3-3 says:  Make /tmp/ a tmpfs by default. Restore the upstream default and make /tmp/ a tmpfs.
Can be overridden with:

Code Select Expand

touch /etc/systemd/system/tmp.mount
or:

Code Select Expand

systemctl mask tmp.mount

So it seems, it's not a bug, but a feature, that was badly communicated. Can someone test this on bare metal please, I can't free any hardware right now.

[dibl]

...
Can be overridden with:
...

LOL -- yes, AFTER YOU HAVE BOOTED YOUR BORKED SYSTEM!

Ha ha ha --- the joke is on us!   

Thanks, @devil -- mystery solved.

[Fellfrosch]

There is something fishy.

I rolled back with timeshift. Disabled the fstab entry. Enabled the tmp.mount unit via

Code Select Expand

systemctl enable tmp.mount rebooted. Everything looks good. tmp is mounted as tmpfs.
Now I did the update to 256~rc3-5
Reboot. tmp isn't mounted anymore as tmpfs.

Code Select Expand

systemctl list-unit-files | grep bad
tmp.mount                                                                 bad             enabled

So what does that mean, that the unit is in a bad state. How do I get more infos on what systemd dislikes?

[reddark]

The changelog for systemd 256~rc3-3 says:  Make /tmp/ a tmpfs by default. Restore the upstream default and make /tmp/ a tmpfs.
Can be overridden with:

Code Select Expand

touch /etc/systemd/system/tmp.mount
or:

Code Select Expand

systemctl mask tmp.mount

So it seems, it's not a bug, but a feature, that was badly communicated. Can someone test this on bare metal please, I can't free any hardware right now.

bedeutet tmpfs in der fstab wieder aktivieren und dann diese befehle ausführen??

[Fellfrosch]

Dear diary. This is the last entry for today.   
I did

Code Select Expand

touch /etc/systemd/system/tmp.mount
Now tmp.mount wasn't listed as bad anymore, but as masked. I unmasked it via

Code Select Expand

systemctl unmask tmp.mount
After that I rebooted, with still deactivated line in fstab.
Result:
tmp is now mounted as tmpfs like it should. So for now I'm at least satisfied.

@devil or anybody else who has more knowledge then me. As I have already written, I'm no expert regarding systemd. Have You some hints for me?
Why is touching an already existing file changing things?
What does masking do? I thought masking means hiding a process. But it seems that I'm wrong with this assumption.

[scholle1]

Can be overridden with:

Bei mir ist /etc/systemd/system/tmp.mount ein toter Link weil /usr/share/systemd/tmp.mount nicht mehr existiert.
Der Link zeigt auf /usr/share und nicht auf /usr/lib , wo tmp.mount tatsächlich ist!

Dazu:
Absichtlich frische Installation siduction.....-kde-amd64-202405152113.iso auf ext4.
In dieser Installation fehlt die "tmp"-Zeile in der fstab.
(Habe die Zeile bei vielen anderen Installationen seit Jahren in der fstab. Benutze meist Btrfs Dateisystem.)
dann DU
systemd von 256~rc2-3 zu 256~rc3-5

Code Select Expand

Vor DU
# journalctl -b | grep "/tmp"
Mai 29 18:06:05 pc2 systemd[1]: tmp.mount: Directory /tmp to mount over is not empty, mounting anyway.
Mai 29 18:06:05 pc2 systemd[1]: Mounting tmp.mount - Temporary Directory /tmp...
Mai 29 18:06:05 pc2 systemd[1]: Mounted tmp.mount - Temporary Directory /tmp.

Nach DU
# journalctl -b | grep "/tmp"
Mai 29 19:12:54 pc2 systemd[1]: tmp.mount: Failed to open /etc/systemd/system/tmp.mount: No such file or directory
Boot fehlerfrei vor und nach DU.
Also will systemd "tmp.mount" ausführen, aber es ist keine "tmp" Zeile in fstab und nach DU die systemd-Unit "tmp.mount" nicht erreichbar, da der tote Link (s.o.) wie eine Maskierung wirkt. Sieht so aus als wissen sie bei systemd selbst nicht genau was sie da veranstalten.

Nach "rm /etc/systemd/system/tmp.mount" ist die Funktion wieder wie vor dem DU.
Aber
1. Ich hatte vor und nach dem DU kein Bootproblem.
2. Das DU hat die systemd-version 256~rc3-2 und 256~rc3-3 übersprungen. (Von rc2-3 zu rc3-5)

[micspabo]

Thanks for all the hints. They were very helpful to me.

I did the same two steps as Fellfrosch, but unmasking tmp.mount showed

Code Select Expand


Unit tmp.mount does not exist, proceeding anyway.
Removed '/etc/systemd/system/tmp.mount'.


Nevertheless,- after a reboot

Code Select Expand


# systemctl is-enabled tmp.mount
  enabled

# systemctl is-active tmp.mount
  active

# systemctl status tmp.mount
  Active: active (mounted) since Wed 2024-05-29 21:08:08 CEST; 14min ago
  Siduction systemd[1]: Mounting tmp.mount - Temporary Directory /tmp...
  Siduction systemd[1]: Mounted tmp.mount - Temporary Directory /tmp.

# mount | grep tmpfs | grep /tmp
  tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=16085924k,nr_inodes=1048576,inode64)


@Fellfrosch
  masking a service will make it impossible for the system to sideload
  it, even if an unforseen part of the system still tries to start it.

[scholle1]

Code Select Expand

touch /etc/systemd/system/tmp.mount

You have overwritten the dead link to /usr/share/.. with this.

Code Select Expand

systemctl unmask tmp.mount

and the command removes file /etc/systemd/system/tmp.mount.
Another way - same result.
Wie sagt man auf englisch: Well done. 

[reddark]

so siehts jetzt bei mir aus ...

Code Select Expand

$ systemctl is-enabled tmp.mount
generated

$ systemctl is-active tmp.mount
active

$ systemctl status tmp.mount
● tmp.mount - /tmp
     Loaded: loaded (/etc/fstab; generated)
     Active: active (mounted) since Wed 2024-05-29 21:56:15 CEST; 2min 7s ago
Invocation: 1391d16a0edc412a9ebc1d67510a78a7
      Where: /tmp
       What: tmpfs
       Docs: man:fstab(5)
             man:systemd-fstab-generator(8)
      Tasks: 0 (limit: 38092)
     Memory: 8.0K (peak: 2.5M)
        CPU: 5ms
     CGroup: /system.slice/tmp.mount

Mai 29 21:56:15 laptop1 systemd[1]: Mounting tmp.mount - /tmp...
Mai 29 21:56:15 laptop1 systemd[1]: Mounted tmp.mount - /tmp.

$ mount | grep tmpfs | grep /tmp
tmpfs on /tmp type tmpfs (rw,noatime,inode64)

[brutor]

Confirm that the solution from Fellfrosch is working on my system.

had to implement the change ro ==> rw in the boot command to be able to boot.

Many Thanks for this help !

[Fellfrosch]

Danke für die Erklärungen. Bin ich ja froh, dass ich als blindes Huhn trotzdem in die richtige Richtung marschiert bin. 

[eriefisher]

I noticed this morning that the tmp.mount link was dead when I was poking around and repaired it as mentioned. At the time though I wasn't aware of the new fstab line. So now with tmp.mount repaired and the fstab line commented I did a D-U to the current systemd and rebooted. Everything seems normal. Wonderful!

[eriefisher]

A little more poking around.

Dmesg shows nothing I can see but:

Code Select Expand

journalctl -b | grep "/tmp"
May 29 17:36:15 hp8470p systemd-tmpfiles[740]: /usr/lib/tmpfiles.d/legacy.conf:13: Duplicate line for path "/run/lock", ignoring.
May 29 17:36:15 hp8470p systemd-tmpfiles[740]: Opening file "/tmp/.ICE-unix/3583" failed, ignoring: No such device or address

I'm not sure if this is related or not.

[burrowsdav]

I have posted a bug report upstream for this issue here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1072187

The steps I used to resolve the issue are in the report itself.  In a nutshell, /etc/systemd/system/tmp.mount is a broken symlink.  I needed to comment out the fstab entry, and "cp /usr/lib/systemd/system/tmp.mount /usr/share/systemd/tmp.mount" to (optionally) restore tmpfs functionality.