Subject: Stretch does not boot properly after a recent upgrade

From: Rainer Dorsch <[email-masked]>
Date: Fri, 02 Sep 2016 21:43:56 +0200

Hello,

I have a virtual server at scaleway (actually it is a physical armhf) on which 
I installed jessie and upgraded to stretch a long time back. After a recent 
upgrade broke the boot process. The first failure I see is

[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.

A full log is below:

Welcome to Scaleway' initrd            http://j.mp/scw-initrd-linux
███████╗ ██████╗ █████╗ ██╗     ███████╗██╗    ██╗ █████╗ ██╗   ██╗
██╔════╝██╔════╝██╔══██╗██║     ██╔════╝██║    ██║██╔══██╗╚██╗ ██╔╝
███████╗██║     ███████║██║     █████╗  ██║ █╗ ██║███████║ ╚████╔╝
╚════██║██║     ██╔══██║██║     ██╔══╝  ██║███╗██║██╔══██║  ╚██╔╝
███████║╚██████╗██║  ██║███████╗███████╗╚███╔███╔╝██║  ██║   ██║
╚══════╝ ╚═════╝╚═╝  ╚═╝╚══════╝╚══════╝ ╚══╝╚══╝ ╚═╝  ╚═╝   ╚═╝
>>> Initrd build: v3.11.1-0-g227373e - 2016/06/13
>>> Kernel version: Linux c1-10-1-6-53 4.5.7-std-4 #1 SMP Tue Jul 12 11:00:06 
UTC 2016 armv7l GNU/Linux
>>> Installing busybox...
>>> Cloning initramfs for shutdown......
>>> Mounting pseudo-filesystems......
>>> Mounting /dev...
>>> Mounting /dev/pts...
>>> Mounting /run...
>>> Mounting /sys...
>>> Mounting /proc...
>>> Retrieving system info...
>>> Kernel cmdline:    earlyprintk=ttyS0 console=ttyS0,9600n8 nousb noplymouth 
ip=:::::eth0: boot=local
>>> Physycal memory:   2070028 kB
>>> CPU model:         ARMv7 Processor rev 2 (v7l)
>>> CPU cores:         4

>>> Configuring network...
>>> Retrieving metadata info...
>>> Server ID:        2e95a5c1-497c-4a93-8e20-871b52ff80ea
>>> Private IP:       10.1.6.53
>>> Public IP:        www.xxx.yyy.zzz
>>> Hostname:         scw-e243aa
>>> Commercial type:  C1
>>> Platform:         10
>>> Organization:     9019f06a-xxxx-xxxx-xxxx-xxxxe2475d93
>>> Timezone:         UTC
>>> Server (user) tags:
>>> Configuring GPIOs...
>>> Checking user input for debug/verbose mode...
>>> Checking user input for debug shell...
>>> Signaling the server is kernel-started to the control plane...
>>> Adjusting time (ntp)...
>>> Initializing 'local' root file system...
>>> Attaching nbd0...
>>> Mounting nbd0 root...
>>> Mounting /newroot...
>>> Checking metadata in case we skip to wait for secondary volumes to be 
ready...
>>> Retrieving NBD devices info...
>>> - nbd0: l_ssd 50G
>>> Image name:     Debian Jessie
>>> Image release:  2016-03-16
>>> Usage on /:     10%
>>> Signaling the server is booted to the control plane...
>>> Creating default configuration files on rootfs......
>>> Checking metadata for debug shell...
>>> Checking metadata for debug sshd (dropbear)...
>>> Mounting /newroot/proc...
>>> Mounting /newroot/sys...
>>> Mounting /newroot/run...
>>> Mounting /newroot/dev...
>>> Moving initramfs backup to /run/initramfs for shutdown...
>>> /sbin/init may be broken
>>> make sure /sbin/init is an executable and not an absolute symlink
>>> Switching to linux...
End of Scaleway' initrd
           _ _       _      _          _ _
 ___ _ _ _|_| |_ ___| |_   | |_ ___   | |_|___ _ _ _ _
|_ -| | | | |  _|  _|   |  |  _| . |  | | |   | | |_'_|
|___|_____|_|_| |___|_|_|  |_| |___|  |_|_|_|_|___|_,_|
Welcome to Debian GNU/Linux stretch/sid!
[  OK  ] Reached target Remote File Systems.
[  OK  ] Listening on Journal Socket.
[  OK  ] Listening on Syslog Socket.
[  OK  ] Listening on udev Control Socket.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Reached target Swap.
[UNSUPP] Starting of Arbitrary Executable Fi...m Automount Point not 
supported.
[  OK  ] Reached target Paths.
[  OK  ] Listening on Journal Audit Socket.
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Created slice System Slice.
[  OK  ] Created slice system-postgresql.slice.
         Starting SCW kernel requirements checker...
         Mounting POSIX Message Queue File System...
         Starting Journal Service...
[  OK  ] Created slice system-serial\x2dgetty.slice.
[  OK  ] Reached target Slices.
[  OK  ] Created slice system-getty.slice.
         Starting Load Kernel Modules...
         Mounting Debug File System...
         Starting Create Static Device Nodes in /dev...
         Starting Remount Root and Kernel File Systems...
[  OK  ] Mounted POSIX Message Queue File System.
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
[DEPEND] Dependency failed for Flush Journal to Persistent Storage.
[  OK  ] Started Load Kernel Modules.
[  OK  ] Mounted Debug File System.
[  OK  ] Stopped Journal Service.
         Starting Journal Service...
         Starting Apply Kernel Variables...
[FAILED] Failed to start Create Static Device Nodes in /dev.
See 'systemctl status systemd-tmpfiles-setup-dev.service' for details.
         Starting udev Kernel Device Manager...
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting udev Coldplug all Devices...
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
         Starting Load/Save Random Seed...
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started SCW kernel requirements checker.
[  OK  ] Stopped Journal Service.
         Starting Journal Service...
         Starting Raise network interfaces...
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Started Entropy daemon using the HAVEGE algorithm.
[FAILED] Failed to start Create Volatile Files and Directories.
See 'systemctl status systemd-tmpfiles-setup.service' for details.
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
[  OK  ] Stopped Journal Service.
         Starting Journal Service...
[  OK  ] Reached target System Time Synchronized.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started udev Kernel Device Manager.
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
[  OK  ] Stopped Journal Service.
         Starting Journal Service...
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
[  OK  ] Stopped Journal Service.
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Found device /dev/ttyS0.
[  OK  ] Started udev Coldplug all Devices.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily apt activities.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Listening on UUID daemon activation socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting LSB: Start/stop sysstat's sadc...
         Starting SCW fetch kernel modules from Scaleway mirror...
         Starting System Logging Service...
[  OK  ] Started D-Bus System Message Bus.
         Starting PostgreSQL Cluster 9.5-main...
         Starting SCW generate ssh keys on first boot...
[  OK  ] Started Regular background program processing daemon.
         Starting Login Service...
         Starting SCW fetch ssh keys from metadata...
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Started System Logging Service.
[FAILED] Failed to start SCW fetch kernel modules from Scaleway mirror.
See 'systemctl status scw-sync-kernel-modules.service' for details.
[  OK  ] Started Login Service.
[  OK  ] Started LSB: Start/stop sysstat's sadc.
         Starting Daily apt activities...
         Starting SCW generate machine id...
[  OK  ] Started SCW fetch ssh keys from metadata.
[  OK  ] Started SCW generate machine id.
[  OK  ] Started Raise network interfaces.
[  OK  ] Reached target Network.
[  OK  ] Started BIND Domain Name Server.
[  OK  ] Reached target Host and Network Name Lookups.
[  OK  ] Reached target Network is Online.
         Starting LSB: Start NTP daemon...
         Starting /etc/rc.local Compatibility...
         Starting LSB: Apache2 web server...
         Starting Permit User Sessions...
[  OK  ] Started /etc/rc.local Compatibility.
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Getty on tty1.
[  OK  ] Started Serial Getty on ttyS0.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started LSB: Start NTP daemon.
Debian GNU/Linux stretch/sid scw ttyS0
scw login: 

The first systemd failure shows

root@scw:~# systemctl status systemd-journald.service
● systemd-journald.service - Journal Service
   Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; 
vendor 
   Active: failed (Result: start-limit-hit) since Fri 2016-09-02 21:24:34 
CEST; 
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 2506 (code=exited, status=1/FAILURE)

All failing services:

root@scw:~# systemctl --failed
  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
● apache2.service              loaded failed failed LSB: Apache2 web server
● exim4.service                loaded failed failed LSB: exim Mail Transport 
Age
● haveged.service              loaded failed failed Entropy daemon using the 
HAV
● [email-masked]  loaded failed failed PostgreSQL Cluster 9.5-
main
● scw-sync-kernel-modules.service loaded failed failed SCW fetch kernel 
modules 
● systemd-journald.service     loaded failed failed Journal Service
● systemd-tmpfiles-clean.service loaded failed failed Cleanup of Temporary 
Direc
● systemd-tmpfiles-setup-dev.service loaded failed failed Create Static Device 
N
● systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files 
and 
● systemd-journald-audit.socket loaded failed failed Journal Audit Socket
● systemd-journald-dev-log.socket loaded failed failed Journal Socket 
(/dev/log)
● systemd-journald.socket      loaded failed failed Journal Socket
LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
12 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'

Any hint what could go wrong is welcome...

Thanks
Rainer

-- 
Rainer Dorsch
http://bokomoko.de/

---

From: Rainer Dorsch <[email-masked]>
Date: Fri, 02 Sep 2016 22:19:55 +0200

Hello,

I found the problem: 

https://community.online.net/t/c1-arch-not-starting-journal/2856

A bad entry in /etc/machine-id caused the issue. Seems that scaleway is to 
blame, not Debian or Arch.

Thanks and sorry for the noise
Rainer

On Friday 02 September 2016 21:43:56 Rainer Dorsch wrote:
> Hello,
> 
> I have a virtual server at scaleway (actually it is a physical armhf) on
> which I installed jessie and upgraded to stretch a long time back. After a
> recent upgrade broke the boot process. The first failure I see is
> 
> ● systemd-tmpfiles-setup-dev.service loaded failed failed Create Static
> Device N

-- 
Rainer Dorsch
http://bokomoko.de/