Solving a refreshing login screen on an Open Media Vault 5 server

Solution

Free up space on your root volume. See this great forum post: [How-To] Fix full OS filesystem – GUI login loop

Avoid the problem in the future

Simple – make sure your root volume doesn’t fill up.

Some options:

• Buy a large hard drive
• Make sure the server’s outbound email / notification services are functioning (and check your email)
• Login and verify the root volume has free space allocated with “df” infrequently
• Use a server monitoring service to report the disk space
• Spend a bunch of time configuring a Graphana instance at home like this or this and put it in a place where it can be observed.

Problem

When I attempted to login to my web interface for my Open Media Vault server, the screen would refresh and bring me back to an empty login screen. No fun! How am I supposed to fiddle around with Node-Red, Home Assistant and PiHole?

After running df, I found my root volume was full. Looking at the list of volumes, / was used at 100%, with 0 blocks available:

Last login: Wed Dec 14 16:46:39 2022
[email protected]:~$ df
Filesystem      1K-blocks      Used  Available Use% Mounted on
udev               990264         0     990264   0% /dev
tmpfs              202936      2992     199944   2% /run
/dev/sda1        72400968  68738728          0 100% /
tmpfs             1014668         0    1014668   0% /dev/shm
tmpfs                5120         0       5120   0% /run/lock

This command lists volumes and sorts to show only the top 10 volumes

[email protected]:/$ sudo du -xhd1 | sort -hr | head -n 10
65G     .
42G     ./sharedfolders
19G     ./var
4.5G    ./usr
286M    ./boot
13M     ./home
8.3M    ./data
8.1M    ./etc
3.5M    ./root
1.7M    ./srv

I can see that my /var directory is taking up 19 gigs.

Using the ncdu utility, /var is reportedly only taking up 4 gigs.

Looking into /var, I see what I expected- a lot of storage taken up in /logs:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /var -----------------------------------------------------------------------------------------------
                         /..
.   3.4 GiB [##########] /log                                                                           . 630.2 MiB [#         ] /cache
. 321.7 MiB [          ] /lib
.  49.5 MiB [          ] /backups
    7.3 MiB [          ] /www
. 128.0 KiB [          ] /spool..

…and /journal is responsible for most of that:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /var/log -------------------------------------------------------------------------------------------
                         /..                                                                                2.8 GiB [##########] /journal
  138.0 MiB [          ]  daemon.log.1
  105.1 MiB [          ]  daemon.log

Another culprit was docker’s filesystems:

ncdu 1.13 ~ Use the arrow keys to navigate, press ? for help
--- /var/lib/docker/overlay2 -------------------------------------------------------------------------
                         /..                                                                              1.1 GiB [##########] /b6be9adc7a3ab5b180028f5a332dc5bbd71bfbaa3784fc7847bba2358b7487e9
  921.3 MiB [#######   ] /401f5e649ba06b0782cf425c45b5dcddeb41351339641806e7b32244c7b9d28a
  916.8 MiB [#######   ] /8c134dd74390b5d10719483cf40a22ae1ebbbf3ff5051555a293d95bc9af43fd
  574.3 MiB [####      ] 

So, I used sudo docker system prune -a -f, resulting in:

sudo docker system prune -a -f

Deleted Containers:
c9e3773f2cbb4ac7067b89bdcf1c0133b0b37c3f6ac446774070d6c754955405
40fd33305c81320dd7a5fb936e1e85042b89c34305fb4a565f97c76f14538fcf
e538f72bb3ca901d42796c9b5fa85afaa0d716dee2bcaac598448a9d63cdb30a

Deleted Images:
untagged: linuxserver/letsencrypt:latest
untagged: homeassistant/home-assistant:latest
<.. Many more entries ..>

Total reclaimed space: 7.122GB

Success!

After freeing up some disk space, I was able to login to Open Media Vault with no issues.