You should first check if selecting a different/previous kernel from grub-menu advanced section would just boot (as this
doesn't require you to change anything atm and is an easy test).
If this is working, you could use the your regular installed linux (mint) for cleanup and fixing issues. Where you could
still use the given information to locate potential reasons.
The potential reason for a kernel to fail to boot is, that an update or installation of a new kernel or it's
related kernel-modules failed, resulting in an broken initial ramdisk of the default-kernel (the most current one).
To test a different kernel, you have to reboot by pressing CTRL+ALT+DEL.
The instructions given at IRC:bootgrub.htm will show how to enter/show grub-menu on boot.
The IRC:bootkernel.htm-page gives directions for selecting a different/previous kernel-version.
Available partitions
To determine available partitions at busybox, you can use cat /proc/partitions, which will return
some similar output:
BusyBox v1.72.2 (Ubuntu 1:1.27.2-2ubuntu3.2) built-in shell (ash)
Enter 'help' for a list of build-in cammands.
(initramfs) cat /proc/partitions
major minor #blocks name
11 0 1989088 sr0
252 0 15728640 vda
252 1 15726952 vda1
(initramfs)
Your shown disk-devices and partitions will likely be named different (as you will in most cases use
linux at a bare metal installation).
Physical disks are typically named sda, sdb, ...
Partitions of the physical disk sda are named accordingly named sda1, sda2, ... - partitions of the physical disk sdb are
named sdb1, sdb2,... etc.
If cat /proc/partitions doesn't list any diskdevices, your are out of luck with using busybox, as you
will not be able to perform any changes at your disks/partitions atm.
Supported filesystem types
To determine which filesystem types are useable at busybox at the moment, you can use cat /proc/filesystems, which will return
some similar output:
# ... removed previous output here
(initramfs) cat /proc/filesystems
nodev sysfs
nodev tmpfs
nodev bdev
nodev proc
nodev cgroup
nodev cgroup2
nodev cpuset
nodev devtmpfs
nodev configfs
nodev debugfs
nodev tracefs
nodev securityfs
nodev sockfs
nodev bpf
nodev pipefs
nodev ramfs
nodev hugetlbfs
nodev devpts
ext3
ext2
ext4
squashfs
vfat
nodev ecryptfs
fuseblk
nodev fuse
nodev fusectl
nodev efivarfs
nodev mqueue
nodev pstore
btrfs
nodev autofs
(initramfs)
Where the rows missing an entry nodev in the first column are the filesytems of interest in most cases.
If cat /proc/filesystems doesn't list your required filesystem, your are out of luck with using busybox, as you
will not be able to perform any changes at the related partitions atm.
Mounted partitions
To determine currently mounted partitions, you can use the mount command without parameters:
# ... removed previous output here
(initramfs) mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1977528k,nr_inodes=494382,mode=755)
devpts on /dev/pts type devtps (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=403060k,mode=755)
(initramfs)
The above output shows a typical initial situation at busybox, with no physical partition being mounted.
Additional physical partitions would be shown after the line: tmpfs on /run ...
There are known situations, which might lead to a corrupted filesystem:
- power drop
- hard-reset by the user (holding power-button or pressing reset-button)
- software-error: application and/or kernel crashed due to a bug
- failing hardware:
- harddisk or ssd is dying
- connection issue - weak cabling/connectors
- psu is failing/dying
- other failing additional hw - interfering with system
- mainboard controller failing
Check first if a related partition is mounted and unmount it, if needed:
# ... removed previous output here
(initramfs) mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1977528k,nr_inodes=494382,mode=755)
devpts on /dev/pts type devtps (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=403060k,mode=755)
/dev/vda1 on /mntvda1 type ext4 (rw,relatime)
(initramfs) cd /
(initramfs) umount /dev/vda1
(initramfs) mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1977528k,nr_inodes=494382,mode=755)
devpts on /dev/pts type devtps (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=403060k,mode=755)
(initramfs)
With busybox, fsck is only able to test partitions for current supported/available
filesystems (read above Supported filesystem types).
To accept suggested repairs does only require you to press ENTER (aka RETURN).
To give give you some idea, what you might get shown, an ext4 fs has been corrupted intentionally
(your output will very likely differ):
# ... removed previous output here
(initramfs) fsck /dev/vda1
fsck from util-linux 2.31.1
/dev/vda1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Directory inode 21288, block #0, offset 0: directory has no checksum.
Fix? yes
Directory inode 21288, block #0, offset 0: directory corrupted
Salvage? yes
Missing '.' in directory inode 21288.
Fix? yes
Setting filetype for entry '.' in ??? (21288) to 2.
Missing '..' in directory inode 21288.
Fix? yes
Setting filetype for entry '..' in ??? (21288) to 2.
Pass 3: Checking directory connectivity
'..' in /usr/src/linux-headers-5.3.0-28/drivers/media/i2c/adv748x (21288) is (0), should be /usr/src/linux-headers-5.3.0-28/drivers/media/i2c (21285).
Fix? yes
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Inode 2 ref count is 23, should be 24. Fix? yes
Inode 21285 ref count is 9, should be 8. Fix? yes
Pass 5: Checking group summary information
/dev/vda1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/vda1: 447108/983040 files (0.2% non-contiguous), 3732994/3931648 blocks
(initramfs)
Before you reboot, you should additionally check for available free space on your partitions too.
To reboot after performing a fsck press CTRL+ALT+DEL.
If your filesystem did run out of free diskspace, this might result in a partial broken boot-setup. In most cases some
previous update or software installation did not complete, leaving behind an imcomplete, partial broken setup.
Besides the root-filesystem itself, any other filesystem, like a separate boot-partition, bios-partition, efi-partition, etc.,
being used as part of your boot-setup, might be affected by running out of diskspace.
A typical candidate is using a separate boot-partition, which might be setup too small to hold a larger number of
kernel-versions.
To mount a partition you need first to create a mount-point (a folder) and then mount the partition
there. After that, using mount without parameters should show the additional partition being
used:
# ... removed previous output here
(initramfs) cat /proc/partitions
major minor #blocks name
11 0 1989088 sr0
252 0 15728640 vda
252 1 15726952 vda1
(initramfs) mkdir /mntvda1
(initramfs) mount /dev/vda1 /mntvda1
(initramfs) mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1977528k,nr_inodes=494382,mode=755)
devpts on /dev/pts type devtps (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=403060k,mode=755)
/dev/vda1 on /mntvda1 type ext4 (rw,relatime)
(initramfs)
To check for available diskspace it's enough to run df and have a look at the columns Available and Use%:
# ... removed previous output here
(initramfs) df
Filesystem 1024-blocks Used Available Use% Mounted on
udev 1977528 0 1977528 0% /dev
tmpfs 403060 128 402932 0% /run
/dev/vda1 15414216 15226552 0 100% /mntvda1
(initramfs)
If you notice there is no free space left on the device, you very likely found the reason for you boot-issue. If there
are other partitions used at your setup, you should check each of them for available free space.
There are three typical cases why a filesystem did run out of free space:
- Filesystem is setup too small for a longer use
- Filled up the filesystem (by mistake)
- An unknown source did consume the diskspace
Filesystem is setup too small for a longer use
If your used filesystem is smaller than 30 GiB, it might be time to stop here and increase the partition
size. If you are using a fresh install, it might be easier to perform a fresh installation, using some
more resonable partition sizes for your setup.
If your partition sizes are big enough, or you cant just give it more space atm, you could try doing some
cleanup to get back to a working linux installation ....
Filled up the filesystem (by mistake)
The easy case - you filled up the filesystem by mistake - should be fixable by just gettin rid of some
less important data.
An unknown source did consume the diskspace
If it's not clear, what did lead to consuming the diskspace, one can use the du command to sum up used
diskspace and sort the output in ascending order.
Note: As the du command does sum up all files/folder below the given start folder it will take some time
to finish. With a slower disk and some more files, this migh take up some minutes, just let it finish.
For a first overview it does make sense to show the summary for 2 nesting levels and investigate the results further for details.
# ... removed previous output here
(initramfs) df
Filesystem 1024-blocks Used Available Use% Mounted on
udev 1977528 0 1977528 0% /dev
tmpfs 403060 128 402932 0% /run
/dev/vda1 15414216 15226552 0 100% /mntvda1
(initramfs) du -d 2 /mntvda1|sort -n
4 /mntvda1/cdrom
4 /mntvda1/dev/mapper
4 /mntvda1/dev/pts
# ... removed output here ....
139524 /mntvda1/home
242256 /mntvda1/var/log
331440 /mntvda1/lib/firmware
444636 /mntvda1/var/lib
480472 /mntvda1/var/cache
488424 /mntvda1/boot
797988 /mntvda1/usr/src
1182012 /mntvda1/var
1515628 /mntvda1/lib/modules
1900108 /mntvda1/lib
2023132 /mntvda1/usr/lib
2035968 /mntvda1/usr/share
5024976 /mntvda1/usr
6400904 /mntvda1/timeshift/snapshots
6400932 /mntvda1/timeshift
15185592 /mntvda1
(initramfs)
Note: if you notice filesystem-errors to show up while running the du command, you should first check the
Errors on filesystem section, before you continue with your cleanup.
While the du command is usefull to find folders cosuming much disk space, there might be situations where single
files are to blame (see below logfiles), which might be not easy to spot, as they are located at some folder
where it's expected to have some more diskspace being used.
Luckily the find command is availabel at busybox, which can be used to search for larger files.
As there are some system files which are quiet large, it helps to search for files being bigger than 300 MiB.
As the busybox version of find, does only support the definition of a filesize in KB - accordingly searching
for files being at least 300 MB, does require to use a value of 300000.
A simpler version of the command, does just list matching files:
# ... removed previous output here
(initramfs) find /mntvda -type f -size +300000
/mntvda1/home/demo/.xsession-errors
/mntvda1/var/log/syslog
(initramfs)
If you prefer to get the file sizes output in one call, instead of having to look them up manually by using
the ls -al command, you can combine this with the find-results into one call:
# ... removed previous output here
(initramfs) find /mntvda -type f -size +300000 -exec ls -al {} \;
-rw-r----- 1 951150 /mntvda1/home/demo/.xsession-errors
-rw-r----- 1 1151150 /mntvda1/var/log/syslog
(initramfs)
There are two cases, which need some extra attention if being the reason for filling up a filesystem:
- timeshift
- log-files
timeshift
If you are using timeshift with a smaller root-fs-partition (anything up to 30 GiB), you might be affected by
a nasty side-effect.
If you ever used timeshift to create a system snapshot, it does turn on to automatically create up to 5 daily
snapshots (without any further notice).
The snapshots are created with some delay by a so called "cron-job". As the used "cron-job" does run as the user
root, it's not limited by the 5% spare and is thus allowed to fill up the entire filesystem.
If you cant find any other larger files, that you could remove to get back to a working installation, you have to
delete one or more of the available snapshots.
# ... removed previous output here
(initramfs) du -d 2 /mntvda1/timeshift
6377928 /mntvda1/timeshift/snapshots/2020-05-23_00-01-23
6377932 /mntvda1/timeshift/snapshots
4 /mntvda1/timeshift/snapshots-monthly
4 /mntvda1/timeshift/snapshots-ondemand
4 /mntvda1/timeshift/snapshots-weekly
4 /mntvda1/timeshift/snapshots-hourly
4 /mntvda1/timeshift/snapshots-daily
4 /mntvda1/timeshift/snapshots-boot
6377960 /mntvda1/timeshift/
(initramfs) rm -rf /mntvda1/timeshift/snapshots/2020-05-23_00-01-23/
(initramfs)
After you did delete one or more snapshots, you should not reboot immediately, but first get rid
of the according cron-job too, to avoid to run into the same issue again.
Timeshift cron-jobs are located below the /etc/cron.d directory, thus seaching all files below there
and deleting the timeshift-related cronjobs does solve this (is the same what timeshift does itself,
if you turn off the related schedule option):
# ... removed previous output here
(initramfs) find /mntvda1/etc/cron.d -type f
/mntvda1/etc/cron.d/.placeholder
/mntvda1/etc/cron.d/timeshift-hourly
/mntvda1/etc/cron.d/anacron
(initramfs) rm /mntvda1/etc/cron.d/timeshift-hourly
(initramfs)
logfiles
Sometimes it happens that an installations does log a huge number of errors or warnings. In extreme corner-cases,
this might fill up the entire available diskspace.
Logfiles are typically written from processes being owned by the root-user and are not limited by the 5% spare limit.
Before blindly deleting logfiles you should first check the contents for the reason leading to this.
To output the last 200 lines of logentries (if it's the assumed flooding of the logfiles by repeated errors/warnings
this should catch it normally) - sticking to to above samples, the according commands would be (you have to
use your findings instead):
# ... removed previous output here
(initramfs) tail -n 200 /mntvda1/home/demo/.xsession-errors
# ... many errors here ....
Cjs-Message: 00:50:24.861: JS LOG: [LookingGlass/info] Loaded applet network@cinnamon.org in 18 ms
Cjs-Message: 00:50:24.893: JS LOG: Unknown network device type, is 14
Cjs-Message: 00:50:25.029: JS LOG: [LookingGlass/info] Loaded applet menu@cinnamon.org in 168 ms
Cjs-Message: 00:50:25.029: JS LOG: [LookingGlass/info] AppletManager started in 566 ms
Cjs-Message: 00:50:25.054: JS LOG: [LookingGlass/info] Cinnamon took 800 ms to start
cinnamon-session[1181]: WARNING: t+7,45185s: Detected that screensaver has appeared on the bus
Cjs-Message: 00:50:44.858: JS LOG: [LookingGlass/info] Adding XAppStatusIcon: mintUpdate.py (org.x.StatusIcon.PID-1599-0)
# ... many errors here ....
(initramfs) rm /mntvda1/home/demo/.xsession-errors
To be clear about it - deleting the logfiles wont make the underlying error/issue go away.
You have to find and fix the reason, as your free diskspace will (depending on the
error-rate) get eaten up very fast again otherwise.
After you finished your cleanup, you should unmount currently used filesystem and reboot by pressing CTRL+ALT+DEL.
![logo - [click to enlarge]](/Entities/pages/main_workarounds/mint_reset_pw/keyboard-1280x552.png "[click to enlarge]")