IRC:busybox [2020-05-23]

Booted to busybox - "(initramfs)"-prompt

If you are stuck at busybox showing a "(initramfs)"-prompt during boot, there are several potential reasons and workarounds to get back to your installed linux (mint). The shown textconsole will look similar to this:

BusyBox v1.72.2 (Ubuntu 1:1.27.2-2ubuntu3.2) built-in shell (ash)
Enter 'help' for a list of build-in cammands.

(initramfs)

Notes (1) - before you start

Use a live-mint - create backups

If you have a working live-mint available (the install media you did use for setup), you should prefer to make use of it for repair, as busybox does only offer a limited set of tools and commands.

Before doing any attempts to repair a broken setup, it's suggested to create backups of your important data first (if they are of any value to you).

If you are going to use a live-mint for repair, you should still read this tutorial to the end, as some of the mentioned hints will be important at a live-mint too.

Keyboard layout en-US

If you are using a foreign keyboard layout, where the expected setup has not happened due to the current issue, you might need to lookup some keys:

logo - [click to enlarge]

[click to enlarge]

If you are using a live-mint, you can add the required keyboard layout using the keyboard settings tool from the menu and activate the layout via the offered panel-icon afterwards.

Overview

There are muliple potential reasons for being stuck at busybox, where you should check the following list in the given order to sort out (and hopefully fix) the issue.

  1. Broken kernel setup - read additionally: Notes (3) - free space and side-effects
  2. Errors on filesystem - read additionally: Notes (2) - busybox - partitions and filesystems
  3. No free space on filesystem - read additionally: Notes (3) - free space and side-effects

Be aware that the underlying reason might be a mixture/combination of above list and you might need to check all given suggestions to word around your boot-failure.

Broken kernel setup

You should first check if selecting a different/previous kernel from grub-menu advanced section would just boot (as this doesn't require you to change anything atm and is an easy test).

If this is working, you could use the your regular installed linux (mint) for cleanup and fixing issues. Where you could still use the given information to locate potential reasons.

The potential reason for a kernel to fail to boot is, that an update or installation of a new kernel or it's related kernel-modules failed, resulting in an broken initial ramdisk of the default-kernel (the most current one).

To test a different kernel, you have to reboot by pressing CTRL+ALT+DEL.

The instructions given at IRC:bootgrub.htm will show how to enter/show grub-menu on boot.

The IRC:bootkernel.htm-page gives directions for selecting a different/previous kernel-version.

Notes (2) - busybox - partitions and filesystems

Available partitions

To determine available partitions at busybox, you can use cat /proc/partitions, which will return some similar output:

BusyBox v1.72.2 (Ubuntu 1:1.27.2-2ubuntu3.2) built-in shell (ash)
Enter 'help' for a list of build-in cammands.

(initramfs) cat /proc/partitions
major minor  #blocks name

 11        0     1989088 sr0
252        0    15728640 vda
252        1    15726952 vda1
(initramfs) 

Your shown disk-devices and partitions will likely be named different (as you will in most cases use linux at a bare metal installation).

Physical disks are typically named sda, sdb, ...

Partitions of the physical disk sda are named accordingly named sda1, sda2, ... - partitions of the physical disk sdb are named sdb1, sdb2,... etc.

If cat /proc/partitions doesn't list any diskdevices, your are out of luck with using busybox, as you will not be able to perform any changes at your disks/partitions atm.

Supported filesystem types

To determine which filesystem types are useable at busybox at the moment, you can use cat /proc/filesystems, which will return some similar output:

# ... removed previous output here
(initramfs) cat /proc/filesystems
nodev	sysfs
nodev	tmpfs
nodev	bdev
nodev	proc
nodev	cgroup
nodev	cgroup2
nodev	cpuset
nodev	devtmpfs
nodev	configfs
nodev	debugfs
nodev	tracefs
nodev	securityfs
nodev	sockfs
nodev	bpf
nodev	pipefs
nodev	ramfs
nodev	hugetlbfs
nodev	devpts
	ext3
	ext2
	ext4
	squashfs
	vfat
nodev	ecryptfs
	fuseblk
nodev	fuse
nodev	fusectl
nodev	efivarfs
nodev	mqueue
nodev	pstore
	btrfs
nodev	autofs
(initramfs) 

Where the rows missing an entry nodev in the first column are the filesytems of interest in most cases.

If cat /proc/filesystems doesn't list your required filesystem, your are out of luck with using busybox, as you will not be able to perform any changes at the related partitions atm.

Mounted partitions

To determine currently mounted partitions, you can use the mount command without parameters:

# ... removed previous output here
(initramfs) mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1977528k,nr_inodes=494382,mode=755)
devpts on /dev/pts type devtps (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=403060k,mode=755)
(initramfs) 

The above output shows a typical initial situation at busybox, with no physical partition being mounted.

Additional physical partitions would be shown after the line: tmpfs on /run ...

Errors on filesystem

There are known situations, which might lead to a corrupted filesystem:

Check first if a related partition is mounted and unmount it, if needed:

# ... removed previous output here
(initramfs) mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1977528k,nr_inodes=494382,mode=755)
devpts on /dev/pts type devtps (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=403060k,mode=755)
/dev/vda1 on /mntvda1 type ext4 (rw,relatime)
(initramfs) cd /
(initramfs) umount /dev/vda1
(initramfs) mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1977528k,nr_inodes=494382,mode=755)
devpts on /dev/pts type devtps (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=403060k,mode=755)
(initramfs) 

With busybox, fsck is only able to test partitions for current supported/available filesystems (read above Supported filesystem types).

To accept suggested repairs does only require you to press ENTER (aka RETURN).

To give give you some idea, what you might get shown, an ext4 fs has been corrupted intentionally (your output will very likely differ):

# ... removed previous output here
(initramfs) fsck /dev/vda1
fsck from util-linux 2.31.1
/dev/vda1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Directory inode 21288, block #0, offset 0: directory has no checksum.
Fix? yes

Directory inode 21288, block #0, offset 0: directory corrupted
Salvage? yes

Missing '.' in directory inode 21288.
Fix? yes

Setting filetype for entry '.' in ??? (21288) to 2.
Missing '..' in directory inode 21288.
Fix? yes

Setting filetype for entry '..' in ??? (21288) to 2.
Pass 3: Checking directory connectivity
'..' in /usr/src/linux-headers-5.3.0-28/drivers/media/i2c/adv748x (21288) is  (0), should be /usr/src/linux-headers-5.3.0-28/drivers/media/i2c (21285).
Fix? yes

Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Inode 2 ref count is 23, should be 24.  Fix? yes

Inode 21285 ref count is 9, should be 8.  Fix? yes

Pass 5: Checking group summary information

/dev/vda1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/vda1: 447108/983040 files (0.2% non-contiguous), 3732994/3931648 blocks
(initramfs)

Before you reboot, you should additionally check for available free space on your partitions too.

To reboot after performing a fsck press CTRL+ALT+DEL.

Notes (3) - free space and side-effects

Minimum free space

The suggested minimum free space for a partition is 15% to allow a fluent operation.

At the common ext4 filesystem a 5% spare is reserved for administrative tasks only.

If the free space at an ext4-filesystem (root-fs-partition or a separate home-fs-partition) is lower than this 5% spare, a user will face issues to login using the gui. This does mean: you have take care to free up at least a bit more than the 5% spare to be get back to a regular gui, as it wont fix the issue entirely if you can boot afer deleting files, but are limited to a console-login afterwards.

Boot partition

Some setups require to use a separate boot-partition to keep kernel versions and related files there. The default setup will typically use a smaller partition size, which might result in running out of diskspace at installation of a new kernel version, or at updating related files after changes (update) of a kernel module.

No free space on filesystem

If your filesystem did run out of free diskspace, this might result in a partial broken boot-setup. In most cases some previous update or software installation did not complete, leaving behind an imcomplete, partial broken setup.

Besides the root-filesystem itself, any other filesystem, like a separate boot-partition, bios-partition, efi-partition, etc., being used as part of your boot-setup, might be affected by running out of diskspace.

A typical candidate is using a separate boot-partition, which might be setup too small to hold a larger number of kernel-versions.

To mount a partition you need first to create a mount-point (a folder) and then mount the partition there. After that, using mount without parameters should show the additional partition being used:

# ... removed previous output here
(initramfs) cat /proc/partitions
major minor  #blocks name

11        0     1989088 sr0
252        0    15728640 vda
252        1    15726952 vda1
(initramfs) mkdir /mntvda1
(initramfs) mount /dev/vda1 /mntvda1
(initramfs) mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1977528k,nr_inodes=494382,mode=755)
devpts on /dev/pts type devtps (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=403060k,mode=755)
/dev/vda1 on /mntvda1 type ext4 (rw,relatime)
(initramfs) 

To check for available diskspace it's enough to run df and have a look at the columns Available and Use%:

# ... removed previous output here
(initramfs) df
Filesystem           1024-blocks    Used Available Use% Mounted on
udev                   1977528         0   1977528   0% /dev
tmpfs                   403060       128    402932   0% /run
/dev/vda1             15414216  15226552         0 100% /mntvda1
(initramfs) 

If you notice there is no free space left on the device, you very likely found the reason for you boot-issue. If there are other partitions used at your setup, you should check each of them for available free space.

There are three typical cases why a filesystem did run out of free space:

  1. Filesystem is setup too small for a longer use
  2. Filled up the filesystem (by mistake)
  3. An unknown source did consume the diskspace

Filesystem is setup too small for a longer use

If your used filesystem is smaller than 30 GiB, it might be time to stop here and increase the partition size. If you are using a fresh install, it might be easier to perform a fresh installation, using some more resonable partition sizes for your setup.

If your partition sizes are big enough, or you cant just give it more space atm, you could try doing some cleanup to get back to a working linux installation ....

Filled up the filesystem (by mistake)

The easy case - you filled up the filesystem by mistake - should be fixable by just gettin rid of some less important data.

An unknown source did consume the diskspace

If it's not clear, what did lead to consuming the diskspace, one can use the du command to sum up used diskspace and sort the output in ascending order.

Note: As the du command does sum up all files/folder below the given start folder it will take some time to finish. With a slower disk and some more files, this migh take up some minutes, just let it finish.

For a first overview it does make sense to show the summary for 2 nesting levels and investigate the results further for details.

# ... removed previous output here
(initramfs) df
Filesystem           1024-blocks    Used Available Use% Mounted on
udev                   1977528         0   1977528   0% /dev
tmpfs                   403060       128    402932   0% /run
/dev/vda1             15414216  15226552         0 100% /mntvda1
(initramfs) du -d 2 /mntvda1|sort -n
4	/mntvda1/cdrom
4	/mntvda1/dev/mapper
4	/mntvda1/dev/pts
# ... removed output here ....
139524	/mntvda1/home
242256	/mntvda1/var/log
331440	/mntvda1/lib/firmware
444636	/mntvda1/var/lib
480472	/mntvda1/var/cache
488424	/mntvda1/boot
797988	/mntvda1/usr/src
1182012	/mntvda1/var
1515628	/mntvda1/lib/modules
1900108	/mntvda1/lib
2023132	/mntvda1/usr/lib
2035968	/mntvda1/usr/share
5024976	/mntvda1/usr
6400904	/mntvda1/timeshift/snapshots
6400932	/mntvda1/timeshift
15185592	/mntvda1
(initramfs)

Note: if you notice filesystem-errors to show up while running the du command, you should first check the Errors on filesystem section, before you continue with your cleanup.

While the du command is usefull to find folders cosuming much disk space, there might be situations where single files are to blame (see below logfiles), which might be not easy to spot, as they are located at some folder where it's expected to have some more diskspace being used.

Luckily the find command is availabel at busybox, which can be used to search for larger files. As there are some system files which are quiet large, it helps to search for files being bigger than 300 MiB.

As the busybox version of find, does only support the definition of a filesize in KB - accordingly searching for files being at least 300 MB, does require to use a value of 300000.

A simpler version of the command, does just list matching files:

# ... removed previous output here
(initramfs) find /mntvda -type f -size +300000
/mntvda1/home/demo/.xsession-errors
/mntvda1/var/log/syslog
(initramfs)

If you prefer to get the file sizes output in one call, instead of having to look them up manually by using the ls -al command, you can combine this with the find-results into one call:

# ... removed previous output here
(initramfs) find /mntvda -type f -size +300000 -exec ls -al {} \;
-rw-r-----    1    951150 /mntvda1/home/demo/.xsession-errors
-rw-r-----    1   1151150 /mntvda1/var/log/syslog
(initramfs)

There are two cases, which need some extra attention if being the reason for filling up a filesystem:

  1. timeshift
  2. log-files

timeshift

If you are using timeshift with a smaller root-fs-partition (anything up to 30 GiB), you might be affected by a nasty side-effect.

If you ever used timeshift to create a system snapshot, it does turn on to automatically create up to 5 daily snapshots (without any further notice).

The snapshots are created with some delay by a so called "cron-job". As the used "cron-job" does run as the user root, it's not limited by the 5% spare and is thus allowed to fill up the entire filesystem.

If you cant find any other larger files, that you could remove to get back to a working installation, you have to delete one or more of the available snapshots.

# ... removed previous output here
(initramfs) du -d 2 /mntvda1/timeshift
6377928	/mntvda1/timeshift/snapshots/2020-05-23_00-01-23
6377932	/mntvda1/timeshift/snapshots
4	/mntvda1/timeshift/snapshots-monthly
4	/mntvda1/timeshift/snapshots-ondemand
4	/mntvda1/timeshift/snapshots-weekly
4	/mntvda1/timeshift/snapshots-hourly
4	/mntvda1/timeshift/snapshots-daily
4	/mntvda1/timeshift/snapshots-boot
6377960	/mntvda1/timeshift/
(initramfs) rm -rf /mntvda1/timeshift/snapshots/2020-05-23_00-01-23/
(initramfs)

After you did delete one or more snapshots, you should not reboot immediately, but first get rid of the according cron-job too, to avoid to run into the same issue again.

Timeshift cron-jobs are located below the /etc/cron.d directory, thus seaching all files below there and deleting the timeshift-related cronjobs does solve this (is the same what timeshift does itself, if you turn off the related schedule option):

# ... removed previous output here
(initramfs) find /mntvda1/etc/cron.d -type f
/mntvda1/etc/cron.d/.placeholder
/mntvda1/etc/cron.d/timeshift-hourly
/mntvda1/etc/cron.d/anacron
(initramfs) rm /mntvda1/etc/cron.d/timeshift-hourly
(initramfs)

logfiles

Sometimes it happens that an installations does log a huge number of errors or warnings. In extreme corner-cases, this might fill up the entire available diskspace.

Logfiles are typically written from processes being owned by the root-user and are not limited by the 5% spare limit.

Before blindly deleting logfiles you should first check the contents for the reason leading to this.

To output the last 200 lines of logentries (if it's the assumed flooding of the logfiles by repeated errors/warnings this should catch it normally) - sticking to to above samples, the according commands would be (you have to use your findings instead):

# ... removed previous output here
(initramfs) tail -n 200 /mntvda1/home/demo/.xsession-errors
# ... many errors here ....
Cjs-Message: 00:50:24.861: JS LOG: [LookingGlass/info] Loaded applet network@cinnamon.org in 18 ms
Cjs-Message: 00:50:24.893: JS LOG: Unknown network device type, is 14
Cjs-Message: 00:50:25.029: JS LOG: [LookingGlass/info] Loaded applet menu@cinnamon.org in 168 ms
Cjs-Message: 00:50:25.029: JS LOG: [LookingGlass/info] AppletManager started in 566 ms
Cjs-Message: 00:50:25.054: JS LOG: [LookingGlass/info] Cinnamon took 800 ms to start
cinnamon-session[1181]: WARNING: t+7,45185s: Detected that screensaver has appeared on the bus
Cjs-Message: 00:50:44.858: JS LOG: [LookingGlass/info] Adding XAppStatusIcon: mintUpdate.py (org.x.StatusIcon.PID-1599-0)
# ... many errors here ....
(initramfs) rm /mntvda1/home/demo/.xsession-errors

To be clear about it - deleting the logfiles wont make the underlying error/issue go away.

You have to find and fix the reason, as your free diskspace will (depending on the error-rate) get eaten up very fast again otherwise.

After you finished your cleanup, you should unmount currently used filesystem and reboot by pressing CTRL+ALT+DEL.

Related information

IRC:bootgrub.htmentering grub-menu on boot
IRC:bootkernel.htmSelect a kernel on boot

© 1998-2020 by ORCUS® GmbH - www.orcus.de (imprint) - we strictly do not track (period) - details: privacy - overview IRC: IRC support

note-1: not tracking does mean too - we are strictly NOT using direct external links within the content = we bother you having to copy&paste offered URLs