A Beast of a Week!

July 14, 2019

Hello again everyone! As promised, another blog post is here.

In the past few years, I’ve been dipping my feet into the world of the open source BSD operating systems. I’ve tried out FreeBSD, OpenBSD, NetBSD, and a little bit of DragonFly BSD as well as well as some TrueOS here and there.

Anyway, before I ramble further, last week was all about FreeBSD. At the start of the week, I finally got a nice surprise in the mail: the FreeBSD stickers I requested from the FreeBSD Foundation! I now have more stickers to accompany the RUNBSD stickers I received a while back, one of which has already made its way to my OpenBSD laptop, the Toshiba Portege M400.

Various FreeBSD stickers on top of the envelope they came in.

Various FreeBSD stickers on top of the envelope they came in.

Toshiba Portege M400 running OpenBSD with a RUNBSD sticker on the arm rest.

Toshiba Portege M400 running OpenBSD with a RUNBSD sticker on the arm rest. Window manager is cwm.

The FreeBSD stickers come in two versions. One is a cutout of the FreeBSD logo, and the other is a hexagon shaped sticker of the same logo. I plan to use one of them on a laptop with FreeBSD that I have at work, an old Dell Latitude D620.

A big thanks to the FreeBSD Foundation for sending these to me.

Unfortunately, all was not rainbows and unicorns as the week progressed, as evidenced by my ordeal with upgrading a FreeBSD server from 11.2 to 12.0.

Going against my gut, I decided to go from 11.2 to 12.0 instead of the newly-released 11.3 on this old Dell PowerEdge 2800 so that my servers and desktops were all version-consistent. Since our district is on summer break (only custodial and office staff are working aside from me), I figured this was a good time to perform the upgrade at this site location. As I began the upgrade process with freebsd-update, all seemed to go well.

All until it was time to reboot.

KVM showing output from the FreeBSD virtual console with "freebsd-update -r 12.0-RELEASE upgrade" at the ready.

Ready to begin the upgrade to FreeBSD 12.0 via “freebsd-update”.

KVM monitor showing patches being applied as part of the FreeBSD 12.0 upgrade.

Patches being applied as part of the FreeBSD 12.0 upgrade process.

Once I rebooted after the first stage of updates took place, my fears were realized…

The system would not boot!

The bootloader came up fine with the FreeBSD logo and it was able to begin the boot process when I hit Enter. However, after it loaded the kernel, ZFS modules (zfs.ko and opensolaris.ko), and entropy, the spinning pipe that would then go into the rest of the boot process just froze after a second. No amount of restarting would fix it. Thankfully (and with a huge sigh of relief), I was able to boot with kernel.old (the 11.2 kernel) and get to a login prompt.

Time for damage control. I began scouring the Internet and searching for solutions. I found some answers on updating the bootcode for the ZFS drives to support 12.0, but that didn’t work. After exhausting all options on the World Wide Web, I decided to hit the #freebsd IRC channel on Freenode. I usually idle in there during the week, but I decided to chime up and ask for help. I also requested some help from the FreeBSD folks on Mastodon. Everyone went beyond the call of duty to help me with my problem (a testament to the entire BSD community, not just the FreeBSD folks). Unfortunately, none of the tips helped me get a bootable 12.0 system. Loading the modules manually from the boot loader prompt didn’t help, nor did replacing the loader binary with the latest one from 12.0-STABLE. Still, I learned a lot thanks to them, and I’m eternally grateful for all of their help.

KVM showing output of the first stage of the 11.3 upgrade. Ready to reboot in order to begin the second stage.

Finishing the first stage of the 11.3 upgrade after rolling back from the botched 12.0 upgrade. Ready to reboot for the second stage with fingers crossed.

Down but not out, I made peace with my gut and decided to attempt an upgrade to 11.3 from the bootable 11.2 kernel (kernel.old). This was my last resort before going thermonuclear, something I was hoping to avoid at all costs.

I remembered that “freebsd-update” has a “rollback” command, so I ran “freebsd-update rollback” to undo all the 12.0 changes. Once that finished, I ran “freebsd-update -r 11.3-RELEASE upgrade” and crossed my fingers. Everything went through fine as with the 12.0 upgrade and I was prompted to reboot after the 11.3 kernel updates were installed. I typed “shutdown -r now” and hit Enter, fingers crossed.

The moment of truth.

Up came the FreeBSD bootloader. I hit Enter to load the new 11.3 kernel and ZFS modules…

KVM showing the FreeBSD 11.3 login prompt and the MOTD after logging in as root.

Success!

HUZZAH! The rest of the FreeBSD boot process proceeded successfully! All devices and modules loaded successfully, the ZFS pool was mounted without issue, and Samba shares were accessible! I was finally greeted with the FreeBSD login and proceeded to log in as root to view the MOTD. I continued the rest of the upgrade until all kernel and userspace updates were installed and then upgraded all ports via the pkg utility.

And peace reigned across the land thereafter.

As a result of this whole ordeal, I’ve decided to keep this server at the 11.x branch until it is completely EOLed, which means I have until 2021 before moving it to the 12.x branch. Hopefully, the boot issue will have been resolved and I can safely upgrade to it, but I’ll be smarter on how I go about doing so, especially with ZFS.

I hear that a certain someone has some books on the subject…

Post-Independence Post

July 5, 2019

A crowd of people facing the fireworks display. Tents, food trucks, and USA-themed flag posts are shown across the crowd along with palm trees and power lines on posts.

Crowds of people look onward beyond the tents and food trucks to enjoy the Independence Day fireworks festivities where we were in the US.

A day late but not a dollar short, I hope everyone in the United States had a safe and wonderful Independence Day on July 4th. Here I am, the day after with coffee in hand, recovering from a wonderful time with friends and family after enjoying the festivities and fireworks as you see in the image above that I took last night. Of course, this is while everyone is working while I enjoy my day off (or almost everyone, unless they’re like me). As a matter of fact, I’ll be enjoying three more days of R&R since I’ve also taken Monday off, and with my sons with me, it’s going to be filled with gaming and other fun.

I’ll be sure to keep things fresh here during this time, assuming I have content to post. 😀