Tag Archives: ZFS

Installing FreeBSD with Root-on-ZFS on Vultr using iPXE

The title is pretty self explanatory, so let’s get to it, shall we?

I was configuring a server for a customer today, and one of the things I noticed is that FreeBSD was not available for bare-metal.

This got me a bit worried, because we use a lot of FreeBSD on Vultr… Well that’s a lie. We only use FreeBSD on Vultr.

I logged into our company account and noticed that our bare-metals does have FreeBSD as an icon for the image.

So I decided to check the docs and found this:

What operating system templates do you offer?

We offer many Linux and Windows options. We do not offer OpenBSD or FreeBSD images for Vultr Bare Metal. Use our iPXE boot feature if you need to install a custom operating system.

Well, that’s sad, but on the other hand, iPXE will be very useful. We can boot a memdisk such as mfsBSD and install FreeBSD from there.

To start, we need a VM that can host the mfsBSD img/ISO file. I have spun up a VM on Vultr running FreeBSD (altho it can run anything else, it wouldn’t matter), installed nginx on it, downloaded the file so we can boot from it. Here’s the copy-pasta

pkg install -y nginx
service nginx enable && service nginx start
fetch -o /usr/local/www/ \
  https://mfsbsd.vx.sk/files/images/14/amd64/mfsbsd-se-14.0-RELEASE-amd64.img

This should be enough to get started. Oh, if you’re not on FreeBSD then the path might be different, like /var/www/nginx, or something alike. Check your nginx configuration for the details.

Now we need to write an iPXE script and add it into our Vultr iPXE scripts. Here’s what it looks like

#!ipxe

echo Starting MFSBSD
sanboot http://your.server.ip.address/mfsbsd-se-14.0-RELEASE-amd64.img
boot

Finally, we can create a bare-metal that uses our script for iPXE boot.

Don’t forget to choose the right location and plan.

After the machine is provisioned, you need to access the console and you will see the boot process.

The default root password is mfsroot.

To install FreeBSD, you can run bsdinstall. The rest will be familiar for you. Yes, you can use Root-on-ZFS. No, it can’t be in UEFI, you must use GPT (BIOS).

Good luck, and special thanks to Vultr for giving us the chance to use our favorite tools on the public cloud.

That’s all folks…

Reply via email.

Antranig Vartanian

April 26, 2024

I love ZFS…

root@evn0:/var/log/named # du -h -d 1
1.4G    .
root@evn0:/var/log/named # du -A -h -d 1
7.4G    .

Reply via email.

BSDCan 2022 Talks and Scary Thumbnail

I don’t know if it’s YouTube that chose this thumbnail or if it was someone from BSDCan, but I’ve gotta say, I love it! xD

But in all seriousness, you can find my talk “Own The Stack: FreeBSD from a Vendor’s Perspective by Antranig Vartanian (ft. Faraz Vahedi)” on YouTube.

There’s a whole playlist, with each talk more interesting than the other.

Looks like I know what I will be doing this weekend ☺️

Reply via email.

ZFS compression is so good that it cost me 2 hours

So we have this build machine (build0) where we build FreeBSD in Jails and then we mount the src and obj dirs via NFS or we sync them using rsync to destinations so we can run make installworld on not-so-powerful servers.

Couple of days ago we had a network issue at the data center, the switches crashed and we had to reboot them. Turns out I was running rsync on one of our servers, so I decided to make sure that the files were copied.

Like a lazy sysadmin, I run the following commands on both the build0 server, as well as the remote host.

root@build0:~ # du -h -d 0 /usr/local/jails/f130/usr/obj/
 13G    /usr/local/jails/f130/usr/obj/

root@illuriasecurity:~ # du -h -d 0 /usr/obj/
5.5G    /usr/obj/

Hmm, maybe files were not copied properly? So I remove the obj dir and I rsync again.

Looks like the size is 5.5G AGAIN!

So I do a little bit of piping!

root@build0:/usr/local/jails/f130/usr/obj # find . | sort > /tmp/obj_build0.txt

root@illuriasecurity:/usr/obj # find . | sort > /tmp/obj.txt

zvartnots:~ $ scp illuria:/tmp/obj.txt  /tmp/
zvartnots:~ $ scp build0:/tmp/obj_build0.txt /tmp/

zvartnots:~ $ diff /tmp/obj.txt /tmp/obj_build0.txt

Um, no difference?

Looks like the size reported by du was… confusing?

Okay, let’s check the manual of du(1):

     -A		Display the apparent size instead of the disk usage.  This	can be
     		helpful when operating on compressed volumes or sparse files.

Oops, looks like ZFS compression is enabled on my machine…

Let’s try this again!

root@build0:~ # du -h -d 0 -A /usr/local/jails/f130/usr/obj/
 12G    /usr/local/jails/f130/usr/obj/

root@illuriasecurity:~ # du -h -d 0 -A /usr/obj/
 12G    /usr/obj/

Ok! This makes more sense 🙂

Let’s also check with ZFS.

root@illuriasecurity:~ # zfs get compression zroot/usr
NAME       PROPERTY     VALUE     SOURCE
zroot/usr  compression  lz4       inherited from zroot

I wonder what’s the build0 server is doing?

root@build0:~ # zfs get compression zroot/usr
cannot open 'zroot/usr': dataset does not exist

Hn o.O ? Oh yeah, I wonder.

root@build0:~ # mount | grep ' / '
/dev/ufs/rootfs on / (ufs, local, journaled soft-updates)

Okay, this makes much more sense now 🙂

That’s all folks!

Reply via email.

VNET Jail HowTo

So Dan has been tweeting that there’s no good example to get started with VNET Jails with jail.conf, I thought it’s time to write one.

In this example I’ve used FreeBSD 12.1-RELEASE

root@jail-host:~ # freebsd-version
12.1-RELEASE
root@jail-host:~ # uname -a
FreeBSD jail-host 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
root@jail-host:~ #

First thing first, let’s setup a bridge on our host

root@jail-host:~ # sysrc cloned_interfaces="bridge0"
cloned_interfaces:  -> bridge0
root@jail-host:~ # sysrc ifconfig_bridge0="inet 10.0.0.1 netmask 0xffffff00 descr jails-bridge"
ifconfig_bridge0:  -> inet 10.0.0.1 netmask 0xffffff00 descr jails-bridge

Start the bridge0 interface without restarting the other interfaces

root@jail-host:~ # service netif start bridge0

Good! let’s setup a ZFS dataset for Jails 😉

root@jail-host:~ # zfs create -o mountpoint=/usr/local/jails zroot/jails

Good! now let’s fetch the base.txz file. I will be using my closest mirror, you should use yours.

root@jail-host:~ # mkdir /usr/local/jails/.dist-files
root@jail-host:~ # fetch -o /usr/local/jails/.dist-files/FreeBSD-12.1-RELEASE-base.txz http://mirror.yandex.ru/freebsd/releases/amd64/12.1-RELEASE/base.txz

Perfect!

Now, we will extract the base into the jail.

root@jail-host:~ # zfs create zroot/jails/www
root@jail-host:~ # tar xf /usr/local/jails/.dist-files/FreeBSD-12.1-RELEASE-base.txz -C /usr/local/jails/www/

Nicely done! Now let’s setup our /etc/jail.conf 🙂

Here’s my configuration.

# vim: set syntax=sh:
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
allow.raw_sockets;
allow.mount.tmpfs;
mount.devfs;


www {
    $id     = "10";
    $ipaddr = "10.0.0.${id}";
    $mask   = "255.255.255.0";
    $gw     = "10.0.0.1";
    vnet;
    vnet.interface = "epair${id}b";

    exec.prestart   = "ifconfig epair${id} create up";
    exec.prestart  += "ifconfig epair${id}a up descr vnet-${name}";
    exec.prestart  += "ifconfig bridge0 addm epair${id}a up";

    exec.start      = "/sbin/ifconfig lo0 127.0.0.1 up";
    exec.start     += "/sbin/ifconfig epair${id}b ${ipaddr} netmask ${mask} up";
    exec.start     += "/sbin/route add default ${gw}";
    exec.start     += "/bin/sh /etc/rc";

    exec.prestop    = "ifconfig epair${id}b -vnet ${name}";

    exec.poststop   = "ifconfig bridge0 deletem epair${id}a";
    exec.poststop  += "ifconfig epair${id}a destroy";

    host.hostname = "${name}.jail-host";
    path = "/usr/local/jails/${name}";
    exec.consolelog = "/var/log/jail-${name}.log";
    persist;
}

Now let’s start our Jail!

root@jail-host:~ # service jail enable
jail enabled in /etc/rc.conf
root@jail-host:~ # service jail start www
Starting jails: www.
root@jail-host:~ # jls
   JID  IP Address      Hostname                      Path
     1                  www.jail-host                 /usr/local/jails/www

Let’s check the networking 🙂

root@jail-host:~ # ping -c 1 10.0.0.10
PING 10.0.0.10 (10.0.0.10): 56 data bytes
64 bytes from 10.0.0.10: icmp_seq=0 ttl=64 time=0.164 ms

--- 10.0.0.10 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.164/0.164/0.164/0.000 ms

We can do the same from the jail.

root@jail-host:~ # jexec www
root@www:/ # ping -c 1 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=0.087 ms

--- 10.0.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.087/0.087/0.087/0.000 ms

We can also stop all the jails.

root@jail-host:~ # service jail stop
Stopping jails: www.

Okay! Couple of notes 🙂

You can have jail.conf at /etc/jail.conf or /etc/something.jail.conf. The problem with the latter is that if you have jail_enable="YES" in rc.conf without defining jail_list then it will run only the jails in /etc/jail.conf

There are more ways to configure VNET Jails, either with jib or jng, an example is here.

Ideally, it would be nice to have /etc/jail.d/myjail.conf, and I wrote a patch for that (D24570), if you are a FreeBSD developer, please have a look 🙂

Reply via email.

FreeBSD Root-on-ZFS Migration

My home server (running this blog) got old, it has only 2G of RAM and a very old hard drive.

I usually like to use laptops as home server, they get the job done! I know I can’t have fancy things like RAID1, but I usually keep my backups on a separate disk/machine anyway.

So, I got my Dell Latitude E5470 next to my 11 years old Dell Inspiron to start the migration process.

Here’s a simple how-to guide for migrating a FreeBSD system from one machine to another.

Snapshotting The Pool

Okay, on the old host let snapshot the pool.

zfs snap -r zroot@migrate

Let’s save that snapshot in a file.

zfs send -R zroot@migrate | gzip > /usr/local/zroot.zfs.gz

This might take a while, grab some coffee 🙂

Now let’s save the ZFS Boot Environment (bootfs) property

# zpool get bootfs zroot
NAME PROPERTY VALUE SOURCE
zroot bootfs zroot/ROOT/default local

Perfect! now let’s go to the new host! 🙂

Installing FreeBSD

Well, this is simple 🙂 first, install FreeBSD on the new machine. DON’T forget to use ZFS! 🙂

The reason why we are doing this so the FreeBSD installer will handle all the boot loader installation process. If you know how to do this manually, then be my guest (and blog about it).

“Reset” ZFS

Okay, now let’s “reset” that pool

zpool destroy zroot
zpool create -R /mnt zroot /dev/ada0p3 # please check the partition index thought
zpool export zroot # so we can import again with -N
zpool import -N -R /mnt zroot

Restore Old Snapshot

Here we go.

ssh antranigv@pingvinashen "gzcat /usr/local/zroot.zfs.gz" | zfs receive -v -u -F zroot

Aaand we wait again.

So, check you restored all the datasets with zfs list -r zroot

Set the bootfs property.

zpool set bootfs=zroot/ROOT/default zroot

Fingers Crossed and reboot

well, now it’s the time.

shutdown -r now

Done

Hope this helps 🙂

Reply via email.

Setting route interface in FreeBSD

I usually blog for myself, so I don’t forget stuff when I need them 😛

I’m usually connected via WiFi to a network, however, FreeBSD’s WiFi drivers are not very… good. I mean they do work and do essential stuff, but it’s slow 🙂 anyways.

That’s why I’m always connected to the network using a wire as well.

The problem that I always encounter and want to solve is to change the default route’s interface after wiring up 🙂

Here’s how you can do that.

# netstat -4rn | grep default
default            10.10.200.254      UGS       wlan0
# route change default 10.10.200.254 -ifp em0
change net default: gateway 10.10.200.254 fib 0
# netstat -4rn | grep default
default            10.10.200.254      UGS         em0

and so on…

Reply via email.