Tag Archives: FreeBSD

TIL zfs-userspace(8)

Antranig Vartanian

September 21, 2025

There’s a lot of work that needs to be done on Jailer.

To start with, I have finally decided on the release model. The main development branch will be main, similar to FreeBSD, and then the releases will be tagged.

I’m not sure how I feel about branches, as I don’t want to have any kind of LTS versions, but I might need to understand if there’s anything missing in FreeBSD that I would need.

Another problem that I have right now is the fact that we don’t have a Port. I’m not happy with Jailer enough to make a Port for it, but maybe in the coming days.

Fix the Git stuff that I don’t care about but I should
Create a stable non-development version/tag
Create more lists like this, because, let’s be honest, no one can work without lists.

I do wonder if I can pull an all nighter and finish everything that I hope to finish. Lets see.

Analyzing FreeBSD’s Package Mirror GeoDNS

One of the things that makes FreeBSD an amazing operating system is its package manager, pkg(8). Many people will keep arguing with me saying that apt or pacman is better, but pkg has some of the best features out there. Don’t even get me started on number of packages. A friend was telling me “but how can I use FreeBSD when you have 1/3rd number of packages that Debian has?”. Again, this is a lie. While Debian has 80K packages, each software is divided into -doc and -dev, while on FreeBSD a simple pkg install will install everything that a package needs.

But let’s not talk about how awesome FreeBSD’s pkg is, let’s talk about it’s issues.

Couple of years ago, I noticed that downloading from FreeBSD package mirror was veeeeeery slow. Turns out that based on GeoDNS Armenia was connecting to the South African package mirror pkg0.jinx.FreeBSD.org by default.

Lucky for me, there was a package to help you figure out which one is fastest for you, named fastest_pkg. I ran it and I learned that the Frankfurt package mirror was the fastest for me.

Initially, I changed the config in FreeBSD.conf as recommended by fastest_pkg, but I’m not a single server guy, I have dozens of hosts with hundreds of jails.

Solution? I asked the Cluster Admins to set Armenia’s GeoDNS to pkg0.fra.FreeBSD.org.

But servers come and go, and the Frankfurt server got decommissioned a while back. Luckily the project now has a server in… Sweden!

But my download speed is slow. Somehow Armenia defaulted back to South Africa.

Is this an issue for me, or is this an issue for everybody?

(Sidenote: turns out fastest_pkg has been broken for a while, ever since pkg moved to a new package format, so we also made a patch and submitted a PR)

During today’s FreeBSD “Ask the experts: AMA” we used RIPE Atlas to measure the speeds from all around the world, to all package mirror hosts.

All of the measurements can be found here.

So, here’s the goal

Measure from around the world to all of the package mirror hosts (done).
Measure from around the world to pkg.FreeBSD.org to get the default GeoDNS host for each area/country/network (in progress, recurring every 6 hours for the next 6 days)
Find mis-configurations between “fastest” and “default” (TODO).

While the big measurements is still running I tried to use my eyes to see if there’s anything GREEN (aka fast connection) in direct package mirror host measurements whilst being YELLOW/RED (aka bad connection) when connecting to pkg.FreeBSD.org.

Unsurprisingly, we found one! It was in Mombasa, Kenya.

Here’s what it looks like while connecting directly:

{
  "fw": 5080,
  "mver": "2.6.2",
  "lts": 25,
  "dst_name": "pkg0.jinx.freebsd.org",
  "ttr": 10639.830509,
  "af": 4,
  "dst_addr": "196.10.53.168",
  "src_addr": "160.119.216.205",
  "proto": "ICMP",
  "ttl": 59,
  "size": 1000,
  "result": [
    {
      "rtt": 47.434297
    },
    {
      "rtt": 47.39957
    },
    {
      "rtt": 47.412379
    },
    {
      "rtt": 47.369089
    },
    {
      "rtt": 47.851894
    }
  ],
  "dup": 0,
  "rcvd": 5,
  "sent": 5,
  "min": 47.369089,
  "max": 47.851894,
  "avg": 47.493445799999996,
  "msm_id": 110487154,
  "prb_id": 22215,
  "timestamp": 1750269643,
  "msm_name": "Ping",
  "from": "160.119.216.205",
  "type": "ping",
  "group_id": 110487154,
  "step": null,
  "stored_timestamp": 1750269644
}

And here it is via GeoDNS’s defaults:

{
  "fw": 5080,
  "mver": "2.6.2",
  "lts": 10,
  "dst_name": "pkg.freebsd.org",
  "ttr": 10465.87807,
  "af": 4,
  "dst_addr": "173.228.147.98",
  "src_addr": "160.119.216.205",
  "proto": "ICMP",
  "ttl": 52,
  "size": 1000,
  "result": [
    {
      "rtt": 240.19025
    },
    {
      "rtt": 244.967071
    },
    {
      "rtt": 236.466414
    },
    {
      "rtt": 238.189775
    },
    {
      "rtt": 233.570938
    }
  ],
  "dup": 0,
  "rcvd": 5,
  "sent": 5,
  "min": 233.570938,
  "max": 244.967071,
  "avg": 238.6768896,
  "msm_id": 110489920,
  "prb_id": 22215,
  "timestamp": 1750270943,
  "msm_name": "Ping",
  "from": "160.119.216.205",
  "type": "ping",
  "group_id": 110489920,
  "step": null,
  "stored_timestamp": 1750270944
}

Clearly, it’s set to pkg0.chi.freebsd.org (173.228.147.98) by default. That’s Chicago. That’s halfway around the world. It took it 238ms on average. While it took the same probe 47ms on average when connecting to pkg0.jinx.freebsd.org directly.

Clearly, I’m not the only one.

Hopefully after collecting data for 6 days, we can start analyzing and improving the GeoDNS setup that we have.

A good suggestion from crest was to have a pkg plugin that does the work of fastest_pkg on the fly. We’ll try that one day.

If anyone is interested in helping me out, feel free to contact me over email, IRC or Discord.

That’s all folks…

Migrating to FreeBSD -CURRENT with pkgbase using Boot Environments

Couple of months ago I got my sister a MacBook Air with the new M4 chip, and she was happier than ever. However, I was the happiest, as she gave me her old MacBook Pro 2015.

As always, I installed FreeBSD on it, it was FreeBSD 14.2-RELEASE at the time, installed the required drivers, attached a Thunderbolt 3 to Ethernet adapter to it (thank you to whoever made it work via the bge drivers) and… nothing. It’s been sitting on the floor for months now.

Last week I decided I’m going to do my FreeBSD work again. I have a todo list with 20+ action items in them.

But first thing first, I need a -CURRENT machine so I can do my development.

So, here’s a simple guid on how to migrate to FreeBSD -CURRENT using PkgBase.

First we need to setup the pkgbase repo, so pkg can use it. I will be using pkg.FreeBSD.am, which is my regional pkg cache, but other than that, everything is pretty standard.

root@mbp15srv0:~ # mkdir -p /usr/local/etc/pkg/repos
root@mbp15srv0:~ # cat - > /usr/local/etc/pkg/repos/FreeBSD-base.conf
FreeBSD-base {
    url = "pkg+https://pkg.freebsd.am/FreeBSD:15:amd64/base_latest/";
    mirror_type = "srv";
    signature_type = "fingerprints";
    fingerprints = "/usr/share/keys/pkg";
    enabled = yes;
}

Next up, we create and mount a Boot Environment, that way, if we screw something up, we can always come back.

root@mbp15srv0:~ # bectl list
BE      Active Mountpoint Space Created
default NR     /          2.99G 2025-04-20 22:45
root@mbp15srv0:~ # bectl create 15.0-CURRENT-pkgbase
root@mbp15srv0:~ # bectl mount 15.0-CURRENT-pkgbase
/tmp/be_mount.6i35

But before we start, let’s make sure we’re on the latest version of pkg, since the devs keep adding cool features for PkgBase.

root@mbp15srv0:~ # pkg install pkg
Updating FreeBSD.am repository catalogue...
Fetching meta.conf:   0%
FreeBSD.am repository is up to date.
Updating FreeBSD-base repository catalogue...
Fetching meta.conf:   0%
FreeBSD-base repository is up to date.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 2.1.2 -> 2.1.4 [FreeBSD.am]

Number of packages to be upgraded: 1

12 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching pkg-2.1.4.pkg: 100%   12 MiB  12.5MB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 2.1.2 to 2.1.4...
[1/1] Extracting pkg-2.1.4: 100%
Updating FreeBSD.am repository catalogue...
Fetching meta.conf:   0%
FreeBSD.am repository is up to date.
Updating FreeBSD-base repository catalogue...
Fetching meta.conf:   0%
FreeBSD-base repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The most recent versions of packages are already installed

We need to do pkg update so we get the repo metadata. We’ll also set the ABI to 15, since we’ll be installing that.

root@mbp15srv0:~ # env ABI="FreeBSD:15:amd64" pkg update -f -r FreeBSD-base
pkg: Setting ABI requires setting OSVERSION, guessing the OSVERSION as: 1500000
pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Updating FreeBSD-base repository catalogue...
pkg: Repository FreeBSD-base has a wrong packagesite, need to re-create database
Fetching meta.conf: 100%    179 B   0.2kB/s    00:01    
Fetching data.pkg: 100%   52 KiB  53.2kB/s    00:01    
Processing entries:   0%
Newer FreeBSD version for package FreeBSD-zoneinfo:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1500043
- running userland: 1500000
Ignore the mismatch and continue? [y/N]: y
Processing entries: 100%
FreeBSD-base repository update completed. 554 packages processed.
FreeBSD-base is up to date.

Now we can do the installation using -r, which will set the root directory. I’m also using -Fy which will fetch-only, and assume yes for everything.

root@mbp15srv0:~ # env ABI="FreeBSD:15:amd64" pkg -r /tmp/be_mount.6i35/ install -Fy -r FreeBSD-base -g 'FreeBSD-*'                                                    
pkg: Setting ABI requires setting OSVERSION, guessing the OSVERSION as: 1500000                                                                                        
pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended                                                                               
Updating FreeBSD-base repository catalogue...                                                                                                                          
Fetching meta.conf:   0%                                                                                                                                               
FreeBSD-base repository is up to date.                                                                                                                                 
FreeBSD-base is up to date.                                                                                                                                            
The following 554 package(s) will be affected (of 0 checked):                                                                                                          
                                                                                                                                                                       
New packages to be INSTALLED:                                                                                                                                          
        FreeBSD-acct: 15.snap20250612001522 [FreeBSD-base]                                                                                                             
        FreeBSD-acct-dbg: 15.snap20250612001522 [FreeBSD-base]                                                                                                         
        FreeBSD-acct-man: 15.snap20241026125659 [FreeBSD-base]                                                                                                         
        FreeBSD-acpi: 15.snap20250612160001 [FreeBSD-base]                                                                                                             
        FreeBSD-acpi-dbg: 15.snap20250612160001 [FreeBSD-base]                                                                                                         
[...]

        FreeBSD-zfs-dev-lib32: 15.snap20250612160001 [FreeBSD-base]
        FreeBSD-zfs-lib32: 15.snap20250612160001 [FreeBSD-base]
        FreeBSD-zfs-man: 15.snap20250531144220 [FreeBSD-base]
        FreeBSD-zoneinfo: 15.snap20250521200023 [FreeBSD-base]

Number of packages to be installed: 554

The process will require 6 GiB more space.
1 GiB to be downloaded.
[1/507] Fetching FreeBSD-kernel-minimal-15.snap20250613001933.pkg: 100%   39 MiB 861.9kB/s    00:47    

[...]

[371/484] Fetching FreeBSD-smbutils-dev-lib32-15.snap20250612160001.pkg: 100%   86 KiB  87.8kB/s    00:01    
[372/484] Fetching FreeBSD-libsqlite3-dbg-lib32-15.snap20250612160001.pkg: 100%    1 MiB   1.2MB/s    00:01    
Checking integrity... done (0 conflicting)

And finally, I can do the actual installation. Again, not forgetting about ABI, and BACKUP_LIBRARIES with BACKUP_LIBRARY_PATH.


root@mbp15srv0:~ # env ABI="FreeBSD:15:amd64" BACKUP_LIBRARIES=true BACKUP_LIBRARY_PATH=/tmp/be_mount.6i35/usr/local/lib/compat/pkg pkg -r /tmp/be_mount.6i35/ install
-Uy -r FreeBSD-base -g 'FreeBSD-*'                                                                
pkg: Setting ABI requires setting OSVERSION, guessing the OSVERSION as: 1500000                                                                                        
pkg: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended                                                                               
Checking integrity... done (0 conflicting)                                                                                                                             
The following 554 package(s) will be affected (of 0 checked):                                                                                                          
                                                                                                                                                                       
New packages to be INSTALLED:                                                                                                                                          
        FreeBSD-acct: 15.snap20250612001522 [FreeBSD-base]                                                                                                             
        FreeBSD-acct-dbg: 15.snap20250612001522 [FreeBSD-base]                                                                                                         
        FreeBSD-acct-man: 15.snap20241026125659 [FreeBSD-base]                                                                                                         
[...]

Number of packages to be installed: 554                                            
                                                                                   
The process will require 6 GiB more space.                                
[1/554] Installing FreeBSD-acct-15.snap20250612001522...                
[1/554] Extracting FreeBSD-acct-15.snap20250612001522: 100%               
[2/554] Installing FreeBSD-acct-dbg-15.snap20250612001522...            
[2/554] Extracting FreeBSD-acct-dbg-15.snap20250612001522: 100%         
[3/554] Installing FreeBSD-acct-man-15.snap20241026125659...       
[3/554] Extracting FreeBSD-acct-man-15.snap20241026125659: 100%          
[4/554] Installing FreeBSD-acpi-15.snap20250612160001...                
[4/554] Extracting FreeBSD-acpi-15.snap20250612160001: 100%              
[5/554] Installing FreeBSD-acpi-dbg-15.snap20250612160001...            

[...]

[553/554] Installing FreeBSD-zfs-man-15.snap20250531144220...
[553/554] Extracting FreeBSD-zfs-man-15.snap20250531144220: 100%
[554/554] Installing FreeBSD-zoneinfo-15.snap20250521200023...
[554/554] Extracting FreeBSD-zoneinfo-15.snap20250521200023: 100%

Done!

Now we can chroot in and do some changes.

chroot /tmp/be_mount.6i35/ /bin/sh

We will see a lot of files ending with .pkgsave, these are the old files. Some things are required for us, such as sshd_config, master.passwd, sysctl.conf. Now, this is a pretty fresh installation, so I don’t need to do other changes, but you might!

root@mbp15srv0:/ # cp /etc/ssh/sshd_config.pkgsave /etc/ssh/sshd_config
root@mbp15srv0:/ # cp /etc/master.passwd.pkgsave /etc/master.passwd
root@mbp15srv0:/ # cp /etc/group.pkgsave /etc/group
root@mbp15srv0:/ # pwd_mkdb -p /etc/master.passwd
root@mbp15srv0:/ # cp /etc/sysctl.conf.pkgsave /etc/sysctl.conf

I think this is a good time to delete the other .pkgsave files. The FreeBSD Wiki says that I also need to delete /boot/kernel/linker.hints

# find / -name '*.pkgsave' -delete
# rm /boot/kernel/linker.hints

Back on the host, outside the chroot, I can finally activate the BE, at least temporarily, so only for the next boot.

root@mbp15srv0:~ # bectl list
BE                   Active Mountpoint         Space Created
15.0-CURRENT-pkgbase -      /tmp/be_mount.6i35 4.62G 2025-06-13 13:05
default              NR     /                  3.07G 2025-04-20 22:45
root@mbp15srv0:~ # bectl unmount 15.0-CURRENT-pkgbase
root@mbp15srv0:~ # bectl activate -t 15.0-CURRENT-pkgbase
Successfully activated boot environment 15.0-CURRENT-pkgbase
for next boot
root@mbp15srv0:~ # bectl list
BE                   Active Mountpoint Space Created
15.0-CURRENT-pkgbase T      -          4.62G 2025-06-13 13:05
default              NR     /          3.07G 2025-04-20 22:45

A reboot… with fingers crossed!

shutdown -r now

The machine is up and running!

root@mbp15srv0:~ # uname -a
FreeBSD mbp15srv0.evn0.loc.illuriasecurity.com 15.0-CURRENT FreeBSD 15.0-CURRENT main-n277903-a14573de29de GENERIC amd64

Let’s make this BE the default now.

root@mbp15srv0:~ # bectl list
BE                   Active Mountpoint Space Created
15.0-CURRENT-pkgbase N      /          4.62G 2025-06-13 13:05
default              R      -          3.08G 2025-04-20 22:45
root@mbp15srv0:~ # bectl activate 15.0-CURRENT-pkgbase
Successfully activated boot environment 15.0-CURRENT-pkgbase
root@mbp15srv0:~ # bectl list
BE                   Active Mountpoint Space Created
15.0-CURRENT-pkgbase NR     /          7.54G 2025-06-13 13:05
default              -      -          666M  2025-04-20 22:45

Finally, I will re-install pkg (not sure if that’s needed) and update my packages.

pkg bootstrap -f
pkg upgrade

All good! Now I can get to work 🙂

Special thanks to everyone who wrote the Wiki article and to ivy on the FreeBSD IRC channel for answering my questions.

That’s all folks…

Antranig Vartanian

June 12, 2025

A while back I needed to get the input voltage from one of our UPSes, so I used bsnmpwalk(1) to get the information needed and ran it in a script with a loop and sleep. Running it in tmux(1), of course.

#!/bin/sh

lastoff="maybe"

while true;
do
  inpvol=$(bsnmpwalk -o quiet -s public@172.20.42.101 1.3.6.1.2.1.33.1.3.3.1.3)

  [ $? != 0 ] && \
    curl -s -X POST \
    https://api.telegram.org/botXXX:YYY/sendMessage \
    -d chat_id=-ZZZ \
    -d text="Something is wrong with the SNMP server"

  [ "${inpvol}" -lt 200 ] && \
    curl -s -X POST \
    https://api.telegram.org/botXXX:YYY/sendMessage \
    -d chat_id=-ZZZ \
    -d text="Power seems to be off. I see Input Voltage as ${inpvol}" && \
    lastoff="true"

  [ "${inpvol}" -ge 200 ] && [ "${lastoff}" == "true" ] && \
    curl -s -X POST \
    https://api.telegram.org/botXXX:YYY/sendMessage \
    -d chat_id=-ZZZ \
    -d text="Power back on. I see Input Voltage as ${inpvol}" && \
    lastoff="false"

  printf "%s  ---  %s\n" "$(date)" "${inpvol}"
  sleep 60
done

This got the job done, but I guess there’s place for improvement (leave a reply).

Anyways, I kept forgetting that I need to run the script in tmux after reboots, so I decided to use daemon(8).

touch /etc/rc.local
chmod +x /etc/rc.local
cat - >> /etc/rc.local
#!/bin/sh

daemon -u nobody -r -R 5 -f -t ups-notifier -o /var/log/ups_notifier.log /usr/local/bin/ups_notifier.sh

Again, there’s a place for improvement, specifically I can use a proper rc.d(8) script, yet again, this gets the job done.

Gotta say, I love the simplicity of FreeBSD.

Enabling DTrace in Python on FreeBSD (again)

Last night I was running our usual Greybeard AMA on FreeBSD’s Discord server, when someone asked “I’ve been using Linux for years, but I also like FreeBSD. what can I do for FreeBSD, and what can FreeBSD do for me, as a Python Full Stack Developer?”

I started talking about FreeBSD Jails, ZFS, Boot Environments and more, but I also wanted to focus on DTrace. The ability to dynamically trace on production still sounds like magic to me. I know it isn’t, I’ve read the code and the papers, but the fact that I can ask the operating system questions on the fly, without recompiling, is amazing.

So when I was demoing DTrace, I also wanted to show the DTrace integrations with Python. Little did I know that the Python package on FreeBSD was not compiled with the --with-dtrace option.

As a sane person, I setup a Jail (using Jailer, of course), cloned the FreeBSD ports tree and added the --with-dtrace option back. It did not compile.

The first issue that we encountered was the following:

--- Include/pydtrace_probes.h ---
dtrace: option requires an argument -- s

ah yes, looking over the Makefile we see the following

Include/pydtrace_probes.h: $(srcdir)/Include/pydtrace.d
    $(MKDIR_P) Include
    $(DTRACE) $(DFLAGS) -o $@ -h -s $<
    : sed in-place edit with POSIX-only tools
    sed 's/PYTHON_/PyDTrace_/' $@ > $@.tmp
    mv $@.tmp $@

(you can also view the source here)

As far as I can tell, the $< thingie does not work with BSD Make, so how about if we try using gmake instead?

I did the following changes to the Makefile in the FreeBSD Ports tree. Basically adding gmake into USES=.

 USES=          compiler:c11 cpe ncurses pathfix pkgconfig \
-               python:${PYTHON_DISTVERSION:R},env readline shebangfix ssl tar:xz
+               python:${PYTHON_DISTVERSION:R},env readline shebangfix ssl tar:xz gmake

Now let’s try compiling again.

We get the following errors now:

ld: error: undefined symbol: __dtraceenabled_python___function__entry

Ah yes, linking issues.

After an hour of digging we learned that the DTRACE_OBJS= variable is set to… nothing. But it needs to be set to Python/pydtrace.o. I was not able to fix this issue properly (in configure, configure.ac, or whatever madness that GNU Autotools use), so I just changed the line manually in the Makefile.

Now let’s try compiling again.

We get the following error:

./Python/sysmodule.c:223:24: warning: passing 'const char *' to parameter of type 'char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]

Someone in the Discord chat recommended that I disable the LTO (Link-Time Optimization) option.

I did make clean, I patched that line again, and here we go, one more time.

It’s compiling, it’s compiling, it’s compiling… aaaaand we’re done! Let’s do make install

===>  Installing for python311-3.11.11
===>  Checking if python311 is already installed
===>   Registering installation for python311-3.11.11
[python.srv0.hackerspace.am] Installing python311-3.11.11...

good! now let’s try DTrace with Python!

In one terminal, I ran Python, and in another one I got the following

# dtrace -l -n 'python*:::'
   ID   PROVIDER            MODULE                          FUNCTION NAME
85302 python8737 libpython3.11.so.1.0                         sys_audit audit
85303 python8737 libpython3.11.so.1.0                  sys_audit_tstate audit
85304 python8737 libpython3.11.so.1.0          _PyEval_EvalFrameDefault function-entry
85305 python8737 libpython3.11.so.1.0            dtrace_function_return function-return
85306 python8737 libpython3.11.so.1.0          _PyEval_EvalFrameDefault function-return
85307 python8737 libpython3.11.so.1.0                   gc_collect_main gc-done
85308 python8737 libpython3.11.so.1.0                   gc_collect_main gc-start
85309 python8737 libpython3.11.so.1.0  PyImport_ImportModuleLevelObject import-find-load-done
85310 python8737 libpython3.11.so.1.0  PyImport_ImportModuleLevelObject import-find-load-start
85311 python8737 libpython3.11.so.1.0          _PyEval_EvalFrameDefault line

Woohoo! Now I’m happy!

Okay, so what did we learn?

There’s a little bit of GNUMake-ism in Python’s Makefile. Either we have to use gmake in FreeBSD when building Python or we need to submit an alternative to the upstream
Die GNU Autotools… I mean… the GNU Autotools usage in Python has an issue, where the generated Makefile does not set the DTRACE_OBJS correctly. Can anyone help with this? As a Pascal guy, reading GNU configure files makes me wanna die.
LTO is problematic. Solutions?

The real question is, how can we enable DTrace by default in FreeBSD packages for Python. DTrace is one of our market advantages and we should find a way to enable it everywhere we can.

This was a fun Greybeard AMA, where we touched multiple parts of the system. Looking forward for next week!

That’s all folks…

Changing FreeBSD’s rcorder without patching

I was upgrading my jails, when I noticed that the WriteFreely instance for օրագիր.հայ was not running. I jexec’d into the jail and noticed that the writefreely process was not running at all, doing a simple service writefreely start made it work. Why?

Turns out that WriteFreely needs MySQL to be running during startup, and I assume it wasn’t. By running rcorder I was able to see the boot process.

# rcorder /usr/local/etc/rc.d/* 2>/dev/null
writefreely
rsyncd
mysql-server
garb

So, my first instinct was to patch the /usr/local/etc/rc.d/writefreely script and add mysql into the REQUIRE line, but then I thought to myself, I can’t be the only person who had this problem, right? I mean, I know that the script will be overwritten during the next upgrade. What’s the actual solution here?

After searching a bit, I found the article Override rc order in FreeBSD, so based on that, I created the following file: /usr/local/etc/rc.d/__writefreely which has the following content

#!/bin/sh

# PROVIDE: __writefreely
# REQUIRE: mysql
# BEFORE: writefreely

This is a much cleaner way to do things, let’s check the rcorder again

# rcorder /usr/local/etc/rc.d/* 2>/dev/null
mysql-server
rsyncd
garb
__writefreely
writefreely

Much, much better. After restarting the jail, however, I noticed that WriteFreely is still not running… huh?

Oh, of course, I just needed to do chmod +x __writefreely

And now it works.

FreeBSD-Update and ~200 Jails

Initially, when I heard about freebsd-rustdate I was very skeptical. I have a fear of “Written in <new hip language>”. I thought, however, I’ll wait, and when the time comes, I will try and see how it works.

For the last couple of days I’ve been updating hosts and jails for my customers and my company, and one of the best resources I found was the FreeBSD Update page on FreeBSD’s Wiki, specially the “freebsd-update Reverse Proxy Cache” section. It has saved me hours when updating the hosts. For some hosts we even did an NFS mount of /var/db/freebsd-update/files directory.

But when it came to upgrading the jails, I realized that this is going to take a very long time. Each host has at least 15 jails, up to 50. There’s a host which has 100+ jails.

Upgrading all of them was going to take a very, very long time. So I ended up doing some research. Here were my options.

Build FreeBSD once and run make install everywhere else using NFS and DESTDIR (I used to do this years ago)
Migrate to PkgBase (we’ve started doing this, but we’re not done yet, and it will take a while)
Nuke the Jails, start fresh, and just move the data (this could work, and I will do that in the future, but now I need to update ~200 jails in the coming 3 days)
Somehow, make freebsd-update run faster.

As you have guessed, I went for the last option. Uncle Dave reminded me of freebsd-rustdate again, and I decided to give it a try. Even before starting, my good friend Daniel wrote in our group chat:

@dch my guy. You just saved me several hours per year of flipping back and forth between terminals waiting for the next part of a freebsd-update upgrades to finish running on a million systems.

I arrived to my parent’s house, installed freebsd-rustdate on a host, and tested it on a single jail. Here is my initial reaction

holy fuck freebsd-rustdate is fucking fast

Like I said, I hate “rewrite in <new hip language>”, but clearly, this time it’s a winner.

And frankly speaking, my Jail manager, jailer, does have the same problems that freebsd-update has. It’s much, much slower when you have to manage 100+ jails. I will, however, not rewrite it in another language (for now, and if I do, it will be in Oberon). Although I might end up spending some good amount of time optimizing it 🙂

Kudos to Matthew Fuller, amazing work. And I have to mention, when I was thinking about moving to FreeBSD more than a decade ago, his rant “BSD for Linux Users” was the deciding factor for me, and I’ve been using FreeBSD ever since.

That’s all folks…

dtrace.conf is back as dtrace.conf(24)

Woke up middle of the night to grab a cup of water, decided to check Mastodon, and what do I see?

dtrace.conf(24) Tickets, Wed, Dec 11, 2024 at 9:00 AM

This makes me very happy! I love seeing DTrace in the wild, and having more DTrace content out there is beneficial to everyone in the DTrace community.

Obviously, being a Syrian with passport issues, I will not be able to attend, but hopefully everything will be recorded and published online. I’ll try to make it to dtrace.conf(28).

Have fun everyone!

Antranig Vartanian

October 6, 2024

Initially, Jailer has had a single image format to download, the “FreeBSD base image”, also known as base.txz.

Now we’re trying to integrate PkgBase, OCI images, Jailer binary images, Jailer source images (jailerfile), Linux bootstrap images, and regular tarballs.

This is the point where I just want to kill myself. This is harder than expected.

Linux has a package management problem. I’m having a “too many registry types” problem.

Let’s see how it goes.

#Jailer #FreeBSD