Category Archives: Tech

Migrating home-servers

As I have mentioned before, I want to blog more, so here it goes.

I’ve been struggling financially lately, with COVID and then the war I’ve thrown away almost all of my savings. One of the decisions that I had to make was moving back to my old place. No one lives here anymore, my parents got their own house, which means I can live with freedom alone and rent-free.

That meant I need to move home servers again. Yes, I’ve always been a home server fan. This blog runs on my home server as well.

While many people argue that running a home server is a complex process compared to the cloud, since you need to pay for electricity and manage hardware, I, however, feel that’s a myth.

My current uptime is

ssh pingvinashen uptime
1:59PM up 48 days, 42 mins, 2 users, load averages: 0.15, 0.18, 0.21

And I only needed to reboot because I had to upgrade since I’m a fan of upgrading whenever there’s a patch to some critical software 🙂

One of the advantages of running a home server in Armenia is the fact that electricity is cheap, so are static IP addresses. I pay 2USD/mo for each IP address and I have many of them.

Usually, I have one static IP per service (Jabber, ZNC, etc.) and one static IP for all web-oriented services such as blogs, websites, etc.

However, norayr also runs a home-server for the community, he runs the Armenian instance of Diaspora*, Mastodon, and SocialHome.

Due to technical limitations at his side of the city, he’s been keeping his home server at my place.

Vartanian LLC, Home-Server as a Service 😛

Anyways, I had to bring his home server to my new/old place as well, which meant that he needs a static IP for his services.

I did not want to call the ISP for a new IP address since the last one I’ve been using was for an Armenian instance of Lobste.rs that I deployed for our community. It’s not very active, but you can’t force people to be active in communities and Armenia does not have the concept of “tech communities” like others do in the west.

That meant that I have to remove an IP from a Jail so norayr can use it.

So I had to migrate some things. I had to use my proxy server IP address and reverse_proxy the traffic to the lobsters’ Jail.

Sounds easy, until I remembered that I run Apache on my host.

I’m not very fluent in Apache, I keep doing mistakes, so I wanted to migrate all of my vhosts to Nginx.

You’d think that it would be easy, and yes it was 🙂

So now, norayr runs his home server and I have migrated the webserver to Nginx in an hour.

For some reason, it feels faster, but I’m still not sure why. I probably had to optimize Apache back in the day, but Nginx’s default configs do seem better.

Now, since many IP addresses have been changed, I have to struggle with SMTP issues. No, SMTP works fine, but Google, just like it keeps breaking the web, it keeps breaking email as well, routing all-good emails to people’s spam folder, eh.

That’s all folks.

* not a footnote but part of the project name.

Blogging Regularly

Ruben blogged recently about blogging regularly and it kind of hit me: Why don’t I blog regularly?

I love blogging. I improved my Armenian by blogging for years, I wanted to be a blogger so bad that I asked my friends to rent me a domain and a hosting service since I didn’t have money when I moved to Armenia after the war.

But yet again, it’s very hard for me to write my thoughts in English. Armenian? Yes, sure, I can write a very complex sentence very easily. English, however, the language that I think in, the language that I grew up having a love&hate relationship with, is not the language that I’m good at writing. I can talk English very well, at least I’ve been told, but writing is not there yet.

There are a lot of points that Ruben made that I love to be more mainstream. Use ANY blogging platform, literally any, as long as they don’t treat you as the product (Medium as an example). Write about anything, everything. I would love to hear about your daily life, how you solve problems, no matter if it’s about that very complex DB issue you’ve been having or the water pipe that has been leaking. They are all interesting.

At the end of the day the internet is the place that allowed everyone to speak. Now we are fighting over who gets to be heard.

But with blogs and RSS, everyone will be.

That’s all folks.

Flickr’s Uploads from everyone from The Future

I like using RSS, I have Miniflux running in a FreeBSD Jail all the time. Lately I noticed that it’s not very handy to open up Flickr every hour to see photos by my friends, so I decided to use their RSS feed.

First of all, I’m so happy that Flickr provides an RSS feed, it’s an amazing standard and I wish more websites and services implemented it, or at least, not remove it 🙂

So I go to Norayr’s Flickr page, and since Firefox removed RSS I do Ctrl-U to see the page source. I search for the term RSS and I find nothing, I search for atom and I find a link!

I copy the link and I add it to my Miniflux feeds. I hit refresh and I see the following.

Okay, so looks like those are photos… from everyone… on Flickr… from the future.

I KNOW that my time is synced, because I always have NTP running. After some digging, I figured that those posts ARE from the future. Flickr calls them “scheduled” posts.

Doing a little more digging, and I realize that I did not copy the full link with id=norayr's_id and now I’m seeing everyone’s feeds.

That’s all folks! 🙂

macOS to FreeBSD migration a.k.a why I left macOS

I think the title tells a lot about the story I’m going to tell you.

This is not a technical documentation for how I migrated from macOS to FreeBSD. This is a high-level for why I migrated from macOS to FreeBSD.

Not so long ago, I was using macOS as my daily driver. The main reason why I got a macbook was the underlying BSD Unix and the nice graphics it provides. Also, I have an iPhone. But they were also the same reasons for why I left macOS.

I did not want to write this post right after the migration, I wanted to take my time, use FreeBSD daily, see if I will ever miss macOS.

Here’s a tweet of mine from 8 months ago

Let’s look at it this way. macOS is becoming less Unix-y every year, date(1) is outdated, there are 100+ Unix processes running by the time the system is booted, most of them are useless for the general user, it has no native package manager (at least MacPorts/homebrew/pkgsrc is out there) and for a power user, there is no proper documentation. Have you ever checked the FreeBSD handbook? Everything is right there!

Okay, the nice graphics part. Have you seen the latest and greatest Big Sur? It feels like eye-candy, it’s not made for power users at all, everything seems to be a distraction now, even the icons. I’m no UI guru, but bringing iOS to the desktop is not for everyone.

So I decided to move to FreeBSD. This is where many people will tell me “Okay but not everything works outside the box”, true! but which OS works outside the box these days anyway? Windows is still a nightmare, setting up macOS took me 3 days the last time, Linux takes way more if you’re building it from scratch. Setting up FreeBSD took me 3 days, however this meant that I will NOT need to change it again for a very, very, VERY long time.

Every time Apple pushed an updated, my pf.conf and automount configs got broken on macOS. They either got deleted or they moved somewhere. Well, the last 2 times it just got deleted.

On FreeBSD, I upgraded from 12.1-RELEASE to 12.2-RELEASE and nothing broke, and in case there were any changes, FreeBSD just asked me what to do about them.

Let’s come back for a second. Unix is outdated and Apple does not care about it, fancy graphics are too fancy now. Doing forensics is almost impossible. And the hardware is, well, not the best out there (have you ever disassembled a MacBook Pro? it’s takes 2 hours to change a battery while I can reassemble my Dell Latitudes and ThinkPads in 30 minutes).

So there was no reason to stay here anymore. I had to migrate. The question is: where?

Linux has systemd, not my favorite thing out there, Windows is privacy nightmare. That left me with 2 major options: Linuxes without systemd (Gentoo, in my case) or BSDs.

Since I run FreeBSD servers anyway, I just migrated to FreeBSD.

Here’s a short review about running FreeBSD on ThinkPad T480s.

  • WiFi: works. not the fastest, but fast enough.
  • Graphics: works.
  • Touchpad: works with multiple fingers AND very configurable via sysctl.
  • BT does discovery and pairs, I still have to try it with non-Apple headphones.
  • COVID-19 era: Zoom, Google Hangouts, Jitsi and all other WebRTC-based video conferencing software works via web as well.
  • Thanks to Linuxulator, I can watch Netflix as well: here’s a screenshot.

Most importantly, it’s Free and Open Source.

It’s been 1 month and 1 day since I last touched my MacBook Pro, so, what do I miss?

  • Better BT support
  • Faster WiFi

That’s it, that’s all missing on a FreeBSD laptop these days. WiFi can do 48Mbps according to ifconfig but I usually get 10-20Mbps. BT pairs with my Apple AirPods but I wish it worked till the end.

Having a nice workstation/laptop is not an easy thing, using macOS means living by Apple rules, Windows is the same for Microsoft. The BSDs gave me the power to be as free as possible 🙂

During the next weeks I’ll try to blog about the actual setup.

P.S. dear Apple employee, in case you’re reading this, please tell your management to update their BSD Unix layer. Some of us still care, some of us are not just Docker people, some of us are not just “modern” web developers. Thanks in advance.

That’s all folks! 🙂

Signal-cli with scli on FreeBSD

So couple of days ago I migrated from macOS on Macbook Pro to FreeBSD on ThinkPad T480s.

Unfortunately, since we are in war, I do not have the time to blog about the migration, although I’m taking notes every day about every change that I do so I can blog later on 🙂

However, one of the biggest concerns for me was running Signal on FreeBSD, as I understnad, Signal people are not interested in supporting the *BSDs.

As any sane person, I started searching the internet for possible solutions and turns out all I need is two pieces of software

The installation is as easy as running

pkg install signal-cli scli

Now for the simple part.

First, you need to link your phone by running

signal-cli link -n "FreeBSD"

It will give an output that says tsdevice:/?uuid=...&pub_key=....

Copy that output, and then in another terminal run

qrencode 'tsdevice:/?uuid=...&pub_key=...' -t ANSI256

You will be represented by a QR Code in the console (cool, aye?).

Using the phone app, link the device by scanning the QR Code.

To receive list of your contacts run

signal-cli -u +myphonenumber receive

Now try to run the TUI interface by running

scli

Side-note: In case you are not able to send or receive messages, you might need to do some DBUS magic.

First, find if you have DBUS running

antranigv@zvartnots:~ $ ps -x -o comm,pid | grep dbus
dbus-launch         53571
dbus-daemon         54064
dbus-daemon         54963

Then, you will need to find the DBUS_SESSION_BUS_ADDRESS environment variable, this is usually set in the DBUS child process, in our case, it’s 54963, so we can use procstat as root

root@zvartnots:~ # procstat -e 54963
  PID COMM             ENVIRONMENT                                          
54963 dbus-daemon      SHELL=/usr/local/bin/bash DBUS_STARTER_ADDRESS=unix:path=/tmp/dbus-TaY0zoKZIb,guid=4f518f874f97170e788a94fb5fa14a3c DISPLAY=:0.0 WMAKER_BIN_NAME=wmaker PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/antranigv/bin WINDOWPATH=9 MAIL=/var/mail/antranigv GTK_THEME=Adwaita:dark DBUS_SESSION_BUS_ADDRESS=unix:path=/tmp/dbus-TaY0zoKZIb,guid=4f518f874f97170e788a94fb5fa14a3c USER=antranigv DBUS_STARTER_BUS_TYPE=session MM_CHARSET=UTF-8 WRASTER_COLOR_RESOLUTION0=4 PWD=/usr/home/antranigv BLOCKSIZE=K LANG=en_US.UTF-8 LOGNAME=antranigv HOME=/home/antranigv

Okay! we have our variable!

Now, we need to set the ENV and we are done, if you use (t)csh then execute

setenv DBUS_SESSION_BUS_ADDRESS unix:path=/tmp/dbus-TaY0zoKZIb,guid=4f518f874f97170e788a94fb5fa14a3c

If you are using bash, run the following

export DBUS_SESSION_BUS_ADDRESS=unix:path=/tmp/dbus-TaY0zoKZIb,guid=4f518f874f97170e788a94fb5fa14a3c

Now, you can run scli again and it will work fine 🙂

Happy Chatting!@#$%

That’s all folks! 🙂

Erlang dbg Intro

If there’s one programming language that changed my life, that’s Erlang. After using Erlang for couple of years, I “moved” to Elixir, which is based on Erlang’s VM.

One the most important aspects of Erlang’s VM is that it’s a “real” VM, there’s a kernel, processes, messaging facilities and many more.

Lately I’ve been debugging a huge Erlang application whose architecture I was not very familiar with and I needed to find a way to see what kind of messages are being sent and received, which Modules and Functions are being called and what are they returning.

So I wanted to write a small How-To for me and you, in case we need it again in the future.

Okay, for this example I’ll be using Elixir TCP Server, a simple TCP server that gets data and sends it back to its origin.

First, let’s clone the repo.

antranigv@zvartnots:prj $ git clone https://github.com/SonaTigranyan/ElixirTcpServer

Okay, now let’s run the server

antranigv@zvartnots:ElixirTcpServer $ iex -S mix
Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Compiling 3 files (.ex)
Generated tcp_server app
Interactive Elixir (1.10.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

Good! By default, the TCP server listens on port 9000, as specified in the Application Tree.

Okay, we can send data now 🙂

antranigv@zvartnots:ElixirTcpServer $ echo test | nc localhost 9000
test

Or in an interactive way!

antranigv@zvartnots:ElixirTcpServer $ nc localhost 9000
First mesage!
First mesage!
Good TCP server!
Good TCP server!
bye
antranigv@zvartnots:ElixirTcpServer $

Good! As you can see the connection is closed when the server gets bye.

Okay, say we want to trace the do_send function, observe what does it get and return.

iex(2)> :dbg.start()
{:ok, #PID<0.191.0>}
iex(3)> :dbg.tracer()
{:ok, #PID<0.191.0>}
iex(4)> :dbg.tpl(TcpServer, :do_send, [{:_, [], [{:return_trace}]}])
{:ok, [{:matched, :nonode@nohost, 1}, {:saved, 1}]}
iex(5)> :dbg.p(:new_processes, :c)
{:ok, [{:matched, :nonode@nohost, 0}]}
iex(6)>
(<0.198.0>) call 'Elixir.TcpServer':do_send(#Port<0.545>,"Message from client!\n")

Okay, first we start the dbg facility, and then we start a tracing server on the local node.

After that, we use function tpl to specify which local calls we want to trace.

And in the end we use the p function to start tracing the calls (c) of all new_processes 🙂

Now, when the do_send function is called, we see what it gets.

And when we send bye, we see the following:

(<0.198.0>) returned from 'Elixir.TcpServer':do_send/2 -> ok

And all of this is happening when the software system is running. In production, we can do the same, by either attaching to the node or connecting to it!

That’s all folks! 🙂

autofs on macOS Catalina

One of the nice things (that used to be) about macOS is how much unix is underneath, but this has been less true each year.

Like any normal human being, I do my development on a real Unix system, FreeBSD. I ended up using FreeBSD VMs that have NFS exports, and I mount those on my macOS. However, there have been issues with Catalina, here’s the main problem:

mount | grep 'map auto_nfs_antranigv'  | wc -l
18

So for some reason in Catalina you can’t do nested mounts, because auto_nfs_antranigv was mounted in /Users/antranigv/nfs, which is inside /Users (also autofs’ed) it was duplicating and ended up eating a lot of CPU 🙂

Here’s the proper way of doing it

First, in /etc/auto_master add the following:

/System/Volumes/Data/netmount		auto_nfs_user

(you might want to change user to your username)

Then, create the following file → /etc/auto_nfs_user

Finally you can set your mounts, for example:

someoneelsecomputer	-fstype=nfs	cloudserver:/usr/home/sysadmin/cloud

here’s a complete example:

antranigv@zvartnots:~ $ cat /etc/auto_master
#
# Automounter master map
#
+auto_master			# Use directory service
#/net				-hosts		-nobrowse,hidefromfinder,nosuid
/home				auto_home	-nobrowse,hidefromfinder
/System/Volumes/Data/nfs	auto_nfs_antranigv
/Network/Servers		-fstab
/-				-static
antranigv@zvartnots:~ $ cat /etc/auto_nfs_antranigv
illuria-dev	-fstype=nfs illuria-dev:/usr/home/antranigv/illuria
devbsd-src	-fstype=nfs devbsd00:/usr/src

Now I’m happy! 🙂

VNET Jail HowTo

So Dan has been tweeting that there’s no good example to get started with VNET Jails with jail.conf, I thought it’s time to write one.

In this example I’ve used FreeBSD 12.1-RELEASE

root@jail-host:~ # freebsd-version
12.1-RELEASE
root@jail-host:~ # uname -a
FreeBSD jail-host 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
root@jail-host:~ #

First thing first, let’s setup a bridge on our host

root@jail-host:~ # sysrc cloned_interfaces="bridge0"
cloned_interfaces:  -> bridge0
root@jail-host:~ # sysrc ifconfig_bridge0="inet 10.0.0.1 netmask 0xffffff00 descr jails-bridge"
ifconfig_bridge0:  -> inet 10.0.0.1 netmask 0xffffff00 descr jails-bridge

Start the bridge0 interface without restarting the other interfaces

root@jail-host:~ # service netif start bridge0

Good! let’s setup a ZFS dataset for Jails 😉

root@jail-host:~ # zfs create -o mountpoint=/usr/local/jails zroot/jails

Good! now let’s fetch the base.txz file. I will be using my closest mirror, you should use yours.

root@jail-host:~ # mkdir /usr/local/jails/.dist-files
root@jail-host:~ # fetch -o /usr/local/jails/.dist-files/FreeBSD-12.1-RELEASE-base.txz http://mirror.yandex.ru/freebsd/releases/amd64/12.1-RELEASE/base.txz

Perfect!

Now, we will extract the base into the jail.

root@jail-host:~ # zfs create zroot/jails/www
root@jail-host:~ # tar xf /usr/local/jails/.dist-files/FreeBSD-12.1-RELEASE-base.txz -C /usr/local/jails/www/

Nicely done! Now let’s setup our /etc/jail.conf 🙂

Here’s my configuration.

# vim: set syntax=sh:
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
allow.raw_sockets;
allow.mount.tmpfs;
mount.devfs;


www {
    $id     = "10";
    $ipaddr = "10.0.0.${id}";
    $mask   = "255.255.255.0";
    $gw     = "10.0.0.1";
    vnet;
    vnet.interface = "epair${id}b";

    exec.prestart   = "ifconfig epair${id} create up";
    exec.prestart  += "ifconfig epair${id}a up descr vnet-${name}";
    exec.prestart  += "ifconfig bridge0 addm epair${id}a up";

    exec.start      = "/sbin/ifconfig lo0 127.0.0.1 up";
    exec.start     += "/sbin/ifconfig epair${id}b ${ipaddr} netmask ${mask} up";
    exec.start     += "/sbin/route add default ${gw}";
    exec.start     += "/bin/sh /etc/rc";

    exec.prestop    = "ifconfig epair${id}b -vnet ${name}";

    exec.poststop   = "ifconfig bridge0 deletem epair${id}a";
    exec.poststop  += "ifconfig epair${id}a destroy";

    host.hostname = "${name}.jail-host";
    path = "/usr/local/jails/${name}";
    exec.consolelog = "/var/log/jail-${name}.log";
    persist;
}

Now let’s start our Jail!

root@jail-host:~ # service jail enable
jail enabled in /etc/rc.conf
root@jail-host:~ # service jail start www
Starting jails: www.
root@jail-host:~ # jls
   JID  IP Address      Hostname                      Path
     1                  www.jail-host                 /usr/local/jails/www

Let’s check the networking 🙂

root@jail-host:~ # ping -c 1 10.0.0.10
PING 10.0.0.10 (10.0.0.10): 56 data bytes
64 bytes from 10.0.0.10: icmp_seq=0 ttl=64 time=0.164 ms

--- 10.0.0.10 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.164/0.164/0.164/0.000 ms

We can do the same from the jail.

root@jail-host:~ # jexec www
root@www:/ # ping -c 1 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=0.087 ms

--- 10.0.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.087/0.087/0.087/0.000 ms

We can also stop all the jails.

root@jail-host:~ # service jail stop
Stopping jails: www.

Okay! Couple of notes 🙂

You can have jail.conf at /etc/jail.conf or /etc/something.jail.conf. The problem with the latter is that if you have jail_enable="YES" in rc.conf without defining jail_list then it will run only the jails in /etc/jail.conf

There are more ways to configure VNET Jails, either with jib or jng, an example is here.

Ideally, it would be nice to have /etc/jail.d/myjail.conf, and I wrote a patch for that (D24570), if you are a FreeBSD developer, please have a look 🙂

FreeBSD Root-on-ZFS Migration

My home server (running this blog) got old, it has only 2G of RAM and a very old hard drive.

I usually like to use laptops as home server, they get the job done! I know I can’t have fancy things like RAID1, but I usually keep my backups on a separate disk/machine anyway.

So, I got my Dell Latitude E5470 next to my 11 years old Dell Inspiron to start the migration process.

Here’s a simple how-to guide for migrating a FreeBSD system from one machine to another.

Snapshotting The Pool

Okay, on the old host let snapshot the pool.

zfs snap -r zroot@migrate

Let’s save that snapshot in a file.

zfs send -R zroot@migrate | gzip > /usr/local/zroot.zfs.gz

This might take a while, grab some coffee 🙂

Now let’s save the ZFS Boot Environment (bootfs) property

# zpool get bootfs zroot
NAME PROPERTY VALUE SOURCE
zroot bootfs zroot/ROOT/default local

Perfect! now let’s go to the new host! 🙂

Installing FreeBSD

Well, this is simple 🙂 first, install FreeBSD on the new machine. DON’T forget to use ZFS! 🙂

The reason why we are doing this so the FreeBSD installer will handle all the boot loader installation process. If you know how to do this manually, then be my guest (and blog about it).

“Reset” ZFS

Okay, now let’s “reset” that pool

zpool destroy zroot
zpool create -R /mnt zroot /dev/ada0p3 # please check the partition index thought
zpool export zroot # so we can import again with -N
zpool import -N -R /mnt zroot

Restore Old Snapshot

Here we go.

ssh antranigv@pingvinashen "gzcat /usr/local/zroot.zfs.gz" | zfs receive -v -u -F zroot

Aaand we wait again.

So, check you restored all the datasets with zfs list -r zroot

Set the bootfs property.

zpool set bootfs=zroot/ROOT/default zroot

Fingers Crossed and reboot

well, now it’s the time.

shutdown -r now

Done

Hope this helps 🙂

Setting route interface in FreeBSD

I usually blog for myself, so I don’t forget stuff when I need them 😛

I’m usually connected via WiFi to a network, however, FreeBSD’s WiFi drivers are not very… good. I mean they do work and do essential stuff, but it’s slow 🙂 anyways.

That’s why I’m always connected to the network using a wire as well.

The problem that I always encounter and want to solve is to change the default route’s interface after wiring up 🙂

Here’s how you can do that.

# netstat -4rn | grep default
default            10.10.200.254      UGS       wlan0
# route change default 10.10.200.254 -ifp em0
change net default: gateway 10.10.200.254 fib 0
# netstat -4rn | grep default
default            10.10.200.254      UGS         em0

and so on…