Tag Archives: DNS

Analyzing FreeBSD’s Package Mirror GeoDNS

One of the things that makes FreeBSD an amazing operating system is its package manager, pkg(8). Many people will keep arguing with me saying that apt or pacman is better, but pkg has some of the best features out there. Don’t even get me started on number of packages. A friend was telling me “but how can I use FreeBSD when you have 1/3rd number of packages that Debian has?”. Again, this is a lie. While Debian has 80K packages, each software is divided into -doc and -dev, while on FreeBSD a simple pkg install will install everything that a package needs.

But let’s not talk about how awesome FreeBSD’s pkg is, let’s talk about it’s issues.

Couple of years ago, I noticed that downloading from FreeBSD package mirror was veeeeeery slow. Turns out that based on GeoDNS Armenia was connecting to the South African package mirror pkg0.jinx.FreeBSD.org by default.

Lucky for me, there was a package to help you figure out which one is fastest for you, named fastest_pkg. I ran it and I learned that the Frankfurt package mirror was the fastest for me.

Initially, I changed the config in FreeBSD.conf as recommended by fastest_pkg, but I’m not a single server guy, I have dozens of hosts with hundreds of jails.

Solution? I asked the Cluster Admins to set Armenia’s GeoDNS to pkg0.fra.FreeBSD.org.

But servers come and go, and the Frankfurt server got decommissioned a while back. Luckily the project now has a server in… Sweden!

But my download speed is slow. Somehow Armenia defaulted back to South Africa.

Is this an issue for me, or is this an issue for everybody?

(Sidenote: turns out fastest_pkg has been broken for a while, ever since pkg moved to a new package format, so we also made a patch and submitted a PR)

During today’s FreeBSD “Ask the experts: AMA” we used RIPE Atlas to measure the speeds from all around the world, to all package mirror hosts.

All of the measurements can be found here.

So, here’s the goal

  • Measure from around the world to all of the package mirror hosts (done).
  • Measure from around the world to pkg.FreeBSD.org to get the default GeoDNS host for each area/country/network (in progress, recurring every 6 hours for the next 6 days)
  • Find mis-configurations between “fastest” and “default” (TODO).

While the big measurements is still running I tried to use my eyes to see if there’s anything GREEN (aka fast connection) in direct package mirror host measurements whilst being YELLOW/RED (aka bad connection) when connecting to pkg.FreeBSD.org.

Unsurprisingly, we found one! It was in Mombasa, Kenya.

Here’s what it looks like while connecting directly:

{
  "fw": 5080,
  "mver": "2.6.2",
  "lts": 25,
  "dst_name": "pkg0.jinx.freebsd.org",
  "ttr": 10639.830509,
  "af": 4,
  "dst_addr": "196.10.53.168",
  "src_addr": "160.119.216.205",
  "proto": "ICMP",
  "ttl": 59,
  "size": 1000,
  "result": [
    {
      "rtt": 47.434297
    },
    {
      "rtt": 47.39957
    },
    {
      "rtt": 47.412379
    },
    {
      "rtt": 47.369089
    },
    {
      "rtt": 47.851894
    }
  ],
  "dup": 0,
  "rcvd": 5,
  "sent": 5,
  "min": 47.369089,
  "max": 47.851894,
  "avg": 47.493445799999996,
  "msm_id": 110487154,
  "prb_id": 22215,
  "timestamp": 1750269643,
  "msm_name": "Ping",
  "from": "160.119.216.205",
  "type": "ping",
  "group_id": 110487154,
  "step": null,
  "stored_timestamp": 1750269644
}

And here it is via GeoDNS’s defaults:

{
  "fw": 5080,
  "mver": "2.6.2",
  "lts": 10,
  "dst_name": "pkg.freebsd.org",
  "ttr": 10465.87807,
  "af": 4,
  "dst_addr": "173.228.147.98",
  "src_addr": "160.119.216.205",
  "proto": "ICMP",
  "ttl": 52,
  "size": 1000,
  "result": [
    {
      "rtt": 240.19025
    },
    {
      "rtt": 244.967071
    },
    {
      "rtt": 236.466414
    },
    {
      "rtt": 238.189775
    },
    {
      "rtt": 233.570938
    }
  ],
  "dup": 0,
  "rcvd": 5,
  "sent": 5,
  "min": 233.570938,
  "max": 244.967071,
  "avg": 238.6768896,
  "msm_id": 110489920,
  "prb_id": 22215,
  "timestamp": 1750270943,
  "msm_name": "Ping",
  "from": "160.119.216.205",
  "type": "ping",
  "group_id": 110489920,
  "step": null,
  "stored_timestamp": 1750270944
}

Clearly, it’s set to pkg0.chi.freebsd.org (173.228.147.98) by default. That’s Chicago. That’s halfway around the world. It took it 238ms on average. While it took the same probe 47ms on average when connecting to pkg0.jinx.freebsd.org directly.

Clearly, I’m not the only one.

Hopefully after collecting data for 6 days, we can start analyzing and improving the GeoDNS setup that we have.

A good suggestion from crest was to have a pkg plugin that does the work of fastest_pkg on the fly. We’ll try that one day.

If anyone is interested in helping me out, feel free to contact me over email, IRC or Discord.

That’s all folks…

Generating SSHFP Records

I added a new server to our hackerspace last week. This new server will be used for research in security. When I was adding the new DNS record when I realized that the previous server had a SSHFP record as well!

I remember that I should use the ssh-keygen command, however, like a normal human being, I forgot which flag to use. A simple search in the manual page says that I should use -r flag, so here we go.

antranigv@srv0:~ % ssh-keygen -r srv0
srv0 IN SSHFP 1 1 785b3fa04870e92bf25f4c7f7092733acf586ffb
srv0 IN SSHFP 1 2 847fd4a76ef7dfcef31ac3fa18c139413ab0017fa17014b3884bff161c3364de
srv0 IN SSHFP 3 1 8268aa7b8dccf4c0e7881472c72093589ca46b2e
srv0 IN SSHFP 3 2 ea0c9f0a50a825f5a0a59cebf8637876970a34000e6e0afd46bf269e08294a88
srv0 IN SSHFP 4 1 2fbe9d0e2ecdbd9dd58576e4683ee70858ca3f25
srv0 IN SSHFP 4 2 a34643bdce1ef3042cdd76fb7e46fcaf108dc436f8fcdb55daf993a27da0654b

All I need to do is to add these into the DNS zone. Luckily I run BIND, so I copy-pastad them into the file, did +1 to the SOA’s serial and done!

Now I can try logging in.

antranigv@zvartnots:~ $ ssh srv0.hackerspace.am -v
OpenSSH_9.3p1, OpenSSL 3.1.3 19 Sep 2023
debug1: Reading configuration data /Users/antranigv/.ssh/config
debug1: Reading configuration data /Users/antranigv/.ssh/personal
[…]
debug1: Server host key: ssh-ed25519 SHA256:OCsizTimnJi1grbxSY5LpvpLozfZ2pk+4Jzwg60WKYA debug1: found 6 secure fingerprints in DNS debug1: verify_host_key_dns: matched SSHFP type 4 fptype 1 debug1: verify_host_key_dns: matched SSHFP type 4 fptype 2 debug1: matching host key fingerprint found in DNS

and I logged in properly!

Antranig Vartanian

March 28, 2023

I’m sure at this point everyone heard that GitHub updated their RSA SSH host key.

This is a common issue, say you re-installed a server at the same IP/hostname. No problem, you remove a line and put in another one.

But honestly, I thought that GitHub had SSHFP records in their DNS.

$ dig SSHFP github.com +short | wc 
       0       0       0

Not so much… But then again, looks like they DO need to sign their zone first.

$ whois github.com | grep DNSSEC
   DNSSEC: unsigned
DNSSEC: unsigned

I have to say, using SSHFP has changed my life. Obviously there are some issues (Windows and macOS, specifically), but if you’re running a normal, modern operating system (hello BSDs, hello Linux) then it’s like magic!

I just setup one for our hackerspace!

$ whois hackerspace.am | grep DS
   DNS servers (zone signed, 1 DS record):
$ dig hackerspace.am SSHFP +short | wc -l
       6

Anyway, I’m sure many CI/CD pipelines are going crazy because of this… Good luck everyone!