Chapter Ⅰ
I know that “Complete Guide” and “Part One” are oxymorons, but hey, be happy that I’m publishing in parts, otherwise I’d completely ignore this blog post.
Two weeks ago I decided to play with illumos again. I was speaking with a friend and we were sharing our frustrations regarding Open-Source contribution. We write the code, we submit, we get feedback, we submit again, and then we’re ghosted. It’s like the LinkedIn or Tinder version of Software Engineering.
Then I asked him about his best open-source experience and he told me “illumos of course!”.
I was amazed. I thought you had to be very technical in order to even build illumos, but turns out they have an amazing documentation on building illumos and OmniOS (an illumos distribution) has done work to make sure that the system can be self-hosted (i.e. The OS can build itself).
So, I decided to fire up OmniOS on our hackerspace server running FreeBSD inside a bhyve VM.
The installation went smoothly, but the IPS packages were slow to download, and I might be wrong (please correct me if I am) but IPS doesn’t seem to be keeping a local copy of the files. It always downloads. Is that configurable?
Regardless. I thought that the best way to contribute is to advocate. In order to do that I needed to make sure that IPS servers are fast in Armenia. Hence the mirroring project started.
Obey!
Requirements
Here are some terminology that I will use in this blog post, just so we are on the same page.
- OmniOS: an illumos distribution
- Origin: OmniOS’s IPS servers at pkg.omnios.org
- Local: A copy of the Origin
- Repository: A collection of software
- Core: The Core Repository of OmniOS
- Extras: The Extra Repository of OmniOS
- IPS or PKG: The Image Packaging System and its utility,
pkg - Zone: an illumos Zone (similar to FreeBSD Jails, Linux Containers, chroot) running on OmniOS
Now that we are on the same page, let’s talk about our setup and what we need.
- An internet connection: duh!
- A domain name: I decided to use pkg.omnios.illumos.am. Yes, I’m lucky like that.
- A publicly accessible IP address.
- A server: I am running OmniOS Stable (
r151048) inside a VM. You can use bare-metal or a cloud VM if you want. - Storage: I am currently using around 50GB of storage, expect that to go around 300GB when we get to Part Three
Pre-Mirroring Setup
Before we setup our mirror, let’s make sure that we have a good infrastructure that we can maintain.
Here’s what we’ll create
- A Zone that will act as the HTTP(s) server using nginx at IP address
10.10.0.80 - A Zone that will do the mirroring using IPS tools at
10.10.0.51 - An virtual dumb switch (etherstub) that will connect the Zones and the Global-Zone (a.k.a The Host) together. The GZ will have an address of
10.10.0.1 - ZFS datasets for each Core and Extras Repository (for each release)
Please note that there are many ways to do this, for example, having everything in a Global Zone, running IPS mirroring and nginx in a single Zone, not using etherstub at all, etc. But I like this setup as it will allow us to “grow” in the future.
From now on, omnios# means that we’re in the Global Zone and zone0# means we’re inside a Zone named zone0.
Let’s start with setting up our etherstub and connecting our Global Zone to it
omnios# dladm create-etherstub switch0 omnios# dladm create-vnic -l switch0 vnic0 omnios# ipadm create-if vnic0 omnios# ipadm create-addr -T static -a 10.10.0.1/24 vnic0/switch0
Done!
Now, we will setup our Zones using the zadm utility. Install zadm by running
omnios# pkg install zadm
After installing zadm, we’ll create a dataset for our Zones
omnios# zfs create -o mountpoint=/zones rpool/zones
This assumes that your ZFS pool is named rpool.
Finally, we can create our Zones. Running
omnios# zadm create -b pkgsrc www0
will open your $, where you need to modify some JSON, here’s what mine looks like!EDITOR
{
"autoboot" : "true",
"brand" : "pkgsrc",
"ip-type" : "exclusive",
"dns-domain" : "omnios.illumos.am",
"net" : [
{
"allowed-address" : "10.10.0.80/24",
"defrouter" : "10.10.0.1",
"global-nic" : "switch0",
"physical" : "www0"
}
],
"pool" : "",
"scheduling-class" : "",
"zonename" : "www0",
"zonepath" : "/zones/www0"
}
After saving the file, zadm will install the Zone.
Now let’s setup our mirroring Zone. Do the same but change the Zone name to repo, the brand to lipkg (and -b lipkg10.10.0.51/24.
All we need now is to forward the HTTP/HTTPS traffic to www0 Zone and allow all Zones to access the internet using NAT.
Create and edit the IPFilter’s NAT file at /etc/ipf/ipnat.conf, here’s an example configuration
map vioif0 10.10.0.0/24 -> 212.34.250.10 rdr vioif0 212.34.250.10/32 port 80 -> 10.10.0.80 port 80 tcp rdr vioif0 212.34.250.10/32 port 443 -> 10.10.0.80 port 443 tcp
Make sure you set the correct interface name and the correct external IP address.
Finally, we can boot our Zones!
omnios# zadm boot www0
omnios# zadm boot repo
You should see the following output when you run zadm again
omnios# zadm NAME STATUS BRAND RAM CPUS SHARES global running ipkg 56G 12 1 repo running lipkg - - 1 www0 running pkgsrc - - 1
Great! Let’s setup the mirroring process.
Mirroring Setup
Let’s create a ZFS dataset for repos for each release
repo# zfs create -o mountpoint=/repo rpool/zones/repo/ROOT/repo repo# zfs create rpool/zones/repo/ROOT/repo/r151048 repo# zfs create rpool/zones/repo/ROOT/repo/r151048/core repo# zfs create rpool/zones/repo/ROOT/repo/r151048/extra
And then we use the pkgrepo command to create a repository
repo# pkgrepo create /repo/r151048/core
repo# pkgrepo create /repo/r151048/extra
And finally, we can start receiving the packages from Origin to Local
repo# pkgrecv -s https://pkg.omnios.org/r151048/core/ -d /repo/r151048/core '*'
repo# pkgrecv -s https://pkg.omnios.org/r151048/extra/ -d /repo/r151048/extra '*'
This will take a while depending on your internet connection speed and the load on OmniOS’s Origin. It’s like a good investment, we spend load and time now so we save traffic and time later 🙂
After it’s done, we need to set the publisher of these repos the same as Origin.
repo# pkgrepo set -s /repo/r151048/core publisher/prefix=omnios
repo# pkgrepo set -s /repo/r151048/extra/ publisher/prefix=extra.omnios
And we’re done!
Now need to serve these repos using IPS’s depot server.
We will create two instances of the depotd server, one for core and one for extra.
- r151048/core will run on 5148
- r151048/extra will run on 1148
- (in the future) r151050/core will run on 5150
- (in the future) r151050/extra will run on 1150
We start with core
repo# svccfg -s pkg/server add r151048_core
repo# svccfg -s pkg/server:r151048_core addpg pkg application
repo# svccfg -s pkg/server:r151048_core setprop pkg/inst_root = /repo/r151048/core/
repo# svccfg -s pkg/server:r151048_core setprop pkg/port = 5148
repo# svccfg -s pkg/server:r151048_core setprop pkg/proxy_base = https://pkg.omnios.illumos.am/r151048/core
And we do the same for extra
repo# svccfg -s pkg/server add r151048_extra repo# svccfg -s pkg/server:r151048_extra addpg pkg application repo# svccfg -s pkg/server:r151048_extra setprop pkg/inst_root = /repo/r151048/extra/ repo# svccfg -s pkg/server:r151048_extra setprop pkg/port = 1148 repo# svccfg -s pkg/server:r151048_extra setprop pkg/proxy_base = https://pkg.omnios.illumos.am/r151048/extra
Finally, we enable the services
repo# svcadm enable pkg/server:r151048_core pkg/server:r151048_extra
repo# svcadm restart pkg/server:r151048_core pkg/server:r151048_extra
Let’s check!
We’re good! Now let’s setup Nginx 🙂
The Web Server
This part is pretty easy, we login into www0, install nginx, and setup some paths. I will be posting a copy-pasta of my configs, I assume you can do the rest 🙂
www0# pkgin update
www0# pkgin install nginx
Thank you SmartOS! 🧡
In my nginx.conf, I added
include vhosts/*.conf;
and then in /opt/local/etc/nginx/vhosts I created a file
named pkg.omnios.illumos.am.conf, which looks like this
server {
listen 80;
server_name pkg.omnios.illumos.am;
location /.well-known/acme-challenge/ {
alias /opt/local/www/acme/.well-known/acme-challenge/;
}
location / {
return 301 "https://pkg.omnios.illumos.am";
}
}
server {
listen 443 ssl;
server_name pkg.omnios.illumos.am;
ssl_certificate /etc/ssl/pkg.omnios.illumos.am/fullchain.pem;
ssl_certificate_key /etc/ssl/pkg.omnios.illumos.am/key.pem;
location /r151048/core/ {
proxy_pass http://10.10.0.51:5148/;
}
location /r151048/extra/ {
proxy_pass http://10.10.0.51:1148/;
}
location / {
# This needs to be changed, later...
add_header Content-Type text/plain;
return 200 "ok...";
}
}
Finally, we just need to enable nginx
www0# svcadm enable pkgsrc/nginx
and check!
Using the Local Repos
This part is actually pretty easy. We just need to remove everything that exists and add our own. I will be running this on a computer named dna0.
dna0# pkg set-publisher -M '*' -G '*' omnios
dna0# pkg set-publisher -M '*' -G '*' extra.omnios
dna0# pkg set-publisher -O https://pkg.omnios.illumos.am/r151048/core omnios
dna0# pkg set-publisher -O https://pkg.omnios.illumos.am/r151048/extra extra.omnios
dna0# pkg publisher PUBLISHER TYPE STATUS P LOCATION extra.omnios origin online F https://pkg.omnios.illumos.am/r151048/extra/ omnios origin online F https://pkg.omnios.illumos.am/r151048/core/
We’re good! 🙂
Fetching Updates
By the time I wanted to publish this I noticed that there’s a new OmniOS Weekly Update, so I thought, hey, maybe I should try updating the Local Repo as well… how do we do that?
Turns out I just need to pkgrecv again, and then run a refresh command.
pkgrecv -v -s https://pkg.omnios.org/r151048/core/ -d /repo/r151048/core/ '*'
pkgrepo -s /repo/r151048/core refresh
And looks like we’re good! Maybe we can setup a simple cronjob 🙂
Final Notes
This has been an amazing experience. Since I started using OmniOS two weeks ago, I’ve setup the mirror, I installed two OmniOS deployments on production for two organization, and I talked about it during our Armenian Hackers Radio Podcast. With this mirror completely setup, I can advocate even more!
I’d like to send my thanks (and later, my money) to the OmniOS team for the amazing work they’re doing, special thanks to andyf for answering all of my questions, neirac for pushing me to try more illumos in my life and everyone who contributed to the docs and blog posts that I used. I’ll leave some links below.
Finally, for the coming (two) posts I will talk about mirroring downloads.OmniOS.org (for ISO/USB/ZFS images) and the pkgsrc repository run by SmartOS/MNX.
Thank you for reading and thank you, illumos-community for being so nice ^_^
That’s all folks…