Tag Archives: Jails

Installing DFIR-IRIS on FreeBSD using Jails

This is a live blogging of the installation process of DFIR-IRIS on FreeBSD 14.0-RELEASE using Jails and Jailer.

The main requirements are:

  • Nginx
  • PostgreSQL
  • Python
  • Some random dependencies we saw in the Dockerfile

I assume you already have nginx up and running, we will just be setting up a vhost under the domain name dfir.cert.am. Donโ€™t worry, this is INSIDE our infrastructure, you will not be able to connect to it ๐Ÿ™‚

Initial Setup

First we create a jail named iris0, using Jailer:

jailer create iris0

Next we install the required software inside of the jail. Looks like everything is available in FreeBSD packages:

jailer console iris0
pkg install \ nginx \ python39 \ py39-pip \ gnupg \ 7-zip \ rsync \ postgresql12-client \ git-tiny \ libxslt \ rust \ acme.sh

Installing DFIR-IRIS

Since weโ€™re using FreeBSD, weโ€™ll be doing things the right way instead of the Docker way, so we will be running IRIS as a user, not as root.

pw user add iris -m

Next we setup some directories and checkout the repo

root@iris0:~ # pw user add iris -m
root@iris0:~ # su - iris iris@iris0:~ $ git clone --branch v2.4.7 https://github.com/dfir-iris/iris-web.git iris-web

Finally, we install some python dependencies using pip.

iris@iris0:~ $ cd iris-web/source
iris@iris0:~/iris-web/source $ pip install -r requirements.txt

Now we have to configure the .env file based on our needs, I will post my version of it, I hope it helps

# -- DATABASE
export POSTGRES_USER=postgres
export POSTGRES_PASSWORD=postgres
export POSTGRES_DB=iris_db
export POSTGRES_ADMIN_USER=iris
export POSTGRES_ADMIN_PASSWORD=longpassword

export POSTGRES_SERVER=localhost
export POSTGRES_PORT=5432

# -- IRIS
export DOCKERIZED=0
export IRIS_SECRET_KEY=verylongsecret
export IRIS_SECURITY_PASSWORD_SALT=verylongsalt
export IRIS_UPSTREAM_SERVER=app # these are for docker, you can ignore
export IRIS_UPSTREAM_PORT=8000

# -- WORKER
export CELERY_BROKER=amqp://localhost
# Set to your rabbitmq instance

# Change these as you need them.
# -- AUTH
#IRIS_AUTHENTICATION_TYPE=local
## optional
#IRIS_ADM_PASSWORD=MySuperAdminPassword!
#IRIS_ADM_API_KEY=B8BA5D730210B50F41C06941582D7965D57319D5685440587F98DFDC45A01594
#IRIS_ADM_EMAIL=admin@localhost
#IRIS_ADM_USERNAME=administrator
# requests the just-in-time creation of users with ldap authentification (see https://github.com/dfir-iris/iris-web/issues/203)
#IRIS_AUTHENTICATION_CREATE_USER_IF_NOT_EXIST=True
# the group to which newly created users are initially added, default value is Analysts
#IRIS_NEW_USERS_DEFAULT_GROUP=

# -- LISTENING PORT
#INTERFACE_HTTPS_PORT=443

Configuring HTTPS

We can use acme.sh to issue a TLS certificate from Lets Encrypt.

root@iris0:~ # acme.sh --set-default-ca --server letsencrypt
root@iris0:~ # acme.sh --issue -d dfir.cert.am --standalone
root@iris0:~ # acme.sh -i -d dfir.cert.am --fullchain-file /usr/local/etc/ssl/dfir.cert.am/fullchain.pem --key-file /usr/local/etc/ssl/dfir.cert.am/key.pem --reloadcmd 'service nginx reload'

Setup nginx

DFIR-IRIS provides a nginx configuration template at nginx.conf, we will be using that, with a little bit of modifications.

The final nginx.conf will look like this:

#user  nobody;
worker_processes  1;

# This default error log path is compiled-in to make sure configuration parsing
# errors are logged somewhere, especially during unattended boot when stderr
# isn't normally logged anywhere. This path will be touched on every nginx
# start regardless of error log location configured here. See
# https://trac.nginx.org/nginx/ticket/147 for more info. 
#
#error_log  /var/log/nginx/error.log;
#

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    # Things needed/recommended by DFIR-IRIS
    map $request_uri $csp_header {
        default "default-src 'self' https://analytics.dfir-iris.org; script-src 'self' 'unsafe-inline' https://analytics.dfir-iris.org; style-src 'self' 'unsafe-inline';";
    }

    server_tokens off;
    sendfile    on;
    tcp_nopush  on;
    tcp_nodelay on;

    types_hash_max_size             2048;
    types_hash_bucket_size          128;
    proxy_headers_hash_max_size     2048;
    proxy_headers_hash_bucket_size  128;
    proxy_buffering                 on;
    proxy_buffers                   8 16k;
    proxy_buffer_size               4k;

    client_header_buffer_size   2k;
    large_client_header_buffers 8 64k;
    client_body_buffer_size     64k;
    client_max_body_size        100M;

    reset_timedout_connection   on;
    keepalive_timeout           90s;
    client_body_timeout         90s;
    send_timeout                90s;
    client_header_timeout       90s;
    fastcgi_read_timeout        90s;
    # WORKING TIMEOUT FOR PROXY CONF
    proxy_read_timeout          90s;
    uwsgi_read_timeout          90s;

    gzip off;
    gzip_disable "MSIE [1-6]\.";

    # FORWARD CLIENT IDENTITY TO SERVER
    proxy_set_header    HOST                $http_host;
    proxy_set_header    X-Forwarded-Proto   $scheme;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;

    # FULLY DISABLE SERVER CACHE
    add_header          Last-Modified $date_gmt;
    add_header          'Cache-Control' 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
    if_modified_since   off;
    expires             off;
    etag                off;
    proxy_no_cache      1;
    proxy_cache_bypass  1;

    # SSL CONF, STRONG CIPHERS ONLY
    ssl_protocols               TLSv1.2 TLSv1.3;

    ssl_prefer_server_ciphers   on;
    ssl_certificate             /usr/local/etc/ssl/dfir.cert.am/fullchain.pem;
    ssl_certificate_key         /usr/local/etc/ssl/dfir.cert.am/key.pem;
    ssl_ecdh_curve              secp521r1:secp384r1:prime256v1;
    ssl_buffer_size             4k;

    # DISABLE SSL SESSION CACHE
    ssl_session_tickets         off;
    ssl_session_cache           none;
    server {
        listen          443 ssl
        server_name     dfir.cert.am;
        root            /www/data;
        index           index.html;
        error_page      500 502 503 504  /50x.html;

        add_header Content-Security-Policy $csp_header;
        
        # SECURITY HEADERS
        add_header X-XSS-Protection             "1; mode=block";
        add_header X-Frame-Options              DENY;
        add_header X-Content-Type-Options       nosniff;
        # max-age = 31536000s = 1 year
        add_header Strict-Transport-Security    "max-age=31536000: includeSubDomains" always;
        add_header Front-End-Https              on;

        location / {
            proxy_pass  http://localhost:8000;

            location ~ ^/(manage/templates/add|manage/cases/upload_files) {
                keepalive_timeout           10m;
                client_body_timeout         10m;
                send_timeout                10m;
                proxy_read_timeout          10m;
                client_max_body_size        0M;
                proxy_request_buffering off;
                proxy_pass  http://localhost:8000;
            }

            location ~ ^/(datastore/file/add|datastore/file/add-interactive) {
                keepalive_timeout           10m;
                client_body_timeout         10m;
                send_timeout                10m;
                proxy_read_timeout          10m;
                client_max_body_size        0M;
                proxy_request_buffering off;
                proxy_pass  http://localhost:8000;
            }
        }
        location /socket.io {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_http_version 1.1;
            proxy_buffering off;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_pass http://localhost:8000/socket.io;
        }
    }
}

Setup PostgreSQL

I assume you know how to do this ๐Ÿ™‚ You donโ€™t need to configure a separate user, by the looks of it, IRIS likes to do that itself. Thanks to Jails I was able to run a separate PostgreSQL instance in the iris0 jail.

P.S. If you are running PostgreSQL inside a jail, make sure that the following variables are set in your jail configuration

  sysvshm         = new;
  sysvmsg         = new;

Running DFIR-IRIS

Now that everything is up and running, we just need to run DFIR-IRIS and it will create the database, needed users, an administration account, etc.

su - iris
cd ~/iris-web/source
. ../.env
~/.local/bin/gunicorn app:app --worker-class eventlet --bind 0.0.0.0:8000 --timeout 180 --worker-connections 1000 --log-level=debug

Assuming everything is fine, now we can setup a rc.d service script to make sure it runs at boot.

For that I wrote two files, the service itself and a helper start.sh script

rc.d script at /usr/local/etc/rc.d/iris

#!/bin/sh

# PROVIDE: iris
# REQUIRE: NETWORKING
# KEYWORD: 

. /etc/rc.subr

name="iris"
rcvar="iris_enable"
load_rc_config ${name}

: ${iris_enable:=no}
: ${iris_path:="/usr/local/iris"}
: ${iris_gunicorn:="/usr/local/bin/gunicorn"}
: ${iris_env="iris_gunicorn=${iris_gunicorn}"}

logfile="${iris_path}/iris.log"
pidfile="/var/run/${name}/iris.pid"

iris_user="iris"
iris_chdir="${iris_path}/source"
iris_command="${iris_path}/start.sh"

command="/usr/sbin/daemon"
command_args="-P ${pidfile} -T ${name} -o ${logfile} ${iris_command}"

run_rc_command "$1"

and the helper script at /home/iris/iris-web/start.sh

#!/bin/sh

export HOME=$(getent passwd `whoami` | cut -d : -f 6)

. ../.env

${iris_gunicorn} app:app --worker-class eventlet --bind 0.0.0.0:8000 --timeout 180 --worker-connections 128

now we set some variables in rc.conf using sysrc and we can start the service.

sysrc iris_enable="YES"
sysrc iris_path="/home/iris/iris-web"
sysrc iris_gunicorn="/home/iris/.local/bin/gunicorn"

Finally, we can start DFIR-IRIS as a service.

service iris start

Aaaaand weโ€™re done ๐Ÿ™‚

Thank you for reading!

There are some issues that Iโ€™d like to tackle, for example, service iris stop doesnโ€™t work, and it would be nice if we ported all of the dependencies into Ports, but for now, this seems to be working fine.

Special thanks to the DFIR-IRIS team for creating this cool platform!

Thatโ€™s all folksโ€ฆ

Reply via email.

FreeBSD Jail booting & running Devuan GNU+Linux with OpenRC

Two years ago I wrote a blog post named VoidLinux in FreeBSD Jail; with init, where we installed and “booted” VoidLinux in a FreeBSD Jail. I think it’s time to revise that post.

This time we will be using Devuan GNU+Linux, boot things using OpenRC and put some native FreeBSD binaries inside the Linux Jail.

Here’s what I’m running at the moment

root@srv0:~ # uname -v
FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC

To bootstrap the Devuan system, we need debootstrap. Specifically, debootstrap that ships with Devuan Chimaera. We can start by installing debootstrap from ports/packages, and then we can modify the rest.

pkg install -y debootstrap

Now we need to fetch Devuan’s debootstrap, extract it, put some files into our debootstrap and set some symbolic links.

# Path might change over time, check https://pkginfo.devuan.org/ for the exact link
fetch http://deb.devuan.org/merged/pool/DEVUAN/main/d/debootstrap/debootstrap_1.0.123+devuan3_all.deb

# .deb files are messy, make a directory
mkdir debootstrap_devuan
mv debootstrap_1.0.123+devuan3_all.deb debootstrap_devuan/
cd debootstrap_devuan/
tar xf debootstrap_1.0.123+devuan3_all.deb
tar xf data.tar.gz

# We need chimaera (latest, symlink) and ceres (origin)
cp usr/share/debootstrap/scripts/ceres usr/share/debootstrap/scripts/chimaera /usr/local/share/debootstrap/scripts/

Now we can bootstrap our system. I will be using a ZFS filesystem, but this can be done without ZFS as well.

Keep in mind that my Jail’s path is going to be /usr/local/jails/devuan0, modify this path as needed ๐Ÿ™‚

zfs create zroot/jails/devuan0

debootstrap --no-check-gpg --arch=amd64 chimaera /usr/local/jails/devuan0/ http://pkgmaster.devuan.org/merged/

The installation should start now but at some point there, we’ll get the following error:

I: Configuring libpam-runtime...
I: Configuring login...
I: Configuring util-linux...
I: Configuring mount...
I: Configuring sysvinit-core...
W: Failure while configuring required packages.
W: See /usr/local/jails/devuan0/debootstrap/debootstrap.log for details (possibly the package package is at fault)

DON’T PANIC! This is fine ๐Ÿ™‚ We just need to chroot inside, fix this manually and install OpenRC


chroot /usr/local/jails/devuan0 /bin/bash
# Fix base packages
dpkg --force-depends -i /var/cache/apt/archives/*.deb
# Set Cache-Start
echo "APT::Cache-Start 251658240;" > /etc/apt/apt.conf.d/00chroot
# Install OpenRC
apt update
apt install openrc

We have almost everything ready. We just need to create a password database file that the jail(8) command uses internally.

cd /usr/local/jails/devuan0/etc/
echo "root::0:0::0:0:Charlie &:/root:/bin/bash" > master.passwd
pwd_mkdb -d ./ -p master.passwd
# Restore the Linux passwd file
cp passwd- passwd

We can also move our statically linked FreeBSD binaries into the Linux Jail so we can use them when needed

cp -a /rescue /usr/local/jails/devuan0/native

Now we just need our Jail configuration file. We can put that at /etc/jail.conf.d/devuan0.conf

(This assumes that you’re network is configured similar to “VNET Jail HowTo Part 2: Networking”

# vim: set syntax=sh:
exec.clean;
allow.raw_sockets;
mount.devfs;

devuan0 {
  # ID == epair index :)
  $id             = "0";
  $bridge         = "bridge0";
  # Set a domain :)
  $domain         = "bsd.am";
  vnet;
  vnet.interface = "epair${id}b";

  mount.fstab     = "/etc/jail.conf.d/${name}.fstab";

  exec.prestart   = "ifconfig epair${id} create up";
  exec.prestart  += "ifconfig epair${id}a up descr vnet-${name}";
  exec.prestart  += "ifconfig ${bridge} addm epair${id}a up";

  exec.start      = "/sbin/openrc default";

  exec.stop       = "/sbin/openrc shutdown";

  exec.poststop   = "ifconfig ${bridge} deletem epair${id}a";
  exec.poststop  += "ifconfig epair${id}a destroy";

  host.hostname   = "${name}.${domain}";
  path            = "/usr/local/jails/devuan0";

  # Maybe mkdir this path :)
  exec.consolelog = "/var/log/jail/${name}.log";

  persist;
  allow.socket_af;
}

As you have guessed, we also need an fstab file, that should go into /etc/jail.conf.d/devuan0.fstab

devfs       /usr/local/jails/devuan0/dev      devfs     rw                   0 0
tmpfs       /usr/local/jails/devuan0/dev/shm  tmpfs     rw,size=1g,mode=1777 0 0
fdescfs     /usr/local/jails/devuan0/dev/fd   fdescfs   rw,linrdlnk          0 0
linprocfs   /usr/local/jails/devuan0/proc     linprocfs rw                   0 0
linsysfs    /usr/local/jails/devuan0/sys      linsysfs  rw                   0 0
tmpfs       /usr/local/jails/devuan0/tmp      tmpfs     rw,mode=1777         0 0

Finally, let’s load some kernel modules (in case they haven’t yet)

service linux enable
service linux start
kldload netlink

Let’s start our Jail!

jail -c -f /etc/jail.conf.d/devuan0.conf

Is it running?

 # jls -N
 JID             IP Address      Hostname                      Path
 devuan0                         devuan0.bsd.am                /usr/local/jails/devuan0

Yes it is!

Now we can jexec into it and run things!

root@srv0:~ # jexec -l devuan0 /bin/bash
root@devuan0:~# uname -a
Linux devuan0.bsd.am 4.4.0 FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC x86_64 GNU/Linux

The process tree looks neat as well!

root@devuan0:~# ps f
  PID TTY      STAT   TIME COMMAND
74682 pts/1    S      0:00 /bin/bash
78212 pts/1    R+     0:00  \_ ps f
48412 ?        Ss     0:00 /usr/sbin/cron
41190 ?        Ss     0:00 /usr/sbin/rsyslogd

Let’s do some networking things! Let’s setup networking and install OpenSSH.
(This assumes that you’re network is configured similar to “VNET Jail HowTo Part 2: Networking”)

# Setup network interfaces
/native/ifconfig lo0 inet 127.0.0.1/8 up
/native/ifconfig epair0b inet 10.0.0.10/24 up
/native/route add default 10.0.0.1

# Install and start OpenSSH server
apt-get --no-install-recommends install openssh-server
rc-service ssh start

You should be able to ping things now

~# ping -n -c 1 bsd.am
ping: WARNING: setsockopt(ICMP_FILTER): Protocol not available
PING  (37.252.73.34) 56(84) bytes of data.
64 bytes from 37.252.73.34: icmp_seq=1 ttl=55 time=2.60 ms

---  ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.603/2.603/2.603/0.000 ms

To make the networking configuration persistent, we can use the rc.local file that OpenRC executes at boot.

chmod +x /etc/rc.local
echo '/native/ifconfig lo0 inet 127.0.0.1/8 up' >> /etc/rc.local
echo '/native/ifconfig epair0b inet 10.0.0.10/24 up' >> /etc/rc.local
echo '/native/route add default 10.0.0.1' >> /etc/rc.local

Do you know what this means? It means that now you can have proper ZFS, DTrace and pf firewalling with Linux. Congrats, now you have clean waters.

That’s all folksโ€ฆ

P.S. I would like to thank my mentor, norayr, for showing me how to start/stop OpenRC manually, and the awesome folks at #devuan for their help.

Reply via email.

Antranig Vartanian

July 1, 2023

A customer asked me to help them setup a tiny lab with many open-source tools. They are planning to move from corporate services to open-source alternatives such as NextCloud, Gitea, etc.

Unfortunately, they run only Linux, Ubuntu to be more specific, and as a UNIX gentlemen, I didn’t want to put everything into a single host, so I decided to use containers, in this case, LXC, a.k.a Linux Containers.

How hard could it be?

Oh god, layers of abstraction on within the system that have no idea about each other.

Like, who would assume that LXC would automatically download and install dnsmasq and assign IP addresses without my knowledge, or that it would push rules into the firewall?

The more I use Linux Container, the more I understand why FreeBSD Jails / illumos Zones didn’t win.

People don’t want automation or control, they want “please do this for me as I don’t wanna do it myself” tools.

I’d expect at least a message post-installation that says “We have installed and configured dnsmasq, reconfigured some systemd things, modified the following file (which is not mentioned in any man page, so you can use Google instead of man/apropos) and will use IP address ranges that you didn’t approve”

Is this why Docker won? Is it because people DIDNโ€™T want to learn how to do software packaging? I hope not. I wanna believe its because developers wanted to “think operationally”

Oh, and from a FreeBSD perspective, what’s even more weird is that

  1. there are no proper manual pages.
  2. the documentation is weird. It talks about a utility named lxc but I’m using 20 utilities named lxc-*, and I still cannot find the proper documentation for that
  3. it’s very much segmented. For example, on FreeBSD, we talk about which is better, jail.conf, BastilleBSD, pot, AppJail or Jailer. Here the same utility (lxc) that has multiple config files with no proper versioning, pretty complex manual pages and the not even examples or HowTos.

I’m looking at this and thinking โ€oh well, if we build a proper tool, I bet we can win some of the marketโ€ until you realize, of course, that when people hear FreeBSD, they will be thinking โ€itโ€™s not Linux? maybe itโ€™s not worth it, otherwise I wouldโ€™ve heard about itโ€

Iโ€™m just angry here. Please ignore my rants.

Cheers yโ€™all.

Reply via email.

Call For Testing: Jailer v0.1.1

Well, it’s finally here! After a week of sleepless work, I cleaned up the Jailer codebase and added many features (and removed some as well!) that I wanted since last year ๐Ÿ™‚

If you are reading this, please consider testing Jailer on FreeBSD. The codebase is at illuria/jailer.

The README.md should have all the info that you need to run Jailer.

If find any issues, please report to illuria/jailer/issues or you can email me personally at antranigv [at] freebsd [dot] am

Here’s the roadmap for what’s coming next

  1. Complete NetGraph support using jng.
  2. Jailerfile, which will be something similar to Dockerfile, allowing developers to create consistent images.
  3. jailerd and jailerctl, for remote jailer automation. This will be an open-source port of what illuria has already developed.
  4. Distributed Jailer, where jailerctl list will show not just what’s on a remote machine, but on a remote datacenter, inspired by Triton. Again, we have this at illuria, but we need to create an open-source port.

This release is dedicated to

Thank you for reading ๐Ÿ™‚

That’s all folksโ€ฆ

Reply via email.

VoidLinux in FreeBSD Jail; with init

Two important things happened this week for me.

First, Faraz asked me if I can rename my Jail manager to something other than Jailio because he got that domain for his Jailer manager already. So I named it

Second, I was able to run a complete Linux system using Jailer. While the repo for Jailer is not released yet (we are auditing for possible security issues), I would like to share how I was able to run VoidLinux in a Jail.

Since Jailer is not announced yet, I will give the examples using jail.conf, as most people either are or should be familiar with its concepts.

I went with VoidLinux because I am able to run the init process without its need to be running as PID1.

Let’s start, shall we?

First, ZFS dataset for our jail!

zfs create zroot/jails/voidlinux

Next we need to fetch the base system of VoidLinux. Luckily they do provide it on their website.

fetch https://alpha.de.repo.voidlinux.org/live/current/void-x86_64-ROOTFS-20210218.tar.xz

Now we can extract this into our dataset

tar xf void-x86_64-ROOTFS-20210218.tar.xz -C /usr/local/jails/voidlinux/

You might get an error that ./usr/bin/iputils-ping: Cannot restore extended attributes: security.capability, which is fine, I think?

If you are on FreeBSD 12.2-RELEASE or later, now you need to enable the Linuxulator.

service linux enable; service linux start

Now you can at least chroot into the system.

chroot /usr/local/jails/voidlinux/ /bin/bash

If everything is fine until now, perfect.

Now we need to add a root user into the system.

root@host:~ # cd /usr/local/jails/voidlinux/etc/
root@host:/usr/local/jails/voidlinux/etc # echo "root::0:0::0:0:Charlie &:/root:/bin/bash" > master.passwd
root@host:/usr/local/jails/voidlinux/etc # pwd_mkdb -d ./ -p master.passwd
pwd_mkdb: warning, unknown root shell

Execute the rest of the commands in Void.

root@host:~ # chroot /usr/local/jails/voidlinux/ /bin/bash
bash-5.1# cd /etc/
bash-5.1# pwconv 
bash-5.1# grpconv 
bash-5.1# passwd 
New password: 
Retype new password: 
passwd: password updated successfully
bash-5.1# exit

If all went fine, then the system is ready to be run as a Jail!

First we need to make an fstab for the system.

Create a file at /usr/local/jails/voidlinux/etc/fstab.pre and insert the following inside

devfs       /usr/local/jails/voidlinux/dev      devfs           rw                      0   0
tmpfs       /usr/local/jails/voidlinux/dev/shm  tmpfs           rw,size=1g,mode=1777    0   0
fdescfs     /usr/local/jails/voidlinux/dev/fd   fdescfs         rw,linrdlnk             0   0
linprocfs   /usr/local/jails/voidlinux/proc     linprocfs       rw                      0   0
linsysfs    /usr/local/jails/voidlinux/sys      linsysfs        rw                      0   0
/tmp        /usr/local/jails/voidlinux/tmp      nullfs          rw                      0   0

Next, let’s create a loopback interface for networking. Oh yes, VNET is not supported yet, but I’m working on a patch ๐Ÿ™‚

ifconfig lo1 create
ifconfig lo1 inet 10.10.0.1/24 up # sorry, 10.0.0.0/24 was unavailable :P

Okay, time to create our Jail conf!

exec.clean;
allow.raw_sockets;
mount.devfs;

voidlinux {
    $id     = "1";
    $ipaddr = "10.10.0.42";
    $mask   = "255.255.255.0";
    $domain = "srv0.bsd.am";
    devfs_ruleset  = 4;
    allow.mount;
    allow.mount.devfs;
    mount.fstab = "${path}/etc/fstab.pre";

    exec.start     = "/bin/sh /etc/runit/2 &";
    exec.stop      = "/bin/sh /etc/runit/3";


    ip4.addr      = "${ipaddr}";
    interface     = "lo1";
    host.hostname = "${name}.${domain}";
    path = "/usr/local/jails/voidlinux";
    exec.consolelog = "/var/log/jail-${name}.log";
    persist;
    allow.socket_af;
}

Let’s check?

# jls
   JID  IP Address      Hostname                      Path
     1  192.168.0.42    voidlinux.srv0.bsd.am         /usr/local/jails/voidlinux

And the process tree?

# ps auxd -J voidlinux
USER   PID %CPU %MEM  VSZ  RSS TT  STAT STARTED    TIME COMMAND
root 35182  0.0  0.1 2320 1428  -  SsJ  21:09   0:00.12 runsvdir -P /run/runit/runsvdir/current log: ot set SO_PASSCRED: Protocol not available\ncould not set SO_PASSCRED: Protocol
root 35190  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty6
root 35397  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty tty6 38400 linux
root 35191  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty1
root 35396  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty --noclear tty1 38400 linux
root 35192  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty5
root 35398  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.01 `-- agetty tty5 38400 linux
root 35193  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty2
root 35393  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty tty2 38400 linux
root 35194  0.0  0.1 2168 1396  -  RsJ  21:09   0:00.12 - runsv udevd
root 35195  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty3
root 35394  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty tty3 38400 linux
root 35196  0.0  0.1 2168 1376  -  SsJ  21:09   0:00.02 - runsv agetty-tty4
root 35390  0.0  0.1 2412 1704  -  SsJ  21:10   0:00.00 `-- agetty tty4 38400 linux

You may jexec now ๐Ÿ™‚

# jexec voidlinux /bin/bash
bash-5.1# uname -a
Linux voidlinux.srv0.bsd.am 3.2.0 FreeBSD 12.2-RELEASE-p6 GENERIC x86_64 GNU/Linux

Let’s check networking?

bash-5.1# ping -c 1 10.10.0.1
ping: WARNING: setsockopt(ICMP_FILTER): Protocol not available
PING 10.10.0.1 (10.10.0.1) 56(84) bytes of data.
64 bytes from 10.10.0.1: icmp_seq=1 ttl=64 time=0.069 ms

--- 10.10.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.069/0.069/0.069/0.000 ms

There you go! Well, things that are related to netlink might not work, but other than that it’s okay.

I did have some problems while installing packages, something about too many levels of symbolic links. Here’s the exact output when I was trying to install the curl package

[*] Unpacking packages
libev-4.33_1: unpacking ...
ERROR: libev-4.33_1: [unpack] failed to extract file `./usr/lib/libev.so.4': Too many levels of symbolic links
ERROR: libev-4.33_1: [unpack] failed to extract files: Too many levels of symbolic links
ERROR: libev-4.33_1: [unpack] failed to unpack files from archive: Too many levels of symbolic links
Transaction failed! see above for errors.

Now, I did not find the time to fix this yet, but if you have any idea, please let me know or comment below ๐Ÿ™‚

So, what do we have here? A Linux Jail, running VoidLinux, with init, so you can also run services, and basic networking for it.

That’s all folksโ€ฆ

Reply via email.

VNET Jail HowTo Part 2: Networking

As always, Dan has been tweeting about VNET Jail issues, which means it’s time for another VNET Jail post.

This post assumes that you’ve read the original post on VNET Jail HowTo.

In Part two we will discuss Networking.

We will use PF as a firewall to do things like NAT.

If you need more help please check the FreeBSD Handbook: Chapter – Firewalls or send me an email/tweet.

At this point (from the last post) we were able to ping from the Jail to the Host.

root@www:/ # ping -c 1 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=0.087 ms

--- 10.0.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.087/0.087/0.087/0.000 ms

Now we will setup PF on the host by adding the following to /etc/pf.conf

ext_if="em0"
jailnet="10.0.0.0/24"

nat pass on $ext_if inet from $jailnet to any -> ($ext_if)

set   skip on { lo0, bridge0 }
pass  inet proto icmp
pass  out all keep state

We also need to enable IP Forwarding in the kernel

Add the following in /etc/sysctl.conf

net.inet.ip.forwarding=1

And now execute

sysctl -f /etc/sysctl.conf
service pf restart

That should be it, now your Jail should be able to ping the outside world

root@zvartnots:~ # jexec -l www
You have mail.
root@www:~ # ping -c 1 9.9.9.9
PING 9.9.9.9 (9.9.9.9): 56 data bytes
64 bytes from 9.9.9.9: icmp_seq=0 ttl=61 time=2.566 ms

--- 9.9.9.9 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.566/2.566/2.566/0.000 ms
root@www:~ # 

If you setup a resolver, you should also be able to ping domain names as well.

root@www:~ # echo 'nameserver 9.9.9.9' > /etc/resolv.conf 
root@www:~ # ping -c 1 freebsd.org
PING freebsd.org (96.47.72.84): 56 data bytes
64 bytes from 96.47.72.84: icmp_seq=0 ttl=53 time=133.851 ms

--- freebsd.org ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 133.851/133.851/133.851/0.000 ms

Now, for a more complicated setup that assumes no firewalls and multiple IP addresses, where each Jail has its own IP address. I have a similar setup at home where my ZNC server Jail has its own IP address by connecting the physical NIC to the same bridge as the ZNC Jail.

In my rc.conf on the host

ifconfig_em0="inet 192.168.0.34 netmask 255.255.255.0"
defaultrouter="192.168.0.1"

cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0"

Here’s an example with jail.conf

znc {
	$id		= "52";
	$addr		= "192.168.0.252";
	$mask		= "255.255.255.0";
	$gw		= "192.168.0.1";
	vnet;
	vnet.interface	= "epair${id}b";

	exec.prestart	= "ifconfig epair${id} create up";
	exec.prestart	+= "ifconfig epair${id}a up descr vnet-${name}";
	exec.prestart	+= "ifconfig bridge0 addm epair${id}a up";

	exec.start	= "/sbin/ifconfig lo0 127.0.0.1 up";
	exec.start	+= "/sbin/ifconfig epair${id}b ${addr} netmask ${mask} up";
	exec.start	+= "/sbin/route add default ${gw}";
	exec.start	+= "/bin/sh /etc/rc";

	exec.poststop   = "ifconfig bridge0 deletem epair${id}a";
	exec.poststop  += "ifconfig epair${id}a destroy";

	host.hostname = "${name}.bsd.am";
	path = "/usr/local/jails/${name}";
 	exec.consolelog = "/var/log/jail-${name}.log";
	persist;
}

And that’s pretty much it!

That’s all folks.

Reply via email.

VNET Jail HowTo

So Dan has been tweeting that there’s no good example to get started with VNET Jails with jail.conf, I thought it’s time to write one.

In this example I’ve used FreeBSD 12.1-RELEASE

root@jail-host:~ # freebsd-version
12.1-RELEASE
root@jail-host:~ # uname -a
FreeBSD jail-host 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
root@jail-host:~ #

First thing first, let’s setup a bridge on our host

root@jail-host:~ # sysrc cloned_interfaces="bridge0"
cloned_interfaces:  -> bridge0
root@jail-host:~ # sysrc ifconfig_bridge0="inet 10.0.0.1 netmask 0xffffff00 descr jails-bridge"
ifconfig_bridge0:  -> inet 10.0.0.1 netmask 0xffffff00 descr jails-bridge

Start the bridge0 interface without restarting the other interfaces

root@jail-host:~ # service netif start bridge0

Good! let’s setup a ZFS dataset for Jails ๐Ÿ˜‰

root@jail-host:~ # zfs create -o mountpoint=/usr/local/jails zroot/jails

Good! now let’s fetch the base.txz file. I will be using my closest mirror, you should use yours.

root@jail-host:~ # mkdir /usr/local/jails/.dist-files
root@jail-host:~ # fetch -o /usr/local/jails/.dist-files/FreeBSD-12.1-RELEASE-base.txz http://mirror.yandex.ru/freebsd/releases/amd64/12.1-RELEASE/base.txz

Perfect!

Now, we will extract the base into the jail.

root@jail-host:~ # zfs create zroot/jails/www
root@jail-host:~ # tar xf /usr/local/jails/.dist-files/FreeBSD-12.1-RELEASE-base.txz -C /usr/local/jails/www/

Nicely done! Now let’s setup our /etc/jail.conf ๐Ÿ™‚

Here’s my configuration.

# vim: set syntax=sh:
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
allow.raw_sockets;
allow.mount.tmpfs;
mount.devfs;


www {
    $id     = "10";
    $ipaddr = "10.0.0.${id}";
    $mask   = "255.255.255.0";
    $gw     = "10.0.0.1";
    vnet;
    vnet.interface = "epair${id}b";

    exec.prestart   = "ifconfig epair${id} create up";
    exec.prestart  += "ifconfig epair${id}a up descr vnet-${name}";
    exec.prestart  += "ifconfig bridge0 addm epair${id}a up";

    exec.start      = "/sbin/ifconfig lo0 127.0.0.1 up";
    exec.start     += "/sbin/ifconfig epair${id}b ${ipaddr} netmask ${mask} up";
    exec.start     += "/sbin/route add default ${gw}";
    exec.start     += "/bin/sh /etc/rc";

    exec.prestop    = "ifconfig epair${id}b -vnet ${name}";

    exec.poststop   = "ifconfig bridge0 deletem epair${id}a";
    exec.poststop  += "ifconfig epair${id}a destroy";

    host.hostname = "${name}.jail-host";
    path = "/usr/local/jails/${name}";
    exec.consolelog = "/var/log/jail-${name}.log";
    persist;
}

Now let’s start our Jail!

root@jail-host:~ # service jail enable
jail enabled in /etc/rc.conf
root@jail-host:~ # service jail start www
Starting jails: www.
root@jail-host:~ # jls
   JID  IP Address      Hostname                      Path
     1                  www.jail-host                 /usr/local/jails/www

Let’s check the networking ๐Ÿ™‚

root@jail-host:~ # ping -c 1 10.0.0.10
PING 10.0.0.10 (10.0.0.10): 56 data bytes
64 bytes from 10.0.0.10: icmp_seq=0 ttl=64 time=0.164 ms

--- 10.0.0.10 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.164/0.164/0.164/0.000 ms

We can do the same from the jail.

root@jail-host:~ # jexec www
root@www:/ # ping -c 1 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=0.087 ms

--- 10.0.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.087/0.087/0.087/0.000 ms

We can also stop all the jails.

root@jail-host:~ # service jail stop
Stopping jails: www.

Okay! Couple of notes ๐Ÿ™‚

You can have jail.conf at /etc/jail.conf or /etc/something.jail.conf. The problem with the latter is that if you have jail_enable="YES" in rc.conf without defining jail_list then it will run only the jails in /etc/jail.conf

There are more ways to configure VNET Jails, either with jib or jng, an example is here.

Ideally, it would be nice to have /etc/jail.d/myjail.conf, and I wrote a patch for that (D24570), if you are a FreeBSD developer, please have a look ๐Ÿ™‚

Reply via email.