Author Archives: Antranig Vartanian

About Antranig Vartanian

Doing things @ illuria, Inc. Unix, BSD, InfoSec, Elixir/Erlang, DNS, XMPP. Mostly harmless.

Unshallow Git

A while back, when I was working on some changes for FreeBSD, I wanted to checkout our source tree. A very typical thing that every developer does every day, that is

git clone https://git.FreeBSD.org/src.git

However, the FreeBSD git server is pretty far from me. There’s a GeoDNS system in the front so I usually hit the one in Frankfurt, Germany.

Still, it’s pretty slow!

root@devbsd14:~ # git clone https://git.FreeBSD.org/src.git
Cloning into 'src'...
remote: Enumerating objects: 4234853, done.
remote: Counting objects: 100% (381211/381211), done.
remote: Compressing objects: 100% (28321/28321), done.
Receiving objects:   3% (152416/4234853), 48.97 MiB | 1.08 MiB/s

Okay, 1.08 MiB/s is not that bad, but I’m sure we can do better.

How about GitHub?

root@devbsd14:~ # git clone https://github.com/freebsd/freebsd-src/
Cloning into 'freebsd-src'...
remote: Enumerating objects: 4793378, done.
remote: Counting objects: 100% (398/398), done.
remote: Compressing objects: 100% (233/233), done.
Receiving objects:  16% (780550/4793378), 223.95 MiB | 2.13 MiB/s

Okay, 2.13 MiB/s is also not bad, but I have a faster connection than that!

Regardless, I needed just the last state of the code, without the history, so in order to save time and bandwidth I can do

git clone --depth 1 https://git.FreeBSD.org/src.git

And now I can work.

The problem is that this was months ago, and I totally forgot about it.

While I was debugging some issue, I ran git blame and I realized that I can’t see anything older than 3 months. what?

Lucky me, I was able to understand what I did by looking into the shell history.

Okay, so two questions.

  1. Can I get the rest of the depth/history?
  2. If GitHub and git.FreeBSD.org is slow, can I setup a local mirror?

Turns out, I had to ask these questions in reverse.

First, I setup a FreeBSD source tree mirror in my home server (which also serves this blog). The connection to that server is fast, the download speed is around 500Mbps, compared to the 50Mbps that I get in this apartment. Yes, I have to apartments, but one of them is only for servers 😀

That was pretty easy to do, I just needed to tell Gitea to mirror https://git.FreeBSD.org/src.git, and in couple of minutes, it was all ready.

Next, I had to make my local checkout… unshallow. After setting up the appropriate remotes, all I had to do was

git pull --unshallow mirror main

and now I have the history all the way back to Jun 12, 1993.

Oh, right! The clone speed test!

root@devbsd14:~ # git clone git@git.bsd.am:antranigv/freebsd-src.git
Cloning into 'freebsd-src'...
remote: Enumerating objects: 4235021, done.
remote: Counting objects: 100% (4235021/4235021), done.
remote: Compressing objects: 100% (824757/824757), done.
Receiving objects:  18% (762304/4235021), 207.13 MiB | 3.53 MiB/s

Okay! now this does use a lot more speed!

Lessons Learned?

  1. Latency matters! If the distance between my two apartments is $2 worth of commute, while the FreeBSD server is $2000 worth of commute, then my apartments are also close to each other digitally.
  2. When you do anything non-standard with git (e.g. --depth=1) make sure to read the docs on how to undo that later.

That’s all folks…

Reply via email.

Reply from National Vulnerability Database Team regarding Legacy Data Feeds

Couple of days ago when I was assisting a customer, I recommended that they follow the National Institute of Standards and Technology’s (a.k.a. NIST) Information Technology Laboratory’s Computer Security Division’s National Vulnerability Database’s (a.k.a. NVD a.k.a. that place that publishes the CVEs) data feeds. (Apologies for the bad intro)

So, these are RSS feeds that “contains the most recent CVE cyber vulnerabilities published within the NVD”

Unfortunately, I saw a notice at the top of the page, which got me really worried. It says

In September 2023, the NVD plans to retire all legacy data feeds while guiding any remaining data feed users to updated application-programming interfaces (APIs).

Usually, I’d panic and start ranting on my blog, but this is the NVD we’re talking about. They are a US government project that has been doing a lot of good and they are sponsored by the CISA, an agency that does many good things not just for US citizens, but citizens of our planet.

I started digging to understand what exactly is going to be retired and most importantly, why?

The NVD has made an amazing change timeline that has the following

The NVD plans to retire the RSS data feeds. The NVD plans to enable reCAPTCHA across all webpages and to retire webpages intended to support web scraping (e.g., Full Listings) before its APIs existed.

Okay, NOW I’m worried.

I’ll break this into two parts.

Why we need RSS feeds

You see, the internet relies on RSS, and I’m not just saying that because most of my audience uses RSS daily. The reason is much deeper than that.

As Dave Winer blogged a month ago

RSS is a thing like roadways and paths of rivers, they change very slowly. Think about qwerty keyboards. That’s what we’re talking about here. Agreements between products to interop. RSS is just like the gauge of rails, or always driving on one side of the street. A convention that makes progress possible. #

Scripting.com, Saturday, January 28, 2023

There are three products/protocols that I use daily, it’s Slack (for work), XMPP (for friends and family) and Telegram (for Armenian tech communities).

There are specific things that I should deliver for all these and that is messages, alerts, notices.

For my work, I should be able to get news if there’s a security issue on FreeBSD, because we use that. For friends and family I should deliver notices if there are any issues or upcoming maintenance to our servers. For my telegram communities I should update them if we’re having any new meetups, events, podcasts.

But, instead of writing a software that fetches, parses, analyzes and does something-something to these messages, I use RSS! FreeBSD has an RSS feed for Security Advisories. All I do in Slack is /feed subscribe https://www.freebsd.org/security/feed.xml and now, every time there an SA for FreeBSD, I get notified in Slack.

For friends and family? I have Huginn agent that parses RSS and send an XMPP message. For Armenian tech communities? I read a website’s RSS and a bot posts it in a group.

You get the idea.

RSS is all about “things working together”, there is no need to write a specific piece of for that specific thing.

And for years, I’ve relied on NVD’s RSS data feed to notify customers, tell them what to upgrade, if they need to upgrade and why to upgrade.

These RSS feeds are part of my professional life, a way for me, and people like me to know if we should be in panic mode or not.

So…

Okay, now what?

I believe in communication. I was very sure that my questions will be answered by the NVD, so I sent a message!

Greetings dear NVD team, NIST team and Computer Security Division,

While browsing your website, I have noticed the following change:

> In September 2023, the NVD plans to retire all legacy data feeds and the 1.0 APIs.

This became very disturbing, as many companies (including mine) rely on the data feed provided by NIST’s NVD.

I have two questions:
1) Is there *any* chance to keep the RSS feeds?
2) Is it okay if others (i.e. I) generate an RSS feed from your new API, if your final decision for q#1 is no?

[ . . . ]

If I may, that being said, I’m sure there’s a good reason, so my other question is:
What are/were the technical issues with RSS? Could it be bypassed or hacked around?

Thank you for all the work that you do, and thank in advance.

Kind regards,

I was right! They did answer all of my questions! I got a reply yesterday, here it is.

1) Is there *any* chance to keep the RSS feeds?

We have no plans to continue providing the RSS feeds located at
https://nvd.nist.gov/vuln/data-feeds#RSS
https://nvd.nist.gov/feeds/xml/cve/misc/nvd-rss.xml
https://nvd.nist.gov/feeds/xml/cve/misc/nvd-rss-analyzed.xml

Additionally, it is important to point out that per our announcement at https://nvd.nist.gov/general/news/change-timeline, the RSS feeds will be retired in March, not September. If you were not aware of these announcements we highly advice joining the NVD Google Group to stay better informed (https://groups.google.com/a/list.nist.gov/g/nvd-news).

2) Is it okay if others (i.e. I) generate an RSS feed from your new API, if your final decision for q#1 is no?

All NIST publications are available in the public domain. Organizations seeking to automate the retrieval of NVD data should use the NVD’s Application Programing Interfaces (APIs).
Services which utilize or access the NVD are asked to display the following notice prominently within the application: “This product uses data from the NVD API but is not endorsed or certified by the NVD.” You may use the NVD name to identify the source of the data. You may not use the NVD name, to imply endorsement of any product, service, or entity, not-for-profit, commercial or otherwise. For information on how to the cite the NVD, including the database’s Digital Object Identifier (DOI), please consult NIST’s Public Data Repository.

3) What are/were the technical issues with RSS? Could it be bypassed or hacked around?

The RSS feeds were considered to be overly simplistic and underutilized, they were determined in scope of retirement for these reasons as part of a larger effort to consolidate our output formats as we move towards the APIs. If you would like to submit a user story explaining the benefits and needs that the APIs currently do not meet we would pass that along to the development team for consideration in the future.

Okay. I agree! RSS is very simplistic, but that’s the point! it’s supposed to be simple. I mean, it’s simple enough that podcasts are RSS feeds.

And to be clear, I DID check the NVD’s new Vulnerabilities API, it’s awesome, it’s nice, it’s documented very well, kudos to the team, they did an amazing work, I’m sure it wasn’t easy. It has, for sure, more features than RSS could provide.

What to do about it?

I understand that the NVD is pushing the REST API, and I also understand why. But I really don’t want to write a “wrapper” for every service and technology that I use.

Here are my two questions.

  1. Will systems break because of this? Are you using these feeds? Do you rely on them for yourself or your organization?
  2. Will there be an interest by the InfoSec community to write a wrapper that generates a new RSS feed from NVDs new API?

Personally, if there’s an interest or not, I will be stopping everything I’m working on to create this NVD-to-RSS generator, as I very much rely on it. It will be open-source, obviously. What I should build is a drop-in replacement, where you change the feed URL, and everything works like before. (Well, I have to finish my other open-source commitments first, then I should work on this 😀 hopefully it wont take long.)

I would like to thank the NVD for keeping these feeds for all these years and congratulate them for their new APIs, I’m sure many good things will come out of these APIs.

And thank you for reading 🙂

That’s all folks…

Reply via email.

Hardcoded Folder Icons in macOS

I saw this post on Mastodon, and it got me thinking.

Screenshot 2023 02 08 at 4 26 58 AM

Are there any other hardcoded folder names in macOS? Where would I even start looking?

Well, based on what I know, every program on macOS is a directory that ends with .app, which means all I have to do is to find Finder’s location.

That should be pretty easy!

antranigv@zvartnots:~ $ cd /System/
antranigv@zvartnots:/System $ find . -type d -name Finder.app 2>/dev/null
./Library/CoreServices/Finder.app
./Volumes/Data/System/Library/CoreServices/Finder.app

Well, that was easy to find! Don’t worry, /System/Library is the same as /System/Volumes/Data/System/Library, Apple has a weird way of doing mount points, and I’m not here to judge. Well, at least not today!

Next I have to dig into Finder.app’s content, let’s see what we have.

antranigv@zvartnots:/System $ cd Library/CoreServices/Finder.app/
antranigv@zvartnots:/System/Library/CoreServices/Finder.app $ find . -type f -name 'Developer*'

Weird. Nothing at all?

I wonder what type of files we have here? I already know that I wanna see all the file types except Mach-O.

$ find . -type f -print0 | xargs -0 -I% -L 1 file -b "%" | sort -u | grep -v 'Mach-O'
ASCII text, with no line terminators
Apple binary property list
IFF data, AIFF audio
Mac OS X Code Directory version 20100 - 203 bytes
Mac OS X Code Directory version 20100 - 213 bytes
Mac OS X Code Directory version 20100 - 215 bytes
Mac OS X Code Directory version 20100 - 217 bytes
Mac OS X Code Requirement Set - 76 bytes
Mac OS X Code Requirement Set - 84 bytes
Mac OS X Code Requirement Set - 88 bytes
Mac OS X bill of materials (BOM) file
Mac OS X icon, 114423 bytes, "ic13" type
Mac OS X icon, 1162872 bytes, "ic12" type
Mac OS X icon, 23712 bytes, "ic13" type
Mac OS X icon, 30805 bytes, "ic13" type
Mac OS X icon, 37834 bytes, "ic13" type
Mac OS X icon, 72729 bytes, "ic13" type
Mac OS X icon, 76602 bytes, "ic13" type
XML 1.0 document text, ASCII text
XML 1.0 document text, Unicode text, UTF-8 text
data

Okay! we’re getting somewhere!

According to Magic file directory on FreeBSD, the filename is usually .icns

I think the rest will be easy, let’s try this again!

$ find /System/Library -type f -name 'Developer*.icns' 2>/dev/null
/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/DeveloperFolderIcon.icns

Bingpot!

What else do we have there?

$ find . -type f -name '*Folder*.icns'
./UsersFolderIcon.icns
./DocumentsFolderIcon.icns
./SidebarHomeFolder.icns
./ApplicationsFolderIcon.icns
./PrivateFolderBadgeIcon.icns
./GenericFolderIcon.icns
./PicturesFolderIcon.icns
./SidebarDesktopFolder.icns
./PublicFolderIcon.icns
./SidebarGenericFolder.icns
./SystemFolderIcon.icns
./ServerApplicationsFolderIcon.icns
./LibraryFolderIcon.icns
./ReadOnlyFolderBadgeIcon.icns
./SidebarBurnFolder.icns
./OpenFolderIcon.icns
./SmartFolderIcon.icns
./BurnableFolderIcon.icns
./SidebarDownloadsFolder.icns
./SidebarMoviesFolder.icns
./SidebarPicturesFolder.icns
./UtilitiesFolder.icns
./SidebarSmartFolder.icns
./HomeFolderIcon.icns
./SidebarApplicationsFolder.icns
./MovieFolderIcon.icns
./SidebarDocumentsFolder.icns
./DropFolderBadgeIcon.icns
./DownloadsFolder.icns
./GroupFolder.icns
./SidebarUtilitiesFolder.icns
./SidebarMusicFolder.icns
./DeveloperFolderIcon.icns
./NewFolderBadgeIcon.icns
./MusicFolderIcon.icns
./DesktopFolderIcon.icns
./SitesFolderIcon.icns
./SidebarDropBoxFolder.icns

These are good! Let’s look at them!

I wrote a script that converts all these .icns files to proper PNGs using the iconutil program.

#!/bin/sh

for icns in *.icns;
do
  iconutil -c iconset "${icns}"
done

for iconset in *.iconset;
do
  cp "${iconset}/icon_512x512@2x.png" "${iconset}-icon_512x512@2x.png" ||
    cp "${iconset}/icon_32x32.png" "${iconset}-icon_32x32.png"
done

Here are the ones that we see basically every day!

Here are some of the exotic ones that we don’t always notice, in an alphabetical order.

The Burnable folder! Still supported even on macOS Ventura!

BurnableFolderIcon iconset icon 512x512 2x

The Developer folder! If you don’t know where you should put your code, this is the right place for it!

DeveloperFolderIcon iconset icon 512x512 2x

The Group folder!

GroupFolder iconset icon 512x512 2x

The Library!

LibraryFolderIcon iconset icon 512x512 2x

The Public folder! Want to share something with your local network? Put it here!

PublicFolderIcon iconset icon 512x512 2x

The Server Applications Folder! Wait, what? I’ve never seen this before. If you know what this is, please leave a reply 🙂

ServerApplicationsFolderIcon iconset icon 512x512 2x

The Sites folder! If you’re new to macOS, this is a kindly reminder that macOS ships with Apache2. Yes, and ~/Sites is the default UserDir, i.e. http://localhost/~yourusername

$ grep '^UserDir' /etc/apache2/extra/httpd-userdir.conf 
UserDir Sites

SitesFolderIcon iconset icon 512x512 2x

The System folder!

SystemFolderIcon iconset icon 512x512 2x

The User folder!

UsersFolderIcon iconset icon 512x512 2x

And finally… the Utilities folder which lives inside the Application folder!

UtilitiesFolder iconset icon 512x512 2x

As sad as it is, these are the old icons, i.e. pre-Ventura (I think). I wish if there was a way to click on a switch and change it back, since it’s all here anyway!

That’s all folks…

Reply via email.

Antranig Vartanian

February 8, 2023

Turns out when you start MariaDB for the first time it prints technical messages and theeen it says:

Please report any problems at https://mariadb.org/jira

The latest information about MariaDB is available at https://mariadb.org/.

Consider joining MariaDB's strong and vibrant community:
⠀https://mariadb.org/get-involved/

Starting mysql.

I love this!

I think we should add something similar to FreeBSD, where after the installation is done it says something like:

Please report any problems at https://bugs.freebsd.org/
The latest Handbook is available at https://freebsd.org/handbook/

Consider joining FreeBSD’s worldwide community:
https://docs.freebsd.org/en/articles/contributing/

Thank you for choosing FreeBSD!

Wait, maybe we have such a message? I have to check and then patch if we don’t 🙂

That’s all folks…

Reply via email.

Antranig Vartanian

February 3, 2023

Turns out this is what happens when you leave me alone at a supermarket.

Basically I start playing with all the electronics I find.

I found this Barcode scanner with Windows 8 (for some reason) and the browser was open.

I was thinking of idling it at my Armenian blog, but I think this was better!

It’s showing the “Get Blogging!” website 🙂

That’s a all folks…

Reply via email.

“You try, you fail, you try, you fail, but the only true failure is when you stop trying.”

Madame Leota, Haunted Mansion

Reply via email.

iPad with Magic Keyboard

When I was cleaning our office couple of days ago I found a Magic Keyboard. I always liked these, I had one years ago when I had a very old iMac (which was running OpenBSD at the time!)

I brought it home and connected it to my iPad. Looks like I have a new blogging station.

I’ve seen people who walk around with their iPads connected to a keyboard case. Some of these cases are very expensive.

While this is a very cheap solution, I would not recommend it to anyone, because you’d need a proper desk.

Unlike a laptop, which can be used while sitting in a corner of a hotel, even without a desk/table/chair. Just put it on your lap.

But then again, this might be a nice setup while on the move.

Actually, I’m writing this from my iPad+MagicKeyboard right now 🙂

And just like, see you all next year 😉

That’s all folks…

Reply via email.

Apple updated macOS’s date(1)!

In 2020 I blogged that

[…] macOS is becoming less Unix-y every year, date(1) is outdated […]

While I was coding, I thought that I’m (SSH’d to) on my FreeBSD machine but I was on macOS, I noticed that the -I flag suddenly works.

I wondered if Apple has updated the date(1) command in macOS Ventura.

Luckily, I have macOS Monterey at home as well.

Here’s the date(1) command on macOS Monterey

Screen Shot 2022 12 29 at 4 14 21 PM

Here it is on macOS Ventura

Screenshot 2022 12 29 at 4 22 45 PM

Ah, so it works!

Did Apple update something? I’m pretty sure it’s mentioned in the history section of the man page.

Here’s the man page of date(1) on macOS Monterey

Screen Shot 2022 12 29 at 4 25 18 PM

Wait, what?

What about the man page on macOS Ventura?

SCR 20221229 mr9

Well… Either someone forgot to update the man page on macOS Ventura, or someone forgot to merge the code properly on macOS Monterey

In either cases, I’m happy that Apple noticed the change and pulled the updated code from FreeBSD!

That’s all folks…

Reply via email.

Antranig Vartanian

December 22, 2022

I’m trying to migrate my life to Obsidian, turns out they have Vim key bindings for the editor. As soon as you enable it, it asks you a question.

Screenshot 2022 12 22 at 6 29 21 PM cropped

Nice!

Reply via email.