Category Archives: Internet

Incident Postmortem: BSD.am home server @ 3-4 July 2023

Incident Information

Between the hours of Mon Jul 3 03:05:59 2023 and Tue Jul 4 01:10:15 2023 the home server named BSD.am (also known as pingvinashen.am) was completely down.

The event was triggered by a battery issue due to high temperature at the apartment where the home server resides.

A battery swell caused the computer to shut down as it produced higher than normal heat into the system.

The event was detected by the monitoring system at mon.bsd.am which notified the operators using email and chat systems (XMPP).

This incident affected 100% of the users of the following services:

  • jabber.am public XMPP server
  • conference.jabber.am public XMPP MUC server
  • օրագիր.հայ public WriteFreely instance
  • սարեան.ցանցառներ.հայ public Lobste.rs instance
  • BIND.am public DNS server and its zones
  • Multiple hosted blogs, including this one you’re reading.
  • A private ZNC server for Armenian Hackers Community
  • git.bsd.am public Gitea server
  • A matterbridge instance connecting multiple communities
  • A Huginn instance automating tasks (such as RSS to Telegram, RSS to newsletter) for Armenian Hackers Communities
  • A newsletter instance running listmonk.app
  • A private Miniflux.app server for Armenian Hackers Community
  • FreeBSD Jail users’ meetup website

Multiple community members contacted the operator (yours truly) asking for an ETA.

Response

After receiving an email at Mon Jul 3 03:06:49 2023, the Chief Debugging Officer (yours truly) started analyzing the possible issue. According to Monit (mon.bsd.am) all the services were unavailable and the server was not reachable by IP (based on ICMP).

The usual possibility, network failure at the ISP level, was ruled out, as the second home server (arnet.am) was functioning properly.

The person closest to the server physically, was the operator’s sibling (lucy.vartanian.am), however she did not have the background in Unix system administration nor in hardware maintenance. Also, she was asleep.

Hours later the siblings (yours truly) organized a FaceTime call to debug the issues remotely.

The system did boot the kernel properly, however it would shutdown before the services could complete their startup.

Clearly, the machine needed to be shipped to the operator (yours truly) to be debugged at the spot.

So that’s what the team did.

IMG 6689
Precise addresses are removed for privacy

Recovery

At the operator’s (yours truly) location, the BIOS logs have listed that the system suffered from a ASF2 Force Off. This usually means a thermal problem.

The operator (yours truly) disassembled the laptop, hoping the system needs a little dust clean-up and a thermal paste update.

Turns out the problem was actually a swollen battery.

IMG 6683
IMG 6684
IMG 6685

After removing the battery, the system booted fine. Just to be sure that the swollen battery was the root cause, a complete system stress test was ran. No issues detected (Well, except “Missing Battery”).

The systems was returned to its residency, connected to the internet and all services were accessible again.

IMG 6690
Precise addresses are removed for privacy

Next Steps

  • Install a new battery in the future, as the laptop is not connected to a UPS
  • Make sure to test the hardware during environmental changes (too cold, too hot, etc)
  • Run a simple status page with an RSS feed in a separate environment and notify users

If you’re new here, then first of all I’d like to thank you for reading this IR Postmortem article.

Yes, this was an IR Postmortem of a home server of a tiny community in a tiny country. This was not about Amazon, Google, Netflix, etc.

I wrote this for two reasons.

First, I wanted to show you how awesome the actual internet is. You see, when Amazon dies, everything dies with it. Your startup infra, your website, your hobby projects, everything.

When my server dies, only my server dies. And that’s the beauty of the internet. If you can, please, keep that beauty going.

Second, I run a small security company, illuria, Inc., where we help companies harden their environment and recover from incidents. It’s been years since I wrote an IR postmortem personally (my team members who do that are way smarter than me!), and I thought it would be a nice exercise to write it all by myself 🙂

I hope you liked this.

That’s all folks…

Reply via email.

Domains as Verification

Couple of days ago when I was browsing the internet I stumbled upon Jim Nielsen’s blog, where at the top it said

Verified ($10/year for the domain)

Screenshot 2023 05 26 at 9 37 20 PM

Luckily, his blog is so organized (unlike mine) where I found the post named Verified Personal Website in which he talked about this.

Personally, I don’t have enough CSS skills to do that, but I added a check mark next to my name on my blog (thank you Unicode!).

IMG 6638

I think this is amazing and it should be used more by bloggers everywhere. If someone opens a blog they should see a check mark. Maybe a cute one in SVG, maybe a CSS trick, maybe it’s just an image, but it should be there.

Why? so we remind people that on the internet, whenever you have a domain, you are already verified.

Can scammers scam and criminals phish? yes, indeed. But unlike the not-very-social-media, it’s hard to do that.

Ironically, having a website on the internet costs less than having a “verified” social media account, say on Twitter.

Currently, Twitter Blue costs $8/month or $84/year.

Let’s see how much would it cost to have a blog on the internet.

First thing first, you need a domain, and it can be anything that you feel awesome with. Awesome-ness is the first and only rule.

Here’s an awesome domain that I found is available using NameCheap.

Screenshot 2023 05 26 at 9 23 37 PM

This is awesome!

Next, we need to host our website. Well, lemme check my favorite server hosting platform, Vultr.

Vultr pricing

A machine with a single CPU and a 1GB of RAM, that’s plenty!

I mean, with that much power, you can easily run WordPress (if you’re using caching).

Or, if you don’t want to get techy-techy at all, you can use a static site generator. You like Markdown and text files? There’s Hugo for you. Do you want to just click on buttons and BOOM, your website is ready? Have a look at Publii!

So, how much does it cost in the end? Here’s how it looks like if you pay annually or monthly, per year.

A/M Twitter Blue Website on the Internet
Monthly $8×12 = $96 $8×12 + $10 = $70
Annually $84 $8×12 + $10 = $70

So yes, it is cheaper to have a website on the internet.

Wait a second, annually vs monthly looks the same? OF COURSE IT DOES! THIS IS THE INTERNET! We want you to think “huh, 70 dollars? well that’s dope” and not about “well, if I pay annually now, I will save 12 dollars” and then completely forget about that service anyway.

Oh, and did I tell you about the features of having a website on the internet? Well we don’t have a list, but here’s some things from the top of my head.

  • You get to be verified, because welcome to the internet
  • You get to post whatever you want
  • you get to edit them! can you believe that?
  • You can upload photos and make it looks like a photo blog
  • Unlike other platforms, which seemed to be for photographers but not anymore, you can tag things, and make albums!
  • You can upload podcasts!
  • Hell, and if you ever want to leave, you can just redirect your domain to somewhere else 🙂

And I’m not even talking about the other awesome features of having a domain, like, custom emails! Be that person that does NOT have a @gmail.com, but @AwesomeIsHere.net!

And hey, Twitter Blue might die, Twitter might die, every other company might die, but the internet will not 🙂

That’s all folks…

Reply via email.

Downtime for the rest of us

If the homebrew server club had an official membership based on technicality, then I would be a very proud member, but it does not have a membership application. That being said, I am still a proud member of HBSC, as I’ve been running a home server for a decade now.

I can’t say that it’s been easy, but it has been evolving. When I tried setting up my first server, I had issues with an ISP that didn’t allow me to have more than a single public static IP address.

Over time, ISPs changed, servers have changed, but the only thing that remained the same is me running my server from my home.

Now, I do have multiple IPs, a VLAN with my ISP that we’ve agreed on the setup, an internal email where they answer my questions without me calling the general support line and finally a publicly available Looking Glass that anyone can use.

Unfortunately, it’s not all sunshine and roses. My biggest request for the last couple of years has been the same: a status page.

You know, that simple web page that tells you if a service is down?

Interestingly, when I was researching ISPs (that’s a post for another day) I noticed that most ISPs don’t provide a status page.

Some ISPs (like Google Fiber) ask for an address, while others ask you to log in.

I understand that an ISP is a complex beast, and it would not be an easy task to say “we have an issue”, but hey, someone has to start trying.

Oh, I forgot, the downtime mentioned in the title!

Well, my personal blogs don’t have a lot of traffic (unless if someone posts a link to the Orange Website, then I get 20K+ viewers per day), but many people use my services, such as my Jabber/XMPP chat server, a publicly available blogging system an Armenian tech forum and so on.

All of the local ISPs had issues this week and their first response was to fix the outbound traffic. So for most people in the country, they didn’t care, as long as they were able to use Telegram and log into their Meta-owned social media services.

But for me and my community, we had to wait almost 18 hours for them to fix the internal network issues.

However, I am still a proud member of HBSC, because unlike Big Tech companies, if I go down, only I go down. But if a cloud goes down, everyone goes down with them.

See you at the next downtime 😉

Reply via email.

Reply from National Vulnerability Database Team regarding Legacy Data Feeds

Couple of days ago when I was assisting a customer, I recommended that they follow the National Institute of Standards and Technology’s (a.k.a. NIST) Information Technology Laboratory’s Computer Security Division’s National Vulnerability Database’s (a.k.a. NVD a.k.a. that place that publishes the CVEs) data feeds. (Apologies for the bad intro)

So, these are RSS feeds that “contains the most recent CVE cyber vulnerabilities published within the NVD”

Unfortunately, I saw a notice at the top of the page, which got me really worried. It says

In September 2023, the NVD plans to retire all legacy data feeds while guiding any remaining data feed users to updated application-programming interfaces (APIs).

Usually, I’d panic and start ranting on my blog, but this is the NVD we’re talking about. They are a US government project that has been doing a lot of good and they are sponsored by the CISA, an agency that does many good things not just for US citizens, but citizens of our planet.

I started digging to understand what exactly is going to be retired and most importantly, why?

The NVD has made an amazing change timeline that has the following

The NVD plans to retire the RSS data feeds. The NVD plans to enable reCAPTCHA across all webpages and to retire webpages intended to support web scraping (e.g., Full Listings) before its APIs existed.

Okay, NOW I’m worried.

I’ll break this into two parts.

Why we need RSS feeds

You see, the internet relies on RSS, and I’m not just saying that because most of my audience uses RSS daily. The reason is much deeper than that.

As Dave Winer blogged a month ago

RSS is a thing like roadways and paths of rivers, they change very slowly. Think about qwerty keyboards. That’s what we’re talking about here. Agreements between products to interop. RSS is just like the gauge of rails, or always driving on one side of the street. A convention that makes progress possible. #

Scripting.com, Saturday, January 28, 2023

There are three products/protocols that I use daily, it’s Slack (for work), XMPP (for friends and family) and Telegram (for Armenian tech communities).

There are specific things that I should deliver for all these and that is messages, alerts, notices.

For my work, I should be able to get news if there’s a security issue on FreeBSD, because we use that. For friends and family I should deliver notices if there are any issues or upcoming maintenance to our servers. For my telegram communities I should update them if we’re having any new meetups, events, podcasts.

But, instead of writing a software that fetches, parses, analyzes and does something-something to these messages, I use RSS! FreeBSD has an RSS feed for Security Advisories. All I do in Slack is /feed subscribe https://www.freebsd.org/security/feed.xml and now, every time there an SA for FreeBSD, I get notified in Slack.

For friends and family? I have Huginn agent that parses RSS and send an XMPP message. For Armenian tech communities? I read a website’s RSS and a bot posts it in a group.

You get the idea.

RSS is all about “things working together”, there is no need to write a specific piece of for that specific thing.

And for years, I’ve relied on NVD’s RSS data feed to notify customers, tell them what to upgrade, if they need to upgrade and why to upgrade.

These RSS feeds are part of my professional life, a way for me, and people like me to know if we should be in panic mode or not.

So…

Okay, now what?

I believe in communication. I was very sure that my questions will be answered by the NVD, so I sent a message!

Greetings dear NVD team, NIST team and Computer Security Division,

While browsing your website, I have noticed the following change:

> In September 2023, the NVD plans to retire all legacy data feeds and the 1.0 APIs.

This became very disturbing, as many companies (including mine) rely on the data feed provided by NIST’s NVD.

I have two questions:
1) Is there *any* chance to keep the RSS feeds?
2) Is it okay if others (i.e. I) generate an RSS feed from your new API, if your final decision for q#1 is no?

[ . . . ]

If I may, that being said, I’m sure there’s a good reason, so my other question is:
What are/were the technical issues with RSS? Could it be bypassed or hacked around?

Thank you for all the work that you do, and thank in advance.

Kind regards,

I was right! They did answer all of my questions! I got a reply yesterday, here it is.

1) Is there *any* chance to keep the RSS feeds?

We have no plans to continue providing the RSS feeds located at
https://nvd.nist.gov/vuln/data-feeds#RSS
https://nvd.nist.gov/feeds/xml/cve/misc/nvd-rss.xml
https://nvd.nist.gov/feeds/xml/cve/misc/nvd-rss-analyzed.xml

Additionally, it is important to point out that per our announcement at https://nvd.nist.gov/general/news/change-timeline, the RSS feeds will be retired in March, not September. If you were not aware of these announcements we highly advice joining the NVD Google Group to stay better informed (https://groups.google.com/a/list.nist.gov/g/nvd-news).

2) Is it okay if others (i.e. I) generate an RSS feed from your new API, if your final decision for q#1 is no?

All NIST publications are available in the public domain. Organizations seeking to automate the retrieval of NVD data should use the NVD’s Application Programing Interfaces (APIs).
Services which utilize or access the NVD are asked to display the following notice prominently within the application: “This product uses data from the NVD API but is not endorsed or certified by the NVD.” You may use the NVD name to identify the source of the data. You may not use the NVD name, to imply endorsement of any product, service, or entity, not-for-profit, commercial or otherwise. For information on how to the cite the NVD, including the database’s Digital Object Identifier (DOI), please consult NIST’s Public Data Repository.

3) What are/were the technical issues with RSS? Could it be bypassed or hacked around?

The RSS feeds were considered to be overly simplistic and underutilized, they were determined in scope of retirement for these reasons as part of a larger effort to consolidate our output formats as we move towards the APIs. If you would like to submit a user story explaining the benefits and needs that the APIs currently do not meet we would pass that along to the development team for consideration in the future.

Okay. I agree! RSS is very simplistic, but that’s the point! it’s supposed to be simple. I mean, it’s simple enough that podcasts are RSS feeds.

And to be clear, I DID check the NVD’s new Vulnerabilities API, it’s awesome, it’s nice, it’s documented very well, kudos to the team, they did an amazing work, I’m sure it wasn’t easy. It has, for sure, more features than RSS could provide.

What to do about it?

I understand that the NVD is pushing the REST API, and I also understand why. But I really don’t want to write a “wrapper” for every service and technology that I use.

Here are my two questions.

  1. Will systems break because of this? Are you using these feeds? Do you rely on them for yourself or your organization?
  2. Will there be an interest by the InfoSec community to write a wrapper that generates a new RSS feed from NVDs new API?

Personally, if there’s an interest or not, I will be stopping everything I’m working on to create this NVD-to-RSS generator, as I very much rely on it. It will be open-source, obviously. What I should build is a drop-in replacement, where you change the feed URL, and everything works like before. (Well, I have to finish my other open-source commitments first, then I should work on this 😀 hopefully it wont take long.)

I would like to thank the NVD for keeping these feeds for all these years and congratulate them for their new APIs, I’m sure many good things will come out of these APIs.

And thank you for reading 🙂

That’s all folks…

Reply via email.

Comments are back

When I started blogging 8 years ago I used WordPress. One of its features was comments. However, when I started my English blog (the one that you are reading right now) I chose Hugo and then migrated my Armenian blog to Hugo as well.

This had two amazing features. First, no more managing PHP and MySQL, since Hugo is a static sigh site generator, second, no more dealing with comments.

During the last years more and more people have been contacting me over email/Twitter/Telegram to give me feedback about a post that they read. This is mostly about my Armenian blog. I don’t get much feedback from the English blog, unless someone posts it on HackerNews (then I get A TON).

I started missing comments, a centralized place to read all the feedback and an easy way for the reader to post them.

In Hugo’s documentation I see there’s a section about comments but it recommends Disqus. I don’t like 3rd party services. Lucky someone on Twitter recommended an alternative, Isso!

Isso was very easy to deploy. I created a FreeBSD Jail, did a pip install isso and then setup a reverse proxy. Add some JS scripts here and there in the template, and it’s all done!

I’m not sure if I’ll be able to fight spam. I still need to setup an SMTP server so it emails the commenters if someone replied to their comments, but that’s a project for the weekend.

That’s all folks…

Reply via email.

Good bloggers write a lot

I’ve been thinking lately that I am NOT able to blog a lot and I always blame external factors, “Oh I don’t have time” or “oh there’s no pagination in my theme so there’s no point of blogging daily, yet.”

But in reality, turns out I’m just being lazy.

I’ve been reading Jamie Zawinski’s blog for years, via RSS, of course. Couple of days ago I opened it via my web browser, an woah those number hit me hard!

As you can see, there are 366 days in a year but jwz happens to have more posts per year than that! Look at year 2012, there are 870 posts!

I mean, I know that my favorite blogger, Rubenerd blogs a lot, but I never knew how much.

I know he has 10 posts per, and his blog currently says

Page 1 of 758 → Older posts

And I know he started blogging since 2004, so if you do the math using bc,

$ echo '(758 * 10) / (2021 - 2004)' | bc -l
445.88235294117647058823

Actually, lately I’ve learned about expr, it’s very handy in command line scripts!

$ expr \( 758 \* 10 \) / \( 2021 - 2004 \)
445

What I’m trying to say is, I don’t know how people blog regularly, it’s not that I don’t have any ideas in my head, there’s always something to say, something to share, something to write about. If it’s not technical then at least it’s political.

Recently Lilith suggested that I should try to allocate 30 minutes a day to write some posts, even if it would end up into the drafts. This is me trying to do that, while drunk 🙂

That’s all folks!

Reply via email.

Blogging Regularly

Ruben blogged recently about blogging regularly and it kind of hit me: Why don’t I blog regularly?

I love blogging. I improved my Armenian by blogging for years, I wanted to be a blogger so bad that I asked my friends to rent me a domain and a hosting service since I didn’t have money when I moved to Armenia after the war.

But yet again, it’s very hard for me to write my thoughts in English. Armenian? Yes, sure, I can write a very complex sentence very easily. English, however, the language that I think in, the language that I grew up having a love&hate relationship with, is not the language that I’m good at writing. I can talk English very well, at least I’ve been told, but writing is not there yet.

There are a lot of points that Ruben made that I love to be more mainstream. Use ANY blogging platform, literally any, as long as they don’t treat you as the product (Medium as an example). Write about anything, everything. I would love to hear about your daily life, how you solve problems, no matter if it’s about that very complex DB issue you’ve been having or the water pipe that has been leaking. They are all interesting.

At the end of the day the internet is the place that allowed everyone to speak. Now we are fighting over who gets to be heard.

But with blogs and RSS, everyone will be.

That’s all folks.

Reply via email.