Tag Archives: debugging

Enabling DTrace in Python on FreeBSD (again)

4 Replies

Last night I was running our usual Greybeard AMA on FreeBSD’s Discord server, when someone asked “I’ve been using Linux for years, but I also like FreeBSD. what can I do for FreeBSD, and what can FreeBSD do for me, as a Python Full Stack Developer?”

I started talking about FreeBSD Jails, ZFS, Boot Environments and more, but I also wanted to focus on DTrace. The ability to dynamically trace on production still sounds like magic to me. I know it isn’t, I’ve read the code and the papers, but the fact that I can ask the operating system questions on the fly, without recompiling, is amazing.

So when I was demoing DTrace, I also wanted to show the DTrace integrations with Python. Little did I know that the Python package on FreeBSD was not compiled with the --with-dtrace option.

As a sane person, I setup a Jail (using Jailer, of course), cloned the FreeBSD ports tree and added the --with-dtrace option back. It did not compile.

The first issue that we encountered was the following:

--- Include/pydtrace_probes.h ---
dtrace: option requires an argument -- s

ah yes, looking over the Makefile we see the following

Include/pydtrace_probes.h: $(srcdir)/Include/pydtrace.d
    $(MKDIR_P) Include
    $(DTRACE) $(DFLAGS) -o $@ -h -s $<
    : sed in-place edit with POSIX-only tools
    sed 's/PYTHON_/PyDTrace_/' $@ > $@.tmp
    mv $@.tmp $@

(you can also view the source here)

As far as I can tell, the $< thingie does not work with BSD Make, so how about if we try using gmake instead?

I did the following changes to the Makefile in the FreeBSD Ports tree. Basically adding gmake into USES=.

 USES=          compiler:c11 cpe ncurses pathfix pkgconfig \
-               python:${PYTHON_DISTVERSION:R},env readline shebangfix ssl tar:xz
+               python:${PYTHON_DISTVERSION:R},env readline shebangfix ssl tar:xz gmake

Now let’s try compiling again.

We get the following errors now:

ld: error: undefined symbol: __dtraceenabled_python___function__entry

Ah yes, linking issues.

After an hour of digging we learned that the DTRACE_OBJS= variable is set to… nothing. But it needs to be set to Python/pydtrace.o. I was not able to fix this issue properly (in configure, configure.ac, or whatever madness that GNU Autotools use), so I just changed the line manually in the Makefile.

Now let’s try compiling again.

We get the following error:

./Python/sysmodule.c:223:24: warning: passing 'const char *' to parameter of type 'char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]

Someone in the Discord chat recommended that I disable the LTO (Link-Time Optimization) option.

I did make clean, I patched that line again, and here we go, one more time.

It’s compiling, it’s compiling, it’s compiling… aaaaand we’re done! Let’s do make install

===>  Installing for python311-3.11.11
===>  Checking if python311 is already installed
===>   Registering installation for python311-3.11.11
[python.srv0.hackerspace.am] Installing python311-3.11.11...

good! now let’s try DTrace with Python!

In one terminal, I ran Python, and in another one I got the following

# dtrace -l -n 'python*:::'
   ID   PROVIDER            MODULE                          FUNCTION NAME
85302 python8737 libpython3.11.so.1.0                         sys_audit audit
85303 python8737 libpython3.11.so.1.0                  sys_audit_tstate audit
85304 python8737 libpython3.11.so.1.0          _PyEval_EvalFrameDefault function-entry
85305 python8737 libpython3.11.so.1.0            dtrace_function_return function-return
85306 python8737 libpython3.11.so.1.0          _PyEval_EvalFrameDefault function-return
85307 python8737 libpython3.11.so.1.0                   gc_collect_main gc-done
85308 python8737 libpython3.11.so.1.0                   gc_collect_main gc-start
85309 python8737 libpython3.11.so.1.0  PyImport_ImportModuleLevelObject import-find-load-done
85310 python8737 libpython3.11.so.1.0  PyImport_ImportModuleLevelObject import-find-load-start
85311 python8737 libpython3.11.so.1.0          _PyEval_EvalFrameDefault line

Woohoo! Now I’m happy!

Okay, so what did we learn?

  • There’s a little bit of GNUMake-ism in Python’s Makefile. Either we have to use gmake in FreeBSD when building Python or we need to submit an alternative to the upstream
  • Die GNU Autotools… I mean… the GNU Autotools usage in Python has an issue, where the generated Makefile does not set the DTRACE_OBJS correctly. Can anyone help with this? As a Pascal guy, reading GNU configure files makes me wanna die.
  • LTO is problematic. Solutions?

The real question is, how can we enable DTrace by default in FreeBSD packages for Python. DTrace is one of our market advantages and we should find a way to enable it everywhere we can.

This was a fun Greybeard AMA, where we touched multiple parts of the system. Looking forward for next week!

That’s all folks…

Reply via email.

dtrace.conf is back as dtrace.conf(24)

Woke up middle of the night to grab a cup of water, decided to check Mastodon, and what do I see?

dtrace.conf(24) Tickets, Wed, Dec 11, 2024 at 9:00 AM

This makes me very happy! I love seeing DTrace in the wild, and having more DTrace content out there is beneficial to everyone in the DTrace community.

Obviously, being a Syrian with passport issues, I will not be able to attend, but hopefully everything will be recorded and published online. I’ll try to make it to dtrace.conf(28).

Have fun everyone!

Reply via email.

Wrong Indicators

Bryan Cantrill has this amazing talk about debugging where he tells the story of Three Mile Island.

After watching that talk all I thought was “well, let’s hope this doesn’t happen in my life”, and by “my life”, I meant my personal or work server, not my AFK life!

55 days ago my girlfriend and I moved to a new apartment downtown the capital. I like everything about this house, specially that many things are electric, including the stove.

Like a sane person, when I see a stove with multiple levels (1, 2, 3) I assume that the lowest number is the lowest and highest number is the highest.

Now you’d think and say “Antranig, didn’t you notice that your cooking was talking 2 hours, so there must be something wrong?”

Oh no, my friend, very much no. As you can see we have two stove tops, a small one and a big one. Now, the small one is working very fine. At the highest level it heats more than at lowest level.

But the big one, the big stove top, not so much.

We thought that there’s a problem with that top and used it only during slow emergencies.

One day I come home from work and Lilith is laughing. I asked “what happened?” and she replied “you’re not gonna believe this!”

Well she was right, it’s been couple of days now and I still can’t believe this. I mean if both of the stove tops were in reverse order, I would understand that someone was very Unix-y and they wanted to design it similar to nice.

But when each of those knobs are the exact opposite of each other, it makes you think, “why me?”

Why me indeed.

That’s all folks!

Reply via email.

Erlang dbg Intro

If there’s one programming language that changed my life, that’s Erlang. After using Erlang for couple of years, I “moved” to Elixir, which is based on Erlang’s VM.

One the most important aspects of Erlang’s VM is that it’s a “real” VM, there’s a kernel, processes, messaging facilities and many more.

Lately I’ve been debugging a huge Erlang application whose architecture I was not very familiar with and I needed to find a way to see what kind of messages are being sent and received, which Modules and Functions are being called and what are they returning.

So I wanted to write a small How-To for me and you, in case we need it again in the future.

Okay, for this example I’ll be using Elixir TCP Server, a simple TCP server that gets data and sends it back to its origin.

First, let’s clone the repo.

antranigv@zvartnots:prj $ git clone https://github.com/SonaTigranyan/ElixirTcpServer

Okay, now let’s run the server

antranigv@zvartnots:ElixirTcpServer $ iex -S mix
Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Compiling 3 files (.ex)
Generated tcp_server app
Interactive Elixir (1.10.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

Good! By default, the TCP server listens on port 9000, as specified in the Application Tree.

Okay, we can send data now 🙂

antranigv@zvartnots:ElixirTcpServer $ echo test | nc localhost 9000
test

Or in an interactive way!

antranigv@zvartnots:ElixirTcpServer $ nc localhost 9000
First mesage!
First mesage!
Good TCP server!
Good TCP server!
bye
antranigv@zvartnots:ElixirTcpServer $

Good! As you can see the connection is closed when the server gets bye.

Okay, say we want to trace the do_send function, observe what does it get and return.

iex(2)> :dbg.start()
{:ok, #PID<0.191.0>}
iex(3)> :dbg.tracer()
{:ok, #PID<0.191.0>}
iex(4)> :dbg.tpl(TcpServer, :do_send, [{:_, [], [{:return_trace}]}])
{:ok, [{:matched, :nonode@nohost, 1}, {:saved, 1}]}
iex(5)> :dbg.p(:new_processes, :c)
{:ok, [{:matched, :nonode@nohost, 0}]}
iex(6)>
(<0.198.0>) call 'Elixir.TcpServer':do_send(#Port<0.545>,"Message from client!\n")

Okay, first we start the dbg facility, and then we start a tracing server on the local node.

After that, we use function tpl to specify which local calls we want to trace.

And in the end we use the p function to start tracing the calls (c) of all new_processes 🙂

Now, when the do_send function is called, we see what it gets.

And when we send bye, we see the following:

(<0.198.0>) returned from 'Elixir.TcpServer':do_send/2 -> ok

And all of this is happening when the software system is running. In production, we can do the same, by either attaching to the node or connecting to it!

That’s all folks! 🙂

Reply via email.