How to Hack Together a Graphical Python Debugger

As a Python developer, one of the most valuable tools in your toolbox is a good debugger. When you‘re stuck on a tricky problem or trying to understand unexpected behavior in your code, being able to pause execution, step through your program line-by-line, and inspect variables can be a lifesaver.

Python ships with a basic command line debugger called pdb, which gets the job done but can be a bit cumbersome to use. Wouldn‘t it be nice to have a graphical interface where you can see your source code, the variables, and control execution with the click of a button? In this post, I‘ll show you how we can hack together our very own graphical Python debugger.

But before we dive into the implementation, let‘s take a quick tour of some debugging concepts and tools.

Debugging in Python

Python‘s built-in pdb module provides a command line interface for debugging. You can launch it by running python -m pdb myfile.py or by importing it and calling pdb.set_trace() anywhere in your code. This will pause execution and give you a (Pdb) prompt where you can type commands to continue (c), step (s), print variables (p), and more.

There are also several popular graphical debuggers for Python, including:

• PyCharm‘s visual debugger
• WingIDE‘s graphical debugger
• Visual Studio Code‘s Python extension debugger

These give you a more intuitive interface with your source code displayed and buttons to control execution. Under the hood, they‘re making use of the same hooks Python provides for pdb; they‘ve just built a user-friendly GUI on top.

A Crash Course in Python Bytecode

To understand how our debugger will work, we need to peek under the hood of the Python interpreter for a moment. When you run a .py file, the Python compiler parses your source code and compiles it to bytecode. This is a lower-level representation of your program as a sequence of instructions.

For example, consider this simple function:

def greet(name):
    print(f"Hello, {name}!")

If we use the dis module to inspect the bytecode, it looks something like this:

 0    LOAD_GLOBAL     0 (print)
 2    LOAD_CONST      1 (‘Hello, ‘)
 4    LOAD_FAST       0 (name)
 6    FORMAT_VALUE    0
 8    BUILD_STRING    2
10    CALL_FUNCTION   1
12    POP_TOP
14    LOAD_CONST      0 (None)
16    RETURN_VALUE

Each line is a single bytecode instruction. LOAD_FAST pushes a local variable onto the stack, LOAD_CONST pushes a constant, etc. These get executed one at a time by the Python virtual machine.

The bytecode is wrapped in a code object, which contains additional metadata about the code, such as the filename, line numbers, and local variables. As the Python VM executes bytecode, it maintains a call stack of frame objects. Each frame represents a function call and points to the code object being executed and the current instruction.

With this background, we‘re ready to think about building a debugger. We need a way to pause execution of the Python VM before any given bytecode instruction is executed. We can then inspect the current frame to see the source code and local variables. Finally, we need a way to control execution by stepping forward or continuing to the next breakpoint.

Luckily, Python exposes some low-level hooks that let us do exactly this! Let‘s see how they work.

Tracing Python Execution with sys

The sys module is mostly used for interacting with the Python interpreter, but it also provides a useful hook for tracing execution. The settrace() function lets us register a callback that will be invoked for various events during code execution.

Here‘s a simple example:

import sys

def trace(frame, event, arg):
    print(f"Executing line {frame.f_lineno}")
    return trace

def greet(name):
    print(f"Hello, {name}!")

sys.settrace(trace)
greet("Guido")

If we run this, we‘ll see:

Executing line 8
Executing line 5
Hello, Guido!
Executing line 6

Our trace function is invoked for each line of code, and it receives the current frame object. We can inspect frame.f_lineno to see the current line number. By returning the trace function at the end, we tell Python to keep calling it for subsequent lines.

We can extend this further to trace every step of execution:

def trace_calls(frame, event, arg):
    if frame.f_code.co_name == "greet":
        return trace_lines
    return None

def trace_lines(frame, event, arg):
    print(f"Executing line {frame.f_lineno}")

sys.settrace(trace_calls)

Now we have two trace callbacks. trace_calls is invoked for each function call. If we‘re entering the greet function, it returns trace_lines to be invoked for subsequent line events. This lets us turn tracing on and off for only the functions we‘re interested in.

We‘re getting close now! We have a way to intercept execution at each line and inspect the frame. Now we just need to build an interface around it.

Building a Web-Based Graphical Debugger

For our debugger interface, we‘ll build a web app using the Flask framework. This will let us display the source code, variables, and control buttons in a browser.

We‘ll structure our app with a few key components:

• MainDebugger: The main Flask app that renders the UI.
• DebuggerBackend: A separate process that runs the user‘s code in the debugger.
• DebuggerUI: A class that generates the HTML for the debugger UI.

The MainDebugger and DebuggerBackend components will communicate via a queue. The backend will send UI updates (current line, variables) via the queue, which the Flask app will poll for and use to update the UI.

Here‘s a simplified version of the DebuggerBackend:

import multiprocessing as mp
import sys

class DebuggerBackend:
    def __init__(self, code_queue):
        self.code_queue = code_queue

    def run(self, code):
        self.code = code
        sys.settrace(self.trace_calls)
        exec(code)

    def trace_calls(self, frame, event, arg):
        if frame.f_code.co_name == "<module>":
            return self.trace_lines
        return None

    def trace_lines(self, frame, event, arg):
        self.code_queue.put(self.format_frame(frame))
        cmd = self.code_queue.get()
        if cmd == "step":
            return self.trace_lines
        elif cmd == "continue":
            return None

When the backend is launched, it sets the trace function and then execs the user‘s code. Each time a line event occurs, it puts a JSON representation of the current frame in the queue. It then blocks waiting for a command on the queue – either "step" to go to the next line or "continue" to keep running.

On the frontend, the Flask app renders the UI by pulling frame data from the queue:

from flask import Flask, render_template
from flask_socketio import SocketIO

app = Flask(__name__)
socketio = SocketIO(app)
code_queue = mp.Queue()

@app.route(‘/‘)
def index():
    return render_template("index.html", code=code)

@socketio.on("step")
def step():
    code_queue.put("step")

@socketio.on("continue")
def cont():
    code_queue.put("continue")

def debug(code):
    backend = DebuggerBackend(code_queue)
    p = mp.Process(target=backend.run, args=(code,))
    p.start()

    while True:
        frame_data = code_queue.get()
        socketio.emit("frame_update", frame_data)

if __name__ == "__main__":
    code = "x = 10\ny = 20\nprint(x + y)"
    socketio.start_background_task(debug, code)
    socketio.run(app)

When the app starts up, it launches the DebuggerBackend in a separate process with the user‘s code. In the main event loop, it pulls frame data from the queue and sends it to the browser via a SocketIO event.

The UI itself is rendered using a HTML/JavaScript template. The DebuggerUI class generates the HTML for the current frame, including the code with the current line highlighted and the local variables. Here‘s a snippet:

{% for line in code %}
    <div class="line {{ ‘current‘ if line.lineno == frame.f_lineno }}">
        <div class="lineno">{{ line.lineno }}</div>
        <div class="code">{{ line.text }}</div>
    </div>
{% endfor %}

<h3>Local Variables</h3>
<table>
    {% for var, val in frame.f_locals.items() %}
        <tr>
            <td>{{ var }}</td>
            <td>{{ val }}</td>
        </tr>
    {% endfor %}
</table>

The JavaScript code sets up listeners for the "step" and "continue" buttons to send the appropriate SocketIO events.

And there we have it! A fully functional graphical debugger for Python, all in less than 200 lines of code. Of course, a real debugger would need a lot more functionality – setting breakpoints, a full variable inspector, conditional breakpoints, etc. But I hope this demonstrates the basic principles.

Reflections and Caveats

Building this debugger was a fascinating exercise in exploring Python‘s internals. A few key takeaways:

• Python bytecode is the secret sauce behind how the interpreter works. Understanding bytecode and code objects is key to advanced Python techniques.
• The inspect and dis modules are incredibly useful for introspecting and understanding Python code.
• sys.settrace() is a powerful hook, but use it wisely. Modifying the trace function can have surprising effects and is not something you‘d want to do in production code.

Also keep in mind that this implementation is specific to CPython, the reference implementation of Python. Other implementations like PyPy or Jython may have different debugging hooks.

I encourage you to dig deeper into Python‘s internals and think about how you can apply these techniques. What other developer tools could you build with code introspection and tracing? How about a code coverage tool or a performance profiler?

I hope this post has given you a taste of the power and flexibility of Python. Happy debugging!