Execute Code in a Remote Python Process

December 28, 2025 - 8 mins read

• python
• debugging
• pep

© Rafael Weingartner-Ortner

Python 3.14 introduces PEP 768, a zero-overhead debugging interface that “allows debuggers and profilers to safely attach to running Python processes without stopping or restarting them” [1]. One main application of this new feature is enabling pdb to attach to live processes (via pdb -p <pid>), which allows interactive debugging of Python apps in real-time.

If you want to learn more about the external debugger interface, I recommend this talk from EuroPython 2025 by Pablo Galindo Salgado.

Besides pdb -p, the new interface allows “anyone”¹ to execute arbitrary Python code within the context of another Python process by using the new API sys.remote_exec(). This function takes a PID and the path to a script containing the code to be executed. Upon calling, it writes the path to the “debug-script” into a specified spot in the memory of the remote process which will load and run the file the next time a safe evaluation point is reached.

This means you can write any Python code and execute it in a remote interpreter — quite 🆒, right?

Imagine

… you have some sort of Python web server that runs in production. Since you are a good software engineer, you added various logging statements using different levels to log what your running application is doing. But in production, you don’t log on DEBUG, of course.

Now, you notice that your application logs errors but you lack the context of the error. Additional (debug) information which would greatly help to identify the causes, is not available. Too bad that we are not on the dev system…

What if you could temporarily change the log level to DEBUG to see what’s really going on? No way, I would have to restart the app for that.

Guess what, with Python 3.14 there is a way! 🔝

Setup

First, create a virtual environment with Python 3.14. For this example, we will use the built-in venv module as it allows to use copies instead of symlinks² (I will explain why we want this later).

$ uv run -p 3.14 python -m venv --copies .venv
$ source .venv/bin/activate

Server

I created a minimal example of a server that logs an error and additional context in a debug message.

 1# file: server.py
 2import logging
 3import os
 4import time
 5
 6logging.basicConfig(level=logging.WARNING)
 7logger = logging.getLogger("main")
 8
 9
10def main() -> None:
11    print(f"Server started. PID: {os.getpid()}")
12    while True:
13        time.sleep(5)
14        logger.debug("This gives the missing context for the error.")
15        logger.error("An error occurred!")
16
17
18if __name__ == "__main__":
19    main()

Let’s start³ the server.

$ 𝜋thon server.py
INFO:main:Server started. PID: 8293
ERROR:main:An error occurred!
ERROR:main:An error occurred!

As expected, we see errors but we lack additional context.

(Note that this is a very simplified example. In reality, you should already give more context in the error message, of course. But there will always be useful information that is “hidden” in debug logs.)

Luckily, we now have a new weapon in our debug arsenal: Remote exec to the rescue! 🦺

Debugger

The following code is a mini-CLI tool that can change the log level of the "main" logger in a remote process to the level of choice.

 1# file: debug.py
 2import argparse
 3import sys
 4import textwrap
 5from tempfile import NamedTemporaryFile
 6
 7
 8def main(pid: int, log_level: str) -> None:
 9    script = textwrap.dedent(f"""
10        import logging
11        logger = logging.getLogger("main")
12        print("Changing log level to {log_level!r} via remote exec...")
13        logger.setLevel(logging.{log_level})
14        """)
15    with NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
16        script_path = f.name
17        f.write(script)
18
19    print("Injecting script to change log level to", log_level)
20    sys.remote_exec(pid, script_path)
21    print("Log level changed ✅")
22
23
24if __name__ == "__main__":
25    parser = argparse.ArgumentParser(
26        description="Change the log level of a remote server."
27    )
28    parser.add_argument(
29        "--log-level",
30        type=str,
31        choices=["DEBUG", "INFO", "WARNING", "ERROR"],
32        required=True,
33        help="Set the log level of the remote server.",
34    )
35    parser.add_argument(
36        "-p",
37        "--pid",
38        type=int,
39        required=True,
40        help="PID of the remote server process.",
41    )
42    args = parser.parse_args()
43    main(pid=args.pid, log_level=args.log_level)

Let’s look into the most important parts of this code — leaving the logic to set up argparse aside.

First, we write the Python code that we want to run in the remote process into a file. In this case, we import the logging module, get the "main" logger, and set the log level to the input argument.

 7    script = textwrap.dedent(f"""
 8        import logging
 9        logger = logging.getLogger("main")
10        print("Changing log level to {log_level!r} via remote exec...")
11        logger.setLevel(logging.{log_level})
12        """)
13    with NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
14        script_path = f.name
15        f.write(script)

Then, we “inject” the code into the remote process:

20    sys.remote_exec(pid, script_path)

The sys.remote_exec function will return immediately and the code will be executed by the target process at the next safe opportunity.

Trial and error

Attempt 1 - my user can do anything

Let’s run the debug script:

$ python debug.py --log-level DEBUG -p 8293
Injecting script to change log level to DEBUG
Traceback (most recent call last):
  File "/home/rafael/repos/website/content/posts/python_remote_exec/debug.py", line 46, in <module>
    main(pid=args.pid, log_level=args.log_level)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rafael/repos/website/content/posts/python_remote_exec/debug.py", line 22, in main
    sys.remote_exec(pid, script_path)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 1] Operation not permitted

Wait… what happened? 🔎 It looks like we lack permissions. Let’s take a look at the section Permission requirements in the Remote debugging attachment protocol documentation:

Attaching to a running Python process for remote debugging requires elevated privileges on most platforms. The specific requirements and troubleshooting steps depend on your operating system:

Linux

The tracer process must have the CAP_SYS_PTRACE capability or equivalent privileges. (…) Try re-running the command with elevated privileges: sudo -E !!

(If you are on macOS or Windows, please check the Permission requirements — sudo should work on macOS.)

Attempt 2 - use sudo, of course

Ok, let’s try with sudo⁴

$ sudo .venv/bin/python debug.py --log-level DEBUG -p 8293
Injecting script to change log level to DEBUG
Log level changed ✅

Cool, it worked! Or did it…? 🔭

# Output of server.py
...
ERROR:main:An error occurred!
ERROR:main:An error occurred!
Can't open debugger script /tmp/tmpdh_kj9nu.py:
Traceback (most recent call last):
  File "/home/rafael/repos/website/content/posts/python_remote_exec/server.py", line 12, in main
    time.sleep(5)
PermissionError: [Errno 13] Permission denied: '/tmp/tmpdh_kj9nu.py'
ERROR:main:An error occurred!

Huh… I guess since we write the temporary file as root, the server process that runs under my user cannot read the file.

Attempt 3 - why do I even have a user separate from root?

Running everything as root should fix the issue — although this does not seem best practice.

$ sudo .venv/bin/python server.py
Server started. PID: 7195
ERROR:main:An error occurred!
ERROR:main:An error occurred!

$ sudo .venv/bin/python debug.py --log-level DEBUG -p 7195
Injecting script to change log level to DEBUG
Log level changed ✅

Looking at the logs of the server… it worked! 🌠

# Output of server.py
ERROR:main:An error occurred!
ERROR:main:An error occurred!
Changing log level to 'DEBUG' via remote exec...
DEBUG:main:This gives the missing context for the error.
ERROR:main:An error occurred!
DEBUG:main:This gives the missing context for the error.
ERROR:main:An error occurred!

Finding better ways

You might have read the following part of the permission requirements for Linux:

The tracer process must have the CAP_SYS_PTRACE capability (…)

This looks interesting! 🧐 In Linux, capabilities are distinct units of the “privileged features” that a process running under root has. While a privileged process (e.g via sudo) bypasses all kernel permission checks, capabilities can be independently enabled and disabled, and are a per-thread attribute.

We can also associate a capability with an executable file. This can be done via the setcap command on the Python executable (not the script!).

(By the way: This is why we used the --copies flag when creating the virtual environment in the beginning, because we don’t want to mess with the “global” uv Python binary.)

Let’s enable CAP_SYS_PTRACE for the Python executable:

sudo setcap 'CAP_SYS_PTRACE=ep' .venv/bin/python3
# You can remove it later via
# sudo setcap -r  .venv/bin/python3

The e stands for effective, p stands for permitted — please read the section “Thread capability sets” in the man page to learn more.

You can view the file capabilities by running getcap

$ getcap .venv/bin/*
.venv/bin/python3 cap_sys_ptrace=ep

Let’s start the server again using python server.py and the debugger using python3 debug.py. Note that we use python for the server, but python3 for the debugger because this has the required capability which only the tracer process needs. 🔐

Final Demo

Outlook

Besides remotely changing the log level of a running Python process, you can imagine other useful scenarios like

• 🩺 printing diagnostics in running web servers (e.g. active connections, open requests)
• 💾 examining in-memory caches or even clearing a cache variable
• 🗑️ analyzing objects tracked by the garbage collector

In this post, we managed to execute code on another running Python process with a simple local setup: both the server and debugger run on a local dev machine. But how does this work in production? How can we poke around in processes running in a Docker container on a remote server? Or in a secure cloud environment? What about Kubernetes clusters?

Great questions, but for another time! ⏭️

The new interface for external debuggers elegantly solves a problem that debugging tools have had for a long time: “What is the point in time where it is safe to attach to a Python process?” Moreover, the PEP authors were so kind to add an API for us Python users to use this feature and execute code after attaching.

Now the door is open for a new era of debugging! 🐛