是否可以在 CPython 中以编程方式构造一个堆栈(一个或多个堆栈帧)并在任意代码点开始执行?想象以下场景:

Is it possible to programmatically construct a stack (one or more stack frames) in CPython and start execution at an arbitrary code point? Imagine the following scenario:

您有一个工作流引擎，其中的工作流可以使用 Python 编写脚本，其中包含一些调用工作流引擎的结构(例如分支、等待/加入).

You have a workflow engine where workflows can be scripted in Python with some constructs (e.g. branching, waiting/joining) that are calls to the workflow engine.

阻塞调用(例如等待或加入)在具有某种持久后备存储的事件调度引擎中设置侦听器条件.

A blocking call, such as a wait or join sets up a listener condition in an event-dispatching engine with a persistent backing store of some sort.

您有一个工作流脚本，它调用引擎中的等待条件，等待稍后将发出信号的某些条件.这会在事件调度引擎中设置监听器.

You have a workflow script, which calls the Wait condition in the engine, waiting for some condition that will be signalled later. This sets up the listener in the event dispatching engine.

工作流脚本的状态、包括程序计数器(或等效状态)在内的相关堆栈帧被保留 - 因为等待条件可能会在数天或数月后发生.

The workflow script's state, relevant stack frames including the program counter (or equivalent state) are persisted - as the wait condition could occur days or months later.

在此期间，工作流引擎可能会停止并重新启动，这意味着必须能够以编程方式存储和重建工作流脚本的上下文.

In the interim, the workflow engine might be stopped and re-started, meaning that it must be possible to programmatically store and reconstruct the context of the workflow script.

事件调度引擎触发等待条件启动的事件.

The event dispatching engine fires the event that the wait condition picks up.

工作流引擎读取序列化状态和堆栈，并使用堆栈重建线程.然后它在调用等待服务的位置继续执行.

The workflow engine reads the serialised state and stack and reconstructs a thread with the stack. It then continues execution at the point where the wait service was called.

问题

这可以用未经修改的 Python 解释器来完成吗?更好的是，谁能指出一些可能涵盖此类事情的文档或以编程方式构造堆栈帧并在代码块中间某处开始执行的代码示例?

Can this be done with an unmodified Python interpreter? Even better, can anyone point me to some documentation that might cover this sort of thing or an example of code that programmatically constructs a stack frame and starts execution somewhere in the middle of a block of code?

为了澄清未修改的 python 解释器"，我不介意使用 C API(PyThreadState 中是否有足够的信息来执行此操作?)但我不想去探索 Python 解释器的内部结构并不得不构建一个修改过的解释器.

To clarify 'unmodified python interpreter', I don't mind using the C API (is there enough information in a PyThreadState to do this?) but I don't want to go poking around the internals of the Python interpreter and having to build a modified one.

更新:通过一些初步调查，可以使用 PyThreadState_Get() 获取执行上下文.这将返回 PyThreadState 中的线程状态(在 pystate.h 中定义)，它具有对 frame 中堆栈帧的引用.堆栈帧保存在 PyFrameObject 的结构类型定义中，该结构在 frameobject.h 中定义.PyFrameObject 有一个字段 f_lasti (props to bobince)，它有一个程序计数器，表示为距代码块开头的偏移量.

Update: From some initial investigation, one can get the execution context with PyThreadState_Get(). This returns the thread state in a PyThreadState (defined in pystate.h), which has a reference to the stack frame in frame. A stack frame is held in a struct typedef'd to PyFrameObject, which is defined in frameobject.h. PyFrameObject has a field f_lasti (props to bobince) which has a program counter expressed as an offset from the beginning of the code block.

最后一点是个好消息，因为这意味着只要您保留实际编译的代码块，您就应该能够根据需要为尽可能多的堆栈帧重建局部变量并重新启动代码.我想说这意味着理论上可以不必修改 python 解释器，尽管这意味着代码仍然可能会与特定版本的解释器紧密耦合.

This last is sort of good news, because it means that as long as you preserve the actual compiled code block, you should be able to reconstruct locals for as many stack frames as necessary and re-start the code. I'd say this means that it is theoretically possible without having to make a modified python interpereter, although it means that the code is still probably going to be fiddly and tightly coupled to specific versions of the interpreter.

剩下的三个问题是:

事务状态和传奇"回滚，这可能可以通过一种用于构建 O/R 映射器的元类黑客来完成.我确实构建过一次原型，所以我对如何实现这一点有一个大致的了解.

Transaction state and 'saga' rollback, which can probably be accomplished by the sort of metaclass hacking one would use to build an O/R mapper. I did build a prototype once, so I have a fair idea of how this might be accomplished.

稳健地序列化事务状态和任意局部变量.这可以通过读取 __locals__(可从堆栈框架中获得)并以编程方式构造对 pickle 的调用来完成.但是，我不知道这里可能存在什么问题(如果有的话).

Robustly serialising transaction state and arbitrary locals. This might be accomplished by reading __locals__ (which is available from the stack frame) and programatically constructing a call to pickle. However, I don't know what, if any, gotchas there might be here.

工作流程的版本控制和升级.这有点棘手，因为系统没有为工作流节点提供任何符号锚.我们只有锚为了做到这一点，必须识别所有入口点的偏移量并将它们映射到新版本.手动操作可能可行，但我怀疑很难自动化.如果您想支持此功能，这可能是最大的障碍.

Versioning and upgrade of workflows. This is somewhat trickier, as the system is not providing any symbolic anchors for workflow nodes. All we have is the anchor In order to do this, one would have to identify the offsets of all of the entry points and map them to the new version. Probably feasible to do manually, but I suspect it would be hard to automate. This is probably the biggest obstacle if you want to support this capability.

更新 2: PyCodeObject (code.h) 有一个 addr 列表 (f_lasti)->PyCodeObject.co_lnotab 中的行号映射(如果此处错误，请纠正我).这可能用于促进将工作流更新到新版本的迁移过程，因为冻结的指令指针可以映射到新脚本中的适当位置，根据行号完成.仍然很混乱，但更有希望.

Update 2: PyCodeObject (code.h) has a list of addr (f_lasti)-> line number mappings in PyCodeObject.co_lnotab (correct me if wrong here). This might be used to facilitate a migration process to update workflows to a new version, as frozen instruction pointers could be mapped to the appropriate place in the new script, done in terms of the line numbers. Still quite messy but a little more promising.

更新 3: 我认为这个问题的答案可能是 Stackless Python.您可以暂停任务并将它们序列化.我还没有弄清楚这是否也适用于堆栈.

Update 3: I think the answer to this might be Stackless Python. You can suspend tasks and serialise them. I haven't worked out whether this will also work with the stack as well.

推荐答案

使用标准 CPython，由于堆栈中 C 和 Python 数据的混合，这很复杂.重建调用堆栈需要同时重建 C 堆栈.这确实把它放在了一个太难的篮子里，因为它可能将实现与特定版本的 CPython 紧密耦合.

With standard CPython this is complicated by the mixture of C and Python data in the stack. Rebuilding the call stack would require the C stack to be reconstructed at the same time. This really puts it in the too hard basket as it could potentially tightly couple the implementation to specific versions of CPython.

Stackless Python 允许对 tasklet 进行腌制，从而提供开箱即用所需的大部分功能.

Stackless Python allows tasklets to be pickled, which gives most of the capability required out of the box.