我有一个计算密集型程序这样做，我打算并行处理计算。它是用Python编写的，我希望用 多进程 模块。我想一些帮助，了解什么，我需要做的有一个程序，从我的笔记本电脑控制整个进程中运行。

I have a computationally intensive program doing calculations that I intend to parallelise. It is written in python and I hope to use the multiprocess module. I would like some help with understanding what I would need to do to have one program run from my laptop controlling the entire process.

我在，我可以用什么电脑术语两种选择。一个是，我可以通过 SSH user@comp1.com 从终端接入电脑（不知道如何通过python的访问它们），然后运行该实例有，虽然我ð更像一个编程的方式来得到他们比。看来，如果我跑远程管理类型的应用程序，将工作？

I have two options in terms of what computers I can use. One is computers which I can access through ssh user@comp1.com from the terminal ( not sure how to access them through python ) and then run the instance there, although I'd like a more programmatic way to get to them than that. It seems that if I ran a remote manager type application it would work?

第二个选择我当时的想法是利用AWS E2C服务器。 （我认为这是我所需要的）。而且我发现博托这是我从来没有使用，但似乎提供了控制AWS系统的接口。我觉得我会再需要一些实际的工作分配在AWS上，大概类似于选项1（？）。我在这里暗了一点。

The second option I was thinking is utilising AWS E2C servers. (I think that is what I need). And I found boto which I've never used but seemed to provide an interface to control the AWS system. I feel that I would then need something to actually distribute jobs on AWS, probably similarly as option 1 (?). I'm a bit in the dark here.

任何帮助AP preciated和原谅我的无知。

Any help appreciated and pardon my ignorance.

感谢您

编辑：

要给你这是多么parallelisable一个想法：

To give you an idea of how parallelisable it is:

res = []
for param in Parameters:
    res.append(FunctionA(param))
Parameters2 = FunctionB(res)
res2 = []
for param in Parameters2:
    res2.append(FunctionC(param))
return res, res2

于是两个循环基本上是在那里我可以送走许多参数值进行并行运行，我知道如何重组他们创造 RES 只要我知道哪个参数他们来自何处。然后，我需要它们组合在一起，以获得 Parameters2 ，然后第二部分是再次parallelisable。

So the two loops are basically where I can send off many param values to be run in parallel and I know how to recombine them to create res as long as I know which param they came from. Then I need to group them all together to get Parameters2 and then the second part is again parallelisable.

推荐答案

你想使用多进程模块只，如果你想进程在内存中共享数据。这是我会建议只如果你确实得有共享存储由于性能方面的考虑。蟒蛇多进程应用程序是不平凡的编写和调试。

you would want to use the multiprocess module only if you want the processes to share data in memory. That is something I would recommend ONLY if you absolutely have to have shared memory due to performance considerations. python multiprocess applications are non-trivial to write and debug.

如果你正在做的事情，如distributed.net和SETI @ home的项目，在那里即使任务是计算intenive他们是相当孤立的，你可以按照下面的过程。

If you are doing something like the distributed.net or seti@home projects, where even though the tasks are computationally intenive they are reasonably isolated, you can follow the following process.

您可以在AWS单核心实例上运行这些程序，如果你想，或使用你的笔记本电脑运行多个副本你有核心，以备用。

you could run these processes on AWS single core instances if you wanted, or use your laptop to run as many copies as you have cores to spare.

编辑：基于更新的问题的

所以，你的主进程将创建文件（或其他数据结构），将有其中的参数信息。正如许多文件，如你有PARAMS处理。该文件将被存储在共享文件夹，名为需要工

So your master process will create files (or some other data structures) that will have the parameter info in them. As many files as you have params to process. This files will be stored in a shared folder called needed-work

每个蟒蛇工人（在AWS实例）将着眼于所需要的工共享文件夹，寻找可用的文件上工作（或等待套接字上的主进程文件分配给它们）。

Each python worker (on AWS instances) will look at the needed-work shared folder, looking for available files to work on (or wait on a socket for the master process to assign the file to them).

蟒蛇的过程，需要在需要的工作文件，将工作就可以了，结果在一个单独的共享文件夹中的参数作为文件结构的一部分存储。

The python process that takes on a file that needs work, will work on it and store the result in a separate shared folder with the the parameter as part of the file structure.

主进程将着眼于在工作完成的文件夹中的文件，处理这些文件，并生成合成反应

The master process will look at the files in the work-done folder, process these files and generate the combined response

这个完整的解决方案，可以为好，那里的工人将听取插座师傅工作分配给他们，师傅会等待一个插座工人因此提出因应实现的插座。

This whole solution could be implemented as sockets as well, where workers will listen to sockets for the master to assign work to them, and master will wait on a socket for the workers so submit response.

在基于文件的方法需要一种方式为工人，以确保他们拿起工作没有采取由另一名工人。这可能是固定的具有独立的工作文件夹，每个工人和主进程将决定当有需要为工人更多的工作。

The file based approach would require a way for the workers to make sure that the work they pick up is not taken on by another worker. This could be fixed by having separate work folders for each worker and the master process would decided when there needs to be more work for the worker.

工人可以删除他们拿起从工作文件夹，主进程可以继续当一个文件夹为空的手表，并添加更多的工作文件，其文件。

Workers could delete files that they pick up from the work folder and master process could keep a watch on when a folder is empty and add more work files to it.

此外更优雅，如果你是舒服，要做到这一点使用套接字。

Again more elegant to do this using sockets if you are comfortable with that.