Pyaesar: A Multi-Noded Multi-Processor API

/ Personal, Post, Programming

As the rate of improvement for processors decreases, the ability to adapt code to leverage the growing number of cores holds an untapped market for performance.  Pyaesar is an API that was developed specifically for High-Performance Computing that enables highly distributed parallel processing. We implemented a two-step distributive process. First distributing across nodes using MPI4Py then distributing across processors using the multiprocessing API. We exhibit the effectiveness of this API on a wide range of computational problems. The current map functionality of Pyaesar is not present in similar packages such as Dask. Our results show an almost 17x speed-up going from one to thirty-two fully subscribed nodes to solve the Sieve of Eratosthenes for finding prime numbers on an HPC.

Presentation: here

Repository:  here

Leave a Comment