Wednesday, November 9, 2011

f2pypy

I posted about f2pypy my mini-project to make a new backend to f2py which generates a ctypes-based extension module for Python. The goal is to see if that is a viable path for PyPy support of the existing shared libraries already wrapped by f2py. I think it is.

Feel free to comment!

4 comments:

Antonio Cuni said...

note that PyPy has also another, (faster) way to call C function: the _ffi module, which ctypes is built on top of.
On my machine:
viper tmp $ cat x.py
import ctypes
m = ctypes.cdll.LoadLibrary("libm.so")
cos = m.cos
cos.argtypes = [ctypes.c_double]
cos.restype = ctypes.c_double
viper tmp $ cat y.py
import _ffi
m = _ffi.CDLL('libm.so')
cos = m.getfunc('cos', [_ffi.types.double], _ffi.types.double)
viper tmp $ pypy-c -m timeit "from x import cos" "cos(0)"
10000000 loops, best of 3: 0.0393 usec per loop
viper tmp $ pypy-c -m timeit "from y import cos" "cos(0)"
100000000 loops, best of 3: 0.0148 usec per loop
viper tmp $ pypy-c -m timeit "from math import cos" "cos(0)"
100000000 loops, best of 3: 0.0076 usec per loop

still not as good as math.cos, but that's because _ffi calls always release the GIL, while math.cos doesn't.

Wim said...

Hi Andrew,

I think you're missing the point on SWIG v.s. Reflex. First of all, someone needs to come up with the manpower, and in our field, SWIG is non-existent (it isn't able to parse our C++ codes, for one). As such, I could not justify working on it, as I can with Reflex (and Cling, later). Second, SWIG does not allow the same performance improvements if it goes through the CPython ABI (cpyext). In our field, with lots of cross-calls, cpyext and SWIG itself would be the main bottleneck. With Reflex, we can peel away those layers for optimal performance.

Cheers,
Wim

Andrew Dalke said...

Hi Wim,

I had no point on SWIG vs. Reflex so it's very easy to have missed the point. I had never heard of it outside of the single PyPy reference, which is what I wrote.

In my field (computational chemistry software), several tools are based on SWIG bindings to Python. These include the OpenEye libraries (C++), Open Babel library (C++) and my own aged PyDaylight (C). Both OpenEye and Open Babel use SWIG to also generate Java and C# bindings.

As you say, there isn't the people time/interest to do the conversion to some other system. Which is the main issue here. CPython has had decades to build up ecosystems which support it, with multiple tools in the niche. PyPy is just now getting to that point.

That PyPy isn't there *yet* is mostly because the lack of strong, specific funding for these tasks means that it must be done piecemeal over 5-10 years.

Wim said...

Hi Andrew,

where I saw you making a comparison between the two, is the description of Reflex as an "existing ABI" and the call for the creation of "some SWIG/ABI interface."

However, Reflex is not and does not provide an ABI for PyPy. Instead, the cppyy module generates bindings at runtime (this code is written in RPython) based on stored reflection information.

This method is more functional (once its finalized :) ) and gives better performance than a SWIG/ABI approach could ever do, since it does not have an extension layer.

Cheers,
Wim