Adapt py2fgen for gt4py programs (#371)

Refactor py2fgen to support compilation and running of simple Python functions and gt4py programs from Fortran.

.gitignore

@@ -6,6 +6,7 @@ _reports
 tmp
 testdata
 simple_mesh*.nc
+/prof
 
 ### GT4Py ####
 .gt_cache/

ci/docker/Dockerfile.build

@@ -6,6 +6,7 @@ ARG DEBIAN_FRONTEND=noninteractive
 RUN apt-get update -qq && apt-get install -qq -y --no-install-recommends \
     strace \
     build-essential \
+    gfortran \
     tar \
     wget \
     curl \

tools/README.md

@@ -310,38 +310,78 @@ OUTPUT_FILEPATH   A path to the output Fortran source file to be generated.
 
 **Note:** The output of f2ser still has to be preprocessed using `pp_ser.py`, which then yields a compilable unit. The serialised files will have `f2ser` as their prefix in the default folder location of the experiment.
 
-## `py2f`
+# py2fgen
 
-Python utility for generating a C library and Fortran interface to call Python icon4py modules. The library [embeds python via CFFI ](https://cffi.readthedocs.io/en/latest/embedding.)
+`py2fgen` is a command-line interface (CLI) tool designed to generate C and Fortran 90 (F90) wrappers, as well as a C library, for embedding a Python module into C and Fortran applications. This tool facilitates the embedding of Python code into Fortran programs by utilizing the [`CFFI`](https://cffi.readthedocs.io/en/latest/embedding) library. `CFFI` instantiates a Python interpreter to execute Python code which is "frozen" into the dynamic library generated by `CFFI`.
 
-This is **highly experimental** and has not been tested from within Fortran code!
+**Note:** `py2fgen` is currently in an **experimental** stage. Simple examples have been successfully tested within Fortran code, but it is not yet production-ready. Further testing with complex Python code is necessary. The performance implications of invoking a Python interpreter from within Fortran are also yet to be fully understood.
 
-### usage
+## Usage
 
-`py2fgen`: Generates a C header file and a Fortran interface and compiles python functions into a C library embedding python. The functions need to be decorated with `CffiMethod.register` and have a signature with scalar arguments or `GT4Py` fields, for example:
+`py2fgen` simplifies the process of embedding Python functions into C and Fortran codebases. Here's how to use it:
 
+```bash
+py2fgen [OPTIONS] MODULE_IMPORT_PATH FUNCTION_NAME
+
+Arguments:
+  MODULE_IMPORT_PATH  The Python module containing the function to embed.
+  FUNCTION_NAME       The function within the module to embed.
+
+Options:
+  -b, --build-path PATH     Specify the directory for generated code and compiled libraries.
+  -d, --debug-mode          Enable debug mode to print additional runtime information.
+  -g, --gt4py-backend TYPE  Set the gt4py backend to use (options: CPU, GPU, ROUNDTRIP).
+  --help                    Display the help message and exit.
 ```
-@CffiMethod.register
-def foo(i:int, param:float, field1: Field[[VertexDim, KDim], float], field2: Field[CellDim, KDim], float])
-```
 
-see `src/icon4pytools/py2f/wrappers` for examples.
+### Example
+
+To create a Fortran interface along with the dynamic library for a Python function named `square` within the module `example.functions`, execute:
 
 ```bash
-py2fgen icon4pytools.py2f.wrappers.diffusion_wrapper  py2f_build
+py2fgen example.functions square
 ```
 
-where the first argument is the python module to parse and the second a build directory. The call above will generate the following in `py2f_build`:
+`py2fgen` can accept two types of functions:
+
+- **Simple Function:** Any Python function can be exposed.
+- **GT4Py Program:** Specifically, a Python function decorated with a `@program` decorator.
+
+**Important:** All arguments in the exposed functions must use GT4Py style type hints.
+
+## Generated Files
+
+Running `py2fgen` generates five key files:
+
+- **.c File**: Contains the generated CFFI code and the frozen Python code.
+- **.so File**: The compiled dynamic C library containing the CFFI code.
+- **.h File**: Declares the function signature of your exposed function.
+- **.f90 File**: Contains a Fortran interface to the C function in the dynamic library.
+- **.o File**: Represents the object code of the CFFI plugin.
+- (Optional) **.py File**: Contains the Python code frozen into the dynamic library (available with `--debug-mode`).
+
+## Running from Fortran
+
+To use the generated CFFI plugin in a Fortran program, call the subroutine defined in the .f90 interface file. Ensure that any arrays passed to the subroutine are in column-major order.
+
+Examples can be found under `tools/tests/py2fgen/fortran_samples`.
+
+## Compilation
+
+Compiling your Fortran driver code requires a Fortran compiler, such as `gfortran`. Follow these steps:
+
+1. Compile (without linking) .f90 interface:
 
 ```bash
- ls py2f_build/
-total 204K
-drwxrwxr-x 2 magdalena magdalena 4.0K Aug 24 17:01 .
-drwxrwxr-x 9 magdalena magdalena 4.0K Aug 24 17:01 ..
--rw-rw-r-- 1 magdalena magdalena  74K Aug 24 16:58 diffusion_wrapper.c
--rw-rw-r-- 1 magdalena magdalena 3.5K Aug 24 17:01 diffusion_wrapper.f90
--rw-rw-r-- 1 magdalena magdalena  955 Aug 24 17:01 diffusion_wrapper.h
--rw-rw-r-- 1 magdalena magdalena  58K Aug 24 17:01 diffusion_wrapper.o
--rwxrwxr-x 1 magdalena magdalena  49K Aug 24 17:01 libdiffusion_wrapper.so
+gfortran -c <function_name>_plugin.f90
+```
+
+2. Compile the Fortran driver code along with the Fortran interface and dynamic library:
 
+```bash
+gfortran -I. -Wl,-rpath=. -L. <function_name>_plugin.f90 <fortran_driver>.f90 -l<function_name>_plugin -o <executable_name>
 ```
+
+Replace `<function_name>`, `<fortran_driver>`, and `<executable_name>` with the appropriate names for your project.
+
+**Note:** When executing the compiled binary make sure that you have sourced a Python virtual environment where all required dependencies to run the embedded Python code are present.

tools/pyproject.toml

@@ -25,7 +25,8 @@ dependencies = [
   'gt4py>=1.0.1',
   'icon4py-common',
   'tabulate>=0.8.9',
-  'fprettify>=0.3.7'
+  'fprettify>=0.3.7',
+  'cffi>=1.5'
 ]
 description = 'Tools and utilities for integrating icon4py code into the ICON model.'
 dynamic = ['version']
@@ -35,14 +36,14 @@ readme = 'README.md'
 requires-python = '>=3.10'
 
 [project.optional-dependencies]
-all = ['icon4pytools[py2f]']
+all = ["icon4pytools[py2f]"]
 py2f = ['cffi>=1.5']
 
 [project.scripts]
 f2ser = 'icon4pytools.f2ser.cli:main'
 icon4pygen = 'icon4pytools.icon4pygen.cli:main'
 icon_liskov = 'icon4pytools.liskov.cli:main'
-py2fgen = 'icon4pytools.py2f.py2fgen:main'
+py2fgen = 'icon4pytools.py2fgen.cli:main'
 
 [project.urls]
 repository = 'https://github.com/C2SM/icon4py'

tools/src/icon4pytools/liskov/codegen/serialisation/deserialise.py

@@ -128,9 +128,11 @@ def _make_fields(named_args: dict, gpu_fields: set) -> list[FieldSerialisationDa
         fields = [
             FieldSerialisationData(
                 variable=variable,
-                association="z_hydro_corr(:,:,1)"  # special case
-                if association == "z_hydro_corr(:,nlev,1)"
-                else association,
+                association=(
+                    "z_hydro_corr(:,:,1)"  # special case
+                    if association == "z_hydro_corr(:,nlev,1)"
+                    else association
+                ),
                 decomposed=False,
                 dimension=None,
                 typespec=None,

tools/src/icon4pytools/liskov/codegen/serialisation/interface.py

@@ -43,7 +43,7 @@ class FieldSerialisationData:
 
 @dataclass
 class SavepointData(CodeGenInput):
-    subroutine: str  # todo: change to name
+    subroutine: str
     intent: str
     fields: list[FieldSerialisationData]
     metadata: Optional[list[Metadata]]