Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build packages for Android #647

Closed
wants to merge 7 commits into from
Closed

Conversation

mansourmoufid
Copy link

@mansourmoufid mansourmoufid commented Dec 7, 2021

Looking for feedback as this is very experimental. :)

Edit: This is in reference to #471.

PR Checklist:

  • All new features have been tested
  • All new features have been documented
  • I have read the CONTRIBUTING.md file
  • I will abide by the code of conduct

First pass the locations of Python.h and libpython.so in the environment
variables CPPFLAGS and LDFLAGS.  Then run pip via the androidenv module.
@freakboy3742
Copy link
Member

Wow - this is awesome... and a lot simpler than I thought it would be - thanks!

I'll need to take a closer look (especially into what exactly androidenv is doing), but this looks like a really good start!

# sysconfig.get_config_var("INCLUDEPY")
includepy = os.path.join(
self.bundle_path(app), "app", "include",
"python{}m".format(self.python_version_tag),
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This path doesn't exist, is the trailing 'm' a mistake?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "m" is a suffix added by Python when it is configured with --with-pymalloc. It seems this is only valid for 3.6 and 3.7: https://github.com/beeware/Python-Android-support/blob/master/main.sh#L84 I'll fix this.

env.update({"CPPFLAGS": " ".join(cppflags)})
# sysconfig.get_config_var("LIBDIR")
libdir = os.path.join(
self.bundle_path(app), "app", "libs", "arm64-v8a"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it hardcoded for arm64-v8a? It should be compatible with all available targets, as the emulator can run with a x86_64 arch for example.

Copy link
Author

@mansourmoufid mansourmoufid Dec 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good point. This will require changes to the beeware/Python-Android-support project to include four architecture-specific "app_packages" directories, as well as changes to the beeware/briefcase-android-gradle-template project to add the right directory to sys.path. I'll try to implement this.

Edit: We may be able to avoid architecture-specific directories by having architecture-specific filename extensions. This is set by the variable EXT_SUFFIX in Python's configure script, and available at run-time with sysconfig.get_config_var("EXT_SUFFIX"). Something like ".arm64-v8a.so" maybe. Hmmm it can't be that easy...

@mansourmoufid
Copy link
Author

I have been using the SDK from Android Studio, so I didn't notice that ANDROID_SDK_ROOT was not in pip's environment for everyone using the SDK downloaded by briefcase. This latest commit should fix that, hopefully.

@mansourmoufid
Copy link
Author

mansourmoufid commented Dec 31, 2021

OK, with the latest commit (f94f62b) and beeware/briefcase-android-gradle-template/pull/44, briefcase builds for every ABI. I only tested this with NumPy, and only on arm64-v8a.

Edit:

(bee-venv) % find android -name '*.so'
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/core/_struct_ufunc_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/core/_simd.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/core/_umath_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/core/_multiarray_umath.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/core/_operand_flag_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/core/_rational_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/core/_multiarray_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/linalg/lapack_lite.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/linalg/_umath_linalg.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/fft/_pocketfft_internal.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/random/bit_generator.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/random/_bounded_integers.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/random/mtrand.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/random/_generator.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/random/_mt19937.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/random/_sfc64.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/random/_philox.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/random/_pcg64.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/armeabi-v7a/numpy/random/_common.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/core/_struct_ufunc_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/core/_simd.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/core/_umath_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/core/_multiarray_umath.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/core/_operand_flag_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/core/_rational_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/core/_multiarray_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/linalg/lapack_lite.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/linalg/_umath_linalg.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/fft/_pocketfft_internal.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/random/bit_generator.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/random/_bounded_integers.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/random/mtrand.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/random/_generator.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/random/_mt19937.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/random/_sfc64.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/random/_philox.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/random/_pcg64.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86/numpy/random/_common.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/core/_struct_ufunc_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/core/_simd.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/core/_umath_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/core/_multiarray_umath.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/core/_operand_flag_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/core/_rational_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/core/_multiarray_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/linalg/lapack_lite.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/linalg/_umath_linalg.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/fft/_pocketfft_internal.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/random/bit_generator.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/random/_bounded_integers.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/random/mtrand.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/random/_generator.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/random/_mt19937.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/random/_sfc64.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/random/_philox.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/random/_pcg64.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/arm64-v8a/numpy/random/_common.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/core/_struct_ufunc_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/core/_simd.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/core/_umath_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/core/_multiarray_umath.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/core/_operand_flag_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/core/_rational_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/core/_multiarray_tests.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/linalg/lapack_lite.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/linalg/_umath_linalg.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/fft/_pocketfft_internal.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/random/bit_generator.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/random/_bounded_integers.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/random/mtrand.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/random/_generator.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/random/_mt19937.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/random/_sfc64.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/random/_philox.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/random/_pcg64.so
android/gradle/Hello World/app/src/main/assets/python/app_packages/x86_64/numpy/random/_common.so

@danyeaw danyeaw changed the title Build packages for Android. Build packages for Android May 30, 2022
@danyeaw danyeaw added the android The issue relates to Android mobile support. label May 30, 2022
@danyeaw
Copy link
Member

danyeaw commented May 30, 2022

@eliteraspberries I fixed the merge conflict and updated the PR. I would really like to see this merged because you went most of the way to get something working. I think this probably needs:

Is there anything else? How can I help?

@freakboy3742
Copy link
Member

@danyeaw There's one big additional requirement: it requires a change to Python's distutils.

That's the biggest impediment at present, because there isn't a "simple fix" we can apply - we need a change made upstream in CPython. Alternatively, we need to work out a way for an end user to patch distutils in a way that enables them to leverage those changes. That might mean we need to maintain a forked version of distutils in the short term while we attempt to get this change upstreamed - although I'm open to any other suggestions.

@danyeaw
Copy link
Member

danyeaw commented May 31, 2022

@freakboy3742 Good point.

Since distutils as part of the standard library is now deprecated and is being updated / migrated as part of setuptools, maybe this hurdle might not be so high.

@eliteraspberries Can androidenv make use of the vendored version of distutils in setuptools if setuptools is installed? I think this is now the default behavior of setuptools as long as it isn't overriden with SETUPTOOLS_USE_DISTUTILS=stdlib.

This change might make the androidenv patch of sysconfig.py not needed any longer, it looks like you can override ranlib if it is in os.environ.

@freakboy3742
Copy link
Member

I'm going to close this PR; thanks for the work you've put into this, but it looks like we have a path forward using the Chaquopy toolchain as a base for Android projects.

@joooeey
Copy link

joooeey commented Aug 9, 2022

I'm going to close this PR; thanks for the work you've put into this, but it looks like we have a path forward using the Chaquopy toolchain as a base for Android projects.

Is there a timeline, pull request, or issue for this change? We are looking at using BeeWare for a research project because of its small complexity and since it only requires one language. However, Numpy support is a must for us and I think we don't have the skills and resources to add support for binary packages ourselves. I'd love to know how this effort comes along.

@mhsmith
Copy link
Member

mhsmith commented Aug 9, 2022

The central PR for this change is beeware/briefcase-android-gradle-template#52, and we're aiming to release it by the end of August.

Binary packages will initially only be available for Python 3.8, and will include all the packages currently supported by Chaquopy. Later we'll add packages for Python 3.9, 3.10 and possibly 3.7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
android The issue relates to Android mobile support.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants