Add support for Java subclassing

[freakboy3742]

#23 added an explicit warning for any attempt to subclass a Java class.

This is a workaround to flag a known error case. We should add support for subclassing.

The underlying problem is that a Java-side object is needed to serve as the recipient for Java invocations. The approach used for interfaces (java.lang.reflect.InvocationHandler) won't work, because InvocationHandler explicitly takes Interfaces, not classes.

One possible solution: Use Python metaclass handling to define the bytecode for a class implementation that can proxy it's calls into native invocations, and use a ClassLoader to instantiate an instance of that class that can be wrapped Python-side.

Some edge cases that will need to be handled:

• Extending an abstract base class
• Extending a class and adding additional interfaces.
• Java code invoking a method on a Python class that overrides the definition on the base class
• Java code invoking a method on a Python class that is not overridden.
• Python code invoking a method from the base class
• Python code invoking the super() implementation on the base class.

Syntactically, something like the following should be possible:

class MyStackClass(
    JavaClass, 
    extends="java.lang.Stack", 
    implements=["org.example.FirstInterface", "org.example.OtherInterface"]
):
    def push(self, item: "java.lang.Object") -> None:
        ...
    def ElementAt(self, index: int) -> "java.lang.Object":
        ...

[xloem]

Pyjnius also had this issue and still has not solved it after years. It seems it is necessary to generate java code to facilitate this. This is a script they started: https://github.com/tshirtman/longface/blob/master/parse.py . It would be great if the android API would recognise this difficulty and offer interfaces to implement rather than abstract base classes to subclass.

[t-arn]

The Chaquopy Python API seems to be able to create subclasses of a Java class:
https://chaquo.com/chaquopy/doc/current/python.html

Could we use that code or the ideas behind that code in Beeware?

[freakboy3742]

If Chaquopy is doing it, then it's clearly possible - but I can't look at it, because Chaquopy is closed source. If I read their code, even for "inspiration", I open BeeWare up to potential claims of copyright infringement.

If you can find an academic or descriptive article of the general technique that isn't covered by Chaquopy's copyright, then it's something we can attempt to replicate independently.

[xloem]

[edit: i found mit-licensed chaquopy demo app, was mistaken]

[xloem]

Comment edited.
Reading the following link may risk copyright issues, see other comment. https://chaquo.com/chaquopy/doc/current/android.html#static-proxy-generator .

In summary, chaquopy does not add anything new over existing approaches that have been discussed or implemented in other comments, threads, and open source projects.

[freakboy3742]

Again - if you're intending to contribute to this ticket, DO NOT look in to "how ChaquoPy does it". If there is any connection between the implementation you develop and ChaquoPy's that contribution becomes a derived work of ChaquoPy, and as such cannot be distributed under and Open Source license without agreement from the ChaquoPy developers.

[xloem]

My understanding is that it is fine to work from an interface specification without access to the implementation, but I am not a lawyer. However, I've updated my comment to summarise that chaquopy basically doesn't add anything new.

[nickmain]

Would a solution to this need to handle the possibility that a Python method could be dynamically added to the subclass that should then override the equivalent method on the Java side for an already instantiated object ?

[freakboy3742]

In an ideal world, sure - but I wouldn't consider it a very high priority if it's even marginally complex to implement that way. The key requirement is to be able to implement handlers that are defined as abstract base classes, rather than interfaces; while it would be nice to be able to preserve all of Python's dynamism, I'd also be happy to call that sort of dynamism out of scope if if it helps us get a working solution for the key use case.

[xloem]

This isn't conceptually complex: the discovered solution, which kivy had begun in pyjnius, is to generate java implementations of every abstract class, and have them proxy access to a python object. If users have extra abstract classes, they need to generate or write more wrappers. Since a python object is wrapped, runtime changes should work fine.

We spent some time looking into other solutions and each one had some major blocking issue in the end. It's possible that something has changed now.

[freakboy3742]

Sure - code generation is the obvious approach; the trick is finding an elegant workflow for generating that code, and declaring any code generation requirements.

I believe there are some options that might exist that dont involve code generation, based in runtime class generation; these are a lot more complicated though, and I'm not sure how viable they are once Android's compilation tooling is involved.

[xloem]

I suspect that an issue should be opened in https://github.com/android/ndk regarding this usecase if there isn't one already. As enough interest develops in the issue they may find a cleaner solution. The stumbling block with runtime class generation is that android says they did not implement jni class definition due to their use of nonstandard bytecode: https://developer.android.com/training/articles/perf-jni#unsupported-featuresbackwards-compatibility

[nickmain]

For API level 26+ there is https://developer.android.com/reference/dalvik/system/InMemoryDexClassLoader that should work for loading generated DEX classes.

[xloem]

This means people can, with effort when needed, solve this problem locally until a solution is integrated into their framework, by precompiling classes and loading them at runtime. It looks like the dex compiler is a command-line utility written in java called 'd8'. Its source is at https://r8.googlesource.com/r8 .

[mhsmith]

See discussion #75.