feat(plugins): spring custom plugins

[david-leifker]

Allow custom plugins to use Spring for injection and autoconfiguration of custom plugins.

This is in addition to the ClassGraph (non-Spring) plugin loader. Spring is optional.

Created a single 20MB dependency for plugins to use. Created a separate metadata-io-api module to isolate most of the non-essential classes.

The artifact is io.acryl:datahub-custom-plugin-lib found in the module :metadata-integration:java:custom-plugin-lib

Notes: The Spring context is isolated so all required classes must be present in the packageScan configuration for Spring to find.

Additional Changes:

• Class graph - optimize class scanning and added cache
• OpenAPIv3 Spec Generator fixes

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[kevin1chun]

@david-leifker in this change then, the plugin classloader will no longer have access to the core application classloader?
So the plugin jar would need to contain a specific version of spring?

I know we discussed this last time, and it seems like this will help by adding isolation between core application and plugin dependencies. It doesn't help us solve for initializing core components like logging and metrics in the core app though. That could be solved separately, but just wanted to confirm my assumption there

[kevin1chun]

Another question, if we are able to reference the main application spring context from the plugin spring context, can we pull in the metrics reporter exclusively?

[david-leifker]

@david-leifker in this change then, the plugin classloader will no longer have access to the core application classloader? So the plugin jar would need to contain a specific version of spring?

I know we discussed this last time, and it seems like this will help by adding isolation between core application and plugin dependencies. It doesn't help us solve for initializing core components like logging and metrics in the core app though. That could be solved separately, but just wanted to confirm my assumption there

I believe that the classes already imported would be available to plugins. The Spring context is isolated and the packageScan can include pages to load into this isolated context which is used for the plugin. The packages are not restricted to any package location so beans or components from anything can be autowired.

[kevin1chun]

@david-leifker in this change then, the plugin classloader will no longer have access to the core application classloader? So the plugin jar would need to contain a specific version of spring?
I know we discussed this last time, and it seems like this will help by adding isolation between core application and plugin dependencies. It doesn't help us solve for initializing core components like logging and metrics in the core app though. That could be solved separately, but just wanted to confirm my assumption there

I believe that the classes already imported would be available to plugins. The Spring context is isolated and the packageScan can include pages to load into this isolated context which is used for the plugin. The packages are not restricted to any package location so beans or components from anything can be autowired.

Awesome, that's helpful context. So in our case in which we want to reference classes that are contained in the core application classloader, we would just initialize our plugin with the directive to load packages in datahub gms that contain beans that we want.
There's probably some more complexity here though since we would need to understand all of the dependent paths to load as well.

[kevin1chun]

Where is the limit being applied?

[david-leifker]

The packages and class names are used in the acceptPackages and acceptClass

new ClassGraph()
.acceptPackages(packageNames.stream().distinct().toArray(String[]::new) .acceptClasses(classNames.stream().distinct().toArray(String[]::new))

[kevin1chun]

Interesting, so this is creating a key like
sum ( classLoaders.hashCode..., aspectPluginConfigClass.hashCode... )

Is this guaranteed to be unique?

[david-leifker]

This is unique within a given jvm execution. It might leak a bit when/if jars are being constantly replaced since each time that happens we'll be creating orphans. This was primarily added to help with tests where we run multiple concurrent test threads and memory usage was triggering OOM.