Pipeline Editor: Allow for configuration of shared memory size

[ptitzler]

Requires #2957 RESOLVED

Extend the pipeline editor to allow for setting a custom shared memory size in generic and custom components. A pipeline default can be configured in the pipeline properties. Support is limited to Kubeflow Pipelines and Apache Airflow.

To define a custom size the user needs to specify a value other than zero. Zero implies that the Kubernetes default is used, which is 64Mi.

When a custom shared memory size is configured, Elyra adds a volume and volume mount to the pod, as shown here for Kubeflow Pipelines:

    volumeMounts:
     - {mountPath: /dev/shm, name: shm}
    volumes:
    - emptyDir: {medium: Memory, sizeLimit: 2G}
      name: shm

Closes #2838

TODO:

• pipeline editor release and merge

What changes were proposed in this pull request?

• See above
• Add new property description to pipeline documentation

How was this pull request tested?

• Added validation tests (for valid and invalid inputs)
• Updated handlers test

pipeline defaults:

• no custom size specified
• invalid custom size specified
• valid size specified

generic nodes (KFP and Airflow):

• no custom size specified and no pipeline default defined
• invalid custom size specified
• valid size specified

custom nodes (KFP and Airflow):

• no custom size specified and no pipeline default defined
• invalid custom size specified
• valid size specified

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

[elyra-bot]

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link:

[ptitzler]

In today's community meeting it was brought up that giving the user the option to choose from multiple units might be complicating things. While I agree that it might not be necessary to support Gi/Mi and G/M by choosing a basis that is applied to all memory units in Elyra, there would benefits to giving the user the option to choose a unit (e.g GB/MB ignoring for now what basis is used) if the units are supposed to be internally interpreted using power of two instead of power of ten.

It's not clear to me which basis is used today for the generic components RAM resources setting. The implementation for the KFP and Airflow processor appear to be in disagreement, with the KFP processor applying the G (power of ten-based) suffix, whereas the Airflow processor applying none (which may or may not be a bug). Anybody have insights whether G or Gi is the "correct" one?

[ptitzler]

Follow-up required: When I merged the main branch into this PR branch I overrode the pipeline editor dependency because I believed the necessary changes were already included already included the the most recent pipeline editor release. I'm not sure if this assumption is true. @marthacryan is elyra-ai/pipeline-editor#202 specific to this PR?

[kevin-bates]

LGTM