Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relax TriggerTemplate limitation of only creating Tekton resources #482

Closed
disposab1e opened this issue Mar 15, 2020 · 17 comments
Closed

Relax TriggerTemplate limitation of only creating Tekton resources #482

disposab1e opened this issue Mar 15, 2020 · 17 comments
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@disposab1e
Copy link

disposab1e commented Mar 15, 2020

Expected Behavior

It should be possible to create a PersistentVolumeClaim resource in TriggerTemplate spec.resourcetemplates:.

Actual Behavior

Only Tekton resources are allowed.

Steps to Reproduce the Problem

We would like to have a possibilty to dynamically create a PVC within TriggerTemplate.

apiVersion: tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
 name: pipeline-tt
 namespace: awesome-pipelines
spec:
 params:
 - name: pipelineName
   description: Pipeline Name (build, deploy, verify)    
resourcetemplates:
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: $(uid)
  spec:
    resources:
      requests:
        storage: 0.2Gi
    accessModes:
      - ReadWriteMany
 - apiVersion: tekton.dev/v1alpha1
   kind: PipelineRun
   metadata:
     generateName: pipeline-run-
   spec:
     serviceAccountName: pipeline-sa
     pipelineRef:
       name: pipeline-$(params.pipelineName)
     workspaces:
     - name: pipeline-workspace
       persistentVolumeClaim:
         claimName: `$(uid)`

We don't think this will break any design decisions but can be very helpful in some dynamic scenarios.

In this context we really miss "support" for workspace lifecycle management. For example cleaning up a workspace (and optional PVC) after:

We also follow the Auto Workspaces discussion.

Additional Info

This issue has its roots here.

@ncskier
Copy link
Member

ncskier commented Mar 16, 2020

/kind feature

@tekton-robot tekton-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Mar 16, 2020
@bitsofinfo
Copy link
Contributor

@disposab1e do you currently have any workaround/hack way you create these PVCs dynamically before a PipelineRun invokes? care to share?

@vtereso
Copy link

vtereso commented Mar 24, 2020

Due to security considerations, we want to make sure we do this right and that the use case is warranted. Auto workspaces might be able to support your use case. We will continue to look at this.

@disposab1e
Copy link
Author

@vtereso What are your security concerns? Where is a auto workspace feature?

@vdemeester
Copy link
Member

@disposab1e auto workspace is there : tektoncd/pipeline#1986
@vtereso can you elaborate on the security concern ? what differs from creating tekton resources and pvcs ? (if the user has the rights to do so)

@disposab1e
Copy link
Author

@vdemeester Perhaps it will be there? Still open, nothing to use right now.

@vdemeester
Copy link
Member

@vdemeester Perhaps it will be there? Still open, nothing to use right now.

On auto-workspace, yeah, there is no auto-workspace yet, which makes using workspace with dynamically provisionned PVC close to impossible with triggers indeed 😓

@disposab1e
Copy link
Author

disposab1e commented Mar 24, 2020

I do not agree! I can use workspace feature with dynamically provisioned PVCs. Ok it's harder without error handling and without lifecycle management. Still two features I'm really mssing in a professionell environment.

@vdemeester
Copy link
Member

I do not agree! I can use workspace feature with dynamically provisioned PVCs. Ok it's harder without error handling and without lifecycle management. Still two features I'm really mssing in a professionell environment.

My comment above says that it's hard as of today (not being able to create PersistentVolumeClaim in a TriggerTemplate), which I think is a problem (hence my question to @vtereso). I didn't mean that it shouldn't be supported, but the opposite, I think it should be supported (otherwise this mean we need to do an extra hoop to create dynamic PVC, like have a Task that creates it and then triggers something else…)

@disposab1e
Copy link
Author

I think there must be "something" when you remove PipelineResources in the future. I totally understand the discussions about not to pin tekton-pipelines to a specific way to store data between tasks. But it's a common use case "the need for storage". I had this discussion in mind when I was asking for PVCs in TriggerTemplates. I don't think this will break your design decissions. But when you create something, you also have to remove something. That's why error handling and/or lifecycle management will be very important in the furture. And to be honest for me it's not understandable that these use cases are NOT part of any design so far.

@dibyom
Copy link
Member

dibyom commented Mar 30, 2020

@vtereso can you elaborate on the security concern ? what differs from creating tekton resources and pvcs ? (if the user has the rights to do so)

I think the security issues are the same as mentioned in #77

jlpettersson added a commit to jlpettersson/pipeline that referenced this issue Apr 4, 2020
An existing PersistentVolumeClaim can currently be used as a Workspace
volume source. There is two ways of using an existing PVC as volume:

 - Reuse an existing PVC
 - Create a new PVC before each PipelineRun.

There is disadvantages by reusing the same PVC for every PipelineRun:

 - You need to clean the PVC at the end of the Pipeline
 - All Tasks using the workspace will be scheduled to the node where
   the PV is bound
 - Concurrent PipelineRuns may interfere, an artifact or file from one
   PipelineRun may slip in to or be used in another PipelineRun, with
   very few audit tracks.

There is also disadvantages by creating a new PVC before each PipelineRun:

 - This can not (easily) be done declaratively
 - This is hard to do programmatically, because it is hard to know when
   to delete the PVC. The PipelineRun can not be set as OwnerReference since
   the PVC must be created first

 This commit adds 'volumeClaimTemplate' as a volume source for workspaces. This
 has several advantages:

 - The syntax is used in k8s StatefulSet and other k8s projects so it is
   familiar in the kubernetes ecosystem
 - It is possible to declaratively declare that a PVC should be created for each
   PipelineRun, e.g. from a TriggerTemplate.
 - The user can choose storageClass (or omit to get the cluster default) to e.g.
   get a faster SSD volume, or to get a volume compatible with e.g. Windows.
 - The user can adapt the size to the job, e.g. use 5Gi for apps that contains
   machine learning models, or 1Gi for microservice apps. It can be changed on
   demand in a configuration that lives in the users namespace e.g. in a
   TriggerTemplate.
 - The size affects the storage quota that is set on the namespace and it may affect
   billing and cost depending on the cluster environment.
 - The PipelineRun or TaskRun with the template is created first, and is used
   as OwnerReference on the PVC. That means that the PVC will have the same lifecycle
   as the PipelineRun.

 Related to tektoncd#1986

 See also:
  - tektoncd#2174
  - tektoncd#2218
  - tektoncd/triggers#476
  - tektoncd/triggers#482
  - kubeflow/kfp-tekton#51
@jlpettersson
Copy link
Member

jlpettersson commented Apr 4, 2020

@disposab1e @vdemeester @vtereso I agree with you all! This is a blocker before using Tekton in a professional environment. I spent some time and submitted tektoncd/pipeline#2326 for the solution I proposed in the auto workspace issue.

@disposab1e
Copy link
Author

@jlpettersson Looks wonderful! I like the idea that the PVC is deleted when TaskRun, PipelineRun is deleted!

jlpettersson added a commit to jlpettersson/pipeline that referenced this issue Apr 9, 2020
An existing PersistentVolumeClaim can currently be used as a Workspace
volume source. There is two ways of using an existing PVC as volume:

 - Reuse an existing PVC
 - Create a new PVC before each PipelineRun.

There is disadvantages by reusing the same PVC for every PipelineRun:

 - You need to clean the PVC at the end of the Pipeline
 - All Tasks using the workspace will be scheduled to the node where
   the PV is bound
 - Concurrent PipelineRuns may interfere, an artifact or file from one
   PipelineRun may slip in to or be used in another PipelineRun, with
   very few audit tracks.

There is also disadvantages by creating a new PVC before each PipelineRun:

 - This can not (easily) be done declaratively
 - This is hard to do programmatically, because it is hard to know when
   to delete the PVC. The PipelineRun can not be set as OwnerReference since
   the PVC must be created first

 This commit adds 'volumeClaimTemplate' as a volume source for workspaces. This
 has several advantages:

 - The syntax is used in k8s StatefulSet and other k8s projects so it is
   familiar in the kubernetes ecosystem
 - It is possible to declaratively declare that a PVC should be created for each
   PipelineRun, e.g. from a TriggerTemplate.
 - The user can choose storageClass (or omit to get the cluster default) to e.g.
   get a faster SSD volume, or to get a volume compatible with e.g. Windows.
 - The user can adapt the size to the job, e.g. use 5Gi for apps that contains
   machine learning models, or 1Gi for microservice apps. It can be changed on
   demand in a configuration that lives in the users namespace e.g. in a
   TriggerTemplate.
 - The size affects the storage quota that is set on the namespace and it may affect
   billing and cost depending on the cluster environment.
 - The PipelineRun or TaskRun with the template is created first, and is used
   as OwnerReference on the PVC. That means that the PVC will have the same lifecycle
   as the PipelineRun.

 Related to tektoncd#1986

 See also:
  - tektoncd#2174
  - tektoncd#2218
  - tektoncd/triggers#476
  - tektoncd/triggers#482
  - kubeflow/kfp-tekton#51
tekton-robot pushed a commit to tektoncd/pipeline that referenced this issue Apr 14, 2020
An existing PersistentVolumeClaim can currently be used as a Workspace
volume source. There is two ways of using an existing PVC as volume:

 - Reuse an existing PVC
 - Create a new PVC before each PipelineRun.

There is disadvantages by reusing the same PVC for every PipelineRun:

 - You need to clean the PVC at the end of the Pipeline
 - All Tasks using the workspace will be scheduled to the node where
   the PV is bound
 - Concurrent PipelineRuns may interfere, an artifact or file from one
   PipelineRun may slip in to or be used in another PipelineRun, with
   very few audit tracks.

There is also disadvantages by creating a new PVC before each PipelineRun:

 - This can not (easily) be done declaratively
 - This is hard to do programmatically, because it is hard to know when
   to delete the PVC. The PipelineRun can not be set as OwnerReference since
   the PVC must be created first

 This commit adds 'volumeClaimTemplate' as a volume source for workspaces. This
 has several advantages:

 - The syntax is used in k8s StatefulSet and other k8s projects so it is
   familiar in the kubernetes ecosystem
 - It is possible to declaratively declare that a PVC should be created for each
   PipelineRun, e.g. from a TriggerTemplate.
 - The user can choose storageClass (or omit to get the cluster default) to e.g.
   get a faster SSD volume, or to get a volume compatible with e.g. Windows.
 - The user can adapt the size to the job, e.g. use 5Gi for apps that contains
   machine learning models, or 1Gi for microservice apps. It can be changed on
   demand in a configuration that lives in the users namespace e.g. in a
   TriggerTemplate.
 - The size affects the storage quota that is set on the namespace and it may affect
   billing and cost depending on the cluster environment.
 - The PipelineRun or TaskRun with the template is created first, and is used
   as OwnerReference on the PVC. That means that the PVC will have the same lifecycle
   as the PipelineRun.

 Related to #1986

 See also:
  - #2174
  - #2218
  - tektoncd/triggers#476
  - tektoncd/triggers#482
  - kubeflow/kfp-tekton#51
@dibyom
Copy link
Member

dibyom commented Jun 15, 2020

The use case mentioned in this issue is now possible using volumeClaimTemplate. I'm going to close this out now but we can re-open if any new use cases pop up that needs Triggers to create non-Tekton resources.

@tekton-robot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

@tekton-robot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

@tekton-robot tekton-robot added the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Aug 14, 2020
@tekton-robot
Copy link

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Development

No branches or pull requests

8 participants