Add new control command for creating control repository

[scotje]

From @bittner on October 24, 2017 14:16

PDK should support creating and extending a control-repository, which is enabled for validation and testing using PDK (just like the modules created by PDK).

Obsoletes #289 and #332 when implemented.
See also: puppetlabs/control-repo#52

Proposed Solution

Ideally,

• the validation and test setup currently present in generated modules should be taken over to the proposed Puppet Labs control-repo, and
• the pdk new control command should generate a control-repo setup that is equivalent to the proposed Puppet Labs control-repo on GitHub.

This way, whether you use PDK or just the GitHub repo as an example, you can trust to get a usable best-practice setup.

Copied from original issue: #333

[scotje]

From @cdenneen on October 25, 2017 16:28

@bittner except for the metadata.json check the validation will work on existing control repo so far that I can tell.
The ability to generate a control repo would definitely need to be added and skip the metadata.json check in those cases.

[scotje]

From @bittner on October 25, 2017 22:36

The Gemfile is needed to make pdk validate work on a control repo, see #289 (comment).

The Rakefile, .fixtures.yml (with links: to the roles and profiles folders, and more dependencies specified in the forge_modules: section), spec/spec_helper.rb and specs in spec/classes/ is needed for pdk test unit.

Not sure how the examples/ would come into play. Then there are also acceptance tests (as seen in the PSICK repo). If that all makes sense it must be generated, and work out of the box. 👍

[scotje]

From @ardrigh on May 3, 2018 4:31

I was talking with my team about adopting PDK, and the files PDK adds when it converts modules, so that it can run the tests and validation.

Currently our control repo has 40+ internal Puppet modules, and converting them all with PDK would potentially add a huge number of duplicate files (Gemfile etc).

It would be good to know if support for testing across control repos is still on the development plan.

I am hopeful we will eventually be able migrate our TravisCI setup to test our control repo solely using simple PDK commands

[scotje]

From @ardrigh on May 3, 2018 5:58

Additional question, has any thought been given to integrating the onceover gem into PDK for control repo testing?

[scotje]

From @Nekototori on May 8, 2018 18:2

I'd second onceover. PDK is to modules as onceover is to control repos (as it stands today).

[ardrigh]

Just as a comment on development of an RFC on this type of feature...

It would be good if a pdk new role class integrated with the onceover testing to create a test for the role, in the manner that classes and defines do.

This requires work on support for testing of control repos, so it's a complicated request.

But looking it from a greenfields user perspective, the starting point for a new site should hopefully be defining a role and some profiles, and have instant feedback that something is working because a test passed successfully.

[reidmv]

Language comment: the term "control" by itself does not have meaning. It's unfortunate, but the compound noun for the set of content and structure we're talking about creating is control-repo.

If we're gonna rename it ...let's just say we can do WAY better than "control". If we're gonna call it what everybody calls it today though, we should use the full compound noun.

What we should not do is truncate the existing compound noun. Doing that doesn't give us any benefit and it will likely cause confusion.

If we stick with this name the CLI should be:

pdk new control-repo

[logicminds]

why not pdk new module control_repo --no-interview When this name is encountered do something extra special for the user.

[cdenneen]

I just stick a dummy metadata.json and use pdk just like you... but this --no-interview flag with a dummy metadata.json would be useful:

{
  "name": "puppet-controlrepo",
  "version": "0.0.1",
  "author": "puppet",
  "summary": "Control Repository",
  "license": "Apache-2.0",
  "source": "",
  "project_page": "",
  "issues_url": "",
  "dependencies": [
    {
      "name": "puppetlabs-stdlib",
      "version_requirement": ">= 4.21.0 < 9.0.0"
    }
  ],
  "operatingsystem_support": [
    {
      "operatingsystem": "CentOS",
      "operatingsystemrelease": [
        "7"
      ]
    }
  ],
  "requirements": [
    {
      "name": "puppet",
      "version_requirement": ">= 4.10.0 < 8.0.0"
    }
  ],
  "description": "Control Repository defaults across platforms.",
  "pdk-version": "2.0.0",
  "template-url": "https://github.com/puppetlabs/pdk-templates#main",
  "template-ref": "heads/main-0-g12a5dea"
}

then I stick in the .sync.yml and the convert/update seems to work as needed.
For those 2 minor tweaks to make pdk work with a control from testing perspective I find it a bit more consistent with results I expect that onceover. I already am writing things for pdk with modules so might as well follow same pattern.

[cdenneen]

Is anyone still working on this?

[scotje]

Is anyone still working on this?

Yes! The very first control-repo related bits have landed recently (#826) and we have been discussing this some at our bi-weekly public planning meetings.

There will be a more formal RFC for things soon, in the meantime you can see and comment on our working doc here: https://docs.google.com/document/d/1oNRjTvkkNSzuESlTazReOdYNgcdXo1Y-vV-LrA72-Cs/edit?usp=sharing

[bittner]

Any more progress to report on this topic?

I had an internal education and hands-on practicing with Puppet's PDK at our office yesterday, and I would have loved to show some related capabilities of PDK. You probably can tell the rest of the story yourselves... 🙄

The feature descriptions in the linked Google doc are intriguing. I'd really love to see them hit the stage soon. 🙏

[cdenneen]

Think sub-task of this will be #1018

[logicminds]

I have just added puppetlabs/pdk-templates#425

My workflow is to always clone the puppetlabs/control-repo then run pdk convert inside of it and treat it like a module.

But there are some gotchas with this setup. Anything under site-modules doesn't get checked and adding new roles and profiles is still manual since pdk new class doesn't work with this setup.

[logicminds]

@cdenneen I purposely did not add onceover here because it pulls in so many dependencies. But that is something that could/should be opt-in.

[cdenneen]

@logicminds I still like the idea of pdk new control vs pdk new module control_repo because I think it gets confusing. Also I think in doing some preparation the modules under site, dist, site-modules are basically same functionality (local modules, not controlled by Puppetfile) therefore need to be handled at the global spec level rather than individually because when you are trying to validate a control repository or run through unit/acceptance testing your not going to want to have to loop through everything in the "site" directory for profile/roles. You want those tests to be written at the control repo level. This is why I use/prefer spec to be at the root of the control repo so it can be used to test the classes (profile/role).

[logicminds]

Litmus testing with site-modules in the control repo is very problematic. So something would need to be done to fix that.

[cdenneen]

could you elaborate more on this? I've forked the pdk image and done pdk with docker in order to spin up litmusimages for testing so far with regular modules for docker-in-docker approach to testing have yet to do it with control repo but would image it to be similar just requires some sort of LitmusHelper to copy the control repo over. I've even considered using the prep from onceover to stage everything and then copy that over from .onceover but haven't committed to anything just yet.

[logicminds]

The litmus::install_module task uses the metadata.json file to install that module and dependency modules. Because site-modules cannot be listed in metadata there is no way to get that directory in the modules directory on the SUT (system under test). LItmus is capable of transferring the control-repo module over as a whole, but puppet doesn't know to look for site-modules due to the modulepath. The hack is to move the site-modules/profile and site-module/role over to the modules directory on the SUT. It might be possible to append the modulespath in Litmus to include site-modules but I never looked.

If litmus knew this was a control module it might be able to just copy the entire directory over to the environmentpath instead of a basemodulepath. Then install all of the dependencies.

On a similar note I also find it useful for unit testing to add this in my control-repo .fixtures.yml file.

symlinks:
  profiles: "File.join(source_dir, 'site-modules', 'profiles')"
  roles: "File.join(source_dir, 'site-modules', 'roles')"

[ardrigh]

If the word control is considered confusing, how about pdk new site then?

This would follow the general idea of having a separate site folder for internal modules, different from the modules directory for externally sourced (Forge/git) modules.

Again, thinking about this from a greenfields perspective for new users, and not copying existing opinionated defaults.

IMO, any new user setting up Puppet for the first time should be able to use PDK the same way an existing module developer would.

This document refers to site-modules https://puppet.com/docs/pe/2019.8/control_repo.html
As does the repo https://github.com/puppetlabs/control-repo

I am sure other people have used different names for the same thing.

[natemccurdy]

site has probably a more troubled history than control does, and I don't recommend using site for basically the same reason that @reidmv recommends not using control.

See the discussion where site was renamed to site-modules in the control-repo for an example of what I mean: puppetlabs/control-repo#71

"control-repo" seems to have become the canonical compound noun to describe the thing in discussion here.