pdk validate: Rubocop config errors are not displayed in the default output level

[sanfrancrisko]

Originally reported in PDK-1680:

Environment

• PDK 1.18.1
• Running on a linux docker container (centos)

Description

Errors from Rubocop (and possibly other tools) are swallowed by PDK, instead of reporting to the user.

To recreate:

1. edit .rubocop.yml to set "TargetRubyVersion" to something invalid eg. 99.0
2. run "pdk validate"
3. Observe that no errors are reported (including any normal rubocop output)
4. run ""pdk validate --debug"
5. Observe that rubocop failed to run at all, due to the config error, and there is stderr output that was not reported to the user

The only indication that there is an issue is that there is a cross instead of a tick in the basic pdk output for ruby:

┌ [✖] Running ruby validators ...
└── [✖] Checking Ruby code style (*/*.rb).

This isn't very obvious and isn't being picked up by our automated tests currently (we use the actual output).

Impact

Ruby issues could go undetected due to a misconfiguration of rubocop

How this was identified

By default rubocop is (for some reason) set to validate against quite an old version of ruby as the minimum version (2.1).

This gave us errors related to newer formats, so we updated the value of TargetRubyVersion parameter to 2.5, this then gave no output at all (as the included rubocop gem is old, it doesn't support 2.5 despite 2.5 being the version puppet uses now).

By running with --debug we identified that pdk had swallowed the rubocop config error.

We suspect pdk behaves the same way for other tool config errors too.

Return code?

It does appear that pdk has a exit code of 2 when this happens, compared to 1 for "normal error output"

The documentation simply says:

"The exit code is non-zero when errors occur."

Is there more detail that can be documented? as using the return code can be very helpful for test automation

[sanfrancrisko]

From last conversation, @DavidS recommended Jon try the --format=junit option to see if it fits the bill. If not, we can resume the conversation here.

[jpsheffield]

@sanfrancrisko the junit format option doesn't help, it too completely omits the rubocop output in this situation.
So if rubocop is misconfigured or not working for any reason nothing reports this and all issues rubocop would have identified go unfound.

[sanfrancrisko]

Thanks for providing that feedback @jpsheffield . We'll look to get this triaged / reproduced and then raised as an issue.