Checkmarx-GitLab Merge Request: 401 Unauthorized

[kayvonkhosrowpour]

Problem

I'm getting a 401 Unauthorized error in the GitLab job logs when running the checkmarx-scan-mr job for a merge request:

2021-06-02 20:50:58.489 ERROR 112 --- [main] c.c.f.c.GitLabIssueTracker [VL6xfeGJ] : 
Error calling gitlab project api {"message":"401 Unauthorized"}
org.springframework.web.client.HttpClientErrorException$Unauthorized: 401 Unauthorized: [{"message":"401 Unauthorized"}]
	...
2021-06-02 20:50:58.491 ERROR 112 --- [main] c.c.f.c.GitLabIssueTracker [VL6xfeGJ] : 
Could not obtain GitLab Project Id for <REDACTED>
2021-06-02 20:50:58.495 ERROR 112 --- [main] c.c.f.CxFlowRunner [VL6xfeGJ] : An error has occurred.
com.checkmarx.flow.exception.MachinaException: Could not obtain GitLab Project Id
	...

Problem Details

I'm following the Checkmarx-GitLab integration as described in the guide here: https://checkmarx.atlassian.net/wiki/spaces/SD/pages/1929937052/GitLab+Integration

I am able to successfully run scans without returning results back to GitLab. The scan results are showing in Checkmarx as expected. However, I am having trouble with the merge request comments. I am using the same template provided in the tutorial and I've added all of the variables that are required, including the personal access token GITLAB_TOKEN with the required scopes (in fact, I've tried providing all scopes).

I am not sure if this is an issue or if there is additional configuration required that the guide does not specify (or that I am missing).

[jbrotsos]

Hey @kayvonkhosrowpour ,

Thanks for the question! Can you do me a favor and try a curl command to create an issue to see if it works without Checkmarx?

https://docs.gitlab.com/ee/api/issues.html

Example:

curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/4/issues?title=Issues%20with%20auth&labels=bug"

[kayvonkhosrowpour]

Hi @jbrotsos,

I successfully ran that command using the same project and I was able to create the issue just fine without any authorization issues. I provided the same access token as I have in the GitLab environment variables.

Let me know what information I can provide that would be helpful. Here are some details:

Variables:

variables:
    CHECKMARX_DOCKER_IMAGE: "cx-flow"
    CX_FLOW_BUG_TRACKER: "None"
    CX_FLOW_BUG_TRACKER_IMPL: ${CX_FLOW_BUG_TRACKER}
    CX_FLOW_EXE: "java -jar /app/cx-flow.jar"
    CX_PROJECT: "$CI_PROJECT_NAME-$CI_COMMIT_REF_NAME"
    CHECKMARX_VERSION: "9.0"
    CHECKMARX_SETTINGS_OVERRIDE: "false"
    CHECKMARX_EXCLUDE_FILES: ""
    CHECKMARX_EXCLUDE_FOLDERS: ""
    CHECKMARX_CONFIGURATION: "Default Configuration"
    CHECKMARX_SCAN_PRESET: "Checkmarx Default"
    CX_FLOW_FILTER_SEVERITY: "High"
    CX_FLOW_FILTER_CATEGORY: ""
    CX_FLOW_FILTER_CWE: ""
    CX_FLOW_FILTER_STATUS: ""
    CX_FLOW_FILTER_STATE: ""
    CX_FLOW_ENABLED_VULNERABILITY_SCANNERS: sast
    CX_TEAM: "<REDACTED>"
    CX_FLOW_BREAK_BUILD: "false"
    SCA_FILTER_SEVERITY: ""
    SCA_FILTER_SCORE: ""
    SCA_THRESHOLDS_SCORE: ""
    SCA_TEAM: ""
    GITLAB_BLOCK_MERGE: "false"
    GITLAB_ERROR_MERGE: "false"
    PARAMS: ""

The merge request job that I am using (taken from the provided template):

checkmarx-scan-mr:
  stage: test
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  image:   
    name: checkmarx/${CHECKMARX_DOCKER_IMAGE}
    entrypoint: ['']
  variables:
    CHECKMARX_INCREMENTAL: "true"
  script:
    - ${CX_FLOW_EXE}
          --scan 
          --bug-tracker="GitLabMerge"
          --app="${CI_PROJECT_NAME}" 
          --namespace="${CI_PROJECT_NAMESPACE}"
          --repo-name="${CI_PROJECT_NAME}"
          --repo-url="${CI_REPOSITORY_URL}" 
          --cx-team="${CX_TEAM}" 
          --cx-project="${CX_PROJECT}" 
          --branch="${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}" 
          --project-id="${CI_PROJECT_ID}" 
          --merge-id="${CI_MERGE_REQUEST_IID}" 
          --spring.profiles.active="${CX_FLOW_ENABLED_VULNERABILITY_SCANNERS}" 
          --f=. 
          ${PARAMS}

GitLab environment variables that are setup:

CHECKMARX_BASE_URL
CHECKMARX_USERNAME
CHECKMARX_PASSWORD
GITLAB_TOKEN

[jbrotsos]

@kayvonkhosrowpour can you try changing this:

CX_FLOW_BUG_TRACKER: "none"
to
CX_FLOW_BUG_TRACKER: "GitLab"

My thought is that the CX_FLOW_BUG_TRACKER_IMPL can't be "none".

Let me know.

[kayvonkhosrowpour]

That does not work either --- "GitLab", "GitLabDashboard", and "None" all produce the same effect -- but does setting that variable matter? In the script of the checkmarx-scan-mr job, there is an argument "bug-tracker" that came from the GitLab template. It is set to GitLabMerge, which I assume overrides the environment variable.

[jbrotsos]

@kayvonkhosrowpour It overrides CX_FLOW_BUG_TRACKER but not CX_FLOW_BUG_TRACKER_IMPL.

Can you reproduce this on a public project that I can fork?

[kayvonkhosrowpour]

I'm not sure I will be able to reproduce this on a public project since our company's Checkmarx server cannot be accessed externally. However, I have a few more details that may be of some interest. I set the logging level to TRACE and found the below:

2021-06-03 21:54:37.403 DEBUG 474 --- [main] o.s.w.c.RestTemplate [qpqusq04] : HTTP GET https://gitlab.com/api/v4/projects/<REDACTED>/merge_requests/<REDACTED>/notes
2021-06-03 21:54:37.403 DEBUG 474 --- [main] o.s.w.c.RestTemplate [qpqusq04] : Accept=[application/json, application/*+json]
2021-06-03 21:54:37.641 DEBUG 474 --- [main] o.s.w.c.RestTemplate [qpqusq04] : Response 401 UNAUTHORIZED
2021-06-03 21:54:37.645 ERROR 474 --- [main] c.c.f.s.RepoService [qpqusq04] : Error while adding or updating GITLAB repo pull request comment

It looks like the gitlab URL is not being set to my company's gitlab URL.

[kayvonkhosrowpour]

I found the isssue! You must specify the following two environment variables:

GITLAB_URL and GITLAB_API_URL
For example:

GITLAB_URL: "https://gitlab-master.<YOUR_COMPANY>.com"
GITLAB_API_URL: "https://gitlab-master.<YOUR_COMPANY>.com/api/v4"

[jbrotsos]

Can you please try this:

variables:
GITLAB_URL: "${CI_SERVER_URL}"
GITLAB_API_URL: "${CI_API_V4_URL}"

And let me know if it works, I'm also trying to add something else to a new version of the template.

[rahul6941]

its not working .
2021-06-18 14:04:20.145 ERROR 10 --- [ main] c.c.f.s.RepoService [yDGV0NzJ] : Error while adding or updating GITLAB repo pull request comment
org.springframework.web.client.HttpClientErrorException$Unauthorized: 401 Unauthorized: [{"message":"401 Unauthorized"}]

[rahul6941]

Please check trace level log . I can see when its call to Gitlab API then its not calling first GitLab Auth API to retrieve tokens and pass token in header or PRIVATE-TOKEN

http-outgoing-1 << "end of stream"
2021-06-18 14:46:49.340 DEBUG 10 --- [ main] o.a.http.wire [tLww8Xkd] : http-outgoing-2 >> "GET /api/v4/projects/<project_id>/merge_requests/<merge_request_id>/notes HTTP/1.1[\r][\n]"
2021-06-18 14:46:49.340 DEBUG 10 --- [ main] o.a.http.wire [tLww8Xkd] : http-outgoing-2 >> "Accept: application/json[\r][\n]"
2021-06-18 14:46:49.340 DEBUG 10 --- [ main] o.a.http.wire [tLww8Xkd] : http-outgoing-2 >> "Content-Type: application/json[\r][\n]"
2021-06-18 14:46:49.340 DEBUG 10 --- [ main] o.a.http.wire [tLww8Xkd] : http-outgoing-2 >> "PRIVATE-TOKEN: xxxx[\r][\n]"

[jbrotsos]

@rahul6941, can you run the following curl command:

curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/4/issues?title=Issues%20with%20auth&labels=bug"

and confirm that https://gitlab.example.com is your GITLAB_URL env variable

[rahul6941]

If we add GITLAB_TOKEN={Personal_access_token} at group level it worked . Thanks for help