Suffix artifactId with amd64/arm64 for the dist jars [skip ci] #7070

NvTimLiu · 2022-11-15T14:04:57Z

Build on x86_64/amd64 hosts to aggregate rapids-4-spark dist jar with -Dcpu_arch=amd64 Suffix 'amd64' for dist artifact, e.g., rapids-4-spark-amd64_2.12

Build on arm64/aarch64 hosts to aggregate rapids-4-spark dist jar with -Dcpu_arch=arm64 Suffix 'arm64' for dist artifact, e.g., rapids-4-spark-arm64_2.12

Signed-off-by: Tim Liu timl@nvidia.com

NOTE：

This PR should be merged along w/ many internal CICD updates, otherwise it may break regular CICD pipelines.
So please allow devops guys to merge this one at the appropriate time, thanks!

Build on x86_64/amd64 hosts to aggregate rapids-4-spark dist jar with -Dcpu_arch=amd64 Suffix 'amd64' for dist artifact, e.g., rapids-4-spark-amd64_2.12 Build on arm64/aarch64 hosts to aggregate rapids-4-spark dist jar with -Dcpu_arch=arm64 Suffix 'arm64' for dist artifact, e.g., rapids-4-spark-arm64_2.12 Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu · 2022-11-15T14:05:10Z

build

NvTimLiu · 2022-11-15T14:08:34Z

The solution is still under discussing. Will update scala/doc files when the final solution is made

build/buildall

pom.xml

pxLi · 2022-11-16T03:18:58Z

If we would like to have UCX on arm, the build must support to correctly switch arch-based jucx dependency

also cc @abellina for help

Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu · 2022-11-16T10:57:19Z

build

jlowe · 2022-11-16T15:17:50Z

dist/pom.xml

@@ -24,7 +24,7 @@
        <artifactId>rapids-4-spark-parent</artifactId>
        <version>22.12.0-SNAPSHOT</version>
    </parent>
-    <artifactId>rapids-4-spark_2.12</artifactId>
+    <artifactId>rapids-4-spark-${cpu_arch}_2.12</artifactId>


Based on an earlier discussion, I thought it was not allowed to have a variable in the artifactId? Seems like we want to split dist into dist-amd64 and dist-arm64, and maybe they rely on a dist-common that builds an intermediate jar using the CPU architecture in the classifier, similar to how we use the Spark version in the classifier of our sql-plugin intermediate jars.

I thought it was not allowed to have a variable in the artifactId?

maven does not allow to put a var in project root module (rapids-4-spark-parent in our case),
but we could add it in submodule artifactID.

do you think directly use artifactID+{cpu.arch} might be easier than parallel world dist-common+classifier{cpu.arch}?

jlowe · 2022-11-16T15:19:02Z

build/buildall

+  aarch64|arm64)
+    cpu_arch='arm64';;
+  *)
+    echo "Non-support cpu architecture: ${arch}"; exit 1;;


Nit, applies to all the places where it was copied. Not must-fix.

Suggested change

echo "Non-support cpu architecture: ${arch}"; exit 1;;

echo "Unsupported CPU architecture: ${arch}"; exit 1;;

Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu · 2022-11-17T03:30:36Z

jenkins/Jenkinsfile-blossom.premerge

@@ -166,7 +166,11 @@ pipeline {
            steps {
                script {
                    // Retrieve PROJECT_VER from version-def.sh, e.g, '<major>.<minor>.<patch>-SNAPSHOT'
-                    PROJECT_VER = sh(returnStdout: true, script: "bash $JENKINS_ROOT/version-def.sh | cut -d ',' -f 3 | cut -d ' ' -f 3")


Change to retrieve PROJECT_VER

The previous way of parsing the bash output by | cut -d ',' -f 3 ... from version-def.sh can not work well, if we change echo string in version-def.sh

The previous output from version-def is:
CUDF_VER: 22.12.0-SNAPSHOT, CUDA_CLASSIFIER: cuda11, PROJECT_VER: 22.12.0-SNAPSHOT, SPARK_VER: 3.1.1 ...

But | cut -d ',' -f 3 ... does not work if we :

add echo $arcg ahead,
--> bash output : amd64 CUDF_VER: 22.12.0-SNAPSHOT, CUDA_CLASSIFIER: cuda11, PROJECT_VER: 22.12.0-SNAPSHOT, SPARK_VER: 3.1.1 ..
--> PROJECT_VER=amd64 22.12.0-SNAPSHOT

or if remove CUDF_VER: 22.12.0-SNAPSHOT, from previous output:
--> bash output : CUDA_CLASSIFIER: cuda11, PROJECT_VER: 22.12.0-SNAPSHOT, SPARK_VER: 3.1.1 ..
--> PROJECT_VER=3.1.1

please revert this file change too

sure, will update jenkins/Jenkinsfile-blossom.premerge in follow up issue PR changing the | cut -d ',' -f 3 ...

NvTimLiu · 2022-11-17T03:32:03Z

build

Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu · 2022-11-17T05:49:34Z

build

pxLi · 2022-11-17T06:02:18Z

NOTE：

This PR should be merged along w/ many internal CICD updates, otherwise it may break regular CICD pipelines.

So please allow devops guys to merge this one at the appropriate time, thanks!

NvTimLiu · 2022-11-17T10:52:23Z

CI IT against the new artifact name rapids-4-spark-amd64_2.12-22.12.0-SNAPSHOT-cuda11.jar, though it failed on some tests

[2022-11-17T06:02:03.861Z] [INFO] Building jar: /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-6095-ci-2/dist/target/rapids-4-spark-amd64_2.12-22.12.0-SNAPSHOT-cuda11.jar
...
[2022-11-17T06:05:32.164Z] AND PLUGIN JARS: /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-6095-ci-2/dist/target/rapids-4-spark-amd64_2.12-22.12.0-SNAPSHOT-cuda11.jar:/home/jenkins/agent/workspace/jenkins-rapids_premerge-github-6095-ci-2/integration_tests/target/rapids-4-spark-integration-tests_2.12-22.12.0-SNAPSHOT-spark311.jar:/home/jenkins/agent/workspace/jenkins-rapids_premerge-github-6095-ci-2/integration_tests/target/dependency/parquet-hadoop-1.10.1-tests.jar

...

[2022-11-17T06:05:50.671Z] 22/11/17 06:05:49 WARN Executor: Issue communicating with driver in heartbeater
[2022-11-17T06:05:50.672Z] org.apache.spark.SparkException: Exception thrown in awaitResult: 
...
[2022-11-17T06:05:50.672Z] Caused by: java.lang.NullPointerException
[2022-11-17T06:05:50.672Z] 	at org.apache.spark.storage.BlockManagerMasterEndpoint.org$apache$spark$storage$BlockManagerMasterEndpoint$$register(BlockManagerMasterEndpoint.scala:524)
[2022-11-17T06:05:50.672Z] 	at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:116)
[2022-11-17T06:05:50.672Z] 	at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103)
[2022-11-17T06:05:50.672Z] 	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)

...
[2022-11-17T06:05:50.672Z] Caused by: java.lang.NullPointerException
[2022-11-17T06:05:50.672Z] 	at org.apache.spark.storage.BlockManagerMasterEndpoint.org$apache$spark$storage$BlockManagerMasterEndpoint$$register(BlockManagerMasterEndpoint.scala:524)
[2022-11-17T06:05:50.672Z] 	at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:116)
[2022-11-17T06:05:50.672Z] 	at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103)
[2022-11-17T06:05:50.672Z] 	at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)

NvTimLiu · 2022-11-17T11:17:26Z

build

pxLi · 2022-11-18T09:43:37Z

build

pxLi · 2022-11-18T09:45:55Z

build/buildall

+  aarch64|arm64)
+    cpu_arch='arm64';;
+  *)
+    echo "Unsupport CPU architecture: ${arch}"; exit 1;;


Unsupported

I think Jason already commented

pxLi · 2022-11-18T10:05:00Z

Convert to draft to avoid accidental merge, thx

pxLi · 2022-11-21T01:08:36Z

also filed #7116 to tracking potential jucx update

Signed-off-by: Tim Liu <timl@nvidia.com>

pxLi · 2022-11-21T01:19:28Z

skip ci for recent typo fix

NvTimLiu · 2022-11-21T01:19:33Z

build

pxLi · 2022-11-21T01:19:34Z

build

pxLi

Let's merge this and focus on corresponding CICD update.

If any issue, please help file new follow-ups, thanks!

NVIDIA#7070)" This reverts commit c7f87f4.

…]] (#7070)(#7120)" (#7135) * Revert "Suffix artifactId with amd64/arm64 for the dist jars [skip ci] (#7070)" This reverts commit c7f87f4. * Revert "Build noSnapshots without cdh shims on arm CPU [skip ci] (#7120)" This reverts commit b7a79d3. Signed-off-by: Tim Liu <timl@nvidia.com> Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu added the build Related to CI / CD or cleanly building label Nov 15, 2022

NvTimLiu requested review from jlowe, revans2, tgravescs, pxLi, zhanga5 and GaryShen2008 November 15, 2022 14:04

NvTimLiu self-assigned this Nov 15, 2022

jlowe reviewed Nov 15, 2022

View reviewed changes

build/buildall Outdated Show resolved Hide resolved

gerashegalov reviewed Nov 15, 2022

View reviewed changes

build/buildall Outdated Show resolved Hide resolved

pom.xml Show resolved Hide resolved

pxLi marked this pull request as draft November 16, 2022 01:37

pxLi marked this pull request as ready for review November 16, 2022 03:07

Fail if non-support cpu architectur

bc1084e

Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu mentioned this pull request Nov 16, 2022

[DOC] Update artifactId with amd64/arm64 for the spark-rapids dist jars in doc and Scala files #7076

Open

jlowe reviewed Nov 16, 2022

View reviewed changes

NvTimLiu added 2 commits November 17, 2022 01:46

Change to 'Unsupport CPU architecture'

e99996b

Signed-off-by: Tim Liu <timl@nvidia.com>

Retrive PROJECT_VER from version-def.sh

35a37c5

Signed-off-by: Tim Liu <timl@nvidia.com>

NvTimLiu commented Nov 17, 2022

View reviewed changes

NvTimLiu added 2 commits November 17, 2022 04:42

Move cpu_arch source block after all the definition of default VARs

4e725a5

Signed-off-by: Tim Liu <timl@nvidia.com>

Revert Jenkinsfile change, will follow up in another issue

80db0b0

Signed-off-by: Tim Liu <timl@nvidia.com>

This was referenced Nov 17, 2022

Support cpu-arch artifact for rapids-4-spark_2.12 NVIDIA/spark-rapids-examples#247

Closed

Support cpu-arch artifact for rapids-4-spark_2.12 NVIDIA/spark-rapids-ml#91

Merged

pxLi reviewed Nov 18, 2022

View reviewed changes

pxLi marked this pull request as draft November 18, 2022 10:05

Unsupported

02be71b

Signed-off-by: Tim Liu <timl@nvidia.com>

pxLi changed the title ~~Suffix artifactId with amd64/arm64 for the dist jars~~ Suffix artifactId with amd64/arm64 for the dist jars [skip ci] Nov 21, 2022

pxLi approved these changes Nov 21, 2022

View reviewed changes

pxLi marked this pull request as ready for review November 21, 2022 01:20

pxLi merged commit c7f87f4 into NVIDIA:branch-22.12 Nov 21, 2022

NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this pull request Nov 22, 2022

Revert "Suffix artifactId with amd64/arm64 for the dist jars [skip ci] (

224bb00

NVIDIA#7070)" This reverts commit c7f87f4.

NvTimLiu mentioned this pull request Nov 22, 2022

Revert "Suffix artifactId with amd64/arm64 for the dist jars [skip ci]] (#7070)(#7120)" #7135

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Suffix artifactId with amd64/arm64 for the dist jars [skip ci] #7070

Suffix artifactId with amd64/arm64 for the dist jars [skip ci] #7070

NvTimLiu commented Nov 15, 2022 •

edited by pxLi

Loading

NvTimLiu commented Nov 15, 2022

NvTimLiu commented Nov 15, 2022

pxLi commented Nov 16, 2022 •

edited

Loading

NvTimLiu commented Nov 16, 2022

jlowe Nov 16, 2022

pxLi Nov 17, 2022 •

edited

Loading

jlowe Nov 16, 2022

NvTimLiu Nov 17, 2022

NvTimLiu Nov 17, 2022

pxLi Nov 17, 2022

NvTimLiu Nov 17, 2022 •

edited

Loading

NvTimLiu commented Nov 17, 2022

NvTimLiu commented Nov 17, 2022

pxLi commented Nov 17, 2022

NvTimLiu commented Nov 17, 2022 •

edited

Loading

NvTimLiu commented Nov 17, 2022

pxLi commented Nov 18, 2022

pxLi Nov 18, 2022

pxLi commented Nov 18, 2022

pxLi commented Nov 21, 2022

pxLi commented Nov 21, 2022

NvTimLiu commented Nov 21, 2022

pxLi commented Nov 21, 2022

pxLi left a comment

	echo "Non-support cpu architecture: ${arch}"; exit 1;;
	echo "Unsupported CPU architecture: ${arch}"; exit 1;;

Suffix artifactId with amd64/arm64 for the dist jars [skip ci] #7070

Suffix artifactId with amd64/arm64 for the dist jars [skip ci] #7070

Conversation

NvTimLiu commented Nov 15, 2022 • edited by pxLi Loading

NvTimLiu commented Nov 15, 2022

NvTimLiu commented Nov 15, 2022

pxLi commented Nov 16, 2022 • edited Loading

NvTimLiu commented Nov 16, 2022

jlowe Nov 16, 2022

Choose a reason for hiding this comment

pxLi Nov 17, 2022 • edited Loading

Choose a reason for hiding this comment

jlowe Nov 16, 2022

Choose a reason for hiding this comment

NvTimLiu Nov 17, 2022

Choose a reason for hiding this comment

NvTimLiu Nov 17, 2022

Choose a reason for hiding this comment

pxLi Nov 17, 2022

Choose a reason for hiding this comment

NvTimLiu Nov 17, 2022 • edited Loading

Choose a reason for hiding this comment

NvTimLiu commented Nov 17, 2022

NvTimLiu commented Nov 17, 2022

pxLi commented Nov 17, 2022

NvTimLiu commented Nov 17, 2022 • edited Loading

NvTimLiu commented Nov 17, 2022

pxLi commented Nov 18, 2022

pxLi Nov 18, 2022

Choose a reason for hiding this comment

pxLi commented Nov 18, 2022

pxLi commented Nov 21, 2022

pxLi commented Nov 21, 2022

NvTimLiu commented Nov 21, 2022

pxLi commented Nov 21, 2022

pxLi left a comment

Choose a reason for hiding this comment

NvTimLiu commented Nov 15, 2022 •

edited by pxLi

Loading

pxLi commented Nov 16, 2022 •

edited

Loading

pxLi Nov 17, 2022 •

edited

Loading

NvTimLiu Nov 17, 2022 •

edited

Loading

NvTimLiu commented Nov 17, 2022 •

edited

Loading