[DB Install] Likely after #2347 `make install` does not install DB files correctly #2701

junliume · 2024-01-27T03:37:12Z

[Observation]:

about make install at this moment:
Install the project...
-- Install configuration: "Release"
-- Installing: /opt/rocm/share/doc/miopen-hip/LICENSE.txt
-- Installing: /opt/rocm/include/miopen/export.h
-- Installing: /opt/rocm/share/miopen/db/gfx908.tn.model
-- Installing: /opt/rocm/share/miopen/db/gfx908_ConvAsm1x1U_decoder.ktn.model
-- Installing: /opt/rocm/share/miopen/db/gfx908_ConvAsm1x1U_encoder.ktn.model
-- Installing: /opt/rocm/share/miopen/db/gfx908_ConvAsm1x1U_metadata.ktn.model
-- Installing: /opt/rocm/share/miopen/db/gfx908_metadata.tn.model
-- Installing: /opt/rocm/share/miopen/db/gfx90a.tn.model
-- Installing: /opt/rocm/share/miopen/db/gfx90a_ConvHipIgemmGroupFwdXdlops_decoder.ktn.model
-- Installing: /opt/rocm/share/miopen/db/gfx90a_ConvHipIgemmGroupFwdXdlops_encoder.ktn.model
-- Installing: /opt/rocm/share/miopen/db/gfx90a_ConvHipIgemmGroupFwdXdlops_metadata.ktn.model
-- Installing: /opt/rocm/share/miopen/db/gfx90a_metadata.tn.model
-- Up-to-date: /opt/rocm/include
-- Up-to-date: /opt/rocm/include/miopen
-- Installing: /opt/rocm/include/miopen/miopen.h
-- Up-to-date: /opt/rocm/include
-- Up-to-date: /opt/rocm/include/miopen
-- Up-to-date: /opt/rocm/include/miopen/export.h
-- Installing: /opt/rocm/include/miopen/config.h
-- Installing: /opt/rocm/include/miopen/version.h
-- Installing: /opt/rocm/lib/libMIOpen.so.1.0.60100
-- Up-to-date: /opt/rocm/lib/libMIOpen.so.1
-- Set runtime path of "/opt/rocm/lib/libMIOpen.so.1.0.60100" to ""
-- Up-to-date: /opt/rocm/lib/libMIOpen.so
-- Installing: /opt/rocm/lib/cmake/miopen/miopen-targets.cmake
-- Installing: /opt/rocm/lib/cmake/miopen/miopen-targets-release.cmake
-- Installing: /opt/rocm/lib/cmake/miopen/miopen-config.cmake
-- Installing: /opt/rocm/lib/cmake/miopen/miopen-config-version.cmake
-- Installing: /opt/rocm/bin/MIOpenDriver
-- Set runtime path of "/opt/rocm/bin/MIOpenDriver" to ""
-- Installing: /opt/rocm/bin/test_perf.py
-- Installing: /opt/rocm/share/miopen/perf_models/Alexnet_v1_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Alexnet_v1_FP16_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Alexnet_v1_FP32_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Alexnet_v1_FP32_BS4.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Alexnet_v1_FP32_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Alexnet_v1_FP32_BS64.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Densenet201_v1_FP16_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Densenet201_v1_FP32_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Densenet_v1_FP16_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Densenet_v1_FP32_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Googlenet_v1_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Googlenet_v1_FP16_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Googlenet_v1_FP32_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Googlenet_v1_FP32_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Inception3_v1_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Inception3_v1_FP32_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Inception3_v1_FP32_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Inception4_v1_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Inception4_v1_FP16_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Inception4_v1_FP32_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Inception4_v1_FP32_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Mobilenet_v1_FP32_BS4.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Mobilenet_v1_FP32_BS64.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet101_v1_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet101_v1_FP16_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet101_v1_FP16_BS32.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet101_v1_FP16_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet101_v1_FP32_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet101_v1_FP32_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet101_v1_FP32_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet152_v1_FP16_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet152_v1_FP32_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet152_v2_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet152_v2_FP16_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet152_v2_FP32_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet152_v2_FP32_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1.5_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1.5_FP16_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1.5_FP16_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1.5_FP32_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1.5_FP32_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1.5_FP32_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1_FP16_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1_FP16_BS32.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1_FP16_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1_FP16_BS64.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1_FP32_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1_FP32_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Resnet50_v1_FP32_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/SSD_v1_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/SSD_v1_FP32_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/Shufflenet_v2_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG11_v1_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG11_v1_FP16_BS256.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG11_v1_FP16_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG11_v1_FP32_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG16_v1_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG16_v1_FP32_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG16_v1_FP32_BS4.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG16_v1_FP32_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG16_v1_FP32_BS64.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG19_v1_FP16_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG19_v1_FP16_BS512.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG19_v1_FP32_BS128.txt
-- Installing: /opt/rocm/share/miopen/perf_models/VGG19_v1_FP32_BS512.txt
-- Installing: /opt/rocm/bin/install_precompiled_kernels.sh

meanwhile:

✗ ls share/miopen/db 
gfx1030_36.db             gfx900_56.HIP.fdb.txt     gfx906_64.HIP.fdb.txt                 gfx908_ConvAsm1x1U_metadata.ktn.model  gfx90a_ConvHipIgemmGroupFwdXdlops_decoder.ktn.model
gfx1030_36.HIP.fdb.txt    gfx900_56.OpenCL.fdb.txt  gfx906_64.OpenCL.fdb.txt              gfx908_metadata.tn.model               gfx90a_ConvHipIgemmGroupFwdXdlops_encoder.ktn.model
gfx803_36.HIP.fdb.txt     gfx900_64.HIP.fdb.txt     gfx90878.db                           gfx908.tn.model                        gfx90a_ConvHipIgemmGroupFwdXdlops_metadata.ktn.model
gfx803_36.OpenCL.fdb.txt  gfx900_64.OpenCL.fdb.txt  gfx90878.HIP.fdb.txt                  gfx90a68.db                            gfx90a_metadata.tn.model
gfx803_64.HIP.fdb.txt     gfx906_60.db              gfx90878.OpenCL.fdb.txt               gfx90a68.HIP.fdb.txt                   gfx90a.tn.model
gfx803_64.OpenCL.fdb.txt  gfx906_60.HIP.fdb.txt     gfx908_ConvAsm1x1U_decoder.ktn.model  gfx90a6e.db                            gfx942130.db
gfx900_56.db              gfx906_60.OpenCL.fdb.txt  gfx908_ConvAsm1x1U_encoder.ktn.model  gfx90a6e.HIP.fdb.txt                   gfx942130.HIP.fdb.txt

So likely after #2347 , these files are "made" in build directly, but not installed at correct location at /opt/rocm.

The text was updated successfully, but these errors were encountered:

junliume · 2024-02-05T21:13:01Z

@JehandadKhan @apwojcik unfortunately it seems that the problem is not fixed as expected. The .HIP.fdb.txt and .db files are still not been installed or be part of the package if we run make package
CC: @atamazov

atamazov · 2024-02-05T21:29:52Z

@JehandadKhan @apwojcik I am not sure but maybe https://github.com/ROCm/MIOpen/pull/2347/files#r1478928388

apwojcik · 2024-02-05T21:50:31Z

@junliume @atamazov, I will look into it tomorrow in my time zone (CET).

junliume · 2024-02-05T22:29:40Z

One observation particular to this issue: we need to completely remove build folder to reproduce and/or confirm a "fix", sometimes it looks like adding some debug info could "trigger" the fdb.txt and .db installation, but when building from scratch the issue re-appears again.

[Reproducing Steps]:

build MIOpen from scratch by deleting old build directory. Run make install, (sample script below).
The observation is that the following file (for example) exists in build directory but not installed:

root:/data/driver/MIOpen/build# ls share/miopen/db/gfx90a6e.HIP.fdb.txt
share/miopen/db/gfx90a6e.HIP.fdb.txt
root:/data/driver/MIOpen/build# ls /opt/rocm/share/miopen/db/gfx90a6e.HIP.fdb.txt
ls: cannot access '/opt/rocm/share/miopen/db/gfx90a6e.HIP.fdb.txt': No such file or directory

However, if we touch the following file:

root@:/data/driver/MIOpen/build# touch ../src/CMakeLists.txt

then the file is installed correctly!

root@:/data/driver/MIOpen/build# ls /opt/rocm/share/miopen/db/gfx90a6e.HIP.fdb.txt
/opt/rocm/share/miopen/db/gfx90a6e.HIP.fdb.txt

P.S. The sample install script:

export LC_ALL=C.UTF-8 && \
export LANG=C.UTF-8 && \
apt-get update -y && \
apt-get install -y sudo && \
sudo apt-get -y install git git-lfs unzip lbzip2 python3.9-venv pkg-config vim ninja-build half && \
sudo apt update && \
mkdir -p /data/driver && cd /data/driver && git clone -b develop https://github.com/ROCmSoftwarePlatform/MIOpen && cd MIOpen && \
mkdir -p install_dir && cmake -P install_deps.cmake --minimum --prefix /data/driver/MIOpen/install_dir && \
mkdir -p build && \
cd build && \
CXX=/opt/rocm/bin/amdclang++ CC=/opt/rocm/bin/amdclang cmake -DMIOPEN_BACKEND=HIP -DCMAKE_PREFIX_PATH="/data/driver/MIOpen/install_dir;/opt/rocm" \
-DMIOPEN_TEST_ALL=On -DMIOPEN_TEST_HALF=On .. && \
make -j$(nproc) install

apwojcik · 2024-02-06T01:08:45Z

@junliume fix for missing files after installation #2728

apwojcik · 2024-02-06T01:14:38Z

@JehandadKhan @junliume I merged the updated Embed.cmake for MIGraphX to accommodate Windows resource files. If you are interested, I would also submit PR for MIOpen.

junliume assigned JehandadKhan and apwojcik Jan 27, 2024

junliume added bug urgency_blocker labels Jan 27, 2024

junliume mentioned this issue Jan 27, 2024

[HotFix] Fix DB install after #2347 #2702

Merged

junliume closed this as completed Jan 31, 2024

junliume reopened this Feb 5, 2024

junliume added this to the MIOpen v3.1.0 milestone Feb 5, 2024

junliume closed this as completed Feb 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DB Install] Likely after #2347 `make install` does not install DB files correctly #2701

[DB Install] Likely after #2347 `make install` does not install DB files correctly #2701

junliume commented Jan 27, 2024

junliume commented Feb 5, 2024

atamazov commented Feb 5, 2024

apwojcik commented Feb 5, 2024

junliume commented Feb 5, 2024 •

edited

Loading

apwojcik commented Feb 6, 2024

apwojcik commented Feb 6, 2024

[DB Install] Likely after #2347 make install does not install DB files correctly #2701

[DB Install] Likely after #2347 make install does not install DB files correctly #2701

Comments

junliume commented Jan 27, 2024

junliume commented Feb 5, 2024

atamazov commented Feb 5, 2024

apwojcik commented Feb 5, 2024

junliume commented Feb 5, 2024 • edited Loading

apwojcik commented Feb 6, 2024

apwojcik commented Feb 6, 2024

[DB Install] Likely after #2347 `make install` does not install DB files correctly #2701

[DB Install] Likely after #2347 `make install` does not install DB files correctly #2701

junliume commented Feb 5, 2024 •

edited

Loading