[Auto Techsupport] Event driven Techsupport Changes #1796

vivekrnv · 2021-09-03T01:02:45Z

What I did

sonic-utilities changes required for feature "Event Driven TechSupport Invocation & CoreDump Mgmt". HLD

Summary of the changes:

Added the AUTO GEN CLI for the CFG DB tables required for this feature
Added the coredump_gen_handler.py & techsupport_cleanup.py scripts.
Added the UT's required for these scripts.
Enhanced coredump-compress & generate-dump scripts

How I did it

How to verify it

tests/coredump_gen_handler_test.py::TestCoreDumpCreationEvent::test_core_dump_cleanup PASSED [ 22%]
tests/coredump_gen_handler_test.py::TestCoreDumpCreationEvent::test_core_dump_with_exit_event_unknown_cmd PASSED [ 22%]
tests/coredump_gen_handler_test.py::TestCoreDumpCreationEvent::test_core_dump_with_no_exit_event PASSED [ 22%]
tests/coredump_gen_handler_test.py::TestCoreDumpCreationEvent::test_feature_table_not_set PASSED [ 22%]
tests/coredump_gen_handler_test.py::TestCoreDumpCreationEvent::test_global_rate_limit_interval PASSED [ 22%]
tests/coredump_gen_handler_test.py::TestCoreDumpCreationEvent::test_invalid_since_argument PASSED [ 22%]
tests/coredump_gen_handler_test.py::TestCoreDumpCreationEvent::test_invoc_ts_after_rate_limit_interval PASSED [ 22%]
tests/coredump_gen_handler_test.py::TestCoreDumpCreationEvent::test_invoc_ts_state_db_update PASSED [ 22%]
tests/coredump_gen_handler_test.py::TestCoreDumpCreationEvent::test_max_core_size_limit_not_crossed PASSED [ 22%]
tests/coredump_gen_handler_test.py::TestCoreDumpCreationEvent::test_per_proc_rate_limit_interval PASSED [ 22%]
tests/coredump_gen_handler_test.py::TestCoreDumpCreationEvent::test_since_argument PASSED [ 22%]
tests/techsupport_cleanup_test.py::TestTechsupportCreationEvent::test_dump_cleanup PASSED [ 79%]
tests/techsupport_cleanup_test.py::TestTechsupportCreationEvent::test_no_cleanup_state_disabled PASSED [ 79%]
tests/techsupport_cleanup_test.py::TestTechsupportCreationEvent::test_no_cleanup_state_enabled PASSED [ 80%]
tests/techsupport_cleanup_test.py::TestTechsupportCreationEvent::test_state_db_update PASSED [ 80%]

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)


admin@sonic:~$ show auto-techsupport global
STATE      RATE LIMIT INTERVAL    MAX TECHSUPPORT SIZE    MAX CORE SIZE  SINCE
-------  ---------------------  ----------------------  ---------------  ----------
enabled                    180                      10                5  2 days ago

admin@sonic:~$ show auto-techsupport-feature 
FEATURE NAME    STATE       RATE LIMIT INTERVAL
--------------  --------  ---------------------
bgp             enabled                     600
database        enabled                     600
dhcp_relay      enabled                     600
lldp            enabled                     600
macsec          enabled                     600
mgmt-framework  enabled                     600
nat             enabled                     600
pmon            enabled                     600
radv            enabled                     600
restapi         disabled                    800
sflow           enabled                     600
snmp            enabled                     600
swss            disabled                    800
syncd           enabled                     600
teamd           enabled                     600
telemetry       enabled                     600
admin@sonic:~$ show auto-techsupport history

TECHSUPPORT DUMP                          TRIGGERED BY    CORE DUMP
----------------------------------------  --------------  -----------------------------
sonic_dump_r-lionfish-16_20210901_221402  bgpcfgd         bgpcfgd.1630534439.55.core.gz
sonic_dump_r-lionfish-16_20210901_203725  snmp-subagent   python3.1630528642.23.core.gz
sonic_dump_r-lionfish-16_20210901_222408  lldpmgrd        python3.1630535045.34.core.gz

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…o event_driven_ts

…ic-utilities into event_driven_ts

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…o event_driven_ts

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

This reverts commit 7def4b7.

This reverts commit 4fdf805.

This reverts commit 7be9ee4.

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…t#1599)" This reverts commit 920bb87.

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…o event_driven_ts

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…o event_driven_ts

vivekrnv · 2021-10-27T08:21:50Z

scripts/coredump-compress

+if [[ ! -z $CONTAINER_ID ]]; then
+   CONTAINER_NAME=$(docker inspect --format='{{.Name}}' ${CONTAINER_ID} | cut -c2-)
+   if [[ ! -z ${CONTAINER_NAME} ]]; then
+      # coredump_gen_handler invokes techsupport if all the other required conditions  are met


@qiluo-msft Regarding the extra space char comment. Do you want the "#" to align with if?

azure-pipelines · 2021-10-28T03:00:54Z

Commenter does not have sufficient privileges for PR 1796 in repo Azure/sonic-utilities

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

vivekrnv · 2021-10-28T17:57:59Z

/azpw run

mssonicbld · 2021-10-28T17:58:01Z

/AzurePipelines run

azure-pipelines · 2021-10-28T17:58:13Z

Azure Pipelines successfully started running 1 pipeline(s).

vivekrnv · 2021-11-06T08:33:51Z

/azpw run

mssonicbld · 2021-11-06T08:33:52Z

/AzurePipelines run

azure-pipelines · 2021-11-06T08:34:04Z

Azure Pipelines successfully started running 1 pipeline(s).

liat-grozovik · 2021-11-11T16:16:45Z

@qiluo-msft and @ganglyu could you please review latest code changes?

qiluo-msft · 2021-11-13T02:09:08Z

scripts/coredump-compress

+   if [[ ! -z ${CONTAINER_NAME} ]]; then
+      # coredump_gen_handler invokes techsupport if all the other required conditions are met
+      # explicitly passing in the env vars because coredump-compress's namespace doesn't have these set by default
+      for path in $(find /usr/local/lib/python[^2]*/dist-packages -maxdepth 0); do


^2

check python and python3* only? #Closed

Only python3.x, Can't involve both python2 & python3 both. imports won't work if i do so

Then check python3* only? you are checking non-2.

qiluo-msft · 2021-11-13T02:11:19Z

scripts/coredump-compress

+      done
+      setsid $(echo > /tmp/coredump_gen_handler.log;
+             export PYTHONPATH=$PYTHONPATH;
+             python3 /usr/local/bin/coredump_gen_handler.py ${PREFIX}core.gz ${CONTAINER_NAME} &>> /tmp/coredump_gen_handler.log) &


python3

If add a shebang to the py file, do you still need to use python3 and PYTHONPATH? #Closed

i can avoid python3 but PYTHONPATH is still required.

qiluo-msft · 2021-11-13T02:13:45Z

show/plugins/auto_techsupport.py

+
+@AUTO_TECHSUPPORT.command(name="global")
+@clicommon.pass_db
+def AUTO_TECHSUPPORT_GLOBAL(db):


AUTO_TECHSUPPORT_GLOBAL

Do you have a strong reason to use UPPER_CASE function name? We prefer lower_case. #Closed

That's auto_generated by SONiC Auto CLI gen. #1644

vivekrnv · 2021-11-15T04:28:10Z

/azpw run

mssonicbld · 2021-11-15T04:28:12Z

/AzurePipelines run

azure-pipelines · 2021-11-15T04:28:23Z

Azure Pipelines successfully started running 1 pipeline(s).

vivekrnv · 2021-11-15T16:42:57Z

/azpw run

mssonicbld · 2021-11-15T16:42:59Z

/AzurePipelines run

azure-pipelines · 2021-11-15T16:43:11Z

Azure Pipelines successfully started running 1 pipeline(s).

dgsudharsan · 2021-11-15T22:18:03Z

@qiluo-msft Can you please sign off if all comments are addressed?

#### Why I did it Changes required for feature "Event Driven TechSupport Invocation & CoreDump Mgmt". [HLD](sonic-net/SONiC#818 ) Requires: sonic-net/sonic-utilities#1796. Merging in any order would be fine. Summary of the changes: - Added the YANG Models for the new tables introduces as a part of this feature. - Enhanced init_cfg.json with the default config required - Added a compile Time flag which enables/disables the config required for this feature inside the init_cfg.json - Enhanced the supervisor-proc-exit-listener script to populate `<feature>:<critical_proc> = <comm>:<pid>` info in the STATE_DB when it observes an proc exit notification for the critical processes running inside the docker.

vivekrnv and others added 30 commits August 7, 2021 23:30

Commit for Ref

999983c

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

TechSupport Tests Completed

104a412

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

auto_techsupport helper added

6bcfb5d

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

coredump_gen_script in progress

6b5ba3f

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

CoredumpHandler UT's completd and script fixed

843d329

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'master' of https://github.com/Azure/sonic-utilities int…

7050c38

…o event_driven_ts

Merge branch 'Azure:master' into event_driven_ts

4cc830d

Merge branch 'event_driven_ts' of https://github.com/vivekreddynv/son…

e32ea79

…ic-utilities into event_driven_ts

Added original Setup.py

c89f15f

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Test Changes

4b1faa2

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'master' of https://github.com/Azure/sonic-utilities int…

4e9a2a2

…o event_driven_ts

CLI GEN-1 merged

7be9ee4

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

CLI GEN-2 merged

4fdf805

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Removed a few tests

7def4b7

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Added setup.py

6bfb465

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Revert "Removed a few tests"

eba8261

This reverts commit 7def4b7.

Revert "CLI GEN-2 merged"

e89fac3

This reverts commit 4fdf805.

Revert "CLI GEN-1 merged"

7b08c27

This reverts commit 7be9ee4.

config feature added

a708f06

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Revert "[config][generic-update] Implementing patch sorting (sonic-ne…

e6122e9

…t#1599)" This reverts commit 920bb87.

Added new state table schema

6c4f96c

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'master' of https://github.com/Azure/sonic-utilities int…

d911b67

…o event_driven_ts

UT's updated for new design

1fbe04e

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Updated the Script and UT's

195b5ad

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Minor Change to test

9a50c0f

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Auto GEN CLI's added

3e66b70

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Updated Setup.py

b6ae7bb

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Tests Updated to use default mock infra

e8bfd2e

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

scripts updated

68f7e5c

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

CLI added

c446e5f

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'master' of https://github.com/Azure/sonic-utilities int…

5a4dcfc

…o event_driven_ts

vivekrnv commented Oct 27, 2021

View reviewed changes

env vars related and other minor comments addressed

86ea3fd

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'Azure:master' into event_driven_ts

0fc5391

vivekrnv requested a review from qiluo-msft November 11, 2021 19:29

Merge branch 'Azure:master' into event_driven_ts

3d37043

qiluo-msft reviewed Nov 13, 2021

View reviewed changes

Update coredump-compress

12f75f1

vivekrnv requested a review from qiluo-msft November 15, 2021 05:23

qiluo-msft approved these changes Nov 16, 2021

View reviewed changes

qiluo-msft merged commit a3e34e3 into sonic-net:master Nov 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Auto Techsupport] Event driven Techsupport Changes #1796

[Auto Techsupport] Event driven Techsupport Changes #1796

vivekrnv commented Sep 3, 2021

vivekrnv Oct 27, 2021 •

edited

Loading

azure-pipelines bot commented Oct 28, 2021

vivekrnv commented Oct 28, 2021

mssonicbld commented Oct 28, 2021

azure-pipelines bot commented Oct 28, 2021

vivekrnv commented Nov 6, 2021

mssonicbld commented Nov 6, 2021

azure-pipelines bot commented Nov 6, 2021

liat-grozovik commented Nov 11, 2021

qiluo-msft Nov 13, 2021 •

edited

Loading

vivekrnv Nov 13, 2021

qiluo-msft Nov 13, 2021

vivekrnv Nov 15, 2021

qiluo-msft Nov 13, 2021 •

edited

Loading

vivekrnv Nov 13, 2021

qiluo-msft Nov 13, 2021 •

edited

Loading

vivekrnv Nov 13, 2021

vivekrnv commented Nov 15, 2021

mssonicbld commented Nov 15, 2021

azure-pipelines bot commented Nov 15, 2021

vivekrnv commented Nov 15, 2021

mssonicbld commented Nov 15, 2021

azure-pipelines bot commented Nov 15, 2021

dgsudharsan commented Nov 15, 2021

[Auto Techsupport] Event driven Techsupport Changes #1796

[Auto Techsupport] Event driven Techsupport Changes #1796

Conversation

vivekrnv commented Sep 3, 2021

What I did

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

vivekrnv Oct 27, 2021 • edited Loading

Choose a reason for hiding this comment

azure-pipelines bot commented Oct 28, 2021

vivekrnv commented Oct 28, 2021

mssonicbld commented Oct 28, 2021

azure-pipelines bot commented Oct 28, 2021

vivekrnv commented Nov 6, 2021

mssonicbld commented Nov 6, 2021

azure-pipelines bot commented Nov 6, 2021

liat-grozovik commented Nov 11, 2021

qiluo-msft Nov 13, 2021 • edited Loading

Choose a reason for hiding this comment

vivekrnv Nov 13, 2021

Choose a reason for hiding this comment

qiluo-msft Nov 13, 2021

Choose a reason for hiding this comment

vivekrnv Nov 15, 2021

Choose a reason for hiding this comment

qiluo-msft Nov 13, 2021 • edited Loading

Choose a reason for hiding this comment

vivekrnv Nov 13, 2021

Choose a reason for hiding this comment

qiluo-msft Nov 13, 2021 • edited Loading

Choose a reason for hiding this comment

vivekrnv Nov 13, 2021

Choose a reason for hiding this comment

vivekrnv commented Nov 15, 2021

mssonicbld commented Nov 15, 2021

azure-pipelines bot commented Nov 15, 2021

vivekrnv commented Nov 15, 2021

mssonicbld commented Nov 15, 2021

azure-pipelines bot commented Nov 15, 2021

dgsudharsan commented Nov 15, 2021

vivekrnv Oct 27, 2021 •

edited

Loading

qiluo-msft Nov 13, 2021 •

edited

Loading

qiluo-msft Nov 13, 2021 •

edited

Loading

qiluo-msft Nov 13, 2021 •

edited

Loading