Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(utils): evaluate date parser multiple holiday results correctly #23685

Merged
merged 5 commits into from
Apr 26, 2023

Conversation

arkid15r
Copy link
Contributor

@arkid15r arkid15r commented Apr 13, 2023

SUMMARY

The purpose of this PR is fixing incorrect datetime_eval for holiday results containing more than one entry (even with observed=False flag set):

  • Bump holidays version range from ">=0.17.2, <0.18" to ">=0.23, <0.24"
  • Change holiday search result evaluation logic for multiple date cases
  • Change holiday lookup from icontains (default) to istartswith.
  • Add tests for the cases

TESTING INSTRUCTIONS

The added unit tests would fail with the currently used approach.

The described cases include:

  • multiple holidays per year (happens for Islamic calendar countries) e.g., "Eid al-Fitr" holidays for Saudi Arabia (JAN and DEC of 2000)
  • multiple holidays per lookup name, e.g., "Independence Day" for US years after 2021 (Independence Day, Juneteenth National Independence Day)

ADDITIONAL INFORMATION

  • Has associated issue: no issue, small PR
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

  - Bump holidays version family from ">=0.17.2, <0.18" to ">=0.22, <0.23"
  - Change holiday search result evaluation logic for multiple date cases
  - Change holiday lookup from `icontains` (default) to `istartswith`.
  - Add tests for described cases
@arkid15r arkid15r changed the title fix (utils/date_parser): evaluate multiple holiday results correctly fix(utils/date_parser): evaluate multiple holiday results correctly Apr 13, 2023
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats on making your first PR and thank you for contributing to Superset! 🎉 ❤️

We hope to see you in our Slack community too! Not signed up? Use our Slack App to self-register.

@arkid15r arkid15r changed the title fix(utils/date_parser): evaluate multiple holiday results correctly fix(utils): evaluate date parser multiple holiday results correctly Apr 18, 2023
@EugeneTorap
Copy link
Contributor

EugeneTorap commented Apr 21, 2023

Hello @arkid15r! Thanks so much for this PR!
Can you use latest 0.23 version and also update the lib in requirements/base.txt?
@villebro @dpgaspar Pls, take a look at the PR.

@arkid15r
Copy link
Contributor Author

Can you use latest 0.23 version

Hi Eugene,
that's absolutely doable. I used the current version range based on my understanding of consistency with other install_requires entries.

and also update the lib in requirements/base.txt?

I've done a couple of half-hearted attempts to update the requirements. It seems it hasn't been done for a while. Or maybe for some other reason a lot of dependencies are getting their version bumped when I run pip-compile-multi -P holidays.

git diff --stat
 requirements/base.txt        |  29 +++++++++++++++++-
 requirements/development.txt |   4 ---
 requirements/docker.txt      | 326 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------
 requirements/integration.txt |   8 ++---

I'd appreciate some guidance on how to properly resolve this sort of issues, e.g.:

Package greenlet was resolved to different versions in different environments: 1.1.3.post0 and 2.0.2
Traceback (most recent call last):
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/cli_v1.py", line 26, in cli
    recompile()
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/actions.py", line 31, in recompile
    compile_topologically(env_confs, deduplicator)
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/actions.py", line 38, in compile_topologically
    if env.maybe_create_lockfile():
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/environment.py", line 51, in maybe_create_lockfile
    self.create_lockfile()
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/environment.py", line 72, in create_lockfile
    self.fix_lockfile()
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/environment.py", line 134, in fix_lockfile
    sections = [
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/environment.py", line 135, in <listcomp>
    self.fix_pin(section)
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/environment.py", line 211, in fix_pin
    raise RuntimeError(
RuntimeError: Please add constraints for the package version listed above

@EugeneTorap, I appreciate you jumping in and tagging the right people as I was within a whisker of creating a post on Slack's #contributing channel asking for help.

@villebro
Copy link
Member

Changes LGTM, but the deps will need to be fixed. @arkid15r do you need help updating the requirements?

@arkid15r
Copy link
Contributor Author

Changes LGTM, but the deps will need to be fixed. @arkid15r do you need help updating the requirements?

Right, I do. The instructions are clear and easy to follow. The output is what bothers me -- a lot of unrelated [to this PR] changes.

@villebro
Copy link
Member

Changes LGTM, but the deps will need to be fixed. @arkid15r do you need help updating the requirements?

Right, I do. The instructions are clear and easy to follow. The output is what bothers me -- a lot of unrelated [to this PR] changes.

@arkid15r I wrote before reading your thorough post properly, sorry for asking what you already had posted. I'm looking into this, but my env seems to be slightly messed up right now, and pip-compile-multi is misbehaving for me. I'll look at this more over the weekend, ok!

@EugeneTorap
Copy link
Contributor

EugeneTorap commented Apr 21, 2023

@arkid15r try to run:

pip-compile-multi --no-upgrade

And commit only base.txt file

@pull-request-size pull-request-size bot added size/M and removed size/S labels Apr 21, 2023
@arkid15r
Copy link
Contributor Author

@arkid15r try to run:

pip-compile-multi --no-upgrade

And commit only base.txt file

It worked, I guess the yesterday's requirements update might have helped. The workflows need to be approved/run one more time.

Thank you!

@codecov
Copy link

codecov bot commented Apr 22, 2023

Codecov Report

Merging #23685 (1e1bc02) into master (b734a0f) will increase coverage by 0.10%.
The diff coverage is 80.49%.

❗ Current head 1e1bc02 differs from pull request most recent head 2e8e9b8. Consider uploading reports for the commit 2e8e9b8 to get more accurate results

@@            Coverage Diff             @@
##           master   #23685      +/-   ##
==========================================
+ Coverage   67.87%   67.98%   +0.10%     
==========================================
  Files        1925     1936      +11     
  Lines       74389    74914     +525     
  Branches     8108     8140      +32     
==========================================
+ Hits        50494    50928     +434     
- Misses      21818    21894      +76     
- Partials     2077     2092      +15     
Flag Coverage Δ
hive 53.00% <62.22%> (+0.07%) ⬆️
mysql 78.79% <83.20%> (+0.05%) ⬆️
postgres 78.87% <83.20%> (+0.05%) ⬆️
presto 52.91% <62.22%> (+0.07%) ⬆️
python 82.66% <83.70%> (+0.01%) ⬆️
sqlite 77.38% <83.20%> (+0.06%) ⬆️
unit 52.80% <62.96%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...chart-echarts/src/Timeseries/EchartsTimeseries.tsx 0.00% <0.00%> (ø)
...uperset/utils/dashboard_filter_scopes_converter.py 15.78% <0.00%> (ø)
superset/views/sql_lab/views.py 63.81% <ø> (ø)
superset/models/sql_lab.py 78.40% <42.85%> (-1.11%) ⬇️
superset-frontend/src/views/routes.tsx 51.06% <50.00%> (-0.05%) ⬇️
superset/db_engine_specs/ocient.py 57.28% <57.28%> (ø)
superset/dao/base.py 92.85% <70.00%> (-2.60%) ⬇️
...-frontend/src/pages/RowLevelSecurityList/index.tsx 72.54% <72.54%> (ø)
...rontend/src/features/rls/RowLevelSecurityModal.tsx 73.39% <73.39%> (ø)
...uperset/row_level_security/commands/bulk_delete.py 87.50% <87.50%> (ø)
... and 13 more

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@EugeneTorap
Copy link
Contributor

@arkid15r Pls, fix the mypy issue in pre-commit CI

Comment on lines +279 to +290
result = datetime_eval(
"holiday('Juneteenth', datetime('2022-01-01T00:00:00'), 'US')"
)
expected = datetime(2022, 6, 19, 0, 0, 0)
assert result == expected

result = datetime_eval(
"holiday('Independence Day', datetime('2022-01-01T00:00:00'), 'US')"
)
expected = datetime(2022, 7, 4, 0, 0, 0)
assert result == expected

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi there, the US holiday case was added to L273 so the duplicated test case is not necessary. Thanks for supporting more widely holiday types!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @zhaoyongjie,

I read L273 as UK's (technically GB) test. And even if it was the US test I'd suggest to keep the L279-289 Juneteenth related tests as they specifically cover the wrong parser behaviour this PR trying to address.

Thanks for reviewing this!

@arkid15r
Copy link
Contributor Author

@arkid15r Pls, fix the mypy issue in pre-commit CI

👍 I reverted the local mypy fixes, hope this fixes the issue.

@villebro villebro merged commit ff12654 into apache:master Apr 26, 2023
@arkid15r arkid15r deleted the fix-date-eval-parser branch April 26, 2023 13:59
sebastianliebscher pushed a commit to sebastianliebscher/superset that referenced this pull request Apr 28, 2023
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.0.0 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/M 🚢 3.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants