Fix ZScoreNorm processor bug #1398
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
waiting for triage
SunsetWolf
added
bug
github-actions
bot
added
the
waiting for triage
you-n-g
reviewed
Dec 25, 2022
you-n-g
reviewed
Dec 26, 2022
you-n-g
reviewed
Dec 29, 2022
qlib/data/dataset/processor.py
Outdated
| @@ -361,7 +359,8 @@ def __init__(self, fields_group=None): | |||
|
|
|||
| def __call__(self, df): | |||
| cols = get_group_columns(df, self.fields_group) | |||
| df[cols] = df[cols].groupby("datetime").apply(lambda x: x.fillna(x.mean())) | |||
| df.index.astype(np.datetime64) | |||
tests/test_processor.py
Outdated
| assert (df == ((origin_df - origin_df.mean()).div(origin_df.std()))).all().all() | ||
|
|
||
|
|
||
| def suite(): |
There was a problem hiding this comment.
Why is this function required?
There was a problem hiding this comment.
I think a simpler implementation should achieve similar effects
tests/test_processor.py
Outdated
|
|
||
| def test_CSZScoreNorm(self): | ||
| st = """ | ||
| 2000-01-01,1,2 |
There was a problem hiding this comment.
The data does not have a instruments column.
It does not align with the desired format.
you-n-g
reviewed
Dec 30, 2022
qlib/data/dataset/processor.py
Outdated
| @@ -361,7 +367,7 @@ def __init__(self, fields_group=None): | |||
|
|
|||
| def __call__(self, df): | |||
| cols = get_group_columns(df, self.fields_group) | |||
| df[cols] = df[cols].groupby("datetime").apply(lambda x: x.fillna(x.mean())) | |||
| df[cols] = df[cols].groupby("datetime", group_keys=False).apply(lambda x: x.fillna(df[cols].mean())) | |||
There was a problem hiding this comment.
I can't understand what is this for.
you-n-g
reviewed
Dec 30, 2022
| @@ -361,7 +368,7 @@ def __init__(self, fields_group=None): | |||
|
|
|||
| def __call__(self, df): | |||
| cols = get_group_columns(df, self.fields_group) | |||
| df[cols] = df[cols].groupby("datetime").apply(lambda x: x.fillna(x.mean())) | |||
| df[cols] = df[cols].groupby("datetime", group_keys=False).apply(lambda x: x.fillna(x.mean())) | |||
There was a problem hiding this comment.
, group_keys=False
Why is this necessary ?
tests/test_processor.py
Outdated
| min_val = np.nanmin(origin_df.values, axis=0) | ||
| max_val = np.nanmax(origin_df.values, axis=0) | ||
| origin_df.loc(axis=1)[origin_df.columns] = (origin_df.values - min_val) / (max_val - min_val) | ||
| assert (df.iloc[:, :-1] == origin_df.iloc[:, :-1]).all().all() |
There was a problem hiding this comment.
origin_df["test"] remains 0 is also a important check
This was referenced Dec 30, 2022
qianyun210603
pushed a commit
to qianyun210603/qlib
that referenced
this pull request
Mar 23, 2023
* fix_ZScoreNorm_bug * fix_CI_error * fix_CI_error * add_test_processor * fix_pylint_error * fix_some_error_and_optimize_code * modify_terrible_code * optimize_code * optimize_code
qianyun210603
pushed a commit
to qianyun210603/qlib
that referenced
this pull request
Mar 23, 2023
* fix_ZScoreNorm_bug * fix_CI_error * fix_CI_error * add_test_processor * fix_pylint_error * fix_some_error_and_optimize_code * modify_terrible_code * optimize_code * optimize_code
qianyun210603
pushed a commit
to qianyun210603/qlib
that referenced
this pull request
Mar 23, 2023
* fix_ZScoreNorm_bug * fix_CI_error * fix_CI_error * add_test_processor * fix_pylint_error * fix_some_error_and_optimize_code * modify_terrible_code * optimize_code * optimize_code
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Because the version of numba is limited to 0.52.0, some old code(e.g. np.long) in numba can not adapt to the new version of numpy, resulting in CI failure, so update numba to the latest version to solve the CI problem.