-
Notifications
You must be signed in to change notification settings - Fork 2.4k
-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fails to parse backslash on line by itself #4261
Comments
So I looked into it and the issue seems to lie here: The tokenizer tracks backslash-escaped newlines by setting the black/src/blib2to3/pgen2/tokenize.py Lines 494 to 559 in e4bfedb
That means that the input class Plotter:
\
pass generates these tokens:
whereas class Plotter:
pass generates these:
But the grammar always expects an indent after the newline: black/src/blib2to3/Grammar.txt Line 127 in e4bfedb
So add_token doesn't find any valid state and raises an error. black/src/blib2to3/pgen2/parse.py Lines 333 to 334 in f03ee11
I feel like it would make more sense to add special handling in the tokenizer than to change the grammar. Anyone have a suggestion? |
I agree this should probably be fixed in the tokenizer. I don't have strong feelings on what the fix should look like, but presumably we should try to match what the CPython tokenizer does. |
Fun fact, this was illegal in earlier versions of Python. I guess the new parser must have somehow made this allowed:
|
Oh, haha! My first thought was even "That doesn't look allowed" but then it just ran. I'll take a look at how this is handled in CPython and try to write a fix! |
can't we modify the add_token function to do so:
|
Feel free to submit a PR with test cases if your proposed change fixes this issue. |
@JelleZijlstra I have proposed a fix, but I feel it will require extensive testing as well. I've tried running it on a few open source projects and haven't seen any unexpected crashes etc. |
THANKS!!!🏋️♂️ |
Saw this in astral-sh/ruff#10099; the example is minified from the repro case in that issue.
The text was updated successfully, but these errors were encountered: