bpo-44317: Improve tokenizer errors with more informative locations #26555
Conversation
Parser/tokenizer.c
Outdated
| @@ -1089,7 +1083,14 @@ syntaxerror(struct tok_state *tok, const char *format, ...) | |||
| if (!errtext) { | |||
| goto error; | |||
| } | |||
| int offset = (int)PyUnicode_GET_LENGTH(errtext); | |||
|
|
|||
| if (col_offset == 0) { | |||
There was a problem hiding this comment.
What if error occurred at the beginning of the line?
There was a problem hiding this comment.
What if error occurred at the beginning of the line?
Oh, I was thinking here about line numbers. This should have been -1. Thanks for the catch
Parser/tokenizer.c
Outdated
| @@ -1552,6 +1585,7 @@ tok_get(struct tok_state *tok, const char **p_start, const char **p_end) | |||
| /* Number */ | |||
| if (isdigit(c)) { | |||
| if (c == '0') { | |||
| const char* number_start = tok->cur; | |||
There was a problem hiding this comment.
Cat tok->start be used instead of a new variable?
Parser/tokenizer.c
Outdated
| @@ -1606,6 +1640,8 @@ tok_get(struct tok_state *tok, const char **p_start, const char **p_end) | |||
| if (c != '0' && c != '1') { | |||
| tok_backup(tok, c); | |||
| if (isdigit(c)) { | |||
| // Move to the actual current token that is incorrect | |||
| tok_nextc(tok); | |||
There was a problem hiding this comment.
Same as above, tok_nextc() cancels tok_backup().
| @@ -1639,6 +1675,7 @@ tok_get(struct tok_state *tok, const char **p_start, const char **p_end) | |||
| } | |||
| c = tok_nextc(tok); | |||
| } | |||
| char* zeros_end = tok->cur; | |||
There was a problem hiding this comment.
Cat tok->start be used instead of a new variable?
There was a problem hiding this comment.
Not here because we want to highlight only the zeros and tok->cur points at the end of the number
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
|
This PR is stale because it has been open for 30 days with no activity. |
|
Thanks @pablogsal for the PR |
…ythonGH-26555) (cherry picked from commit f24777c) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
|
GH-27079 is a backport of this pull request to the 3.10 branch. |
https://bugs.python.org/issue44317