Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove TokenStream::JointTree. #56964

Merged
merged 1 commit into from
Dec 24, 2018
Merged

Conversation

nnethercote
Copy link
Contributor

This is done by adding a new IsJoint field to TokenStream::Tree,
which simplifies a lot of match statements. And likewise for
CursorKind.

The commit also adds a new method TokenTree:stream() which can replace
a choice between .into() and .joint().

@rust-highfive
Copy link
Collaborator

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 18, 2018
@nnethercote
Copy link
Contributor Author

After these commits, TokenStream looks like this:

pub enum TokenStream {
    Empty,
    Tree(TokenTree, IsJoint),
    Stream(Lrc<Vec<TokenStream>>),
}

I wonder if the recursion via the Stream variant is necessary. Perhaps it could instead be this:

    Stream(Lrc<Vec<TokenTree>>),

To break it down further, perhaps TokenStream is conceptually just this:

Vec<(TokenTree, IsJoint)>

(Possibly with the use of Lrc and Empty/singleton variants for performance
reasons.)

There's a comment at the top of tokenstream.rs that suggests this is true:

TokenStreams represent syntactic objects before they are converted into
ASTs. A TokenStream is, roughly speaking, a sequence (eg stream) of
TokenTrees, which are themselves a single Token or a Delimited
subsequence of tokens.

@petrochenkov
Copy link
Contributor

petrochenkov commented Dec 18, 2018

perhaps TokenStream is conceptually just this: Vec<(TokenTree, IsJoint)>

More that that, IsJoint is not even a property of TokenTree, but only punctuation tokens, so TokenStream is conceptually just Vec<TokenTree>.
See stable public interfaces in proc_macro for what "canonical" token streams look like in terms of interface.
In general, it would be nice to reuse types from proc_macro in the compiler if possible, or at least align with them.

@nnethercote
Copy link
Contributor Author

@petrochenkov: I tried looking at those public interfaces but I'm having trouble understanding them, particularly the bridge and define_handles! stuff. Can you give me some hints? Thanks.

@petrochenkov
Copy link
Contributor

Oh, there's the bridge thing now.
Then my first hint would be to look at the snapshot of libproc_macro preceding merge of #49219.

@petrochenkov
Copy link
Contributor

(libproc_macro was much more straightforward and closer to its "conceptual" form before all that machinery for client-server communication was added.)

@eddyb
Copy link
Member

eddyb commented Dec 18, 2018

You can still look at the public API, IMO, it's not like the old code was self-contained.

Group, for example, used syntax::tokenstream::TokenStream before (back then wrapped by proc_macro::TokenStream), and it still uses that today, albeit the actual definition of Group now lives in syntax_ext::proc_macro_server (as does the rest of the compiler side of the implementation).

src/libsyntax/tokenstream.rs Outdated Show resolved Hide resolved
@petrochenkov
Copy link
Contributor

r=me with #56964 (comment) addressed.
@bors delegate+

@bors
Copy link
Contributor

bors commented Dec 19, 2018

✌️ @nnethercote can now approve this pull request

@petrochenkov petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 19, 2018
This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
@nnethercote
Copy link
Contributor Author

@bors r=petrochenkov

@bors
Copy link
Contributor

bors commented Dec 19, 2018

📌 Commit e7c5146 has been approved by petrochenkov

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Dec 19, 2018
@nnethercote
Copy link
Contributor Author

perhaps TokenStream is conceptually just this: Vec<(TokenTree, IsJoint)>

More that that, IsJoint is not even a property of TokenTree, but only punctuation tokens, so TokenStream is conceptually just Vec<TokenTree>.

Perhaps it's more like this: (Vec<TokenTree>, IsJoint). The IsJoint applies only to the final TokenTree, and should be used if the Vec gets extended. (All the TokenTrees prior to the last one should have already been joined if necessary.)

At least, that's the case for any TokenStream that may still be extended. A TokenStream that is known to be complete can drop the IsJoint.

pietroalbini added a commit to pietroalbini/rust that referenced this pull request Dec 21, 2018
…petrochenkov

Remove `TokenStream::JointTree`.

This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
bors added a commit that referenced this pull request Dec 21, 2018
Rollup of 20 pull requests

Successful merges:

 - #56802 (Add DoubleEndedIterator::nth_back)
 - #56842 (Add unstable VecDeque::rotate_{left|right})
 - #56869 (Reduce search-index.js size)
 - #56887 (Disable field reordering for repr(int).)
 - #56892 (rustc: Update Clang used to build LLVM on Linux)
 - #56909 (static eval: Do not ICE on layout size overflow)
 - #56914 (Ignore ui/target-feature-gate on sparc, sparc64, powerpc, powerpc64 and powerpc64le)
 - #56917 (Simplify MIR generation for logical operations)
 - #56919 (Remove a wrong multiplier on relocation offset computation)
 - #56933 (Add --progress to git submodule commands in x.py)
 - #56941 (deny intra-doc link resolution failures in libstd)
 - #56964 (Remove `TokenStream::JointTree`.)
 - #56970 (Mem uninit doc ptr drop)
 - #56973 (make basic CTFE tracing available on release builds)
 - #56979 (Adding unwinding support for x86_64_fortanix_unknown_sgx target.)
 - #56981 (miri: allocation is infallible)
 - #56984 (A few tweaks to dropck_outlives)
 - #56989 (Fix compiletest `trim` deprecation warnings)
 - #56992 (suggest similar lint names for unknown lints)
 - #57002 (Stabilize Vec(Deque)::resize_with)

Failed merges:

r? @ghost
bors added a commit that referenced this pull request Dec 21, 2018
Rollup of 20 pull requests

Successful merges:

 - #56802 (Add DoubleEndedIterator::nth_back)
 - #56842 (Add unstable VecDeque::rotate_{left|right})
 - #56869 (Reduce search-index.js size)
 - #56887 (Disable field reordering for repr(int).)
 - #56892 (rustc: Update Clang used to build LLVM on Linux)
 - #56909 (static eval: Do not ICE on layout size overflow)
 - #56914 (Ignore ui/target-feature-gate on sparc, sparc64, powerpc, powerpc64 and powerpc64le)
 - #56917 (Simplify MIR generation for logical operations)
 - #56919 (Remove a wrong multiplier on relocation offset computation)
 - #56933 (Add --progress to git submodule commands in x.py)
 - #56941 (deny intra-doc link resolution failures in libstd)
 - #56964 (Remove `TokenStream::JointTree`.)
 - #56970 (Mem uninit doc ptr drop)
 - #56973 (make basic CTFE tracing available on release builds)
 - #56979 (Adding unwinding support for x86_64_fortanix_unknown_sgx target.)
 - #56981 (miri: allocation is infallible)
 - #56984 (A few tweaks to dropck_outlives)
 - #56989 (Fix compiletest `trim` deprecation warnings)
 - #56992 (suggest similar lint names for unknown lints)
 - #57002 (Stabilize Vec(Deque)::resize_with)

Failed merges:

r? @ghost
Centril added a commit to Centril/rust that referenced this pull request Dec 22, 2018
…petrochenkov

Remove `TokenStream::JointTree`.

This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
Centril added a commit to Centril/rust that referenced this pull request Dec 23, 2018
…petrochenkov

Remove `TokenStream::JointTree`.

This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
bors added a commit that referenced this pull request Dec 23, 2018
Rollup of 10 pull requests

Successful merges:

 - #56188 (enum type instead of variant suggestion unification )
 - #56916 (Fix mutable references in `static mut`)
 - #56917 (Simplify MIR generation for logical operations)
 - #56953 (Mark tuple structs as live if their constructors are used)
 - #56964 (Remove `TokenStream::JointTree`.)
 - #56966 (Correct strings for raw pointer deref and array access suggestions)
 - #56999 (AST/HIR: Introduce `ExprKind::Err` for better error recovery in the front-end)
 - #57020 (Point to cause of `fn` expected return type)
 - #57032 (fix deprecation warnings in liballoc benches)
 - #57053 (Fix alignment for array indexing)

Failed merges:

r? @ghost
Centril added a commit to Centril/rust that referenced this pull request Dec 23, 2018
…petrochenkov

Remove `TokenStream::JointTree`.

This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
Centril added a commit to Centril/rust that referenced this pull request Dec 23, 2018
…petrochenkov

Remove `TokenStream::JointTree`.

This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.

The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
bors added a commit that referenced this pull request Dec 24, 2018
Rollup of 14 pull requests

Successful merges:

 - #56188 (enum type instead of variant suggestion unification )
 - #56342 (Improve docs for collecting into `Option`s)
 - #56916 (Fix mutable references in `static mut`)
 - #56917 (Simplify MIR generation for logical operations)
 - #56939 (Pin stabilization)
 - #56953 (Mark tuple structs as live if their constructors are used)
 - #56964 (Remove `TokenStream::JointTree`.)
 - #56966 (Correct strings for raw pointer deref and array access suggestions)
 - #57020 (Point to cause of `fn` expected return type)
 - #57032 (fix deprecation warnings in liballoc benches)
 - #57053 (Fix alignment for array indexing)
 - #57062 (Fix a comment)
 - #57067 (Stabilize min_const_unsafe_fn in 1.33)
 - #57078 (Ignore two tests on s390x)

Failed merges:

r? @ghost
@bors bors merged commit e7c5146 into rust-lang:master Dec 24, 2018
@nnethercote nnethercote deleted the TokenStream-IsJoint branch April 2, 2020 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants