Named existentials and impl Trait variable declarations

[cramertj]

Add the ability to create named existential types and support impl Trait in let, const, and static declarations.

// existential types
existential type Adder: Fn(usize) -> usize;
fn adder(a: usize) -> Adder {
    |b| a + b
}

// existential type in associated type position:
struct MyType;
impl Iterator for MyType {
    existential type Item: Debug;
    fn next(&mut self) -> Option<Self::Item> {
        Some("Another item!")
    }
}

// `impl Trait` in `let`, `const`, and `static`:

const ADD_ONE: impl Fn(usize) -> usize = |x| x + 1;
static MAYBE_PRINT: Option<impl Fn(usize)> = Some(|x| println!("{}", x));
fn my_func() {
    let iter: impl Iterator<Item = i32> = (0..5).map(|x| x * 5);
    ...
}

Rendered

[aturon]

Thanks so much, @cramertj, for taking this on!

I wanted to follow up on two points that @nikomatsakis, @eddyb, @cramertj and I have been discussing but didn't make it in full detail in the RFC.

Going fully expressive

First, the question of pursuing this feature as a next step, rather than adding direction support for fn-level impl Trait in traits: as we explored the design space, it turns out that the design for this "more advanced" feature is actually far more straightforward than for the "simpler", more sugary version. In addition, there are some open questions around the sugary version that we could get insight into with more experience.

I do want to emphasize that I personally am uncomfortable with the situation where you can use impl Trait in fn signatures, unless you are defining or implementing a trait. I think we should strive to support that ASAP. But @cramertj has convinced me that it's wisest to start with this RFC as the first step.

Implementation concerns

The RFC strives to stay pretty high-level in terms of the specification, but it has some pretty significant implementation impact. In particular, the fact that a single type Foo = impl SomeTrait; definition may be determined jointly by multiple functions in a module implies that we're doing some amount of module-level type inference or checking. However, this is far less scary than it might sound at first.

The idea we've discussed involves doing type inference as usual for each function, while treating instances of an impl Trait type alias as an inference variable. In today's system, for type checking to succeed for a function, by the end of the process we must be able to fulfill all outstanding "obligations" (basically: things to verify about the types, most commonly checking for trait implementations). In this setup, though, we may not have enough information within a single function to know for sure that all obligations have been fulfilled, since we may not know the full identity of the type alias.

There are a number of options for how to proceed, falling on a spectrum. Here are the two extremes:

• At the most conservative, we could force the type variable we introduced to be fully resolved by the end of type checking. That would mean each function using the alias, by itself, must contain enough information to fully infer the concrete type for the alias. In this approach, we'd still be able to require that all obligations are fulfilled by the end of type checking a function. As a final step, we then ensure that all functions using the alias agree on the concrete type they nailed down.

• At the most liberal, we could fulfill as many obligations as possible when checking each function, and then store the remaining ones. Then, after checking all of the functions within a module, we would combine their remaining obligations and ensure that they can all be resolved. That is, in effect, module-level type inference, but done in a way that pushes as much locally as possible. The impact on incremental compilation is not entirely clear.

The RFC is deliberately leaving the precise resolution of these questions up in the air, since they are best resolved through implementation and experimentation.

I personally think we should start with the most conservative approach and go from there.

[Ixrec]

Is this : meant to be a =?

[cramertj]

Yes-- same for the other one. Thanks for catching that. I'll fix it as soon as I get to a computer.

[Ixrec]

Similarly, is this -> meant to be a =?

[scottmcm]

impl Trait in let, const, and static looks amazing 🎉

Can I cast (or type ascribe) impl Trait?

let displayable = "Hello, world!" as impl Display;

Can an impl trait alias be self-referential?

type Foo = impl Add<Foo, Output=Foo>;

The impl Trait type "alias" syntax somewhat scares me. While technically they do create synonyms for some type, they're very different in that manually substituting the right hand side in place of the type alias into signatures changes behaviour. I'm also not certain what type Foo<T> = impl Bar<T>; would (eventually) mean in the "module-level type inference" model. Would it infer a single concrete type constructor?

[eddyb]

Can an impl trait alias be self-referential?

In the current implementation, at least, yes, as the trait bounds are associated to, but distinct from, the type itself, i.e. it's something like this in the compiler:

type_of(Foo) = Anon0;
predicates_of(Anon0) = [Anon0: Add, <Anon0 as Add>::Output == Anon0];

[cramertj]

@scottmcm

Can I cast (or type ascribe) impl Trait?

This RFC wouldn't allow either of those. It's not totally obvious to me what either let x = "Hello world!" as impl Display; or let x = "Hello world!": impl Display; would mean. My gut reaction is that the type ascription (:) one should behave the same as an impl Trait let binding, which is basically a no-op except for providing some hints to type inference.

[cramertj]

@scottmcm

I'm also not certain what type Foo<T> = impl Bar<T>; would (eventually) mean in the "module-level type inference" model. Would it infer a single concrete type constructor?

Yes, Foo<T> must resolve to a single concrete type constructor:

trait MyTrait {}
type Foo<T> = impl MyTrait;

struct MyStruct<A, B, C> {
    a: A,
    b: B,
    c: C,
}
impl<A, B, C> MyTrait for MyStruct<A, B, C> {}

fn foo<T>(t: T) -> Foo<T> {
    // This tells the compiler that `for<T> Foo<T> == MyStruct<i32, T, &'static str>`
    MyStruct { a: 1i32, b: t, c: "" } 
}

[Ixrec]

My way-more-than-two cents on this RFC:

I am strongly in favor of the functionality being proposed here.

This makes it a lot easier to properly distinguish public APIs from implementation details in a way the compiler reliably enforces, makes it far more feasible to work with otherwise unnameable types like closures or messy "I really don't care what's in here" types like iterator combinators, and the syntax seems about as concise, obvious and ergonomic to me as it could possibly get.

---

This seems very relevant to the "publicly unnameable types" issue, but the RFC never mentions that.

I have no idea how widely known that issue is, or what everyone else is calling it these days, so I should probably explain what I mean by it:

First, backstory: The only objection I'm aware of to having impl Trait be in the language at all is that it makes unnameable types more common. In particular, it causes types which were nameable within a function or module to become unnameable outside that function or module. I'm calling those "publicly unnameable types" to distinguish them from "everywhere unnameable types" like closures (which this RFC does mention). Note that being able to name a type does not mean being able to rely on that type never changing. For instance, if your library has a function returning i32, and my code puts that i32 in one of my structs, today I need to be able to write "i32" as part of my struct's type definition. If you change that i32 to impl Debug, then I can no longer rely on you always returning i32 anymore (which IS a good thing), but I also can't put your i32 in one of my structs anymore because Rust doesn't provide a way to say "whatever type that function returns" (which is NOT a good thing).

Previously, I thought the only solution to this would be adding a typeof operator. Then I could write typeof(foo(x, y, z)) in my struct definition to tell the compiler I want whatever foo's concrete return type is. But in this RFC, every new usage of impl Trait being proposed comes with a name, which seems like a far better solution to that problem since it doesn't require the massive syntax bikeshed that typeof would (e.g., should I have even put "x, y, z" in that pseudocode just now?) and we automatically get all the trait bounds needed for type safety without any duplicate where clauses in my struct definition (I'm not actually sure if typeof would require extra annotations like that, but I assume it wouldn't be quite as trivial/ergonomic as Trait::FooType).

However, this RFC also alludes to a "sugar" for "impl Trait in traits", though it's never stated exactly what that is since it's not part of this proposal. I assume this hypothetical feature would mean making code like this:

trait Foo {
	fn foo() -> impl Debug;
}

be sugar for this:

trait Foo {
	type __FooSecretAssocType1__ = impl Debug;
	fn foo() -> __FooSecretAssocType1__;
}

If this sugar is added, then using it makes the type unnameable again. So there's an argument we should never actually add this sugar unless we also add something like typeof, or unless there's some reason why trait authors would need the ability to forbid clients from storing their return types in structs (is there one? I'm not aware of one).

Now what I actually wanted to say:

I think the RFC should address the publicly unnameable types issue. It should at least be an unresolved question, but if the author(s?) actually intended for this to make a typeof operator unnecessary, or less necessary, that should be made explicit. I have no strong opinion on whether we should in the long run add a typeof operator or rely on the proposed impl Trait type aliases or simply reject the idea that all returned types should be nameable in client code, but we shouldn't be committing to or ruling our any of those options by accident.

---

Readability of impl Trait type aliases

If we ignore interactions with other features for a moment, the only concern I have with this RFC in isolation is that it won't always be obvious what the concrete type of an associated impl type alias is intended to be. I'm fine with making the compiler do a limited form of module-level type inference (assuming the compiler team is confident it won't cause any problems), but there's a risk of also requiring every human who reads the code to do module-level type inference in their heads. If I were to start using this feature as currently proposed, I'd probably add the intended concrete type in a comment every time I wrote such an alias.

pub type Foo = impl Debug; // i32

It's hard to come up with a counter-proposal though. Adding the concrete type directly to the type alias statement feels bad because it breaks the principle that the signatures of a module's public items contain exactly what client code needs to know and no more (that is a thing in Rust, right? I'm not making that up?), and as cramertj explained "casting" or "ascribing" syntaxes would be pretty confusing here, but if we put the concrete type anywhere else that's not much of an improvement on requiring it to be explicit in every method's return values.

So at the moment, I think aturon's suggestion that the conservative implementation would be "each function using the alias, by itself, must contain enough information to fully infer the concrete type for the alias" seems like the best solution to this concern, since the human would probably only need to look at the first method after the associated impl type alias to figure out what the concrete type is. Consider this a vote in favor of "we should start with the most conservative approach and go from there".

[Ericson2314]

What about full abstract type: bound = concrete;? #1951 assumes all the "rigorous phase" stuff would happen eventually, but IMO this is still beating around the bush with the impl shorthand.

[cramertj]

@Ericson2314 I'm not sure I understand your proposal. With abstract <type>: <bound> = <concrete>;, you'd still have to specify the concrete type, which breaks the "unnameable types" use case. If you allow users to just write abstract <type>: <bound>; and have the concrete type inferred, that seems to be the same feature as type <type> = impl <bound>;.

Is your goal just to use a different syntax?

[dlight]

The type Item = impl Debug syntax is confusing, at least in the top level, because it isn't just a type alias: it also introduces a type equality constraint ensuring all uses of Item are the same concrete type. However, a top level type alias normally means that if you substituted it by its definition, the program would continue to work (for example, if you do type A = u32; then you can substitute all uses of A by u32).

So, I think that this should have another syntax, or at least this should be noted in the "drawbacks" section.

[eddyb]

@dlight That interpretation is incorrect though. It's not a special syntax. Also, syntactical substitution is not guaranteed in Rust, and semantically, an impl Trait syntactical node has an identity (i.e. it's a declaration like struct Foo;) which it refers back to whenever mentioned.

The RFC probably needs more examples such as type Sequence = Vec<impl Element>;, or pairs, etc. to show that each impl Trait is independent from the type alias it happens to be declared in.

The only reason to write it as type Foo = impl Trait; most of the time is for interoperability but IMO typeof is better suited and easier to "implement" (it got left in the AST a while back, even if it's a syntax error or something, and most of the compiler ended up doing the right thing for it) - just needs an RFC now.

[dlight]

In my mental model a type alias is just syntax sugar (not an associated type; just a top-level alias). Replacing aliases by their definition should never change whether a program typechecks. Could you point out some stable Rust code where my intuition is incorrect?

Anyway, even if it's incorrect, people may still be misled by it.

[eddyb]

@dlight Path resolution is the most obvious one, since it's done in the scope of the definition.
Sadly "type blocks" isn't a thing otherwise type Foo = Vec<{ struct FooImpl; FooImpl }>; would be a decent example. You can put literally anything in the length of an array, e.g.:

type Bar = [(); {
    // Any top level item, including nested modules.
    mod foo {
        // There is exactly one instance of this module.
        pub struct Foo;
        impl Foo {
            pub const X: usize = 123;
        }
    }
    foo::Foo::X
}];

A more meaningful example, although not available on stable yet:

type Baz = [u8; { struct Quux(u8, u16); std::mem::size_of::<Quux>() }];

If we had randomized field reordering, that one one would be guaranteed to always be the same type (i.e. its length would always be evaluated for the same struct Quux definition).

Of course these would make more sense with const generics, with the usize array length not being the only value you can have in any type, anymore.

---

Alright, so I don't have a perfect example. Still, a type alias has the same semantics as an associated type, once the latter has been resolved through the trait system, and they will likely grow even closer together. Having a syntactic alias would be limiting and wasteful.

---

There's another angle, I suppose - we can show that type alias expansion cannot be syntactic, given a definition such as this, that duplicates a type parameter:

type Double<T> = (T, T);

Lifetime elision behaves independently of the expansion of Double (playpen):

fn elision_alias((x, _): Double<&str>) -> &str { x }

// error: "this function's return type contains a borrowed value, but the signature
//         does not say which one of `(x, _)`'s 2 lifetimes it is borrowed from"
fn elision_syntax((x, _): (&str, &str)) -> &str { x }

impl Trait (although not stable) has an identity that's not duplicated (playpen):

fn impl_trait_alias() -> Double<impl ToString> {
    (String::new(), Default::default())
}

// error: "type annotations needed"
fn impl_trait_syntax() -> (impl ToString, impl ToString) {
    //                                    ^^^^^^^^^^^^^ cannot infer type for `_`
    (String::new(), Default::default())
}

[dlight]

@eddyb

Thanks. Since type aliases aren't just syntactical already, it makes less sense to add different syntax just for being able to name a impl Trait. Also it seems cleaner / less ad-hoc than typeof (since you can give a meaningful name to the impl Trait, instead of referring to it indirectly).

I'm unable to find some kind of documentation for this subtle semantics around type aliases and their expansion. I looked on the first book, the second book, and the reference, but perhaps I should look at more advanced stuff (which I'm not sure exists yet?).

[eddyb]

The issue is that there is no formal specification, otherwise I could link to that.
I would argue subtly surprising semantics would arise from a syntactic expansion (e.g. unhygienic macros), whereas the semantic expansion interprets the type where it is defined, just like field types in a struct definition, argument types in a function, etc.

There is a similar situation with const items: they are evaluated once (at the definition) and the value is (byte-wise) copied everywhere they're used, whereas a macro is far less rigorous, could expand to code with side-effects, etc.

Really, only macros should involve syntactic expansion and that's why they are invoked with a bang, i.e. ! after the macro name.

[dlight]

Perhaps this RFC should mention this sketch of explicit existentials from RFC 1951 in the "Alternatives" section. I'm not sure if its semantics is a subset of this RFC (it looks like it is).

[petrochenkov]

impl Trait in const and static

A logical extension of impl Trait in functions, and quite reasonable one. 👍

Issue: The type of impl Trait in consts and statics is "revealed" in the current module, but the type of impl Trait in functions is not. They should behave identically, preferably "not revealed".

impl Trait in let

Kinda reasonable, by analogy with const/static, but I don't see enough motivation.
let bindings are usable only in the current module, if the type of impl Trait in let is revealed in the current module, like the RFC suggests, then it's always revealed, so what's the point.
Given that it also requires special rules for lifetimes to be usable, diverging from rules for other impl Trait, I'm not sure it worths it.

impl Trait in type aliases

type Alias = impl Trait; is a very counter-intuitive notation.
Even if I already know that the impl Trait has "identity" internally, it's still counter-intuitive.
Yes, after reading the @eddyb's long explanation it may look reasonable, but I'm not sure every person encountering this feature for the first time will go and read and understand it.
Can something similar to

type Alias: Trait; // No initializer

be used instead?

I'm also unhappy about module-level inference, as a human reader in particular.
Can a single use of Alias be marked as "canonical", so all the other uses could be forced to match it?

type Alias: Trait;

fn f1() -> marked_as_canonical Alias { ... }
fn f2() -> Alias { ... }

And yeah, this example reminds me about typeof too, and can be reworded into

fn f1() -> impl Trait { ... }

type Alias = typeof(f1()); // f1 is "canonical"

fn f2() -> Alias { ... }

Looks like not revealing the underlying type of Alias in the current module is impractical by definition, so it has to differ from impl Trait in functions in this respect, but I haven't looked into this deeper.

[cramertj]

@petrochenkov

Issue: The type of impl Trait in consts and statics is "revealed" in the current module, but the type of impl Trait in functions is not. They should behave identically, preferably "not revealed".

My motivation for making consts and statics "leak" was to prevent users from having to sprinkle complex bounds in order to make unnameably-typed constants usable. However, I'm not sure how often this would be a problem in practice, and you could always work around it by using type Foo = impl Trait; const X: Foo = ...;. I'm definitely willing to be persuaded here, though I think my personal preference still tends towards allowing them to leak. Can you say more about why you'd prefer them to be private?

impl Trait in let...I'm not sure it worths it.

I think that it would be confusing to new users if we allow impl Trait in const and static but not let. It also just feels "wrong" to me from a more philosophical perspective.

WRT type alias syntax: there have been a lot of ideas floated in both this thread and others, so i'll try to briefly outline what I see as some of the main advantages and disadvantages of each proposed syntax:

• abstype / abstract type Foo: I'm not super fond of the word "abstract" here. It doesn't seem like it adds much from a user's perspective (what does it mean for a type to be "abstract"?). However, it's a distinct keyword which makes it recognizable and easy to search for on the internet. I personally place a pretty high value on feature Google-ability, as it makes it much easier to discover and learn.
• type Foo = impl Trait;: this syntax seems straightforward and resembles existing type alias syntax. It's an obvious transition from fn foo() -> impl Trait { ... } to type Foo = impl Trait; fn foo() -> Foo { ... }.
• type Foo: Trait;: I like that this separates the identity/assignment of the type from its declaration, which makes it clear that the type is being inferred. It could also allow things like type Foo: Trait = MyStruct; which would allow users to keep the module-based abstraction boundary while still explicitly stating the concrete type. However, this could be "too much power"/too complex, and overall this syntax seems hard to Google for as it doesn't contain any impl Trait / abstype-esque special keyword.

I also gave some consideration to type Foo: impl Trait, which has the same advantages as type Foo: Trait while retaining Google-ability and a clear relationship to the impl Trait feature. However, this opens up all sorts of questions around "what the heck is type Foo: MyType = MyType;?" or similar.

Overall, my preference is towards type Foo = impl Trait;. I feel like it's the most natural syntax and will be the easiest for new users to recognize and make use of.

Can a single use of Alias be marked as "canonical", so all the other uses could be forced to match it?

This seems to me like an unnecessary complication. It's relatively easy for the compiler to determine if a function contains enough information to infer the concrete type, so I'd prefer to save users the extra work.

[eddyb]

FWIW eliding the type of a const/static should always be possible (#2010), except for i32 default for integer literals being unpopular (so it might be turned off if we accept that RFC), and for static items being potentially recursive (e.g. a circular linked list). Both cases can just use explicit types.

IMO that is a much better approach than impl Trait unless you want to limit the API surface of the value being placed in the global and/or if the type is a private implementation detail, e.g.:

struct MyAlloc {...}
pub static MY_ALLOC: impl Allocator = MyAlloc {...};

[glaebhoerl]

Would I be allowed to do things like:

type Foo = (impl Bar, impl Baz);

or

type IterDisplay = impl Iterator<Item=impl Display>;

?

If yes, that's a significant difference relative to the other two syntaxes, where you'd have to introduce a separate abstype (or whichever) by hand for each internal "impl Trait node" in the above definitions (corresponding to the fact that impl Trait is "side-effecting" and introduces new hidden items to the top-level scope, unlike plain type aliases, as discussed). For that matter: I can't remember whether or not the accepted impl Trait RFC for functions allows nested use like above; I assume that the same rule, whatever it may be, would apply to all of the positions where impl Trait is legal.

[eddyb]

@glaebhoerl Those are allowed and I've previously mentioned the RFC should be more explicit on it.
They have always been part of my implementations, including nice things such as:

fn parse_csv<'a>(s: &'a str) -> impl Iterator<Item = impl Iterator<Item = &'a str>> {
    s.split('\n').map(|line| line.split(','))
}

[petrochenkov]

@cramertj

My motivation for making consts and statics "leak" was to prevent users from having to sprinkle complex bounds in order to make unnameably-typed constants usable. However, I'm not sure how often this would be a problem in practice, and you could always work around it by using type Foo = impl Trait; const X: Foo = ...;. I'm definitely willing to be persuaded here, though I think my personal preference still tends towards allowing them to leak. Can you say more about why you'd prefer them to be private?

What makes impl Trait in functions different then? Why don't they suffer from the "complex bounds" problem? Constants are basically pure functions with no parameters.
I prefer the "not revealed" variant because it's consistent with functions and more conservative, if experience shows that it's impractical and causes boilerplate, it could be relaxed for both functions and constants.

[Boscop]

I often have the situation that I want to write extension traits, e.g.:

I wish it would just work like that..

[Boscop]

Hm, it doesn't even work with existential type:

error[E0700]: hidden type for `impl Trait` captures lifetime that does not appear in bounds
  --> src/main.rs:13:2
   |
13 |     existential type R: Iterator<Item = (A, &'a mut T)> /*+ 'a*/;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
note: hidden type `std::iter::FilterMap<I, [closure@src/main.rs:15:19: 15:54]>` captures the lifetime 'a as defined on the impl at 11:6
  --> src/main.rs:11:6
   |
11 | impl<'a, A, T: 'a, I: Iterator<Item = (A, &'a mut Option<T>)> /*+ 'a*/> FilterMap2<'a, A, T> for I {
   |      ^^

error: aborting due to previous error

https://play.rust-lang.org/?gist=b13f589b6f8e7813a689e98319bed0f5&version=nightly&mode=debug&edition=2015

So what's the right way to do this? :)

[earthengine]

@cramertj

It's this bit that I think is an antipattern.

Can you please give a concrete example to show this is not good? IMO removing the ability to name the type and making constraint on it is not a bad thing.

It clearly tells the code reader: "Here is a type, you should only use its Trait ability, no to assume anything else about it", this relaxes the code flexibility - the code writer can adjust the return type freely, without the fear of breaking backward compatibility.

[eddyb]

@Boscop Huh, I would think that would work, because 'a shows up in the Iterator bound.
But even with Captures<'a> it doesn't work. cc @oli-obk

(also, shouldn't this be posted on the tracking issue?)