Dyno: add a type to represent iterators, implement loop expressions (#…

…25994)

This PR adds a new type sub-hierarchy for iterable things. It creates
some scaffolding for promoted expressions, though it does not currently
implement them. It also matches production's implementation of
iterators, which, as we have discussed with @bradcray, @mppf and
@benharsh may not be the direction we want to take in the long term.

## What?
This PR adds a new `IteratorType` type, as well as subclasses for it for
various situations that produce iterables (promotion, loop expressions,
calls to `iter` procedures). This way, we can unify the logic for
resolving iterators on records/classes and "iterables", invoking a
`these` method. The `these` methods for user records/classes are, as
before, provided explicitly by the user; the `these` methods for
iterators are compiled-provided (production generates them, in Dyno we
simply have special logic to mimic their resolution).

The three subclasses of `IteratorType` are as follows:
* `FnIteratorType`: the class of iterators created from invocations to
`iter` procedures. Just like in production, this type contains a
reference to the function that created it. When the iterator is
traversed (using `these`), the compiler resolves a call to a function
with the same name as the original `iter` procedures, and with the same
formal types. This matches the production compiler's implementation,
down to the ability to provide additional overloads for serial/parallel
iterators outside of where they are originally defined (for better or
for worse).
* `LoopExprIteratorType`: the class of iterators created from loop
expressions. In production, there are some tricky cases that make the
"yield type" of a loop expression be fluid, since it can depend on the
instantiation of the follower iterator it uses, which can change
depending on the leader. Thus, it's possible that in `forall i in
(loopExpr)` and `forall (_, i) in zip(.., loopExpr)`, the index variable
`i` is a different type. To make this work, we could need to capture the
outer variables of the loop expression and re-resolve it with different
leaders. This seems undesirable as per discussions with Brad et. al.,
and as a result, I instead make the assumption that loop expressions
always yield the same type, and that they do not re-resolve their
leaders and followers. There's some future work to make this more
principled.
* `PromotionIteratorType`: the class of iterators created from promoted
expressions. Since promotion is not implemented, this is just left as a
scaffold.

This PR removes some speculative resolution logic originally introduced
by @dlongnecke-cray in #24915.
Specifically, calls to iterators like `foo()` are no longer re-resolved
with tags if original resolution fails. This has several motivators:

1. Production doesn't support this, and instead requires a serial
iterator to be present for `foo()` to successfully invoke
`foo(standalone)` etc.
2. In David's PR, the re-resolution only happens at the site of `zip`s /
loop iterands. However, if we truly wanted to implement this feature, we
would need to handle invocations of iterators in arbitrary contexts
(e.g., as sub-expressions of promoted expressions), and thus would
likely need to hoist the "insert tags if needed" logic to the more
general call resolution world.
3. The need to speculatively resolve an expression before inserting tags
(and re-emit errors later if needed) made it difficult to resolve arrays
types early (instead of resolving them as bracket loops).
* Resolving array types late no longer works under this PR, but I think
that's fine. Previously, we would resolve a call to `_domain.these` with
a generic receiver (normally not allowed), and fail to infer its return
type despite succeeding in call resolution. This PR requires knowing the
return type (to make use of the fact that it's iterable), so this no
longer flies. Anyway, relying on resolving call with an accidentally
generic receiver seems unfortunate.
* Resolving array types early also means we avoid unnecessary
invocations of 'these' which is a nice bonus.

## Future Work
Following some discussion, we'd like to change the following properties
of the loop expressions (at least):

1. The return type may not depend on the type of the leader, or on
whether the serial or parallel overload was called. This way, loop
expressions _will_ always have a consistent yield type, no matter the
context in which they are used.
2. Implementing (1) would still allow a different follower iterator to
be resolved depending on the leader type (it would just need to return a
value of the right type). We would like to rule out this "context
dependence", and make it so the same follower function is invoked for
any given iterator expression.
* This would likely require eagerly resolving the leader/follower of the
iterator in the place where it is found, and then re-using these resolve
calls whereever needed.
* Loop expressons that need one follower type but are used with a leader
type yielding other things will be rejected.
3. We do find it desirable for `foo()` to be able to invoke
`foo(standalone)` even if `foo()` doesn't exist. This PR removes David's
original form of this, but does not yet provide a replacement.

Reviewed by @dlongnecke-cray and @mppf -- thanks!

# Testing
- [x] dyno tests

compiler/passes/convert-uast.cpp

@@ -4530,6 +4530,11 @@ Type* Converter::helpConvertType(types::QualifiedType qt) {
     case typetags::UnknownType:   return dtUnknown;
     case typetags::VoidType:      return dtVoid;
 
+    // subclasses of IterableType
+    case typetags::FnIteratorType: return dtUnknown; // a lie
+    case typetags::LoopExprIteratorType: return dtUnknown; // a lie
+    case typetags::PromotionIteratorType: return dtUnknown; // a lie
+
     // subclasses of BuiltinType
 
     // concrete builtin types
@@ -4592,6 +4597,8 @@ Type* Converter::helpConvertType(types::QualifiedType qt) {
     case typetags::END_CompositeType:
     case typetags::START_PrimitiveType:
     case typetags::END_PrimitiveType:
+    case typetags::START_IteratorType:
+    case typetags::END_IteratorType:
     case typetags::NUM_TYPE_TAGS:
       INT_FATAL("should not be reachable");
       return dtUnknown;

frontend/include/chpl/framework/Context.h

@@ -879,6 +879,47 @@ class Context {
   #endif
   ;
 
+  /**
+    Note an warning for the currently running query.
+    This is a convenience overload.
+    This version takes in a Location and a printf-style format string.
+   */
+  void warning(Location loc, const char* fmt, ...)
+  #ifndef DOXYGEN
+    // docs generator has trouble with the attribute applied to 'build'
+    // so the above ifndef works around the issue.
+    __attribute__ ((format (printf, 3, 4)))
+  #endif
+  ;
+
+  /**
+    Note an warning for the currently running query.
+    This is a convenience overload.
+    This version takes in an ID and a printf-style format string.
+    The ID is used to compute a Location using parsing::locateId.
+   */
+  void warning(ID id, const char* fmt, ...)
+  #ifndef DOXYGEN
+    // docs generator has trouble with the attribute applied to 'build'
+    // so the above ifndef works around the issue.
+    __attribute__ ((format (printf, 3, 4)))
+  #endif
+  ;
+
+  /**
+    Note an warning for the currently running query.
+    This is a convenience overload.
+    This version takes in an AST node and a printf-style format string.
+    The AST node is used to compute a Location by using a parsing::locateAst.
+   */
+  void warning(const uast::AstNode* ast, const char* fmt, ...)
+  #ifndef DOXYGEN
+    // docs generator has trouble with the attribute applied to 'build'
+    // so the above ifndef works around the issue.
+    __attribute__ ((format (printf, 3, 4)))
+  #endif
+  ;
+
   /**
     Sets the enableDebugTrace flag. This was needed because the context
     in main gets created before the arguments to the compiler are parsed.

frontend/include/chpl/resolution/resolution-types.h

@@ -1105,6 +1105,13 @@ class TypedFnSignature {
     return fetchIterKindStr(context, str) && str == USTR("follower");
   }
 
+  /** Returns 'true' if this signature is for a parallel iterator. */
+  bool isParallelIterator(Context* context) const {
+    return isParallelStandaloneIterator(context) ||
+           isParallelLeaderIterator(context) ||
+           isParallelFollowerIterator(context);
+  }
+
   /** Returns 'true' if this signature is for a serial iterator. */
   bool isSerialIterator(Context* context) const {
     UniqueString str;

frontend/include/chpl/types/FnIteratorType.h

@@ -0,0 +1,69 @@
+/*
+ * Copyright 2024 Hewlett Packard Enterprise Development LP
+ * Other additional copyright holders may be indicated within.
+ *
+ * The entirety of this work is licensed under the Apache License,
+ * Version 2.0 (the "License"); you may not use this file except
+ * in compliance with the License.
+ *
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef CHPL_TYPES_FN_ITERATOR_TYPE_H
+#define CHPL_TYPES_FN_ITERATOR_TYPE_H
+
+#include "chpl/types/Type.h"
+#include "chpl/types/QualifiedType.h"
+#include "chpl/resolution/resolution-types.h"
+#include "chpl/types/IteratorType.h"
+
+namespace chpl {
+namespace types {
+
+class FnIteratorType final : public IteratorType {
+ private:
+  /*
+    The iterator procedure that was invoked to construct an iterator of this
+    type. E.g,, the type of `foo()` will contain `iter foo()` as the iterator
+    function.
+   */
+  const resolution::TypedFnSignature* iteratorFn_;
+
+  FnIteratorType(QualifiedType yieldType, const resolution::TypedFnSignature* iteratorFn)
+    : IteratorType(typetags::FnIteratorType, std::move(yieldType)), iteratorFn_(iteratorFn) {}
+
+  bool contentsMatchInner(const Type* other) const override {
+    auto rhs = (FnIteratorType*) other;
+    return this->yieldType_ == rhs->yieldType_ &&
+           this->iteratorFn_ == rhs->iteratorFn_;
+  }
+
+  void markUniqueStringsInner(Context* context) const override;
+
+  static const owned <FnIteratorType>&
+  getFnIteratorType(Context* context,
+                    QualifiedType yieldType,
+                    const resolution::TypedFnSignature* iteratorFn);
+
+ public:
+  static const FnIteratorType* get(Context* context,
+                                   QualifiedType yieldType,
+                                   const resolution::TypedFnSignature* iteratorFn);
+
+  const resolution::TypedFnSignature* iteratorFn() const {
+    return iteratorFn_;
+  }
+};
+
+} // end namespace types
+} // end namespace chpl
+
+#endif

frontend/include/chpl/types/IteratorType.h

@@ -0,0 +1,57 @@
+/*
+ * Copyright 2024 Hewlett Packard Enterprise Development LP
+ * Other additional copyright holders may be indicated within.
+ *
+ * The entirety of this work is licensed under the Apache License,
+ * Version 2.0 (the "License"); you may not use this file except
+ * in compliance with the License.
+ *
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef CHPL_TYPES_ITERATOR_TYPE_H
+#define CHPL_TYPES_ITERATOR_TYPE_H
+
+#include "chpl/types/Type.h"
+#include "chpl/types/QualifiedType.h"
+
+namespace chpl {
+namespace types {
+
+class IteratorType : public Type {
+ protected:
+  /*
+    The type produced by this iterator. E.g., in a loop such
+    as the right hand side in the below assignment:
+
+       var A = foreach i in 1..10 do (i,i);
+
+    the yield type is (int, int).
+   */
+  QualifiedType yieldType_;
+
+  IteratorType(typetags::TypeTag tag, QualifiedType yieldType)
+    : Type(tag), yieldType_(std::move(yieldType)) {}
+
+  Genericity genericity() const override {
+    return CONCRETE;
+  }
+
+ public:
+  const QualifiedType& yieldType() const {
+    return yieldType_;
+  }
+};
+
+} // end namespace types
+} // end namespace chpl
+
+#endif

frontend/include/chpl/types/LoopExprIteratorType.h

@@ -0,0 +1,124 @@
+/*
+ * Copyright 2024 Hewlett Packard Enterprise Development LP
+ * Other additional copyright holders may be indicated within.
+ *
+ * The entirety of this work is licensed under the Apache License,
+ * Version 2.0 (the "License"); you may not use this file except
+ * in compliance with the License.
+ *
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef CHPL_TYPES_LOOP_EXPR_ITERATOR_TYPE_H
+#define CHPL_TYPES_LOOP_EXPR_ITERATOR_TYPE_H
+
+#include "chpl/types/Type.h"
+#include "chpl/types/QualifiedType.h"
+#include "chpl/types/IteratorType.h"
+
+namespace chpl {
+namespace types {
+
+class LoopExprIteratorType final : public IteratorType {
+ private:
+  /*
+     Whether the loop expression is zippered. In production, zippered
+     loop expressions contain all of their iterands and separately invoke
+     leader/follower iterators on each one
+   */
+  bool isZippered_;
+  /*
+     Whether this loop can be iterated over in parallel. Some loops
+     ('for' and 'foreach') can't, even when their iterands support parallel
+     iteration.
+   */
+  bool supportsParallel_;
+  /*
+     The type of the thing being iterated over, used for resolving the
+     needed leader/follower iterators. If the loop is zippered, iterand
+     should be a tuple type.
+   */
+  QualifiedType iterand_;
+  /*
+     The ID of the AST node that caused this loop type to be constructed.
+     This is part of the type so that loop expressions created at different
+     places are considered to have a different type.
+
+     Note: this may need to change if we compiler-generated loop expressions,
+     since the IDs of those would be nebulous.
+   */
+  ID sourceLocation_;
+
+  LoopExprIteratorType(QualifiedType yieldType,
+                       bool isZippered,
+                       bool supportsParallel,
+                       QualifiedType iterand,
+                       ID sourceLocation)
+    : IteratorType(typetags::LoopExprIteratorType, std::move(yieldType)),
+      isZippered_(isZippered), supportsParallel_(supportsParallel),
+      iterand_(std::move(iterand)), sourceLocation_(std::move(sourceLocation)) {
+    if (isZippered_) {
+      CHPL_ASSERT(iterand_.type() && iterand_.type()->isTupleType());
+    }
+  }
+
+  bool contentsMatchInner(const Type* other) const override {
+    auto rhs = (LoopExprIteratorType*) other;
+    return yieldType_ == rhs->yieldType_ &&
+           isZippered_ == rhs->isZippered_ &&
+           supportsParallel_ == rhs->supportsParallel_ &&
+           iterand_ == rhs->iterand_ &&
+           sourceLocation_ == rhs->sourceLocation_;
+  }
+
+  void markUniqueStringsInner(Context* context) const override {
+    yieldType_.mark(context);
+    iterand_.mark(context);
+    sourceLocation_.mark(context);
+  }
+
+  static const owned<LoopExprIteratorType>&
+  getLoopExprIteratorType(Context* context,
+                          QualifiedType yieldType,
+                          bool isZippered,
+                          bool supportsParallel,
+                          QualifiedType iterand,
+                          ID sourceLocation);
+
+ public:
+  static const LoopExprIteratorType* get(Context* context,
+                                         QualifiedType yieldType,
+                                         bool isZippered,
+                                         bool supportsParallel,
+                                         QualifiedType iterand,
+                                         ID sourceLocation);
+
+  bool isZippered() const {
+    return isZippered_;
+  }
+
+  bool supportsParallel() const {
+    return supportsParallel_;
+  }
+
+  const ID& sourceLocation() const {
+    return sourceLocation_;
+  }
+
+  const QualifiedType& iterand() const {
+    return iterand_;
+  }
+};
+
+} // end namespace types
+} // end namespace chpl
+
+#endif