Context. Appendix to the documentation proposal. This personal draft describes GT4Py
mainatf3b3ab28; it is not an accepted normative standard.
gt4py.next computation DSL specification
Status, scope, and evidence
This document specifies the Python-facing computation language implemented by
gt4py.next at repository revision
f3b3ab28ecfc4de6ef914d6e7bed17a5447fd43f. It covers field operators, scan
operators, programs, and their interface with Python. Internal representations
are evidence for these semantics, not additional user languages specified here.
The package remains experimental; this is a descriptive specification of this
checkout, not a promise of compatibility with later releases.
Rules stated below describe the frontend and its intended mathematical model, with observed execution differences called out explicitly. A rejection by the frontend is different from a limitation of a particular backend. A historical ADR records design intent; it does not establish current support by itself. Recommendations in the conflict register are proposals for interpreting or reconciling existing sources, not changes made to the implementation.
The investigation follows three evidence streams: user documentation and examples (including notebook cells), frontend/data-model/embedded/lowering implementation, and unit/integration/regression tests. The final source index records their coverage. Accepted and rejected examples in tests are executable evidence, subject to their backend skips and expected failures. An untested implementation path is identified as such where it affects the contract.
Notation
d is a named dimension; D is an ordered sequence of dimensions; τ is a
scalar dtype; F[D, τ] is a field type; T₀ × … × Tₙ is a fixed product type.
Γ ⊢ e : T means that expression e has type T in environment Γ.
R(d) = [a, b) is an integer interval. p|D projects a labeled index onto
D. Mathematical rules describe values, not allocation or fusion decisions.
A rule applies only when its side conditions hold. Unless a section explicitly states an execution difference, storage must contain every element read and outputs must contain every element written. No rule implies automatic halo exchange, padding, clipping, or boundary conditions.
1. Semantic objects
1.1 Dimensions and grids
A dimension is the immutable pair (name, kind). Kinds are HORIZONTAL
(default), LOCAL, and VERTICAL. Equality includes both name and kind;
separately created equal dimensions identify the same axis. Dimensions label
logical coordinates, not array positions or units of physical distance.
Horizontal dimensions may represent Cartesian coordinates or unstructured
location sets such as cells, edges, and vertices. A local dimension labels
neighbor slots; a vertical dimension may carry a sequential scan.
Canonical field dimension order groups horizontal, local, then vertical
axes, ordering base names lexicographically within a group. Dimension promotion
merges compatible dimension sequences into this order. Field/DSL dimension
checks reject noncanonical order, duplicates, more than one local dimension, and
coexistence of a base dimension and its staggered counterpart. A field annotation
must already list its dimensions in canonical order: for horizontal I and
vertical K, Field[Dims[K, I], τ] raises InvalidAnnotationError at
decoration, and promotion does not reorder it. A host Domain object has weaker
construction checks and is not proof that a field annotation is admissible.
For non-staggered d, flip_staggered(d) represents locations half a cell below
d: its integer index i denotes physical position i - 1/2 in base-grid
coordinates. is_staggered and as_non_staggered inspect/normalize this pairing.
The _Staggered name prefix is implementation encoding; names such as IHalf
in examples are Python variables, not a required suffix. Staggering preserves
kind. It changes labeled coordinate spaces, not array index types.
GridType.CARTESIAN and GridType.UNSTRUCTURED select grid constraints.
Unstructured offsets/local dimensions require the unstructured grid model;
unstructured programs can also perform Cartesian-like calculations. When the
user does not select a grid type, the frontend deduces it from referenced
dimensions, offsets, and operators.
Evidence: data model (Dimension, order_dimensions, check_dims,
promote_dims, stagger helpers), grid deduction in program lowering
using transform utilities,
dimension tests, staggered tests, and
ADR 26.
1.2 Ranges, domains, and coordinates
A unit range is a contiguous integer interval [a, b) with stride one.
UnitRange normalizes a >= b to the empty range [0, 0). Semantic bounds
may be infinite, for example on broadcast dimensions; allocated storage needs
finite extents. A domain is an ordered product
R = {d₁: [a₁,b₁), …, dₙ: [aₙ,bₙ)}
|R| = {p | aᵢ ≤ p[dᵢ] < bᵢ for every i}.The domain with no dimensions contains the single empty coordinate. A domain with an empty constituent interval has no coordinates. Bounds are runtime properties, not part of a field’s static type.
Domain intersection aligns by dimension labels, adds dimensions present on only one side, and intersects shared intervals. Equivalently, an absent dimension has an unbounded interval for this operation. This is the domain algebra of pointwise field evaluation; it is not NumPy positional shape broadcasting.
The host domain(...) conversion accepts a Domain, a mapping, or an ordered
sequence of (Dimension, range-like) pairs. An integral extent n means
[0,n); a pair (a,b) means absolute bounds; a Python range must have unit
stride. These conveniences are broader than captured program domain syntax
(section 6).
d(i) is a named absolute index. Relative host array indexing and program
output slices use positions relative to existing bounds; explicit domain pairs
use logical coordinates. For a field with logical interval [-2,8), the slice
[0:3] selects [-2,1), whereas domain {d: (0,3)} selects [0,3).
Evidence: data model (UnitRange, unit_range, Domain, domain),
model tests (empty ranges/domains and domain intersections), and
domain tests.
1.3 Scalars, fields, and collections
A scalar is a value of a concrete dtype. A field F[D,τ] is a partial function
from labeled coordinates in a domain over D to scalar values of dtype τ.
Storage layout, strides, padding, device, and allocator do not change this
mathematical value. The Python spelling is gtx.Field[gtx.Dims[d₁, …, dₙ], τ];
gtx.Field[[d₁, …, dₙ], τ] also occurs in existing code.
A zero-dimensional field F[[],τ] contains one value but remains a field.
A scalar and a zero-dimensional field are not generally identical: only a
scalar boolean controls if. .as_scalar() is a host extraction operation.
At call boundaries, promote_zero_dims allows a matching scalar for an annotated
zero-dimensional field, recursively inside supported collections. It does not
implicitly convert a scalar argument into an arbitrary nonzero-dimensional
field.
Tuples are fixed, heterogeneous products, possibly nested. They can contain scalars and fields and can be arguments, local bindings, results, and scan carries. A tuple of fields is not a field with tuple-valued elements. Supported typed named collections additionally preserve member names and original Python type. Their admissibility is checked by the named-collection/type-translation layer; arbitrary Python objects are not DSL values. Member access, construction, unpacking, and selected structure-preserving builtins operate on these statically known products (section 3.4).
Evidence: type specifications, type translation,
call compatibility (promote_zero_dims), named collections,
tuple tests, named-collection tests, and
ADR 13.
1.4 Connectivities and missing neighbors
A connectivity is an integer index map C : R_C → Z_d, with codomain dimension
d. A neighbor table has origin dimension a, local dimension l, and
codomain d; its entries C[a,l] identify neighbors in d. Multiple slots
may contain the same neighbor; multiplicity matters in reductions.
A declaration FieldOffset(tag, source=d, target=(a,l)) names such a remapping.
Its source is the dimension read in the input field; its target describes
the replacement dimensions in the result. By contrast, a neighbor table’s
source_dim names its origin a. These two uses of “source” have opposite
orientations and must not be interchanged.
The runtime offset_provider maps tags to neighbor tables. Its compile-time
descriptor includes domain/codomain dimensions, integer dtype, skip metadata,
and maximum neighbor count; actual table entries are runtime data. Current
Cartesian dimension shifts describe themselves and need no provider entry.
A declared skip value, normally -1, denotes an absent neighbor. Local
reductions omit skipped slots. Ordinary fixed-neighbor dereferences need valid
indices; no portable value is specified for reading a skipped neighbor or an
out-of-bounds table entry. The connectivity is not an automatic boundary
condition.
Evidence: connectivity model (Connectivity, NeighborTable,
NeighborConnectivityType, OffsetProvider, get_offset),
constructors (as_connectivity), ADR 19, and
reduction tests.
2. Language layers and syntax
Python code outside decorated functions is ordinary host Python. Decorated functions form three related constructs:
| Construct | Meaning | Result/effect |
|---|---|---|
@field_operator | Pure composition of operations on whole fields/scalars | Returns a DSL value inside an operator; writes through out at an output-writing entry point |
@scan_operator(axis=K, forward=b, init=z) | A scalar recurrence lifted over columns | Field/collection of fields containing successive updated carries |
@program | Ordered composition of output-writing operator calls | Mutates supplied output fields; returns None |
These are source-captured, statically analyzed Python subsets, not arbitrary
Python functions accelerated opportunistically. The same frontend validation
runs when selecting field-level embedded execution. Ordinary host methods such
as .ndarray, .asnumpy(), allocation, or I/O do not thereby become DSL
operations.
2.1 Abstract grammar
The following grammar describes constructs, not Python precedence or indentation
lexing. Python’s parser supplies those; static side conditions below further
restrict the productions. suite means an indented sequence of statements.
call-arguments are statically bound positional/keyword arguments, not arbitrary
Python argument expansion.
field-definition = "def" name "(" typed-parameters ")"
[ "->" data-type ] ":" field-suite ;
field-suite = { field-statement } ;
field-statement = assignment | conditional | "return" expression
| "pass" | docstring ;
assignment = targets [ ":" data-type ] "=" expression ;
targets = name | tuple-pattern ;
conditional = "if" expression ":" field-suite
{ "elif" expression ":" field-suite }
[ "else" ":" field-suite ] ;
expression = scalar-literal | name | qualified-reference
| tuple-expression | collection-construction
| unary-op expression
| expression binary-op expression
| expression comparison-op expression
| expression "if" expression "else" expression
| callable "(" call-arguments ")"
| field "(" remappings ")"
| expression "[" restricted-index "]" ;
unary-op = "+" | "-" | "~" | "not" ;
binary-op = "+" | "-" | "*" | "/" | "//" | "%" | "**"
| "&" | "|" | "^" ;
comparison-op = "==" | "!=" | "<" | "<=" | ">" | ">=" ;
program-suite = { operator-name "(" call-arguments ","
"out=" output [ ", domain=" domain-spec ] ")" } ;
output = field-name | field-name "[" output-slices "]"
| output-collection ;
domain-spec = "{" domain-entry { "," domain-entry } "}"
| domain-tuple ;
domain-entry = dimension ": (" integer-expression ","
integer-expression ")" ;The program production abbreviates argument ordering and a trailing comma;
it does not require an ordinary argument before out. Domain tuples must
match output structure. Supported module-qualified dimensions, offsets, and constants may be resolved
from closure references. Callable calls require directly bound names: forms
such as gtx.broadcast(...) or module.fieldop(...) in a captured body are
currently rejected, as pinned by module-import tests. Import
the callable into a directly bound name outside the DSL function.
Evidence: field parser, program parser, dialect parser, frontend AST passes, field parser tests, and program parser tests.
2.2 Binding, assignments, and control flow
Every ordinary DSL function parameter needs a supported type annotation.
A field-operator return annotation is optional; when supplied, it must equal the
deduced return type. The safe signature subset is ordinary positional-or-keyword
parameters. Full Python support for positional-only parameters, keyword-only
parameters, defaults, *args, and **kwargs must not be inferred from the
Python function object: the current parsers primarily consume ast.arguments.args
(see conflict C12).
Locals bind immutable values; rebinding a name expresses a new value, not a
mutation of field storage. The frontend converts bindings to static single
assignment form. It supports tuple/named-collection unpacking; a starred
unpacking target receives a tuple. Chained name assignment (b = c = a) is not
supported: it fails at decoration with an AssertionError, because the
single-assignment-target pass creates a synthetic load without
the source location the dialect parser requires. Use separate
assignments instead.
Local annotated assignment is partially supported. Builtin scalar annotations
such as x: float = 1.0 work, and once an annotation resolves, a mismatch with
the assigned value is a DSL type error. The field parser evaluates
local annotations only against the DSL builtins and captured closure variables,
and source capture excludes names used only in annotations. Dims, a
module prefix such as gtx., or a dimension used only in a local annotation can
therefore raise a raw NameError at decoration, even when imported at module
level. The annotated-assignment cases in the parser tests and
lowering tests are skipped or expected to fail. Prefer
ordinary inferred bindings. Assignment through a field subscript or object attribute is not a
field-operator operation. Function/operator definition names cannot shadow
reserved DSL builtin names; this explicit check does not establish a separate
blanket prohibition on every local/parameter spelling.
An if statement or conditional expression requires scalar bool, not a
boolean field or zero-dimensional boolean field. Only the chosen branch
contributes its runtime result, but both branches must be statically valid.
All reachable paths of a field operator return, and all returned types agree
exactly. Conditional-expression alternatives likewise have equal types; no
branch dtype/dimension promotion is implied. A new name is available after a
branch only when the binding/SSA analysis can establish it on all relevant
paths with compatible types. where provides elementwise field selection.
Closure references may name supported constants, dimensions, offsets, callable DSL objects, and typed collections. Module attribute references are resolved against captured Python objects; arbitrary method calls or Python callbacks are not implied. Closure-mutation behavior differs by execution mode (section 7 and C11).
2.3 Rejected Python constructs
The supported subsets do not include loops (for, while), comprehensions,
generators, lambdas, exception handling, context managers, assertions, classes,
pattern matching, import statements in the DSL body, or nested ordinary Python
function definitions. Augmented assignment such as += is also rejected.
Arbitrary expression statements are invalid in field
operators. Boolean and and or are rejected even for scalar booleans; use
boolean & and |. Chained comparisons such as a < b < c are rejected;
write the individual comparisons with & and explicit parentheses.
Programs have a smaller subset: output-writing field/scan-operator calls in
sequence. Programs do not introduce local temporaries with assignment, branch,
loop, return a value, or call another program. Field expressions belong in
field operators. Scalar expressions used for call arguments or bounds use the
restricted program expression grammar: scalar arithmetic, negative scalar
literals, and supported scalar minimum/maximum, not arbitrary field
computation or unrestricted unary expressions.
Evidence: dialect rejection catalogue, diagnostic guide,
diagnostic tests, parser tests
(test_domain_chained_comparison_failure,
test_field_chained_comparison_failure), and program parser.
3. Static types
3.1 Type universe and dtype identity
τ ::= bool | int8 | int16 | int32 | int64
| uint8 | uint16 | uint32 | uint64 | float32 | float64
T ::= τ | F[D,τ] | (T₀, …, Tₙ) | NamedProduct[type, members: T]Dimensions, domains, offsets, callable types, and constructors additionally have compile-time/meta-level types. Their presence does not make them arbitrary runtime field elements. Strings are represented by the type system and can occur as scalar-constructor literals; this is not a promise of string field arithmetic.
The Python int annotation/constructor maps to signed int64, and float to
float64, independent of platform integer width. In contrast, a plain Python
integer value is inferred as int32 when representable, otherwise int64;
values outside signed 64-bit range are rejected. Source integer constants are
classified before unary signs: unary minus preserves its operand’s dtype.
Consequently the source expression -2147483648 has type int64 because its
positive operand exceeds int32, and -9223372036854775808 is rejected because
its positive operand exceeds int64. A host value already equal to either
negative minimum is classified by its signed value instead. A decimal floating
literal is float64. NumPy scalar constants preserve their concrete kind. IndexType aliases int32; frontend alias support
for constructor spelling is incomplete. Export/type-enum availability of
small/unsigned integer widths is broader than the subset handled by some
as_dtype/from_dtype and backend paths. These paths commonly support only
bool, int32, int64, float32, and float64; portable use of other widths
requires backend-specific evidence (C13).
Scalar constructors are intended for typed literals. Reserved constructor
spellings enforce literal arguments; use astype for data conversions.
Constructor aliases may bypass the initial spelling check, but literal-only
validation also occurs during lowering. They do not establish a general
computed-value conversion interface.
Evidence: scalar/type definitions, type translation
(get_scalar_kind), builtins, typing rules, and
conversion tests.
3.2 Dimension promotion
Write dims(τ)=[], dims(F[D,τ])=D, and dtype(T)=τ for primitive scalar/field
types. merge(D₁,D₂) is dimension promotion under the constraints of section
1.1. For admissible ordinary pointwise arithmetic:
Γ ⊢ a : T₁ Γ ⊢ b : T₂ dtype(T₁) = dtype(T₂) = τ
τ is arithmetic D = merge(dims(T₁), dims(T₂))
────────────────────────────────────────────────────────────
Γ ⊢ a op b : τ if both operands are scalars
Γ ⊢ a op b : F[D,τ] otherwiseShared dimensions align by label, not positional index. A scalar or a field missing an axis is constant along that axis. At shared axes the evaluated domain is the intersection of available ranges. Distinct extents on shared axes are not a request to broadcast length one positionally.
There is no general implicit dtype promotion. For example, combining an
int64 field and a small integer literal (int32) requires an explicitly
typed literal or astype; an int32 field plus 1 needs no dtype conversion.
Combining float32 and float64 is likewise not accepted by the ordinary rule.
Bool is not an arithmetic dtype.
** currently bypasses the ordinary result-promotion path and is a recorded
exception, not a model for other binary operations (C5).
Evidence: type utilities (promote), typing rules,
model (promote_dims), and type deduction tests.
3.3 Calls and exact matching
Callable signatures determine argument count, names, supported keyword binding,
and structural types. In captured calls, passing an argument with the wrong
dtype, dimension list, or collection structure is an error; dimension promotion of an expression does
not make every field argument interchangeable with every annotation. The
zero-dimensional scalar compatibility of section 1.3 is a specific exception.
Return annotations constrain the full inferred type. Output fields must match
the computed result’s dimensions/dtype/collection structure; a domain
restriction does not change a field’s type. A scalar return is not generally
assignable to a Field output through the compiled frontend: use broadcast
to produce a statically field-valued result.
These matching rules are also caller obligations at host entry points. Direct field-level embedded operator calls do not perform all runtime dtype, dimension, or nominal-collection checks; invalid calls can execute or cast on assignment. This accidental acceptance is not supported coercion (C21).
Scan calls have a distinct promotion rule: a scalar recurrence parameter may receive a matching scalar or field leaf, and the result is lifted over the merged invocation dimensions plus the scan axis (section 5.5).
Evidence: call compatibility, type utilities, argument interface tests, and scalar argument tests.
3.4 Products and named collections
Tuple element access uses a statically known integer index; negative tuple
indices are supported. Tuple slicing and dynamic tuple indexing are not a
portable DSL operation. Tuple unpacking is statically expanded, including one
starred target. For named collections, supported Python declarations include
typed named tuples and admissible dataclasses/custom named-collection types.
Dataclasses must satisfy the constructor/type layer’s restrictions: at least
one member, no custom __init__, no InitVar, and no default-valued fields.
Collection construction and member access preserve the nominal collection
type; plain tuples and distinct named types are not freely interchangeable.
where recurses through compatible products, including named collections;
astype supports scalars, fields, and recursive tuples, without equivalent
named-collection frontend support. These are products
of DSL data, without arbitrary Python object behavior, member mutation, or
user-defined methods.
Evidence: named collections, type translation
(make_type, make_constructor_type), unpacking pass,
named-collection tests, and tuple tests.
4. Pointwise operations and builtins
4.1 Evaluation rule
For a pointwise operation on fields/scalars, let R be the intersection of
operand domains under the dimension algebra, treating scalars as constant.
For each p ∈ |R|:
(op(e₁,…,eₙ))(p) = op_scalar(e₁(p|dims(e₁)), …, eₙ(p|dims(eₙ))).This equation specifies values and dimensional alignment. It does not prescribe
intermediate materialization, iteration order, fusion, floating-point
reassociation, or evaluation of invalid coordinates outside R.
4.2 Operator and builtin inventory
The table lists the Python-facing function inventory of fbuiltins, together
with expression operators. “Same type” includes dimension preservation;
“promote” means dimension promotion with matching dtypes, not dtype widening.
The experimental functions are specified separately in section 5.
| Spelling | Operand constraint | Static result / meaning |
|---|---|---|
Unary +, -; abs, neg | Arithmetic scalar/field | Same type; identity, negation, magnitude, negation respectively |
+, -, *, /, // | Arithmetic scalar/fields with matching dtype | Promoted dimensions, same dtype; / does not acquire a floating static type automatically |
% | Ordinary promotion plus integral right operand (therefore both integral under matching-dtype rule) | Same dtype over promoted dimensions |
** | Arithmetic base and exponent; special frontend result path | Base type is returned directly; floating base with integral scalar exponent is tested; additional exponent dimensions are not validated/promoted consistently (C5) |
==, !=, <, <=, >, >= | Matching arithmetic dtypes | bool scalar/field over promoted dimensions; bool equality is not covered by this arithmetic comparison rule |
&, |, ^ | Bool scalar/fields | Bool conjunction, disjunction, exclusive-or over promoted dimensions; not integer bitwise arithmetic |
~, not | Bool scalar/field in frontend typing | Logical negation; prefer ~ on fields because not field is an explicitly disputed accidental acceptance (C6) |
sin, cos, tan, arcsin, arccos, arctan | Floating scalar/field | Same floating type |
sinh, cosh, tanh, arcsinh, arccosh, arctanh | Floating scalar/field | Same floating type |
sqrt, exp, log, gamma, cbrt | Floating scalar/field | Same floating type |
floor, ceil, trunc | Floating scalar/field | Same floating dtype, rather than an integer result |
isfinite, isinf, isnan | Floating scalar/field | Bool over the same dimensions |
minimum, maximum, fmod, power | Arithmetic scalar/fields under ordinary promotion | Matching dtype over merged dimensions; integral power prints a result-type consistency diagnostic to standard output, not a Python warning |
astype(value, dtype) | Scalar/field or recursively nested tuple | Convert every primitive leaf to dtype, preserving dimensions and tuple structure |
broadcast(value, dimensions) | Scalar/field; target dimensions contain every input dimension | Field of the same dtype over exactly the target dimensions |
where(mask, true_value, false_value) | Bool field mask; matching branch structure and corresponding leaf dtypes | Recursive field result with branch/mask dimension promotion |
neighbor_sum(value, axis=L) | Local-neighbor field | Sum valid neighbors; remove L, preserve static dtype |
max_over(value, axis=L), min_over(value, axis=L) | Local-neighbor field | Maximum/minimum valid neighbors; remove L, preserve static dtype |
Scalar constructors bool, int, float, the explicitly sized numeric
constructors, and tuple are also exposed names. The collection/literal/type
rules apply to these names; their export is not an unrestricted re-export of
all Python constructor behavior.
Numerical edge cases are only partly specified by the repository. In
particular, integer / has integer static type while embedded array true
division can produce floating values. Negative modulo and floor division carry
backend exclusions. Division by zero may yield IEEE values or raise, and the
basic tests accept both. Overflow/narrowing, NaN and signed-zero propagation,
and reproducible accumulation order have no uniform complete contract here.
Use explicit types and the selected backend’s established behavior; C14 records
the unresolved numerical contract.
Evidence: builtin definitions, typing rules, expression lowering, math test data, math execution tests, unary tests, basic feature tests, and backend exclusions.
4.3 Broadcasting and conversion
For Γ ⊢ v : T of dtype τ, if dims(T) ⊆ D and D is admissible:
Γ ⊢ broadcast(v, D) : F[D,τ]
broadcast(v,D)(p) = v(p|dims(T)).Broadcasting introduces no coordinate dependence on the added dimensions. Embedded scalar broadcast currently returns its scalar unchanged rather than materializing the statically described field (C7); later output assignment may still give the expected values. This matters to field-only operations on the broadcast result, particularly zero-dimensional ones.
astype changes dtype explicitly, not domain or coordinate dependence. For a
tuple it converts leaves independently while retaining tuple structure. Named
collections lack the corresponding generic frontend acceptance.
It is the supported operation for computed-value conversion, unlike the
literal constructors. A cast does not repair missing input coordinates or
change connectivity bounds.
Evidence: typing rules, builtins, embedded field implementation, broadcast tests, and conversion tests.
4.4 Elementwise selection
For primitive branches t, f, and boolean field m, let
D = merge(dims(m), dims(t), dims(f)). Corresponding branch dtypes must agree.
Then where(m,t,f) has type F[D,τ] and, on the common valid domain:
where(m,t,f)(p) = t(p|dims(t)) if m(p|dims(m))
= f(p|dims(f)) otherwise.For products the rule applies to corresponding leaves, preserving the product
and any named type. Even scalar branch leaves become statically field-valued
through the mask. A boolean scalar is not the ordinary where mask type.
where selects among field expressions; it does not promise lazy Python
short-circuit evaluation of branch expressions. Both expressions must be
well-typed and valid for the required evaluation region. In particular,
where is not a general mechanism for hiding an out-of-bounds shift in the
unselected branch. Region-based concatenation has a different domain rule
(section 5.4).
Evidence: typing rules (_visit_where), where tests,
tuple tests, and Quickstart.
5. Coordinate and sequential operations
5.1 General remapping
For a field f containing dimension d, and connectivity c with codomain
d, remapping substitutes that coordinate:
f(c)(p) = f(p with its d-coordinate replaced by c(p|dims(c))).Its type replaces d by the connectivity’s domain dimensions and retains
unaffected dimensions, subject to dimension constraints. Connectivity entries
must name valid input coordinates for every access. Halo allocation belongs
to the caller; domain inference does not synthesize halo values.
For Off = FieldOffset(tag, source=d, target=(a,l)) and provider table C:
f(Off)(a,l,…) = f(C[a,l],…) ; result replaces d by (a,l)
f(Off[q])(a,…) = f(C[a,q],…) ; result replaces d by a.The first gathers all neighbor slots; the second selects a fixed neighbor.
A Cartesian offset needs a displacement, so the analogous unindexed
single-target f(Off) is rejected. A field already carrying a local axis
can select a slot as field[LocalDim(q)], removing that axis. This restricted
field subscript differs from tuple element access and host array slicing.
Multiple Cartesian remappings may occur in one call or chained calls. Fixed unstructured remaps can compose. A remapping call that creates a local neighbor dimension permits only one remapping argument; arbitrary mixtures are not established by the generic equation. Runtime selection of an unstructured neighbor slot is not a general supported operation.
Evidence: type utilities (return_type_field,
function_signature_incompatibilities_field), expression lowering
(_visit_shift), shift tests, reduction tests,
and local-field tests.
5.2 Cartesian and staggered shifts
For integral literal displacement n:
f(d+n)(…,i,…) = f(…,i+n,…)
dom(f(d+n))[d] = dom(f)[d] - n.d-n reverses the displacement. Use dimension arithmetic for current fixed
Cartesian shifts. Indexed single-target FieldOffset remains accepted by parts
of the frontend but retains legacy tagged-provider behavior; it is not an
equivalent portable, self-contained shift spelling (C4). This limitation does
not remove the single-target declaration used by as_offset below.
The displacement must be an inline literal or an attribute of a constant
namespace that closure folding recognizes (FrozenNamespace or
Enum). A plain captured integer such as SHIFT = 1 is rejected in
f(I + SHIFT): folding does not turn arbitrary captured scalar names into the
literal that fixed-shift typing requires. A runtime scalar displacement is not
the same syntax as a fixed shift.
For half-integral h, f(d+h) reads the opposite staggered dimension and
returns dimension d. With result index i, the input integer index is
i + floor(h) + 1 when d is nonstaggered, and i + floor(h) when d is
staggered. Thus a field on IHalf = flip_staggered(I) read as f(I+0.5) uses
IHalf(i+1), while a field on I read as f(IHalf+0.5) uses I(i).
Other fractional parts are rejected. Integer shifts retain staggering.
Experimental as_offset(Off,q) from gt4py.next.ffront.experimental makes a
coordinate-dependent Cartesian offset. Off is single-target with equal
source/target dimension d; q is an integral field containing d:
f(as_offset(Off,q))(p) = f(p with d := p[d] + q(p)).All dynamically chosen input coordinates must be valid. The current static
result-dimension rule does not uniformly account for extra dimensions of q;
treat such extra-dimensional cases as unresolved rather than a general
broadcasting guarantee. This API does not establish arbitrary dynamic
unstructured neighbor selection.
Evidence: Cartesian connectivity construction, experimental builtins, typing rules, shift tests, and staggered tests.
5.3 Neighbor reductions
The portable reduction model is a reduction over one LOCAL neighbor axis
with a known connectivity association. For valid-slot set
S(a) = {l | C[a,l] is not the skip sentinel}:
neighbor_sum(v, axis=L)(a,…) = Σ {v(a,l,…) | l ∈ S(a)}
max_over(v, axis=L)(a,…) = max {v(a,l,…) | l ∈ S(a)}
min_over(v, axis=L)(a,…) = min {v(a,l,…) | l ∈ S(a)}.The result removes L; other dimensions and the static dtype remain.
Repeated neighbor indices contribute once per slot. These operations are not
a general horizontal/vertical global reduction API. Frontend axis checking is
weaker than this portable model: it checks presence, whereas embedded execution
requires LOCAL (C8).
Missing neighbors are excluded rather than replaced by caller-provided physical boundary values. The implementation disagrees about min/max identities for all-skipped neighborhoods and infinite data; no uniform result is specified for those cases (C9). The frontend’s preserved sum dtype may also differ from array-library accumulator widening.
Externally supplied local coefficient fields are supported in expressions with
shifted neighborhood fields. Reduction of a standalone external sparse/local
field is explicitly skipped because compiled unrolling lacks the corresponding
neighborhood evidence. Sparse output writes and arbitrary mismatches between
offset tag and local dimension name also have backend exclusions. Use the
feature matrix, not the broad static Field type, to determine portability.
Evidence: reduction lowering (_visit_neighbor_sum,
_visit_max_over, _visit_min_over), embedded reductions
(_make_reduction), reduction tests,
local-field tests, and backend exclusions.
5.4 Region selection: experimental concat_where
concat_where(R,t,f) is imported from gt4py.next.ffront.experimental.
Its condition is a coordinate region, not a boolean field. Dimension/index
comparisons describe regions; with n of IndexType (int32):
| Expression | Region along d |
|---|---|
d < n | (-∞, n) |
d <= n | (-∞, n+1) |
d > n | [n+1, +∞) |
d >= n | [n, +∞) |
d == n | [n, n+1) |
d != n | Complement of [n, n+1); not embedded-evaluable |
Here the d > n row means integer indices i >= n+1; infinite endpoints are
only notation for unbounded intervals. Reversed comparisons reverse the
relation. Parenthesized domain &, |, ^ expressions have frontend typing,
but disjoint and multidimensional regions are not uniformly executable.
Coordinate arithmetic such as d % 2 is not an index-field expression.
For primitive branches and a one-dimensional region R along dimension d,
let Q be the intersection of the branch domains projected onto all dimensions
other than d. Missing branch dimensions are unbounded for this operation.
Extend Q without restriction along d when intersecting it below:
Q = projection_except_d(dom(t)) ∩ projection_except_d(dom(f))
dom(concat_where(R,t,f)) = Q ∩ ((dom(t) ∩ R) ∪ (dom(f) ∩ complement(R)))
concat_where(R,t,f)(p) = t(p) if p ∈ R, otherwise f(p).This accounts for the orthogonal intersection performed before embedded
concatenation. For example, true/false branches on {I:(0,1), J:(0,3)} and
{I:(1,2), J:(1,4)} with R = I < 1 produce {I:(0,2), J:(1,3)} rather than
the nonrectangular union of their unrestricted contributions. Missing branch
dimensions follow broadcasting; products are processed leafwise with matching
structure/dtypes.
Each branch need only cover its contributed region, so nonoverlapping input
fields can concatenate. This is materially different from where’s common
branch-domain requirement.
Embedded execution requires a one-dimensional condition and a contiguous
resulting domain; interior gaps raise NonContiguousDomain. It cannot express
every compiled disjoint condition. In particular, d != n is frontend syntax with
backend-specific support: in field-level embedded execution, evaluating it raises
NotImplementedError in dimension comparison before concat_where
runs. The equivalent selection concat_where(d < n, t, concat_where(d > n, t, f))
avoids constructing a disjoint condition, subject to the ordinary branch-domain
and contiguity requirements. Empty branches,
infinite broadcast domains, scalar branches, and more elaborate concatenations
have feature-specific exclusions. The frontend accepts some region forms that
embedded execution cannot represent; section 8 records this boundary rather
than treating all typed regions as portable.
Evidence: experimental definitions, typing rules, embedded domain operations, concat tests, and ADR 22.
5.5 Scans
A declaration
@gtx.scan_operator(axis=K, forward=True, init=z)
def recurrence(carry: C, x: X) -> C: ...defines a scalar recurrence lifted over a vertical axis K. axis is
required, and K must have kind VERTICAL. forward is a boolean defaulting to
True; init defaults to 0.0. The first parameter, return value, and init
have exactly matching scalar/product types, so the default initializer fits only
a scalar float64 carry: integer, differently typed, or product carries need an
explicit compatible init. C may be a nested
tuple or supported named collection with scalar leaves. The body uses the
scalar DSL subset and may compose lowerable scalar operators; it is not an
arbitrary Python loop or a field remapping of the carry.
At invocation the caller supplies inputs after carry; the implementation
supplies the carry. Scalar pass parameters accept matching scalar/field leaves
at the field-level call. Invocation dimensions merge input dimensions and K.
Each scalar carry leaf becomes a field over that merged set, even if no input
contains K. Collection inputs and outputs participate as composite values.
For each orthogonal coordinate p and requested interval [l,u):
forward: c(l-1) = z
c(k) = recurrence(c(k-1), x(p,k), …), k=l,…,u-1
result(p,k) = c(k)
backward: c(u) = z
c(k) = recurrence(c(k+1), x(p,k), …), k=u-1,…,l
result(p,k) = c(k).The scan is inclusive: it stores the updated carry. init resets independently
for every column and invocation, including an invocation on a subdomain. It is
not implicitly taken from a neighboring output value outside that subdomain.
The enclosing output/domain context supplies the scan interval.
Embedded scans require a single shared domain/column interval. Explicit
per-leaf domain tuples can fail, both at direct host entry and when called from
a program; use a shared domain or separate invocations (C2). Direct scan
operators can invoke/JIT but cannot be explicitly precompiled with .compile().
For ahead-of-time compilation, wrap the scan in a program with concrete field
annotations and compile that program (section 7.4).
All scans in one lowered program must share the same scan axis. General horizontal/local scans, per-point initial state, multiple scan axes, and arbitrary nested scans are not a portable extension of this rule. Scalar-only scans exist, but embedded execution excludes scan-without-field-input cases. Iterator-level nested-scan tests do not establish Python DSL nesting support.
Evidence: scan typing (visit_ScanOperator), call rules
(return_type_scanop, _scan_param_promotion), embedded operators,
program lowering (_column_axis), scan tests,
named-collection tests, and scan regression.
6. Programs, domains, and writes
For nonconflicting output storage, an operator call denotes evaluation followed
by assignment over the selected domain. If program state is σ, result is
v, output field is o, and selected domain is R:
σ'(o,p) = v(p) for p ∈ |R|
σ'(o,p) = σ(o,p) for p outside |R|.For products apply the rule to each output leaf and its selected domain. Program statements execute in source order; a later statement observes writes from earlier statements. Local field-operator assignments do not write host fields. The equation does not resolve conflicting overlapping writes within one call (section 6.3).
6.1 Explicit domains
Captured program domain= accepts a nonempty dictionary literal of dimensions
to integral (start, stop) expressions. It names absolute, half-open logical
ranges. Current lowering requires keys to match output dimensions in order;
omitted, extra, or reordered axes are not a supported shorthand. Runtime scalar
parameters can supply bounds; field bounds need not all be static compilation
constants.
A single outer domain dictionary may constrain all output leaves, or a domain tuple can recursively match the output structure and supply different leaf domains. Within nested structures, a dictionary is not a general substitute for an entire nested tuple of domains. Embedded scans require a single shared domain and do not support this per-leaf tuple workaround (section 5.5). Each write domain must be covered by its output and by all input reads required by the expression, including shifted and dynamically indexed coordinates.
6.2 Output slices and omitted domains
A program output may be a field name or a restricted relative slice of it.
Captured output slicing requires one slice per dimension with its step
omitted. Integer indexing, ellipsis, and explicit steps (including ::1)
are not the output-domain syntax. For current output bounds [a,b), a nonnegative bound n means a+n,
a negative bound means b+n, and an omitted bound selects the corresponding
endpoint. domain= cannot be combined with sliced output arguments.
Captured slice bounds must be integer literals (including supported negative
literal spellings) or omitted. Runtime names and arithmetic expressions such as
1+1 are unsupported bounds. Use an explicit absolute domain for runtime bounds,
for example domain={I: (0,n)}; account for nonzero logical origins when
replacing a relative slice.
Without a domain restriction, lowered execution derives a domain independently for each output leaf. Field-level embedded execution instead intersects all output-leaf domains before assignment. Equal output domains give the same choice; unequal extents can differ (C2). Equal domains avoid that difference; explicit per-leaf domain tuples also do so for ordinary field operators. Embedded scans instead require a single shared domain/column interval.
Output fields must match result leaf types without a general scalar-to-field result conversion. Lowering rejects local output dimensions, and sparse output tests have backend exclusions. Host construction of such a field is not proof that a portable program may write it.
Evidence: program typing, program lowering
(_compute_field_slice, _visit_slice_bound, _visit_stencil_call_out_arg),
embedded assignment (_get_out_domain, _tuple_assign_field),
domain tests, program tests, and
multiple-output domains.
6.3 Aliasing, boundaries, and execution obligations
Some in-place calls are established by tests, including pointwise output reuse and scans writing an input. This does not specify every case of shifted reads sharing output storage, intersecting views, or multiple aliased output leaves. In particular, distinct allocations in tuple-output tests do not establish a simultaneous in-place swap guarantee. The coherent mathematical interpretation would evaluate right-hand sides from the call’s input state, but full snapshot semantics for arbitrary aliases is a recommendation, not a verified contract. Use independent outputs or caller-managed temporaries where overlapping reads and writes could affect results.
The caller supplies valid halos, valid connectivity entries, and sufficient output coverage. The DSL does not perform mesh communication, time stepping, physical boundary conditions, or dynamic allocation as implicit effects of an operator. A domain restricts writes and determines required reads; it does not turn an invalid read into a defined value. Error timing and out-of-bounds behavior can differ between validation, embedded indexing, and native code; no portable value is specified for invalid accesses.
Evidence: scan tests (test_scalar_scan),
domain tests (test_single_value_field),
decorator tests, and multiple-output tests.
7. Python interface
7.1 Construction and storage
| Host API | Contract relevant to the DSL |
|---|---|
empty, zeros, ones, full | Allocate fields on an explicit domain. Default dtype is float64 except full, which derives dtype from the fill value. |
as_field(domain_or_dimensions, data, ...) | Copy supported array data into new field storage, with optional dtype/device/allocator selection. There is no copy=False contract. |
as_connectivity(..., codomain=d, skip_value=...) | Copy integer index data and attach connectivity metadata. Default auto-detection selects -1 when present, else no skip sentinel; explicit None disables auto-detection. |
Field.domain, .dtype | Expose logical bounds/dimensions and dtype. |
.ndarray, .asnumpy(), .as_scalar() | Host storage access, host NumPy conversion, and zero-dimensional scalar extraction. |
| Host field indexing and mutable-field assignment | Relative/named/absolute views and storage updates according to the Field protocol; not unrestricted DSL subscript/store syntax. |
With a dimension-only as_field argument, data shape supplies [0,n) bounds.
An origin origin[d]=o changes bounds to [-o,n-o): array index o is logical
index zero. A concrete domain cannot be combined with a nonempty origin, and
its extents must match data shape. Default dtype follows input data; default
allocation follows its device, otherwise CPU. Namespace allocators such as
NumPy/CuPy/JAX and GT4Py allocator/backend objects are host configuration.
aligned_index is ignored with a warning for namespace allocation.
Constructors copy, but subsequent field slices may share underlying storage.
The general as_connectivity signature admits broader shapes and skip
sentinels than the portable compiled neighbor-table boundary. Current remapping
paths require None or -1 skip metadata; constructing another sentinel does
not establish that it can execute (C10).
Evidence: constructors, Field protocols, constructor tests, and embedded field tests.
7.2 Decoration, capture, and execution modes
program, field_operator, and scan_operator consume Python function objects.
They retrieve source and resolve annotations using Python introspection and
captured globals/nonlocals. Nonlocals override globals. Syntax/type validation
runs during frontend construction even for embedded execution. Arbitrary
dynamically generated functions without retrievable source are not a guaranteed
entry path.
Captured constants and qualified namespace values are folded for lowering; enums and supported frozen namespaces have test coverage. Embedded execution calls the original Python function and may see later changes to its environment, whereas parsed/cached compiled stages retain captured values. Uniform closure mutation/recompilation semantics are not specified. Treat closure values as immutable configuration; pass changing values explicitly.
Immutability alone does not guarantee closure composition. Recursive closure
collection currently rejects a parent and referenced child that capture the
same name with unequal values, raising NotImplementedError during collection
or lowering even though Python gives them distinct lexical scopes. Use distinct
captured names or explicit parameters for such compositions. Evidence:
recursive closure collection (_get_closure_vars_recursively) and
program lowering.
program and field_operator accept backend/grid/compilation configuration
and can be used bare or with arguments. scan_operator requires axis
(forward and init have defaults, section 5.5) and accepts backend/grid selection, but does not expose the same
compilation-option keywords at decoration time; its resulting entry point can
use .with_compilation_options. An omitted backend uses DEFAULT_BACKEND,
initially None; explicit None chooses field-level embedded execution.
A backend value is a Backend object, not a backend-name string. Configured
copies are produced by .with_backend, .with_grid_type, and
.with_compilation_options.
| Execution selection | Meaning |
|---|---|
backend=None | Original Python field-operator/program execution over registered field implementations, typically NumPy/CuPy/JAX; no iterator lowering |
gtx.itir_python | Lower to iterator IR and execute the roundtrip Python backend |
gtx.gtfn_cpu, gtx.gtfn_gpu, other backend objects | Lower and run generated native/backend code subject to backend feature support |
An operator called inside a program/operator uses the enclosing execution context; its own configured backend applies when it is the host entry point. Field-level embedded execution is distinct from executing lowered iterator IR in Python. Neither one is a universal oracle for every accepted construct.
Evidence: decorators, source capture, closure folding, closure tests, public API, and decorator tests.
7.3 Calling convention
| Call site | Required behavior |
|---|---|
| Ordinary Python calls a program | Supply typed parameter arguments; writes output arguments; returns None |
| Ordinary Python calls a field/scan operator | Supply out=; optional domain= controls writes; returns None |
| A program calls a field/scan operator | out= is required; optional domain/slice determines writes |
| A field operator calls another operator | Expression call returns a DSL value; no out= |
Omitting offset_provider at a host entry point becomes {}. Programs also
normalize explicit None to {}; a field/scan operator requires a mapping
when the keyword is explicitly supplied, so None is not equivalent there.
Named neighbor offsets require the necessary table mapping; Cartesian-only
computations can omit it. Nested calls inherit the provider from the execution
context. Omitting out on a host operator call is diagnosed even in embedded
mode; internal expression calls return values because they run inside an
existing context.
Arguments support positional/keyword binding for ordinary annotated parameters.
Captured calls validate names, count, dtype, dimensions, and product structure.
Host callers must also supply matching types, but direct field-level embedded
operator calls bypass some runtime type checks and may execute or cast invalid
arguments/results during assignment (C21).
out is a field or recursively matching output collection. The compiled frontend requires field result leaves matching the output
fields. Embedded assignment can store a scalar (for example the scalar
broadcast shortcut), but that is not a general scalar-return entry-point rule. Host domain arguments use domain conversion interfaces; the captured
program literal syntax remains the narrower rule of section 6.
Omit domain to infer writes from outputs. Explicit domain=None is treated
as omission only by field-level embedded operator execution; compiled operator
host calls pass it to domain conversion and reject it (C22).
Evidence: decorators (Program.__call__, FieldOperator.__call__),
operator-to-program adaptation, embedded call wrapper,
and argument interface tests.
7.4 Specialization and compilation
.compile(offset_provider=..., **static_arguments) requires a non-None
backend and runtime neighbor-table mappings, or a list of such mappings. An
empty mapping suffices when no tables are used; connectivity configuration may
supply the mapping instead. It submits variants
and returns the same entry-point object, updating its compiled-variant pool.
Although the public annotation admits provider type descriptors, nonempty
descriptor-only mappings are currently rejected, including when supplied
through connectivity configuration (C20). Static
argument values are supplied as lists, producing combinations of variants.
static_params specializes selected scalar parameters; static_domains
specializes field bounds. Otherwise bounds may be runtime data.
Direct scan operators cannot be explicitly precompiled through this interface:
their generic invocation types are not supplied by .compile(). Wrap the scan
in a program with concrete field annotations and compile that program instead.
Direct scan invocation can still compile a variant through JIT (C20).
enable_jit controls compilation of a missing variant at invocation. Disabling
it requires a matching precompiled variant. The global gtx.wait_for_compilation() waits for
submitted compilation work. These host controls do not change the meaning of
a well-defined DSL expression.
Program.with_bound_args(**kwargs) creates a reduced interface with bound
scalar arguments. Unknown names and repeated bound arguments are errors;
explicit .compile() on ProgramWithBoundArgs is currently unimplemented.
Compilation options, caches, workers, allocator selection, and backend builders
are host infrastructure. Caller-owned DaCe workspace memory has backend-specific
lifetime/concurrency requirements, not a DSL allocation primitive.
Evidence: decorators (compile, with_static_params,
with_bound_args), compilation options, compiled pool,
compiled-program tests, bound-argument tests,
ADR 27, and advanced guides.
7.5 Errors and conformance
Unsupported syntax, undeclared names, invalid annotations, incompatible types,
and invalid call signatures are DSL errors. The frontend uses DSLError and
subclasses with source locations and structured explanatory data. The exact
rendered message is a diagnostic interface, not a different language meaning.
Host construction/conversion errors can instead use Python exceptions; some
unsupported lower/backend paths still fail later with assertions or
NotImplementedError.
Frontend acceptance is necessary but does not establish backend conformance.
Feature tests run a backend matrix with explicit skips/expected failures.
Use definitions.py and the feature tests together. Iterator-only
features (lift, generic iterator reductions, nested iterator scans, and
internal runtime-domain primitives) are not Python DSL constructs simply
because they occur in the same test tree.
Evidence: diagnostic guide, diagnostic tests, backend matrix, and ADR 15.
8. Contradictions, interpretation, and unresolved contracts
This register separates disagreements from ordinary absence of support. The recommendations are the most coherent interpretation of the collected evidence; they are not implemented by this documentation change. “Portable” below means avoiding the particular disagreement, not certification of every backend.
C1. Historical frontend/domain API versus the accepted API
Definitions: ADR 1 rejects an explicit field-operator domain=
interface in favor of output slicing and presents AST-root decorator entry
points. Current decorators, domain tests, and
workshop material use Python-function decorators and explicit domains.
Interpretation: retain the ADR as architecture history; the current surface
includes explicit absolute domains and relative output slicing. Do not revive
its rejected syntax decision as a current prohibition.
C2. Default domains of multiple outputs
Definitions: ADR 10 requires same-shaped/same-sliced outputs and
infers a shared domain from the first field. Current program lowering
(_visit_stencil_call_out_arg) derives each output-leaf domain independently;
multiple-output tests and named outputs
accept distinct explicit leaf domains. Embedded execution
(_get_out_domain) instead intersects all output domains when none is supplied.
Observation: the probe in section 9 produces different writes in embedded
and itir_python execution. Interpretation: per-leaf domains are the
coherent model consistent with the current multiple-output API. Use equal output
domains or, for ordinary field operators, explicit per-leaf domain tuples until default execution agrees;
the old blanket same-shape restriction is not current frontend law.
Embedded scans are an additional limitation: _get_vertical_range in
embedded execution produces a tuple of ranges for a domain tuple,
while ScanOperator.__call__ expects one range. Use a single shared
domain/column interval or separate invocations for embedded scans, including
scans called from a program; per-leaf tuples are not a universal workaround.
C3. where dtype promotion
Definitions: Quickstart, “Using conditionals on fields,”
says dimensions and dtype are promoted. Type promotion and
frontend typing require matching corresponding leaf dtypes.
Interpretation: dimensions promote; dtypes do not implicitly widen. Use
astype explicitly. Product leaves may have different dtypes from one another,
but each true/false leaf pair must match; there is no single global product dtype.
C4. Offset-provider values and optionality
Definitions: the history in ADR 19 includes dimension-valued
Cartesian provider entries and older provider classes. The current
model defines provider elements as neighbor tables; dimension shifts
are self-describing. Quickstart says host operator calls require
both out and offset_provider, while decorators and
call-interface tests allow an omitted provider.
Interpretation: provide a tag-to-neighbor-table mapping when a named neighbor
lookup needs it; otherwise omission means {}. out remains required at host
operator entry points. Explicit offset_provider=None is accepted by programs,
while operator calls require a mapping when supplied. Old dimension-valued
provider examples do not establish a portable current API. Indexed single-target
FieldOffset is still accepted by frontend typing and appears in
diagnostic hints, but lowering emits a tagged
shift and embedded offset handling looks up a neighbor table. It is
not equivalent to a self-contained Cartesian shift: use field(I+n) or
field(I-n) for fixed shifts. Single-target declarations used by as_offset
remain a separate supported form.
C5. ** versus ordinary promotion and power
Definitions: typing returns the base type immediately for **,
while power follows common-dtype/dimension promotion. The parser test
test_binary_pow in field parser tests deliberately allows
a floating field raised to an integral scalar exponent. A focused parser probe
also accepts float64[I] ** int32[J] and reports only float64[I], losing the
exponent’s extra dimension. Interpretation: preserve the tested floating
base/integral scalar exponent behavior. Additional field-exponent dimension
and compatibility checks are needed before a general field-exponent rule can
be promised; indiscriminately imposing equal dtypes on ** would contradict
the intentional scalar exponent case.
C6. Boolean negation across Python and lowered execution
Definitions: typing/lowering accept field not and scalar/field
~ as logical negation. Unary tests explicitly xfail field not
as accidental support that should be rejected. Python not field truth
conversion and Python ~True == -2 differ from elementwise/logical semantics.
The scalar-to-0d probe in section 9 reproduces the latter difference through
an accepted call. Interpretation: use not for scalar bool and ~ for
actual bool fields. Field not acceptance and Python-bool inversion must not
be used to infer a coherent cross-mode boolean contract.
C7. Broadcast fields and scalar-to-zero-dimensional compatibility
Definitions: ADR 13 describes annotation-driven implicit scalar
broadcast. Call typing implements scalar/0d compatibility, but
embedded operators pass the original scalar to the Python body.
Separately, the scalar broadcast implementation returns its input
unchanged despite a statically field-valued result. Field-only behavior can
therefore differ even when simple arithmetic agrees.
Interpretation: retain the distinct scalar/field types and the explicit
compatibility rule. Abstract broadcast denotes a constant field; the scalar
embedded representation is a shortcut with observable limitations. Pass an
actual zero-dimensional field when field operations matter. A uniform wrapper
or representation-independent primitive semantics would be needed to turn
annotation compatibility into uniform runtime behavior.
C8. Reduction-axis acceptance and sparse-only reductions
Definitions: frontend typing checks that the requested axis
exists, whereas embedded reductions require it to be LOCAL,
and lowering constructs neighborhood-list reductions. The external-local-field
suite explicitly skips standalone local-field reduction.
Interpretation: specify neighbor reductions over a local axis with known
connectivity association, spelling axis= explicitly. Neither arbitrary
global reductions nor standalone sparse-only reductions follow from the weaker
static check. The positional-axis Python signature is also broader than the
frontend implementation, which accesses kwargs['axis'].
C9. Reduction identity, skipped slots, and dtype
Definitions: lowering seeds max/min with finite dtype
minimum/maximum (arithmetic_bounds in type utilities);
embedded reduction fills skipped slots with global input extrema.
All-skipped neighborhoods and all--inf maximum/all-+inf minimum can therefore
differ. Array-library integer sum can widen while frontend result typing
preserves input dtype. Interpretation: normal nonempty finite neighborhoods
follow section 5.3. Recommend uniform typed identities (zero for sum,
-∞/+∞ for floating max/min, integer extrema), an explicit empty-neighborhood
rule, and an explicit accumulator dtype. Current sources do not settle these
edge cases uniformly, including NaN behavior.
C10. General skip-sentinel API versus executable remapping
Definitions: constructors and connectivity metadata accept
integral skip_value arguments; embedded fields require None
or -1 on current paths: remapping asserts skip_value is None or skip_value == -1,
and _make_reduction compares against the fixed _DEFAULT_SKIP_VALUE rather than
connectivity metadata.
A constructor/remapping probe accepts a table with skip_value=-99, then
raises AssertionError on remapping it. Interpretation: -1 is the
portable missing-neighbor sentinel; the generic constructor signature is
broader than established execution support. Either execution should honor
metadata generally or construction should reject unsupported sentinels earlier.
C11. Closure lifetime
Definitions: source capture and cached frontend stages retain
captured values for lowering; embedded execution calls Python
functions that can observe changed globals/nonlocals. No common reparse or
invalidation rule establishes mutation semantics. Interpretation: closure
values should be immutable configuration; dynamic values are parameters.
There is also a concrete composition restriction independent of mutation:
recursive closure collection rejects unequal same-named captures
in a parent and referenced child. A child capturing factor=1 and a program
capturing factor=2 can construct successfully, then fail lowering with
NotImplementedError. Distinct capture names or explicit parameters avoid that
restriction. Python lexical scope independence is therefore not fully preserved
by current recursive collection. The lifetime behavior remains unresolved;
no rule establishes that every mode snapshots every Python object.
C12. Python declaration signatures
Definitions: Python functions can declare defaults, positional-only,
keyword-only, and variadic parameters; frontend parsers primarily
consume ordinary node.args, constructing positional-or-keyword signatures.
Interface tests validate permutations of ordinary supplied arguments,
not every Python declaration form. Interpretation: annotate ordinary
parameters and supply all values for portable declarations. Do not characterize
all other forms as consistently rejected at decoration time: some are omitted
or interpreted differently by the captured path versus Python invocation.
C13. Exported dtypes versus conversion/backend coverage
Definitions: exports, ScalarKind, and scalar classification
include small and unsigned integer widths; type conversion helpers
as_dtype/from_dtype cover only bool/int32/int64/float32/float64.
Interpretation: type representation and export do not certify all execution,
allocator, or conversion paths. State the wider type universe, and require
backend evidence for the smaller/unsigned widths; do not label them categorically
unparseable. Plain integer values also use int32/int64 inference, while the
int annotation denotes int64: this distinction is current behavior, not an
implicit contextual conversion rule. In a negative literal, the positive
magnitude is classified before unary minus is applied (no unsigned dtype is
inferred), so source-negative dtype minima differ from host values
as described in section 3.1; the representable-value rule alone does not describe
every source spelling.
C14. Numerical semantics
Definitions: frontend typing keeps integer / integer, unlike
normal Python/array true division. Basic tests tolerate both
IEEE inf/NaN and ZeroDivisionError; backend exclusions cover floor
division and negative modulo. Interpretation: no uniform Python/NumPy
numerical contract can be inferred. A future specification should decide
integer division dtype, signed division/modulo, narrowing/overflow, exceptional
values, NaN/signed-zero selection, and reduction ordering explicitly. Until
then retain the static rules and label these runtime cases backend-dependent.
C15. Dimension identity and domain-map ordering
Definitions: ADR 26 describes dimensions as identified by name;
Dimension equality and grid tests distinguish kind
as well. Domains are conceptually keyed by dimension, yet
initialized-domain lowering checks dictionary key order against
output dimensions. Interpretation: equality is (name,kind). Keep the
current order constraint for accepted program domains, while recommending
key-based lookup as the coherent future map semantics. Backend-generated tags
may impose additional practical name constraints not established here.
C16. Region algebra versus embedded interval representation
Definitions: frontend region typing accepts some disjoint and
multidimensional expressions; concat tests exercise compiled
!= and mark embedded exclusions. Embedded dimension inequality rejects a
disjoint region; concat requires one-dimensional, contiguous representable
output (ADR 22). Interpretation: the abstract coordinate-region
algebra is wider than the common execution subset. Keep those distinctions
explicit; one-dimensional contiguous concatenations are the shared model,
not every well-typed region expression.
C17. Backend selector examples and internal representation terminology
Definitions: some decorator docstrings show a string such as
backend="roundtrip", but execution expects a backend object with transforms.
Historical tuple IR ADRs discuss equivalences involving tuples inside fields,
while current frontend values are products of scalar-valued fields.
Interpretation: use backend objects such as gtx.itir_python; IR tuple
representations do not create Python fields with arbitrary tuple-valued elements.
The same boundary excludes internal get_domain_range, lift, and generic
iterator primitives from this DSL specification.
C18. Aliases and dimension-changing remaps
Definitions: existing in-place tests establish selected reuse cases; the sources do not establish arbitrary overlapping-output or shifted snapshot semantics. Remap lowering visits multiple remaps, while type result deduction bases substitution on the first remap on some paths. Ordinary fixed Cartesian shifts leave dimensions unchanged, masking this discrepancy; multi-axis staggering or extra-dimensional dynamic offsets need more coverage. Interpretation: preserve tested in-place and fixed-shift cases, but do not generalize to arbitrary aliases or dimension-changing remap combinations. Recommend snapshot right-hand-side semantics and complete remap dimension deduction as future rules backed by dedicated tests.
C19. Named-collection declaration and conversion limits
Definitions: type-translation diagnostics describe
restricted dataclasses without custom initialization; the structural hook in
named collections checks field metadata but does not directly prove
all custom-init exclusions. Locally defined collection classes are explicitly
xfailed in named tests because lowering requires module-qualified
resolution. where/concat_where accept named products, whereas astype’s
frontend signature is limited to scalars, fields, and tuples.
Interpretation: use module-resolvable data-only named collections satisfying
the documented restrictions. Do not extend one builtin’s structure support to
all others, or infer arbitrary dataclass support from incomplete rejection.
C20. Compilation annotations versus executable precompilation
Definitions: decorators annotate provider arguments with runtime
tables or provider type descriptors. The compiled pool
(_compile_variant, compile) explicitly rejects nonempty descriptor-only mappings.
It also rejects direct scan precompilation without argument types; the public
.compile() path supplies none, and compiled-program tests
explicitly xfail scan precompilation. Interpretation: use runtime table
mappings (or {} when no tables are needed), including configured connectivities.
Compile a concretely annotated program wrapping a scan for ahead-of-time use;
direct scan invocation/JIT remains available. Broader annotations or inherited
methods do not establish implementation support.
C21. Captured type rules versus embedded host-call validation
Definitions: frontend typing enforces captured-call compatibility, but direct embedded decorator calls pass actual values to the original Python function. Embedded assignment does not consistently check collection types and can cast values into output storage. Program host calls have a separate debug validation path. Interpretation: exact matching remains a caller obligation; accidental embedded acceptance is not implicit coercion or a supported relaxation of the DSL type rules. Do not promise uniform runtime rejection or error timing for invalid direct host calls.
C22. Omitted versus explicit None operator domains
Definitions: embedded calls treat domain=None like omission.
Compiled host calls in decorators convert any explicitly supplied
domain through domain conversion, which rejects None.
Interpretation: omit the keyword to request output-derived domains.
Document explicit None as an embedded-only convenience, not a common default
value accepted by all operator entry points.
9. Evidence coverage and focused observations
9.1 Documentation and examples reviewed
| Repository source | Knowledge used / boundary |
|---|---|
| Quickstart | Fields, pure operators, imperative programs, connectivities, local reductions, where, tuples, pseudo-Laplacian; wording conflicts C3/C4 |
| WorkflowPatterns, HackTheToolchain | Toolchain stages, configurable/cached workflows, builders; host infrastructure rather than new DSL syntax |
| Workshop slides 1–4 | Fields/decorators/programs; Cartesian and neighbor shifts/reductions; where/domains; scan recurrence and composition |
| Workshop exercises and solutions 1–5 | Addition, divergence, gradient, curl, vector Laplace; array/field conversion, orientation/local coefficients, provider wiring |
| Workshop exercises and solutions 6–8 | Where/domain halos, tuple scan carry, diffusion composition; exercises intentionally incomplete, solutions supply intent |
| Workshop helpers, Jupyter introduction | Host mesh/allocation helpers and notebook usage; no additional language productions |
| Cartesian/Next Laplacian comparison | Named Cartesian shifts and storage interoperability in the example, not syntax compatibility between the two DSLs |
| Diagnostic guide | Error vocabulary, rejected-construct hints, source-location expectations |
All 21 Next-relevant notebooks (workshop slides, exercises/solutions, Jupyter
introduction, and the Laplacian comparison) had cell sources inspected. They
were not rerun in full. examples/cartesian/* belongs to the separate Cartesian
DSL and is not normative evidence for Next syntax.
The Next ADR collection was reviewed as follows:
- Direct semantic evidence: 0001 (frontend), 0010 (domains), 0013 (scalar/0d), 0019 (connectivities), 0020 (runtime domains), 0022 (embedded concat limits), 0026 (staggering), cross-checked against current implementation/tests.
- Internal-language history: 0002 (deprecated lowering), 0003/0004 (tuple representation), 0005 (IR extensions), 0006/0008 (C++ indices/domains), 0007 (deprecated processors), 0009 (superseded backend integration). Their primitives and historical restrictions do not enlarge/restrict the current Python language without current evidence.
- Execution/host boundary: 0011/0012 (OTF), 0014 (DaCe), 0015 (test exclusions), 0016/0017 (backend/configuration), 0018 (SDFG transforms), 0021 (argument descriptors), 0023 (fingerprints), 0024 (runners), 0025 (cache), 0027 (workspace), 0028 (builders). Prototype restrictions there are not treated as current universal frontend limitations.
9.2 Implementation and test coverage
This index records investigation coverage, not a claim that every test in these trees was executed. Low-level iterator tests were consulted only as supporting implementation evidence; their complete internal language is out of scope.
9.3 Focused cross-mode and frontend probes
The following small cases were run against this checkout with uv, Python
3.12.13, NumPy embedded fields, and the itir_python lowered backend. Native
C++/GPU/DaCe variants were not exercised. No runtime code was changed.
For default tuple-output domains, the essential case is:
import numpy as np
import gt4py.next as gtx
I = gtx.Dimension("I")
@gtx.field_operator
def two(a: gtx.Field[gtx.Dims[I], gtx.int32]):
return a + 1, a + 2
if __name__ == "__main__":
for backend in (None, gtx.itir_python):
a = gtx.as_field([I], np.arange(5, dtype=np.int32))
x = gtx.as_field([I], np.full(5, -7, dtype=np.int32))
y = gtx.as_field([I], np.full(3, -7, dtype=np.int32))
two.with_backend(backend)(a, out=(x, y))
print(x.asnumpy(), y.asnumpy())| Mode | x after call | y after call |
|---|---|---|
| Embedded | [1, 2, 3, -7, -7] | [2, 3, 4] |
itir_python | [1, 2, 3, 4, 5] | [2, 3, 4] |
A second operator demonstrates scalar/0d compatibility without uniform wrapping:
@gtx.field_operator
def complement(x: gtx.Field[gtx.Dims[()], bool]) -> gtx.Field[gtx.Dims[()], bool]:
return ~x
# In the guarded entry point above, for each backend:
# out = gtx.as_field([], np.array(False))
# complement.with_backend(backend)(True, out=out)Embedded assignment produces True, whereas itir_python produces False.
Passing an actual bool Field avoids the Python-scalar inversion involved here.
A third probe constructs a neighbor table with entries [[0,-99],[1,2]] and
skip_value=-99: construction succeeds, then direct embedded remapping raises
AssertionError.
Five focused parser/lowering probes additionally established:
float64[I] ** int32[J]is accepted with deduced resultfloat64[I];- floating field to integral scalar exponent is accepted;
float64("inf")is accepted and lowered;- integer field
/ 2retains integer inferred dtype; - augmented assignment (
+=) is rejected as unsupported syntax.
These observations substantiate the conflict register; they are not an all-backend conformance suite. The remaining gaps are explicit in sections 6.3 and 8: arbitrary aliasing, extreme/empty reductions, complete numeric edge semantics, closure invalidation, richer signatures, and dimension-changing remap combinations.