Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

parallel distributed div av #754

Open
anderson2981 opened this issue Sep 19, 2022 · 2 comments
Open

parallel distributed div av #754

anderson2981 opened this issue Sep 19, 2022 · 2 comments
Assignees
@matthiasdiener
Labels
Lazy-eval

Comments

@anderson2981
Copy link
Contributor

anderson2981 commented Sep 19, 2022 •
edited by MTCam
Loading

When running the isolator driver with the new artificial dissipation operator in parallel I get the following errors:

production mirgecom using the new_art_visc branch of the isolator driver

To reproduce (with isolator driver):

mpirun -n 2 python -u -O -m mpi4py isolator_injection_init.py -i run_params_scalar.yaml --lazy

mpirun -n 2 python -u -O -m mpi4py isolator_injection_run.py -i run_params_scalar.yaml -r restart_data/isolator_init-000000 --log --lazy

This error can also be reproduced with a physical AV example from mirgecom@production:
mpiexec -n 2 python -u -O -m mpi4py doublemach_physical_av-mpi.py --lazy

The error:

`Traceback (most recent call last):
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/arraycontext/arraycontext/impl/pytato/__init__.py", line 390, in freeze
    pt_prg = self._freeze_prg_cache[normalized_expr]
KeyError: DictOfNamedArrays({'_ary': IndexLambda(expr=Product((Subscript(Variable('_in0'), (Variable('_0'), Variable('_1'))), Subscript(Variable('_in1'), (Variable('_0'), 0)))), shape=(26800, 6), dtype='float64', bindings={'_in0': IndexLambda(expr=Product((Subscript(Variable('_in0'), (Variable('_0'), Variable('_1'))), 1.5)), shape=(26800, 6), dtype='float64', bindings={'_in0': IndexLambda(expr=Quotient(Subscript(Variable('_in0'), (Variable('_0'), Variable('_1'))), 100), shape=(26800, 6), dtype='float64', bindings={'_in0': IndexLambda(expr=Call(Variable('pytato.c99.log'), (Subscript(Variable('in_0'), (Variable('_0'), Variable('_1'))),)), shape=(26800, 6), dtype='float64', bindings={'in_0': (...)})})}), '_in1': Placeholder(axes=(Axis(tags=frozenset({DiscretizationElementAxisTag()})), Axis(tags=frozenset())), tags={PrefixNamed(prefix='char_lscales')}, shape=(26800, 1), dtype='float64', name='_actx_dw')})})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 138, in __call__
    method = getattr(self, expr.mapper_method)
AttributeError: 'tuple' object has no attribute 'mapper_method'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 138, in __call__
    method = getattr(self, expr.mapper_method)
AttributeError: 'tuple' object has no attribute 'mapper_method'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 150, in rec
    method = getattr(self, expr._mapper_method)
AttributeError: 'CodeGenMapper' object has no attribute 'map_distributed_send_ref_holder'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/someguy/software/Install/Conda/envs/mirgeDriver.Y2isolator/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/someguy/software/Install/Conda/envs/mirgeDriver.Y2isolator/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/someguy/software/Install/Conda/envs/mirgeDriver.Y2isolator/lib/python3.9/site-packages/mpi4py/__main__.py", line 7, in <module>
    main()
  File "/Users/someguy/software/Install/Conda/envs/mirgeDriver.Y2isolator/lib/python3.9/site-packages/mpi4py/run.py", line 198, in main
    run_command_line(args)
  File "/Users/someguy/software/Install/Conda/envs/mirgeDriver.Y2isolator/lib/python3.9/site-packages/mpi4py/run.py", line 47, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "/Users/someguy/software/Install/Conda/envs/mirgeDriver.Y2isolator/lib/python3.9/runpy.py", line 288, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Users/someguy/software/Install/Conda/envs/mirgeDriver.Y2isolator/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/someguy/software/Install/Conda/envs/mirgeDriver.Y2isolator/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "isolator_injection_run.py", line 1750, in <module>
    main(restart_filename=restart_filename, target_filename=target_filename,
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/mirgecom/mirgecom/mpi.py", line 157, in wrapped_func
    func(*args, **kwargs)
  File "isolator_injection_run.py", line 1615, in main
    advance_state(rhs=my_rhs, timestepper=timestepper,
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/mirgecom/mirgecom/steppers.py", line 425, in advance_state
    _advance_state_stepper_func(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/mirgecom/mirgecom/steppers.py", line 154, in _advance_state_stepper_func
    state, dt = pre_step_callback(state=state, step=istep, t=t, dt=dt)
  File "isolator_injection_run.py", line 1448, in my_pre_step
    fluid_state = create_fluid_state(cv=cv,
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/arraycontext/arraycontext/impl/pytato/compile.py", line 328, in __call__
    return compiled_f(arg_id_to_arg)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/grudge/grudge/array_context.py", line 359, in __call__
    input_args_for_prg = _args_to_device_buffers(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/arraycontext/arraycontext/impl/pytato/compile.py", line 554, in _args_to_device_buffers
    arg = actx.freeze(arg)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/arraycontext/arraycontext/impl/pytato/__init__.py", line 415, in freeze
    pt_prg = pt.generate_loopy(transformed_dag,
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 983, in generate_loopy
    insn_id = add_store(name, expr, cg_mapper(expr, state), state, cg_mapper)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in map_product
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in map_product
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 516, in map_quotient
    numerator = self.rec(expr.numerator, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 626, in map_call
    self.rec(expr.parameters, prstnt_ctx, local_ctx))
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 151, in __call__
    return self.map_foreign(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 206, in map_foreign
    return self.map_tuple(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 583, in map_tuple
    children = tuple([self.rec(child, *args, **kwargs) for child in expr])
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 583, in <listcomp>
    children = tuple([self.rec(child, *args, **kwargs) for child in expr])
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in map_sum
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 626, in map_call
    self.rec(expr.parameters, prstnt_ctx, local_ctx))
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 151, in __call__
    return self.map_foreign(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 206, in map_foreign
    return self.map_tuple(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 583, in map_tuple
    children = tuple([self.rec(child, *args, **kwargs) for child in expr])
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 583, in <listcomp>
    children = tuple([self.rec(child, *args, **kwargs) for child in expr])
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in map_product
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in map_sum
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 516, in map_quotient
    numerator = self.rec(expr.numerator, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in map_product
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in map_sum
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 516, in map_quotient
    numerator = self.rec(expr.numerator, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in map_sum
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in map_product
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 652, in map_reduce
    inner_expr = self.rec(inner_expr, prstnt_ctx,
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in map_product
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in map_sum
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in map_product
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
2022-09-19 10:57:18,457 - INFO - meshmode.array_context - Transforming kernel with 2 statements.
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 652, in map_reduce
    inner_expr = self.rec(inner_expr, prstnt_ctx,
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in map_product
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in map_sum
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in map_sum
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in map_sum
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 654, in map_if
    then = self.rec(expr.then, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in map_product
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 507, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 516, in map_quotient
    numerator = self.rec(expr.numerator, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in map_sum
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in map_sum
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 498, in <listcomp>
    children = [self.rec(child, *args, **kwargs) for child in expr.children]
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 165, in rec
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 390, in map_index_lambda
    loopy_expr = self.exprgen_mapper(expr.expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 584, in __call__
    return self.rec(expr, prstnt_ctx, local_ctx)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pymbolic/pymbolic/mapper/__init__.py", line 153, in __call__
    return method(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/target/loopy/codegen.py", line 591, in map_subscript
    result: ImplementedResult = self.codegen_mapper(
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 169, in __call__
    return self.rec(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 160, in rec
    return self.handle_unsupported_array(expr, *args, **kwargs)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/pytato/pytato/transform/__init__.py", line 135, in handle_unsupported_array
    raise UnsupportedArrayError("%s cannot handle expressions of type %s"
pytato.transform.UnsupportedArrayError: CodeGenMapper cannot handle expressions of type <class 'pytato.distributed.DistributedSendRefHolder'>`
@inducer
Copy link
Contributor

inducer commented Sep 19, 2022

The issue appears to be that a DAG that includes MPI communication is being passed to freeze:

  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/arraycontext/arraycontext/impl/pytato/compile.py", line 554, in _args_to_device_buffers
    arg = actx.freeze(arg)
  File "/Users/someguy/work/CEESD/MirgeCom/Drivers/CEESD-Y2_isolator/emirge/arraycontext/arraycontext/impl/pytato/__init__.py", line 415, in freeze
    pt_prg = pt.generate_loopy(transformed_dag,

This is something we could support, but we currently do not.

This was referenced Sep 19, 2022
Closed
Merged
@MTCam
Copy link
Member

MTCam commented Sep 20, 2022

The issue appears to be that a DAG that includes MPI communication is being passed to freeze:

This particular error comes from the fact that we are using grad_cv (the gradient of the conserved vars) to compute the smoothness, or to just visualize the gradient itself for debugging. Apparently, I've never tried to do this when running distributed before now. Indeed, even nsmix-mpi.py in mirgecom@production fails to run distributed when it tries to compute and viz the gradients. I'm shocked we haven't encountered this problem before now.

Is there any way we can get around this @inducer?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@matthiasdiener matthiasdiener

Labels
Lazy-eval
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@inducer @matthiasdiener @MTCam @anderson2981