Handling repated loop variables

[thilinarmtb]

Hi All,

I am porting CEED BP5 (which is solving pressure Poisson equation using CG) with
nomp to GPUs. I am running into a case where there are repeated loop variables in C
kernels like the following gather-scatter kernel:

#define scalar double
#define uint unsigned

inline static void gs(scalar *v, const uint *gs_off, const uint *gs_idx,
                      const uint gs_n) {
#pragma nomp for transform("bp5", "gs")
  for (uint i = 0; i < gs_n; i++) {
    scalar s = 0;
    for (uint j = gs_off[i]; j < gs_off[i + 1]; j++)
      s += v[gs_idx[j]];
    for (uint j = gs_off[i]; j < gs_off[i + 1]; j++)
      v[gs_idx[j]] = s;
  }
}

I am splitting i and tagging the resulting inames as g.0 and l.0. Both the j
loops are serial loops.

If I use the above C-code as is, it is an error in nomp right now since the variable
name j is repeated. As a workaround, I have renamed the second j loop to k
but I think this needs a better solution.

For example in the Ax kernel (which is the poisson operator), there will be about 12
loop variables if the user had to rename the loops:
https://github.com/nomp-org/benchmarks/blob/main/CEED/BP5/backends/bp5-backend-nomp.c#L126

Only solution we (@kaushikcfd and I) could think of other than asking the user to
use different loop variables for distinct loops is to assume the same loop variable
in C implies the same loop domain in loopy (so basically, we are fusing the loops
with same loop variables).

For the SEM/FEM code, I think it is an okay solution. We can document this as how
nomp interprets code and ask the user to use different loop variables if they are
actually different and wants to avoid fusing.

We would love to hear from the loopy community if there is a better solution to
resolve this matter in a more elegant manner.

[inducer]

What you report is by design, where the goal is that, for transformation purposes (e.g. split_iname), loops are uniquely identified by their name. Unfortunately, that's at odds with what you're asking for here, which is being able to reuse the same loop variable for multiple loops.

As far as I'm concerned, there's no easy win here: either loop identifiers become more complicated, or we disallow their repeated use. Fusing loops behind the user's back is not a good idea IMO, as it's quite likely to lead to non-equivalent code.

The renaming solution you propose is also what I would call the "official" remedy. It is also what the "Fortran frontend" does under the hood, see this code here:

loopy/loopy/frontend/fortran/translator.py

Lines 765 to 787 in bf3091b

	# {{{ find a usable loopy-side loop name
	loopy_loop_var = loop_var
	loop_var_suffix = None
	while True:
	already_used = False
	for iset in scope.index_sets:
	if loopy_loop_var in iset.get_var_dict(dim_type.set):
	already_used = True
	break
	if not already_used:
	break
	if loop_var_suffix is None:
	loop_var_suffix = 0
	loop_var_suffix += 1
	loopy_loop_var = loop_var + "_%d" % loop_var_suffix
	loopy_loop_var = intern(loopy_loop_var)
	# }}}

[thilinarmtb]

Thank you for the reply, insights and the code!

I think I can do something similar to Fortran frontend with nomp as well by adding a suffix.
I just found in loopy docs that lp.tag_inames() supports wildcards, so user actually can
use that to tag all j_* names with the same axis easily. I think it is a good solution rather
than fusing all the loops with the same loop variable.