Skip to content

Commit

Permalink
Merge pull request #519 from fossell/update_v11_2D_docs
Browse files Browse the repository at this point in the history
Update v11 2-d docs
  • Loading branch information
fossell authored Jun 14, 2022
2 parents 79f68d9 + bb8ed1a commit 6b5c589
Show file tree
Hide file tree
Showing 2 changed files with 58 additions and 9 deletions.
48 changes: 47 additions & 1 deletion docs/2D-decomp.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,51 @@ The structural changes needed for the 2D decomposition are implemented in MPI_FI

Both MPI_FIRST.f and EXCH.f contain significant additional test code to exchange arrays containing grid coordinates and ensure EXACT matches for all exchanges before the domain exchanges are performed. This is intended to trap errors in the larger variety of 2D decomposition layouts that are possible and most of it can eventually be removed or made conditional at build and run time.

Indices and variables to facilitate the 2D decomposition are found in CTLBLK.f and shared in the rest of UPP through use of CTLBLK.mod.
The following is found in CTLBLK.f and shared in the rest of UPP through use of CTLBLK.mod:

| Variable | Type | Description |
|---------|------|-------------|
| im | integer | full longitude domain|
| jm | integer | full latitude domain|
| | | |
| jsta | integer | start latitude on a task subdomain|
| jend | integer | end latitude on a task subdomain|
| ista | integer | start longitude on a task subdomain|
| iend | integer | end longitude on a task subdomain|
| | | |
| ista_2l | integer |start longitude -2 of the subdomain|
| iend_2u | integer |end longitude +2 of the subdomain|
| jsta_2l | integer |start latitude -2 of the subdomain|
| jend_2u | integer |end latitude +2 of the subdomain|

The shape of the subdomain is ista_2l:iend_2u,jsta_2l:jend_2u so it includes the halos although the halos are not populated until exchange is done in EXCH.f. Because of halos we need more bounds defined:

| Variable | Type | Description |
|---------|------|-------------|
| jsta_m | integer | Beginning latitude loop index in subdomain for halo depth 1 |
| jend_m | integer | ending latitude loop index in subdomain for halo depth 1 |
| jsta_m2 | integer | second latitude below begin latitude of subdomain for halo depth 2 (in NGMFLD.f) |
| jend_m2 | integer | second latitude above end latitude of subdomain for halo depth 2 ( in NGMFLD.f) |

Note:<ul><li>In interior subdomains these are the same as jsta and jend.</li><li>In boundary subdomains these loop indices define a smaller subset of the subdomain since halos are not defined on the full domain boundaries and stencils must be restricted to valid full domain points. </li></ul>

| Variable | Type | Description |
|---------|------|-------------|
| ista_m | integer | begining longitude loop index in subdomain for halo depth 1|
| iend_m | integer | end longitude loop index in subdomain for halo depth 1 |
| ista_m2 | integer | second longitude before begin longitude for halo depth 2 (not used as of 6/22)|
| iend_m2 | integer | second longitude after end longitude for halo depth 2 (not used as of 6/22) |

Note:<ul><li>In interior subdomains these are the same as ista and iend.</li><li>In boundary subdomains these loop indices define a smaller subset of the subdomain since halos are not defined on the full domain boundaries and stencils must be restricted to valid full domain points.</li></ul>


| Variable | Type | Description |
|---------|------|-------------|
| ileft | integer | MPI rank containing the last longitude before ista
| iright | integer | MPI rank containing the first longitude after iend
| iup | integer | MPI rank containing the first latitude after jend
| idn | integer | MPI rank containing the last latitude before jsta
| | | |
| ileftb | integer |MPI rank containing the last longitude before ista but for cyclic boundary conditions where "last" at the beginning is the other end of the domain (apparently unused and replaced with local calculation) |
| irightb | integer | MPI rank containing the first longitude after iend but for cyclic boundary conditions where "first" at the beginning is the other end of the domain (apparently unused and replaced with local calculation) |

19 changes: 11 additions & 8 deletions sorc/ncep_post.fd/MPI_FIRST.f
Original file line number Diff line number Diff line change
Expand Up @@ -130,14 +130,17 @@ SUBROUTINE MPI_FIRST()

! global loop ranges
!
! para_range2 supports a 2D decomposition. The rest of the post
! supports 1D still and the call here is the special case where each
! processor gets all of the longitudes in the latitude 1D subdomain
! jsta:jend. The X decomposition will be specified by the third
! argument (currently 1) and the Y decoposition will be specified by
! the fourth argument (currently all of the ranks) When X is
! subdivided the third and fourth arguments will have to be integral
! factors of num_procs
! para_range2 supports a 2D decomposition.
! The X decomposition is specified by the third
! argument and the Y decoposition is specified by
! the fourth argument. The product of the third and fourth arguments
! must be num_procs and the third and fourth arguments must be integral
! factors of num_procs.
!
! for the special case of 1D decomposition, numx is set to 1 and the
! fourth argument becomes the number of MPI ranks for the job. numx=1
! makes the code fully compatible with the old 1D decomposition.


call para_range2(im,jm,numx,num_procs/numx,me,ista,iend,jsta,jend)

Expand Down

0 comments on commit 6b5c589

Please sign in to comment.