For developers - Revision history

Yjzhang at 17:04, 18 December 2014

2014-12-18T17:04:11Z

Yjzhang at 17:01, 18 December 2014

2014-12-18T17:01:33Z

Yjzhang: /* Making sense of MPI inside SELFE */

2014-10-11T11:04:48Z

‎Making sense of MPI inside SELFE

Yjzhang: /* Making sense of MPI inside SELFE */

2014-10-09T19:39:48Z

‎Making sense of MPI inside SELFE

Yjzhang: /* Making sense of MPI inside SELFE */

2014-10-09T19:39:31Z

‎Making sense of MPI inside SELFE

Yjzhang: /* Making sense of MPI inside SELFE */

2014-10-09T19:39:21Z

‎Making sense of MPI inside SELFE

Yjzhang: /* Making sense of MPI inside SELFE */

2014-10-09T19:38:47Z

‎Making sense of MPI inside SELFE

Yjzhang: /* Making sense of MPI inside SELFE */

2014-10-09T19:34:29Z

‎Making sense of MPI inside SELFE

Yjzhang: /* Making sense of MPI inside SELFE */

2014-10-09T09:32:01Z

‎Making sense of MPI inside SELFE

Yjzhang: /* Making sense of MPI inside SELFE */

2014-10-09T09:29:30Z

‎Making sense of MPI inside SELFE

@@ Line 69: / Line 69: @@
 Consider this figure:
-[[File:SELFEwiki-domain-decomp-ghost.jpg|600px|Fig. 1: Domain decomposition in SCHISM]]
+[[File:SCHISMwiki-domain-decomp-ghost.png|600px|Fig. 1: Domain decomposition in SCHISM]]
 For example, you want to do averaging at each node of the sub-domain around its ball of elements.

@@ Line 1: / Line 1: @@
 All svn account holders need to familiarize themselves with the following rules.
-==Rules for contributing your code to be included in the SELFE package==
+==Rules for contributing your code to be included in the SCHISM package==
 Here are some rules for preparing your own code:
@@ Line 8: / Line 8: @@
 *Try to use the I/O channel number directly, e.g., read(61, etc instead of assigning a number to a variable (e.g. read(ich,). This'd facilitate others searching for conflicts.
-==Sharing I/O channels in SELFE==
+==Sharing I/O channels in SCHISM==
 You need to exercise caution when dealing with parallel I/O especially for writing. For writing outputs, you’d generally let only 1 process do the job, e.g.
@@ Line 16: / Line 16: @@
 If you do need to have all processes write e.g. debug messages, you’d then use channel 12 (see below).
-Below are all I/O channel numbers used in different sub-models of SELFE (bold numbers mean they have been used by other sub-models and you’d avoid using them). Please add your numbers as soon as you change the code so as to prevent others from using it. A good way to find out if a number is available is to issue the following cmd from src/:
+Below are all I/O channel numbers used in different sub-models of SCHISM (bold numbers mean they have been used by other sub-models and you’d avoid using them). Please add your numbers as soon as you change the code so as to prevent others from using it. A good way to find out if a number is available is to issue the following cmd from src/:
 grep "61" */*.F90  -->Looks for '61'
@@ Line 41: / Line 41: @@
 If you need to profile some parts of the code, you can use the custom timer: timer_ns(). Here are the steps to set up a new timer:
-*In Core/elfe_glbl.F90: increase the dimension of timer_ns() for your number of timers needed;
+*In Core/schism_glbl.F90: increase the dimension of timer_ns() for your number of timers needed;
 *Insert the following segments around the portion of the code you want to profile:
@@ Line 63: / Line 63: @@
 On many clusters, more sophisticated profilers exist. Check your local system.
-==Making sense of MPI inside SELFE==
+==Making sense of MPI inside SCHISM==
-If you are working on the code you may be confused about the exchanges inside SELFE. When should you  use these?
+If you are working on the code you may be confused about the exchanges inside SCHISM. When should you  use these?
 The first thing you need to remember when writing MPI code with domain decomposition is that a rank (or MPI 'process') only has info in its 'augmented' (=resident + ghost) domain, and knows absolutely ''nothing'' outside this region.
 Consider this figure:
-[[File:SELFEwiki-domain-decomp-ghost.jpg|600px|Fig. 1: Domain decomposition in SELFE]]
+[[File:SELFEwiki-domain-decomp-ghost.jpg|600px|Fig. 1: Domain decomposition in SCHISM]]
 For example, you want to do averaging at each node of the sub-domain around its ball of elements.

@@ Line 66: / Line 66: @@
 If you are working on the code you may be confused about the exchanges inside SELFE. When should you  use these?
-The first thing you need to remember when writing MPI code with domain decomposition is that a rank (or MPI 'process') only has info in its 'augmented' domain, and knows absolutely ''nothing'' outside this region.
+The first thing you need to remember when writing MPI code with domain decomposition is that a rank (or MPI 'process') only has info in its 'augmented' (=resident + ghost) domain, and knows absolutely ''nothing'' outside this region.
 Consider this figure:
@@ Line 92: / Line 92: @@
 However, for a ghost node 'Q', some surrounding elements are located outside the augmented  domain (and in this case, indel() are actually negative!), and so if you use the same operation, erroneous values at ghost nodes would be generated. Therefore you cannot use npa in the outer loop.
-Now, after this loop is executed, what happens to the ghost nodes? Since they are excluded, the 'stokes_nd' will be wrong there. Fortunately, some neighboring ranks have the correct values for these nodes, which are resident nodes in those neighboring ranks. So now different ranks will have different values at some nodes, and this is to be avoided. In order to make sure each rank has correct (and up-to-date) values in its augmented (=resident + ghost) domain, you need to follow this loop with an exchange routine. Remember that for the exchange routines to work, you need to define the array in the augmented domain:
+Now, after this loop is executed, what happens to the ghost nodes like 'Q'? Since they are excluded, the 'stokes_nd' will be wrong there. Fortunately, some neighboring ranks have the correct values for these nodes, which are resident nodes in those neighboring ranks; e.g., 'Q' is a resident node of Rank 0. So now different ranks will have different values at some nodes, and this is to be avoided. In order to make sure each rank has correct (and up-to-date) values in its augmented  domain, you need to follow this loop with an exchange routine. Remember that for the exchange routines to work, you need to define the exchanged array in the augmented domain:
 allocate(stokes_nd(npa)) !not 'np'
@@ Line 98: / Line 98: @@
 in order to allow each rank to receive info in the ghost zone.
-The above is just one example of when exchanges are needed. Usually this involves queries into a neighborhood, but beware that there are other circumstances where exchanges are necessary. The most important thing to remember is, again,
+The above is just one example of when exchanges are needed. Usually this involves some kind of queries into a neighborhood, but beware that there are other circumstances where exchanges are necessary. The most important thing to remember is, again,
 '''A rank only has info in its 'augmented' domain, and knows absolutely ''nothing'' outside this region.'''

@@ Line 69: / Line 69: @@
 Consider this figure:
-[[File:SELFEwiki-domain-decomp-ghost.jpg|300px|Fig. 1: Domain decomposition in SELFE]]
+[[File:SELFEwiki-domain-decomp-ghost.jpg|600px|Fig. 1: Domain decomposition in SELFE]]
 For example, you want to do averaging at each node of the sub-domain around its ball of elements.

@@ Line 69: / Line 69: @@
 Consider this figure:
-[[File:SELFEwiki-domain-decomp-ghost.jpg|100px|Fig. 1: Domain decomposition in SELFE]]
+[[File:SELFEwiki-domain-decomp-ghost.jpg|300px|Fig. 1: Domain decomposition in SELFE]]
 For example, you want to do averaging at each node of the sub-domain around its ball of elements.

← Older revision		Revision as of 09:32, 9 October 2014
Line 112:		Line 112:
	enddo !i		enddo !i

−	Since all ranks have info on 'cde' and 'area' locally available, 'qdl_e' will be correctly calculated even inside the ghost zones. So in this case you'd use 'nea' instead of 'ne' in the loop. Of course, there is nothing wrong with using 'ne' in the loop followed by an exchange routine, but doing so would be less efficient and incur unnecessary communication cost.	+	Since all ranks have info of 'cde' and 'area' locally available, 'qdl_e' will be correctly calculated even inside the ghost zones, and its values are the same cross ranks there (since 'cde' and 'area' are same cross ranks in the overlapping zones). So in this case you'd use 'nea' instead of 'ne' in the loop. Of course, there is nothing wrong with using 'ne' in the loop followed by an exchange routine, but doing so would be less efficient and incur unnecessary communication cost.

@@ Line 100: / Line 100: @@
 The above is just one example of when exchanges are needed. Usually this involves queries into a neighborhood, but beware that there are other circumstances where exchanges are necessary. The most important thing to remember is, again,
-'''A rank only has info in its 'augmented' domain, and knows absolutely ''nothing'' outside this region.
+'''A rank only has info in its 'augmented' domain, and knows absolutely ''nothing'' outside this region.'''
-   In MPI code, it's crucial to make sure all ranks have identical values in overlapping zone.
+'''In MPI code, it's crucial to make sure all ranks have identical values in overlapping zone.'''
 Now let's consider an example where no exchanges are needed. Say you want to simply go thru all elements and update some arrays: