Debugging after a run crashes - Revision history

Yjzhang at 14:43, 30 June 2014

2014-06-30T14:43:00Z

Yjzhang at 18:08, 17 September 2013

2013-09-17T18:08:51Z

Yjzhang at 01:03, 16 September 2013

2013-09-16T01:03:35Z

Yjzhang at 01:02, 16 September 2013

2013-09-16T01:02:50Z

Yjzhang at 01:00, 16 September 2013

2013-09-16T01:00:00Z

Yjzhang: Created page with "aa"

2013-09-16T00:49:34Z

Created page with "aa"

New page

← Older revision		Revision as of 14:43, 30 June 2014
Line 1:		Line 1:
	The best way to find out the reason for a crash is to visualize the surface velocity with ACE/xmvis6. Usually you'll see some large/noisy velocity somewhere, which may give you some hints on forcing etc.		The best way to find out the reason for a crash is to visualize the surface velocity with ACE/xmvis6. Usually you'll see some large/noisy velocity somewhere, which may give you some hints on forcing etc.

−	Sometimes you want to visualize the problem right before the crash. ~~Here~~ is ~~the way using the hotstart option~~.	+	Sometimes you want to visualize the problem right before the crash. You cannot use autocombine_MPI_elfe.pl as the last stack of output is incomplete. But you can use the core FORTRAN combine script (e.g., combine_output6) to directly combine an incomplete stack. Just follow the instruction in the header of combine_output6.f90 to prepare the inputs and run. Then visualize the combined outputs with xmvis6.
−
−	~~Suppose you run crashed right after time step it=1005 (~~you can ~~find out this in mirror.out; note that "TIME STEP= " is written AFTER a step is completed), and~~ the ~~closest hotstart output~~ (~~in outputs/) has a step of 900.~~
−
−	~~First save any outputs that may be overwritten upon ihot=2:~~
−	~~<UL>~~
−	~~<LI>mv mirror~~.~~out mirror~~.~~out.0~~
−	~~<LI>mv hotstart.in hotstart.in.0~~
−	~~<LI>mv outputs outputs.0~~
−	~~<LI>mkdir outputs~~
−	~~</UL>~~
−	~~....~~
−
−	~~The third move is necessary as we are going~~ to ~~change the~~ stack ~~size (ihfskip).~~
−
−	Combine hotstart outputs at it=900 using combine_hotstart*.~~f90 to generate a new hotstart.in, and then move it to~~ the ~~same dir as hgrid.gr3.~~
−	~~Then set ihot=2 in param.~~in~~. Also set ihfskip, and hotout_write to 1005, and nspool to a number that is divisible by 1005. Start~~ the ~~run with same number~~ of ~~CPUs~~. ~~Occasionally,~~ the ~~hotstarted~~ run ~~will crash at a different step, say 1006, and if this is the case, reset ihfskip, and hotout_write to 1006 (and nspool accordingly) and redo it~~. ~~The 2nd time should work.~~
−
−	~~You'll see 2 stacks coming out after~~ the ~~crash. Combine the 1st stack and then viz~~.

@@ Line 17: / Line 17: @@
 Combine hotstart outputs at it=900 using combine_hotstart*.f90 to generate a new hotstart.in, and then move it to the same dir as hgrid.gr3.
-Then set ihot=2 in param.in. Also set nspool and ihfskip, and hotout_write to 1005. Start the run with same number of CPUs. Occasionally, the hotstarted run will crash at a different step, say 1006, and if this is the case, reset nspool and ihfskip, and hotout_write to 1006 and redo it. The 2nd time should work.
+Then set ihot=2 in param.in. Also set ihfskip, and hotout_write to 1005, and nspool to a number that is divisible by 1005. Start the run with same number of CPUs. Occasionally, the hotstarted run will crash at a different step, say 1006, and if this is the case, reset ihfskip, and hotout_write to 1006 (and nspool accordingly) and redo it. The 2nd time should work.
 You'll see 2 stacks coming out after the crash. Combine the 1st stack and then viz.

@@ Line 6: / Line 6: @@
 First save any outputs that may be overwritten upon ihot=2:
+<UL>
-mv mirror.out mirror.out.0
+  <LI>mv mirror.out mirror.out.0
-mv hotstart.in hotstart.in.0
+  <LI>mv hotstart.in hotstart.in.0
-mv outputs outputs.0
+  <LI>mv outputs outputs.0
-mkdir outputs
+  <LI>mkdir outputs
 ....

@@ Line 3: / Line 3: @@
 Sometimes you want to visualize the problem right before the crash. Here is the way using the hotstart option.
-Suppose you run crashed at time step it=1005 (you can find out this in mirror.out), and the closest hotstart output (in outputs) has a step of 900. First save any outputs that may be overwritten upon ihot=2:
+Suppose you run crashed right after time step it=1005 (you can find out this in mirror.out; note that "TIME STEP= " is written AFTER a step is completed), and the closest hotstart output (in outputs/) has a step of 900.
 mv mirror.out mirror.out.0
@@ Line 11: / Line 13: @@
 ....
-The 2nd move is necessary as we are going to change the stack size (ihfskip).
+The third move is necessary as we are going to change the stack size (ihfskip).
-First, combine hotstart outputs at it=900 using combine_hotstart*.f90 to generate a new hotstart.in, and then move it to the same dir as hgrid.gr3.
+Combine hotstart outputs at it=900 using combine_hotstart*.f90 to generate a new hotstart.in, and then move it to the same dir as hgrid.gr3.
 Then set ihot=2 in param.in. Also set nspool and ihfskip, and hotout_write to 1005. Start the run with same number of CPUs. Occasionally, the hotstarted run will crash at a different step, say 1006, and if this is the case, reset nspool and ihfskip, and hotout_write to 1006 and redo it. The 2nd time should work.
 You'll see 2 stacks coming out after the crash. Combine the 1st stack and then viz.

← Older revision		Revision as of 01:00, 16 September 2013
Line 1:		Line 1:
−	aa	+	The best way to find out the reason for a crash is to visualize the surface velocity with ACE/xmvis6. Usually you'll see some large/noisy velocity somewhere, which may give you some hints on forcing etc.
		+
		+	Sometimes you want to visualize the problem right before the crash. Here is the way using the hotstart option.
		+
		+	Suppose you run crashed at time step it=1005 (you can find out this in mirror.out), and the closest hotstart output (in outputs) has a step of 900. First save any outputs that may be overwritten upon ihot=2:
		+
		+	mv mirror.out mirror.out.0
		+	mv hotstart.in hotstart.in.0
		+	mv outputs outputs.0
		+	mkdir outputs
		+	....
		+
		+	The 2nd move is necessary as we are going to change the stack size (ihfskip).
		+
		+	First, combine hotstart outputs at it=900 using combine_hotstart*.f90 to generate a new hotstart.in, and then move it to the same dir as hgrid.gr3.
		+	Then set ihot=2 in param.in. Also set nspool and ihfskip, and hotout_write to 1005. Start the run with same number of CPUs. Occasionally, the hotstarted run will crash at a different step, say 1006, and if this is the case, reset nspool and ihfskip, and hotout_write to 1006 and redo it. The 2nd time should work.
		+
		+	You'll see 2 stacks coming out after the crash. Combine the 1st stack and then viz.