Debugging after a run crashes

The best way to find out the reason for a crash is to visualize the surface velocity with ACE/xmvis6. Usually you'll see some large/noisy velocity somewhere, which may give you some hints on forcing etc.

Sometimes you want to visualize the problem right before the crash. Here is the way using the hotstart option.

Suppose you run crashed right after time step it=1005 (you can find out this in mirror.out; note that "TIME STEP= " is written AFTER a step is completed), and the closest hotstart output (in outputs/) has a step of 900.

First save any outputs that may be overwritten upon ihot=2:

mv mirror.out mirror.out.0 mv hotstart.in hotstart.in.0 mv outputs outputs.0 mkdir outputs ....

The third move is necessary as we are going to change the stack size (ihfskip).

Combine hotstart outputs at it=900 using combine_hotstart*.f90 to generate a new hotstart.in, and then move it to the same dir as hgrid.gr3. Then set ihot=2 in param.in. Also set nspool and ihfskip, and hotout_write to 1005. Start the run with same number of CPUs. Occasionally, the hotstarted run will crash at a different step, say 1006, and if this is the case, reset nspool and ihfskip, and hotout_write to 1006 and redo it. The 2nd time should work.

You'll see 2 stacks coming out after the crash. Combine the 1st stack and then viz.