o /jF@sdZddlmZddlmZddlmZddlmZddl Z ddl Z ee  j dZer8e jdeeddlZedddlmZddlmZddlZddlZdd lm Z m!Z!ddl"m#Z$ed Z%e%d Z&d Z'e%d Z(e%dZ)dZ*dZ+dZ,e%dZ-e-dZ.e-dZ/e-dZ0e-dZ1e-dZ2dZ3dZ4dZ5dZ6dZ7dZ8dZ9dZ:eGdddZ;eGd d!d!Zdd.d/Z?dd1d2Z@dd4d5ZAdd7d8ZBddks&z"safe_station_id..)r-stripjoin)r+textrrrsafe_station_idis r8 input_filer pd.DataFramecCs|s td|dt|}tttth}t| |j }|r(t d||ttttg }|t tj|t<tj|tdd|t<tj|tdd|t<tj|tdd|t<||td|t@ }|jrwt d|jttgdd ttgttg}|S) NzInput file not found: z,. Edit INPUT_FILE at the top of this script.z(Input file is missing required columns: coerceerrorsr.z*No rows remain after station/time parsing.F)Zas_index)existsFileNotFoundErrorpdZread_csv STATION_COLTIME_COLNOAA_COL STOFS_COLsorted differencecolumns ValueErrorcopyastyper-r5 to_datetime to_numericnenotnaemptygroupbymean sort_values)r9dfrequiredmissingrrr read_inputns,   " rVrScCs|}|tjdj|d<|tdgjtdft dftdft dfd }||d |d @|dt k@|d t k@}| tdgS) NMmonthrQcountnoaa_monthly_mstofs_monthly_mnoaa_valid_hoursstofs_valid_hoursr[r\r]r^)rIrBdt to_period to_timestamprPrAaggrCrD reset_indexrNMIN_VALID_HOURS_PER_MONTHrR)rSZworkmonthlyrrrhourly_to_common_monthlys.     rfpd.DatetimeIndexcCst|dd}tj|ddtd}|dr9|ddddd}tj|dd d}t |tj |d d Sd d l m }|||d }t tdd|DS)Nunitsr.dtypezseconds since z+0000T)Zutcs)Zunitr)num2date)rhcSsg|]}|qSr)Z isoformat)r3daterrr sz+netcdf_time_to_datetime..)getattrnpasarrayr startswithreplacer5r@rKZ tz_localizeZ DatetimeIndexZ to_timedeltanetCDF4rm)Ztime_varrhvaluesZ origin_textoriginrmdatesrrrnetcdf_time_to_datetimes    ry list[str]cCsNddlm}|dd}|jjdvr |jdkr dd||DSdd|DS)Nr) chartostring>UScSg|]}t|qSrr-r5r3itemrrrroz-decode_netcdf_station_ids..cSrrrrrrrror)rur{rjkindndim)Zstation_id_varr{rvrrrdecode_netcdf_station_idss  rnoaa_ncstofs_nctuple[pd.DataFrame, list[str]]cs|r|std|d|g}tt|dd}tt|dd}z tttfD] }||jvrsz3paired_netcdf_to_common_monthly..csg|]}|vr|qSrr)r3rZ stofs_indexrrrorz3paired_netcdf_to_common_monthly..z8No common station_id values between paired NetCDF files.rWrXrQrYrZr[r\r]r^rGT)Z ignore_index)&r>r?rr-NETCDF_TIME_VARNETCDF_STATION_ID_VARNETCDF_WATER_VARZ variablesrHrylenrqZ array_equalrvr enumerater`rar@ DataFrameZmaZfillednanrJrrIrCrDrPrbrcrNrdrAappendcloseconcatrR)rrrowsZnoaa_dsZstofs_dsZvar_nameZ noaa_timesZ stofs_timesZnoaa_idsZ stofs_idsZ station_idsZ month_indexbaseZnoaa_varZ stofs_varZnoaa_idxrZ stofs_idxZ noaa_valuesZ stofs_valuesZ station_framereZ monthly_allrGrrrpaired_netcdf_to_common_monthlys     ""       rrxpd.Series | pd.DatetimeIndexrcCs.t|}|}||tjddjtdS)Ng{Gv@)Zdaysri)r@rKminZ Timedeltato_numpyr)rxr_startrrryears_since_starts r pd.Seriesrvc Cstj|ddjtd}t|}t|t|@}|dkr#td||}||}t |t |j ddt id}tj|tjtd}|t |||<|jd d } t|jd } t| | d t|jd t| dd t| dd t|jd t|j|dS)Nr;r<riz,Need at least 3 valid points for time trend.ZHACZmaxlags)Zcov_typeZcov_kwds皙?alpharkg@@rrkrrkrk)rrrrrrrr)r@rLrrrrqisfinitesumrHsmOLS add_constantfitTREND_HAC_MAXLAGS full_likerpredictZconf_intparamsr pvaluesrsquared) rxrvyxmaskx_validy_validmodelrZcir rrrfit_time_trend s2    rx_valuespd.Series | np.ndarrayy_valuesc Cstjt|ddjtd}tjt|ddjtd}t|t|@}|dkr0td||}||}t |t | }tj |tjtd}|t |||<tt|jdt|jdt|jdt|j|dS) Nr;r<rirz1Need at least 3 valid points for pair regression.rkr)r r!rrr)r@rLSeriesrrrqrrrHrrrrrrrrrrr) rrrrrrrrrrrrfit_pair_regression*s"    rnoaastofsdict[str, float]cCstjt|ddjtd}tjt|ddjtd}t|t|@}|dkr7tjtjtjtjdSt ||||}t|j t|j t|j t|j dS)Nr;r<rir~) pearson_rbias_mrmse_mmae_m)r@rLrrrrqrrrrRZMEZRMSDZMAE)rrZobsmodrmetricsrrr pair_metrics@s rrrdigitsintcCs&|dus t|s dS|d|dS)NNAr1frqr)r+rrrr format_floatUsrcCs.|dus t|s dS|dkr|dS|dS)Nrg-C6?z.2ez.4fr)r+rrrformat_p[s raxplt.AxescCs8|jtjdd|jtd|jddddS)Nr~)rz%YT?r)ZxaxisZset_major_locatormdatesZ YearLocatorZset_major_formatterZ DateFormattergrid)rrrrset_month_axiscsrQ?ab pad_fractiontuple[float, float]cCsttj|tdtj|tdg}|t|}|jdkrdStt|}tt|}t||r=tt |dd}n|||}||||fS)Nrir)g?g?) rqZ concatenaterrrrsizermaxZiscloseabs)rrrrvlohiZpadrrr equal_limitsis"   rrercCsbd|d|ddd|dddt|ddd t|d d t|d d t|ddt|ddt|d dd t|dd t|dd t|ddt|ddt|d dt|ddt|dddt|ddt|d d!t|d"d#t|d$d%t|d&d't|d(d)t|d*d+ g }d,|S)-N station_id: zperiod: rX%Y-%m to common months: r.z NOAA trend:z raw_noaa_trend_mm_per_yearz mm/yr [!raw_noaa_trend_ci_low_mm_per_yearr"raw_noaa_trend_ci_high_mm_per_yearz], p=raw_noaa_trend_p_valuez, R2=raw_noaa_trend_r_squaredz STOFS trend:raw_stofs_trend_mm_per_year"raw_stofs_trend_ci_low_mm_per_year#raw_stofs_trend_ci_high_mm_per_yearraw_stofs_trend_p_valueraw_stofs_trend_r_squaredzdifference trend: raw_difference_trend_mm_per_yearz mm/yr, p=raw_difference_trend_p_valuez(STOFS = intercept + slope x NOAA: slope=raw_pair_slope , intercept=raw_pair_intercept_mz m, R2=raw_pair_r_squaredz Pearson r= raw_pearson_rz, bias= raw_bias_mz m, RMSE= raw_rmse_mz m, MAE= raw_mae_m m )rrrrrr6rrerlinesrrrstats_text_rawws\"                  (rc Csd|dt|ddt|dddt|ddd t|d d t|d dd t|ddt|ddt|dddt|ddg }d|S)Nrrr.zNOAA detrended anomaly SD: detrended_noaa_sd_mrzSTOFS detrended anomaly SD: detrended_stofs_sd_mzregression: slope=detrended_pair_sloperdetrended_pair_intercept_mz Pearson r: detrended_pearson_rzR2: detrended_pair_r_squaredzRMSE: detrended_rmse_mzMAE: detrended_mae_mr)rrr6rrrrstats_text_detrendeds     r  noaa_trend stofs_trend diff_trendpair_regout_pngc Csz|d}|djtd} |djtd} | | } tjddddd\} } | j|d d d | d }|j|| d ddd|j|| dddd|j||jd dddd|j||jddddd|d|d|j dd t || d}|j|| dddd|j||jddddd|j d d!d"d#|d$|d%|j dd t || d&}|j | | d d'd(d)d*d+t | | \}}|j||g||gd,d-d.dt| }|j| ||j|ddd/d|||||||jd0d1d2|d3|d4|d5|jdd6d7|j dd | d8}|d9|d:|jd;dd?d@|jdA| j|tdBt| dS)CNrXr[rir\r~ TZfigsizeZconstrained_layoutz Raw Monthly Mean Regressionfontsizerr#1f77b4g?zNOAA monthly meancolorlwlabel#d62728zSTOFS monthly mean@--z NOAA trendrrZlsrz STOFS trendzA. Monthly Mean Time SerieszWater level (m NAVD88)rrk#2ca02cz STOFS - NOAAblackzdifference trend0.45皙?rrzB. Difference Time SerieszSTOFS - NOAA (m)r?#4c78a8none common monthsrlrrZ edgecolorsr0.35333333?1:1 line regressionequalboxZ adjustablezC. STOFS vs NOAAzNOAA monthly mean (m NAVD88)zSTOFS monthly mean (m NAVD88)rrroffzD. Station Statistics{Gz?\(\?topleft monospaceg!@ZvaZhaZfamilyr transformZdpi)rrpltsubplotssuptitleplotr set_title set_ylabellegendraxhlinescatterrrqargsortset_xlimset_ylim set_aspect set_xlabelraxisr7r transAxessavefigFIG_DPIr)rrer r r r rrrxrrdifffigaxesrrrorderrrrplot_raw_stationsh                  rS-tuple[pd.DataFrame, TrendResult, TrendResult]cCs|}|djj|d<|ddd}|ddd}|d||d<|d||d<t|d|d}t|d|d}|d|j|d<|d|j|d <|||fS) NrXZcalendar_monthr[rQr\Znoaa_anomaly_mZstofs_anomaly_mnoaa_detrended_anomaly_mstofs_detrended_anomaly_m)rIr_rXrPr;rr)redetZ noaa_climZ stofs_climZnoaa_anom_trendZstofs_anom_trendrrrcompute_detrended_anomaliess rXrWc Csn|d}|djtd}|djtd}||}tt|} tjddddd\} } | j|d d d | d } | j||d ddd| j||dddd| jdddd| d| d| j dd t | | d} | j||dddd| j| dddd | d!d"d#| jdddd| d$| d%| j dd t | | d&} | j ||d d'd(d)d*d+t||\} }| j| |g| |gd,d-d.dt|}| j|||j|dd/d0d| | || | || jd1d2d3| d4| d5| d6| jdd7d8| j dd | d9} | d:| d;| jdd?d@dA| jdB| j|tdCt| dS)DNrXrUrirVr~rTrz% Detrended Monthly Anomaly RegressionrrrrrzNOAA detrended anomalyrrzSTOFS detrended anomalyr$r%r&r'zA. Detrended Monthly AnomalieszDetrended anomaly (m)r r!r"zdetrended STOFS - NOAAr#g?rzmean=z.3frrzB. Detrended DifferencezDifference (m)rr(r)r*r+r,r-r.r/rr0r1r2r3zC. Detrended STOFS vs NOAAzNOAA detrended anomaly (m)zSTOFS detrended anomaly (m)rrrr4zD. Detrended Statisticsr5r6r7r8r9rr:r<)rrrqZnanmeanr=r>r?r@rDrArBrCrrErrFrrGrHrIrJrrKr7r rLrMrNr)rrWr rrrxrrrOZ diff_meanrPrQrrrrRrrrplot_detrended_station sh                 rYdict[str, object]cCs,|djdd}t|}t|d}t|d}|d}|d}||}t|d|}t|d|} t|d|} t||} t||} id|d |d d d |d d d t t |d | td| td|jd|jd|jd|jd|jd| jd| jd| jd| jd| jd| j| j| j| j| j| j| j| j| j| d| d| d| dd } t|||| | | | |t|\}}}|d}|d }t||}t||}| ttj|d!d"ttj|d!d"|j|j|j|j|d|d|dd# t|||| || S)$NrXT)Zdropz_raw_regression.pngz_detrended_regression.pngr[r\rfirst_common_monthrlast_common_month common_months raw_figuredetrended_figurerrrrrrrrrrrrrrr) Z'raw_difference_trend_ci_low_mm_per_yearZ(raw_difference_trend_ci_high_mm_per_yearrZraw_difference_trend_r_squaredrrZraw_pair_p_valuerrrrrrUrVrk)Zddof) rrrrZdetrended_pair_p_valuerrrr)rRrcr8r(r)rrrrstrftimerrrZ relative_tor&Zas_posixrrrrrr r!rSrXupdaterrqZnanstdrY)rreZ station_safeZraw_pngZdet_pngrrrOr r r Z raw_pair_regZraw_pair_metricsrrWr0Zdet_noaaZ det_stofsZ det_pair_regZdet_pair_metricsrrranalyze_stationPs            rblog_rowslist[dict[str, object]]cCs(gd}t|j|djtdddS)Nrstatusreasonr] first_month last_monthrFindex)r@rZreindexto_csvPROCESSING_LOG_CSV)rcrGrrrwrite_processing_logs rn summary_rowscCs,t|}|js |d}|jtdd|S)NrFrj)r@rrOrRrl SUMMARY_CSV)rosummaryrrr write_summarys  rrFis_pboolc CsPzt|}Wn ttfyYdSwt|sdSt|r"t|St||S)Nr.)r TypeErrorrHrqrrrr)r+rrsvalrrr html_numbers  rwc Cs<zt|}Wn ttfyYdSwt|sdS|dS)Nr.ZNaNz.12grrurHrqr)r+rvrrrhtml_sort_values  rycCs dt|dt|||ddS)Nz)rrs)ryrw)r+rrsrrr html_num_cells r|r rtuple[str, str]c Csbz t|}t|}Wn ttfyYdSwt|r!t|s#dS|dkr)dS|dkr/dSdS)N)znot significantnot-significantrr)zsignificant increasezsignificant-increase)zsignificant decreasezsignificant-decreaserx)r rZ slope_valZp_valrrr trend_labels rrqcCst|}tdd|D}tdd|D}g}|D]\}}tt|dd}tt|dd} t|d|d\} } |dgd tt|d dd tt|d dd tt|d ddt |dddt |dt |dt |dt |dddd| dt| dt |dt |dt |dt |dt |dt |dt |dt |d t |d!d"|d#|d$| d#| d%qg} |D]Y} t| d&d}d'd(d)d*|d+}| d tt| d dd,|dt|d-tt| d.ddt | dddd/tt| d0dd tt| d1dd2qd| }|j rud3}ntt|d  d4tt|d  d5}d6|d7|d8|d9|d:t d;tdd|d?|d@}tj|dAdBdS)CNcs"|] }|ddkrdVqdSrfskippedrkNgetr3rowrrrr4 z$write_html_report..csrrffailedrkNrrrrrr4rr^r.r_rrzrz r[r\r{r]r)rrrT)rszraw regressionrfsuccessZ insufficientr)rrrr~z { const tbody = table.tBodies[0]; table.querySelectorAll('th').forEach((th, index) => { th.addEventListener('click', () => { const rows = Array.from(tbody.rows); const ascending = th.dataset.asc !== 'true'; table.querySelectorAll('th').forEach(h => h.dataset.asc = ''); th.dataset.asc = ascending ? 'true' : 'false'; rows.sort((a, b) => { const av = cellValue(a, index); const bv = cellValue(b, index); if (Number.isFinite(av) && Number.isFinite(bv)) return ascending ? av - bv : bv - av; return ascending ? String(av).localeCompare(String(bv)) : String(bv).localeCompare(String(av)); }); rows.forEach(row => tbody.appendChild(row)); }); }); }); const search = document.getElementById('search'); const stationBody = document.getElementById('stationTable').tBodies[0]; search.addEventListener('input', () => { const q = search.value.toLowerCase(); Array.from(stationBody.rows).forEach(row => { row.style.display = row.innerText.toLowerCase().includes(q) ? '' : 'none'; }); }); zutf-8)encoding)rrZiterrowsrr-rrrr6r|rOrrMIN_COMMON_MONTHSrdrpnamerm REPORT_HTMLZ write_text)rqrcZok_countZ skipped_countZ failed_countrr0rZraw_linkZdet_linkrZ label_classZlog_table_rowsZlog_rowrfZ status_classZlog_htmlZ period_texthtmlrrrwrite_html_reports                  !!!"#%&FXrc Cstz;trtt}t|}t|t}n$tr-t dt dt t t t \}}ntt}t|}t|t}Wn(t yf}zddt|ddddg}t|tgtt|d}~wwg}g}t|D]}||t|k}tt|}|r|dd nd} |r|dd nd} |tkr||d d |d t|| | dqoz|t||||d d|| | dWqot y}z||d|dtjdd|| | dWYd}~qod}~wwt|} t|t| |t dt t dt!t dt"t dt#dt dt|t dt$dd|Dt dt$dd|DdS)Nz0Input CSV not found, using paired NetCDF files: r__all__rrr.rerXrrzonly z common months; need rz; rk)limitzWrote summary: zWrote processing log: zWrote HTML report: zWrote figures under: ZfigureszSuccessful stations: zSkipped stations: cs |] }|ddkrdVqdSrrrrrrr4zrun_analysis..zFailed stations: csrrrrrrrr4r)%r* INPUT_FILEr>rVrfrErAuniqueUSE_NETCDF_IF_CSV_MISSINGprint NOAA_NC_FILE STOFS_NC_FILEr Exceptionr-rnrrrr@rrIrrrr`rrrrb traceback format_excr5rprmrr&r) ZhourlyreZall_station_idsexcrcrorZstation_monthlyr]rhrirqrrr run_analysiss        r__main__)r"r#)r+r,r"r-)r9rr"r:)rSr:r"r:)r"rg)r"rz)rrrrr"r)rxrr"r)rxrrvrr"r )rrrrr"r)rrrrr"r)r)r+rrrr"r-)r+rr"r-)rrr"r#)r)rrrrrrr"r)rr-rer:rrr"r-)rr-rer:r r r r r r r rrrrrr"r#)rer:r"rT) rr-rWr:r rrrrrr"r#)rr-rer:r"rZ)rcrdr"r#)rordr"r:)rF)r+r,rrrsrtr"r-)r r,rr,r"r})rqr:rcrdr"r#)[__doc__Z __future__rZ dataclassesrrrZpathlibrsysr__file__ZresolveparentZ LOCAL_DEPSr>pathinsertr-Z matplotlibZuseZmatplotlib.datesrxrZmatplotlib.pyplotZpyplotr=ZnumpyrqZpandasr@ZpylibrrZstatsmodels.apiZapirZROOTrrrrrrrr&r(r)rprmrrArBrCrDrrNrrdr rr*r8rVrfryrrrrrrrrrrrr rSrXrYrbrnrrrwryr|rrrrrrrrs                I         ,  F  C Y     J Z