Improvement for TEC correction by seongsujeong · Pull Request #369 · isce-framework/isce3

seongsujeong · 2026-08-25T17:08:25Z

This PR adds several options to ionospheric TEC correction

Add option into runconfig to use total TEC only
Add option to turn on / off the slant range / azimuth correction individually
Add logic to ignore topside TEC (i.e. use total TEC only) based on the data flag inside IMAGEN TEC
Apply polynomial fitting to TEC profile to get rid of spikes
Add the additional TEC profiles to GCOV and InSAR workflows

EDIT
09/10/2026: RANSAC-based outlier detection is implemented and being applied before polynomial fitting. Test is ongoing internally.

…to TEC profile

…pside TEC rejection logic

hfattahi

some comments up to what we reviewed today.

hfattahi · 2026-09-09T18:58:55Z

+        # NOTE: Just returning an empty LUT2d() will disrupt the downstream procedure,
+        # especially comparing the LUT's axis with the RSLC's radargrid
+        zero_arr = np.zeros(tec_correction.data.shape)
+        tec_correction = isce3.core.LUT2d(tec_correction.x_axis,


Same comment as above.

bhawkins

Polynomial fitting might not be robust to outliers, so another approach like median filter or RANSAC algorithm might be better.

@gshiroma suggests checking that the related output layers (az and rg corrections, but not TEC, which get geocoded) in the GCOV product still make sense in all cases.

seongsujeong · 2026-09-10T21:49:58Z

@gshiroma suggests checking that the related output layers (az and rg corrections, but not TEC, which get geocoded) in the GCOV product still make sense in all cases.

In GCOV, disabling either correction (i.e., the azimuth or slant-range correction) causes the corresponding correction LUT to be omitted from the output. @gshiroma Please confirm if this is expected behavior.

rad-eng-59

LGTM. Thanks for the updates. Just some minor comments.

rad-eng-59 · 2026-09-11T22:12:09Z

    return delta_r


+def _ransac_polyfit(x: np.ndarray, y: np.ndarray, degree: int,


IMO, this is overkill. You could have just use MAD with either unity scalar (strict filtering) or factor of 2 scalar (looser filtering) to remove outliers from your data population prior to polyfitting for a certain degree.
It seems like it is assumed that the noise in measured TEC values has normal distribution by default and thus outliers are treated as tails beyond 3xSigma. Not sure whether this is correct assumption about TEC measurements and their source of error given small number of samples.

After testing and tuning the algorithm, it look like MAD-based filtering gives better results. I changed the implementation to MAD-based.

gshiroma · 2026-09-16T22:21:51Z

+                 sigma: float=1.5) -> np.ndarray:
+    '''
+    Robustly fit a polynomial to a TEC profile by rejecting noisy samples using
+    a MAD-based threshold.


Please, introduced the acronym MAD (Median Absolute Deviation).

gshiroma · 2026-09-16T22:23:45Z

+    coeffs = np.polyfit(x, y, degree)
+    resid = y - np.polyval(coeffs, x)
+
+    # Robust noise scale: k * 1.4826 * MAD of the residuals.


What's this "k * 1.4826"?

gshiroma · 2026-09-16T22:24:32Z

+    # Keep only inliers, but fall back to the plain fit if too few survive.
+    inliers = np.abs(resid) <= threshold
+    if np.count_nonzero(inliers) <= degree:
+        info_channel.log('Not enough iners from TEC sample. Keeping all TEC samples for polynomial fitting.')


"iners"? Should it be "inliers"?

gshiroma · 2026-09-16T22:26:28Z

+
+    # Keep only inliers, but fall back to the plain fit if too few survive.
+    inliers = np.abs(resid) <= threshold
+    if np.count_nonzero(inliers) <= degree:


Shouldn't it be "degrees + 1" instead? A polynomial with degree-n should require at least n + 1 samples.

gshiroma · 2026-09-16T22:27:06Z

+        Multiplier `k` on the robust noise scale that sets the inlier
+        threshold: `k * MAD`. Default 1.5.


k non longer needed?

bhawkins · 2026-09-16T22:13:47Z

+        zero_arr = np.zeros(tec_correction.data.shape)
+        tec_correction = isce3.core.LUT2d(tec_correction.x_axis,
+                                          tec_correction.y_axis,
+                                          zero_arr)


Suggested change

zero_arr)

zero_arr, b_error=False)

Like @hfattahi mentioned, this would prevent an error occurring when our dummy extent isn't wide enough for whatever reason, replicating the behavior of LUT2d() (no arg ctor)

bhawkins · 2026-09-16T22:16:00Z

+        Sample values (the TEC profile) to fit.
+    degree: int
+        Degree of the polynomial to fit.
+    sigma: float


Suggested change

sigma: float

num_sigma: float

Name is a little confusing since it's a sigma multiplier (unitless) rather than something with the units of the data.

bhawkins · 2026-09-16T22:21:20Z

+    # Degenerate noise scale (near-perfect fit); nothing to reject.
+    if not np.isfinite(threshold) or threshold <= 0:
+        info_channel.log('TEC MAD Threshold is not valid. Keeping all TEC samples for polynomial fitting.')
+        return np.polyval(coeffs, x)


Suggested change

return np.polyval(coeffs, x)

raise RuntimeError()

If we get here I'm not sure it's safe to evaluate the polynomial. So I'd suggest either return y or just throw an exception.

bhawkins · 2026-09-16T22:23:43Z

+    inliers = np.abs(resid) <= threshold
+    if np.count_nonzero(inliers) <= degree:
+        info_channel.log('Not enough iners from TEC sample. Keeping all TEC samples for polynomial fitting.')
+        return np.polyval(coeffs, x)


Suggested change

return np.polyval(coeffs, x)

return y

This condition suggests the fit is bad, so maybe it's better not to apply polynomial smoothing and just return the data. Should mention the behavior in the docstring.

Now I'm having some doubts about what to do in this case.

bhawkins · 2026-09-16T22:26:12Z

    return delta_r


+def _mad_polyfit(x: np.ndarray, y: np.ndarray, degree: int,


Suggested change

def _mad_polyfit(x: np.ndarray, y: np.ndarray, degree: int,

def _smooth_mad_polyfit(x: np.ndarray, y: np.ndarray, degree: int,

The function doesn't return fit coefficients like numpy.polyfit, so I think it's clearer if the name is more different. The function performs smoothing, so _smooth_mad_polyfit makes more sense to me, at least.

bhawkins · 2026-09-16T22:38:02Z

+                apply_azimuth_correction: True
+                use_total_tec_only: False
+                ignore_invalid_topside_tec: True
+                polyfit_tec_profile: False


Suggested change

polyfit_tec_profile: False

polyfit_tec_profile: False

num_sigma: 1.5

bhawkins · 2026-09-16T22:38:14Z

+                apply_azimuth_correction: True
+                use_total_tec_only: False
+                ignore_invalid_topside_tec: True
+                polyfit_tec_profile: False


Suggested change

polyfit_tec_profile: False

polyfit_tec_profile: False

num_sigma: 1.5

bhawkins · 2026-09-16T22:40:33Z

+tec_correction_options:
+    use_total_tec_only: bool(required=False)
+    ignore_invalid_topside_tec: bool(required=False)
+    polyfit_tec_profile: bool(required=False)


Suggested change

polyfit_tec_profile: bool(required=False)

polyfit_tec_profile: bool(required=False)

num_sigma: num(min=0.0, required=False)

bhawkins · 2026-09-16T22:41:00Z

+    apply_azimuth_correction: bool(required=False)
+    use_total_tec_only: bool(required=False)
+    ignore_invalid_topside_tec: bool(required=False)
+    polyfit_tec_profile: bool(required=False)


Suggested change

polyfit_tec_profile: bool(required=False)

polyfit_tec_profile: bool(required=False)

num_sigma: num(min=0.0, required=False)

bhawkins · 2026-09-16T22:41:09Z

+    apply_azimuth_correction: bool(required=False)
+    use_total_tec_only: bool(required=False)
+    ignore_invalid_topside_tec: bool(required=False)
+    polyfit_tec_profile: bool(required=False)


Suggested change

polyfit_tec_profile: bool(required=False)

polyfit_tec_profile: bool(required=False)

num_sigma: num(min=0.0, required=False)

seongsujeong added 2 commits August 6, 2026 23:42

add switch to control TEC correction's behavior

71d234e

refine the TOP TEC selection logic; add option to polynomial fitting …

40a155c

…to TEC profile

hfattahi added this to the R05.03.0 milestone Aug 25, 2026

seongsujeong added 3 commits September 3, 2026 20:05

TEC polynomial fittinf

11760d0

fix GSLC schema

96e646d

add TEC correction paramters to GCOV and InSAR workflows; refactor to…

0a30640

…pside TEC rejection logic

seongsujeong requested review from gshiroma, hfattahi and xhuang-jpl September 8, 2026 16:40

rad-eng-59 reviewed Sep 9, 2026

View reviewed changes

Comment thread python/packages/nisar/workflows/geocode_corrections.py Outdated

Comment thread python/packages/nisar/workflows/geocode_corrections.py Outdated

Comment thread python/packages/nisar/workflows/geocode_corrections.py Outdated

hfattahi reviewed Sep 9, 2026

View reviewed changes

bhawkins reviewed Sep 9, 2026

View reviewed changes

Comment thread python/packages/isce3/atmosphere/tec_product.py

Comment thread python/packages/isce3/atmosphere/tec_product.py

seongsujeong added 2 commits September 10, 2026 20:16

applying comments from PR review

0d191e9

Apply suggestion from PR review

e8b16f5

seongsujeong added 2 commits September 11, 2026 16:44

add RANSAC implementation, address comments from PR review

2cf355f

turnon ignore_invalid_topside_tec

56a9db6

rad-eng-59 approved these changes Sep 11, 2026

View reviewed changes

seongsujeong added 3 commits September 15, 2026 21:03

Check if TEC file exists in the path provided in runconfig

eea2ecf

test core for MAD thresholding

137ef41

MAD parameter tuning

e7edaa1

gshiroma reviewed Sep 16, 2026

View reviewed changes

bhawkins reviewed Sep 16, 2026

View reviewed changes

		return delta_r


		def _ransac_polyfit(x: np.ndarray, y: np.ndarray, degree: int,

		Multiplier `k` on the robust noise scale that sets the inlier
		threshold: `k * MAD`. Default 1.5.

		return delta_r


		def _mad_polyfit(x: np.ndarray, y: np.ndarray, degree: int,

	def _mad_polyfit(x: np.ndarray, y: np.ndarray, degree: int,
	def _smooth_mad_polyfit(x: np.ndarray, y: np.ndarray, degree: int,

	polyfit_tec_profile: False
	polyfit_tec_profile: False
	num_sigma: 1.5

	polyfit_tec_profile: bool(required=False)
	polyfit_tec_profile: bool(required=False)
	num_sigma: num(min=0.0, required=False)

Conversation

seongsujeong commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hfattahi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bhawkins left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

seongsujeong commented Sep 10, 2026

Uh oh!

rad-eng-59 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

seongsujeong commented Aug 25, 2026 •

edited

Loading