Incorrect DNS settings for mirrors

[joa-quim]

With the black down of the Hawaii server (Forum is down) many CI tests fail, all GMT.jl tests fail too.
Tried to
set (GMT_DATA_SERVER china)
but no luck, and local runs fail with

C:\v>grdinfo @earth_relief_01d_p
unknown [ERROR]: Bad record counter in file C:/Users/j/.gmt/gmt_data_server.txt
unknown [ERROR]: Deleting C:/Users/j/.gmt/gmt_data_server.txt so it can get regenerated - please try again
grdinfo [ERROR]: Unable to open file C:/Users/j/.gmt/gmt_data_server.txt
grdinfo [ERROR]: Remote download is currently deactivated
grdinfo [ERROR]: Unable to obtain remote file @earth_relief_01d_p
grdinfo [ERROR]: Must specify one or more input files

[joa-quim]

Found part of the problem. When one do
set (GMT_DATA_SERVER china)
it tries to download http://china.generic-mapping-tools.org/gmt_data_server.txt, which fails and returns some html file. Setting the explicit address, works. That is works
set (GMT_DATA_SERVER https://mirrors.ustc.edu.cn/gmtdata/)

[joa-quim]

I don't know how to solve this. @redyank is this related to what you want to report?

[redyank]

No, this is not related to the current server outage. The issue I found is a security vulnerability (stack-based buffer overflow) caused by missing input length validation in gmt_remote_dataset_id.

[joa-quim]

Was not referring to the sever outage but to the GMT_DATA_SERVER variable issue.
From your description the vulnerability, once identified, should not be difficult to fix. Please provide more details.

[redyank]

Understood. I have already reproduced the stack-based buffer overflow using AddressSanitizer (ASan) and identified the problematic code in src/gmt_remote.c.

To ensure a responsible disclosure, I will email you the full technical report, including the ASan logs and a minimal PoC script, right away. Please check your inbox shortly.

[seisman]

Found part of the problem. When one do set (GMT_DATA_SERVER china) it tries to download http://china.generic-mapping-tools.org/gmt_data_server.txt, which fails and returns some html file. Setting the explicit address, works. That is works set (GMT_DATA_SERVER https://mirrors.ustc.edu.cn/gmtdata/)

It's likely some protection or misconfiguration on the USTC mirror. Other mirrors may have the same problem which resulted in the issue report in #290.

Perhaps we should always recommend using the actual URLs rather than relying on the aliases?

[Esteban82]

I think I could take a look at this.

One thing I'm not entirely sure about: GMT currently uses a formula to build the server URL. As I understand it, this is what causes the failure in some situations. I can try to fix that.

However, would it make more sense to store the server URL directly in the GMT code instead of generating it from a formula?

Please let me konw what you prefer.

[joa-quim]

Did not noticed the formula. Formulas are good in the sense that more servers can be added without further code changes, but risky if some do not follow the formula. Don't know what is our current situation.

[Esteban82]

@seisman (or anyone else), do you have any thoughts or suggestions?

[seisman]

GMT currently uses a formula to build the server URL. As I understand it, this is what causes the failure in some situations. I can try to fix that.

I'm unsure what "formula" means here. Internally, GMT maps "china" to "https://china.generic-mapping-tools.org", and then DNS settings map "https://china.generic-mapping-tools.org" to "https://mirrors.ustc.edu.cn/gmtdata/". It's more likely a DNS issue, rather than GMT issue.

[Esteban82]

I'm unsure what "formula" means here. Internally, GMT maps "china" to "https://china.generic-mapping-tools.org", and then DNS settings map "https://china.generic-mapping-tools.org" to "https://mirrors.ustc.edu.cn/gmtdata/". It's more likely a DNS issue, rather than GMT issue.

I agree that this looks like a DNS issue rather than a GMT issue.

From my side, I can access https://mirrors.ustc.edu.cn/gmtdata/ immediately without any problems. However, I cannot access https://china.generic-mapping-tools.org; the browser returns "This site can’t be reached".

[Esteban82]

I now think the issue is probably entirely DNS-related.

I tested the script below, and it fails when using china.generic-mapping-tools.org but works when using the final mirror URL directly.

gmt set GMT_DATA_SERVER http://china.generic-mapping-tools.org   # Fails
gmt set GMT_DATA_SERVER https://mirrors.ustc.edu.cn/gmtdata/       # Works

gmt begin test png
    gmt grdimage -R0/20/0/20 @earth_relief_01d -Baf
gmt end

Given this, I think a more robust solution may be to maintain a small alias table inside GMT that maps server names directly to their final URLs, bypassing the DNS indirection. That would avoid failures when the DNS mapping is unavailable while still allowing users to select a mirror by name.

[seisman]

a more robust solution may be to maintain a small alias table inside GMT that maps server names directly to their final URLs, bypassing the DNS indirection.

The issue is that, the URLs to mirrors may change and it will break old GMT versions.

[Esteban82]

Ok. I see.

I did some additional DNS testing and found an inconsistency between the mirror hostnames.

For oceania.generic-mapping-tools.org, DNS returns a CNAME that ultimately resolves to the actual server:

dig oceania.generic-mapping-tools.org

;; ANSWER SECTION:
oceania.generic-mapping-tools.org. 900 IN CNAME gmtserver.soest.hawaii.edu.
gmtserver.soest.hawaii.edu.      10800 IN A     128.171.156.218

However, for several other mirrors (brasil, china, australia, noaa), DNS returns only the same A record:

dig australia.generic-mapping-tools.org

;; ANSWER SECTION:
australia.generic-mapping-tools.org. 900 IN A 216.40.34.41

I would have expected these mirror hostnames to resolve via a CNAME chain to the actual mirror host (or at least to a hostname associated with the final mirror URL), similar to how oceania is configured. Instead, they all point to the same IP address, and the corresponding hostnames do not appear to serve or redirect to the mirror content correctly.

Solution?

My understanding is that fixing this would require someone with administrative access to the Hover account that manages generic-mapping-tools.org. If I understand correctly, that account is now managed by EarthScope (@ES-Solar @rcasey-earthscope) so they may be the ones who can verify and update the DNS/forwarding configuration for these mirror hostnames.