Data Acquisition

Sidereon.GNSS.Data fetches GNSS archive products and DTED terrain tiles into a verified local cache. It is cache-first, supports offline reads, records provenance next to each data file, and derives product names, URLs, terrain paths, and DTED bytes through the core NIF.

Terrain Quick Start

Fetch the tile first, then pass the terrain cache root to the terrain reader:

{:ok, tile_path} = Sidereon.GNSS.Data.fetch_dted(36.75, -106.25)
terrain_root = tile_path |> Path.dirname() |> Path.dirname()

{:ok, terrain} = Sidereon.Terrain.dted(terrain_root)
{:ok, height_m} = Sidereon.Terrain.height(terrain, -106.25, 36.75)

For fully ocean or no-data tiles, fetch_dted/3 returns:

{:ok, {:no_coverage, tile_id}}

The terrain reader treats absent tiles as sea level, so callers can decide whether to prefetch no-coverage markers or simply rely on the reader fallback.

Bulk Terrain Cache

Populate a region while online:

terrain_root = "/tmp/sidereon-terrain"

{:ok, report} =
  Sidereon.GNSS.Data.prefetch_dted_bbox({36.0, -107.0, 37.0, -106.0},
    cache_dir: terrain_root
  )

report.fetched
report.cached
report.no_coverage
report.errors

Tile lists accept core Skadi ids or {lat_index, lon_index} pairs:

{:ok, report} =
  Sidereon.GNSS.Data.prefetch_dted_tiles(["N36W107", {36, -106}],
    cache_dir: terrain_root
  )

Later, run without network:

{:ok, tile_path} =
  Sidereon.GNSS.Data.fetch_dted(36.75, -106.25,
    cache_dir: terrain_root,
    offline: true
  )

GNSS Products

Build products through the catalog wrappers and fetch them to the GNSS cache:

{:ok, sp3_product} = Sidereon.GNSS.Data.mgex_sp3(:esa, ~D[2020-06-24])
{:ok, sp3_path} = Sidereon.GNSS.Data.fetch(sp3_product)
{:ok, sp3} = Sidereon.GNSS.SP3.load(sp3_path)

{:ok, ionex_path} = Sidereon.GNSS.Data.fetch_ionex(:cod_rap, ~D[2026-06-13])
{:ok, ionex} = Sidereon.GNSS.Ionosphere.load_ionex(ionex_path)

{:ok, nav_product} = Sidereon.GNSS.Data.mgex_nav(:igs, ~D[2020-06-25])
{:ok, nav_path} = Sidereon.GNSS.Data.fetch(nav_product)

{:ok, clk_product} = Sidereon.GNSS.Data.mgex_clk(:gfz, ~D[2020-06-24])
{:ok, clk_path} = Sidereon.GNSS.Data.fetch(clk_product)

Exact Product and Explicit Source Selection

The exact-acquisition API treats product identity and distribution as separate public concepts. First build the catalog product, then list only the acceptable sources in priority order:

alias Sidereon.GNSS.Data
alias Sidereon.GNSS.Distribution

{:ok, product} = Data.mgex_sp3(:cod, ~D[2026-07-12])

{:ok, request} =
  Data.request(product, [
    Distribution.nasa_cddis(),
    Distribution.direct()
  ])

{:ok, result} =
  Data.acquire(request,
    earthdata_auth:
      Distribution.EarthdataAuth.bearer(System.fetch_env!("EARTHDATA_TOKEN"))
  )

result.path
result.provenance.distribution_source
result.provenance.sha256

Only the listed sources are tried. If CDDIS returns 404, the example may try the direct archive, but it retains the same publisher, analysis-center product line, solution class, issue, date, cadence, family, format, and official filename. Earlier failures appear in result.provenance.attempts.

When several exact products are required, declare the complete identity set and gate dependent processing on the resolved acquisition identities:

expected = [request_a.identity, request_b.identity]

available = [
  result_a.provenance.resolved_identity,
  result_b.provenance.resolved_identity
]

:ok = Data.validate_exact_product_set(expected, available)

The gate rejects empty declarations, duplicates, missing products, undeclared products, and same-filename identities with different prediction metadata. For SP3 observed/predicted timing, use Sidereon.GNSS.SP3.prediction_summary/1; issue times and catalog fields are not substitutes for the product's record flags.

Caller-provided inputs use the same validation path:

Distribution.local_file("/data/exact-product.sp3.gz")
Distribution.in_memory(product_bytes, compression: :none)

These sources do not grant permission to relabel their contents. SP3 and IONEX bytes must parse and match the complete exact request.

Product Eras and Exact SP3 Validation

Catalog classification is product-aware. In particular, IGS final SP3 is "final", while IGS merged broadcast navigation remains "broadcast":

{:ok, "final"} = Data.product_solution_class(:igs, :sp3)
{:ok, "broadcast"} = Data.product_solution_class(:igs, :nav)

IGS final SP3 starts at GPS week 0730. Through GPS week 2237 its exact filename is igs<week><day>.sp3, and CDDIS packages it as Unix compress (.Z). From week 2238, beginning 2022-11-27, the identity uses the IGS long filename and CDDIS packages it with gzip. Distribution.location/2 returns the exact URL, archive filename, and compression without changing the identity. Sidereon does not derive a historical direct-BKG URL because the reviewed BKG listings do not establish one uniform historical layout. Unix-compress decoding rejects detectable partial terminal codes and nonzero terminal padding before product parsing. The format has no end marker, so a code-aligned truncation is not structurally distinguishable from a shorter stream; exact product validation and any caller-supplied digest remain the final integrity checks.

CODE routing is also product-specific. Current MGEX final SP3 and clock files use CODE_MGEX/CODE/<year>, final IONEX uses CODE/<year>, and rapid, ultra-rapid, and predicted products retain their documented CODE-family paths. GFZ rapid SP3 defaults to 15M through 2021-05-17 and 05M from 2021-05-18; dated derivation uses Data.default_sample_for_date/3 automatically.

Use Data.supported_samples/4 when a caller needs the complete evidenced set for an exact catalog line rather than its default. The result follows the GFZ rapid date boundary and the issue-level ESA/GFZ ultra-rapid transitions, and product constructors enforce the same result before deriving a filename, URL, identity, or cache key. Omitting the issue for an issue-based query selects its 0000 publication; constructing that product still requires the issue.

The catalog also enforces the verified start of each modeled long-name SP3 family: ESA final starts on 2014-01-05, GFZ rapid on 2020-05-13, ESA ultra-rapid on 2022-10-04, and GFZ ultra-rapid on 2020-10-06. IGS ultra-rapid is modeled only from GPS week 2238. ESA ultra-rapid uses 15M through the 2025-02-02 0600 issue and 05M from the 1200 issue; the date-only default uses the start-of-day (0000) convention. GFZ ultra-rapid uses 15M through 2021-05-15 and 05M from 2021-05-16.

Ultra-rapid acquisition enumerates only dated cadence/span variants evidenced for the exact center, date, and issue. The GFZ 2021-05-15 0000 issue is the only cataloged two-candidate overlap. AIUB's separately documented COD0OPSULT.SP3 is a moving two-day snapshot, not an exact alias for CODE's dated 01D_05M product, so it is not a fallback candidate.

For historical GFZ ultra-rapid products, the filename epoch and first content epoch are not always the same. Query the product-aware core catalog instead of inferring from either the filename or retrieval time:

{:ok, convention} =
  Data.sp3_content_start_convention(:gfz_ult, ~D[2022-09-04], "0000")

convention.value == :filename_epoch_minus_one_day
convention.content_start_offset_s == -86_400

The transition is issue-specific: on 2022-09-07 the 0000 issue is aligned and later issues retain the prior-day convention; on 2022-09-08 the 0000 issue is aligned, the 0300 and 0600 issues revert to the prior-day convention, and 0900 onward are aligned. All GFZ ultra issues are aligned from 2022-09-09. Exact requests built from catalog identity apply this rule automatically while keeping strict header, GPS-week, MJD, cadence, epoch-grid, span, and agency validation.

CDDIS requests for every pre-week-2238 long-name SP3 identity are rejected. The historical IGS combined-final product remains supported because its exact identity uses the documented short filename and Unix compress packaging, not a long-name alias.

The general SP3.parse/1 reader remains permissive. To validate bytes against a declared date, issue, span, cadence, format revision, and optional producing agency, use an exact request:

alias Sidereon.GNSS.SP3

{:ok, exact} =
  SP3.ExactRequest.new(~D[2026-07-12], "01D", "05M",
    issue: "0000",
    expected_agency: "AIUB"
  )

{:ok, product, coverage} = SP3.parse_exact(decompressed_bytes, exact)
coverage in [:half_open, :inclusive]

The exact gate rejects unknown, zero, non-finite, noncanonical, or mismatched cadence; incomplete mandatory SP3 structure; a mismatched agency, declared start, parsed start, epoch count, or identity; and nonascending, irregular, short, or long epoch grids. A one-day five-minute request accepts 288 epochs for half-open coverage or 289 epochs when the boundary epoch is present. The permitted counts are calculated from the validated request cadence and span, not from untrusted header values.

Fallback is deliberately narrow. An ordinary HTTP 404/not-yet-published result or offline cache absence may advance to the next explicitly selected source or official catalog candidate. A retired endpoint may advance to another explicitly selected distributor for the same exact identity. An exhausted timeout, connection, 408, 429, or server failure may do the same, but the first such availability failure is preserved if no source succeeds. Malformed content, parse failure, checksum or digest failure, cadence/span/identity mismatch, cache integrity failure, and caller configuration errors are terminal. No official source reviewed for this release documented a moving-alias validation race that justified a validation-error exception. Invalid :http_client options or callback responses, and callbacks that raise, throw, or exit, return {:http_client_failure, kind, sanitized_url} and are terminal. Only explicit timeout or connection atoms, typed Req.TransportError values, and HTTP 408/429 or server-status results from a custom client enter the retry and availability-fallback policy.

Acquisition byte limits are mandatory positive integers. Invalid :max_archive_bytes or :max_product_bytes values—including zero, negatives, fractions, and infinity-like atoms—return {:invalid_option, option} before cache or transport work begins.

CDDIS and Earthdata Login

SP3 CDDIS URLs use the GNSS-products GPS-week directory and preserve the identity's era-specific .Z or .gz packaging. Current long-name IONEX URLs use ionex/<year>/<day-of-year> and gzip.

Credentials are supplied by the caller. Besides bearer tokens, Earthdata's documented netrc mechanism is available:

machine urs.earthdata.nasa.gov login EARTHDATA_USERNAME password EARTHDATA_PASSWORD

auth = Distribution.EarthdataAuth.netrc()
{:ok, result} = Data.acquire(request, earthdata_auth: auth)

The bearer token, netrc path and password, cookies, authorization headers, and URL queries are excluded from errors and provenance. Redirects are restricted to approved HTTPS hosts. Cookies retain their documented host/domain, path, and secure restrictions.

The path and authentication decisions follow these official public sources:

• NASA CDDIS archive access
• Earthdata curl and wget access
• Earthdata bearer-token example
• NASA precise orbit products
• NASA GNSS atmospheric products
• IGS long product filename guidelines
• RFC 1952 gzip file format (accessed 2026-07-21)

Public evidence for the 0.33 catalog audit

All sources in this table were accessed on 2026-07-20.

Exact Acquisition Provenance and Cache

Distribution.Provenance records requested and parsed/resolved identity, publisher, source, official filename, sanitized original and final URLs, retrieval time, decompressed and archive lengths and SHA-256 hashes, compression, ETag/Last-Modified when present, cache-hit state, and earlier source failures.

The cache key includes the source and every exact identity discriminator. Each accepted entry is one immutable transaction containing the decompressed product, original archive bytes, and JSON provenance. A SHA-256-bound commit record names that transaction and is atomically replaced only after the files and directories are synchronized. Hits follow only that record, then recheck both hashes and lengths, full requested and resolved identities, source, caller checksum, and a fresh SP3/IONEX semantic parse.

On Linux and macOS, acquisition delegates to the shared Rust transaction implementation. Threads, BEAM instances, and other cooperating processes use its per-entry advisory lock across cache validation, acquisition, and commit. Waiters therefore reuse the completed entry instead of downloading it again. :cache_lock_timeout_ms bounds the wait (30,000 milliseconds by default); a timeout or cache write failure is terminal and never authorizes trying another distributor. The operating system releases a dead owner's lock, so abandoned transactions are removed only after a new owner holds it. Low-level callers can use Sidereon.GNSS.ExactCache directly while retaining responsibility for transport and product-format validation.

Publication relies on same-filesystem atomic rename, synchronized regular files, and synchronized entry, entries, and commit-record directories. A process death or power loss at a boundary leaves the prior complete commit or no acceptable commit. Fully valid cache triples from 0.29.0-0.29.2 are revalidated and migrated without another acquisition. result.path keeps the official filename inside the immutable transaction directory. These guarantees apply to local filesystems on Linux and macOS that honor POSIX advisory locks, atomic same-directory rename, and directory synchronization. The legacy Data.fetch/2 API and cache layout remain unchanged.

Failures remain typed tagged tuples, including authentication required/failed, authorization denied, product not published, retired endpoint, redirect policy, malformed URL, transport timeout/connection, caller HTTP-client failure, HTML or invalid content, content length, download size, decompression, checksum, semantic validation, and cache read/write errors. When every attempted source ends in ordinary publication absence, multiple attempts return {:all_distributors_failed, failures}. The first exhausted availability failure is preserved if no source succeeds. The first integrity or configuration failure is returned directly and stops fallback. Retries are bounded to meaningful transport errors, HTTP 408/429, and server errors; callers should retain the verified cache and keep public-service concurrency modest.

For multi-center merge acquisition, a center that supports SP3 but has no publicly verified catalog convention for the requested date is recorded as reason: "catalog_unavailable" without attempting transport. This does not weaken the configuration boundary: unsupported center/product combinations, invalid dates or issues, and other catalog errors remain terminal.

Merged SP3 acquisition fetches each contributing center, parses the products, and uses the existing SP3 merge implementation:

{:ok, merged_sp3, report} =
  Sidereon.GNSS.Data.fetch_merged_sp3(~D[2024-09-03], [:igs_ult, :gfz_ult])

Ultra-rapid acquisition probes only the dated variants returned by the exact catalog, and advances only when an archive returns ordinary publication absence. A second candidate currently exists only for GFZ's documented 2021-05-15 0000 cadence overlap. Integrity failure does not mark that center absent and does not authorize another candidate or center; it is returned to the caller. report.contributors records the pattern that succeeded. Merge policy options are forwarded unchanged, including guarded cell precedence:

{:ok, merged_sp3, report} =
  Sidereon.GNSS.Data.fetch_merged_sp3(
    ~N[2026-07-12 18:30:00],
    [:igs_ult, :cod_ult, :esa_ult, :gfz_ult],
    combine: :precedence,
    min_agree: 1,
    precedence_scope: :cell,
    outlier_reject: [position_m: 0.5, clock_ns: 5.0]
  )

prediction = Sidereon.GNSS.SP3.prediction_summary(merged_sp3)
prediction.observed_through

The default merge timeline uses the finest commensurate input cadence, with lower-precedence centers filling missing cells. No interpolation is introduced. Even one successful contributor passes through the same merge path so system filters and cadence validation remain consistent.

Each successful contributor now has two deliberately separate records: artifact_identity contains the requested and content-resolved product identities, explicit distributor, official filename, product and archive SHA-256 digests and lengths, and compression; acquisition contains cache-hit status, retrieval time, sanitized public URLs, HTTP metadata, and earlier attempts. Only the first record enters report.stable_input_identity.

That versioned identity canonically binds the complete contributor set and merge policy. It is unchanged by a warm-cache hit, process restart, map order, or non-semantic contributor enumeration order, and changes when an artifact, resolved identity, contributor set, or merge option changes. Precedence source order is bound explicitly because reversing it can change merged bytes. Persist a secret-free map without inspecting cache directories:

persisted = Sidereon.GNSS.Data.merge_report_to_map(report)
json = Jason.encode!(persisted)
:ok = json |> Jason.decode!() |> Sidereon.GNSS.Data.verify_merge_report()

The compatibility file helper still returns only the written path. Call fetch_merged_sp3_file_with_report/4 when the report must be retained:

{:ok, path, report} =
  Sidereon.GNSS.Data.fetch_merged_sp3_file_with_report(
    ~D[2024-09-03],
    [:igs_ult, :gfz_ult],
    "merged.SP3"
  )

Merged-SP3 acquisition accepts only complete, verified exact-acquisition records. It does not scan cache directories or infer a contributor identity from a filename, timestamp, or local path.

Every fetch returns either {:ok, value} or {:error, reason}. Offline misses, checksum failures, redirects, archive 404s, no-coverage terrain, cache failures, and catalog validation all use typed tagged reasons.

The product-aware catalog and exact-SP3 APIs are additive, while stricter acquisition rejection is intentional. The permissive parser remains available; the new date-free default-sample query supports compatibility-oriented catalog inspection. These changes therefore ship as minor version 0.33.0. Public material did not resolve a uniform historical BKG direct layout or document a safe moving-alias integrity fallback; Sidereon leaves both unsupported rather than guessing.