use egrid elec data in ghg FBS - #911
Merged
Merged
Conversation
Member
|
@catherinebirney I'm seeing if I can generate eGRID 2017 |
WesIngwersen
added a commit
that referenced
this pull request
Sep 17, 2026
…#906) Adds row 14 to the driver tracker for the UMD-to-eGRID switch in #911, with #913 and #914 against it. The switch covers 2018-2024 only, because EPA never published an eGRID 2017 - the series runs 2014, 2016, then 2018. So the base year keeps the UMD series and the methodology change lands on the 2017/2018 boundary that every indexed series and the Make-reallocation measure are anchored on. eGRID also carries Puerto Rico, ~12 Mt CO2e a year and 0.8% of its total, where the US GHG inventory does not. Marked not justified, because a source switch is not a movement, and it lands on 221100 electric power - the largest cell in the panel at 609 Mt gross, and the one row 8 currently classifies as justified decarbonisation on the strength of the UMD series. That verdict has to be re-derived once #911 merges.⚠️ The vintage pinning does not protect against this. resolve_span_vintage guards against assembling a span from different build vintages; it cannot see a source change inside one vintage at a year boundary. #914 proposes the guard that would - the generalisation of ATTRIBUTION_ROLE_ALIAS - which would also have caught the EPA soils renumbering. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
catherinebirney
added this pull request to stack #940
September 17, 2026 22:42
|
|
||
| # Drop plants with no FIPS (territories like PR) so national update_geoscale | ||
| # does not reassign null Location to 00000. | ||
| df = df.loc[df['Location'].notna()].reset_index(drop=True) |
Member
There was a problem hiding this comment.
@catherinebirney There are some facility locations in the different stewi inventories that will not have FIPS but we do want to include them. stewi has built in filters which I'm reviewing right now and will post another PR on top of this with a proposed fix.
Member
|
Needs #946 approval first, then it's good |
Member
|
Do we still need the older GHG FBS w/ egrid from 2023 and 2024 @jvendries? |
Contributor
I don't think we will need the non-nowcasted 2023 and 2024 eGRID FBSs. |
…016 and 2018 (#947) * new function to average flowby data * update 2017 nowcasted data to use avg egrid 2016/18 * fix(ghg): index eGRID 2017 to the GHG inventory instead of averaging 2016 and 2018 EPA published no eGRID 2017, so the 2017 nowcast builds it from the two neighbouring inventories. Averaging them in levels assumes 2017 sits on the straight line between 2016 and 2018. It does not. UMD GHGIA table 2-S1 puts electric power carbon dioxide at 1,808.9 million metric tons in 2016, 1,732.0 in 2017 and 1,753.4 in 2018 - a 4.3% fall and then a 1.2% rise. The straight average put 2017 at 1,818.3 million metric tons, 2.8% above what the inventory implies, and turned the observed 2017-to-2018 rise into a 1.5% fall. Table 3-7 locates the miss in natural gas, whose carbon dioxide dips to 505.6 million metric tons in 2017 from 545.0 in 2016 and rebounds to 577.9 in 2018, while coal runs close to linear over the same three years. average_flowby now takes an optional index_source. Each source year is scaled onto the target year by that series' own ratio between the two years before the average is taken, so eGRID supplies the plant-level composition and the inventory supplies the timing. The 2016 anchor gives 1,767.3 million metric tons for 2017 and the 2018 anchor gives 1,769.0, 0.10% apart, against an eGRID-to-inventory level offset of 1.0203 in 2016 and 1.0214 in 2018. The mean of the two is 1,768.1 million metric tons of carbon dioxide, 2.76% below the previous estimate; methane and nitrous oxide move by the same factor. The function logs the spread between the indexed estimates and warns when it exceeds index_agreement_tolerance, which defaults to 5%. Adds 2016 to the UMD GHGIA year list. It is outside the model span and is there only as the second anchor for this estimate. Closes #914 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: catherinebirney <60186515+catherinebirney@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…essful FIPS join (#945) `egrid_to_sector` dropped plants whose county FIPS lookup came back null. On today's eGRID that is the right set - 63 Puerto Rico plants and one nameless stub in 2024 - but it gets there by accident. A null `Location` out of `apply_county_FIPS` conflates three unrelated things: a facility outside the boundary, a facility inside it that has no county to match, and plain missing data. Offshore is the second kind: NEI codes the US maritime zone `DM` with a `County` of `Federal Waters`, so it has no county FIPS and never will. BEA's economic territory is what governs here, because these emissions become B = E / x and x is BEA gross output. That territory runs past the 50 states to the Exclusive Economic Zone and the Outer Continental Shelf, so offshore oil, gas, wind and fishing are already in the denominator; dropping them from the numerator would put the two on different geographies. Territories go the other way - Puerto Rico, Guam and the Virgin Islands sit outside the NIPA economic territory, so they have no x to divide into. State the boundary once, in `location.py`, and ask the question directly. The filter runs on the facility list, before `apply_county_FIPS` overwrites `State` with a full name, and logs what it drops and why. The merge becomes inner, so emission rows with no facility record are dropped loudly rather than by way of a null `Location`. eGRID 2023 and 2024 come out byte-identical - same 11,669 and 12,392 facilities, same per-flow totals. The change is what happens when eGRID starts coding offshore wind `DM`, or when a county is renamed. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
WesIngwersen
self-requested a review
September 18, 2026 19:07
WesIngwersen
approved these changes
Sep 18, 2026
WesIngwersen
added a commit
that referenced
this pull request
Sep 18, 2026
* refactor(analysis): attribute the E-versus-x gap to FBS sources (#906)
Rebuilds the B time-series analysis on the nowcast models for 2017-2024 and
replaces the by-gas breakdown with a breakdown by the two FBS source columns,
MetaSources and AttributionSources. A gas split says what was emitted; it
cannot say why the number moved. The source columns can, because together they
name the inventory table a row came from and the vector that spread it across
sectors.
What changed:
- One 2025_usa_cornerstone_v0_4_nowcast_<year> config per calendar year, so E,
x and Vnorm all sit on the same year's detail MUT. The old script ran
2019-2023 against the published 2017 benchmark Vnorm and a BEA gross-output
series deflated to a base year.
- E is now a long frame of sector x MetaSources x AttributionSources CO2e
rather than a 7-gas pivot. The gas mapping, the fossil versus non-fossil CH4
split and the GWP conversion are unchanged from the production path, and the
CH4 split is kept because the two carry different GWPs.
- An additive decomposition: for each sector, the emissions that would have
been reported had every source grown with output are E[t-1] * g, and the
shortfall splits exactly over the source strata. verify_identity asserts the
decomposition closes against the economy-wide gap; it does, to 1e-14 percent.
- output_elasticity regresses dlog(E) on dlog(x) per stratum so a source's
co-movement with output is measured rather than assumed from its attribution
label. A stratum attributed on a row of the Use table rides a sector's fuel
purchases, not its output, so it need not track x.
- Artifacts resolve local-first at one vintage pinned across the whole span. A
cross-year comparison assembled from mixed build vintages measures the
rebuild, not the years.
Two things the run surfaces that change how the tables read, both now reported
rather than left implicit:
- x is nominal on the nowcast path, because each config sets
apply_io_year_adjustments to False. Nominal gross output runs 34.5 to 50.7
trillion dollars over the span against 40.9 trillion in constant 2017
dollars, and that price wedge is 73% of both the 2021 and the 2022 gap.
Every divergence table now has a real companion built against the deflated
series, and price_effect splits the two.
- F01000 personal consumption expenditures is a final-demand column with no
gross output, so household fuel burning and personal vehicles cannot enter
E / x and are dropped by the production path too. That is 21% of published
CO2e. emissions_without_output reports the excluded mass per year instead of
letting the totals come up quietly short against an inventory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(analysis): satisfy mypy, and drop the F01000 note to a reconciliation line
Typecheck failures on #907, all three the same shape: a mapping typed on a
Literal of the keys it knows, indexed with a plain str.
- GWP100_AR6_CEDA is keyed on a Literal of its gas names, and the two basket
rows the production path adds, HFCs and PFCs, are not among them. Copy it
into an explicit dict[str, float] before adding them.
- INDUSTRY_DESC is keyed on a Literal of the 405 industry codes, while these
frames carry codes as ordinary strings. Same fix, once at module level.
- DataFrame.columns.get_loc returns int, slice or a boolean mask. Check for the
int before adding one to it, and raise with a reason if the column is not a
single column, rather than casting past a real ambiguity.
Also: F01000 personal consumption expenditures having no gross output is
expected behaviour, not a finding. Its warning is now a plain reconciliation
note, and it no longer prints in the run log; the per-year table stays so the
totals still tie out against a published inventory.
Verified: black, ruff, mypy all clean, and a full rebuild from an empty output
directory still closes both identity checks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(analysis): chart the divergence by inventory table x attribution (#906)
The divergence figure was stacked by attribution source alone, which says how
emissions were spread across sectors but not what was being spread. It is now
stacked by the MetaSources x AttributionSources pair, labelled
"<inventory table> -> <what spread it across sectors>", with the source names
abbreviated for the legend. Pairs that stay under a share threshold of gross
divergence in every year are bundled into "other"; the default is 3%, which
keeps 22 of 65 pairs and bundles 43 of them.
The threshold is measured against GROSS divergence, the sum of absolute terms,
not the signed total. The signed total nets opposing movements to something far
smaller than the movements themselves, so a share of it would let a large
mover through as "small" whenever the year happened to cancel.
Building it surfaced a defect in the existing tables, now fixed:
EPA renumbered two attribution tables mid-span. The soils vector is T_5_17 in
2017-18 and T_5_18 from 2019, and the indirect-soils vector moved T_5_18 to
T_5_19 at the same time, so T_5_18 means direct in some years and indirect in
others. Non-energy use moved T_3_25b to T_3_25 for 2023. Keyed on the raw table
number, 2019 booked -297 Mt on T_5_17 against +255 Mt on T_5_18 while the
emissions themselves ran flat at ~290 Mt, and 2023 did the same at ~90 Mt for
non-energy use. Those were the two largest single terms in the 2019 and 2023
columns and both were relabellings.
Rollups now key on a vintage-stable `attribution` role. Merging on the table
number alone would be wrong, because the number does not identify the role;
merging onto the role is safe, because the pair's MetaSources half already
carries it and each MetaSource pairs with exactly one EPA table in any year.
The raw value stays in AttributionSources, and divergence_by_attribution_raw
keeps the renumbering visible as evidence about vintages. Span.__post_init__
re-derives the role on load, so a cache written before an alias was added
cannot re-introduce the artifact. The same class of artifact on the MECS
vector, -660/+646 Mt across the 2019 boundary, was already handled by the
year-suffix stripping.
Two plot fixes while here: tab20 repeats once past 20 series and was putting
two sources in the same blue, so the palette now chains tab20/b/c; and
autoscale sees each half-stack rather than the stacked extent, so the limits
are set from the stacks and the tallest bar is no longer clipped.
Both a nominal and a constant-dollar version are written, and the plotted
frame is returned and saved so the bundling is inspectable rather than only
visible in the picture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(analysis): add divergence_pct_of_E_abs, and stop truncating saved tables (#906)
Adds `divergence_pct_of_E_abs`, the absolute value of the signed percentage
column, so a CSV can be sorted on "which sectors moved furthest relative to
their own size" without a helper column. It comes from divergence_by, so every
rollup gets it, not only the sector one.
The more important half: the saved tables were truncated and should not have
been. divergence_by_sector kept only the 15 largest absolute movers per year,
105 rows of the 2,758 the rollup produces, and divergence_by_metasource and
divergence_by_sector_stratum were cut the same way. Absolute size is the wrong
ranking for a percentage question - a small sector can move a long way in
percentage terms and never make a top-15-by-magnitude cut - so sorting the
truncated file on the new column would have been sorting a pre-filtered file
without any sign of it. Every saved table is now complete; truncation stays in
the log preview, which says so.
All 394 sectors that carry emissions in any year are present. The 11 missing
against the 405 industries carry none, and F01000 has no gross output.
Also adds divergence_by_sector_real, since the sector question deserves the
constant-dollar denominator as much as the source question does.
⚠️ Read the new column next to E_from. Unfiltered, the top of an absolute
percentage sort is sectors whose emissions round to zero: for 2023 it leads
with watch and clock manufacturing at -1208% off a 0.0004 Mt base. Above 1 Mt
the same sort gives other financial investment activities -68.5%, manufacturing
structures +67.3% and funds and trusts -52.9%, which are real movements.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(analysis): add B_change_real, the EF movement table (#906)
Every table so far ranked the emissions behind the factor. This one ranks the
factor itself: B is the published EF, CO2e per dollar of a commodity with
indirect effects already in it via Vnorm, so B_change_real is what to sort when
the question is where the EF moved rather than where the emissions did.
Long rather than wide, one row per commodity per year-pair, so it sorts on year
and commodity together: B_from, B_to, delta_B, pct_change and abs_pct_change.
All 405 commodities and all 7 year-pairs, 2,835 rows, nothing truncated. The 21
rows where B_from is zero carry NaN rather than a fabricated percentage.
Defaults to constant first-year dollars, and B_total gained a `real` switch to
supply them. A nominal EF falls whenever prices rise, and over this span that
is not a small correction: the median absolute EF change in 2022 is 13.4% on
nominal output against 9.3% real, so a third of the typical nominal move is
inflation. 2024 runs the other way, 3.8% nominal against 7.6% real. Ranking EF
movement on nominal dollars would answer a different question than the one
being asked. B_total stays nominal, since that is the B the production model
publishes, and B_total_real is saved beside it.
⚠️ Sort on abs_pct_change, filter on B_from. A commodity whose factor rounds to
zero posts a large percentage off a rounding-scale numerator. The log preview
filters at 1e-4 kg CO2e/$ for that reason and says so.
For 2022 the ranking leads with automotive equipment rental and leasing, whose
EF goes 0.0151 to 0.0402 kg CO2e/$, up 166%, then general and consumer goods
rental at +119%. Median commodity moves 4.7% to 16.9% a year depending on the
year, with 2021 the worst.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(analysis): save the plotted frames, and let one locked CSV not cost the rest
Two ordering and robustness defects, both found by a run that looked like it
had simply not produced a table.
save_tables ran before the two plot calls, and those calls return the frames
they charted for `tables`. Adding them after the save meant they were never
written, so divergence_by_source_plotted and its real companion were missing
from every run - including the one whose commit message claimed the bundling
was saved and inspectable. Plotting now happens first and saving last.
save_tables also aborted on the first write it could not make. A CSV open in a
spreadsheet is locked on Windows, so one open file silently cost every table
after it in the iteration order, which looks exactly like the analysis never
produced them. Each write is now attempted on its own and the failures are
raised together at the end, naming the files to close. A run with one locked
file now reports "27 of 28 tables written" and names the 28th, instead of
leaving the reader to guess which of the missing tables were real.
Verified on a full run: 28 tables, B_change_real.csv at 2,835 rows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): map the B smoothing plan's approaches onto the diagnostics (#906)
Adds a Methods section to the smoothing plan giving, per approach, the
diagnostic that answers it and the file it lands in. Approach 1 is served by
the divergence decomposition, the indexed E-against-x figure, the output
elasticities and the price effect; approach 2 by B_change_real.csv. The two
rank in different units, Mt CO2e against % of the factor, and will not agree —
stated so nobody treats one as a check on the other.
Three things the evidence says the plan should absorb:
The 5% gate in approach 2 is not currently a filter. On real-dollar commodity
EFs it admits 48% to 86% of commodities depending on year, 342 of 400 in 2021,
because the median commodity moves about as much as the gate. Either raise it —
20% admits 6-43% — or keep 5% for reporting and prioritise within it on
|delta_B| times output, which is what "reduces cumulatively the most" actually
asks for and what a percentage ranking does not give.
"Numerator or denominator" is a false dichotomy for B. B = (E/x) @ Vnorm has
three moving parts, and each year's Vnorm comes from that year's nowcast Make,
so a factor can move with E/x flat. Holding one side at a time: the Make is
7-11% of gross B movement through 2022 and then 26.6% in 2023 and 21.6% in
2024. For 133 commodity-years it moved the factor more than the intensity did.
Those cannot be remediated on either the emissions or the output side.
"Unjustified" gets four concrete mechanisms rather than being left to
judgement: vintage relabelling, prices in a nominal denominator,
attribution-vector churn, and Make-mix churn. The first is definitionally
unjustified and already handled; the second is 73% of the 2021 and 2022 gap.
Acceptance also gets numbers to move: median absolute interannual EF change and
the share over the gate, both already computed, plus a diff against the
pre-change B_change_real.csv so an improvement bought by displacing movement
elsewhere stays visible.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): split the B-smoothing findings out of the plan (#906)
The plan keeps the methods: the approach-to-diagnostic-to-output table, the
note that the two approaches rank in different units and will not agree, and
the acceptance test. Everything that was a measured result moves to
About_B_change_drivers.md, following the About_*.md pattern the repo already
uses for analysis findings beside the code that produced them.
The findings note carries: what a 5% gate selects (48% to 86% of commodities
depending on year, 342 of 400 in 2021, because the median commodity moves
about as much as the gate); Vnorm's share of gross B movement (under 12%
through 2022, then 26.6% in 2023 and 21.6% in 2024, and larger than E/x for
133 commodity-years); the price share of the E-versus-x gap by year, now all
seven rather than the five that fit the earlier table; and four mechanisms
that move B with nothing underneath it.
The Vnorm section is reframed as one of the three drivers the plan now names,
rather than as a third driver the plan was missing. Two claims tightened
against the CSVs: the IO-derived share of E is 29.7% to 31.7%, not "30-32%",
and the price table now runs 2018-2024 including 19% for 2023 and 40% for
2024.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): finish the inflation sentence and mark E, x and Vnorm as code
Wes's edits to the smoothing plan: the trailing "separated from the" now ends
on "the real change", the three sources of change are named in the sentence
that introduces them, and the project is stated to start without bias toward
any of the three. E, x, Vnorm and the identity itself are marked up as code.
One slip fixed while committing: `vnorm` lowercase in the new sentence, against
`Vnorm` in the identity and the sentence after it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(analysis): gate the factor change on its share of N, not on B alone (#906)
The bottom-up gate ranked commodities on the percentage change in their direct
factor. That does not discriminate: on real-dollar factors a 5% gate admits
48% to 86% of commodities depending on year, because the median commodity
moves about as much as the gate.
The target is a steady N, so a move in B matters in proportion to how much of
N it drives. Weighting the percentage change in B by its own-direct share of N
cancels to something simple:
delta_B_pct_of_N = pct_change_B * own_direct_share_of_N
= (dB / B) * (B * L[j,j] / N)
= dB * L[j,j] / N
own_direct_share_of_N has a median of 0.077, so for the typical commodity only
8% of its footprint is its own direct emissions, which is why the unweighted
gate let nearly everything through. The same 5% gate on the weighted metric
admits 5.2% to 22.5%, and 2021 is still visibly the worst year.
The ranking changes, not just the count. For 2022 the weighted metric leads
with fruit and tree nut farming, metal ore mining, vegetable farming and water
transportation, all with own-shares of 0.67 to 0.89 and weighted moves that
track the actual N move closely. The unweighted metric led with automotive
equipment rental and put internet publishing fourth on a +105% factor move
that shifts its N by 0.45%, because its own direct emissions are 0.4% of its
footprint.
Mechanics: Span gains L, the Leontief inverse (I - (Adom + Aimp))^-1, built
per year inside that year's config and cached. N = B @ L, matching the
production total-requirements form. B_change gains N_from, N_to, pct_change_N,
L_own_loop, own_direct_share_of_N, delta_B_pct_of_N and its absolute value,
and now sorts on the last of those. pct_change_N is carried so the weighted
figure can be checked against what N actually did; they diverge when a
commodity's supply chain moves rather than its own factor, as motorcycle
manufacturing does in 2022 at -21.2% weighted against +81.7% on N.
Two corrections while here. pct_change and abs_pct_change are renamed
pct_change_B and abs_pct_change_B: the old names shadowed DataFrame.pct_change,
so `df.pct_change` in downstream pandas silently returned the method rather
than the column, and the new names are symmetric with pct_change_N. load_span
now raises when the cache predates L rather than returning a span whose
N-weighted columns would be silently absent.
⚠️ This weights within a commodity, not across them. A change in B[j] also
moves N for everything that buys from j, and a commodity nobody buys still
ranks the same as electricity. The cross-commodity version is dB[j] propagated
through row j of L, output-weighted, and is not built.
Verified: the identity delta_B_pct_of_N == pct_change_B * own_direct_share_of_N
holds to 5.7e-14 across the panel.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(analysis): split the N change into factor-driven and L-driven (#906)
Adds pct_change_N_L_held and pct_change_N_L_effect to B_change: the N move
with L frozen at the prior year, and the remainder. Makes measurable what was
previously only a docstring caveat.
L moves N two to four times as much as the emission factors do. Median
absolute N movement against the part that survives holding L fixed:
year N move factors L
2018 3.7% 1.4% 3.1%
2020 11.8% 3.4% 9.3%
2021 18.2% 3.8% 14.9%
2023 16.1% 5.4% 10.5%
2024 6.2% 1.9% 3.9%
Some of that is real structural change and some is relative prices, not
separated here because A is not deflated on this path — the nowcast configs
set apply_io_year_adjustments: False, so each year's A and therefore L is at
that year's prices.
Records the scope boundary Wes set, in both the plan and the findings note.
B = (E/x) @ Vnorm has exactly three inputs and L is not one of them; it enters
only through N. L comes from A = U_norm @ V_norm, the nowcast's own IO
product, so no emissions-side change can move it. Diagnosed here, remediated
in Nowcast Phase 2. Vnorm falls on the same side of that line for remediation
even though it is a direct term in B, since its 26.6% share of B movement in
2023 is the nowcast Make moving and no inventory or attribution change will
touch it.
⚠️ Consequence worth keeping: with L out of scope this project can deliver the
factor-driven column, 1.4% to 5.4% a year, and not the L column. A flat median
factor change must not later be read as "N is now smooth".
Confirmed while adding this that everything the gate uses is deflated: nominal
mean factor falls 0.2186 to 0.1470 over the span against 0.2186 to 0.1979
real, identical in the base year, and delta_B_pct_of_N is a ratio in which the
dollar year cancels.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): add the top 30 weighted factor movements, and flag two negative EFs (#906)
Adds the 30 largest commodity-year movements on abs_delta_B_pct_of_N to the
findings note, with own_direct_share_of_N, the unweighted change and what N
actually did beside each.
The list is led by commodities that are their own footprint — water
transportation, fertilizer manufacturing, fruit and nut farming, metal ore
mining, air transportation, own-shares 0.63 to 0.89 — which is the weighting
doing its job. Own-shares across the 30 run 0.10 to 0.89, so a low-share
commodity can still make the list on a large enough move; what it can no
longer do is make it on a large move that barely touches N. 2021 supplies 11
of the 30, consistent with it being the worst year on every other measure.
Building the table surfaced a defect that has to be recorded: 114000 fishing,
hunting and trapping and 327910 abrasive product manufacturing carry NEGATIVE
emission factors in several years. 114000 runs 1.44, 0.77, 0.67, -0.18, 0.68,
0.34, -0.27, -0.42 Mt CO2e; 327910 is negative 2020 through 2024. No other
commodity is negative in any year.
It is a defect rather than a sink. The negative mass sits on fossil-combustion
inventory rows — UMD_GHGIA_T_3_11.petroleum_industrial,
natural_gas_nonmanufacturing, ng_manufacturing, UMD_GHGIA_T_2_S1.carbonate_use
— and a negative quantity of burned petroleum does not exist.
Ruled out: gross output, positive for both in every year; and negative cells in
the nowcast Use table for those columns, of which there are none, with 324110
petroleum refineries into 114000 at 256.3, 81.5 and 131.0 $M in 2017, 2020 and
2024. The Use table does carry 383 to 699 negative cells overall out of 170,910,
but not in these columns. The remaining candidate is a subtractive step inside
the attribution, a national total allocated net of an already-allocated
portion, which goes negative when the subtracted part exceeds the total. Not
confirmed.
Until it is resolved these two commodities cannot be ranked or gated — a
percentage off a negative base has no interpretation — and 114000 otherwise
takes 6 of the top 30 slots, including the first. The four affected rows are
marked in the table rather than dropped from it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): link #912 and explain the shrinking commodity count (#906)
The gate table's commodity count falls from 402 to 400 across the span and the
note did not say why. Three of the 405 carry no direct emissions in any year
and are always absent - 4200ID, 814000 and S00402 - and from the 2021 row the
negative-factor commodities drop out as well, since the filter reads the prior
year's factor. That is why negatives beginning in 2020 first appear as a
smaller count in the 2021 row.
Also links the negative-factor section to #912, filed on the B Smoothing
Phase 1 board at Priority High.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): drop the negative-factor commodities from the top-30 ranking (#906)
114000 and 327910 are excluded from the top 30 pending #912. Their factors go
negative, so a percentage change has no interpretation for them, and 114000
was taking 6 of the 30 slots including first place at a nominal +379% that is
entirely an artefact of crossing zero. 327910 never reached the 30. The
exclusion is marked in the table's preamble with a pointer to #912 and an
instruction to restore them once the defect is fixed rather than leaving the
omission to be rediscovered.
The cleaned list runs +87.6% to 16.7% on the gate metric and contains no
negative factor on either side of any pair.
One thing the clean list makes visible that the old one did not: the span is
not evenly represented. 2021 supplies 14 of the 30 and 2022 seven, while 2018
and 2024 supply none at all. That is the COVID rebound appearing as a genuine
one-year movement in real output and fuel use together, not something to
smooth away, and prioritising on this ranking alone would spend the project's
effort on 2021. Noted in the section rather than left for someone to trip over.
Twelve of the 30 sit at own-shares of 0.63 or above - water transportation,
fertilizer, fruit and nut farming, metal ore mining, petrochemicals, air
transportation and others - which is the weighting selecting sectors whose
emissions happen on their own site.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(analysis): visualize divergence_by_sector_stratum as drift against oscillation (#906)
The finest-grained divergence table is 14k rows and had no view. It now has
sector_stratum_divergence.png, two panels answering the question the smoothing
project actually needs: is this movement going anywhere?
oscillation = 1 - |sum(divergence)| / sum(|divergence|)
0 means every year pointed the same way; 1 means the movements cancel and the
cell ends where it started. Left panel places all 1,991 cells by gross movement
against that score, so the top-right corner is large-and-rocky and the
bottom-right is large-and-trending. Right panel shows the 25 largest cells year
by year, signed, so the pattern behind the score is visible rather than taken
on trust.
The distinction matters and does not fall out of magnitude. The single largest
cell is electric power's own direct emissions against 221100 at 609 Mt gross,
and it is not a smoothing target: oscillation 0.42, net -355 Mt, emissions
falling faster than output year after year. Smoothing it would erase
decarbonisation. Of the 63 cells above 10 Mt gross, 33 oscillate above 0.75 and
carry 1,158 Mt between them against 7 trending cells carrying 270 Mt. Oil and
gas extraction takes the top two rocky slots, moving 228 Mt across two cells
and arriving 13 Mt from where it started, both on industrial-fuel rows
attributed through the Use table or the MECS energy FBS.
Building it caught a bug in my own aggregation, of the same family as the EPA
renumbering. The detail frame is keyed on the raw AttributionSources, and the
MECS energy FBS carries a year in its name, so one cell-year is split across an
..._2017 row holding the E_from side and an ..._2018 row holding the E_to side,
each with a large divergence that cancels against its sibling. Summing
|divergence| over raw rows counts both halves of that cancellation as movement.
It put petroleum refineries' gas combustion at 740 Mt of gross movement against
an actual 21 Mt, a factor of 35, and put that cell top of the ranking.
sector_stratum_span now nets within the cell-year before taking absolute
values. Corrected concentration: 3,304 Mt of gross movement over 1,991 cells,
top 10 holding 40%.
Two plot fixes the first render made obvious: labels now sit to the left of
their points, since the big cells are against the right edge of a log axis and
right-hand labels ran into each other; and the heatmap is row-normalised to
each cell's own gross, with the magnitude moved into the row label, because on
a shared scale electric power's 131 Mt year set the limits and rendered every
other row white.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(analysis): put the Make on E_vs_x_indexed, embed both figures (#906)
E_vs_x_indexed.png showed E and x but not the third driver. It now carries the
Make on a second axis, and both figures are embedded in the findings note.
Vnorm cannot go on an indexed-levels chart, and q is not a way around it. The
output-weighted total of B is total emissions by construction - measured, 5,023
Mt against 5,017 Mt in 2022, the 0.13% gap being the scrap correction - and
freezing Vnorm at 2017 moves that total by 0.1%. The Make redistributes
emissions across commodities without creating or destroying any, so an indexed
Vnorm line would sit flat at 100 and say nothing. q would not help either: q
and x are the same money counted along two axes and total to the dollar, 46.692
T against 46.692 T in 2022.
What is visible is the redistribution. make_reallocation gives the mass sitting
on a different commodity than the base-year Make would have put it on, zero in
2017 by construction and rising to 25 Mt by 2024, about 0.5% of emissions. That
is the green line.
⚠️ Corrects a number I reported earlier. The Make's share of gross B movement
was given as 9.0% rising to 26.6% in 2023 and 21.6% in 2024. That was
UNWEIGHTED - every commodity counted equally, so a kg/$ swing on something
nobody buys weighs the same as one on electricity. Weighted by commodity
output the series is 3.8, 5.2, 6.4, 3.7, 2.8, 10.7, 13.0 percent. The 2023 step
up is real and the Make is still the driver this project cannot remediate, but
it is half the size the unweighted figure implied, and the note now leads with
the weighted row and says why.
Span gains q, which the reallocation measure needs and which the plan's
"reduce cumulatively the most" criterion has been waiting on as a weight.
load_span raises when a cache predates it rather than silently dropping the
columns that depend on it.
Adds the .gitignore negation for the new images directory, alongside the
existing ones and for the reason the existing comment gives: *.png is ignored
repo-wide, so without it the embeds break on GitHub while looking fine locally.
Both figures are LFS blobs like every other committed png here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(analysis): make the E_vs_x legend say what each line is (#906)
Wes could not read the figure, and the labels were the reason. They were the
raw attribution_class keys, which read as five unrelated quantities when in
fact all five are one: total emissions partitioned by what vector spread them
across sectors. Two of the keys actively misled - io_gross_output looked like
it might be x, and inventory_table looked like it might be the GHG inventory
total, when it is the 6% attributed using another inventory table's shares.
Each line now says what it is and what share of E it carries:
inventory names the sector directly 54%
spread by a Use table row (fuel purchases) 30%
spread by the MECS manufacturing energy survey 9.5%
spread by another GHG inventory table 6.4%
spread by gross output 0.018%
A total-E line is added, drawn first, so the partition is visible rather than
inferred, and the two x lines say "dollars, not emissions". The title now names
the three drivers and the units each is in.
⚠️ io_gross_output is 1.2 Mt, eighteen thousandths of a percent of E. Drawn as
a peer of direct at 3,538 Mt it was the most dramatic line on the chart,
swinging from 100 to 83 to 107 on a fifth of a megatonne. Classes under 1% are
now thin and faded, with the share in the label, rather than dropped - a
silently missing class is worse than a de-emphasised one.
The note gains the same table beneath the figure, since the legend cannot hold
the full explanation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): commit the third embedded figure (#906)
divergence_by_source_real.png is embedded in the PR but was only in the
gitignored output/ directory. Committed alongside the other two so all three
embeds have a tracked counterpart that moves with the code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): correct the electric power reading — the trend is not monotone
Wes caught this: I wrote that Direct emissions are "falling faster than output,
year after year, in the same direction", which does not square with the cell
being positive in 2021 and 2024. It was wrong on two counts.
Not persistent: two of the seven years reverse, which is why the cell scores
0.42 on oscillation rather than near zero. And the two reversals have different
causes, which the divergence alone does not reveal:
year E growth real x growth divergence
2021 +7.0% +0.4% +96.3 Mt
2024 +0.4% -1.8% +30.9 Mt
In 2021 emissions rose 103 Mt on flat real output and direct intensity went UP
6.6%, from 3.19 to 3.40 kg CO2e per constant-2017 dollar. In 2024 emissions were
flat and real output contracted. So the 2021 reversal is on the emissions side
and the 2024 one is on the output side. Fuel switching in the generation mix is
the obvious candidate for 2021 but is not measured by anything here, and is
marked as unmeasured rather than asserted.
The span trend is still real - intensity falls 3.82 to 3.06, net -355 Mt on 609
gross - so the conclusion that the cell should not be smoothed stands. What was
wrong was the characterisation of how it gets there.
2021 also turns out to be the sharpest illustration of the price finding:
nominal electricity output rose 19.8% that year, $473bn to $567bn, against 0.4%
real. On nominal x this cell would post a large negative divergence in the very
year its emissions rose.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): correct three claims in the E-versus-x interpretation
Checked the paragraph against the figure's own numbers after Wes updated the
embed. Three things were wrong.
"A third of that denominator movement is prices" understated it by a factor of
nearly two. Nominal output rises 47.2% against 18.7% real, a 24% price wedge,
which is 56% of the nominal growth in log terms. Reading the black line as
activity overstates the denominator effect about two and a half times.
"Rising monotonically" for the Make reallocation is wrong: 0, 6.8, 10.3, 14.8,
20.0, 25.9, 24.2, 25.0 Mt. It peaks in 2022 and flattens, so the mix drift is a
2017-2022 story that has since stalled. Worth noting because it sits oddly
beside the Make's SHARE of gross B movement, which steps up in 2023-24 — both
are measured and neither is wrong, since cumulative displacement away from the
2017 mix and annual churn within it are different quantities. That
reconciliation is now in the note.
"Emissions are flat to falling on every attribution class" is not true of two
of the five. MECS-attributed emissions end 1.8% HIGHER than 2017 and
io_gross_output ends flat; the whole of the net decline is Direct, down 9.3%,
with Use-table and inventory-table attribution ending within 2% of where they
started. The per-class table is now in the PR rather than a summary adjective.
Also states what the decline actually decomposes to: total emissions fall 5.4%
while real output rises 18.7%, so the ~20% real intensity decline is roughly
three quarters denominator and one quarter numerator. And that every class dips
together in 2020 and rebounds in 2021 — the common COVID shock, the same event
that puts 14 of the bottom-up top 30 in 2021.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(analysis): add the driver investigation tracker, rename the module (#906)
Adds B_driver_investigation.md, a running tracker of the departures the
diagnostics surface: what each is worth, whether it has been shown to be
justified, and where it stands. Thirteen rows populated from the current run.
The Justified? field is the one that takes real work, and the document says so.
The diagnostics establish THAT something departed and by how much; they cannot
say whether it should have. Four verdicts are available - justified, not
justified, out of scope, unresolved - and an unresolved driver is explicitly
not a smoothing candidate. Five rows are already resolved on evidence
established in this PR: #912 and the EPA renumbering are not justified, prices
and electric power and COVID are justified, and L and Vnorm are out of scope
for Nowcast Phase 2.
Eight rows are unresolved and carry the potency to rank them. The two leading
candidates are oil and gas extraction's industrial fuel, 228 Mt moved across
two cells arriving 13 Mt from where it started, and grain farming soils at 71.8
Mt. Row 12 records a possible common cause rather than another cell: Use-table
attribution has a measured output elasticity of 1.51, and 348 Mt of the rocky
mass routes through it.
⚠️ One thing the route table needs stated so it is not misread: Direct carries
302 Mt of rocky mass, which is not a contradiction of the electric power row.
Electric power is not in that group - its oscillation is 0.42. The 302 Mt is
eight other Direct cells led by air transportation and coal mining.
Potency measures are not comparable across rows and the document says not to
sum them: top-down drivers are in Mt of gross divergence, bottom-up ones in
delta_B_pct_of_N.
Also renames derive_B_time_series.py to B_change_diagnostics.py, which is what
the module now does - it stopped being a B time-series builder several commits
ago. References updated in both docs and the module's own usage lines; verified
the module imports and a full --use-cache run writes all 29 tables under the
new name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): track the eGRID electricity source switch as a driver (#906)
Adds row 14 to the driver tracker for the UMD-to-eGRID switch in #911, with
#913 and #914 against it.
The switch covers 2018-2024 only, because EPA never published an eGRID 2017 -
the series runs 2014, 2016, then 2018. So the base year keeps the UMD series
and the methodology change lands on the 2017/2018 boundary that every indexed
series and the Make-reallocation measure are anchored on. eGRID also carries
Puerto Rico, ~12 Mt CO2e a year and 0.8% of its total, where the US GHG
inventory does not.
Marked not justified, because a source switch is not a movement, and it lands
on 221100 electric power - the largest cell in the panel at 609 Mt gross, and
the one row 8 currently classifies as justified decarbonisation on the strength
of the UMD series. That verdict has to be re-derived once #911 merges.
⚠️ The vintage pinning does not protect against this. resolve_span_vintage
guards against assembling a span from different build vintages; it cannot see a
source change inside one vintage at a year boundary. #914 proposes the guard
that would - the generalisation of ATTRIBUTION_ROLE_ALIAS - which would also
have caught the EPA soils renumbering.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): put tracker row 14 after row 13, not before
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): number the diagnostics, and track the fuel-combustion drivers (#906)
The tracker said *that* a driver departed but never which output found it,
so a row could not be re-checked without re-deriving how it was measured.
Diagnostic register:
- Extend the plan's methods table into a register of every diagnostic the
module writes, each with a stable D# handle. It previously listed 8 of 13.
- Add a Diagnostic column to the tracker, plus a key beneath it saying what
to open for each ID and what to re-run after a fix.
- D6a is flagged as the acceptance measure, so a fix is read against the
pooled gate as well as against the row's own diagnostic; a single-row
re-check cannot see movement displaced onto other commodities.
- Row 14 has no diagnostic at all, and says so: a source switch is invisible
to every diagnostic here, which is itself worth recording.
New rows, all EPA table 3-11 (industrial stationary fuel combustion):
- 15, the family as an umbrella: 1,009 Mt of gross movement, 30.5% of the
panel. Its route labels mislead - petroleum_industrial is BEA Use money
wearing a MECS label, so ~71% of the family rides the Use table, not the
29% the labels imply.
- 16, oil and gas extraction's petroleum input coefficient: 3.11x to 2022
then -29%, the 98th percentile of its own row, against a GHGRP series flat
to 1.08x over seven years.
- 17, byproduct fuel: still gas, coke oven gas and blast furnace gas are
never purchased, so no Use row can represent them. A category error rather
than an inaccuracy.
Rows 1, 2 and 12 are corrected for the same provenance finding: #918 cannot
fix rows 1 and 2, because smoothing the MECS cutoff moves a scalar
multiplier and not the allocation shape.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(analysis): implement D13-D15, the three diagnostics that had no code (#906)
D13 and D14 were quoted in the findings but existed only as ad-hoc scripts,
so their numbers could not be re-derived. Formalising D13 caught a bad one:
none of the three defensible index forms reproduces the Vnorm shares that
were published in About section 2, on the same span and the same vintages.
2023 was printed as 26.5% unweighted and is 35.7%.
- D13 vnorm_share_of_B_movement: the Make's share of gross B movement per
year. Both one-sided index directions are reported because they disagree
materially - 4.8% against 6.1% in 2021 - and the headline is their average.
- D14 combustion_floor_test: GHGRP subpart C as a measured lower bound on
each sector's table 3-11. A sector below the floor in EVERY year is a
boundary definition difference, not a defect; only the intermittent ones
carry a volatility finding. That distinction is the whole value of the
test, and 211000 leads the intermittent group at a 12.7x ratio spread.
- D15 facility_combustion / facility_basis_comparison: stationary combustion
as facilities reported it, GHGRP first and NEI for the sub-threshold
remainder, deduplicated on FRS_ID. Sector comes from the facility, never
from the SCC - a combustion SCC names the boiler and the fuel, so the same
code appears in every industry.
fuel_class separates purchased fuel from fuel the facility made itself.
Byproduct gas is never a purchase, so no Use row can carry it; the two
cannot ride the same vector. GHGRP is only published at subpart granularity,
so the process-gas share of a matched NEI record is carried onto the GHGRP
total - without that step every refinery lands unclassified and the figure
collapses from 37.0 Mt to 12.5.
Geography follows BEA's economic territory, because the denominator is BEA
gross output: offshore stays, territories go. "Limit it to the 50 states and
DC" reads like the obvious tidy-up and would delete 627 Gulf of Mexico
platforms carrying 4.9 Mt, all of it oil and gas extraction.
D14 and D15 are opt-in behind --facility-data: they are the only diagnostics
here that reach outside the repository. stewi needs download_if_missing=True
or it regenerates from the live GHGRP API and fails for 2017-2021.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): record the fuel-combustion findings and correct section 2 (#906)
Section 2's Vnorm shares are replaced by D13's, and the supersession is
stated rather than the digits quietly swapped: the previous figures came
from an ad-hoc calculation that no defensible index form reproduces.
New sections:
- The MECS split is fuel-only, and petroleum_industrial is not MECS. MECS
numbers its tables the opposite way to intuition - 2.x is nonfuel
feedstock, 3.x is fuel - and every combustion row correctly draws the fuel
side. But petroleum_industrial is BEA Use money scaled by a MECS fuel
share that is constant within a survey vintage, so ~71% of table 3-11
moves with the Use table rather than the 29% its labels imply.
- The MECS vintage cutoff lands between 2020 and 2021, measured against the
non-MECS route as a control: ~10pp of 2021 reallocation the cutoff
explains and the COVID rebound does not.
- Oil and gas extraction's petroleum coefficient against a physical
yardstick, and D14's floor, with the boundary-offset group called out as
explicitly not a defect list.
- D15's facility basis: 67.8% of table 3-11 could be facility-backed, and
the hybrid marks oil and gas DOWN 4.6pp - independent corroboration of
#922 reached from facility records rather than from the coefficient.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): give every unresolved tracker row an issue (#906)
Six rows carried a verdict and no owner, so nothing would have picked them
up once this PR merged.
On B Smoothing Phase 1:
- #933, row 12: Use-table attribution amplifies output rather than following
it, elasticity 1.51 against Direct's 0.54. High, because provenance makes
it the common cause behind rows 1 and 2 rather than a peer of them - ~716
Mt of the 1,009 Mt table 3-11 family moves with the Use table, not the 303
Mt the route labels suggest.
- #934, row 3: grain farming soils, 71.8 Mt gross and 10.6 Mt net, the
largest non-fuel oscillating cell. Measured after the table-renumbering
fix, so it is a separate movement.
- #935, row 4: air transportation jet fuel, 99.5 Mt with three quarters
cancelling. Notes that the denominator moved too, which the easy COVID
reading skips.
- #936, row 13: water transportation, +87.6% then -25.8% at an own-share of
0.79-0.89, so it passes almost undamped to buyers.
Handed to Nowcasting Phase 2, where they are remediable and here they are
not:
- #937, row 10: L moves N two to four times as much as the factors do, and
A is not deflated on this path.
- #938, row 11: the Make's share of B movement doubles in 2023 while its
cumulative drift flattens - two measured quantities moving opposite ways.
Row 8's one open thread, the 2021 electric power reversal, now points at
#936: if both are the same fuel-price event they close together.
Rows 6-9 keep an empty Issue cell and the note says why - 6 and 7 were
resolved in the change that created this tracker, 8 and 9 are justified
verdicts, and a justified driver needs no issue because the correct action
is to leave it alone. An unresolved row with no issue is now a visible gap
rather than an ambiguity.
Also annotates D13's row dict, which mypy flagged on this branch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): rows 1 and 2 are not justified, and point at the basis that replaces them (#906)
Both rows sat at `unresolved` after the evidence that resolves them had
already been gathered.
Row 1, oil and gas extraction. Three external checks agree: GHGRP subpart C
for these facilities holds a 1.08x span across 2017-2023 while the
allocation spans 12.8x; the allocation drops BELOW that measured floor in
2017 and 2020, once by a factor of 5.4; and the driving coefficient moves at
the 98th percentile of its own Use row against a median buyer of 0.91x.
Row 2, petroleum refineries. GHGRP alone - one source, so it cannot double
count itself - reports 176.4 Mt against the 86.9 Mt this method assigns.
⚠️ That is a LEVEL disagreement, not a movement study like row 1's, and the
row now says so: its 79.4 Mt of churn is attributed to the shared Use-table
mechanism rather than separately measured.
Both now link #923, the basis that replaces the vector, with #933 named as
the cause, and both carry "reassess after #929 lands". #918 is dropped from
them - it was never the lever, since smoothing the MECS cutoff moves a
scalar multiplier and not the allocation shape.
Row 15, the umbrella, is `split`: the Use-table 71% is not justified via
rows 1 and 2, the MECS 29% stays unresolved. Both halves resolve the same
way, because manufacturing combustion is 100% inside the facility scope, so
#918, #920 and #921 retire with #929.
`split` is added to the verdict legend rather than left undefined - it is
only for an umbrella whose parts landed differently, and it must state the
share each way or the row says nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): say what the tracker is for in the plan's workflow (#906)
The plan pointed at the tracker as a place drivers are listed. It is more
than that now: it is the bridge between diagnosing and acting, carrying the
diagnostic that found each driver, its potency, a verdict and the issue that
owns the fix.
Adds the rule the tracker enforces and the plan did not state - a driver is
not prioritised until it has a verdict, and not actioned until it has an
issue. Ranking on potency alone would spend effort on movements that turn
out to be real: three rows are `justified` and their correct action is to
leave them alone, and two are `out of scope` here and went to Nowcasting
Phase 2.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(analysis): resolve the span vintage on provenance, not file mtime (#912)
#912 reported negative fossil-combustion emission factors on `114000` and
`327910`. They do not reproduce on the published FBS. The defect was in which
build the module resolved to.
`resolve_span_vintage` ranked span-covering vintages by file mtime. The
`v0.3.0_796a6ca` parquets were written 33 minutes after the published
`v0.3.0_99655e9` set, so they won — despite being a local-only build off a
commit that no ref reaches, whose tree does not contain the per-year
`Energy_manufacturing_national_nowcast_<year>.yaml` methods the parquet it
produced names. An mtime says when a file was written and nothing about what
wrote it.
The resolver now classifies each vintage with `vintage_provenance` — on_main,
on_branch, dangling, no_such_commit — prefers on_main ordered by commit date,
and raises rather than guessing when several span-covering vintages are on
disk and none is reachable from origin/main. A lone off-main vintage still
resolves, because there is no ambiguity to settle, and a dangling one warns.
`--list-vintages` shows the provenance and dates of every build on disk.
`--use-cache` now refuses a pin the cache cannot honour instead of ignoring it.
Re-ran the whole note on `v0.3.0_99655e9` and restated every figure. Yearly
national totals are identical between the two builds, but up to 1.8% of E moves
between commodities (2020), with 300-340 commodities shifting over 1%. Four
findings change materially:
- No commodity is negative in any year. `114000` is a genuine mover, second in
the panel; `327910` is unremarkable at rank 655 of 2,835.
- D14: oil and gas extraction's floor spread is 2.93x, not 13.0x, and it leads
pulp mills by 1.26x rather than 5.7x. It still breaches in 2017 and 2020.
- D15: total facility-basis share movement is 14.9pp, not 31.9pp; oil and gas
is marked down 2.6pp, not 4.6pp.
- Use-table output elasticity is 1.02, not 1.51, so the finding is the gap to
`Direct` at 0.54 rather than amplification.
Also fixes three mypy errors in `nei_combustion_proxy_test.py`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): say what the figure's legend does and does not identify (#906)
The legend reads `<UMD table>.<stratum> -> <what spread it across sectors>`,
and both halves invite a wrong grouping. The attribution half is a route, not
a provenance — `petroleum_industrial -> MECS` is BEA Use money — and a fuel
name is not a combustion category, since `petroleum` appears on mobile,
commercial, industrial and non-combustion fugitive tables alike.
Name the table each entry belongs to, state that #923 is table 3-11 and only
table 3-11 at 291.0 Mt, and measure what a legend-text filter gets instead:
418 Mt that catches half the family, pulls in 209.5 Mt of mobile and
commercial combustion, and drops the 82.4 Mt that is natural gas and coal.
Also reconcile the two aggregations, so 291.0 Mt against this figure and
697.9 Mt against `sector_stratum_span.csv` stop reading as a contradiction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Facility-reported basis for industrial combustion emissions: a quarter of the year-to-year churn (#924)
* feat(analysis): score the facility basis against the whole inventory, not table 3-11 (#923)
D15 restricted the facility side to GHGRP subpart C and NEI combustion SCCs,
and compared it against table 3-11. Both halves of that were wrong for the
same reason: a cement kiln burns fuel and calcines limestone in one vessel,
and both inventories report the vessel. Subpart H is a single CO2 number
with no combustion field, and NEI puts 69.1 of cement's 69.2 Mt on process
SCCs. Scoring that against a combustion-only total produced a coverage of
0.01 for cement and a 150% overshoot for refineries - neither of them real.
- Facility side takes every GHGRP subpart except D, which is electricity and
runs on eGRID here, and SCC branches 1, 2 and 3.
- Comparison is against everything the inventory assigns the sector.
- fuel_class gains `process`, and the GHGRP imputation now carries the full
matched NEI mix rather than only the process-gas share.
Iron and steel and fertilizer land at 0.96 coverage on the widened basis,
against 1.77 and 0.96 on the old one. Cement moves from 0.01 to 1.54.
⚠️ Widening the comparison does not widen what can be improved. The 433 Mt
of process emissions this pulls in are 100% Direct-attributed - the
inventory names the sector itself, with no vector in between - so the frame
keeps inventory_Mt, allocated_Mt and direct_Mt apart and ranks on the
allocated column.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(analysis): scope the facility comparison, and correct two misleading tables (#923)
The coverage table could not be read, and two things in it were wrong.
Scope. The comparison ran over every sector, so the uncovered band was 5,007
Mt - 58% of which was electric power and households, neither ever a
candidate. Electric power runs on eGRID and is filtered out of the facility
union by construction; F01000 has no gross output and is outside this
module. They are now excluded from BOTH sides via NOT_FACILITY_COMPARABLE,
and an in_scope column marks mining, utilities and manufacturing - the
industries whose emissions happen at a plant somebody reports.
⚠️ In scope the basis reaches **94%** of the vector-allocated mass; across
every sector it reaches 34%. The unscoped figure measures the boundary
rather than the basis, and the note now says which denominator it is using.
The "does it agree" table implied an authoritative denominator that does not
exist. For these sectors the inventory does not attribute fossil combustion
directly: the column was the GHGI total pushed through MECS and BEA Use, the
derivation under suspicion. A facility total above it is a disagreement, not
an overcount, and the ratio cannot say which side moved.
So the comparison now leads on GHGRP alone, which is one source and
therefore cannot double count itself whatever #925 does to the union. **In
14 in-scope sectors GHGRP alone exceeds the current method, 368 Mt against
231 Mt**, refineries at 2.03x. Only the like-for-like subpart C against
table 3-11 comparison settles direction, and the note now says so.
Also fixes five mypy errors: an int-typed dict taking floats in D13, and
two unannotated accumulators plus an unnarrowed Optional in the proxy test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(analysis): link the prerequisite issues and project to the facility basis (#923)
Section 7 was a prose list of gaps with no way to act on it. It is now the
prerequisite map, grouped by what each item blocks and linked to project 34
and the eight issues.
- Blocking: #925 deduplication and #926 the NEI reclassification. The levels
cannot be quoted until both land. ⚠️ Neither touches the churn result -
that is measured on shares, and a stable double count does not move a
share - so the case for the approach stands while the levels wait.
- Coverage: #931 GHGRP 2024 and #932 NEI 2023-24, which together take the
basis from 2017-2022 to the full nowcast span.
- Method: #927 lease fuel, #928 the residual rule.
- Integration: #929, the one issue cross-listed onto B Smoothing Phase 1, so
that project depends on a single item while the rest of 34 feeds it. Plus
#930 for location.
Also separates what is NOT a prerequisite: the 203 sectors with no facility
data are government, agriculture, trucking and buildings - a boundary, not a
backlog, and 6% of allocated mass once scoped to mining, utilities and
manufacturing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cc: @jvendries @bl-young
Closes: #908 #913
What changed? Why?
Updated the 2018 - 2024 nowcast GHG FBS method yamls (e.g.,
GHG_national_Cornerstone_nowcast_2024.yaml) to call on eGRID electricity dataDrop UMD T_2_S1 / T_3_7 electric-power activity sets, add EPA_eGRID_electric to replace this data
Still call on UMD: SF6 via electricity_transmission (Electrical Equipment), not replaced with eGRID
Did not update 2017 FBS method, as we do not yet have the 2017 eGRID data
update
Cornerstone_2025_target- add NAICS 221111–221118, 221121, 221122 under NAICS6addresses Remove Puerto Rico from eGRID electricity emissions in the GHG FBS, and reconcile against the GHG inventory #913 - the eGRID FBS now contain only US facility data, no PR
Note - all egrid data is stored in a user's local directory, not in the bedrock directory:
C:\Users\user\AppData\Local\stewi\flowbyfacility\Testing
The 2024 nowcast FBS results match the
GHG_national_Cornerstone_nowcast_2024_egridresults onmain