doc/FBMC_Methodology_Explanation.md

# Flow-Based Market Coupling (FBMC) Methodology Explanation

## Quick Reference for FBMC Flow Forecasting MVP

---

## 1. What is FBMC?

**Flow-Based Market Coupling (FBMC)** is a European electricity market methodology that:
- Calculates cross-border trading capacity based on **network physics** (power flows)
- Replaces simple border-to-border capacity limits with **network constraints**
- Enables **hub-to-hub trading** between ANY two zones (not just physical neighbors)
- Maximizes market efficiency by considering the entire interconnected AC grid

### Traditional ATC vs FBMC

| Aspect | Traditional ATC | Flow-Based Market Coupling (FBMC) |
|--------|----------------|-----------------------------------|
| **Capacity Model** | Border-to-border limits | Network-wide constraints (CNECs) |
| **Trading Allowed** | Only between physically connected zones | Between ANY two zones (hub-to-hub) |
| **Network Physics** | Simplified, ignores loop flows | Fully modeled via PTDFs |
| **Example** | FR can only trade with direct neighbors | FR can trade with HU despite no physical interconnector |
| **Optimization** | Sub-optimal (ignores network capacity) | Optimal (uses full network capacity) |

---

## 2. Core FBMC Concepts

### 2.1 MaxBEX (Maximum Bilateral Exchange)

**Definition**: Commercial hub-to-hub trading capacity between two zones

**Key Points**:
- MaxBEX ≠ Physical interconnector ratings
- MaxBEX = Result of optimization considering ALL network constraints
- Calculated for ALL zone pairs: 12 × 11 = 132 bidirectional combinations
- Includes both physical borders and virtual borders

**Physical Border Example** (DE→FR):
```
- Physical interconnector: 3,000 MW capacity
- MaxBEX value: 2,450 MW
- Why lower? Network constraints (CNECs) in DE and FR limit capacity
- DE→FR exchange affects transmission lines in both countries
```

**Virtual Border Example** (FR→HU):
```
- Physical interconnector: NONE (no direct FR-HU cable)
- MaxBEX value: 1,200 MW
- How is this possible? Power flows through AC grid via DE, AT, CZ
- FR exports 1,200 MW, HU imports 1,200 MW
- Physical reality: Power flows through intermediate countries' grids
```

### 2.2 CNECs (Critical Network Elements with Contingencies)

**Definition**: Transmission line + contingency scenarios that constrain power flows

**Structure**:
```
CNEC = Transmission line + "What if X fails?"
Example: "German DE_CZ_LINE_123 under contingency: Czech power plant outage"
```

**Key Metrics**:
- **RAM (Remaining Available Margin)**: How much flow capacity is left (MW)
- **Shadow Price**: Economic value of relaxing this constraint (€/MWh)
- **Presolved**: Boolean indicating if CNEC was binding (limiting)
- **Fmax**: Maximum allowed flow on this line (MW)

**Why CNECs Matter**:
- CNECs are the **physical constraints** that limit MaxBEX
- Each CNEC affects multiple borders simultaneously via PTDFs
- Top 50 CNECs account for ~80% of binding events

### 2.3 PTDFs (Power Transfer Distribution Factors)

**Definition**: Sensitivity coefficient showing how a zone's injection/withdrawal affects each CNEC

**Interpretation**:
```
PTDF_DE for a German CNEC = 0.45
→ If DE increases export by 1000 MW, this CNEC's flow increases by 450 MW

PTDF_FR for same CNEC = -0.22
→ If FR increases export by 1000 MW, this CNEC's flow decreases by 220 MW
```

**Why PTDFs Enable Virtual Borders**:
- FR→HU exchange has NO direct physical path
- But it affects CNECs in DE, AT, CZ via PTDFs
- PTDF_FR = +0.35, PTDF_HU = -0.28 for a German CNEC
- FR exports → increases German CNEC flow
- HU imports → decreases German CNEC flow
- Net effect: FR→HU exchange feasibility depends on German CNEC margin

**PTDF Properties**:
- Sum of all PTDFs ≈ 0 (Kirchhoff's law - flow conservation)
- High absolute PTDF = strong influence on that CNEC
- PTDFs are constants (depend only on network topology, not on flows)

---

## 3. How MaxBEX is Calculated

### 3.1 Optimization Problem

JAO solves this optimization problem daily:

```
Maximize: Σ (MaxBEX_ij) for all zone pairs (i→j)

Subject to:
1. For each CNEC k:
   Σ(PTDF_i^k × Net_Position_i) ≤ RAM_k  (Network constraint)

2. For each zone i:
   Σ(MaxBEX_ij) - Σ(MaxBEX_ji) = Net_Position_i  (Flow balance)

3. MaxBEX_ij ≥ 0  (Non-negative capacity)

Where:
- MaxBEX_ij = Capacity from zone i to zone j (WHAT WE FORECAST)
- PTDF_i^k = Zone i's PTDF for CNEC k
- RAM_k = Remaining Available Margin for CNEC k
- Net_Position_i = Net export from zone i
```

### 3.2 Why 132 Zone Pairs Exist

**FBMC Core Bidding Zones** (12 total):
- AT (Austria)
- BE (Belgium)
- CZ (Czech Republic)
- DE (Germany-Luxembourg)
- FR (France)
- HR (Croatia)
- HU (Hungary)
- NL (Netherlands)
- PL (Poland)
- RO (Romania)
- SI (Slovenia)
- SK (Slovakia)

**All Permutations**:
```
Total bidirectional pairs = 12 × 11 = 132

Examples:
- AT→BE, AT→CZ, AT→DE, ..., AT→SK  (11 directions from AT)
- BE→AT, BE→CZ, BE→DE, ..., BE→SK  (11 directions from BE)
- ...
- SK→AT, SK→BE, SK→CZ, ..., SK→SI  (11 directions from SK)
```

**Physical vs Virtual**:
- ~40-50 physical borders (zones with direct interconnectors)
- ~80-90 virtual borders (zones without direct interconnectors)

---

## 4. Network Physics: Power Flow Reality

### 4.1 AC Grid Fundamentals

**Key Principle**: Power flows through ALL available paths, not just the intended route

**Example**: DE→PL bilateral exchange
```
Intended:  DE → PL (direct interconnector)
Reality:   Power also flows through CZ and SK (parallel paths)
Result:    CZ and SK CNECs are affected, limiting DE→PL capacity
```

### 4.2 Loop Flows

**Definition**: Unintended power flows through neighboring countries

**FR→HU Exchange Example**:
```
Commercial transaction: FR exports 1000 MW, HU imports 1000 MW

Physical reality (power flow percentages):
- 0% flows directly (no FR-HU interconnector)
- 35% flows through DE grid (PTDF_DE = +0.35)
- 28% flows through AT grid (PTDF_AT = +0.28)
- 22% flows through CZ grid (PTDF_CZ = +0.22)
- 15% flows through other paths (SI, HR, SK)

Impact:
- German CNECs see +350 MW load (may become binding)
- Austrian CNECs see +280 MW load (may become binding)
- Czech CNECs see +220 MW load (may become binding)
- MaxBEX(FR→HU) limited by most constraining CNEC
```

### 4.3 Why Virtual Borders Have Lower Capacity

**Physical Border** (DE→FR):
- Direct interconnector: 3,000 MW rating
- MaxBEX: Often 2,200-2,800 MW
- Reason: Local CNECs in DE and FR

**Virtual Border** (FR→HU):
- Direct interconnector: None
- MaxBEX: Often 800-1,500 MW
- Reason: Power flows through DE, AT, CZ (affects many CNECs)
- More CNECs affected → more constraints → lower capacity

---

## 5. FBMC Data Series Relationships

### 5.1 Data Hierarchy

```
MaxBEX (TARGET)
    ↑ Result of optimization
CNECs + PTDFs + RAM
    ↑ Network constraints
LTN (Long-Term Nominations)
    ↑ Pre-allocated capacity
Net Positions (Min/Max)
    ↑ Zone-level limits
Planned Outages
    ↑ Reduce RAM availability
```

### 5.2 Causal Chain

```
1. Planned Outages → Reduce RAM for affected CNECs
2. Reduced RAM → Tighter CNEC constraints
3. Tighter constraints + PTDFs → Limit MaxBEX
4. MaxBEX optimization → 132 capacity values
```

### 5.3 What We Forecast

**Forecasting Task**: Predict MaxBEX for all 132 zone pairs, D+1 to D+14 horizon

**Input Features** (~1,735 features):
- Historical MaxBEX (past 21 days)
- CNEC binding patterns (200 CNECs × 8 features)
- PTDFs (200 CNECs × 12 zones, aggregated)
- RAM time series (200 CNECs)
- Shadow prices (200 CNECs)
- Planned outages (200 CNECs, future covariates)
- Weather forecasts (52 grid points, future covariates)
- LTN allocations (known in advance)
- Net positions (min/max bounds)

**Output**: MaxBEX forecast for 132 zone pairs × 336 hours (14 days)

**Evaluation Metric**: MAE (Mean Absolute Error) in MW, aggregated across all borders

---

## 6. Why This Matters for Forecasting

### 6.1 Multivariate Dependencies

**Key Insight**: You cannot forecast MaxBEX(DE→FR) independently of MaxBEX(FR→DE) or MaxBEX(AT→CZ)

**Reason**: All borders share the same CNEC constraints via PTDFs

**Example**:
```
If German CNEC "DE_NORTH_LINE_5" is binding with RAM = 200 MW:
- MaxBEX(DE→FR) is limited
- MaxBEX(DE→NL) is limited
- MaxBEX(PL→DE) is limited
- MaxBEX(FR→CZ) is affected (loop flows through DE)

All of these borders compete for the same 200 MW of remaining margin!
```

### 6.2 Network Constraints Drive Capacity

**Not driven by**:
- Historical MaxBEX averages (too simplistic)
- Physical interconnector ratings (not the binding constraint)
- Bilateral flow patterns (ignores network physics)

**Driven by**:
- Which CNECs are binding (top 50 account for ~80% of binding events)
- How much RAM is available (affected by outages, weather, generation patterns)
- PTDF patterns (which zones affect which CNECs)
- LTN pre-allocations (reduce available capacity)

### 6.3 Why Chronos 2 is Well-Suited

**Chronos 2 Strengths** (for zero-shot FBMC forecasting):
1. **Multivariate context**: Sees all 132 borders + 1,735 features simultaneously
2. **Temporal patterns**: Learns hourly, daily, weekly cycles in CNEC binding
3. **Attention mechanism**: Focuses on top binding CNECs for each forecast horizon
4. **Pre-trained on diverse time series**: Generalizes to electricity network physics
5. **Zero-shot**: No fine-tuning needed for MVP (target: 134 MW MAE)

**Why CNEC features are critical**:
- CNECs = physical constraints that determine MaxBEX
- Without CNEC context, model would miss network bottlenecks
- Top 50 CNECs × 20 features = 1,000 features capturing network state

---

## 7. Practical Example Walkthrough

### Scenario: Forecasting DE→FR MaxBEX for Tomorrow (D+1)

**Step 1: Gather Historical Context** (21 days lookback)
```
- MaxBEX(DE→FR) past 21 days: avg 2,450 MW, std 320 MW
- Top 10 binding CNECs affecting DE→FR:
  * German CNEC "DE_SOUTH_1": Binding 60% of time, avg shadow price 45 €/MWh
  * French CNEC "FR_EAST_3": Binding 40% of time, avg shadow price 38 €/MWh
- Historical RAM for these CNECs: trending down (more congestion)
- Recent outages: None planned for DE or FR
```

**Step 2: Future Covariates** (D+1 to D+14)
```
- Planned outages: French line "FR_EAST_3" scheduled maintenance D+3 to D+7
  → Expect lower MaxBEX(DE→FR) during this period
- Weather forecast: High winds in DE (high renewables) → Higher DE export pressure
- LTN allocations: 400 MW pre-allocated for long-term contracts
```

**Step 3: CNEC Impact Analysis**
```
German CNEC "DE_SOUTH_1":
- PTDF_DE = +0.42 (DE export increases flow)
- PTDF_FR = -0.35 (FR import decreases flow)
- Current RAM = 450 MW
- DE→FR exchange adds: 0.42 × 1000 - 0.35 × (-1000) = 770 MW to CNEC flow
- Therefore: MaxBEX(DE→FR) ≤ 450 / 0.77 = 584 MW (if this CNEC is limiting)

French CNEC "FR_EAST_3":
- PTDF_DE = +0.38
- PTDF_FR = -0.40
- Current RAM = 600 MW
- DE→FR exchange adds: 0.38 × 1000 - 0.40 × (-1000) = 780 MW to CNEC flow
- Therefore: MaxBEX(DE→FR) ≤ 600 / 0.78 = 769 MW

Most constraining: German CNEC → MaxBEX(DE→FR) ≈ 584 MW
```

**Step 4: Chronos 2 Inference**
```
Input features (1,735-dim vector):
- Historical MaxBEX context (132 borders × 21 days)
- CNEC features (200 CNECs × 8 metrics)
- PTDF aggregates (132 borders × PTDF sums)
- Future outages (200 CNECs × 14 days)
- Weather forecasts (52 grid points × 14 days)

Chronos 2 output:
- MaxBEX(DE→FR) forecast: 620 MW (D+1, hour 12:00)
- Confidence: Model attention focused on "DE_SOUTH_1" CNEC
- Interpretation: Slightly above CNEC-derived limit due to other borders absorbing some CNEC load
```

**Step 5: Validation**
```
Actual MaxBEX(DE→FR) = 605 MW
Forecast = 620 MW
Error = 15 MW (within 134 MW target MAE)
```

---

## 8. Common Misconceptions

### Misconception 1: "MaxBEX = Interconnector Capacity"
❌ **Wrong**: MaxBEX is often much lower than interconnector ratings
✅ **Correct**: MaxBEX is the result of network-wide optimization considering all CNECs

### Misconception 2: "Virtual borders have zero capacity"
❌ **Wrong**: Virtual borders can have significant capacity (e.g., FR→HU: 800-1,500 MW)
✅ **Correct**: Virtual borders represent feasible commercial exchanges via AC grid network

### Misconception 3: "Each border can be forecasted independently"
❌ **Wrong**: All borders are coupled via shared CNEC constraints
✅ **Correct**: Multivariate forecasting is essential (Chronos 2 sees all 132 borders simultaneously)

### Misconception 4: "PTDFs change with power flows"
❌ **Wrong**: PTDFs are NOT flow-dependent
✅ **Correct**: PTDFs are constants determined by network topology (linearity assumption in DC power flow)

### Misconception 5: "Only physical borders matter for trading"
❌ **Wrong**: FBMC enables trading between ANY zone pairs
✅ **Correct**: All 132 zone-pair combinations have commercial capacity via grid network

---

## 9. References and Further Reading

### Official JAO Documentation
- JAO Publication Tool User Guide: [https://publicationtool.jao.eu/help](https://publicationtool.jao.eu/help)
- JAO FBMC Methodology: Available via JAO website
- Core FBMC Practitioners Guide: `doc/practitioners_guide.pdf`

### ENTSO-E Resources
- ENTSO-E Transparency Platform: [https://transparency.entsoe.eu/](https://transparency.entsoe.eu/)
- FBMC Overview: ENTSO-E publications on flow-based market coupling

### Academic References
- Ehrenmann, A., & Neuhoff, K. (2009). A comparison of electricity market designs in networks. *Operations Research*, 57(2), 274-286.
- Pellini, E. (2012). Measuring the impact of market coupling on the Italian electricity market. *Energy Policy*, 48, 322-333.

### Project Documentation
- `doc/JAO_Data_Treatment_Plan.md`: Complete data collection and feature extraction guide
- `doc/FBMC_Flow_Forecasting_MVP_ZERO_SHOT_PLAN.md`: 5-day MVP implementation plan
- `notebooks/01_data_exploration.py`: Interactive data exploration with sample data

---

## 10. Summary: Key Takeaways

1. **MaxBEX ≠ Physical Capacity**: MaxBEX is a commercial metric derived from network optimization
2. **132 Zone Pairs**: All 12 × 11 bidirectional combinations exist (physical + virtual borders)
3. **CNECs Are Key**: Network constraints (CNECs) determine MaxBEX via optimization
4. **PTDFs Enable Virtual Borders**: Power flows through AC grid network affect distant CNECs
5. **Multivariate Forecasting Required**: All borders share CNEC constraints via PTDFs
6. **Network Physics Matters**: Loop flows, congestion patterns, and outages drive capacity
7. **Chronos 2 Zero-Shot Approach**: Pre-trained model leverages multivariate context without fine-tuning

---

**Document Version**: 1.0
**Created**: 2025-11-03
**Project**: FBMC Flow Forecasting MVP (Zero-Shot)
**Purpose**: Comprehensive reference for understanding FBMC methodology and MaxBEX forecasting