PCIe Gen2 + RGMII Ethernet integration test for validating the combined data path from Ethernet receive through AXI-Stream CDC FIFO to XDMA DMA transfer. This project bridges Project 21 (pure PCIe XDMA validation) and Project 23 (full order book with PCIe output) by proving the Ethernet → PCIe pipeline before introducing full trading logic.
Key features:
- RGMII Gigabit Ethernet receive (JL2121 PHY, 125 MHz)
- Async CDC FIFO: 125 MHz Ethernet domain → 250 MHz PCIe/AXI domain
- AXI-Stream to XDMA C2H streaming
- End-to-end data path validation: Ethernet frame → PCIe DMA → host
+----------------+ +------------------+ +-------------------+ +------------------+
| Host PC | | XDMA IP Core | | bbo_cdc_fifo | | RGMII RX |
| Linux | | (C2H Stream) | | (CDC FIFO) | | (125 MHz) |
| | | | | | | |
| /dev/xdma0_c2h |◄────│ S_AXIS_C2H |◄────│ 125 MHz → 250 MHz|◄────│ JL2121 PHY |
| | | (250 MHz AXI) | | Gray code ptrs | | RGMII DDR |
+----------------+ +------------------+ +-------------------+ +------------------+
| ▲
| PCIe Gen2 x1 |
| (AX7203 via TB4) |
| Gigabit Ethernet
| (test traffic)
▼
bbo_verify / dd
(data validation)
Key difference from Project 21: Adds RGMII Ethernet RX and CDC FIFO — real network data replaces the test pattern generator. No order book or ITCH parsing yet.
Key difference from Project 23: No ITCH parser, no order book, no BBO extraction. Raw Ethernet payload forwarded directly to PCIe. Trading logic added in Project 23.
| Domain | Frequency | Source | Signals |
|---|---|---|---|
| RGMII RX | 125 MHz | JL2121 PHY RX_CLK | Ethernet receive data |
| AXI / PCIe | 250 MHz | XDMA axi_aclk | AXI-Stream, XDMA DMA |
CDC crossing handled by bbo_cdc_fifo.vhd — async FIFO with Gray code pointers and
2-stage synchronizer chains. No combinatorial paths cross the clock boundary.
PCIe Link:
- Max Link Speed: 5.0 GT/s (Gen2)
- Max Link Width: X4 (negotiates to X1 via TB4 adapter)
- Device ID: 0x7024
- Vendor ID: 0x10EE (Xilinx)
AXI Interface:
- Data Width: 64-bit
- Clock Frequency: 250 MHz (axi_aclk)
- Mode: AXI-Stream C2H streaming
1. Gigabit Ethernet frame arrives at JL2121 PHY
2. RGMII RX @ 125 MHz — DDR ODDR sampling, MMCM clock alignment
3. MAC layer strips preamble/SFD, validates FCS
4. Payload written to async CDC FIFO (write side: 125 MHz)
5. CDC FIFO read side: 250 MHz AXI domain
6. AXI-Stream tvalid/tready handshake to XDMA S_AXIS_C2H
7. XDMA DMA transfer → PCIe Gen2 → /dev/xdma0_c2h_0
8. Host reads raw payload from character device
Raw Ethernet payload forwarded without modification:
| Offset | Size | Field |
|---|---|---|
| 0-5 | 6 | Destination MAC |
| 6-11 | 6 | Source MAC |
| 12-13 | 2 | EtherType |
| 14+ | Variable | Payload (forwarded to PCIe) |
No ITCH parsing or BBO extraction — payload forwarded as-is for integration validation.
IMPORTANT: Linux has TWO different xdma drivers — use the correct one:
| Driver | Type | Purpose |
|---|---|---|
Kernel built-in xdma.ko |
Platform driver | SoC DMA (Zynq etc.) — NOT for PCIe |
Xilinx dma_ip_drivers xdma.ko |
PCIe driver | PCIe endpoint DMA — correct one |
# Clone Xilinx dma_ip_drivers
cd /work/projects
git clone https://github.com/Xilinx/dma_ip_drivers
# Build PCIe XDMA driver
cd dma_ip_drivers/XDMA/linux-kernel/xdma
make clean && make
# Unload wrong driver if loaded
sudo rmmod xdma 2>/dev/null
# Load correct driver
sudo insmod /work/projects/dma_ip_drivers/XDMA/linux-kernel/xdma/xdma.ko
# Verify device nodes
ls -la /dev/xdma*# 1. PCIe device detected
lspci -d 10ee:
# Expected: XX:00.0 Memory controller: Xilinx Corporation Device 7024
# 2. Driver bound
lspci -d 10ee: -k
# Expected: Kernel driver in use: xdma
# 3. Device nodes exist
ls /dev/xdma0_*
# Expected: /dev/xdma0_c2h_0, /dev/xdma0_h2c_0, /dev/xdma0_control
# 4. Test DMA read — send Ethernet traffic first, then:
dd if=/dev/xdma0_c2h_0 of=/tmp/eth_capture.bin bs=64 count=10
xxd /tmp/eth_capture.binIMPORTANT: AX7203 requires external 12V power for PCIe + GTP transceiver operation. Without external power: PCIe device not detected, GTP transceivers unreliable.
Power options (diode-OR on board):
- CN1: External 12V DC barrel jack
- J14: ATX PSU 4-pin Molex
- PCIe slot: 12V from PCIe edge connector
Connect CN1 or J14 in addition to the PCIe slot.
XDMA IP auto-generates GTP lane constraints with wrong order for AX7203.
Auto-generated (WRONG): AX7203 correct order:
Lane 0 → X0Y7 Lane 0 → X0Y5
Lane 1 → X0Y6 Lane 1 → X0Y4
Lane 2 → X0Y5 Lane 2 → X0Y6
Lane 3 → X0Y4 Lane 3 → X0Y7
Fix applied in TCL script:
set_property IS_ENABLED false [get_files *pcie2_ip-PCIE_X0Y0.xdc]Use constraints/ax7203_pcie.xdc which has the correct lane order.
# Open project in Vivado
vivado vivado_project/pcie_eth_test.xpr
# Run Synthesis → Implementation → Generate Bitstream
# Or via TCL:
vivado -mode batch -source scripts/build.tclcd dma_ip_drivers/XDMA/linux-kernel/xdma
make && sudo insmod xdma.kocd test
make
./eth_pcie_loopback_test22-fpga-order-book-pcie/
├── src/
│ ├── eth_pcie_top.vhd # Top-level wrapper
│ ├── rgmii_rx.vhd # RGMII receive (125 MHz, DDR ODDR)
│ ├── mac_rx.vhd # MAC layer — preamble strip, FCS check
│ ├── bbo_cdc_fifo.vhd # CDC FIFO: 125 MHz → 250 MHz
│ └── axi_stream_adapter.vhd # CDC FIFO → XDMA AXI-Stream bridge
├── constraints/
│ └── ax7203_pcie.xdc # PCIe GTP lane constraints (corrected order)
├── scripts/
│ ├── create_block_design.tcl # Block design: XDMA + AXI infrastructure
│ └── build.tcl # Full build script
├── test/
│ ├── tb_cdc_fifo.vhd # Testbench: CDC FIFO crossing
│ ├── eth_pcie_loopback_test.cpp # Host validation app
│ └── Makefile
└── docs/
└── integration_notes.md
| Symptom | Cause | Solution |
|---|---|---|
lspci -d 10ee: empty |
No power or bitstream not loaded | Connect 12V, program FPGA |
/dev/xdma* not created |
Wrong kernel xdma driver | Unload, load dma_ip_drivers version |
reg_rw returns 0xFF |
PCIe link not trained | Check power, rescan: echo 1 > /sys/bus/pci/rescan |
| DMA read hangs | No Ethernet traffic arriving | Send test traffic: ping or iperf |
| Data corruption | CDC FIFO timing issue | Check Gray code pointer implementation, verify WNS > 0 |
| RGMII not locking | MMCM not locked | Check RX_CLK from PHY, verify MMCM constraints |
CDC FIFO is mandatory — 125 MHz and 250 MHz are not synchronous despite the 2× ratio. A direct connection causes metastability. Always use proven async FIFO primitives.
Two xdma drivers — the kernel built-in and Xilinx dma_ip_drivers have the same module
name. The wrong one loads silently and creates no /dev/xdma* nodes. Always verify with
modinfo xdma | grep filename.
GTP lane order — XDMA IP auto-generates wrong constraints for AX7203. Must disable auto-generated XDC and substitute corrected constraints file. Symptom: PCIe detected but link negotiates to wrong width.
External 12V required — PCIe slot power alone insufficient for GTP transceivers on AX7203. Half-bright LEDs = insufficient power.
End-to-end data path validated: Ethernet frame received → CDC FIFO → AXI-Stream →
XDMA DMA → /dev/xdma0_c2h_0 readable on host.
$ lspci -d 10ee:
11:00.0 Memory controller: Xilinx Corporation Device 7024
$ cat /sys/bus/pci/devices/0000:11:00.0/current_link_*
5 GT/s PCIe (Gen2)
1 (x1 lane via TB4)
This project serves as the integration checkpoint between:
- Project 21 (PCIe XDMA validated, no Ethernet)
- Project 22-eth-test (PCIe test pattern generator, no Ethernet)
- Project 23 (full ITCH parser + order book + PCIe output)
Isolates Ethernet → PCIe CDC issues from trading system complexity. If Project 23 has PCIe data integrity issues, revert to this baseline to isolate the fault.
- Vivado 2019.1+
- Xilinx dma_ip_drivers (XDMA Linux driver)
- GCC for host tools
- AX7203 board with JL2121 PHY and PCIe edge connector
Status: Complete — end-to-end data path validated on hardware
Hardware: ALINX AX7203 (Artix-7 XC7A200T)
Author: Adilson Dias
Project: https://github.com/adilsondias-engineer/fpga-trading-systems