Data Pipeline
This document traces the end-to-end data flow from a user constructing a transaction to that transaction being queryable through the explorer. The pipeline is intentionally simple: blockchain data flows in one direction through a series of stateless or append-only services, with polling rather than an event bus at each seam.
Example: Issuing a Token
The walkthrough below follows a single ISSUE action; a user creating a new token called MYTOKEN with 1,000,000 units and 8 decimal places.
Step 1: SDK Constructs the ACTION String
The developer calls the SDK’s issue() method:
sdk.issue({ tick: 'MYTOKEN', supply: 1000000, decimals: 8, ... })
The SDK formats this into a pipe-delimited ACTION string:
ISSUE|0|MYTOKEN|1000000|8|...
Before encoding, the SDK queries xchain-hub to retrieve current network configuration (fee rates, gas requirements, supported formats) and queries xchain-utxo-tracker to retrieve spendable UTXOs for the sender’s address. For fee-bearing actions, the SDK also calls the hub’s getfeequote method to calculate the native coin fee amount via the decentralized oracle (gas → XCHAIN → USD → native coin). These reads give the SDK everything it needs to build a valid encoding request, including the protocol fee output.
Step 2: Encoder Embeds the ACTION in a PSBT
The SDK calls the encoder’s JSON-RPC API, passing the ACTION string, the sender’s UTXOs, and their public key. The encoder:
-
Selects a format. With
encodingomitted the choice is by payload length alone (OP_RETURN, elseP2SH); the other lanes are requested explicitly, or viaencoding: AUTO, which picks the cheapest lane the network and signer support:OP_RETURN: up to 80 bytes per output (76 bytes user data + 4-byte XCHN prefix), single transactionmultisig: 60 bytes per output (two 32-byte key slots carry one chunk), single transactionP2SH: 476 bytes per chunk, split across as many outputs as needed up to the 8,192-byte compiled-payload ceiling, two-transaction patternP2WSH: the same 476-byte chunking up to the 8,192-byte compiled-payload ceiling, two-transaction patternTAPROOT: the envelope, up to 390,000 payload bytes in one tapscript witness, commit/reveal pair returned together; segwit chains only, and requested explicitly or viaencoding: AUTOrather than reached by the size fallback
-
Marks the payload with the magic
XCHN(4 bytes), in a way that depends on the lane (Encoding is the canonical rule):OP_RETURNandmultisig:XCHNis prepended to the ACTION string, so the decoder identifies the payload once it has deobfuscated the output. The marker sits inside the obfuscated bytes, never ahead of them.P2SHandP2WSH: the ACTION chunks go raw into the redeem or witness scripts, and the spend transaction carries one separate markerOP_RETURNoutput holdingXCHNplus ap2shorp2wshtag.TAPROOT: the payload goes raw into the envelope behind a cleartextXCHNmagic and format byte, with no marker output (Taproot envelope).
-
Obfuscates with AES-128-CTR only the
OP_RETURNandmultisigpayloads and theP2SH/P2WSHmarker output; script chunks and the envelope payload stay raw. The key is the first 16 hex characters of the first input’s txid; the IV is the next 16. This is deterministic (any observer with the txid can reverse it) but it filters casual blockchain scanners. -
Returns an unsigned PSBT (Partially Signed Bitcoin Transaction). For two-transaction formats, the encoder returns both PSBTs in sequence: a funding transaction and a reveal transaction.
The encoder is stateless in the sense that it holds no database and keeps no state between calls, but it is not free of network I/O. It calls the coin node over JSON-RPC (configured by the NODE_* variables) for estimatesmartfee-based fee estimation and for the optional broadcast_tx method, and it calls xchain-utxo-tracker for UTXOs when the caller omits utxos. Its output therefore depends on more than the request: with the default node-sourced fee rate or tracker-sourced UTXOs, the same call can return a different PSBT as mempool conditions and the sender’s UTXO set change.
Step 3: User Signs and Broadcasts
The caller (SDK, wallet, or custom client) signs the PSBT with the sender’s private key and broadcasts the resulting transaction to the coin node’s mempool via standard JSON-RPC (sendrawtransaction). For two-transaction formats, the funding transaction is broadcast first and the reveal transaction follows it straight away. The reveal spends an output of the funding transaction, so the node must have the funding transaction before it accepts the reveal, but nothing waits for a confirmation in between. Both can confirm in the same block; for TAPROOT that is the expected, fee-optimal shape (see Taproot envelope: Reorgs).
Step 4: Transaction Enters the Mempool and Gets Mined
From this point, the transaction is a normal blockchain transaction. Miners pick it up, include it in a block, and the block propagates across the network. In regtest environments, xchain-regtest-miner handles this step automatically (see Regtest Variant below).
Step 5: Decoder Polls, Identifies, and Stores
The decoder polls the coin node every few seconds via JSON-RPC (getblockcount, getblockhash, getblock, getrawtransaction). When it finds a new block:
-
Parses each transaction using bitcoinjs-lib. Coin-specific pre-processing is applied first:
- Litecoin: strip the MWEB/HogEx marker and flag from each raw transaction before parsing
- Dogecoin: strip AuxPoW merge-mining headers before parsing
-
Recognizes XChain payloads lane by lane. It deobfuscates an
OP_RETURNormultisigoutput and checks for theXCHNprefix; forP2SH/P2WSHit deobfuscates the markerOP_RETURNand reassembles the raw chunks from the scripts the inputs reveal; forTAPROOTit pattern-matches the cleartextXCHNenvelope in input 0’s witness. Deobfuscation uses the first input’s txid, so the decoder must retrieve the full transaction (including inputs) to reconstruct the key and IV. -
Writes raw data to the Decoder MariaDB (
XChain_{CHAIN}_{NETWORK}_Decoder). The decoder stores the block, the transaction metadata, and the raw decoded ACTION string. It does not interpret or validate the ACTION content; that is the indexer’s job. -
Detects reorgs by comparing stored block hashes to the chain tip. If a reorg is detected, the decoder rolls back to the last common block before continuing.
-
Tracks the mempool separately for the dispenser protocol, which needs to respond to unconfirmed transactions.
Step 6: Indexer Polls, Validates, and Executes
The indexer polls the Decoder DB every 5 seconds. When it finds a new decoded action it has not yet processed:
-
Routes the ACTION string to the appropriate handler class (one of 48 dedicated handler classes:
IssueAction,SendAction,OrderAction, etc.). -
Validates all fields. For
ISSUE, this means checking: the ticker does not already exist, the sender holds enough XCHAIN gas to pay the fee, the supply and decimals are within protocol limits, and the format string is well-formed. -
Executes the business logic atomically:
- Creates a token record in the tokens table
- Creates ledger entries: a credit for the minted supply to the sender’s address
- Deducts the gas fee (a debit from the sender’s XCHAIN balance, credit to the gas address)
- Records the action in the actions table with status
valid
-
Writes to the Indexer DB (
XChain_{CHAIN}_{NETWORK}_Indexer) inside a single database transaction. All writes for a block either commit together or roll back together. -
Handles validation failures gracefully: an action that fails validation is written to the actions table with status
invalid, and no ledger entries are created. The block still commits. -
Processes expirations after each block: open orders, active dispensers, COINPay obligations, and other time-bounded objects are checked against the current block’s time and expired if necessary.
-
Detects reorgs by monitoring the Decoder DB for block hash changes. On reorg, the indexer rolls back across 80+ tables in a single transaction.
The indexer is deterministic: given the same Decoder DB contents and the same local Hub DB mirror rows, it will always produce the same Indexer DB state. There is no network I/O during block processing; the hub-mirrored cross-chain tables are read locally over SQL, never fetched from the hub mid-block.
Step 7: Explorer Serves the Data
The explorer reads directly from the Indexer DB. It exposes:
- REST endpoints: over 200 routes covering tokens, balances, orders, transactions, dispensers, and more
- JSON-RPC 2.0: for programmatic access following the same interface used by Counterparty-compatible tools
- Bootstrap web UI: browser interface with Highcharts for market data visualization
A query like GET /token/MYTOKEN triggers a SQL read against the Indexer DB and returns the token record created in Step 6. Because the explorer reads directly from MariaDB with no external cache tier, it generally reflects the state of the last committed indexer block. It does keep in-process LRU caches (address IDs, ticker IDs, and immutable action records) that key on immutable data and so never serve stale state, plus a short-TTL result cache for getHolders (EXPLORER_HOLDERS_CACHE_MS, default 15s) whose output can lag the last committed block by up to that TTL.
The explorer syncs configuration from xchain-hub every 60 seconds: fee schedules, supported chains, and oracle price data. The hub is a decentralized validator network providing PBFT-consensus config, price oracle (trimmed median aggregation across 36 COIN/FIAT pairs), on-chain PRICE v0 publishing via the oracle_publish capability, cross-chain attestation, external attestation framework (http_get/llm providers), and governance. Consumers connect to multiple hub endpoints via HUB_VALIDATORS for high availability.
PRICE Oracle Data Flow
In addition to the validation pipeline above, oracle price data flows separately:
flowchart TD
FETCH["Validators with price capability fetch from<br>CoinGecko / Kraken / Coinbase (CoinMarketCap optional)"]
PBFT["PBFT consensus on prices<br>(signs canonical PRICE v0 payload)"]
PUBLISH["A validator with oracle_publish capability<br>writes PRICE v0 to a chain"]
PROCESS["That chain's decoder + indexer<br>process the action"]
VALIDATE["Indexer validates PBFT signatures,<br>writes to local prices table"]
PUSH["Indexer pushes validated round to xchain-hub"]
DEDUPE["Hub deduplicates by round_number,<br>writes to price_snapshots"]
BROADCAST["Hub broadcasts new row to all indexers'<br>local hub DB copies"]
QUERY["Indexers query their local hub DB for fee validation,<br>FIAT dispenser settlement, and VM oracle queries"]
FETCH --> PBFT --> PUBLISH --> PROCESS --> VALIDATE --> PUSH --> DEDUPE --> BROADCAST --> QUERY
Capability assignment is governed entirely by stake amount; a validator with sufficient aggregate stake against its pubkey qualifies for every capability whose min_stake it meets. Self-tests are local to each hub and gate participation, not the federation-wide quorum count.
User TOKEN/FIAT oracles (PRICE v1) follow a similar flow but skip the PBFT signature requirement; any address can publish, and the hub enforces a 24-hour lock window on subsequent updates to prevent dispenser front-running.
Step 7b: WebSocket Pushes Real-Time Events
The explorer’s WebSocket server polls the Indexer DB every 5 seconds for new blocks and actions (the same DB the REST API reads from). When changes are detected, it pushes events to subscribed clients:
- NEW_BLOCK / NEW_ACTION, raw block and action events
- Lifecycle events: ORDER_MATCH, COINPAY_REQUIRED, SWAP_MATCH, DISPENSE, etc.
- Entity updates: ADDRESS_UPDATE, TOKEN_UPDATE, MARKET_UPDATE, DISPENSER_UPDATE
Clients subscribe to channels with filters (action types, token tickers) so they only receive relevant events. On reconnect, clients can request catch-up of missed events via since_action_index.
See the Explorer WebSocket API Reference and SDK WebSocket Client for details.
Pipeline Diagram
flowchart TD
DEV["Developer / Wallet"]
SDK["xchain-sdk"]
HUB["xchain-hub<br>(validator network)"]
UTXO["xchain-utxo-tracker<br>(LevelDB)"]
ENC["xchain-encoder<br>(stateless; per-lane embedding)"]
SIGN["Developer signs + broadcasts"]
NODE["Coin node (bitcoind / litecoind / dogecoind)<br>mempool → block"]
DECODER["xchain-decoder"]
DECDB[("Decoder MariaDB<br>XChain_{C}_{N}_Decoder")]
INDEXER["xchain-indexer"]
IDXDB[("Indexer MariaDB<br>XChain_{C}_{N}_Indexer")]
EXPLORER["xchain-explorer"]
SYNC["xchain-sync"]
CLIENTS["Clients"]
REPLICAS["Validator replicas"]
DEV -->|ACTION string| SDK
SDK -->|config, oracle prices, fee quotes| HUB
SDK -->|UTXOs for sender| UTXO
SDK -->|"PSBT request (ACTION + UTXOs + pubkey)"| ENC
ENC -->|unsigned PSBT| SIGN
SIGN --> NODE
NODE -->|JSON-RPC polling| DECODER
DECODER --> DECDB
DECDB -->|"SQL polling (every 5s)"| INDEXER
INDEXER --> IDXDB
IDXDB -->|direct SQL reads| EXPLORER
IDXDB -->|"SQL polling (every 3s)"| SYNC
EXPLORER -->|REST / JSON-RPC / Web UI| CLIENTS
SYNC -->|REST / WebSocket API| REPLICAS
Polling Intervals and the Absence of an Event Bus
Each seam in the pipeline uses polling rather than push notifications or a message queue. This is a deliberate trade-off:
- Simplicity: no broker infrastructure (Kafka, RabbitMQ, Redis Pub/Sub) to deploy, monitor, or tune. Each service can be started, stopped, or restarted independently without affecting others.
- Auditability: the Decoder DB is a complete, queryable record of every raw ACTION the decoder has ever seen. The Indexer DB is a complete record of every validated state transition. Both are inspectable with standard SQL tools.
- Determinism: because the indexer reads only local databases, the Decoder DB and the Hub DB mirror, and applies deterministic logic, running it again from scratch against the same Decoder DB and an equivalent mirror always produces identical output.
The cost is latency: a transaction confirmed in a block will not appear in the explorer until the decoder poll finds the block (~seconds), the indexer poll picks up the decoded row (~5 seconds), and the explorer serves the next query. In practice this is 10–30 seconds of additional latency beyond block confirmation, which is acceptable for a protocol where block times are measured in minutes.
Regtest Variant
In a local development environment, the full pipeline runs identically but with two additions:
-
xchain-regtest-miner polls the coin node’s mempool every 100 ms. When it detects pending transactions, it waits up to 30 seconds (resetting to 5 seconds on each new arrival) and then calls
generatetoaddressto mine a block. This means developers do not have to manually mine blocks. -
xchain-e2e-test drives the entire stack using a Mocha test suite. Tests construct actions via BIP39/BIP32 wallets, broadcast them, wait for the pipeline to process them, and assert the resulting explorer state. Tests run in order and share state across the suite; each test builds on the blockchain and indexer state left by the previous one.
In regtest, all services point to Regtest network databases (XChain_BTC_Regtest_Decoder, XChain_BTC_Regtest_Indexer, etc.), keeping regtest data fully isolated from testnet and mainnet.
Determinism
The indexer’s output is fully determined by its inputs and its code. Those inputs are the Decoder DB and the local Hub DB mirror, the read-only copy of the cross-chain tables described in Database Design; block processing consults the mirror for fee validation, FIAT dispenser settlement, and VM oracle queries. There is no randomness, no external API calls during block processing (the mirror is a local SQL read, not a hub round-trip), and no dependency on wall-clock time beyond block heights. This means:
- Replay: destroy the Indexer DB, run the indexer from block 0 against the existing Decoder DB and an equivalent Hub DB mirror, and the result is bit-for-bit identical.
- Verification: multiple independent indexer instances reading the same Decoder DB and the same hub-mirrored rows will converge to the same state.
- Auditability: a disputed balance or token state can be traced back through ledger entries to the exact action and block that caused it.
Both inputs can themselves be rebuilt. The Decoder DB is rebuilt from the blockchain: destroy it, run the decoder from block 0, and it re-derives the same rows from on-chain data. The hub-mirrored rows are aggregated across chains by the hub: prices arrive as PRICE v0 and v1 actions, and the cross-chain settlement and federation-state rows (matches, calls, capability, policy and list snapshots, bridge transfers, state checkpoints, anchor reward attestations, attestation responses) are records the hub finalizes. Most of them carry validator quorum signatures, while capability snapshots are unsigned validator-set membership derived from on-chain BTC stake state. The mirror is rebuilt from the hub’s snapshot (Database Design). The staking tables (stakes, delegations, validator_rewards) are not mirrored; they are indexer state derived from the Decoder DB. So the full indexer state is reproducible from the Decoder DB plus an equivalent hub mirror, though not from one chain’s Decoder DB in isolation.
Copyright © 2025–2026 Dankest, LLC
Based on XChain Platform by Dankest, LLC – https://dankest.llc
Licensed under the GNU Affero General Public License v3.0 (AGPL-3.0-or-later) with a commercial license available for proprietary use.
You may use, modify, and distribute this material under the terms of the License. See LICENSE and NOTICE for full terms. See the licensing overview.