Skip to content

Index Repair CLI (hyp-repair)

The hyp-repair command-line interface (CLI) is a specialized tool for diagnosing and repairing potential data integrity issues within Hyperion's Elasticsearch block indices. Primarily, it addresses two common problems:

  1. Forked Blocks: Blocks that were indexed but whose prev_id does not correctly link to the block_id of the preceding block number in the index. This often happens due to brief network forks where the indexer might initially follow a non-canonical chain segment. While Hyperion's live indexer attempts to handle forks automatically, past issues or interruptions might leave orphaned blocks.
  2. Missing Blocks: Gaps in the sequence of indexed blocks, where specific block numbers are entirely absent from the Elasticsearch *-block-* indices. This can occur due to indexer interruptions, network issues during indexing, or incomplete historical processing.

Use Cases

This tool is typically used when:

  • You suspect data inconsistency due to past indexing interruptions or known issues.
  • You want to verify the integrity of your indexed block sequence.
  • You need to recover from a situation where the live indexer's fork handling might not have fully corrected an issue.

Info

You can also check missed blocks in the V2 Healthcheck /v2/health

Prerequisites & Preparation

  1. Hyperion Installation: Ensure you are running commands from the root directory of your Hyperion installation (e.g., ~/hyperion-history-api).
  2. Elasticsearch Connection: The tool needs direct access to your Elasticsearch cluster as configured in config/connections.json.
  3. Nodeos Connection: For scan and repair commands involving fork validation, the tool needs access to the chain's Nodeos HTTP endpoint configured in config/connections.json to fetch canonical block information.
  4. Indexer Control Port Connection: For the fill-missing and repair commands (which trigger re-indexing ranges), the tool needs WebSocket access to the running Hyperion Indexer's control port (configured in config/connections.json under the chain's control_port, default 7002).
  5. Backup Elasticsearch (Optional): Before running commands that modify data (repair, fill-missing), creating an Elasticsearch snapshot is recommended as a safety measure.

Workflow Overview

The general workflow for using hyp-repair is:

  1. Scan: Identify potential issues (forks or missing blocks) within a specified block range.
    • scan: Performs a detailed scan comparing indexed blocks against each other (previous block ID linkage) and optionally against a live nodeos RPC endpoint to detect forks and inconsistencies.
    • quick-scan: Performs a faster scan focused only on identifying missing block number ranges using a binary search approach on indexed data. These commands output JSON files detailing the found discrepancies.
  2. Analyze: Review the output files generated by the scan to understand the scope of the issues.
  3. Repair/Fill: Execute commands to fix the identified issues (delete forked data, trigger re-indexing of missing ranges).
    • repair: Deletes forked/incorrect data from various Hyperion indices (blocks, actions, deltas, etc.) based on a "forked blocks" JSON file generated by scan. It then triggers the Hyperion indexer (via its control port) to re-fetch and re-index the correct data for these ranges.
    • fill-missing: Triggers the Hyperion indexer to fetch and index blocks for ranges identified as missing, based on a "missing blocks" JSON file generated by scan or quick-scan.
  4. Verify: Re-scan the affected range to confirm the repairs were successful.

.repair/ Directory

The hyp-repair tool, when run in scan or quick-scan mode, will typically save its findings (lists of forked or missing block ranges) into JSON files within a .repair/ directory created at the root of your Hyperion project. These files are then used as input for the repair and fill-missing commands.

Step 1: Scan for Issues

scan <chainName>

Performs a comprehensive scan of the indexed blocks for the specified chain to detect forked blocks and missing block number sequences. It compares block_id with prev_id of subsequent blocks and can validate against a live nodeos RPC endpoint.

  • Purpose: To generate a detailed report (JSON file) of blocks that are part of a fork or block ranges that are missing.
  • Behavior:
    • Reads blocks from Elasticsearch in batches.
    • Compares block_id of a block with the prev_id of the next block in sequence.
    • If an inconsistency is found, it consults the chain's configured nodeos HTTP API endpoint to determine the canonical block.
    • Identifies ranges of purely missing block numbers.
    • Outputs findings to JSON files (e.g., .repair/<chain>-<start>-<end>-forked-blocks.json, .repair/<chain>-<start>-<end>-missing-blocks.json).

Usage: 

./hyp-repair scan <chainName> [options]

Arguments:

  • <chainName>: The alias of the chain to scan.

Options:

  • -f, --first <blockNumber>: The block number to start the scan from. Defaults to the first block found in the index.
  • -l, --last <blockNumber>: The block number to end the scan at. Defaults to the last block found in the index.
  • -b, --batch <size>: The number of blocks to fetch from Elasticsearch in each batch during the scan. Default is 2000.
  • -o, --out-file <baseFilePath>: Base path and filename for the output JSON files (without extension). Defaults to ./.repair/<chain>-<first>-<last>.

Examples: 

# Full scan (forks & missing) for 'eos' chain using entire indexed range and default batch size
./hyp-repair scan eos

# Scan the WAX chain from block 100,000,000 to 110,000,000
./hyp-repair scan wax -f 100000000 -l 110000000

# Scan with custom output file base name and batch size
./hyp-repair scan wax --first 100000000 --last 150000000 --batch 5000 --out-file /tmp/wax-repair-scan
After the scan, check the .repair/ directory or your specified output path for the JSON report files.

quick-scan <chainName>

Performs a faster scan focused solely on identifying ranges of missing block numbers by checking for sequence continuity. It does not validate block IDs against a live node or detect forks.

  • Purpose: To quickly find gaps in the indexed block sequence. Useful as a first pass or if forks are not suspected.
  • Behavior:
    • Uses a binary search-like approach on Elasticsearch to find block number gaps.
    • Outputs findings to a JSON file (e.g., .repair/<chain>-<start>-<end>-missing-blocks.json).

Usage: 

./hyp-repair quick-scan <chainName> [options]

Arguments:

  • <chainName>: The alias of the chain to scan.

Options:

  • -f, --first <blockNumber>: The block number to start the scan from. Defaults to the first block found in the index.
  • -l, --last <blockNumber>: The block number to end the scan at. Defaults to the last block found in the index.
  • -o, --out-file <baseFilePath>: Base path and filename for the output JSON files (without extension). Defaults to ./.repair/<chain>-<first>-<last>.

Example: 

# Quick scan (missing blocks only) for 'eos' chain
./hyp-repair quick-scan eos

# Quick scan with custom output
./hyp-repair quick-scan eos --first 1 --last 200000000 --out-file /repairs/eos-missing

Results

<output-path>-forked-blocks.json

Lists ranges where block linkage (prev_id -> block_id) is broken. Includes the block IDs identified as non-canonical within that range. Generated only by scan. 

[
  { "start": 123456, "end": 123458, "ids": ["forked_block_id_1", "forked_block_id_2"] }
]


<output-path>-missing-blocks.json:

Lists ranges of consecutive missing block numbers. 

[
  { "start": 10001, "end": 10005, "count": 5 }
]

Step 2: Analyze Scan Results

Review the generated JSON files to understand the extent of missing or forked data.

view <filePath>

Use the view command for a formatted table output:

Usage: 

./hyp-repair view <filePath>

Arguments:

  • <filePath>: Path to the JSON report file.

Example: 

# View missing blocks file
./hyp-repair view .repair/wax-100000000-150000000-missing-blocks.json

# View forked blocks file
./hyp-repair view .repair/wax-100000000-150000000-forked-blocks.json

Step 3: Repair Issues

3.1 Repairing Forked Blocks

repair <chainName> <filePath>

Repairs forked blocks based on a JSON file generated by the scan command. This command will:

  1. It reads the -forked-blocks.json file.
  2. For each range, it deletes the identified non-canonical block IDs from the *-block-* index in Elasticsearch.
  3. It deletes all associated actions, deltas, and state table entries (accounts, voters, etc.) within the block number range (start to end) of the fork from all relevant indices.
  4. After deletions, it instructs the indexer (via its control port) to re-fetch and re-index the block range (start to end), effectively replacing the forked data with the canonical data.


  • Purpose: To correct instances where Hyperion has indexed non-canonical blocks due to forks.

Data Deletion

This command permanently deletes data from your Elasticsearch indices. Always use --dry first and ensure you have backups.

Usage: 

./hyp-repair repair <chainName> <filePath> [options]

Arguments:

  • <chainName>: The alias of the chain.
  • <filePath>: Path to the JSON file containing the forked block ranges (generated by hyp-repair scan).

Options:

  • -d, --dry: Dry run. Shows what data would be deleted but does not perform any deletions or trigger re-indexing. Highly recommended for the first run.
  • -h, --host <hostAddress>: The hostname or IP address and port of the Hyperion indexer's control port (e.g., localhost:7002 or ws://indexer.internal:7002). If only host is given, it assumes the default control port for that chain.
  • -t, --check-tasks: Checks if previous ES delete_by_query tasks are still running. Lists active Elasticsearch tasks.

Example: 

# Dry run: See what would be repaired for WAX using a specific report file
./hyp-repair repair wax ./.repair/wax-100000000-110000000-forked-blocks.json --dry

# Actual repair (after verifying dry run output):
./hyp-repair repair wax ./.repair/wax-100000000-110000000-forked-blocks.json -h localhost:7002

3.2 Fill Missing Blocks

fill-missing <chainName> <filePath>

Instructs the Hyperion indexer (via its control port) to fetch and index blocks for ranges identified as missing. This uses a JSON file generated by hyp-repair scan or hyp-repair quick-scan.

  • Purpose: To fill gaps in Hyperion's indexed block data.
  • Behavior: Sends commands to the indexer's control port to process the missing block ranges.

Usage: 

./hyp-repair fill-missing <chainName> <filePath> [options]

Arguments:

  • <chainName>: The short name of the chain.
  • <filePath>: Path to the JSON file containing the missing block ranges.

Options:

  • -d, --dry: Dry run. Simulates the process and indicates what would be done, but does not actually send commands to the indexer to fill blocks.
  • -h, --host <hostAddress>: The hostname or IP address and port of the Hyperion indexer's control port.

Example: 

# Dry run: See which missing block ranges would be processed for 'telos'
./hyp-repair fill-missing telos ./.repair/telos_missing_scan-missing-blocks.json --dry

# Actual fill operation:
./hyp-repair fill-missing telos ./.repair/telos_missing_scan-missing-blocks.json -h localhost:7012 # Assuming Telos control port is 7012

Monitor the Hyperion indexer logs to observe the progress of filling missing blocks.

Step 4: Verify Repair

Verify the results

Run the scan again to verify that there are no more missing blocks or forks.

After the fill-missing or repair commands complete and the indexer has processed the requested ranges, re-run the scan or quick-scan command on the same block range(s) to confirm that no further issues are detected.

# Example re-scan
./hyp-repair scan wax --first 100000000 --last 150000000
If the scan reports no errors, the repair was successful.

Important Considerations

  • Resource Intensive: Both scanning (especially full scans) and repairing (deleting and re-indexing) can be resource-intensive on your Elasticsearch cluster and the Nodeos instance. Plan these operations during periods of low load if possible.
  • Time: Repairing large ranges or filling many missing blocks can take a significant amount of time.
  • --dry Runs: it is recommended to perform a --dry run for repair and fill-missing before executing them destructively to understand the scope of changes.
  • Log Monitoring: Closely monitor Hyperion indexer logs and Elasticsearch logs during any repair or fill operations.