TokenHub by FNB Solutions: Canonicalizing Digital Product Feeds at Scale

Introduction

TokenHub is FNB Solutions’ data and commerce backbone for distributing and selling digital game tokens across a fragmented ecosystem of providers.
It operates across regions where global payment infrastructure is uneven and local resellers remain the backbone of digital commerce. By connecting these sellers with digital providers, TokenHub enables customers to purchase tokens through trusted local shops using prepaid balances, while maintaining consistent fulfillment and reconciliation at scale.

It aggregates hundreds of titles - ranging from PUBG and Free Fire to Call of Duty - across multiple regional partners, each publishing independently managed catalogs. Provider feeds vary widely in format, identifiers, naming conventions, and currency semantics. Some describe packages by value (“70,000 Credits”), others by SKU (“Pack 70k”), and others by composition (“60k + 10k Bonus”).
Without normalization, TokenHub can’t reliably compare prices, route purchases, or produce accurate analytics.

To solve this, TokenHub implements a deterministic canonicalization system that reconciles provider-specific SKUs into stable, auditable canonical entities - minimizing manual curation while ensuring consistency at scale.

Architectural Overview

TokenHub’s canonicalization engine follows a two-tier architecture:

Ephemeral Snapshot Layer - Ingests, normalizes, clusters, and inspects provider data into immutable, per-run snapshots.
Canonical Persistence Layer - Stores the approved canonical products, packages, bindings, and curation rules used by storefront and pricing services.

Each provider import runs as an immutable CanonJob - a point-in-time, queryable snapshot that can be safely replayed without altering canonical data.

Core Entities

Idempotency: Identical inputs always produce identical canonical outputs. Verified relationships remain stable across re-ingestions.

Provider Integration and Normalization

Each integration defines a JSON configuration describing the provider’s API structure and parameters:

json
{
  "products": { "path": "/products", "method": "GET", "optionalParams": ["page", "offset"] },
  "purchase": { "path": "/order", "method": "POST", "requiredParams": ["id", "number", "quantity"] }
}

Translator modules (e.g., translateProviderProducts) map provider responses into a standardized internal format:

json
{
  "name": "BOLI",
  "category": "provider-category-299",
  "packages": [{
    "name": "BOLI",
    "offerings": [{
      "externalProductId": "878",
      "priceCents": 0,
      "currency": "USD",
      "metadata": {
        "minAmount": 10000,
        "maxAmount": 50000000,
        "requirements": [{ "name": "player_number", "label": "Player number" }]
      }
    }]
  }]
}

Normalization is deterministic. All identifiers are lowercased and Unicode-normalized, numeric quantities are parsed with explicit units, and currency values are stored as integer cents. Provider-defined requirements and metadata are preserved in full.

Each offering’s region attribute reflects the store or sales market (e.g., iTunes Italy or iTunes France), not the player’s physical location. This distinction preserves accurate regional pricing, taxation, and catalog visibility without assumptions about end-user geography.

Provider-Level Canonicalization

Before cross-provider alignment, TokenHub performs provider-level canonicalization. Each provider’s catalog is internally unified so that duplicate entries - such as Free Fire, Garena Free Fire, and free fire - collapse into one provider-level canonical product. This ensures internal consistency within each provider’s feed before global canonical mapping.

Within each provider, TokenHub also performs package consolidation, identifying packages that are semantically identical even if presented differently - such as “70k coins,” “60k + 10k bonus,” or “70,000 credits.” By deriving a semantic signature for each package (covering normalized quantity, unit, duration, and variant flags like bonus or subscription), the system merges equivalent offerings while preserving original SKU references for auditability.

Canonicalization Pipeline

Each canonicalization run executes within a single database transaction per CanonJob.

Pass 1 - Product Clustering

The first stage, Product Clustering, identifies which incoming products likely represent the same underlying game or digital entity. Products are grouped by normalized text tokens and semantic similarity - so variants like “FREE FIRE,” “FREE FIRE CODE,” and “Garena Free Fire” are clustered together into a single candidate group.

This process combines deterministic normalization with fuzzy matching and alias heuristics. It considers normalized names, token frequency, and prior confirmed merges to avoid false positives. When available, metadata such as store category or provider channel also contributes to clustering confidence.

Each cluster is recorded as a CanonSnapProduct, complete with a computed canonicalName, a normKey (the normalized token signature), a list of memberIndexes, and links to source feed entries. These clusters become the atomic units of review and curation during the next phase.

Beyond reducing duplication, clustering provides semantic compression - turning thousands of loosely related names into a few hundred coherent groups. This enables faster subsequent processing, predictable suggestion generation, and scalable manual review workflows.

Pass 2 - Package Consolidation

Within each product cluster, TokenHub performs Package Consolidation, which normalizes and merges redundant or equivalent packages under a unified semantic signature.

A package signature is computed using key normalized attributes: package name, quantity, unit (credits, diamonds, etc.), value range, required parameters (such as player IDs), and type flags (bonus, subscription, time-limited, etc.). By representing packages through signatures rather than surface names, the system can reliably detect that “70k Credits,” “60k + 10k Bonus,” and “70,000” are equivalent offerings.

Packages with matching signatures merge into a single CanonSnapPackage record, which becomes the canonical representative for that configuration. Offerings remain distinct to preserve price and currency differences, while the lowest valid active price within each group is marked as isPrimary for downstream use.

Package consolidation dramatically reduces catalog noise and prevents over-segmentation. It also improves the accuracy of cross-provider comparisons by ensuring that equivalent value propositions are recognized as such, regardless of local naming or formatting differences.

Pass 3 - Suggestion Graph

After package consolidation, TokenHub builds a Suggestion Graph, a network of potential equivalences and relationships between canonical candidates across clusters. Each edge in this graph represents a machine-generated similarity hint - for example, that "Free Fire Diamonds 70k" from one provider may match "70,000 FF Credits" from another.

The graph is computed using a combination of normalized token overlap, semantic distance metrics, numeric equivalence, and learned patterns from prior accepted merges. These similarity scores are persisted as CanonSnapSuggestion records, each annotated with the source cluster IDs, confidence values, and audit metadata linking back to their origin.

Operationally, the Suggestion Graph is the backbone of TokenHub’s admin review and curation workflow. It allows curators to act quickly and confidently, presenting high-probability merge candidates first while suppressing redundant or previously rejected comparisons. When an admin accepts or rejects a suggestion, that feedback is immediately incorporated into future graph computations, allowing the system to “learn” from human input and continuously improve accuracy.

For large imports - often encompassing tens of thousands of product variants - the graph transforms what would otherwise be a manual comparison exercise into an efficient, targeted review process. Instead of combing through all SKUs, administrators can focus only on meaningful merge candidates, reducing decision latency and preventing fatigue-related errors.

The Suggestion Graph also plays a key role in auditability and reproducibility. Every edge is timestamped, versioned, and traceable to a specific ingestion run, ensuring that admins and auditors can understand why two records were presented as similar, even months later.

In short, the Suggestion Graph serves as TokenHub’s cognitive map: it connects semantically related nodes, guides human reviewers through the shortest path to canonical convergence, and ensures that future runs are faster, cleaner, and more accurate than the last.

Admin Editing, Rules, and Precedence

All administrative edits occur on the snapshot layer and remain fully reversible. When an admin renames a product, merges clusters, or suppresses unwanted entries, the system records each action as a structured edit object:

json
{ "type": "rename", "target": "canonSnapProduct:42", "value": "FREE FIRE" }

During editing, TokenHub maintains a local buffer - canon:job:<id>:edits - that supports undo, redo, and crash recovery. When the Apply action is executed, TokenHub upserts or updates relevant canonical records (CanonicalProduct, CanonicalPackage, and Offering), establishes provider-to-canonical links in CanonicalBinding, and persists all curation logic as deterministic rules.

Rename rules correct product labels, package rename rules adjust package naming, and suppression rules hide items from the storefront or search index. Negative pairs record explicitly rejected merges to prevent redundant future suggestions. Every applied change is captured in the immutable CanonEditJournal, ensuring complete traceability.

Rule precedence follows a predictable order: suppression rules override everything, followed by rename and package rename rules, with raw canonical names used as a fallback. On subsequent imports, these rules and bindings are automatically applied before generating new suggestions, minimizing manual work and ensuring consistent, stable naming across re-ingestions.

Human-in-the-loop curation is an essential part of this process. Each verified correction improves the machine’s ability to make accurate automatic suggestions in future runs, allowing automation quality to evolve organically without compromising oversight.

Example: Snapshot to Persistence

Input Snapshot

json
{
  "canonicalName": "FREE FIRE",
  "packages": [{
    "key": "110|free|fire",
    "offerings": [
      {"providerId": "provider", "externalProductId": "878", "priceCents": 10000},
      {"providerId": "provider", "externalProductId": "879", "priceCents": 12000}
    ]
  }]
}

Resulting Database Writes

Refresh, Architecture, and Versioning

Provider imports are safe, idempotent, and replayable. Existing bindings automatically reattach SKUs, updating prices and availability in place. Newly added SKUs appear as fresh snapshot rows, while deprecated entries are hidden or deactivated through suppression rules. This design ensures continuous, non-destructive evolution of the catalog without manual rework.

TokenHub’s persistence model is implemented on PostgreSQL with Prisma ORM, providing a transparent, explicit schema. Each ingestion or edit cycle is logged and versioned for reproducibility and rollback. Alias dictionaries maintain both provider-specific and global naming states, while negative pairs record explicit “do not merge” constraints that protect canonical precision. All operations are transactional and verifiable, ensuring a reproducible and auditable data lineage from raw feeds to canonical entities.

Auditability, Provenance, and Security

TokenHub maintains full lineage from the provider’s raw payloads through snapshot records to final canonical entities and offerings. Every modification is wrapped in a single atomic transaction and logged in CanonEditJournal, allowing administrators to trace the origin and reasoning behind every record. Cryptographic hashes guarantee integrity, and deterministic replay ensures that any historical state can be reconstructed and verified with precision.

In production, TokenHub enforces strict role-based access control, isolates provider credentials from application logic, and avoids collecting personal user data. Operational metrics - such as ingestion success rates, alias coverage, duplication ratio, and curation throughput - are continuously monitored to maintain catalog quality and reliability.

Operational Characteristics

The system operates with strict transactional consistency: each canonicalization cycle commits as one atomic unit, ensuring no partial states. Deterministic mapping rules guarantee that identical inputs always yield identical outputs, while immutable snapshots preserve reproducible, point-in-time views of the data. Learned rules and negative pairs suppress noise from redundant or low-confidence matches, keeping the suggestion graph precise and actionable. Storefront safety is guaranteed by suppression rules, ensuring deprecated or excluded items remain hidden across all surfaces.

Outcome

TokenHub’s canonicalization framework delivers a verifiable, repeatable foundation for managing complex digital product ecosystems. By transforming inconsistent provider feeds into a unified canonical layer, it allows every downstream system - pricing, storefront, analytics, and reconciliation - to operate on a single source of structured truth.

The result is a catalog that evolves safely: prior curation persists automatically, deterministic rules prevent regression, and every transformation remains auditable from source payload to canonical record. Operators gain predictable import cycles and clean diffs; engineers gain an architecture that is both traceable and reproducible; and data governance teams gain confidence that identical inputs always yield identical, verifiable outcomes.

Rather than a static mapping exercise, TokenHub functions as a continuous normalization engine - one that improves with every review cycle, scales across new providers without redesign, and enforces consistency as a property of the system itself.