Project Record

Crypto Reputation Economy Audit

Phase-1 audit and scoring prototype for a crypto social platform: heavy-tail distribution analysis, rarity-driven achievement automation, and no-downgrade rollout simulation across 24k+ users.

Executive Summary

A crypto social reputation platform had grown to tens of thousands of users, but its point economy was still tuned by hand. Every new on-chain or social achievement needed manual REP assignment, and the relationship between badge rarity and point value was weak and inconsistent. Product and engineering needed to understand whether the economy was healthy before automating it or layering in a continuous social score.

I ran a Phase-1 audit on a snapshot of roughly 24,000 users, 88 achievement definitions, and a four-level referral tree. The exploratory work surfaced extreme inequality in social-derived points, heavy-tailed referral growth, and a rank distribution concentrated in the middle tiers. That evidence justified moving from fixed per-badge tuning to a rarity-driven assignment model, with explicit rollout math so existing users would not be downgraded.

The deliverable was a modular scoring prototype: distribution notebooks, a Model v1 pipeline that improved REP–rarity correlation from r ≈ 0.33 to r ≈ 0.71, a graph-based social score overlay, blend/floor simulation scenarios, a stakeholder demo package, and a config dashboard for transparent parameter review. This demonstrates heavy-tail exploratory analysis, scoring-system design under production constraints, and rollout planning for live reputation economies.

Problem

Given a platform where users accumulate reputation from achievements, referrals, and social signals, characterize the current economy and design an automatic assignment rule that scales with badge rarity rather than per-achievement manual tuning. The system must support multiple achievement semantics (fixed badges, USD-threshold tiers, NFT floors, gift levels), remain auditable, and allow a social-score overlay without slashing existing user standings.

The immediate technical questions were: how unequal is the current distribution, which generative models fit the tails, whether rarity alone can explain point values better than the legacy manual table, and how to migrate without a disruptive budget or rank shock.

Distribution & Tail Analysis

The audit started from platform exports: user records with achievement lists, referral-tree nodes, and achievement definitions. Exploratory scripts and a Phase-1 notebook computed quantile tables, histograms, complementary CDFs, Lorenz curves, and inequality summaries across achievement REP, social REP, total REP, referral depth counts, and achievements per user.

Key structural findings on the ~24k-user snapshot:

Tail and count models were compared with AIC/BIC and Kolmogorov–Smirnov checks:

• Direct invites and subtree size — best heavy-tail fit: Pareto
• Social REP (non-zero) — best heavy-tail fit: lognormal
• Achievements per user — best count fit: negative binomial
• Direct invites (including zeros) — negative binomial with high zero mass

The referral tree was shallow in practice: most users sit at depth zero with sparse downstream growth, while a thin tail carries large subtrees. Social REP behaves like a separate concentrated layer on top of the broader achievement economy. These shapes matter for rollout: a rarity-only model will not automatically reproduce a hyper-concentrated legacy social layer without an explicit overlay or recalibration step.

Achievement Rarity Model

Model v1 assigns automatic reputation from live user distribution rather than a static manual table. For achievement $a$ with $n_a$ holders among $N$ users, rarity weight is

Higher $w_a$ means fewer users hold the badge today. Base REP depends on pointsType:

• Static: $\text{base} = s_{\text{static}} \cdot w_a^{p}$
• AmountUSD: $\text{base} = s_{\text{amount}} \cdot t_a^{\alpha} \cdot w_a^{p}$

where $t_a$ is a USD threshold parsed from achievement metadata, $\alpha \in (0,1)$ encodes diminishing returns as thresholds rise, and $p$ is a global rarity exponent. Type, snapshot, and token-risk multipliers scale the result before clamping to configured floors and ceilings.

Evaluation against the legacy manual table showed the core improvement: Pearson correlation between contributed REP and rarity rose from r ≈ 0.33 (manual) to r ≈ 0.71 (automatic). Total contributed REP under a straight replacement was much lower than the legacy budget (~45M vs ~144M on the same snapshot), which is expected when manual inflation is removed — and which motivates phased rollout rather than a hard cutover.

Residual analysis exported achievements whose manual values were far above or below what rarity and threshold structure would predict, giving product a prioritized retuning list.

Social Score Overlay

Alongside achievement-based REP, the platform needed a continuous 0–100 social score derived from third-party Twitter graph APIs, replacing categorical social tiers. The composite blends three sub-scores:

where $C$ measures crypto-native graph embedding, $G$ measures follower quality (mean score of top followers), and $T$ captures account stability and behavioural trust signals. Each feature uses log1p normalization with fixed caps so scores stay stable without global percentile recomputation on every run.

The module includes labelled calibration fixtures, JSON caching with TTL for API cost control, drift monitoring on mean and tail quantiles, and an explainability payload per handle. Final user standing composes achievement reputation and social score with configurable weights (default 60% / 40%). For a related standalone profiler built on the official X API with a different feature set, see the X Account Profiler project.

Rollout Simulation

Because Model v1 reprices achievements relative to rarity, a direct replacement would reshuffle ranks and shrink total REP. Two guardrails were specified for migration:

Blend (phased):

Floor (no-downgrade):

Example parameters: $\lambda = 0.7$ keeps 30% of legacy value in the mix; $\beta = 0.85$ guarantees no badge drops below 85% of its current REP. A third option recalibrates global scale $S$ so total contributed REP matches a budget target at the current user distribution, then transitions new achievements to rarity-only assignment.

Scenario tables and before/after rank simulations were packaged for stakeholder review alongside the distribution charts and correlation report.

Implementation

The prototype is modular Python with JSON configuration and a small Next.js dashboard:

1. Export & EDA — user and referral-tree snapshots; terminal summaries and Phase-1 distribution notebook (histograms, CCDF, Lorenz, model fits).
2. Reputation pipeline — build_achievement_features → assign_rep_model_v1 → evaluate_rep_model_v1; outputs feature tables, auto-assigned REP, correlation/budget report, and diagnostic plots.
3. Social score — batch CLI with caching, calibration checks, and drift reports.
4. Composition — weighted merge of reputation and social layers into a final score.
5. Demo package — scripted stakeholder bundle: slides narrative, scenario tables, charts, and runnable notebook.
6. Config dashboard — read-only UI for model weights, multipliers, and calibration fixtures at build time.

Configs separate scales, type multipliers, snapshot overrides, risk coefficients, and social-score caps so product can inspect parameters without reading pipeline code.

Trade-offs

Rarity counts were joined on achievement titles in the export, which breaks when product renames badges — a stable achievement key in the user snapshot is a prerequisite for production automation. Alpha for USD-threshold curves was initially fitted from legacy REP, which is circular; a fixed policy alpha or one-time calibration is cleaner for a live pipeline. Snapshot and risk multipliers remain partly heuristic until the achievement schema carries structured tags.

The audit used a single export snapshot, so rarity is point-in-time; a scheduled refresh cadence is needed for automation. Social REP concentration (Gini ≈ 0.98) means a separate overlay or recalibration step is required — rarity-based achievement scoring alone will not reproduce that legacy layer. The blend/floor math trades mathematical purity for user trust during migration, which is the right trade in a live reputation product.

Related Work