bumicerts-monorepo · marketplace-data-loads
Bumicerts marketplace data loading audit
41 autoresearch runs across 7 benchmark segments diagnosed why /explore and /organizations felt slow — hero image bloat and N-serial GraphQL calls were the dominant culprits, now fixed.
Headline result
Summary. Warm server TTFB was never the bottleneck — both routes return in under 2 ms each when the Next.js data cache is warm. The real user-visible slowness had two roots: (1) enormous uncompressed PNG hero images (7.6 MB total) that stall LCP, and (2) the organizations route issuing one GraphQL request per DID before hydrating organization cards. Both are now fixed: heroes are 900 px AVIF with only light-theme variants prioritised (21.5 KB total, 16.4 KB preload-critical), org accounts are fetched in a single batched query, the explore listing uses a card-only data adapter skipping unused fields, and both pages use 5-minute unstable_cache revalidation instead of force-dynamic. Strict Playwright data-card selectors confirm warm combined data-visible at ~285 ms. Cold production latency and client JS bundle remain the next open work.
Objective & scope
The user reported that /explore and /organizations felt slow for real data to appear after the loading skeleton. The autoresearch benchmark campaign set out to:
- Measure whether slow warm TTFB, heavy hero assets, or slow data fetches were responsible.
- Find and validate structural optimisations that reduce time-to-data-visible for real card selectors.
- Leave animations and UI behaviour untouched.
All 41 runs targeted commit 95b88da on branch autoresearch/slow-explore-organizations-2026-05-13.
Methodology
Seven benchmark segments were run in sequence, each with a dedicated primary metric:
| Segment | Metric | Runs | Goal |
|---|---|---|---|
| 0 | route_ms (warm TTFB) | 1–2 | Establish baseline; confirm warm server is not the problem |
| 1 | hero_kb (total hero bytes) | 3–8 | Compress hero images PNG→WebP→AVIF |
| 2 | preload_kb (crash) | 9 | Parser bug; segment aborted |
| 3 | preload_kb (priority hero bytes) | 10–18 | Reduce priority-preloaded hero bytes |
| 4 | browser_ms (full load event) | 19–26 | Browser-measured load; found org GraphQL batching win |
| 5 | route_ms (server data path) | 27–31 | Reduce SSR data-adaptation cost |
| 6 | data_visible_ms (generic selectors) | 32–36 | First selector-based data-visible metric |
| 7 | data_visible_ms (strict card selectors) | 37–41 | Strict [data-testid] selectors; final optimisation target |
Each iteration ran on a warm dev server (Next.js on port 3001). The autoresearch.sh script collected warm TTFB via curl (median of 3 iterations). The strict browser benchmark (bench-browser-loads.mjs) drove Chromium via Playwright, waiting for [data-testid="explore-bumicert-card"] and [data-testid="organization-card"] to appear in the DOM before recording timings.
Kept fixes
Hero images: PNG → AVIF 900 px/q32 (runs 3–13)
Fixed. Both explore and organizations hero backgrounds were large PNGs (~7.6 MB total). They were converted to WebP, then AVIF, progressively compressed to 900 px wide at quality 32. Total hero payload fell from 7,632 KB → 21.5 KB (99.7% reduction). Because the images are decorative backgrounds placed behind strong gradient overlays, the compression is visually acceptable; mandatory visual QA at retina/wide is still recommended.
public/images/explore/*.avif, public/assets/organizations/*.avif
Priority preload: light-theme only (run 11)
Fixed. Both light and dark hero variants were marked priority in Next.js <Image>, causing the browser to eagerly preload hidden variants. Dark images are now className="hidden dark:block" without priority so they lazy-load when the dark theme is active. Priority-preloaded bytes fell from 45 KB → 16.4 KB (−64%). A future theme-aware <picture>/media preload would do better still.
5-minute unstable_cache revalidation (run 2)
Fixed. Both pages previously used export const dynamic = "force-dynamic" and cache: "no-store" for the labeler fetch, defeating Next.js data cache reuse. Both now wrap their data calls with unstable_cache(..., { revalidate: 300 }) and export revalidate = 300. Warm TTFB dropped from ~14 ms to <4 ms.
Batched organization GraphQL (run 20)
Fixed. requestOrganizationAccountsByDidBatch previously looped over each DID and called requestOrganizationAccounts individually inside the 25-DID chunk loop, issuing N serial indexer requests. It was changed to pass all DIDs in one call with where did in [...]. Under the browser load metric, combined load time improved from 540 ms → 478 ms. Code lives in lib/account/list.ts:558.
Skip org long-description normalization for listing (run 27)
Fixed. listOrganizationData built full organization data including longDescription normalization for every org, even though listing cards only show displayName, shortDescription, media, country, and bumicertCount. The builder now accepts { includeLongDescription: false } for the listing path (lib/account/list.ts:897). Under the route_ms metric this dropped combined route time from ~14 ms to ~3 ms on the warm server.
Card-only explore adapter (runs 40–41)
Fixed. The explore listing adapter (activitiesToBumicertDataArray) normalised full Leaflet document descriptions, short-description facets, contributor lists, and locationRefs for every activity, even though explore grid cards never render those fields. Passing { cardOnly: true } skips that work (lib/adapters.ts:302). Under strict card-selector data-visible timing, combined data_visible_ms improved 339 ms → 267 ms in autoresearch (validated twice). Detail and account pages still receive the full data shape.
Strict card-selector browser benchmark (run 37)
Fixed. Earlier browser metrics used generic a[href^='/account/'] selectors which could match navigation chrome, not actual data cards. Run 37 switched to [data-testid="explore-bumicert-card"] and [data-testid="organization-card"]. This did not change app behaviour but made the benchmark correctly measure real data visibility rather than incidental DOM content.
Discarded hypotheses
Skip per-activity funding-config GraphQL (runs 22, 33)
Explore listing cards never read fundingConfig, but normalizeActivities fetches ActivityFundingConfig for each listed activity. Skipping those calls regressed primary metrics in both the browser_ms and data_visible_ms segments. Warm caching/concurrency likely hides the cost in a local dev environment. Worth retrying with a cold or production instrumented trace.
Increase org account batch size to 100 (run 23)
After batching org DIDs into a single query (run 20), raising the chunk size from 25 to 100 was expected to reduce serial pagination. Primary browser_ms regressed. Either the current dataset is small enough that batch size does not matter, or the larger payload increases indexer response time. Do not increase batch size without direct DID-count instrumentation.
Org labeler fetch: no-store → next revalidate (run 24)
The external organization labeler fetch inside requestRecentOrganizations used cache: "no-store". Switching to next: { revalidate: 300 } was plausible but regressed browser_ms. The outer unstable_cache wrapper may already absorb the labeler latency; the inner fetch option may be irrelevant in the warm local path.
Org activity-count page size 500 → 1000 (run 25)
Counting Bumicerts per organization pages through orgHypercertsClaimActivity. Raising the page size from 500 to 1000 was expected to reduce pagination round trips, but primary metric regressed. Instrument actual page count and indexer request durations before tuning page size again.
Lightweight org listing GraphQL projection (run 29)
A separate GraphQL document omitting longDescription for the listing query was implemented to reduce raw payload bytes from the indexer. Added code complexity and a new uncached query path; primary route_ms regressed. Prefer payload reduction at the indexer API layer rather than duplicating query logic.
Disable Next Link prefetching for card links (run 38)
Explore and organizations grids render many <Link> components that auto-prefetch in viewport. Disabling prefetch improved the post-load event browser time but worsened strict data-visible time, confirming prefetch contention is not a bottleneck for data becoming initially visible.
Force-static both listing routes (run 36)
Explicitly marking both pages force-static regressed data-visible time badly (339 ms → 689 ms). The routes appear to need streaming/dynamic behavior in the dev path; force-static interacts poorly with current data fetch and streaming configuration.
Precompute country lookup map (run 28)
Organization adaptation scans all countries for each org to find a location code. Pre-building the map was a correct micro-optimisation but did not move route_ms; country lookup is not a meaningful bottleneck compared with data fetch work.
Direct Bun instrumentation script (run 35 — crash)
An attempt to time server page functions directly via a Bun script crashed because the script could not resolve Next.js server-only imports outside the Next runtime. Instrument via the Next.js environment or tsx, not bare Bun execution.
Current metrics (verified 2026-05-13)
Cold-cache vs warm-cache gap. The first benchmark run of a fresh server session produced data_visible_ms=586.9 ms vs 285.1 ms on the second (warmed) run. This gap is the primary remaining user-visible issue: the Next.js data cache is cold on first requests and must hit remote indexer/labeler endpoints. Production cold-start is likely worse still. The warm figures above represent what repeat visitors experience after the first SSR warms the cache.
Next recommendations
Cold-start instrumentation
The warm dev server doesn't reproduce the slow user experience. Add server-side timing logs around listHighQualityBumicerts and listHighQualityOrganizations (the unstable_cache-wrapped functions) to measure actual cold indexer/labeler request durations in production. This should reveal whether the 5-minute cache TTL is hitting often enough or whether cache warming on startup would help.
Indexer-side listing projection
The org listing still fetches longDescription from the indexer even though normalization is now skipped. A dedicated lightweight listing GraphQL operation (omitting longDescription, fundingConfig) would reduce network payload from the indexer to the Next.js server. Measure indexer response bytes before and after to confirm the value.
Theme-aware image preloads
Currently both light and dark hero variants are in the DOM, but only one is prioritised. A <picture> element with media="(prefers-color-scheme: dark)" would let the browser preload only the active variant, halving priority preload bytes (16.4 KB → ~8 KB) without any dark-mode LCP regression.
Skip funding-config fetches — cold instrumentation only
The hypothesis that per-activity fundingConfig GraphQL is redundant for explore listing is structurally sound and worth validating in production. Run with explicit indexer call counting rather than relying on warm benchmark timing where concurrent caching obscures serial request costs.
Visual QA for AVIF hero quality
Hero images are now 900 px wide at quality 32, which is near the lower acceptable bound for desktop hero backgrounds. Perform visual review on large retina displays in both light and dark mode before shipping to production. If banding/softness is visible, try quality 38–42 (prior kept iterations were 1100 px/q38 and 1200 px/q42).
Reproducing
bun --cwd apps/bumicerts dev --port 3001
Warm TTFB benchmarkPORT=3001 BENCH_ITERATIONS=3 bash autoresearch.sh
Strict browser benchmarkPORT=3001 BENCH_ITERATIONS=3 node apps/bumicerts/scripts/bench-browser-loads.mjs
Branchautoresearch/slow-explore-organizations-2026-05-13
Commit95b88da — Add autoresearch benchmark for slow marketplace routes
Raw dataautoresearch.jsonl at repo root — 41 entries
Benchmark run log (run 2, warm cache)
TRACE iteration=1 route=/explore data_visible_ms=152.5 browser_ms=240.7 TRACE iteration=1 route=/organizations data_visible_ms=132.2 browser_ms=241.7 TRACE iteration=2 route=/explore data_visible_ms=175.1 browser_ms=271.8 TRACE iteration=2 route=/organizations data_visible_ms=174.9 browser_ms=254.6 TRACE iteration=3 route=/explore data_visible_ms=152.8 browser_ms=347.1 TRACE iteration=3 route=/organizations data_visible_ms=121.5 browser_ms=182.4 METRIC data_visible_explore_ms=152.8 METRIC data_visible_organizations_ms=132.2 METRIC data_visible_ms=285.1 METRIC browser_explore_ms=271.8 METRIC browser_organizations_ms=241.7 METRIC browser_ms=513.5
autoresearch.sh (warm TTFB, 2026-05-13)
TRACE using_existing_dev_server=1 port=3001 TRACE iteration=1 explore_ms=1.9 organizations_ms=3.6 TRACE iteration=2 explore_ms=1.7 organizations_ms=1.4 TRACE iteration=3 explore_ms=1.5 organizations_ms=1.4 METRIC preload_kb=16.4 METRIC hero_kb=21.5 METRIC explore_ms=1.7 METRIC organizations_ms=1.4 METRIC route_ms=3.1
All 41 autoresearch runs (summary)
| # | Status | Metric | Description |
|---|---|---|---|
| 1 | keep | route_ms=14 | Baseline warm TTFB — not the bottleneck |
| 2 | keep | route_ms=13.8 | 5-min unstable_cache + remove force-dynamic/no-store |
| 3 | keep | hero_kb=7632.6 | Baseline hero PNG payload |
| 4 | keep | hero_kb=304.7 | PNG → WebP (−96%) |
| 5 | keep | hero_kb=184.8 | WebP → AVIF (−39%) |
| 6 | keep | hero_kb=103.4 | AVIF 1280 px/q48 (−44%) |
| 7 | keep | hero_kb=65.4 | AVIF 1200 px/q42 (−37%) |
| 8 | keep | hero_kb=45 | AVIF 1100 px/q38 (−31%) |
| 9 | crash | preload_kb=0 | preload_kb parser bug — segment aborted |
| 10 | keep | preload_kb=45 | Baseline priority preload (all 4 variants) |
| 11 | keep | preload_kb=34.9 | Remove priority from dark hero variants (−22%) |
| 12 | keep | preload_kb=22.6 | AVIF 1000 px/q34 (−35%) |
| 13 | keep | preload_kb=16.4 | AVIF 900 px/q32 (−27%) ← current kept |
| 14 | discard | preload_kb=11.6 | 800 px/q30 — too quality-risky |
| 15 | discard | preload_kb=11.6 | Remove orgs light priority — harms orgs LCP |
| 16 | discard | preload_kb=13.9 | 850 px/q31 explore only — marginal gain, quality risk |
| 17–18 | discard | preload_kb=16.4 | Control reruns — no code change |
| 19 | keep | browser_ms=540.3 | Browser baseline (load event via Playwright) |
| 20 | keep | browser_ms=477.8 | Batch all org DIDs in one GraphQL call |
| 21 | discard | browser_ms=656.5 | Label cache reuse — warm benchmark regressed |
| 22 | discard | browser_ms=581.9 | Skip funding-config — warm benchmark regressed |
| 23 | discard | browser_ms=635 | Batch size 100 — regressed |
| 24 | discard | browser_ms=669.7 | Org labeler cache — regressed |
| 25 | discard | browser_ms=603.9 | Activity page size 1000 — regressed |
| 26 | discard | browser_ms=664.7 | Skip org long-desc (under browser_ms) — regressed |
| 27 | keep | route_ms=3.3 | Skip org long-desc normalization (under route_ms) ← kept |
| 28 | discard | route_ms=14.2 | Country lookup precompute — not a bottleneck |
| 29 | discard | route_ms=13.8 | Lightweight org listing GraphQL — regressed |
| 30 | discard | route_ms=13.4 | Card-only explore adapter (under route_ms) — regressed |
| 31 | discard | route_ms=10.1 | Control rerun — noise estimate |
| 32 | keep | data_visible_ms=323.3 | Selector-based data-visible baseline (generic selectors) |
| 33 | discard | data_visible_ms=378.1 | Skip funding-config — regressed under data_visible |
| 34 | discard | data_visible_ms=380.8 | Control rerun — noise estimate |
| 35 | crash | data_visible_ms=0 | Bun instrumentation script — module resolution crash |
| 36 | discard | data_visible_ms=689 | force-static — badly regressed |
| 37 | keep | data_visible_ms=339.1 | Strict card-selector benchmark ← measurement fix |
| 38 | discard | data_visible_ms=378.2 | Disable Link prefetch — data-visible worsened |
| 39 | discard | data_visible_ms=345.3 | Remove cache-busting query param — no gain |
| 40 | keep | data_visible_ms=315.8 | Card-only explore adapter (under strict selector) |
| 41 | keep | data_visible_ms=267.7 | Validation rerun — improvement confirmed |