Case study 02
Protecting search over a 4 TB Elasticsearch corpus from runaway API clients
WhiteCrow Research · Software Engineer · Mumbai · Jan 2022 — Mar 2025
4TB
I also automated application deployments to the AWS instances through deployment scripts, implemented Google login for authentication, and used MySQL events for scheduled database maintenance.
Context and what was at risk
Talent search ran over a 4 TB JSON corpus, and an unthrottled API surface meant one heavy client could degrade the platform for everyone.
What I owned and built
I owned backend development for three years — REST APIs, data models, and client integrations — optimised the Elasticsearch queries and database operations behind search, and built an anti-scraping rate limiter with per-client request budgets and Fibonacci-escalating lockout windows for repeat offences inside 24 hours.
Stack
- Node.js
- Elasticsearch
- MySQL
- AWS
- REST API design
Built out of this work
Related writing
Sharding didn't just scale us. It cut the bill.
Vertical scaling has a brutal cost curve — every tier up charges a premium on top of the extra CPU and memory. Notes on how sharding a 4 TB search corpus turned that curve into a fleet of commodity nodes, and where the savings actually came from.
Fibonacci lockouts: rate limiting a scraper that keeps coming back
Per-client request budgets over a 4 TB search corpus, and lockout windows that escalate along a Fibonacci sequence — why that curve rather than exponential backoff, and what the 24-hour window is really for.