The Challenge
Feature.io, a dynamic Web3 platform, faced mounting pressure to scale its data infrastructure. With an expanding user base and increasingly complex digital asset campaigns, the company needed a way to
- Reliably ingest Avalanche L1 / subnet blockchain data for advanced analytics.
- Ensure scalability as blockchain activity grows.
- Data cross-pollination from other information systems to provide real-time insights to marketing, operations, and product teams,
- Maintain tight security, observability, and uptime in a decentralized environment.
Their existing system lacked the robustness and flexibility to support the demands of data-hungry business users. Key stakeholders couldn't easily access on-chain activity, campaign performance, or user segmentation data, limiting data-driven growth strategies.
Codimite's Solution
Codimite engineered a GCP-native, serverless ETL pipeline, purpose-built to meet Feature.io's analytics goals. Instead of running dedicated blockchain infrastructure, the team managed Google Cloud services into a pipeline that scales with on-chain activity and needs minimal operational effort.
How the pipeline works
- Poll: Cloud Scheduler triggers Cloud Run services at configured 5–10-minute intervals.
- Track: The ingestion service retrieves the latest Avalanche block information and checks the processing cursor stored in Cloud Datastore.
- Queue: New blocks are published to Pub/Sub to decouple ingestion from downstream processing.
- Process: Cloud Run services process blockchain transactions and token-related data, loading the results into BigQuery.
- Analyze: BigQuery enables blockchain analytics, while dashboards and Cloud Monitoring support visibility into data activity and pipeline health.
Core services and why they matter
- Cloud Run runs scalable, containerized services for block ingestion and enrichment, with no servers to manage.
- Cloud Scheduler orchestrates timely polling intervals (5–10 minutes).
- Pub/Sub is the messaging backbone, decoupling ingestion and processing so each stage can scale independently.
- Cloud Datastore tracks the cursor, ensuring no duplication or missed blocks.
- BigQuery is the centralized analytics warehouse, structured with partitioning and clustering for high-performance querying.
- Secret Manager handles keys and credentials securely.
Security, governance and observability
- Cloud Monitoring & Alerting was set up for uptime tracking, pipeline health and email error notifications.
- Audit logs and compliance features were embedded to meet enterprise-grade standards.
- Dashboards in BigQuery and Cloud Monitoring enable easy tracking of smart contract usage, NFT interactions and user activity.
The architecture was fully walked through with the client and received positive feedback for clarity, robustness and security.
Business Impact
The implementation delivered a highly scalable, secure and reliable blockchain analytics pipeline for Feature.io:
- Streamlined ingestion of blockchain data from the Avalanche network.
- Stable and reliable operations, with monitoring and cursor recovery for resiliency.
- Performance-optimized analytics using BigQuery for deep event queries.
- Improved stakeholder visibility, allowing product and marketing teams to track engagement metrics, campaign completion rates and token usage patterns.
- The Feature.io team praised the architecture for its clarity, scalability and operational simplicity.