Lawrence Jengar
Aug 01, 2024 16:06
Dune has significantly improved its API by integrating DuckDB, enabling pagination, filtering, and handling larger datasets efficiently.
Dune has announced substantial enhancements to its API through the integration of DuckDB, a move that promises to address user requests for pagination, larger query results, and additional functionalities such as filtering and sorting, according to Dune.com.
Motivation and Context
The improvements stemmed from user feedback highlighting the need for pagination and the ability to handle larger query results, which were previously capped at 1GB. Dune’s original architecture was designed primarily for visualizations on dashboards, which did not necessitate large datasets or pagination. This limitation prompted the development team to prioritize these features towards the end of 2023.
Expanding Use Cases
To better serve the developer community, Dune adopted a holistic approach, focusing on real-world use cases rather than specific feature requests. This strategy involved creating user stories that detailed various needs, such as searching and filtering large datasets, feeding data into charts, and performing data science on crypto transactions. These insights guided the development of a more flexible and robust API.
DuneSQL and Query Results
All data on Dune is queryable using DuneSQL, which employs Trino, a distributed query engine. While DuneSQL is powerful, it posed challenges for the new requirements, such as low-latency responses and cost-effective execution. This led the team to seek alternative solutions.
Adoption of DuckDB
After evaluating various options, Dune chose DuckDB for its ability to support JSON and Parquet data formats, high-performance SQL engine, flexibility, and ease of use. DuckDB allowed Dune to load and cache query results quickly, enabling API requests to be served within sub-100 millisecond response times. This integration significantly improved the API’s performance and cost-efficiency.
New Features and APIs
The integration of DuckDB has enabled several new features for the Dune API:
- Pagination: Retrieve data in manageable chunks.
- Filtering: Apply filters based on specific columns and criteria.
- Sorting: Organize query results in a specified order.
- Sampling: Retrieve a uniform sample of the dataset for efficient analysis.
Moreover, these features integrate seamlessly with other Dune functionalities, such as the Query Scheduler and Materialized Views, enhancing the overall developer experience.
Conclusion
Dune’s integration of DuckDB represents a significant step forward in addressing user needs and expanding the capabilities of its API. This strategic move not only resolves immediate feature requests but also sets the stage for future enhancements, making DuneAPI a more powerful tool for developers.
For more details, visit Dune’s API documentation.
Image source: Shutterstock