{"id":7618,"date":"2025-10-12T19:59:23","date_gmt":"2025-10-12T19:59:23","guid":{"rendered":"https:\/\/techtrendfeed.com\/?p=7618"},"modified":"2025-10-12T19:59:24","modified_gmt":"2025-10-12T19:59:24","slug":"we-benchmarked-duckdb-sqlite-and-pandas-on-1m-rows-right-heres-what-occurred","status":"publish","type":"post","link":"https:\/\/techtrendfeed.com\/?p=7618","title":{"rendered":"We Benchmarked DuckDB, SQLite, and Pandas on 1M Rows: Right here\u2019s What Occurred"},"content":{"rendered":"


\n<\/p>\n

\n

\"DuckDB\"DuckDB
Picture by Creator<\/span><\/center>
\n\u00a0<\/p>\n

#\u00a0<\/span>Introduction<\/h2>\n

\u00a0
There are quite a few instruments for processing datasets right now. All of them declare \u2014 after all they do \u2014 that they\u2019re one of the best and the precise alternative for you. However are they? There are two predominant necessities these instruments ought to fulfill: they need to simply carry out on a regular basis knowledge evaluation operations and achieve this rapidly, even underneath the stress of huge datasets.<\/p>\n

To find out one of the best software amongst DuckDB<\/a><\/strong>, SQLite<\/a><\/strong>, and Pandas<\/a><\/strong>, we examined them underneath these circumstances.<\/p>\n

First, we gave them solely on a regular basis analytical duties: summing values, grouping by classes, filtering with circumstances, and multi-field aggregations. This mirrored how analysts really work with actual datasets, in comparison with eventualities designed to showcase one of the best traits of a software.<\/p>\n

Second, we carried out these operations on a Kaggle dataset with over 1 million rows. It\u2019s a sensible tipping level \u2014 sufficiently small to run on a single machine, but giant sufficient that reminiscence stress and question pace begin to reveal clear variations between instruments.<\/p>\n

Let\u2019s see how these exams went.<\/p>\n

\u00a0<\/p>\n

#\u00a0<\/span>The Dataset We Used<\/h2>\n

\u00a0<\/p>\n

\/\/\u00a0<\/span>Dataset Overview<\/h3>\n

We used the Financial institution dataset from Kaggle<\/a><\/strong>. This dataset comprises over 1 million rows, comprising 5 columns:<\/p>\n

\u00a0<\/p>\n\n\n\n\n\n\n\n\n\n
Column Title<\/th>\nDescription<\/th>\n<\/tr>\n<\/thead>\n
Date<\/td>\nThe date the transaction occurred<\/td>\n<\/tr>\n
Area<\/td>\nThe enterprise class or sort (RETAIL, RESTAURANT)<\/td>\n<\/tr>\n
Location<\/td>\nGeographic area (Goa, Mathura)<\/td>\n<\/tr>\n
Worth<\/td>\nTransaction worth<\/td>\n<\/tr>\n
Transaction_count<\/td>\nThe whole variety of transactions on that day<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

\u00a0<\/p>\n

This dataset is generated utilizing Python. Whereas it could not totally resemble real-life knowledge, its measurement and construction are enough to check and examine the efficiency variations between the instruments.<\/p>\n

\u00a0<\/p>\n

\/\/\u00a0<\/span>Peeking Into the Information with Pandas<\/h3>\n

We used Pandas<\/a><\/strong> to load the dataset right into a Jupyter pocket book and look at its normal construction, dimensions, and null values. Right here is the code.<\/p>\n

\n
import pandas as pd
\ndf = pd.read_excel('bankdataset.xlsx')
\n
\nprint(\"Dataset form:\", df.form)
\n
\ndf.head()<\/code><\/pre>\n<\/div>\n
\u00a0<\/p>\n
Right here is the output.<\/p>\n
\u00a0
\"DuckDB\"DuckDB
\u00a0<\/p>\n
In order for you a fast reference to widespread operations when exploring datasets, take a look at this helpful Pandas Cheat Sheet<\/a>.<\/p>\n
Earlier than benchmarking, let\u2019s see  arrange the setting.<\/p>\n
\u00a0<\/p>\n
#\u00a0<\/span>Setting Up a Truthful Testing Setting<\/h2>\n
\u00a0
All three instruments \u2014 DuckDB, SQLite, and Pandas \u2014 had been arrange and run in the identical Jupyter Pocket book setting to make sure the check was honest. This ensured that the circumstances throughout runtime and the usage of reminiscence remained fixed all through.<\/p>\n
First, we put in and loaded the mandatory packages.<\/p>\n
Listed below are the instruments we wanted:<\/p>\n