Python 1991 → 2026: From a Hobby Language to PySpark — Do Finance Teams Need Spark for 12 Million Rows?
Same ledger, four engines, identical answers. One took a second; Spark took twenty.
Python went from a hobby project in 1991 to a cell in Excel, a script in Power BI and a notebook in Microsoft Fabric. We traced that history — then put plain Python, pandas, DuckDB and PySpark through the same four finance questions on a general ledger of about 12 million lines from DATA DARE.
For a ledger of 12 million lines you do not need Spark. DuckDB answered our four finance questions in about 1 second, pandas in about 4, PySpark in about 20 — 5 of them just starting up — and plain Python in 26, after the 368 MB Parquet file was turned into a 1.8 GB CSV. Spark earns its place when data no longer fits one machine.
The test in three numbers
Four ways to run Python on the same general ledger from DATA DARE DD-MFG-005 — about 12 million lines, 368 MB of Parquet.
~1 sDuckDBAll four finance questions, straight from the Parquet file.
~20 sPySparkAbout 5 of those seconds just to start Spark.
4 of 4identical answersPlain Python, pandas, DuckDB and PySpark agreed on every question.
Python 1991 → 2026: from a hobby language to finance's second tool
1991First public releaseGuido van Rossum started Python at CWI in the Netherlands in December 1989 and posted the code in February 1991.
2000Python 2List comprehensions, garbage collection and Unicode.
2006NumPyFast arrays — the base every data library still stands on.
2008pandas — born in a hedge fundWes McKinney started pandas at AQR Capital Management to analyse financial data; open source from 2009.
2008Python 3A cleaner language that did not run Python 2 code unchanged.
2014Spark becomes an Apache top-level projectBorn at UC Berkeley's AMPLab in 2009; built to spread data across many machines.
2018Python in Power BIPython scripts as a data source and Python visuals in Power BI Desktop.
2020Python 2 retiresNo more fixes from 1 January 2020.
2024Python in ExcelGenerally available in September 2024 for Microsoft 365 Business and Enterprise on Windows.
2025Databricks Free EditionSpark notebooks free for learning, launched in June 2025.
2026Python 3.15Released 9 October 2026 — lazy imports, UTF-8 by default and a faster JIT.
May 2027Python visuals in Power BI retireR and Python visuals stop working in published reports; Python in Power Query is not affected.
Plain Python vs libraries
Plain Python is the language and its standard library — reading a CSV, adding numbers, working with dates. It runs everywhere with nothing to install, but it is slow on big data and cannot read a Parquet file at all.
Libraries are tools other people built on top of it: NumPy for arrays, pandas for tables, DuckDB for SQL on files, PySpark for Spark, matplotlib for charts, openpyxl for Excel files. Think of plain Python as Excel without formulas; the libraries are the SUMIFS, pivot tables and Power Query.
Where Python lives for a finance team in 2026
Where
What you can do
Good to know
Excel
Type =PY( in a cell and use pandas and charts inside the workbook.
Runs in Microsoft's cloud; Microsoft 365 Business and Enterprise.
Power Query (Power BI)
A Python script as a data source or a transformation step.
Not affected by the 2027 retirement.
Power BI visuals
Charts drawn with matplotlib inside a report.
Retire in May 2027 — Microsoft points to Fabric notebooks.
Microsoft Fabric
Python notebooks (one machine) and Spark notebooks.
Tables are stored as Delta Parquet in OneLake.
Databricks Free Edition
Spark and Python notebooks in the browser.
Free for learning, not for commercial work.
Your laptop
Positron, VS Code or Jupyter with pandas and DuckDB.
Free, private, and fast enough for tens of millions of rows.
The test: four engines, one ledger, the same four questions
We ran each engine in a fresh process on the DD-MFG-005 general ledger and asked the questions a finance team asks every month:
Total amount by entity and period.
Number of lines by source (sales, purchases, payroll and so on).
The 20 accounts with the largest movements.
Number of distinct journals per entity.
Time to answer all four questions
Plain Python26 s
PySpark21 s
pandas4.3 s
DuckDB1.0 s
Median of three runs (two for PySpark, one for plain Python). Machine: 2 virtual CPUs, 7 GB of memory — slower than most office laptops.
Full results
Engine
Total time
Before the first answer
Slowest question
Peak memory
DuckDB 2.0 (SQL on the Parquet file)
1.0 s
nothing to load — reads only the columns it needs
distinct journals, 0.7 s
0.5–0.6 GB
pandas 3.0 (read 6 columns of Parquet)
4.3 s
read 0.8 s
distinct journals, 2.1 s
3.1 GB
PySpark 4.0 (local, 2 cores)
20–22 s
start Spark 4.5–5.9 s
distinct journals, 7.9 s
2.2–2.3 GB (with Java)
Plain Python (csv module)
26 s
needs the ledger as a 1.8 GB CSV first
all four in one pass
0.7 GB
DD-MFG-005 is a Level 5 set with hidden answers, so only timings, sizes and memory are shown — never amounts or counts.
Which one should a finance team pick?
Decided by the test and by what each tool was built for.
DuckDB vs pandas
Pick for analysis: DuckDB for big files
4× faster in our test and used a fifth of the memory, because it reads only the columns a question needs.
Keep the other when: you want the most tutorials, or the data already sits in a pandas table.
pandas vs PySpark
Pick for analysis: pandas on one machine
5× faster here; Spark spent about 5 seconds just starting up.
Keep the other when: the data no longer fits one machine, or your company runs on Databricks or Fabric Spark.
Plain Python vs Libraries
Pick for analysis: Libraries
Plain Python got the right answers, but needed a 1.8 GB CSV instead of the 368 MB Parquet file — and took 26 seconds.
Keep the other when: you cannot install anything, or the file is small.
The same question in DuckDB and pandas
Distinct journals per entity — the slowest question for every engine:
import duckdb, pandas as pd
# DuckDB: SQL straight on the Parquet file
duckdb.sql("""
SELECT EntityCode, COUNT(DISTINCT JournalID) AS journals
FROM 'DD-MFG-005_FactGLLines.parquet'
GROUP BY EntityCode
""").df()
# pandas: load the columns you need, then group
gl = pd.read_parquet('DD-MFG-005_FactGLLines.parquet', columns=['EntityCode', 'JournalID'])
gl.groupby('EntityCode')['JournalID'].nunique()
The test pack has all four engines and a script that times them on your own laptop.
So when does a finance team need Spark?
When the data no longer fits on one machine, when several teams share one lakehouse, or when pipelines run every hour on a schedule. That is what Databricks and Microsoft Fabric's Spark notebooks are built for — and learning PySpark is worth it if your company runs on them.
For a general ledger of tens of millions of lines on a laptop, the right file format (Parquet) and the right library (DuckDB or pandas) matter far more than Spark.
Download the ledger and race them yourself
The test pack has the script, a runner that times each engine in a fresh process, our results and a README. Get the ledger from DATA DARE DD-MFG-005, point the script at it, and compare your laptop with ours.
No. On one machine, DuckDB answered four finance questions on a 12-million-line ledger in about 1 second and pandas in about 4, while PySpark took about 20. Spark is built for data spread across many machines.
Is DuckDB faster than pandas?
On large Parquet files, usually yes. In our test DuckDB was about 4× faster and used about a fifth of the memory, because it reads only the columns each question needs. pandas has more tutorials and works well up to a few million rows.
What is the difference between Python and pandas?
Python is the programming language. pandas is a library — an add-on — that gives Python tables (DataFrames) with grouping, joins and pivots, much like Excel and Power Query.
Can I use Python in Excel?
Yes. Python in Excel became generally available in September 2024 for Microsoft 365 Business and Enterprise on Windows, and later on the web and Mac. You type =PY( in a cell; the code runs in Microsoft's cloud.
Can I use Python in Power BI?
Yes, as a Python script in Power Query (a data source or a transformation step). Python visuals in Power BI reports retire in May 2027; Microsoft points to Fabric notebooks instead.
What is PySpark?
PySpark is the Python interface to Apache Spark, an engine that spreads data and work across many machines. It runs on Databricks, Microsoft Fabric and other platforms, and also on a single laptop.
Is Databricks free?
Databricks Free Edition, launched in June 2025, gives free Spark and Python notebooks for learning. It is not for commercial use and has compute limits.
When was Python created?
Guido van Rossum started Python in December 1989 and released it publicly in February 1991. Python 3 followed in December 2008, and Python 2 was retired on 1 January 2020.
Get the data and share the story
Test pack (Python scripts + results) — the four-engine test script, a runner that times each engine on your laptop, our results and a README.