Skip to main content
Data story

Python 1991 → 2026: From a Hobby Language to PySpark — Do Finance Teams Need Spark for 12 Million Rows?

Same ledger, four engines, identical answers. One took a second; Spark took twenty.

Python went from a hobby project in 1991 to a cell in Excel, a script in Power BI and a notebook in Microsoft Fabric. We traced that history — then put plain Python, pandas, DuckDB and PySpark through the same four finance questions on a general ledger of about 12 million lines from DATA DARE.

By findi360 · Updated 18 November 2026 · Sources · Discussion (0)

The short answer

For a ledger of 12 million lines you do not need Spark. DuckDB answered our four finance questions in about 1 second, pandas in about 4, PySpark in about 20 — 5 of them just starting up — and plain Python in 26, after the 368 MB Parquet file was turned into a 1.8 GB CSV. Spark earns its place when data no longer fits one machine.

The test in three numbers

Four ways to run Python on the same general ledger from DATA DARE DD-MFG-005 — about 12 million lines, 368 MB of Parquet.

  • ~1 s DuckDB All four finance questions, straight from the Parquet file.
  • ~20 s PySpark About 5 of those seconds just to start Spark.
  • 4 of 4 identical answers Plain Python, pandas, DuckDB and PySpark agreed on every question.

Python 1991 → 2026: from a hobby language to finance's second tool

  1. 1991 First public release Guido van Rossum started Python at CWI in the Netherlands in December 1989 and posted the code in February 1991.
  2. 2000 Python 2 List comprehensions, garbage collection and Unicode.
  3. 2006 NumPy Fast arrays — the base every data library still stands on.
  4. 2008 pandas — born in a hedge fund Wes McKinney started pandas at AQR Capital Management to analyse financial data; open source from 2009.
  5. 2008 Python 3 A cleaner language that did not run Python 2 code unchanged.
  6. 2014 Spark becomes an Apache top-level project Born at UC Berkeley's AMPLab in 2009; built to spread data across many machines.
  7. 2018 Python in Power BI Python scripts as a data source and Python visuals in Power BI Desktop.
  8. 2020 Python 2 retires No more fixes from 1 January 2020.
  9. 2024 Python in Excel Generally available in September 2024 for Microsoft 365 Business and Enterprise on Windows.
  10. 2025 Databricks Free Edition Spark notebooks free for learning, launched in June 2025.
  11. 2026 Python 3.15 Released 9 October 2026 — lazy imports, UTF-8 by default and a faster JIT.
  12. May 2027 Python visuals in Power BI retire R and Python visuals stop working in published reports; Python in Power Query is not affected.

Plain Python vs libraries

Plain Python is the language and its standard library — reading a CSV, adding numbers, working with dates. It runs everywhere with nothing to install, but it is slow on big data and cannot read a Parquet file at all.

Libraries are tools other people built on top of it: NumPy for arrays, pandas for tables, DuckDB for SQL on files, PySpark for Spark, matplotlib for charts, openpyxl for Excel files. Think of plain Python as Excel without formulas; the libraries are the SUMIFS, pivot tables and Power Query.

Where Python lives for a finance team in 2026

WhereWhat you can doGood to know
Excel Type =PY( in a cell and use pandas and charts inside the workbook. Runs in Microsoft's cloud; Microsoft 365 Business and Enterprise.
Power Query (Power BI) A Python script as a data source or a transformation step. Not affected by the 2027 retirement.
Power BI visuals Charts drawn with matplotlib inside a report. Retire in May 2027 — Microsoft points to Fabric notebooks.
Microsoft Fabric Python notebooks (one machine) and Spark notebooks. Tables are stored as Delta Parquet in OneLake.
Databricks Free Edition Spark and Python notebooks in the browser. Free for learning, not for commercial work.
Your laptop Positron, VS Code or Jupyter with pandas and DuckDB. Free, private, and fast enough for tens of millions of rows.

The test: four engines, one ledger, the same four questions

We ran each engine in a fresh process on the DD-MFG-005 general ledger and asked the questions a finance team asks every month:

  • Total amount by entity and period.
  • Number of lines by source (sales, purchases, payroll and so on).
  • The 20 accounts with the largest movements.
  • Number of distinct journals per entity.

Time to answer all four questions

  • Plain Python 26 s
  • PySpark 21 s
  • pandas 4.3 s
  • DuckDB 1.0 s
Median of three runs (two for PySpark, one for plain Python). Machine: 2 virtual CPUs, 7 GB of memory — slower than most office laptops.

Full results

EngineTotal timeBefore the first answerSlowest questionPeak memory
DuckDB 2.0 (SQL on the Parquet file) 1.0 s nothing to load — reads only the columns it needs distinct journals, 0.7 s 0.5–0.6 GB
pandas 3.0 (read 6 columns of Parquet) 4.3 s read 0.8 s distinct journals, 2.1 s 3.1 GB
PySpark 4.0 (local, 2 cores) 20–22 s start Spark 4.5–5.9 s distinct journals, 7.9 s 2.2–2.3 GB (with Java)
Plain Python (csv module) 26 s needs the ledger as a 1.8 GB CSV first all four in one pass 0.7 GB

DD-MFG-005 is a Level 5 set with hidden answers, so only timings, sizes and memory are shown — never amounts or counts.

Which one should a finance team pick?

Decided by the test and by what each tool was built for.

DuckDB vs pandas

Pick for analysis: DuckDB for big files

4× faster in our test and used a fifth of the memory, because it reads only the columns a question needs.

Keep the other when: you want the most tutorials, or the data already sits in a pandas table.

pandas vs PySpark

Pick for analysis: pandas on one machine

5× faster here; Spark spent about 5 seconds just starting up.

Keep the other when: the data no longer fits one machine, or your company runs on Databricks or Fabric Spark.

Plain Python vs Libraries

Pick for analysis: Libraries

Plain Python got the right answers, but needed a 1.8 GB CSV instead of the 368 MB Parquet file — and took 26 seconds.

Keep the other when: you cannot install anything, or the file is small.

The same question in DuckDB and pandas

Distinct journals per entity — the slowest question for every engine:

import duckdb, pandas as pd

# DuckDB: SQL straight on the Parquet file
duckdb.sql("""
    SELECT EntityCode, COUNT(DISTINCT JournalID) AS journals
    FROM 'DD-MFG-005_FactGLLines.parquet'
    GROUP BY EntityCode
""").df()

# pandas: load the columns you need, then group
gl = pd.read_parquet('DD-MFG-005_FactGLLines.parquet', columns=['EntityCode', 'JournalID'])
gl.groupby('EntityCode')['JournalID'].nunique()
The test pack has all four engines and a script that times them on your own laptop.

So when does a finance team need Spark?

When the data no longer fits on one machine, when several teams share one lakehouse, or when pipelines run every hour on a schedule. That is what Databricks and Microsoft Fabric's Spark notebooks are built for — and learning PySpark is worth it if your company runs on them.

For a general ledger of tens of millions of lines on a laptop, the right file format (Parquet) and the right library (DuckDB or pandas) matter far more than Spark.

Download the ledger and race them yourself

The test pack has the script, a runner that times each engine in a fresh process, our results and a README. Get the ledger from DATA DARE DD-MFG-005, point the script at it, and compare your laptop with ours.

Questions people ask

Do I need PySpark for 12 million rows?

No. On one machine, DuckDB answered four finance questions on a 12-million-line ledger in about 1 second and pandas in about 4, while PySpark took about 20. Spark is built for data spread across many machines.

Is DuckDB faster than pandas?

On large Parquet files, usually yes. In our test DuckDB was about 4× faster and used about a fifth of the memory, because it reads only the columns each question needs. pandas has more tutorials and works well up to a few million rows.

What is the difference between Python and pandas?

Python is the programming language. pandas is a library — an add-on — that gives Python tables (DataFrames) with grouping, joins and pivots, much like Excel and Power Query.

Can I use Python in Excel?

Yes. Python in Excel became generally available in September 2024 for Microsoft 365 Business and Enterprise on Windows, and later on the web and Mac. You type =PY( in a cell; the code runs in Microsoft's cloud.

Can I use Python in Power BI?

Yes, as a Python script in Power Query (a data source or a transformation step). Python visuals in Power BI reports retire in May 2027; Microsoft points to Fabric notebooks instead.

What is PySpark?

PySpark is the Python interface to Apache Spark, an engine that spreads data and work across many machines. It runs on Databricks, Microsoft Fabric and other platforms, and also on a single laptop.

Is Databricks free?

Databricks Free Edition, launched in June 2025, gives free Spark and Python notebooks for learning. It is not for commercial use and has compute limits.

When was Python created?

Guido van Rossum started Python in December 1989 and released it publicly in February 1991. Python 3 followed in December 2008, and Python 2 was retired on 1 January 2020.

Get the data and share the story

Test pack (Python scripts + results) — the four-engine test script, a runner that times each engine on your laptop, our results and a README.

The story in 10 slides (PDF) — open to everyone, ready to share.

Downloads need a findi360 account — create one in a minute and the file starts straight away. Already a member? Sign in

What does your finance team use for data too big for Excel?

  • Still Excel 0
  • Power BI 0
  • Python / pandas 0
  • DuckDB or SQL 0
  • Spark (Databricks or Fabric) 0

0 members have picked so far. Each member counts once, with their latest pick.

No comments yet — be the first to answer.

Sources

FigureSourceDate
Test data: general ledger, about 12 million lines, 368 MB ParquetDATA DARE DD-MFG-005 (findi360)Oct 2026
Timings and memory (pandas 3.0.5, DuckDB 2.0.0, PySpark 4.0.1, Python 3.13)findi360 test18 Nov 2026
Python started December 1989; code published February 1991; Python 3.0 on 3 December 2008Wikipedia — History of Python2026
Python 2.0 released 16 October 2000; Python 2 end of life 1 January 2020Wikipedia — History of Python; python.org2026
pandas started at AQR Capital Management in 2008, open-sourced 2009pandas documentation; Wikipedia — pandas (software)2026
Spark: UC Berkeley AMPLab 2009; Apache top-level project February 2014Apache Software Foundation press release27 Feb 2014
Python scripts and visuals in Power BI Desktop (preview)InfoWorld — Microsoft adds Python support to Power BI22 Aug 2018
Python in Excel generally available for Microsoft 365 Business and Enterprise (Windows)Microsoft Excel blog — Python in Excel, available now16 Sep 2024
Databricks Free Edition launchedDatabricks press release11 Jun 2025
Python 3.15.0 releasedpython.org — What's new in Python 3.159 Oct 2026
R and Python visuals in Power BI retire in May 2027; Power Query scripts unaffectedMicrosoft Learn — Create Power BI visuals with Python2026

Read before quoting a figure

  • Timings come from one machine (2 virtual CPUs, 7 GB of memory). Your seconds will differ; the order of the engines is what matters.
  • Spark's start-up time is fixed, so it matters less on bigger jobs and on a cluster.
  • Plain Python read a CSV copy of the ledger, because it cannot read Parquet without a library.

Keep going on findi360

Compiled by findi360 from the public sources listed above. Not investment advice. © findi360 — share freely with attribution. Visit findi360 to learn data and finance.