Delta Table: Parquet Files Plus a Transaction Log

A Delta table is, at its core, a directory of Parquet data files plus a separate transaction log (the _delta_log folder) that records every change made to the table in order. When you query a Delta table, the engine reads the log to determine which Parquet files are part of the current table state, then reads only those files. This simple addition of a log transforms plain Parquet files into a table with ACID transactions, time travel, and scalable metadata handling.

Formula

The calculator computes two related values from a finish time and a distance:

• Average pace (min/km) = finish time (minutes) ÷ distance (km)
• Average speed (km/h) = distance (km) ÷ (finish time (minutes) ÷ 60)

Variables:

• t – finish time in minutes (e.g., 50 min)
• d – distance in kilometers (e.g., 10 km)

Worked Example

Using the default values: t = 50 minutes, d = 10 km.

1. Pace = 50 ÷ 10 = 5.00 min/km
2. Speed = 10 ÷ (50 ÷ 60) = 10 ÷ 0.8333 = 12.00 km/h

The calculator shows an average pace of 5.00 min/km and an average speed of 12.00 km/h.

Why Parquet Alone Isn’t Enough

Parquet is a columnar storage format that compresses data well and speeds up analytical queries by reading only the columns you need. However, a folder of Parquet files has no built-in way to handle concurrent writes, enforce constraints, or track changes over time. If two processes write to the same folder at once, you can end up with corrupted or mixed data. The Delta transaction log solves this by acting as a single source of truth: every operation (insert, update, delete, merge) is recorded as an atomic entry in the log. The log uses optimistic concurrency – writers try to add a new entry, and if two try at the same time, one waits and retries. This gives you the same durability guarantees you’d expect from a database, but on top of cheap cloud storage.

How the Transaction Log Works

Inside the _delta_log directory, you’ll find JSON files with incrementing numbers (e.g., 00000000000000000000.json, then ...01.json, etc.). Each JSON file is a commit that describes what changed: which files were added, which were removed, and any metadata updates (like schema changes). After a certain number of commits, Delta creates a checkpoint – a Parquet file that contains the full state of the table at that point. This lets the engine skip replaying hundreds of small JSON files; it just reads the latest checkpoint and then applies any commits after it. When you run a query, the engine reads the log to build a list of valid Parquet files, then reads those files in parallel. This design is why Delta can handle tables with billions of files – the log is small and efficient to scan.

What You Gain: ACID, Time Travel, and More

The transaction log is the foundation for several features that plain Parquet lacks:

• ACID transactions – Atomicity, consistency, isolation, and durability for concurrent reads and writes.
• Time travel – You can query a table as it existed at any point in time by specifying a version number or timestamp, because the log retains all historical commits.
• Schema enforcement and evolution – The log stores the schema, so writes that don’t match are rejected (unless you explicitly allow evolution).
• Scalable metadata – Instead of listing thousands of files in a directory, the engine reads a compact log, making operations fast even with huge tables.
• Upserts and deletes – You can run MERGE, UPDATE, and DELETE commands that rewrite only the affected files, not the whole table.

Common Misconceptions

One frequent misconception is that Delta tables are a separate storage format. They’re not – the data is still stored as Parquet files. Delta is a “storage layer� that adds a log and management logic on top. Another misconception is that the transaction log is a database. It’s just a series of files, but because they’re written atomically (using atomic rename operations on cloud storage), they provide a reliable record. Some people also think you need a special engine to read Delta tables – in fact, many tools like Apache Spark, Presto, and Trino can read them via Delta connectors, and you can even read the Parquet files directly if you ignore the log, though you’d lose the transactional guarantees.

How to Choose Between Parquet and Delta

If you’re doing simple batch writes and reads where data is written once and read many times, plain Parquet may be sufficient. But if you need:

• Concurrent writes from multiple jobs or users
• Update or delete operations on existing rows
• Audit history or the ability to roll back to a previous version
• Schema enforcement to catch data quality issues early

then Delta is the better choice. The performance overhead is minimal – the log adds a small amount of I/O, but the benefits in reliability and functionality usually outweigh it. Also, Delta tables are compatible with the Parquet ecosystem, so you can still use tools that understand Parquet, as long as they respect the log.

Real-World Example: A Simple Update

Imagine you have a Delta table of customer orders. You want to update the status of order #123 to “shipped.� Without Delta, you’d have to rewrite the entire Parquet file that contains that row. With Delta, the engine reads the log, identifies the file(s) containing order #123, reads only those files, applies the change, writes new Parquet files, and then adds a commit to the log that marks the old files as removed and the new files as added. This operation is atomic – if it fails halfway, the log isn’t updated, so the table remains unchanged. This is how Delta provides efficient updates while maintaining consistency.

Frequently Asked Questions

Can I convert an existing Parquet table to Delta?

Yes. You can use the CONVERT TO DELTA command in Spark, or write a script that reads the Parquet files and creates a Delta table with a new log. The conversion is usually fast because it doesn’t rewrite the data – it just creates the log entries that point to the existing files.

Does Delta table work with pandas or other non-Spark tools?

Yes, there are libraries like deltalake (Python) and delta-rs that allow reading and writing Delta tables without Spark. These libraries implement the Delta protocol natively, so you can use them in pandas, Polars, or other data tools.

What is the difference between Delta Lake and Delta table?

Delta Lake is the open-source storage layer framework, while a Delta table is the actual table format created by that framework. The terms are often used interchangeably, but Delta Lake refers to the software, and Delta table refers to the data structure.

How long does time travel history last?

By default, Delta retains the last 30 days of history, but you can configure this with delta.logRetentionDuration. After that period, old log files are cleaned up, and you can no longer time travel to versions older than the retention window.

Is Delta table free to use?

Yes, Delta Lake is open source under the Apache 2.0 license. You can use it without paying licensing fees, though you’ll still pay for the underlying cloud storage and compute resources.

Can I use Delta table with SQL only?

Yes, many SQL engines like Databricks SQL, Snowflake (via integration), and Trino support querying Delta tables using standard SQL. You can create, read, update, and delete tables using SQL commands without writing any code.

Related

• Set Time All Day Reminders iPhone: Full Guide
• WhatsApp Web Not Downloading Files? Fix It
• Milk Pasteurization: How It Works & Pathogens Controlled

in t | Finish time | min | 50 | 0
in d | Distance | km | 10 | 0
out pace | Average pace | min/km | t / d | 2
out speed | Average speed | km/h | d / (t / 60) | 2