Delta Lake vs Spark DataFrames: Databricks Exam Guide

Updated 6 November 2026 · Standard formula, shown below

Delta Lake and Spark DataFrames are two distinct but complementary technologies you must understand for Databricks certification exams. Delta Lake is a storage layer that adds ACID transactions, time travel, and schema enforcement to data lakes, while Spark DataFrames are the distributed in-memory data structure you use to process data. Exam candidates need to know when each is used, how they interact, and which questions target one versus the other.

Background: Why the Confusion Exists

Many Databricks exam questions blur the line between these two concepts because they are used together in nearly every pipeline. A DataFrame is created by reading from a Delta table, transformed, then written back to Delta. The confusion arises because Delta Lake is often described as a “table format” or “storage layer” while DataFrames are described as “APIs” or “distributed collections.” They are not competitors; they operate at different layers of the stack.

Delta Lake sits on top of parquet files and manages metadata, transactions, and versioning. Spark DataFrames are the programming abstraction you use to manipulate data in memory. In a typical Databricks workflow, you load a Delta table into a DataFrame, perform transformations, and write the result back to a Delta table. The exam tests whether you understand which layer provides which guarantee.

Core Differences You Must Know

Storage vs. Processing: Delta Lake is a storage format with transactional guarantees; DataFrames are an in-memory data structure for distributed processing.
Persistence: Delta tables persist to disk and survive cluster restarts; DataFrames are ephemeral and exist only during a Spark session.
Schema: Delta Lake enforces a schema on write; DataFrames have a schema that is inferred or defined at creation time, but it is not enforced when writing to non-Delta sources.
Time Travel: Delta Lake supports versioning and time travel; DataFrames do not retain historical versions.
ACID Transactions: Delta Lake provides atomicity, consistency, isolation, and durability; DataFrames have no transaction semantics.

Exam questions often ask which feature belongs to which layer. For example, “Which technology provides ACID guarantees?” The answer is Delta Lake, not DataFrames.

How They Work Together in Practice

In Databricks, you rarely use one without the other. Reading a Delta table returns a DataFrame. Writing a DataFrame to a Delta table triggers Delta Lake’s transactional write path. The two are linked by the spark.read.format("delta") and df.write.format("delta") APIs.

Consider a simple pipeline:

Load a Delta table: df = spark.read.format("delta").table("sales")
Transform: filtered = df.filter(df.amount > 100)
Write: filtered.write.format("delta").mode("overwrite").save("/mnt/delta/sales_high")

The DataFrame filtered is a lazy evaluation plan—no data is read until an action like count() or show() is called. Delta Lake handles the actual I/O, transaction log, and versioning when the write occurs.

Exam Scenarios: What Questions Actually Ask

Databricks certification exams (like the Associate or Professional Data Engineer) include scenario-based questions. You might see:

“A team needs to ensure that concurrent writes to a table do not corrupt data. Which feature should they use?” — Delta Lake ACID transactions.
“Which method allows you to view a table as it existed yesterday?” — Delta Lake time travel via TIMESTAMP AS OF or VERSION AS OF.
“What is the result of calling df.collect()?” — Returns an array of Row objects to the driver, not a Delta table.
“Which operation triggers a shuffle?” — groupBy() or join(), not a Delta read.

You must also know the difference between transformations (lazy) and actions (eager). DataFrames are lazy; Delta Lake writes are actions that trigger computation.

Common Misconceptions and Mistakes

A frequent mistake is assuming DataFrames are stored on disk. They are not—they are logical plans that execute on executors. Another is thinking Delta Lake is a database engine; it is a storage layer that works with Spark’s engine, not a replacement for it.

Another misconception: “Delta Lake is a file format.” Technically, Delta Lake uses Parquet for data files, but the Delta table itself is a transaction log plus Parquet files. The exam may test this nuance.

When writing to a Delta table, the default mode is error, which fails if the table already exists. Candidates often confuse overwrite and append. Overwrite replaces the entire table, while append adds new rows. Delta Lake also supports merge for upserts, which is not a DataFrame operation but a Delta-specific one.

How to Choose the Right Concept for a Question

When answering exam questions, ask yourself: “Is this about how data is stored and managed, or how data is processed?” If the question mentions transactions, versioning, schema enforcement, or time travel, the answer is Delta Lake. If it mentions transformations, actions, lazy evaluation, or partitioning in memory, the answer is DataFrames.

For example, a question about “ensuring no partial writes” is about Delta Lake. A question about “filtering rows efficiently” is about DataFrames. Some questions combine both: “Which approach minimizes data shuffling when joining two large datasets?” This is about DataFrame optimization (broadcast join, bucketing), not Delta Lake.

Be careful with questions about performance. Delta Lake can improve read performance with data skipping (via statistics in the transaction log), but DataFrame operations like filter and select are where you tune execution. Exam answers often expect you to identify that Delta Lake’s optimizations (like Z-order) are table-level, while DataFrame optimizations are query-level.

Alternatives and Related Technologies

Databricks exams also cover other storage and processing options. Apache Parquet is the raw file format that Delta Lake builds upon. Hive tables are an older metastore-based approach without ACID. Spark SQL provides a SQL interface to DataFrames. You should know that Delta Lake can be used with Spark SQL via CREATE TABLE USING DELTA, but the underlying engine is still Spark DataFrames.

Another related concept is the Delta Engine (now part of Photon), which accelerates queries on Delta tables. This is not a separate API but a vectorized execution engine. Exam candidates should know that Photon is an optimization, not a replacement for DataFrames.

Frequently Asked Questions

Is Delta Lake a replacement for Spark DataFrames?

No. They serve different purposes. Delta Lake is a storage layer that provides reliability features; DataFrames are the programming abstraction for processing. You use DataFrames to read and write Delta tables, so they are complementary, not competing.

Can I use Delta Lake without DataFrames?

Yes, you can use Spark SQL to create and query Delta tables without explicitly writing DataFrame code. However, under the hood, Spark SQL executes as DataFrames. So while you may not see the DataFrame API, the engine still uses the same execution model.

What is the difference between a DataFrame and a Delta table in terms of schema?

A DataFrame has a schema that is either inferred from data or explicitly defined. Delta Lake enforces a schema on write, meaning it will reject data that does not match the table’s schema. You can also evolve the schema with ALTER TABLE, but the DataFrame schema is fixed for the duration of that DataFrame.

Does time travel work on DataFrames?

No. Time travel is a Delta Lake feature that lets you query previous versions of a table. DataFrames are ephemeral and do not have version history. To use time travel, you must read from a Delta table with TIMESTAMP AS OF or VERSION AS OF.

Which one should I use for a streaming pipeline?

For streaming, you use Structured Streaming, which produces DataFrames (or Datasets) in micro-batches. You can write those streaming DataFrames to Delta tables, which support streaming reads and writes. So both are used: DataFrames for the streaming computation, Delta Lake for the storage sink.

Are there exam questions that ask for code snippets?

Yes. You may be asked to identify the correct API to read a Delta table or to perform a merge. Knowing the syntax like spark.read.format("delta") and DeltaTable.forPath(spark, path).merge() is essential. Practice writing these snippets to avoid confusion.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.