Databricks Data Engineer Practice Test

Databricks Certified Data Engineer Associate

Databricks Data Engineer overview

  • 200 practice questions
  • 32 flashcards
  • 7 study guide topics
  • 40 key terms

What you'll study

  • Databricks Intelligence Platform6%
  • Data Ingestion and Loading21%
  • Data Transformation and Modeling22%
  • Working with Lakeflow Jobs16%
  • Implementing CI/CD10%
  • Troubleshooting, Monitoring, and Optimization10%
  • Governance and Security15%

The Databricks Certified Data Engineer Associate is a proctored, entry-level certification that assesses a person's ability to use the Databricks Data + AI Platform to execute foundational data engineering tasks. It is aimed at practitioners with hands-on experience building ETL pipelines - data engineers, analytics engineers, and developers moving onto the lakehouse - and serves as a structured way to demonstrate baseline platform skills. The exam consists of 45 scored questions with a 90-minute limit, is multiple choice with no test aides allowed, and is delivered online or at a test center in English, Japanese, Brazilian Portuguese, and Korean. Coverage is split across seven areas: Databricks Intelligence Platform (6%), Data Ingestion and Loading (21%), Data Transformation and Modeling (22%), Working with Lakeflow Jobs (16%), Implementing CI/CD (10%), Troubleshooting, Monitoring, and Optimization (10%), and Governance and Security (15%). Data manipulation code is presented in SQL where possible and otherwise in Python. There are no prerequisites, though Databricks strongly recommends related training and hands-on experience with the tasks described in the exam guide. To prepare, review the exam guide, complete the recommended training, use the AI Prep Guide, and run the system check before an online proctored appointment. The credential is valid for two years, after which recertification requires passing the current version of the exam.

Common mistakes

Selecting an all-purpose cluster (or a single-node cluster) as the compute for a scheduled Lakeflow Job run, or assuming a serverless SQL warehouse can execute Python/Spark code in a task.

Candidates learn on interactive all-purpose clusters and carry that habit into production job definitions; the exam scenarios describe scheduled pipelines where the compute choice affects isolation, autoscaling, and cost, and where SQL-warehouse-only tasks cannot run notebook/Spark code.

What to do instead

In job definitions use job compute (or serverless job compute) so each run gets isolated, right-sized resources; use a serverless SQL warehouse only for pure SQL tasks, and move Python/Spark logic to a notebook or Python wheel task on job compute.

Implementing incremental ingestion by re-reading the whole landing directory each run with a DataFrame read or CREATE TABLE AS SELECT, then overwriting the target.

It is the fastest pattern to write locally, and candidates do not distinguish 'the query returned new data' from 'the platform tracked which files were already consumed' - the ingestion state is implicit and lost between runs.

What to do instead

Use Auto Loader (cloudFiles) with a required checkpoint location, or COPY INTO for SQL/SQL-warehouse ingestion; both persist file-level ingestion state so re-runs are idempotent and only new files are processed.

Assuming that because Auto Loader inferred a new column in the source JSON, the downstream Delta table will automatically gain that column, and ignoring the rescued-data column when records arrive with unexpected shapes.

Schema inference on the stream and schema of the persisted sink table are two separate things, and malformed records land in a rescue column rather than failing loudly, so candidates believe data was loaded when it was quarantined.

What to do instead

Explicitly enable schema evolution on the write (e.g. mergeSchema) or use a controlled schema-evolution step, add cloudFiles.schemaLocation so the inferred schema is stable across restarts, and monitor/query _rescued_data to catch records that were not parsed.

Using MERGE INTO to upsert from a source that still contains duplicate keys, expecting the last row to win.

MERGE reads as 'SQL upsert' so candidates assume duplicate handling is implicit; in Delta, when more than one source row matches the same target row for an UPDATE or DELETE action the statement fails (DELTA_MULTIPLE_SOURCE_ROW_MATCHING_TARGET_ROW_IN_MERGE).

What to do instead

Deduplicate the source before the merge - a window function with ROW_NUMBER() over the key ordered by an update timestamp, keeping rn = 1 - or pre-aggregate to one row per key; never rely on statement ordering.

Defaulting to PARTITION BY on a low-cardinality or rarely filtered column (or partitioning a small table at all), instead of choosing clustering keys based on query predicates.

Partitioning is the classic Hadoop-era habit and looks like 'good modelling'; over-partitioning creates many tiny files, and partition columns are also immutable in place, which blocks later design changes.

What to do instead

Use liquid clustering (CLUSTER BY) on the high-cardinality columns that appear in filter/join predicates, leave small tables unpartitioned, and reserve classic partitioning for very large tables filtered on the same low-cardinality column; note liquid clustering and PARTITION BY are mutually exclusive.

Assuming 'Repair run' re-executes the entire job from the first task, or conversely that a retry on a single task will automatically re-run the downstream tasks that consumed its output.

Candidates conflate job-level reruns with task-level retries and do not reason about the task dependency graph declared via depends_on, so they mis-predict which tasks run and which are skipped.

What to do instead

Model the DAG explicitly with depends_on; use Repair run to re-execute only the failed tasks and their downstream dependents while leaving succeeded tasks untouched, and set task-level retries/timeouts for transient failures rather than rerunning the whole job.

Hardcoding workspace URLs, catalog/schema names, and cluster IDs directly into notebooks and job JSON, so the same artifact is deployed to every environment.

It works in a single dev workspace and candidates never introduce a parameterization layer, so promotion to staging/prod either fails or silently writes into the wrong catalog.

What to do instead

Package the project as a Databricks Asset Bundle with per-environment targets and ${var.*} variables (resolved from target-specific values or environment variables), validate with databricks bundle validate -t <target> and deploy with databricks bundle deploy -t <target> so no environment-specific literal lives in source.

Putting transformation logic directly inside notebook cells and then trying to satisfy the CI test gate by executing notebooks against a live cluster (or skipping tests because 'notebooks are untestable').

Notebook-only development is the default on the platform, and spinning up compute inside CI is slow and credential-heavy, so teams either accept flaky integration tests or no coverage at all.

What to do instead

Move transformation logic into importable Python modules/wheels that take DataFrames as input, unit-test them with a local Spark session and tiny fixtures in CI, and keep notebooks as thin orchestration wrappers that pass bundle-resolved parameters.

Diagnosing a failed/long-running Spark job from the top-level 'Job aborted due to stage failure' message or by enlarging the cluster, without reading the innermost Caused by and the stage/task metrics in the Spark UI.

The platform surfaces the outermost exception prominently, and adding compute is the intuitive first response; the actual root cause (data skew, OOM in a specific stage, a permissions error on a path) is buried in the cause chain or visible only per-stage.

What to do instead

Walk the full exception chain to the root cause, open the Spark UI / query profile for the failing stage, and check task duration distribution for skew and spill/memory metrics; then fix the cause (AQE skew join, repartition on a better key, broadcast join, file compaction) instead of scaling out.

Granting SELECT on a table to a group and assuming access works, and validating row filters/column masks by querying as the table owner.

Unity Catalog requires a privilege chain (USE CATALOG, USE SCHEMA, then the object privilege), and owners bypass row filters and column masks, so a test run as the owner always shows unfiltered data and the privilege gap only appears in production for other users.

What to do instead

Grant USE CATALOG on the catalog and USE SCHEMA on the schema in addition to SELECT on the table; verify filtering with a non-owner test identity (or the actual group member) and confirm which roles are exempt from the policy before declaring it correct.

Exam requirements can change. Confirm current details with the exam provider.

Databricks Data Engineer practice

200 questions. Each one lists the choices, the correct answer, and an explanation. With JavaScript, this list opens one question at a time. The checker at the top of this page saves progress in this browser when it is available.

  1. Question 1Databricks Intelligence Platform

    In Delta Lake, which component records every committed change to a table and serves as the basis for its ACID guarantees?

    1. The _delta_log directory Correct answer.
    2. The Spark driver's in-memory catalog of tables
    3. The Hive metastore entry holding the table definition
    4. The collection of Parquet data files in the table root

    Explanation

    Each commit to a Delta table is written atomically as a JSON entry in the _delta_log directory, with periodic Parquet checkpoints summarizing older entries. Readers reconstruct the current snapshot by replaying that log, which is what enables ACID semantics and time travel. The data files themselves carry no transactional metadata.

  2. Question 2Databricks Intelligence Platform

    Which fully qualified name correctly references a table through the Unity Catalog three-level namespace?

    1. database.catalog.schema.table
    2. catalog.schema.table Correct answer.
    3. schema.table
    4. workspace.catalog.table

    Explanation

    Unity Catalog organizes securables as catalog, then schema, then table or view, so a table is addressed as catalog.schema.table. Two-part names work only when a default catalog and schema are in effect, and the three-part form is the canonical qualified reference.

  3. Question 3Databricks Intelligence Platform

    Which magic command executes another notebook inline within the current notebook and shares its variables and functions with the calling notebook?

    1. %sh
    2. %fs
    3. %pip
    4. %run Correct answer.

    Explanation

    %run includes the referenced notebook in the current execution context, so its defined variables and functions become available to the caller. This differs from %pip and %sh, which act on the environment or shell, and from %fs, which issues file system commands.

  4. Question 4Databricks Intelligence Platform

    What is Photon in the Databricks Lakehouse Platform?

    1. A notebook scheduling service for recurring dashboards
    2. A change data capture connector for streaming sources
    3. A vectorized C++ query engine Correct answer.
    4. A governance layer that enforces column-level access control

    Explanation

    Photon is Databricks' native execution engine, written in C++ with vectorized processing, that accelerates SQL and DataFrame workloads on Delta tables. It is enabled per cluster or warehouse and is compatible with Spark APIs, so existing code does not need rewriting.

  5. Question 5Databricks Intelligence Platform

    A data engineer creates an all-purpose cluster with autoscaling enabled, a minimum of 2 workers, and a maximum of 8 workers. How does the cluster respond as its workload changes?

    1. It automatically moves the cluster between serverless and classic compute types
    2. It adds or removes worker nodes within the configured bounds Correct answer.
    3. It scales the driver node's core count between the configured minimum and maximum values
    4. It keeps node count fixed but resizes each worker's memory as needed

    Explanation

    Autoscaling adjusts the number of worker nodes between the minimum and maximum you set, allocating more workers when tasks are queued and releasing them when demand drops. The driver is not scaled by autoscaling, and the compute type of an existing cluster does not change.

  6. Question 6Databricks Intelligence Platform

    In a Delta Live Tables pipeline, which decorator enforces a data quality expectation by discarding records that violate it while allowing the pipeline update to continue?

    1. @dlt.expect_all_or_fail
    2. @dlt.table
    3. @dlt.expect_or_fail
    4. @dlt.expect_or_drop Correct answer.

    Explanation

    The expect_or_drop decorator records the violation in metrics and removes the offending rows from the result set, so the pipeline update keeps running. Failure variants such as expect_or_fail and expect_all_or_fail stop the update instead, while @dlt.table only defines the dataset.

  7. Question 7Databricks Intelligence Platform

    A Delta table has ten committed versions and the default retention settings. Which statement returns the table contents as they existed at version 6 without altering the table?

    1. SELECT * FROM t TIMESTAMP AS OF 6
    2. RESTORE TABLE t TO VERSION AS OF 6
    3. SELECT * FROM t AT SNAPSHOT 6
    4. SELECT * FROM t VERSION AS OF 6 Correct answer.

    Explanation

    Time travel uses the clause VERSION AS OF or TIMESTAMP AS OF after the table name, so SELECT * FROM t VERSION AS OF 6 reads historical data. RESTORE changes the current state of the table rather than only querying a past version, and AT SNAPSHOT is not valid Delta syntax.

  8. Question 8Databricks Intelligence Platform

    A managed table is created in a Unity Catalog schema. Where do its data files reside?

    1. In the default DBFS root of the workspace that ran the query
    2. In the managed storage location of the containing schema or catalog Correct answer.
    3. In the /tmp directory of the cluster's driver node
    4. In the external location attached to the notebook's workspace

    Explanation

    In Unity Catalog, managed tables are stored in a managed storage location defined at the schema or catalog level, with the metastore as the authority for lifecycle. Dropping such a table also removes its data files. DBFS root and driver-local paths are not used for managed tables under Unity Catalog.

  9. Question 9Databricks Intelligence Platform

    What is the primary benefit of running OPTIMIZE with Z-ORDER on a Delta table?

    1. It converts the table into a streaming source with exactly-once semantics
    2. It only merges small files into larger ones without changing row order
    3. It permanently deletes old file versions to reduce storage cost
    4. It co-locates related data in the same files to improve data skipping Correct answer.

    Explanation

    Z-ORDER is a multi-dimensional clustering technique applied during OPTIMIZE that colocates rows with similar values across the z-order columns into the same files. Queries filtering on those columns can then skip far more files. Compaction alone is what plain OPTIMIZE without Z-ORDER performs.

  10. Question 10Databricks Intelligence Platform

    A multi-task job completes except for its final task, which fails. Which action reruns only that failed task and its downstream dependencies without rerunning the tasks that succeeded?

    1. Repairing the failed run Correct answer.
    2. Deleting and recreating the job with identical settings
    3. Resetting the job's schedule
    4. Cloning the job into a different workspace

    Explanation

    A repair run re-executes the failed task and any tasks that depend on it while reusing the completed results of upstream tasks, which saves time and compute. It operates on an existing run and can be started with modified parameters. Recreating or cloning the job starts everything from the beginning.

  11. Question 11Databricks Intelligence Platform

    By default, for which columns of a Delta table does Databricks collect the statistics used for data skipping?

    1. The first 32 columns of the table schema Correct answer.
    2. Only numeric and date-typed columns
    3. All columns regardless of their position in the schema
    4. Only columns declared as partition keys

    Explanation

    Delta collects per-file statistics such as min, max, and null counts for the first 32 columns of the schema by default, and these statistics drive data skipping. Columns beyond that limit can be included by reordering the table or by configuring additional statistics columns explicitly. Partition columns are handled separately through directory pruning.

  12. Question 12Databricks Intelligence Platform

    An engineer needs to create an external table at a cloud storage path in Unity Catalog. Which objects must already exist and be granted before the table can be created at that path?

    1. Only a catalog and schema, since the path is validated automatically by the workspace admin
    2. A cluster configured with an instance profile that can read the bucket
    3. A storage credential and an external location covering the path Correct answer.
    4. A DBFS mount point plus a table ACL grant on the target directory

    Explanation

    Unity Catalog governs access to cloud storage through storage credentials that hold the cloud identity, and external locations that bind a credential to a storage path. A user needs a privilege on the external location plus CREATE TABLE on the schema. Mount points and cluster instance profiles are the legacy workspace-level approach and are not the governing objects in Unity Catalog.

  13. Question 13Data Ingestion and Loading

    Which value of the format/read source activates Auto Loader in a Structured Streaming read?

    1. cloudFiles Correct answer.
    2. autoLoad
    3. streamFiles
    4. fileStream

    Explanation

    Auto Loader is enabled by setting the source format to cloudFiles and supplying options prefixed with cloudFiles. (e.g. cloudFiles.schemaLocation). It is a streaming source designed for incremental file arrival in cloud storage.

  14. Question 14Data Ingestion and Loading

    Which statement describes the idempotency behavior of COPY INTO when run repeatedly against the same source directory?

    1. It tracks previously loaded files and skips them on later runs. Correct answer.
    2. It reloads every file found in the path on each execution of the command.
    3. It requires that source files be deleted manually before each subsequent run.
    4. It deletes source files from cloud storage once they have been loaded.

    Explanation

    COPY INTO keeps a record of the files it has already ingested for a target table, so re-running the same command against the same path loads only new files. This makes the command safe to schedule without duplicating data.

  15. Question 15Data Ingestion and Loading

    Which two file discovery modes does Auto Loader support?

    1. polling and push notifications
    2. schema inference and schema evolution
    3. directory listing and file notification Correct answer.
    4. batch and streaming reads

    Explanation

    Auto Loader can discover new files either by listing the input directory or by subscribing to file arrival events from the cloud storage notification service. Directory listing is the default and requires no extra cloud setup, while file notification scales better for very large volumes.

  16. Question 16Data Ingestion and Loading

    When reading a CSV file with the DataFrame reader, which option causes the first line to be used as column names?

    1. firstRowAsHeader
    2. columnNames
    3. header Correct answer.
    4. inferHeader

    Explanation

    The CSV data source uses the option header set to true to treat the first row as the column names. Without it, columns are named _c0, _c1, and so on.

  17. Question 17Data Ingestion and Loading

    Why must every Structured Streaming query use its own unique checkpoint location?

    1. It defines the output schema used when writing to the sink table.
    2. It stores the query results in a durable location for reuse.
    3. It caches the source files on the cluster for faster reads.
    4. It records stream progress so processing resumes where it stopped. Correct answer.

    Explanation

    The checkpoint directory holds the progress information (which files or offsets were consumed) and streaming state for a query. Sharing a checkpoint between queries corrupts that bookkeeping, so each query needs its own path.

  18. Question 18Data Ingestion and Loading

    Which of these file formats stores data in a columnar layout and is commonly used as a landing format for ingestion into Delta Lake?

    1. JSON
    2. CSV
    3. Text
    4. Parquet Correct answer.

    Explanation

    Parquet is a columnar format that stores data with an embedded schema, which makes reads selective and compression efficient. CSV, JSON, and plain text are row-oriented text formats without a typed schema.

  19. Question 19Data Ingestion and Loading

    In a MERGE INTO statement, which clause specifies what to do when a source row matches an existing target row?

    1. USING <source>
    2. ON <merge condition>
    3. WHEN MATCHED THEN UPDATE SET Correct answer.
    4. WHEN NOT MATCHED BY TARGET THEN INSERT *

    Explanation

    The WHEN MATCHED clause defines the action taken for source rows that satisfy the merge condition and already exist in the target. Actions for new rows are defined by WHEN NOT MATCHED, while ON supplies the join condition.

  20. Question 20Data Ingestion and Loading

    In the medallion architecture, which layer holds raw ingested data with little or no transformation applied?

    1. Silver
    2. Platinum
    3. Gold
    4. Bronze Correct answer.

    Explanation

    The bronze layer stores data exactly as it arrived from the source, preserving the raw records for replay and auditing. Cleansing and conforming happen in silver, and business-level aggregates are produced in gold.

  21. Question 21Data Ingestion and Loading

    Which value of cloudFiles.schemaEvolutionMode is the default for Auto Loader?

    1. failOnNewColumns
    2. rescue
    3. addNewColumns Correct answer.
    4. none

    Explanation

    By default Auto Loader uses addNewColumns, which updates the tracked schema and restarts the stream when a new column appears. The other modes either fail the stream, park unknown data, or ignore changes entirely.

  22. Question 22Data Ingestion and Loading

    While ingesting JSON files, Auto Loader encounters fields that do not match the tracked schema. Where is that data placed by default?

    1. The stream fails on the first mismatched record.
    2. In a column named _rescued_data. Correct answer.
    3. In a separate quarantine table named bad_records.
    4. The record is dropped and logged as an error.

    Explanation

    When a record cannot be parsed against the schema, Auto Loader preserves it by putting the mismatched fields into a column called _rescued_data instead of failing or dropping the record. This lets downstream logic inspect and remediate the bad data later.

  23. Question 23Data Ingestion and Loading

    A COPY INTO statement targets a Delta table name that does not exist yet. What is the result?

    1. The rows are written to a managed staging table instead.
    2. The table is created automatically using an inferred schema.
    3. A temporary view is created to hold the incoming rows.
    4. It fails; the target Delta table must exist first. Correct answer.

    Explanation

    COPY INTO loads data into an existing Delta table and does not create the target. The table must be created beforehand with the desired schema, after which COPY INTO can append to it.

  24. Question 24Data Ingestion and Loading

    A pipeline must continuously ingest millions of small JSON files that land in cloud storage, and the source occasionally adds new columns. Which ingestion approach best fits?

    1. A nightly full read of the directory with spark.read
    2. COPY INTO with a file PATTERN clause
    3. A JDBC batch read of the landing directory
    4. Auto Loader with the cloudFiles format Correct answer.

    Explanation

    Auto Loader is built for continuously arriving files at high volume and tracks schema changes incrementally, adding new columns without reprocessing everything. COPY INTO can work but becomes inefficient at very large file counts because it must evaluate the directory contents on each run.

  25. Question 25Data Ingestion and Loading

    Which permission is required to use Auto Loader file notification mode on AWS?

    1. A Kafka cluster running in the same region and account
    2. IAM permissions to create an SNS topic and an SQS queue Correct answer.
    3. A Unity Catalog volume mounted at the checkpoint path
    4. The landing files rewritten into Delta format first

    Explanation

    File notification mode subscribes to storage events through a cloud notification service, so the identity running the pipeline needs permissions to create and manage the topic and queue used for those events. Without them, Auto Loader must fall back to directory listing.

  26. Question 26Data Ingestion and Loading

    A streaming query is restarted with the same checkpoint location but pointed at a different source directory. What happens?

    1. It silently ignores the new source and keeps reading the old one.
    2. It reprocesses every file in the new source directory from the very beginning.
    3. It reads both sources and unions the results together.
    4. The query fails because the checkpoint is bound to the original source. Correct answer.

    Explanation

    The checkpoint stores the identity and progress of the source it was created for, so changing the source while reusing the checkpoint causes the query to fail. A new checkpoint location is required whenever the source definition changes.

  27. Question 27Data Ingestion and Loading

    In a Delta Live Tables Python pipeline, what determines whether a defined dataset is a streaming table rather than a materialized view?

    1. Whether the query reads a stream via dlt.read_stream Correct answer.
    2. The catalog and schema in which the table is created
    3. The prefix used in the table name, such as stream_ or mv_
    4. The storage location assigned to the table in pipeline settings

    Explanation

    DLT infers the dataset type from the defining query: a query that reads a streaming source produces a streaming table, while a query that reads a static snapshot produces a materialized view. Naming, location, and catalog choices do not affect that classification.

  28. Question 28Data Ingestion and Loading

    Which reader options allow a JDBC read to be split into multiple parallel partitions?

    1. pushDownPredicate and pushDownAggregate set to true
    2. fetchsize, batchsize, and a custom data source class
    3. query and dbtable, with an optional isolation level
    4. partitionColumn, numPartitions, and both bounds Correct answer.

    Explanation

    Specifying a numeric partition column together with a lower bound, upper bound, and partition count lets Spark issue parallel range queries against the remote database. Without them, the read is performed through a single connection.

  29. Question 29Data Ingestion and Loading

    Which approach ingests only new or changed rows from a relational source on each scheduled run?

    1. Read the entire remote table on every run and then drop duplicate rows
    2. Enable change data capture on the target Delta table
    3. Run COPY INTO directly against the remote database
    4. Filter the JDBC query on an increasing column like updated_at Correct answer.

    Explanation

    The common incremental pattern is to parameterize the JDBC read with a predicate on a monotonically increasing column and pass the highest value seen in the previous run. This transfers only newly modified rows instead of the whole table.

  30. Question 30Data Ingestion and Loading

    Which Structured Streaming feature allows a MERGE or a full overwrite to be applied inside a streaming query?

    1. foreachBatch Correct answer.
    2. option('mergeSchema', 'true')
    3. outputMode('complete')
    4. trigger(once=True)

    Explanation

    foreachBatch hands each micro-batch to a user function that receives a static DataFrame, so arbitrary table operations such as MERGE, delete, or overwrite can be performed per batch. Native streaming sinks only support append, update, or complete semantics.

  31. Question 31Data Ingestion and Loading

    Which approach removes duplicate events from a streaming DataFrame keyed on an event id?

    1. dropDuplicates on the event id key, plus a watermark Correct answer.
    2. Repartitioning the stream to a single partition
    3. A broadcast join against a static deduplication table
    4. Writing the stream with outputMode complete

    Explanation

    dropDuplicates on the key column keeps one row per key within the streaming state, and a watermark bounds how long that state is retained so it does not grow without limit. Without a watermark the state grows indefinitely for an unbounded stream.

  32. Question 32Data Ingestion and Loading

    Which write option controls the physical directory layout of a table by the values of a column?

    1. repartition
    2. clusterBy
    3. bucketBy
    4. partitionBy Correct answer.

    Explanation

    partitionBy writes one subdirectory per distinct value of the chosen column, which enables partition pruning on later reads. bucketBy hashes rows into files and repartition reshuffles in memory without affecting the on-disk directory structure.

  33. Question 33Data Ingestion and Loading

    A CSV column containing mixed values is inferred as STRING, but it must be ingested as INTEGER. Which Auto Loader setting forces the type?

    1. Set cloudFiles.schemaHints to col INT Correct answer.
    2. Set cloudFiles.schemaEvolutionMode to 'rescue'
    3. Set mergeSchema to 'true' on the read
    4. Set cloudFiles.inferColumnTypes to 'false'

    Explanation

    Schema hints let you declare the data type of specific columns while leaving the remaining columns to inference. This is the supported way to override an inferred type for a named column in a cloudFiles read.

  34. Question 34Data Ingestion and Loading

    An Auto Loader job using trigger(availableNow=True) fails partway through a large file backlog and is restarted with the same checkpoint and schema location. Why are rows not duplicated?

    1. Writing with mode overwrite deletes any duplicate rows
    2. The durable checkpoint records which files were already processed Correct answer.
    3. Delta's transaction log automatically removes duplicated rows
    4. Auto Loader deduplicates rows by a primary key in the target table

    Explanation

    Auto Loader persists the set of processed files in the checkpoint and schema log on durable storage, so a restarted query resumes with the unprocessed files only. Exactly-once behavior depends on keeping those metadata locations stable and on a transactional sink such as Delta Lake.

  35. Question 35Data Ingestion and Loading

    Which option causes COPY INTO to reload files that were previously ingested into the target table?

    1. COPY_OPTIONS ('force' = 'true') Correct answer.
    2. Delete the checkpoint directory before each run
    3. SET TBLPROPERTIES ('force' = 'true')
    4. Add FILEFORMAT = 'force' to the statement

    Explanation

    COPY INTO normally skips files already loaded, but the force copy option makes it re-ingest all files matched by the path, including previously loaded ones. This is useful after fixing a data problem in the source files, at the cost of duplicating rows.

  36. Question 36Data Ingestion and Loading

    When reading from a Kafka topic with the default configuration, what does the value column of the resulting DataFrame contain?

    1. A base64-encoded text value
    2. A parsed JSON string
    3. Raw bytes that must be deserialized or cast Correct answer.
    4. An Avro record with a schema

    Explanation

    Kafka messages arrive as byte arrays, so the value column holds binary data until an explicit cast or a custom deserializer converts it. key, topic, partition, and offset columns are provided alongside value for bookkeeping.

  37. Question 37Data Ingestion and Loading

    A user needs to create an external table over a cloud storage path in Unity Catalog. What must exist first?

    1. A Hive metastore deployed in the same cloud region
    2. A personal access token embedded in the table path
    3. A global temporary view referencing the storage path
    4. An external location backed by a storage credential the user can use Correct answer.

    Explanation

    Unity Catalog governs cloud storage access through storage credentials and external locations, and the principal must be granted privileges on them. Only then can external tables or volumes be created over that path.

  38. Question 38Data Ingestion and Loading

    Which format string is passed to Spark's readStream to enable Auto Loader?

    1. cloud_files
    2. delta
    3. cloudFiles Correct answer.
    4. autoLoader

    Explanation

    Auto Loader is activated by using cloudFiles as the data source format on readStream, and all of its behavior is then tuned through options prefixed with cloudFiles. The format name is case-sensitive and spelled as a single camel-case word.

  39. Question 39Data Ingestion and Loading

    With the default configuration of the Kafka source, what does the value column of the resulting DataFrame contain?

    1. The raw message payload as a binary column. Correct answer.
    2. The message payload already parsed into typed fields from the schema registry.
    3. The message payload decoded as a UTF-8 string.
    4. The key and value combined into a single struct column.

    Explanation

    The Kafka source exposes key and value as separate binary columns and performs no deserialization itself. Decoding requires an explicit cast, from_json, or a custom deserializer applied after the read.

  40. Question 40Data Ingestion and Loading

    What does the FROM clause of a COPY INTO statement reference?

    1. A temporary view in the current Spark session.
    2. Another Delta table in the same metastore.
    3. A cloud storage path containing the source files. Correct answer.
    4. A Kafka topic that streams records.

    Explanation

    COPY INTO loads files from a storage location into a Delta table, so the FROM clause takes a path (or an external location or volume path) rather than a relation. Copying between tables is done with INSERT ... SELECT, MERGE, or CREATE TABLE AS.

  41. Question 41Data Ingestion and Loading

    A developer writes a streaming DataFrame to a Delta table using writeStream. Which option must be supplied for the query to start?

    1. checkpointLocation Correct answer.
    2. queryName
    3. maxFilesPerTrigger
    4. trigger

    Explanation

    Every streaming write must record progress somewhere, so checkpointLocation is mandatory; the query cannot start without a durable offset and commit log. The other options influence scheduling, rate limiting, or identification but are optional.

  42. Question 42Data Ingestion and Loading

    Auto Loader is implemented on top of which Spark capability?

    1. The Spark SQL catalog and metastore.
    2. Photon vectorized execution.
    3. Structured Streaming sources. Correct answer.
    4. Delta Sharing.

    Explanation

    Auto Loader registers a Structured Streaming source that incrementally discovers and reads new files, so it inherits checkpointing, exactly-once file tracking, and trigger semantics from Structured Streaming.

  43. Question 43Data Ingestion and Loading

    A COPY INTO command is executed twice against the same source directory with no new files arriving between runs. Why does the target table not receive duplicate rows?

    1. COPY INTO compares every column value against the target and drops rows that already exist.
    2. Delta Lake refuses to insert rows whose values are identical to existing rows.
    3. COPY INTO deletes source files after a successful load, so nothing remains to be read.
    4. COPY INTO records the files it has already loaded and skips them on subsequent runs. Correct answer.

    Explanation

    COPY INTO keeps track of which source files have been consumed and processes only files not seen before, which makes repeated runs idempotent at the file level. New appends to an already-loaded file are not picked up, so files should be immutable.

  44. Question 44Data Ingestion and Loading

    Which statement correctly contrasts Auto Loader's directory listing mode with its file notification mode?

    1. Listing mode requires a cloud queue, while notification mode lists the directory on every trigger.
    2. Notification mode transfers file contents through the queue, while listing mode reads files directly from storage.
    3. Notification mode supports only Parquet files, while listing mode supports any file format.
    4. Listing mode calls the storage list API each trigger, while notification mode consumes file events from a queue and scales better to very large numbers of files. Correct answer.

    Explanation

    In listing mode the source repeatedly asks cloud storage what files exist, which becomes expensive as the directory grows. In notification mode the source subscribes to file-arrival events through a cloud queue and uses the event payloads to identify new files.

  45. Question 45Data Ingestion and Loading

    A developer creates a new Auto Loader stream over a directory that already contains 10,000 historical files but wants the stream to begin with only files that land after it starts. Which option should be set?

    1. cloudFiles.useNotifications = false
    2. cloudFiles.includeExistingFiles = false Correct answer.
    3. cloudFiles.maxFilesPerTrigger = 1
    4. cloudFiles.schemaEvolutionMode = 'addNewColumns'

    Explanation

    By default Auto Loader processes files already present when a fresh stream starts; setting cloudFiles.includeExistingFiles to false makes the first run ignore that backlog. The setting only affects the initial discovery, after which any new file is picked up normally.

  46. Question 46Data Ingestion and Loading

    A streaming query must begin at the oldest available offsets of a Kafka topic the first time it runs. Which option accomplishes this?

    1. includeHeaders = 'true'
    2. kafka.auto.offset.reset = 'earliest'
    3. startingOffsets = 'earliest' Correct answer.
    4. failOnDataLoss = 'true'

    Explanation

    The Kafka source decides its initial position from startingOffsets, which accepts 'earliest', 'latest', or a JSON map of per-partition offsets. Consumer-level offset reset properties are not used because Spark itself tracks consumed offsets in the checkpoint.

  47. Question 47Data Ingestion and Loading

    What is the purpose of the cloudFiles.schemaLocation option in an Auto Loader stream?

    1. It stores the inferred schema and tracks schema changes across runs. Correct answer.
    2. It stores the checkpointed offsets of files already processed.
    3. It is the directory where records that failed parsing are written.
    4. It holds the transaction log of the target Delta table.

    Explanation

    Auto Loader persists the inferred schema plus a history of detected changes in schemaLocation so that restarts and schema evolution are consistent. Malformed records are not written there; they are surfaced through the rescued data column.

  48. Question 48Data Ingestion and Loading

    Which COPY INTO clause restricts a load to a specific list of named files in the source directory?

    1. FILEFORMAT
    2. COPY_OPTIONS
    3. FILES Correct answer.
    4. PATTERN

    Explanation

    The FILES clause takes an explicit list of file names relative to the source path, which is useful for backfills of known files. PATTERN offers regular-expression matching over paths rather than an explicit list.

  49. Question 49Data Ingestion and Loading

    What is the effect of supplying an explicit schema through .schema(...) when reading with Auto Loader?

    1. Auto Loader requires that schemaHints also be supplied for every column.
    2. Auto Loader uses the supplied schema and skips inferring one from the data. Correct answer.
    3. Auto Loader rejects any file whose columns do not exactly match the supplied schema.
    4. Auto Loader ignores the supplied schema and always infers from a sample of files.

    Explanation

    Providing a schema turns off inference, so the stream starts with the declared types and does not pay the cost of sampling files. Columns that do not fit the declared schema are not fatal; they are captured in the rescued data column when that option is enabled.

  50. Question 50Data Ingestion and Loading

    Which JDBC read option lets you ingest the result of a SQL subquery rather than an entire table?

    1. query Correct answer.
    2. partitionColumn
    3. fetchsize
    4. dbtable

    Explanation

    The query option accepts a parenthesized subquery whose result becomes the source, which is how filtering and joins are pushed to the remote database. It is mutually exclusive with dbtable in the same read.

  51. Question 51Data Ingestion and Loading

    CSV files being loaded with COPY INTO contain a header line that must not be ingested as data. Which clause supplies the reader option for this?

    1. FORMAT_OPTIONS ('header' = 'true') Correct answer.
    2. COPY_OPTIONS ('header' = 'true')
    3. TBLPROPERTIES ('header' = 'true')
    4. OPTIONS ('header' = 'true')

    Explanation

    Reader-specific settings such as header, delimiter, and inferSchema are passed through FORMAT_OPTIONS, which forwards them to the underlying file reader. COPY_OPTIONS handles load behavior instead, such as forcing a reload or enabling validation.

  52. Question 52Data Ingestion and Loading

    Which Kafka source option sets a floor on the number of Spark partitions used to consume a topic within a micro-batch?

    1. maxPartitions
    2. maxOffsetsPerTrigger
    3. minPartitions Correct answer.
    4. numPartitions

    Explanation

    The minPartitions option forces the Kafka source to split the offset ranges for a batch into at least that many Spark partitions, which helps when the default parallelism underutilizes the cluster. It does not change how many records are read.

  53. Question 53Data Ingestion and Loading

    In a Delta Live Tables pipeline, which statement applies a change feed from a CDC source into a target table using SCD type 1 semantics by default?

    1. CREATE OR REFRESH STREAMING TABLE AS SELECT
    2. APPLY CHANGES INTO Correct answer.
    3. COPY INTO
    4. MERGE INTO

    Explanation

    APPLY CHANGES INTO consumes a streaming change feed, using a sequencing column to order events and applying the latest version of each key. Its default is SCD type 1, and SCD type 2 must be requested explicitly with the appropriate clause.

  54. Question 54Data Ingestion and Loading

    A Kafka streaming query was stopped and is now restarted with the same checkpointLocation but with startingOffsets set to 'latest'. What happens?

    1. It rereads the topic from the earliest offset and duplicates previously processed records.
    2. It fails to start because startingOffsets conflicts with the existing checkpoint.
    3. It resumes from the offsets recorded in the checkpoint, and startingOffsets is ignored. Correct answer.
    4. It jumps to the newest offsets and skips all records produced while it was stopped.

    Explanation

    Once a checkpoint exists, the Kafka source continues from the offsets stored there, so startingOffsets applies only to a brand-new query. This is what allows a restarted stream to process only records it has not yet consumed.

  55. Question 55Data Transformation and Modeling

    In a Delta Lake MERGE INTO statement, which clause identifies the source table, view, or DataFrame?

    1. USING Correct answer.
    2. SET
    3. ON
    4. WHEN MATCHED

    Explanation

    The USING clause introduces the source relation that is joined to the target. The ON clause then supplies the join predicate used to determine matches.

  56. Question 56Data Transformation and Modeling

    Which window function returns 1, 2, 3, ... with no gaps even when multiple rows tie on the ORDER BY value?

    1. NTILE(4)
    2. RANK
    3. DENSE_RANK
    4. ROW_NUMBER Correct answer.

    Explanation

    ROW_NUMBER assigns a unique sequential integer to each row in the window, breaking ties arbitrarily. RANK leaves gaps after ties, while DENSE_RANK gives tied rows the same value.

  57. Question 57Data Transformation and Modeling

    When one side of a join is small enough to fit in executor memory, which Spark SQL hint encourages broadcasting it?

    1. /*+ REPARTITION(t) */
    2. /*+ SHUFFLE_HASH(t) */
    3. /*+ MERGE(t) */
    4. /*+ BROADCAST(t) */ Correct answer.

    Explanation

    The BROADCAST hint tells Spark to send the hinted relation to every executor, avoiding a shuffle of the larger side. This is appropriate when the hinted table is small.

  58. Question 58Data Transformation and Modeling

    In a medallion architecture, which layer typically holds cleaned, conformed, and often joined data that downstream marts consume?

    1. Bronze
    2. Landing
    3. Gold
    4. Silver Correct answer.

    Explanation

    The silver layer stores validated and conformed data, often joining or enriching bronze records. Gold then contains business-level aggregates and presentation-ready tables.

  59. Question 59Data Transformation and Modeling

    Which SQL syntax reads the Delta table sales as it existed at version 12?

    1. SELECT * FROM sales AT VERSION 12
    2. SELECT * FROM sales TIMESTAMP AS OF 12
    3. SELECT * FROM sales AS OF VERSION 12
    4. SELECT * FROM sales VERSION AS OF 12 Correct answer.

    Explanation

    Delta time travel accepts VERSION AS OF for a specific table version and TIMESTAMP AS OF for a point in time. Version 12 is a version number, so VERSION AS OF is the correct form.

  60. Question 60Data Transformation and Modeling

    Which function returns one row per element of an array and drops rows where the array is null or empty?

    1. explode Correct answer.
    2. posexplode
    3. explode_outer
    4. flatten

    Explanation

    explode produces one output row per array element and omits rows whose array is null or empty. explode_outer behaves similarly but preserves those rows with nulls.

  61. Question 61Data Transformation and Modeling

    By default, a table created with CREATE TABLE in Unity Catalog without a LOCATION is what kind of object?

    1. an external table
    2. a temporary view
    3. a streaming table
    4. a managed table Correct answer.

    Explanation

    Without an explicit LOCATION, Unity Catalog creates a managed table and controls both metadata and data storage. Dropping a managed table removes its underlying data files.

  62. Question 62Data Transformation and Modeling

    Which column is generally a good choice for partitioning a large Delta table?

    1. A free-text comment column
    2. A date column used in filters Correct answer.
    3. A column updated on every row
    4. A high-cardinality primary key

    Explanation

    Partitioning works best on relatively low-cardinality columns that queries commonly filter on, such as a date used in range predicates. High-cardinality keys can create too many small directories and files.

  63. Question 63Data Transformation and Modeling

    Which approach returns exactly one row per customer_id using the most recent updated_at value?

    1. RANK() with ORDER BY updated_at DESC, keep rnk = 1
    2. GROUP BY customer_id, taking MAX(updated_at)
    3. SELECT DISTINCT customer_id, * FROM transactions
    4. ROW_NUMBER() with ORDER BY updated_at DESC, keep rn = 1 Correct answer.

    Explanation

    ROW_NUMBER partitioned by customer_id and ordered by updated_at descending assigns 1 to the latest row, and filtering rn = 1 keeps exactly one row per customer. RANK can return multiple rows when timestamps tie.

  64. Question 64Data Transformation and Modeling

    A MERGE INTO target USING source fails with a message that multiple source rows matched the same target row. What is the most likely cause?

    1. The target table contains duplicate keys
    2. The source has multiple rows matching one target row Correct answer.
    3. The ON clause is missing
    4. The source is a view instead of a table

    Explanation

    Delta MERGE requires that each target row match at most one source row, otherwise update semantics are ambiguous. Duplicate keys in the source must be deduplicated before the MERGE.

  65. Question 65Data Transformation and Modeling

    A table is partitioned by date, but queries frequently filter by customer_id across many dates. Which optimization best improves data skipping on customer_id?

    1. Z-ORDER BY customer_id Correct answer.
    2. VACUUM the table
    3. PARTITION BY customer_id
    4. CACHE TABLE sales

    Explanation

    Z-ORDER co-locates related customer_id values within files so file-level statistics can skip more data during filtering. Partitioning by customer_id instead would likely create too many small partitions.

  66. Question 66Data Transformation and Modeling

    An append writes a DataFrame with a new column to an existing Delta table, and the new column should be added automatically. Which write option enables this?

    1. overwriteSchema=true
    2. schemaEvolution=true
    3. autoMerge=true
    4. mergeSchema=true Correct answer.

    Explanation

    For Delta writes, mergeSchema=true allows the schema of the incoming data to be merged into the existing table schema. This adds new columns while preserving existing columns.

  67. Question 67Data Transformation and Modeling

    Which pair of columns is essential for representing SCD Type 2 history in a dimension table?

    1. a row hash and a version number
    2. created_at and updated_at audit timestamps
    3. effective start and end dates Correct answer.
    4. a surrogate key plus the natural business key

    Explanation

    SCD Type 2 tracks validity periods, so each historical version needs an effective start and an effective end date or an equivalent current-row indicator. Audit timestamps alone do not define which version was valid for a business event.

  68. Question 68Data Transformation and Modeling

    Which configuration specifically enables Adaptive Query Execution to handle skewed join partitions?

    1. spark.sql.shuffle.partitions
    2. spark.sql.adaptive.skewJoin.enabled Correct answer.
    3. spark.sql.adaptive.enabled
    4. spark.sql.autoBroadcastJoinThreshold

    Explanation

    AQE skew join handling is controlled by spark.sql.adaptive.skewJoin.enabled, with AQE itself enabled as a prerequisite. It splits oversized partitions so one task does not dominate the join stage.

  69. Question 69Data Transformation and Modeling

    A Delta table has the CHECK constraint amount >= 0. A write includes one row with amount = -5. What happens by default?

    1. The row is written with NULL in amount
    2. Only the offending row is dropped
    3. The write fails and no rows are committed Correct answer.
    4. The constraint is ignored for that row

    Explanation

    Delta enforces CHECK constraints atomically: if any row violates the constraint, the transaction fails and none of the write is committed. The offending row is not silently discarded.

  70. Question 70Data Transformation and Modeling

    Which Delta table property must be enabled to read row-level changes made to a table since a version or timestamp?

    1. delta.enableChangeDataFeed = true Correct answer.
    2. delta.logRetentionDuration = '30 days'
    3. delta.appendOnly = true
    4. delta.autoOptimize.optimizeWrite = true

    Explanation

    Change Data Feed records row-level insert, update, and delete events that can be queried by version or timestamp. It is enabled with the table property delta.enableChangeDataFeed = true.

  71. Question 71Data Transformation and Modeling

    Which Auto Loader option stores the inferred schema and tracks schema evolution across multiple runs?

    1. cloudFiles.useNotifications
    2. cloudFiles.format
    3. cloudFiles.schemaLocation Correct answer.
    4. cloudFiles.inferColumnTypes

    Explanation

    cloudFiles.schemaLocation gives Auto Loader a durable location to store the inferred schema and schema evolution history. Without it, each run may infer the schema independently.

  72. Question 72Data Transformation and Modeling

    A streaming query must apply MERGE INTO to a Delta table. Which sink pattern is appropriate?

    1. foreachBatch with a function that runs MERGE Correct answer.
    2. trigger(once=True) on a batch query
    3. writeStream with a Delta sink in append mode
    4. foreach with a custom sink

    Explanation

    foreachBatch exposes each micro-batch as a DataFrame, allowing arbitrary batch logic such as MERGE INTO. A plain Delta streaming sink only supports append operations.

  73. Question 73Data Transformation and Modeling

    Given an array column items, which expression keeps only elements greater than 10?

    1. exists(items, x -> x > 10)
    2. filter(items, x -> x > 10) Correct answer.
    3. aggregate(items, x -> x > 10)
    4. transform(items, x -> x > 10)

    Explanation

    The higher-order filter function returns a new array containing only elements that satisfy the predicate. transform maps every element, while exists returns a boolean.

  74. Question 74Data Transformation and Modeling

    Which operation reduces the number of partitions without performing a full shuffle?

    1. distribute by
    2. repartition(n)
    3. coalesce(n) Correct answer.
    4. sortWithinPartitions

    Explanation

    coalesce merges adjacent partitions and avoids a full shuffle, which makes it useful for reducing output files. repartition performs a shuffle and can increase or decrease partitions.

  75. Question 75Data Transformation and Modeling

    Which statement about liquid clustering in Delta Lake is correct?

    1. Liquid clustering requires a Hive metastore
    2. Clustering keys must be partition columns
    3. Liquid clustering and Z-ORDER cannot coexist on different tables
    4. Clustering keys can be changed without rewriting data Correct answer.

    Explanation

    Liquid clustering lets you change clustering keys without immediately rewriting existing data; new writes use the updated keys. It is an alternative to partitioning and Z-ORDER for many new tables.

  76. Question 76Data Transformation and Modeling

    When a Delta MERGE has multiple WHEN MATCHED clauses with different conditions, how are they evaluated?

    1. All matching WHEN MATCHED clauses run in order
    2. Delta rejects more than one WHEN MATCHED clause
    3. Only the first matching WHEN MATCHED clause runs Correct answer.
    4. A matched row always runs the DELETE clause first

    Explanation

    Delta evaluates WHEN MATCHED clauses in the order written and executes only the first clause whose condition is true. This lets a table apply conditional updates and deletes in one statement.

  77. Question 77Data Transformation and Modeling

    In Spark SQL, what is the default window frame when a window has ORDER BY but no explicit frame clause?

    1. ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
    2. RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW Correct answer.
    3. RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING
    4. ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING

    Explanation

    With ORDER BY and no frame specification, Spark SQL uses RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. This includes peers with the same ORDER BY value, unlike the ROWS equivalent.

  78. Question 78Data Transformation and Modeling

    Which approach correctly deduplicates an event-time stream by event key while bounding state growth?

    1. Call distinct() on the streaming DataFrame
    2. Use a batch MERGE after the query stops
    3. Call withWatermark then dropDuplicates on the event key Correct answer.
    4. Use dropDuplicates without a watermark on an unbounded state

    Explanation

    Applying withWatermark before dropDuplicates lets the engine expire old deduplication state once events are past the watermark. Without a watermark, deduplication state can grow without bound.

  79. Question 79Data Transformation and Modeling

    Which setting allows a Delta MERGE INTO to automatically add new source columns to the target table?

    1. spark.sql.sources.mergeSchema = true
    2. spark.sql.mergeSchema.enabled
    3. delta.autoMerge.schema = true
    4. spark.databricks.delta.schema.autoMerge.enabled Correct answer.

    Explanation

    MERGE schema evolution is controlled by spark.databricks.delta.schema.autoMerge.enabled. When enabled, new columns in the source can be added to the target during the MERGE.

  80. Question 80Data Transformation and Modeling

    Which higher-order function reduces every element of an array to a single value by carrying an accumulator from one element to the next?

    1. zip_with
    2. filter
    3. transform
    4. aggregate Correct answer.

    Explanation

    aggregate walks the array with an accumulator, applying a merge step per element and a finish step to the accumulator, so it returns a scalar. transform and filter both return arrays rather than a single value.

  81. Question 81Data Transformation and Modeling

    Two rows tie on the ORDER BY column of a window. Which function assigns both the same value and then skips the next value, producing the sequence 1, 1, 3?

    1. ntile
    2. dense_rank
    3. rank Correct answer.
    4. row_number

    Explanation

    rank gives tied rows the same rank and then leaves a gap equal to the number of tied rows, so the next distinct value receives 3. dense_rank leaves no gap, and row_number never ties.

  82. Question 82Data Transformation and Modeling

    Which SQL statement returns the commit history of a Delta table, including version numbers, timestamps and operation types?

    1. DESCRIBE EXTENDED
    2. SHOW CREATE TABLE
    3. DESCRIBE HISTORY Correct answer.
    4. SHOW TABLES

    Explanation

    DESCRIBE HISTORY reads the Delta transaction log and returns one row per commit with version, timestamp, operation and operation metrics. The other statements describe catalog metadata or the table definition.

  83. Question 83Data Transformation and Modeling

    Running OPTIMIZE on a Delta table with no ZORDER clause primarily accomplishes which of the following?

    1. Deletes data files older than the retention period
    2. Recomputes column statistics used by the optimizer
    3. Converts the table's storage format from Delta to Parquet
    4. Rewrites many small files into fewer larger files Correct answer.

    Explanation

    OPTIMIZE performs bin-packing, rewriting small files into fewer, larger files of roughly a target size to reduce per-file overhead. Adding ZORDER BY also co-locates related data, which is a separate effect.

  84. Question 84Data Transformation and Modeling

    Which join returns only the rows from the left DataFrame that have no matching row on the right?

    1. full_outer
    2. left_semi
    3. left_anti Correct answer.
    4. left_outer

    Explanation

    A left anti join keeps left rows whose key is absent from the right side and returns only left columns. It is the DataFrame equivalent of a NOT EXISTS filter.

  85. Question 85Data Transformation and Modeling

    Which SQL clause turns the distinct values of a column into separate output columns?

    1. LATERAL VIEW
    2. UNPIVOT
    3. PIVOT Correct answer.
    4. GROUPING SETS

    Explanation

    PIVOT aggregates values and spreads the distinct values of a chosen column across new columns. It is the inverse of UNPIVOT, which rotates columns into rows.

  86. Question 86Data Transformation and Modeling

    A dimension attribute is corrected in place so that the previous value is overwritten and no history is kept. Which slowly changing dimension type does this describe?

    1. Type 3
    2. Type 0
    3. Type 1 Correct answer.
    4. Type 2

    Explanation

    SCD Type 1 overwrites the attribute value, so the prior value is lost and no historical row is created. Type 2 preserves history by inserting new rows, and Type 3 keeps a limited previous value in an added column.

  87. Question 87Data Transformation and Modeling

    An events table has one row per event with user_id and event_time. You need the elapsed time between each event and the previous event for the same user. Which approach is correct?

    1. Use lag over the user partition ordered by event_time and subtract Correct answer.
    2. Use count(*) over the user partition ordered by event_time and subtract the row counts
    3. Join the table to itself on user_id and on equal event_time values, then subtract the two timestamps
    4. Assign row_number per user, self-join on the previous row_number, then subtract the timestamps

    Explanation

    lag(col) OVER (PARTITION BY user_id ORDER BY event_time) returns the prior row's timestamp within each user, so subtracting it from the current timestamp yields the gap. A self-join on equal timestamps only matches rows that happen to share a timestamp.

  88. Question 88Data Transformation and Modeling

    Two DataFrames share an identical schema. Which pair of operations produces the set union of both, removing all duplicate rows?

    1. Union the DataFrames, then call dropDuplicates on the key column only
    2. Join the DataFrames on all columns with an inner join
    3. Union the DataFrames, then call distinct Correct answer.
    4. Intersect the DataFrames, then union the result back to each input

    Explanation

    In Spark, union keeps duplicates, so a following distinct is required for set-union semantics. SQL UNION without the ALL keyword applies the same deduplication.

  89. Question 89Data Transformation and Modeling

    A model keeps product, subcategory and category in three separate related dimension tables instead of a single product dimension. Which schema pattern is this?

    1. Fact constellation
    2. Flat denormalized schema
    3. Snowflake schema Correct answer.
    4. Star schema

    Explanation

    A snowflake schema normalizes a dimension into a hierarchy of related tables, which is exactly the product to subcategory to category chain. A star schema keeps each dimension as one denormalized table joined directly to the fact.

  90. Question 90Data Transformation and Modeling

    Which statement describes a shallow clone of a Delta table?

    1. It is a view over the source table that always shows the source's latest data
    2. It can only be created from a table that has change data feed enabled
    3. It copies all data files to a new location and is fully independent of the source
    4. It copies the table metadata and references the source's existing data files Correct answer.

    Explanation

    A shallow clone copies the transaction log and metadata to a new table but points at the source's existing Parquet files, so it is cheaper but depends on those files remaining. A deep clone copies both metadata and data.

  91. Question 91Data Transformation and Modeling

    A very large fact table is joined to a dimension table of a few megabytes. Which technique avoids shuffling the fact table?

    1. Disable broadcast by setting the broadcast threshold to -1
    2. Broadcast hint on the dimension table Correct answer.
    3. Repartition both inputs by the join key before the join
    4. Bucket both tables by the join key before the join

    Explanation

    Broadcasting ships the small dimension to every executor so each partition of the large fact table is joined locally without a shuffle. Repartitioning or bucketing both sides forces a shuffle of the fact table.

  92. Question 92Data Transformation and Modeling

    A string column contains some non-numeric values. Which expression returns NULL for those values instead of raising an error in Databricks SQL?

    1. to_number(col, '999')
    2. cast(col AS INT)
    3. try_cast(col AS INT) Correct answer.
    4. col::INT

    Explanation

    try_cast performs the conversion but returns NULL when the value cannot be parsed, which suits dirty source data. cast and the :: shorthand raise an error instead of returning NULL when ANSI mode is on.

  93. Question 93Data Transformation and Modeling

    Which clause filters rows on the result of a window function without wrapping the query in a subquery?

    1. LIMIT
    2. WHERE
    3. HAVING
    4. QUALIFY Correct answer.

    Explanation

    QUALIFY is evaluated after window functions are computed, so predicates on rank or row_number can be applied in the same SELECT. WHERE runs before windows exist, so it cannot reference them.

  94. Question 94Data Transformation and Modeling

    A window is defined with ORDER BY day ROWS BETWEEN 6 PRECEDING AND CURRENT ROW and SUM(amount) is computed over it. What does each row receive?

    1. The sum of the current row and the six preceding rows Correct answer.
    2. The sum of all rows in the partition regardless of position
    3. The sum of the current row and the six following rows
    4. The sum of all rows from the start of the partition through the current row

    Explanation

    A ROWS frame counts physical rows relative to the current row, so this frame spans the current row plus the six before it, producing a seven-row moving total. Using UNBOUNDED PRECEDING instead would give a running total from the partition start.

  95. Question 95Data Transformation and Modeling

    A Delta table is vacuumed with a retention of zero hours. What is the outcome?

    1. Time travel is unaffected, but queries against older versions return null column values
    2. Time travel continues to work because VACUUM only removes entries from the transaction log
    3. Time travel to versions whose files were removed fails, and concurrent readers may fail Correct answer.
    4. Time travel works after a zero retention because Delta keeps a full copy of prior versions

    Explanation

    VACUUM physically deletes data files no longer referenced by the current table state, so versions that relied on those files can no longer be read. A zero-hour retention also risks deleting files that running readers still need, causing file-not-found errors.

  96. Question 96Data Transformation and Modeling

    A Delta table has a generated column derived from a timestamp column. Which statement about writing to that table is correct?

    1. Writers must supply the generated value and Delta validates the expression
    2. Writers must not supply a value for the generated column Correct answer.
    3. The generated column is computed only when the table is optimized
    4. The generated column can be updated directly as long as the expression still holds

    Explanation

    Delta computes generated columns itself on insert, so providing a value for that column is rejected as an error. The expression is part of the table definition and is evaluated automatically for every write.

  97. Question 97Data Transformation and Modeling

    A monthly sales total computed from a sales fact table joined to a customer dimension comes out inflated. The dimension holds two rows per customer, current and historical. What is the cause?

    1. The two tables use different data types for the join key, causing implicit duplication
    2. The dimension is not unique on the join key, so fact rows are duplicated Correct answer.
    3. The aggregation used SUM where COUNT was needed to preserve the grain
    4. The fact table is partitioned on a high-cardinality column, so partition pruning duplicates rows

    Explanation

    Joining a fact row to two dimension rows produces two output rows, so each measure is counted twice, a fan-out or chained-dimension trap. The fix is to filter the dimension to the current version, or join on a surrogate key that is unique.

  98. Question 98Data Transformation and Modeling

    Which Delta table feature must be enabled before a column can be renamed or dropped without rewriting the underlying Parquet data files?

    1. Column mapping Correct answer.
    2. Change data feed
    3. Deletion vectors
    4. Liquid clustering

    Explanation

    Column mapping stores a physical-to-logical name mapping in the table metadata, so rename and drop operations only change metadata and leave data files untouched. Without it, Delta must rewrite the files to change the physical schema.

  99. Question 99Working with Lakeflow Jobs

    A Lakeflow Job is configured to create a new job cluster for each run. What happens to that cluster after the run completes?

    1. It becomes an all-purpose cluster available to other users.
    2. It is retained for 24 hours so later runs can reuse it.
    3. It is terminated automatically. Correct answer.
    4. It remains running until manually terminated.

    Explanation

    Job clusters are dedicated compute owned by the job: Databricks starts the cluster when the run begins and terminates it when the run ends, which is why job compute is cheaper than all-purpose compute.

  100. Question 100Working with Lakeflow Jobs

    Task transform depends on task ingest with the default setting Run if: All succeeded. Ingest fails after exhausting its retries. What happens to transform and to the run?

    1. Transform runs and the run is marked succeeded with warnings.
    2. Transform is skipped and the run is marked failed. Correct answer.
    3. Transform runs using data from the previous successful run.
    4. Transform waits until ingest succeeds in a later run.

    Explanation

    With the default Run if setting of All succeeded, a downstream task runs only when every upstream task completes successfully. When an upstream task fails, its dependents are skipped and the overall run is reported as failed.

  101. Question 101Working with Lakeflow Jobs

    A team keeps Python source files in a workspace Git folder and wants a job task that executes one of those files directly. Which task type should be used?

    1. SQL
    2. Python script Correct answer.
    3. Notebook
    4. Python wheel

    Explanation

    A Python script task points at a .py file, such as one checked out from Git, and runs it as a script, so no packaging step is required. Wheel tasks instead require a built distribution, and notebook tasks expect notebook-format sources.

  102. Question 102Working with Lakeflow Jobs

    Which format does Databricks use for job schedule expressions?

    1. Five-field cron with minute-level granularity only
    2. An ISO-8601 duration such as PT24H
    3. Quartz cron with six fields, where the first field is seconds Correct answer.
    4. A natural-language phrase such as 'every day at 9am'

    Explanation

    Databricks scheduled triggers use Quartz cron syntax, which includes a leading seconds field and therefore has six fields. Five-field Unix cron expressions are not accepted as-is.

  103. Question 103Working with Lakeflow Jobs

    A job keeps the default value of 1 for maximum concurrent runs, and a run is still active when the next scheduled time arrives. What does Databricks do with the new trigger?

    1. It queues the new run to start as soon as the active run ends.
    2. It runs both concurrently on separate clusters.
    3. It skips the new run. Correct answer.
    4. It cancels the active run and starts the new one.

    Explanation

    Maximum concurrent runs caps how many runs of a job can be active at once. When the cap is already reached, the newly triggered run is skipped rather than queued or executed.

  104. Question 104Working with Lakeflow Jobs

    A task is configured with 2 retries. Attempt 1 fails, attempt 2 fails, and attempt 3 succeeds. How many attempts are made and what is the outcome?

    1. Three attempts; the task is marked failed.
    2. Two attempts; the task is marked failed.
    3. Three attempts; the task succeeds. Correct answer.
    4. Two attempts; the task succeeds.

    Explanation

    The retry setting is the number of additional attempts after the initial one, so 2 retries allows up to 3 attempts in total. If any attempt succeeds the task succeeds and the run continues.

  105. Question 105Working with Lakeflow Jobs

    Which destination types can receive notifications for a Lakeflow Job run?

    1. Email addresses and notification destinations such as Slack or webhooks. Correct answer.
    2. Only email addresses configured on the job.
    3. Only Databricks user accounts listed in the job permissions.
    4. Only external ticketing systems connected through the Jobs API.

    Explanation

    Job notifications can target individual or group email addresses and also system notification destinations, which deliver to services such as Slack or generic webhooks. Both are configured in the job's notification settings.

  106. Question 106Working with Lakeflow Jobs

    A job uses a continuous trigger. How does Databricks operate the job?

    1. It runs only when new files land in the watched volume.
    2. It runs a single time and then pauses itself.
    3. It keeps the compute running while active and restarts it after failures. Correct answer.
    4. It runs once per day at the configured cron time.

    Explanation

    Continuous jobs are designed for always-on processing such as streaming ingestion: the compute stays running while the job is active and Databricks restarts it if it fails or is terminated.

  107. Question 107Working with Lakeflow Jobs

    An upstream task named extract calls dbutils.jobs.taskValues.set(key="row_count", value=1000). Which reference returns that value in a downstream task?

    1. {{tasks.extract.values.row_count}} Correct answer.
    2. {{taskValues.extract.row_count}}
    3. {{tasks.extract.row_count}}
    4. {{job.values.extract.row_count}}

    Explanation

    Task values set with dbutils.jobs.taskValues are read with the tasks.<task_name>.values.<key> dynamic value reference. A task can read only values set by tasks it depends on.

  108. Question 108Working with Lakeflow Jobs

    A notebook task must record the identifier of the run it belongs to. Which dynamic value reference should be passed as a parameter?

    1. {{task.run_id}}
    2. {{job.start_time}}
    3. {{job.id}}
    4. {{job.run_id}} Correct answer.

    Explanation

    The job.run_id reference resolves to the identifier of the current job run, which is what audit tables typically record. job.id returns the job definition identifier instead, and job.start_time returns a timestamp.

  109. Question 109Working with Lakeflow Jobs

    Task cleanup must execute only when an upstream task fails, and must be skipped when the upstream task succeeds. Which Run if setting should cleanup use?

    1. At least one failed Correct answer.
    2. At least one succeeded
    3. All succeeded
    4. All done

    Explanation

    Run if: At least one failed causes the task to run when any of its upstream dependencies failed, and to be skipped when they all succeed. All done would run the task regardless of the upstream outcome.

  110. Question 110Working with Lakeflow Jobs

    A nightly job has six dependent tasks and the run fails at task D while tasks A, B, and C succeeded. Which statement describes the default behavior of Repair run?

    1. It re-runs only the failed task, leaving its dependents skipped.
    2. It creates a new job containing only the failed task.
    3. It re-runs the failed task and its downstream dependents, reusing successful upstream results. Correct answer.
    4. It re-runs all tasks in the job from the beginning.

    Explanation

    Repair run re-executes the failed task and the tasks that depend on it, while tasks that completed successfully in the original run are not recomputed. This shortens recovery time for long pipelines.

  111. Question 111Working with Lakeflow Jobs

    A data engineer wants a job to start when files arrive in a Unity Catalog volume. What must the target storage support for a file arrival trigger?

    1. A DBFS mount created by a workspace administrator.
    2. Change data feed enabled on a Delta table.
    3. Autoscaling enabled on the job cluster.
    4. File events enabled on a Unity Catalog external location or volume. Correct answer.

    Explanation

    File arrival triggers depend on cloud storage file events, which must be enabled on the Unity Catalog external location or volume that the job watches. The trigger then fires when matching files land in the path.

  112. Question 112Working with Lakeflow Jobs

    Which job permission level lets a user view runs, repair runs, and cancel runs, but not edit the job definition?

    1. CAN MANAGE RUN Correct answer.
    2. IS OWNER
    3. CAN VIEW
    4. CAN MANAGE

    Explanation

    CAN MANAGE RUN grants operation of existing runs, including viewing results, repairing, and cancelling, without allowing changes to the job's tasks or configuration. Editing the definition requires CAN MANAGE, and IS OWNER additionally controls permissions.

  113. Question 113Working with Lakeflow Jobs

    A job is configured with a service principal as its Run as identity, and a user with CAN MANAGE RUN starts a run. Which identity's permissions are used when the tasks read data?

    1. The permissions of a workspace administrator.
    2. The permissions of the user who started the run.
    3. The permissions of the configured service principal. Correct answer.
    4. The combined permissions of the service principal and the triggering user.

    Explanation

    The Run as setting determines the identity a job uses for data access and for creating compute, which is commonly a service principal for production jobs. The triggering user needs only enough permission to start the run.

  114. Question 114Working with Lakeflow Jobs

    A task has no timeout configured. What is its default timeout behavior?

    1. It is terminated after 24 hours.
    2. It is terminated after 1 hour.
    3. It is terminated after 30 minutes.
    4. It has no timeout and can run indefinitely. Correct answer.

    Explanation

    The task timeout defaults to 0, which means no timeout is applied and the task runs until it completes or fails. A non-zero timeout must be set explicitly to bound the task duration.

  115. Question 115Working with Lakeflow Jobs

    A job defines the parameter env with default value dev. A run is triggered with an override of env = prod, and a notebook task reads the widget named env. Which value does the task receive?

    1. The run is rejected because job defaults cannot be overridden.
    2. dev, then prod on the next run
    3. dev
    4. prod Correct answer.

    Explanation

    A job parameter default is used only when the trigger does not supply a value. Values passed at trigger time override the default and are what the task's widget receives.

  116. Question 116Working with Lakeflow Jobs

    Job runs take several minutes to start because each run provisions new job clusters. Which change reduces that startup time while keeping the compute dedicated to the job?

    1. Move the tasks onto a shared all-purpose cluster.
    2. Increase the driver node size for each task.
    3. Enable autoscaling on the job clusters.
    4. Attach the job clusters to an instance pool. Correct answer.

    Explanation

    Instance pools keep pre-provisioned, idle instances ready so job clusters can acquire nodes without waiting for cloud provisioning. The clusters remain job-scoped while startup latency drops.

  117. Question 117Working with Lakeflow Jobs

    An upstream notebook task finishes with dbutils.notebook.exit("42"). Which reference returns that value to a downstream task?

    1. {{tasks.upstream.output}} Correct answer.
    2. {{job.exit_value}}
    3. {{tasks.upstream.values.exit}}
    4. {{tasks.upstream.result}}

    Explanation

    The value passed to dbutils.notebook.exit becomes the notebook task's output, which downstream tasks read with the tasks.<task_name>.output reference. Task values set with dbutils.jobs.taskValues are a separate mechanism.

  118. Question 118Working with Lakeflow Jobs

    A schedule is created without specifying a time zone. Which time zone governs when the job runs?

    1. The time zone of the region hosting the workspace's compute.
    2. The browser's local time zone at creation time.
    3. The job owner's profile time zone.
    4. UTC Correct answer.

    Explanation

    Databricks schedules default to UTC when no time zone is provided, both in the UI and through the API. A specific IANA time zone can be set on the trigger when local time is required.

  119. Question 119Working with Lakeflow Jobs

    Which statement describes serverless compute for Lakeflow Jobs?

    1. The customer's VPC must first be peered with the control plane.
    2. Databricks provisions and manages the compute for the run. Correct answer.
    3. It supports only JAR and Python wheel tasks.
    4. A cluster policy must permit all-purpose cluster creation.

    Explanation

    With serverless compute, Databricks provisions and manages the compute so the job definition contains no cluster configuration. It is not limited to particular task types and does not depend on customer-managed cluster policies.

  120. Question 120Working with Lakeflow Jobs

    What is the default retention period for job run history in a Databricks workspace?

    1. 60 days Correct answer.
    2. 7 days
    3. 365 days
    4. 90 days

    Explanation

    Run history is retained for 60 days by default, and the value can be adjusted within the supported range of 30 to 90 days. Runs older than the retention window are removed from the history.

  121. Question 121Working with Lakeflow Jobs

    A run fails at a task because of an incorrect parameter value. The engineer must re-execute that task and its dependents with a corrected value while keeping the successful upstream results. Which approach fits?

    1. Repair the run and override the parameter for the repaired run. Correct answer.
    2. Re-run the entire job from the beginning with the new value.
    3. Cancel the run and trigger a fresh run with the new value.
    4. Clone the job, edit the parameter, and run the clone.

    Explanation

    Repair run re-executes only the failed task and its downstream dependents, and it accepts parameter overrides for the repaired run. That preserves the completed upstream work instead of recomputing it.

  122. Question 122Working with Lakeflow Jobs

    An engineer needs to read the printed output and the error message produced by a notebook task in a completed run. Which Jobs API call returns this information?

    1. jobs.list_jobs with a name filter
    2. jobs.get_run_output with the run_id Correct answer.
    3. jobs.list_runs with the job_id
    4. jobs.get_run with the run_id

    Explanation

    jobs.get_run_output returns the notebook output, error details, and metadata for a single task run, which is why it is the call used to retrieve task results. jobs.get_run returns run and task states but not the output payload.

  123. Question 123Working with Lakeflow Jobs

    A job runs task load and then task cleanup, whose Run if setting is At least one failed. Load fails and cleanup runs successfully. What is the final state of the run?

    1. The run is marked failed. Correct answer.
    2. The run stays pending until an engineer resolves it.
    3. The run is marked succeeded because cleanup completed.
    4. The run is marked succeeded with warnings.

    Explanation

    A run fails if any task in it fails, even when a cleanup task configured with At least one failed completes successfully afterwards. The cleanup task's success does not change the failed status of the run.

  124. Question 124Working with Lakeflow Jobs

    A job is configured to create a job cluster rather than using an existing cluster. What happens to that compute?

    1. It is shared with other jobs for as long as the job is scheduled.
    2. It is created at schedule creation time and reused by every run.
    3. It is created once and stays running until an administrator deletes it.
    4. It is created for the run and terminated when the run finishes. Correct answer.

    Explanation

    Job clusters are dedicated compute provisioned for a job run and torn down when that run ends, so each run gets clean compute and you pay only for the run duration. They are not long-lived like all-purpose clusters; pools can be used to reduce the provisioning delay.

  125. Question 125Working with Lakeflow Jobs

    A scheduled job has its schedule status set to Paused. What is the effect at the next scheduled time?

    1. No run is triggered until the schedule is unpaused. Correct answer.
    2. The run is triggered and stored as a pending run.
    3. The run starts but immediately enters a paused waiting state.
    4. The schedule definition is removed and must be recreated.

    Explanation

    Pausing a schedule suspends future triggers while keeping the schedule definition intact, so nothing fires at the next matching time. Resuming the schedule restores triggering from the next matching occurrence onward.

  126. Question 126Working with Lakeflow Jobs

    A task of type Run Job is used to launch a second job from the current job. How are the two executions recorded?

    1. The child job's tasks are merged into the parent run's task list and share its run ID.
    2. The child job run supersedes the parent run as soon as the child job starts executing.
    3. Only the parent run is recorded, and the child job's tasks appear as logs of the triggering task.
    4. The parent run records a task run linked to a separate run in the child job's history. Correct answer.

    Explanation

    A Run Job task starts the target job as its own run, so the child job's run history holds that run independently. The parent run still shows a task run for the triggering task so the lineage between the two runs is traceable.

  127. Question 127Working with Lakeflow Jobs

    Which Quartz cron expression schedules a job to run at 8:00 AM Monday through Friday?

    1. 0 8 * * * MON-FRI
    2. 0 8 0 ? * MON-FRI
    3. 0 0 8 ? * MON-FRI Correct answer.
    4. 8 0 0 ? * MON-FRI

    Explanation

    Quartz cron fields are ordered seconds, minutes, hours, day-of-month, month, day-of-week, so 8:00 AM is seconds 0, minutes 0, hours 8. Exactly one of day-of-month or day-of-week must be set to ?, and the weekday range MON-FRI restricts execution to business days.

  128. Question 128Working with Lakeflow Jobs

    A job defines a job-level parameter named env, and a task's parameter field must resolve that value when the run executes. Which reference should be used?

    1. {{job.parameters.env}} Correct answer.
    2. {{job.env.value}}
    3. {{parameters.job.env}}
    4. {{task.parameters.env}}

    Explanation

    Job parameters are surfaced to task fields through the {{job.parameters.<name>}} syntax, which resolves at run time from the run's supplied values and otherwise from the job default. Task parameters themselves are referenced with {{task.parameters.<name>}}, not for job-level values.

  129. Question 129Working with Lakeflow Jobs

    An engineer must retain the application logs written by the nodes of a job cluster after the run ends. What should be configured?

    1. A cluster log delivery path in the cluster's advanced options. Correct answer.
    2. A longer run history retention period on the job.
    3. A higher timeout on the tasks that use the cluster.
    4. A notification destination of type webhook on the job's tasks.

    Explanation

    Cluster log delivery writes driver and worker logs to a specified storage location, which keeps them readable after the cluster is terminated. Run history retention and notification settings govern run metadata and alerts, not node-level log files.

  130. Question 130Working with Lakeflow Jobs

    A job is scheduled every 10 minutes with Max concurrent runs set to 2, and each run takes about 45 minutes. What happens to trigger times that arrive while two runs are already active?

    1. They start a third and fourth run, which wait in a queue until a slot frees.
    2. They cancel the oldest active run and start a fresh run in its place.
    3. They are skipped, and no run is created for those trigger times. Correct answer.
    4. They pause the schedule until an active run completes, then fire once.

    Explanation

    Max concurrent runs caps how many runs of a job may be active at once, and triggers that arrive after the cap is reached are skipped rather than queued. When runs last longer than the interval, this prevents a backlog from building but leaves gaps in coverage for the skipped trigger times.

  131. Question 131Implementing CI/CD

    Which file is the default configuration file for a Databricks Asset Bundle?

    1. databricks.yml Correct answer.
    2. project.json
    3. bundle.yaml
    4. workspace.tf

    Explanation

    Databricks Asset Bundles use databricks.yml at the bundle root as the entry point for targets, variables, and resources. The CLI reads this file by default for validate, deploy, and destroy operations.

  132. Question 132Implementing CI/CD

    Which command deploys the resources defined in a Databricks Asset Bundle to the selected target?

    1. databricks bundle build
    2. databricks bundle deploy Correct answer.
    3. databricks jobs deploy
    4. databricks workspace deploy

    Explanation

    `databricks bundle deploy` applies the bundle definition to the target workspace, creating or updating jobs, pipelines, and other declared resources. The build command prepares artifacts but does not push resources to the workspace.

  133. Question 133Implementing CI/CD

    What is the primary purpose of Databricks Git folders (formerly Repos)?

    1. To convert SQL notebooks into dashboards
    2. To automatically run notebooks whenever a commit is pushed
    3. To replace Unity Catalog for data governance
    4. To link a workspace folder to a remote Git repo Correct answer.

    Explanation

    Git folders clone and sync a remote Git repository into the workspace so notebooks and files are version-controlled through normal Git workflows. They provide source control integration, not orchestration or governance replacement.

  134. Question 134Implementing CI/CD

    How should a CI/CD pipeline supply Databricks credentials during an automated deployment?

    1. Commit a personal access token to the repository
    2. Hard-code credentials in the deployment notebook
    3. Use the CI system's secret store Correct answer.
    4. Embed a service principal secret in databricks.yml

    Explanation

    Secrets should live in the CI/CD platform's encrypted secret store and be injected as environment variables at runtime. Committing or hard-coding credentials exposes them to anyone with repository or workspace access and makes rotation difficult.

  135. Question 135Implementing CI/CD

    What does `databricks bundle validate` do?

    1. Executes all jobs defined in the bundle
    2. Deletes resources that were previously deployed
    3. Deploys every resource to the target workspace
    4. Validates the config without deploying Correct answer.

    Explanation

    Validation parses databricks.yml, resolves variables and targets, and checks for configuration errors without changing workspace state. It is commonly run as an early gate in CI before deployment.

  136. Question 136Implementing CI/CD

    Which identity is recommended for automated CI/CD deployments to Databricks?

    1. The workspace admin's personal access token
    2. A service principal Correct answer.
    3. A cluster owner's login
    4. A personal user account

    Explanation

    Service principals are non-human identities designed for automation, with scoped permissions and credentials that can be rotated independently of any employee. Using a person's account ties deployments to that individual and complicates auditing.

  137. Question 137Implementing CI/CD

    In a bundle with multiple targets, what does `databricks bundle deploy -t prod` do?

    1. Creates a new target called prod
    2. Deploys using the target named prod Correct answer.
    3. Deploys to the bundle's default target
    4. Validates the bundle without deploying

    Explanation

    The `-t` (or `--target`) flag selects a named target from databricks.yml, so the deploy uses that target's workspace host, variables, and resource overrides. Targets must already be declared in configuration.

  138. Question 138Implementing CI/CD

    What is the effect of the `--var` flag on a bundle command?

    1. Locks a variable to its default
    2. Overrides a variable value Correct answer.
    3. Deletes a bundle variable
    4. Prints all bundle variables

    Explanation

    `--var name=value` supplies or overrides a bundle variable at run time, letting CI pipelines inject environment-specific values without editing databricks.yml. Multiple --var flags can be passed in one command.

  139. Question 139Implementing CI/CD

    Which Databricks CLI command updates a Git folder to the latest commit of its remote branch?

    1. databricks bundle sync
    2. databricks workspace sync
    3. databricks git pull
    4. databricks repos update Correct answer.

    Explanation

    `databricks repos update` pulls the latest changes for a Git folder into the workspace, optionally to a specified branch or tag. Pipelines use it to refresh a deployment checkout to a desired revision.

  140. Question 140Implementing CI/CD

    How are Databricks jobs typically declared in an Asset Bundle?

    1. As YAML resource definitions Correct answer.
    2. Through SQL DDL statements
    3. In a separate JSON file only
    4. Only through the workspace UI

    Explanation

    Jobs, pipelines, and other objects are declared under the resources section of databricks.yml or in additional YAML files included via the include directive. The CLI translates these declarations into workspace API calls during deploy.

  141. Question 141Implementing CI/CD

    Which approach best promotes a data pipeline from staging to production?

    1. Copy notebooks by hand into production
    2. Manually recreate resources in production
    3. Promote one versioned artifact across stages Correct answer.
    4. Rebuild the artifact separately in each environment

    Explanation

    Building once and promoting the same versioned artifact ensures production runs exactly the code that was tested in earlier stages. Rebuilding or hand-copying introduces drift and untested differences between environments.

  142. Question 142Implementing CI/CD

    What does the `permissions` block in a bundle resource definition control?

    1. Resource access levels Correct answer.
    2. Job timeout values
    3. Cluster autoscaling limits
    4. Git branch protection rules

    Explanation

    The permissions block assigns access levels such as CAN_VIEW or CAN_MANAGE to users, groups, or service principals for deployed resources. It is applied during deploy so access is consistent across environments.

  143. Question 143Implementing CI/CD

    Which practice gives fast, reliable unit tests of PySpark logic in a CI pipeline?

    1. Skip tests in CI to save time
    2. Run pytest on extracted functions Correct answer.
    3. Execute every notebook in production
    4. Rely on manual code review only

    Explanation

    Moving transformation logic into importable Python functions lets pytest exercise them with a local Spark session or Databricks Connect in CI. This is faster and more deterministic than running whole notebooks against production data.

  144. Question 144Implementing CI/CD

    Why are cluster policies useful in a CI/CD deployment strategy?

    1. They encrypt notebooks at rest
    2. They automatically merge Git branches
    3. They constrain cluster configuration Correct answer.
    4. They replace Unity Catalog permissions

    Explanation

    Cluster policies limit which cluster attributes users and jobs can set, so deployed workloads meet cost and governance standards. Bundles can reference a policy ID to ensure jobs launch with approved settings.

  145. Question 145Implementing CI/CD

    Which environment variables are commonly used by the Databricks CLI to authenticate in a GitHub Actions job?

    1. GIT_TOKEN and GIT_HOST variables
    2. DBX_USER and DBX_PASSWORD variables
    3. DATABRICKS_HOST, DATABRICKS_TOKEN Correct answer.
    4. WORKSPACE_URL and API_KEY variables

    Explanation

    The Databricks CLI reads DATABRICKS_HOST and DATABRICKS_TOKEN, or OAuth-related variables, for authentication. CI systems inject these values from their secret stores before running bundle commands.

  146. Question 146Implementing CI/CD

    What is a primary use of the Databricks Terraform provider in CI/CD?

    1. Declarative infrastructure as code Correct answer.
    2. Generating billing reports
    3. Editing notebooks in the web browser
    4. Optimizing SQL query plans

    Explanation

    The Terraform provider manages Databricks resources such as clusters, jobs, and Unity Catalog objects through declarative infrastructure-as-code. It complements bundles when an organization standardizes on Terraform for provisioning.

  147. Question 147Implementing CI/CD

    A platform team wants to create service principals for automated deployments across multiple workspaces. Which API level must they use?

    1. Jobs API with a cluster-scoped token
    2. SQL warehouse API with a personal token
    3. Account-level API with an account admin Correct answer.
    4. Workspace-level API with any workspace user

    Explanation

    Service principals are account-level identities, so creating and managing them requires the account API and an account admin or a principal with the appropriate account-level entitlement. Workspace APIs manage workspace objects rather than account identities.

  148. Question 148Implementing CI/CD

    Which credential type is used for OAuth machine-to-machine authentication from a CI pipeline to Databricks?

    1. A notebook-scoped temporary token
    2. A client ID and client secret Correct answer.
    3. A browser login performed on each pipeline run
    4. A cluster init script credential

    Explanation

    OAuth M2M uses a service principal's client ID and client secret to obtain short-lived access tokens from the OAuth token endpoint. The pipeline exchanges these credentials for a token instead of relying on a long-lived personal access token.

  149. Question 149Implementing CI/CD

    What does `databricks bundle destroy` do when run against a target?

    1. Clears all historical job run data
    2. Removes local bundle files
    3. Removes resources for the target Correct answer.
    4. Deletes the remote Git repository

    Explanation

    Destroy tears down the workspace resources that the bundle previously deployed to the selected target, such as jobs and pipelines. It is commonly used to clean up ephemeral test environments rather than to alter source control or local files.

  150. Question 150Implementing CI/CD

    Why is idempotency important for bundle deployments in a CI/CD pipeline?

    1. Runs are repeatable and converge Correct answer.
    2. It blocks automatic Git merges
    3. It guarantees zero-downtime deploys
    4. It skips all permission checks

    Explanation

    Idempotent deployment means applying the same bundle definition repeatedly produces the same end state without duplicating resources. This lets pipelines retry failed runs safely and keeps environments consistent with source control.

  151. Question 151Troubleshooting, Monitoring, and Optimization

    A Spark job in the Spark UI shows twelve separate stages. What creates the boundary between consecutive stages?

    1. A change in the number of executors attached to the cluster during the job, which splits execution
    2. A commit of the Delta transaction log between two actions on the same table, which splits the job
    3. A garbage collection pause on the driver long enough for Spark to checkpoint the plan
    4. A shuffle boundary where data must be redistributed across partitions, such as a wide transformation Correct answer.

    Explanation

    Spark starts a new stage at every shuffle boundary, where data must be redistributed across partitions (for example a join, groupBy or repartition). Narrow transformations such as filter and select are pipelined within a single stage and do not create new ones.

  152. Question 152Troubleshooting, Monitoring, and Optimization

    A table has not had VACUUM run with a custom retention. What is the default retention threshold that VACUUM uses before deleting unreferenced data files?

    1. 7 days since the file was last referenced by a commit Correct answer.
    2. 30 days, matching the default retention interval of the Delta transaction log
    3. Until the table is dropped, because VACUUM only runs when explicitly enabled
    4. 0 hours, so VACUUM removes every unreferenced file immediately after the write

    Explanation

    VACUUM removes data files that are no longer referenced and that are older than the retention threshold, which defaults to 7 days. The threshold exists to protect readers that may still be using an older snapshot of the table.

  153. Question 153Troubleshooting, Monitoring, and Optimization

    Which statement describes what OPTIMIZE with a Z-ORDER BY clause does on a Delta table?

    1. It rewrites the table as plain Parquet and deletes the Delta transaction log entirely.
    2. It deletes data files that are no longer referenced by the transaction log.
    3. It compacts small files and co-locates related rows to enable more file skipping. Correct answer.
    4. It recomputes column statistics without rewriting any data files.

    Explanation

    OPTIMIZE rewrites many small files into fewer larger ones, and Z-ORDER additionally sorts rows by the given columns within each file. The resulting min/max statistics let later queries skip more files.

  154. Question 154Troubleshooting, Monitoring, and Optimization

    Which storage level caches a DataFrame in memory only, as deserialized JVM objects, and recomputes partitions that do not fit?

    1. DISK_ONLY
    2. OFF_HEAP
    3. MEMORY_AND_DISK
    4. MEMORY_ONLY Correct answer.

    Explanation

    MEMORY_ONLY stores partitions as deserialized JVM objects in memory and drops partitions that do not fit, recomputing them when needed. MEMORY_AND_DISK is the level that spills the overflow to disk instead.

  155. Question 155Troubleshooting, Monitoring, and Optimization

    Where can you find the Spark driver log for a running cluster in a Databricks workspace?

    1. In the query history of the SQL warehouse that executed the statements
    2. In the Delta transaction log of the default database on that cluster
    3. In the workspace /Users folder, attached to the notebook that started the cluster
    4. In the Spark driver log, reachable from the cluster details page or the CLI/API Correct answer.

    Explanation

    Databricks exposes the Spark driver log (stdout, stderr and log4j output) from the cluster details page, and the same logs are reachable through the CLI or REST API. Executor logs are available through the same interface.

  156. Question 156Troubleshooting, Monitoring, and Optimization

    Which command returns one row per commit on a Delta table, with the version, timestamp, operation, user and operation metrics?

    1. DESCRIBE HISTORY Correct answer.
    2. ANALYZE TABLE
    3. SHOW TABLES
    4. EXPLAIN

    Explanation

    DESCRIBE HISTORY lists the commit log of a Delta table, one row per version, including who performed the operation and the metrics recorded for it. It is also the reference for time travel queries by version or timestamp.

  157. Question 157Troubleshooting, Monitoring, and Optimization

    A 40 MB dimension table is joined to a 5 TB fact table and the plan shows a SortMergeJoin. Which change is most likely to make Spark broadcast the small side?

    1. Lower spark.sql.files.maxPartitionBytes so each task reads less input
    2. Raise autoBroadcastJoinThreshold above 40 MB or add a broadcast hint Correct answer.
    3. Increase spark.sql.shuffle.partitions so the join runs with more tasks
    4. Disable adaptive query execution so the planner keeps the sort-merge join

    Explanation

    Spark broadcasts a side automatically only when its estimated size is below spark.sql.autoBroadcastJoinThreshold, which is 10 MB by default. Raising the threshold above the small table's size, or adding a broadcast hint, converts the sort-merge join into a broadcast hash join.

  158. Question 158Troubleshooting, Monitoring, and Optimization

    With adaptive query execution enabled, which feature splits oversized shuffle partitions caused by a skewed join key?

    1. Delta deletion vectors, which avoid rewriting whole data files
    2. Dynamic partition pruning, which eliminates partitions at runtime
    3. Adaptive skew join handling (spark.sql.adaptive.skewJoin) Correct answer.
    4. Speculative execution, which relaunches slow tasks on other executors

    Explanation

    When adaptive query execution is on, skew join handling compares shuffle partition sizes against a threshold derived from the median and splits the oversized partitions into smaller sub-partitions. It is controlled by spark.sql.adaptive.skewJoin.enabled.

  159. Question 159Troubleshooting, Monitoring, and Optimization

    In the Spark UI, a stage reports a large Spill (disk) value. What does this indicate?

    1. Delta data skipping excluded most files, so far less data was read than the plan expected
    2. The query result was materialized to the output table while the stage was running
    3. Cached blocks were evicted from memory and had to be recomputed from the source
    4. The stage could not hold shuffle or aggregation data in memory and wrote some to disk Correct answer.

    Explanation

    The spill metric counts bytes written to local disk when the in-memory structures used for shuffle, sort or aggregation exceed the memory available to a task. Large spill usually points to partitions that are too large, key skew, or insufficient executor memory.

  160. Question 160Troubleshooting, Monitoring, and Optimization

    A Delta table is partitioned by a date column, yet a query filtering on a single date scans every file in the table. Which cause is most likely?

    1. Table statistics were collected with ANALYZE TABLE after the most recent write
    2. The table has not been Z-ORDERed by the date column recently
    3. A function wraps the partition column in the filter, so pruning cannot occur Correct answer.
    4. The cluster is configured with autoscaling enabled

    Explanation

    Partition pruning applies only when the filter references the partition column directly. Wrapping the column in a function such as to_date() hides it from the optimizer, so the scan falls back to reading all partitions.

  161. Question 161Troubleshooting, Monitoring, and Optimization

    A Structured Streaming job appends data to a Delta table every few seconds, and read latency on the table keeps growing. Which change addresses the root cause?

    1. Increase the shuffle partition count used by the streaming query
    2. Run OPTIMIZE to compact files and lengthen the trigger interval Correct answer.
    3. Cache the target table in memory before each read
    4. Enable VACUUM with zero retention hours so old data files are removed

    Explanation

    Very frequent appends produce many tiny files, and per-file overhead dominates read time. Compaction with OPTIMIZE merges them into larger files, while a longer trigger interval reduces how often new files are created.

  162. Question 162Troubleshooting, Monitoring, and Optimization

    What does Photon accelerate on a Databricks cluster?

    1. SQL and DataFrame execution via a vectorized C++ engine Correct answer.
    2. Python UDF execution only, by compiling user functions to native code
    3. Delta transaction log commits only, by batching file additions together
    4. Unity Catalog metadata operations only, by caching grants in memory

    Explanation

    Photon is Databricks' vectorized query engine, written in C++, that executes SQL and DataFrame operators such as joins, aggregations and scans. It speeds up execution without changing query semantics.

  163. Question 163Troubleshooting, Monitoring, and Optimization

    Adaptive query execution is enabled and a query is configured with 2000 shuffle partitions, but the shuffle produces only 30 MB of data. What does Spark do at runtime?

    1. It removes the shuffle entirely and broadcasts both join sides
    2. It increases the shuffle partition count so each task handles less data
    3. It fails the query so the user can lower the partition count manually
    4. It coalesces small adjacent shuffle partitions into fewer partitions Correct answer.

    Explanation

    With adaptive query execution, Spark reads the actual shuffle statistics and merges adjacent small partitions into fewer, larger ones (spark.sql.adaptive.coalescePartitions.enabled). This avoids scheduling thousands of near-empty tasks.

  164. Question 164Troubleshooting, Monitoring, and Optimization

    Which column is the better candidate for Z-ORDER rather than for table partitioning?

    1. A low-cardinality region column with only four distinct values
    2. A high-cardinality customer_id used in most point lookups Correct answer.
    3. A date column with one distinct value for each of the last five years
    4. A column that is never referenced in any query filter

    Explanation

    Partitioning creates one directory per distinct value, so it suits low-cardinality columns. Z-ORDER improves file skipping by clustering values within files and is the better fit for high-cardinality columns that appear frequently in filters.

  165. Question 165Troubleshooting, Monitoring, and Optimization

    Which streaming progress measurements best show that a microbatch query is falling behind the arrival rate of its input?

    1. numInputRows, which counts the number of rows read in the most recent batch
    2. numOutputRows, which counts the rows written by the most recent batch
    3. stateOperators.numRowsUpdated, which counts the state rows changed by the batch
    4. batchDuration versus the trigger interval and processedRowsPerSecond Correct answer.

    Explanation

    If batchDuration approaches or exceeds the trigger interval while processedRowsPerSecond stays below inputRowsPerSecond, the query cannot keep up and the backlog grows. Together those metrics show throughput relative to arrival.

  166. Question 166Troubleshooting, Monitoring, and Optimization

    Which information does Delta data skipping use to decide that a data file can be ignored for a query?

    1. The query history of the SQL warehouse that executes the statement
    2. The retention period configured on the Delta table
    3. The number of clusters attached to the workspace when the query runs
    4. Per-file min/max statistics and null counts in the log Correct answer.

    Explanation

    For each data file, Delta records minimum and maximum values and null counts for the leading columns (32 by default). If the filter range cannot overlap a file's range, that file is skipped entirely.

  167. Question 167Troubleshooting, Monitoring, and Optimization

    Two tables are joined on a key where a single value accounts for 60 percent of the fact rows. Broadcast is not possible and adaptive skew handling alone is not enough. Which technique directly spreads that key's rows across partitions?

    1. Cache the large table in memory before perfoRming the join
    2. Salt the hot key on the large side and replicate those rows on the small side Correct answer.
    3. Increase spark.sql.shuffle.partitions to a very large value
    4. Set spark.sql.autoBroadcastJoinThreshold to -1 so broadcast is disabled

    Explanation

    Salting appends a random suffix to the hot key on the large side and duplicates the matching key across the salt values on the small side. Each salted value then lands in a different shuffle partition, so no single task processes the entire hot key.

  168. Question 168Troubleshooting, Monitoring, and Optimization

    A stage shows task time dominated by garbage collection, and executors spill even though the cluster has generous memory. Which adjustment most directly reduces GC pressure in this situation?

    1. Move the job to a driver node with more memory and more cores
    2. Cache less data per executor and use a serialized storage level Correct answer.
    3. Set spark.sql.autoBroadcastJoinThreshold to zero to avoid all broadcasts
    4. Enable VACUUM on the source table so fewer files are read per task

    Explanation

    Caching deserialized JVM objects increases garbage collection and competes with execution memory. Caching fewer partitions and switching to a serialized level such as MEMORY_AND_DISK_SER shrinks the heap footprint and frees memory for shuffles and sorts.

  169. Question 169Troubleshooting, Monitoring, and Optimization

    A large Delta table receives frequent small MERGE updates and read performance has degraded. Which change best improves both write and read performance?

    1. Disable optimized writes and rely on VACUUM to remove old files
    2. Replace the table with a view over the underlying Parquet files
    3. Add a high-cardinality partition column to spread the writes
    4. Enable deletion vectors and optimized writes for the MERGE Correct answer.

    Explanation

    Deletion vectors let MERGE mark rows as deleted without rewriting entire files, and optimized writes reduce the number of small files produced per operation. Together they cut write amplification and keep read performance stable.

  170. Question 170Troubleshooting, Monitoring, and Optimization

    A stateful Structured Streaming aggregation writes to a Delta table, and the state store grows with every batch until batch durations become unmanageable. What is the most likely cause?

    1. The query runs in append output mode instead of update mode
    2. No advancing watermark is defined, so state for every key is retained Correct answer.
    3. The state store is configured to live only in executor memory
    4. The checkpoint location is on DBFS rather than in cloud object storage

    Explanation

    Stateful aggregations keep state per key until a watermark lets Spark evict state for event times that are too old to matter. Without an event-time column and an advancing watermark, no state is ever evicted and the store grows without bound.

  171. Question 171Governance and Security

    In Unity Catalog, which namespace pattern uniquely identifies a table?

    1. catalog.schema.table Correct answer.
    2. schema.catalog.table
    3. metastore.catalog.schema
    4. workspace.catalog.table

    Explanation

    Unity Catalog organizes objects in a three-level hierarchy of catalog, schema, and object such as a table or view, referenced as catalog.schema.table.

  172. Question 172Governance and Security

    A principal holds USE CATALOG on a catalog. Which additional privilege is required to create a new schema inside it?

    1. MODIFY on the catalog
    2. USE SCHEMA on the catalog
    3. CREATE SCHEMA on the catalog Correct answer.
    4. SELECT on the catalog

    Explanation

    Creating a schema requires the CREATE SCHEMA privilege on the parent catalog, together with USE CATALOG so the catalog itself is visible to the principal.

  173. Question 173Governance and Security

    Which secret scope type stores secrets in an external Azure Key Vault rather than inside Databricks?

    1. Workspace-local scope
    2. Databricks-backed scope
    3. Key Vault-backed scope Correct answer.
    4. Replica backed scope

    Explanation

    A Key Vault-backed secret scope keeps the secret material in Azure Key Vault and surfaces it through Databricks, while a Databricks-backed scope stores secrets in an encrypted database owned by the workspace.

  174. Question 174Governance and Security

    Which cluster access mode is dedicated to a single user while still supporting Scala and RDD APIs with Unity Catalog enforcement?

    1. No isolation shared
    2. Single user Correct answer.
    3. Custom
    4. Shared

    Explanation

    Single user access mode assigns dedicated compute to one principal, supports Scala and RDD APIs, and still enforces Unity Catalog permissions.

  175. Question 175Governance and Security

    What does the owner of a Unity Catalog table automatically hold?

    1. Only the SELECT privilege on the table
    2. Only the MODIFY privilege on the table
    3. All privileges on the table Correct answer.
    4. No privileges at all until explicitly granted

    Explanation

    Ownership of a securable object in Unity Catalog confers every privilege on that object and the right to grant privileges on it to other principals.

  176. Question 176Governance and Security

    Which mechanism governs access to tables registered in the legacy Hive metastore rather than in Unity Catalog?

    1. Table access control lists Correct answer.
    2. Catalog-level GRANT statements
    3. Governed tags and policies
    4. Storage credential mappings

    Explanation

    Tables in the legacy Hive metastore are governed by table access control lists, whereas Unity Catalog objects are governed by GRANT statements on securables.

  177. Question 177Governance and Security

    How is table and column lineage obtained for tables registered in Unity Catalog?

    1. It is enabled per catalog by an admin
    2. It is collected automatically at query runtime Correct answer.
    3. It is captured only on single user clusters
    4. It is declared manually with ALTER TABLE

    Explanation

    Unity Catalog records lineage automatically from query execution, capturing table- and column-level relationships without any manual declaration.

  178. Question 178Governance and Security

    Which account-level feature automatically synchronizes users and groups from an identity provider into Databricks?

    1. SCIM provisioning Correct answer.
    2. Cluster policies
    3. Personal access tokens
    4. IP access lists

    Explanation

    SCIM provisioning connects the account to an identity provider so users and groups are created and kept in sync automatically.

  179. Question 179Governance and Security

    Which statement grants the analysts group permission to read rows from the table prod.sales.orders?

    1. GRANT USE ON TABLE prod.sales.orders TO analysts
    2. GRANT SELECT ON prod.sales.orders TO GROUP analysts
    3. GRANT READ ON TABLE prod.sales.orders TO analysts
    4. GRANT SELECT ON TABLE prod.sales.orders TO analysts Correct answer.

    Explanation

    Unity Catalog GRANT statements name the securable type and object, then the principal, for example GRANT SELECT ON TABLE catalog.schema.table TO principal.

  180. Question 180Governance and Security

    A user was granted SELECT on prod.sales.orders but queries of that table fail with a permission error. Which privileges are most likely missing?

    1. USE CATALOG on prod and USE SCHEMA on sales Correct answer.
    2. CREATE TABLE and MODIFY on the schema
    3. SELECT and MODIFY on the catalog
    4. EXECUTE and READ FILES on the schema

    Explanation

    Reading a table requires USE CATALOG on the containing catalog and USE SCHEMA on the containing schema so the object can be resolved, in addition to SELECT on the table.

  181. Question 181Governance and Security

    Which construct restricts the rows a user sees based on the identity running the query, without maintaining separate copies of the table?

    1. A cluster policy attached to the workspace
    2. A dynamic view using current_user() in its WHERE clause Correct answer.
    3. A storage credential scoped to a schema
    4. A table comment documenting the access rule

    Explanation

    A dynamic view evaluates identity functions such as current_user() at query time, so each caller sees only the rows the view definition permits.

  182. Question 182Governance and Security

    A user has SELECT on a Unity Catalog view but holds no privileges on the tables the view references. What happens when the user queries the view?

    1. The query fails until the user is granted access to the tables
    2. The query succeeds only if the user owns the schema
    3. The query succeeds but always returns no rows
    4. The query succeeds, running with the view owner's privileges Correct answer.

    Explanation

    Unity Catalog views execute with the owner's privileges, so principals granted SELECT on the view do not need privileges on the underlying tables.

  183. Question 183Governance and Security

    Compliance requires Social Security numbers in prod.customers to appear in full only to the hr group and in masked form to everyone else. Which Unity Catalog feature fits?

    1. A column mask function applied to the SSN column Correct answer.
    2. A read-only external location
    3. A row filter on the customers table
    4. An IP access list on the workspace

    Explanation

    A column mask applies a function to a column that returns a masked value unless the caller satisfies the condition, which fits per-group visibility of sensitive fields.

  184. Question 184Governance and Security

    Which pair of Unity Catalog objects controls access to cloud storage paths used for external tables?

    1. Secret scope and cluster policy
    2. Storage credential and external location Correct answer.
    3. Catalog and volume
    4. Metastore and workspace

    Explanation

    A storage credential encapsulates a cloud identity, and an external location pairs that credential with a storage path that Unity Catalog objects may reference.

  185. Question 185Governance and Security

    A Unity Catalog managed table is dropped. What happens to its underlying data files?

    1. They remain in storage until manually deleted
    2. They are copied into metastore storage
    3. They are deleted from managed storage Correct answer.
    4. They are moved to the workspace DBFS root

    Explanation

    Dropping a managed table removes both the metadata and the data files in the managed storage location, unlike an external table whose files remain in place.

  186. Question 186Governance and Security

    An admin wants to limit which cluster configurations users can create, such as enforcing autoscaling limits and required tags. Which feature enforces this?

    1. Secret scopes
    2. Governed tags
    3. Cluster policies Correct answer.
    4. IP access lists

    Explanation

    Cluster policies define which configuration attributes users may set, including enforced limits, tags, and permitted instance types.

  187. Question 187Governance and Security

    A user is granted SELECT on the schema sales in catalog prod. Which tables in that schema can the user read?

    1. All current and future tables in the schema Correct answer.
    2. No tables until table-level grants are added
    3. Only the tables listed at grant time
    4. Only the tables the user owns

    Explanation

    Privileges granted on a schema apply to all objects currently in that schema and to objects created there later.

  188. Question 188Governance and Security

    A scheduled production job must authenticate without depending on any employee's credentials. Which identity type should be used?

    1. A personal access token of the workspace admin
    2. A service principal with its own credentials Correct answer.
    3. An external location owned by the job's author
    4. A shared cluster scoped to all users

    Explanation

    A service principal is a non-human identity with its own credentials that can be granted Unity Catalog privileges and used by automated workloads.

  189. Question 189Governance and Security

    Which capability lets an organization supply its own cloud key management service key to encrypt managed services data at rest?

    1. Customer-managed keys Correct answer.
    2. IP access lists
    3. Secret scopes backed by Key Vault
    4. Storage credentials

    Explanation

    Customer-managed keys allow an organization to provide its own key from a cloud KMS to encrypt managed services data, including workspace storage and managed disks.

  190. Question 190Governance and Security

    What is the purpose of an IP access list in a Databricks account?

    1. Restrict workspace access by network origin Correct answer.
    2. Encrypt workspace data with customer keys
    3. Audit Unity Catalog privilege changes
    4. Automatically rotate personal access tokens

    Explanation

    IP access lists restrict which network origins may reach the workspace UI and APIs, and can also apply to the account console.

  191. Question 191Governance and Security

    An admin attempts to grant a Unity Catalog privilege to a workspace-local group and the statement fails. What is the cause?

    1. Unity Catalog grants require a service principal
    2. The group has no metastore assigned
    3. Only account-level groups can receive Unity Catalog privileges Correct answer.
    4. The group must own the catalog first

    Explanation

    Unity Catalog privileges can be granted only to account-level users, service principals, and account-level groups; workspace-local groups are not supported for grants.

  192. Question 192Governance and Security

    Which statement about Unity Catalog metastore attachment is correct?

    1. A workspace attaches to exactly one metastore in its region Correct answer.
    2. A workspace can attach to multiple metastores in the same region
    3. Each schema requires its own metastore
    4. A metastore can span multiple cloud regions

    Explanation

    A workspace is attached to a single Unity Catalog metastore, and a metastore is created in one region and serves workspaces in that region.

  193. Question 193Governance and Security

    A user must create an external table at a path covered by an external location. Beyond CREATE TABLE on the schema, which privilege is required?

    1. CREATE EXTERNAL TABLE on the location Correct answer.
    2. SELECT on the schema and MODIFY on the catalog
    3. USE on the metastore and READ FILES on the catalog
    4. Ownership of the storage credential

    Explanation

    Creating an external table requires the CREATE EXTERNAL TABLE privilege on the external location covering the path, in addition to USE CATALOG, USE SCHEMA, and CREATE TABLE on the schema.

  194. Question 194Governance and Security

    Which statement reflects the security restrictions of the Shared cluster access mode?

    1. It disallows Scala and access to the DBFS root Correct answer.
    2. It requires a dedicated cluster for every user
    3. It allows only R users to connect
    4. It disables Unity Catalog enforcement

    Explanation

    Shared access mode supports multiple users with Unity Catalog isolation but blocks Scala, RDD APIs, and access to the DBFS root to prevent data exfiltration.

  195. Question 195Governance and Security

    A row filter function must check whether the current caller belongs to the account-level group finance. Which function should the filter use?

    1. has_tag_value('finance')
    2. current_database()
    3. is_member('finance')
    4. is_account_group_member('finance') Correct answer.

    Explanation

    is_account_group_member() tests membership in account-level groups, which are the group type Unity Catalog grants and row filters rely on.

  196. Question 196Governance and Security

    A data engineer needs a group of analysts to be able to read the values of secrets stored in a Databricks-backed secret scope, but must not be able to add, delete, or change the scope's configuration. Which secret scope permission should be granted to the group?

    1. READ Correct answer.
    2. USE
    3. MANAGE
    4. WRITE

    Explanation

    Secret scopes use their own ACL model with three permissions: MANAGE, WRITE, and READ. READ lets a principal list scopes they can access and read secret values, while WRITE additionally allows creating and deleting secrets, and MANAGE allows administering the scope. Granting READ is therefore the least-privilege choice for consumption only.

  197. Question 197Governance and Security

    An administrator has already granted a user the USE CATALOG privilege on the Unity Catalog catalog named prod. Which additional privilege must be granted on that same catalog for the user to create new schemas inside it?

    1. CREATE EXTERNAL LOCATION
    2. CREATE TABLE
    3. CREATE SCHEMA Correct answer.
    4. USE SCHEMA

    Explanation

    Schema creation is controlled by the CREATE SCHEMA privilege at the catalog level, and USE CATALOG is separately required to work within the catalog. Table-level privileges such as CREATE TABLE apply to objects inside an existing schema, not to creating the schema itself. USE SCHEMA is a schema-level privilege and cannot be granted on a catalog.

  198. Question 198Governance and Security

    An auditor asks which users queried the Unity Catalog table prod.sales.orders during the past week, and from which workspaces. Which source should the engineer query to answer this?

    1. The system.access.audit system table Correct answer.
    2. The information_schema.column_tags view for the table
    3. The table's Delta transaction log at the storage location
    4. The driver logs of the clusters that ran the queries

    Explanation

    Unity Catalog audit events, including data access events that record who queried which table, are recorded in the account-level audit log and exposed through the system.access.audit system table. Delta logs capture table operations but not the identity of the querying user or workspace. Column tags describe classification metadata, not access activity.

  199. Question 199Governance and Security

    A nightly production job runs as a service principal and its code uses Spark RDD APIs together with Scala and Python UDFs. Which cluster access mode should the engineer select for this job?

    1. Shared access mode with table access control enabled
    2. Custom access mode configured through a cluster policy
    3. Shared access mode, since the run-as identity is a service principal
    4. Dedicated access mode assigned to the service principal Correct answer.

    Explanation

    Shared access mode restricts workloads that are not supported in a multi-tenant environment, including RDD APIs and certain language constructs, so those workloads require dedicated (single-user) access mode. A dedicated cluster is assigned to exactly one user or service principal, and service principals are supported as that single user. Custom is not an access mode; it describes cluster configuration freedom.

  200. Question 200Governance and Security

    A platform team must be able to grant and revoke SELECT on any table inside the catalog prod for other users, but must not itself be able to read the table data. Which privilege on the catalog prod satisfies this requirement?

    1. MANAGE Correct answer.
    2. OWNERSHIP
    3. USE CATALOG
    4. SELECT WITH GRANT OPTION

    Explanation

    In Unity Catalog, MANAGE on a catalog permits granting and revoking privileges on that catalog and its child securables, including schemas and tables, without conferring the ability to read data. USE CATALOG only allows navigation and object reference. Unity Catalog has no per-privilege grant option syntax, and OWNERSHIP would convey full control over the catalog.

Databricks Data Engineer flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Unity Catalog three-level namespace
Catalogs contain schemas; schemas contain tables, views, volumes, functions, and models. Queries reference objects as catalog.schema.object, and privileges are granted at each level.
Control plane vs data plane
The Databricks control plane manages notebooks, jobs, and cluster configurations. The data plane is the cloud storage and compute in your own account where table data lives.
Cluster access modes
Shared mode allows multiple users with Python and SQL but restricts Scala and RDD APIs. Single-user mode gives one principal full language support; custom mode offers flexible isolation options.
Photon engine
A vectorized C++ execution engine available on Databricks clusters and SQL warehouses that accelerates SQL and DataFrame workloads against Delta tables.
Delta Lake
An open table format that adds ACID transactions, schema enforcement, time travel, and efficient upserts to Parquet files stored in cloud object storage.
COPY INTO idempotency
COPY INTO records the files it has already loaded and skips them on subsequent runs, so re-running the same statement over a directory does not duplicate rows.
Auto Loader cloudFiles
Incremental file ingestion that detects new files via directory listing or cloud file notifications. It needs a schema location for tracking and a checkpoint directory for stream state.
availableNow trigger
An Auto Loader trigger option that processes every file new since the last checkpoint in a single micro-batch and then stops, suitable for scheduled batch ingestion.
Lakehouse Federation
Unity Catalog foreign catalogs that let you query external databases such as PostgreSQL, MySQL, SQL Server, or Snowflake in place without copying data into Delta tables.
MERGE INTO upsert
A Delta Lake statement that matches source rows to target rows on a join key and performs updates, inserts, or deletes atomically in one transaction.
Medallion architecture
A layered data design where bronze stores raw ingested data, silver stores cleaned and conformed tables, and gold stores business-level aggregates used for reporting.
ZORDER BY
A Delta OPTIMIZE clause that colocates rows with similar values in the specified columns into the same files, improving data skipping for filtered queries.
Liquid clustering
A Delta table layout defined with CLUSTER BY columns that replaces fixed partitioning and ZORDER; clustering keys can be changed later without rewriting existing data.
AUTO CDC APPLY CHANGES
A Lakeflow pipeline operation that applies a change data capture feed to a target table, supporting SCD type 1 upserts and type 2 history with a sequencing column.
Expectations in Lakeflow
Data quality constraints declared on pipeline datasets using expect, expect_or_drop, expect_or_fail, or expect_all_or_quarantine; metrics are recorded on each pipeline update.
run_if ALL_DONE
A task dependency condition that runs the downstream task after all upstream tasks finish, whether those upstream tasks succeeded or failed.
Repair run
A job run option that re-executes only failed or skipped tasks, reusing the original run's parameters and the outputs of tasks that already succeeded.
For each task
A job task type that runs a nested task once per element of a JSON array input, with a configurable maximum concurrency.
Job cluster
An ephemeral cluster created for a single job run and terminated when it finishes; typically cheaper than an all-purpose cluster for scheduled workloads.
Dynamic value reference
Templated job syntax such as {{job.parameters.name}} or {{tasks.task.values.key}} that Databricks resolves at runtime from job or task context.
Databricks Asset Bundle
A declarative YAML project (databricks.yml) that defines jobs, pipelines, and permissions so the same resources deploy consistently to dev, staging, and prod targets.
bundle deploy command
The CLI command databricks bundle deploy -t <target> that validates the bundle and uploads its resources to the workspace defined by the selected target.
OAuth machine-to-machine
Service principal authentication using a client ID and secret to obtain OAuth tokens for automated pipelines, avoiding personal access tokens tied to an individual.
Secret scope
A named collection of secrets backed by Databricks or Azure Key Vault, read in code with dbutils.secrets.get(scope, key) so credentials never appear in notebooks.
Adaptive Query Execution
A Spark setting that re-optimizes shuffle partition counts, join strategies, and skew handling at runtime using statistics gathered after each shuffle stage.
Task skew symptom
In the Spark UI, one task's duration or shuffle read far exceeds the median, indicating uneven key distribution that may need salting or repartitioning.
Broadcast join hint
A SQL hint, /*+ BROADCAST(table) */, that replicates a small table to every executor so the join avoids shuffling the large table.
Delta auto compaction
A Delta table property, delta.autoOptimize.optimizeWrite, that coalesces small files written during a transaction to reduce small-file overhead.
Row filter
A Unity Catalog function attached to a table that returns a boolean per row, restricting which rows each querying user is allowed to see.
Column mask
A Unity Catalog function attached to a column that transforms values at query time, for example hashing, nulling, or partially showing sensitive data.
External location
A Unity Catalog object that pairs a cloud storage path with a storage credential, enabling creation of external tables and volumes at that path.
system.access.audit
A system table recording Unity Catalog and workspace audit events, including who accessed which object or ran which query and when.

Databricks Data Engineer study guide

Databricks Intelligence Platform

Architecture

Every Databricks deployment splits into a control plane (Databricks-managed: notebook storage, job scheduler, cluster manager, Unity Catalog metadata, web UI) and a data plane (your cloud account for classic compute, or Databricks-managed for serverless). Data stays in your cloud storage; only metadata and control traffic cross into the control plane. Questions love to ask where notebooks live (control plane) versus where Spark drivers and executors run (data plane, in your VPC).

Compute choices

  • All-purpose clusters: interactive, shared by people, terminated manually - expensive if left running.
  • Job clusters: created for a run, terminated at completion; cheapest for scheduled work.
  • SQL warehouses (classic / pro / serverless): for SQL and BI; serverless scales instantly.
  • Serverless compute for notebooks and Lakeflow jobs: no cluster configuration, fastest startup, good for short spiky work.

Photon is a vectorized C++ engine that accelerates SQL and DataFrame operators on Delta tables. Trap: Photon does not speed up Python UDFs, which execute in the Python worker and often force a fallback.

Rule of thumb

Match compute to workload lifetime: if it runs on a schedule and nobody interacts with it, use a job cluster (or serverless); if an analyst clicks Run all all day, use an all-purpose cluster. Autoscaling adjusts worker count within min/max - it never scales to zero, and there is always exactly one driver.

Data Ingestion and Loading

Two ingestion paths

Auto Loader (spark.readStream.format('cloudFiles')) incrementally discovers new files using directory listing (cheap at small scale) or file notification via SNS/SQS or Event Grid (efficient at large scale). It infers and stores schema in a checkpoint, and evolves it via cloudFiles.schemaEvolutionMode (addNewColumns default, rescue, failOnNewColumns, none) plus schemaHints to pin types. A rescuedDataColumn retains rows that do not match the schema instead of dropping them. Auto Loader is append-only: it never updates or deletes rows.

COPY INTO is SQL-based and idempotent - it tracks which files were already loaded, making reruns safe. It works well for thousands of files but struggles with millions of files and with frequent schema drift.

Loading patterns

Bronze is a raw append; silver uses MERGE INTO for upserts and CDC; gold holds aggregates. MERGE fails when the source has more than one row matching a target row - deduplicate with ROW_NUMBER() OVER (PARTITION BY key ORDER BY updated_at DESC) first. JDBC reads parallelize with numPartitions, partitionColumn, lowerBound, upperBound.

Traps

  • Assuming COPY INTO re-reads files each run; it skips already-loaded files unless forced.
  • Assuming Auto Loader handles updates or deletes.
  • Forgetting that the checkpoint and schema location must be a UC volume the job identity can write to.

Rule of thumb

Growing, continuously arriving, schema-drifting files → Auto Loader with the availableNow trigger. A bounded one-time load of a few thousand files → COPY INTO.

Data Transformation and Modeling

Medallion and Delta fundamentals

Bronze preserves raw data, silver holds conformed and deduplicated entities, gold holds business aggregates. Delta Lake provides ACID transactions, schema enforcement (with mergeSchema opt-in), time travel, MERGE, deletion vectors, and liquid clustering. Column statistics enable data skipping; OPTIMIZE compacts small files; Z-ORDER co-locates related values; VACUUM removes unreferenced files (default 7-day retention, and time travel past that horizon stops working).

Modeling moves

  • Deduplicate before merging using ROW_NUMBER() over the business key ordered by ingestion timestamp.
  • SCD Type 1 overwrites attributes; SCD Type 2 inserts a new version and closes the prior row with valid_to and is_current. In pipelines use APPLY CHANGES INTO with a sequencing column.
  • Partition only large tables (roughly >1 TB) on low-cardinality columns; prefer liquid clustering over partitioning plus Z-ORDER for new tables because clustering keys can change without rewriting history.
  • Never partition on high-cardinality columns - you get small files and slow listing.

Traps

  • MERGE erroring on duplicate source keys; fix the source, not the merge.
  • Using INSERT OVERWRITE where late-arriving data must be preserved.
  • Writing many tiny files from a streaming job without compaction.
  • Assuming VACUUM deletes rows from history - it deletes files, and older queries then fail.

Worked example

To keep a silver customers table current: stage the batch, dedupe by customer_id, then MERGE on customer_id with a matched-update branch setting updated_at = current_timestamp() and a not-matched insert branch, followed by a nightly OPTIMIZE.

Working with Lakeflow Jobs

Anatomy of a job

A Lakeflow Job is a DAG of tasks. Task types include notebook, Python script or wheel, SQL query or file, pipeline (Lakeflow / Delta Live Tables), dbt, For each (loop over an array) and If/else (conditional branch). Dependencies define ordering; independent tasks run in parallel. Each task can set retries, timeouts, and parameters.

Compute and triggers

Job compute options: a dedicated job cluster created per run and terminated afterward (never reused between runs), serverless compute, or an existing SQL warehouse for SQL tasks. Triggers include scheduled (cron with timezone), continuous, file arrival (requires a Unity Catalog external location or volume), and table update. max_concurrent_runs and queueing control overlapping schedules.

Parameters

Job-level parameters reach notebooks as widgets (dbutils.widgets.get('k')) and Python scripts as argparse arguments. Task-level parameters override job-level values for that task only; dynamic references such as {{job.parameters.x}} and {{tasks.y.values.z}} pass values between tasks.

Repair and recovery

A repair run re-executes only failed tasks (or failed plus downstream) and preserves successful task outputs. It does not re-run upstream tasks that already succeeded, and it will not pick up code changes in tasks it does not re-run. If upstream logic changed, do a full run.

Traps

  • Expecting a job cluster to stay warm for interactive use - it terminates.
  • Assuming file-arrival triggers work on plain DBFS paths; they need UC volumes or external locations.
  • Confusing widget defaults with job parameters - a missing parameter silently falls back to the default.

Implementing CI/CD

Bundles are the unit of deployment

Databricks Asset Bundles (DABs) describe jobs, pipelines, notebooks, and clusters as code in databricks.yml. The file contains a bundle name, variables, resources (jobs, pipelines, and so on), and one or more targets (dev, staging, prod), each specifying a workspace host and overrides. Commands:

  • databricks bundle validate - schema and reference checks, changes nothing.
  • databricks bundle deploy -t prod - uploads files and creates or updates resources in the target workspace, storing state under .databricks/bundle/.
  • databricks bundle run <resource> -t prod - executes the deployed resource (deploying first if needed).
  • databricks bundle destroy - removes deployed resources.

Override variables with --var key=value or per-target variables blocks. Resource paths are relative to the bundle root, so notebook_path: ./src/ingest.py resolves inside the bundle.

CI/CD flow

Commit → CI runs bundle validate plus unit tests → deploy to the dev target → integration tests → promote the same commit by deploying to the prod target. Authenticate CI with a service principal using OAuth machine-to-machine credentials (or a PAT held in CI secrets); never commit tokens. Use Git folders for interactive development and separate workspaces per environment so production is unreachable from development.

Traps

  • Thinking deploy runs the job - it only deploys; run executes.
  • Hard-coding hosts and catalog names instead of variables, which breaks promotion.
  • Using a personal user's token in CI, so the job owner becomes a person who may leave the team.
  • Sharing one workspace across dev and prod with no target isolation.

Troubleshooting, Monitoring, and Optimization

Where to look

Spark UI / query profile: stages → tasks. Look for shuffle read/write volume, spill (memory versus disk), and task duration skew (max versus median). Delta history (DESCRIBE HISTORY) exposes operation metrics such as numFiles, numOutputRows, and whether OPTIMIZE ran. System tables (system.billing.usage, system.compute.*, system.access.audit, system.query.history) show cost and usage trends. Driver and cluster logs are delivered to a UC volume or DBFS destination you configure.

Optimization levers

  • AQE (on by default) coalesces shuffle partitions, splits skewed partitions, and can switch join strategy.
  • Broadcast join the small side via hint or autoBroadcastJoinThreshold to eliminate a shuffle.
  • Skew hints for known hot keys that AQE cannot split.
  • Cache only DataFrames reused many times; unpersist when finished.
  • Compaction: OPTIMIZE for small files, liquid clustering instead of partitioning for new tables, and predictive optimization on UC managed tables.
  • Photon accelerates native scans, joins, and aggregates, not Python UDFs - rewriting a Python UDF as native functions usually beats resizing the cluster.

Traps

  • Reading spill as a disk failure; it signals executor memory pressure.
  • Expecting AQE to fix non-shuffle skew from one giant input file.
  • Assuming a larger cluster fixes bad join order or missing predicate pushdown.
  • Running VACUUM with retention shorter than the longest reader's time-travel horizon.

Rule of thumb

When a stage is slow, compare slowest task to median. One dominating task means skew; all tasks slow with spill means memory or partition sizing; huge shuffle bytes means a join-strategy problem.

Governance and Security

The Unity Catalog model

The namespace has three levels: catalog.schema.table, plus volumes and functions. One metastore per region attaches to workspaces. Privileges are granted on securables: USE CATALOG, USE SCHEMA, SELECT, MODIFY, CREATE TABLE, ALL PRIVILEGES. Reading a table requires USE CATALOG on the catalog, USE SCHEMA on the schema, and SELECT on the table - a classic exam trap. Ownership implies all current and future privileges plus the right to grant them.

Data objects

  • Managed tables: Unity Catalog owns the storage location; DROP TABLE deletes the data.
  • External tables: point at a path; dropping removes metadata only.
  • External locations require a storage credential (cloud IAM role) plus CREATE EXTERNAL LOCATION; users also need READ FILES / WRITE FILES on the location.
  • Volumes provide governed file access without DBFS mounts.

Fine-grained controls

Row filters and column masks are SQL UDFs bound to a table, so callers see only permitted rows or masked values. Tags and comments drive discovery and cost attribution. Audit logs land in system.access.audit. Delta Sharing shares data outside the account without copying it.

Cluster access modes

Single user (one principal, all Spark APIs), Shared (many users, UC-only, no DBFS root, limited RDD/MLlib APIs), and Custom (jobs, service principals). Secrets belong in secret scopes, never in notebook code or widget defaults.

Traps

  • Granting SELECT and expecting access without USE CATALOG and USE SCHEMA.
  • Assuming dropping a managed table keeps its files.
  • Writing PII to a shared cluster's local storage path.
  • Granting ALL PRIVILEGES on a catalog when only one schema was intended.

Databricks Data Engineer cheat sheet

Quick reference

Databricks Intelligence Platform

  • Unity Catalog uses a three-level namespace: catalog.schema.table, plus volumes, functions, and models registered at the schema level.

  • Workspace assets (notebooks, jobs, repos) are managed in the Databricks control plane; table data resides in cloud storage in the customer's data plane.

  • Compute selection rule: all-purpose clusters for interactive and ad hoc work, job clusters for scheduled runs, SQL warehouses for SQL/BI, serverless for elastic managed compute.

  • Cluster access modes: shared (multi-user, Python and SQL, no Scala/RDD), single user (full language support for one principal), and custom.

  • Magic commands: %sql, %python, %scala, %r, %md, %sh, %pip, %run; %pip installs notebook-scoped libraries and requires a Python restart to take effect.

  • Delta Lake adds ACID transactions, schema enforcement, time travel, and MERGE support on top of Parquet files in cloud object storage.

  • Photon is a vectorized C++ query engine that accelerates SQL and DataFrame operations on Delta tables and is enabled per cluster or SQL warehouse.

  • Lakeflow is the data engineering suite combining ingestion (Auto Loader, Lakehouse Federation), declarative pipelines, and Lakeflow Jobs.

  • A Unity Catalog metastore is regional; one metastore can be assigned to multiple workspaces in the same region, sharing catalogs and privileges.

Quick reference

Data Ingestion and Loading

  • COPY INTO is idempotent: it tracks files already loaded and skips them on re-execution, so re-running over the same directory does not duplicate rows.

  • Auto Loader (cloudFiles) incrementally discovers new files via directory listing or file notification mode and requires a schemaLocation plus a checkpoint location.

  • cloudFiles.format supports json, csv, parquet, avro, orc, text, and binaryFile; binaryFile exposes path, modificationTime, and length metadata columns.

  • Trigger availableNow processes all files new since the last checkpoint in one micro-batch and then stops; trigger once is deprecated in favor of availableNow.

  • read_files(path, format => 'json') is the SQL function for reading files directly in SQL and in declarative pipeline queries.

  • Schema evolution modes for Auto Loader: addNewColumns (default), rescue, failOnNewColumns, and none; rescued fields land in the _rescued_data column.

  • Lakehouse Federation queries external databases (PostgreSQL, MySQL, SQL Server, Snowflake, Redshift) through foreign catalogs without copying data into Delta.

  • Streaming sources include Kafka, Kinesis, Event Hubs, Pulsar, and cloud file sources; each requires a checkpointLocation to track progress.

  • MERGE INTO target USING source ON key WHEN MATCHED THEN UPDATE / WHEN NOT MATCHED THEN INSERT is the standard pattern for upserts and deduplication.

Quick reference

Data Transformation and Modeling

  • Medallion layers: bronze = raw ingested data, silver = cleaned and conformed entities, gold = business-level aggregates for reporting and BI.

  • OPTIMIZE compacts small files; ZORDER BY colocates rows with similar values so queries skip files; liquid clustering (CLUSTER BY) replaces partitioning plus ZORDER on new tables.

  • VACUUM deletes unreferenced files older than the retention threshold (default 7 days); time travel beyond that window fails unless files were retained.

  • Unity Catalog enforces NOT NULL and CHECK constraints; PRIMARY KEY and FOREIGN KEY constraints are informational and not enforced.

  • APPLY CHANGES INTO (AUTO CDC) in Lakeflow pipelines implements SCD type 1 via upsert and SCD type 2 via history tracking with a sequencing column.

  • Window functions: ROW_NUMBER over a partition for deduplication, LAG/LEAD for prior and next values, SUM OVER for running totals.

  • from_json with an explicit schema plus explode converts nested JSON arrays into rows; schema_of_json derives a schema from a sample literal.

  • Materialized views recompute on refresh, streaming tables append incrementally, and both are defined inside declarative Lakeflow pipelines rather than as standalone DDL.

  • Pipeline expectations: @dp.expect records metrics, expect_or_drop discards bad rows, expect_or_fail halts the update, expect_all_or_quarantine routes bad rows to a quarantine table.

Quick reference

Working with Lakeflow Jobs

  • Task types include notebook, Python script, Python wheel, JAR, SQL, pipeline, dbt, Run Job, and For each.

  • run_if values: ALL_SUCCESS (default), AT_LEAST_ONE_SUCCESS, ALL_DONE, AT_LEAST_ONE_FAILED, ALL_FAILED.

  • Dynamic value references: {{job.parameters.name}}, {{job.id}}, {{tasks.task_name.values.key}} resolve at runtime from job or task context.

  • Trigger options: scheduled (Quartz cron), continuous, file arrival, table update, and manual.

  • Repair run re-executes only failed or skipped tasks using the original run's parameters; a full rerun re-executes every task from scratch.

  • For each tasks iterate a nested task over a JSON array input with a configurable concurrency limit.

  • Job clusters are created for a run and terminated at completion; all-purpose clusters can be reused across runs at higher cost.

  • max_concurrent_runs caps parallel executions of a job; when queueing is enabled, extra runs wait instead of failing immediately.

  • Retries are set per task with retry_on_timeout; notifications support email, webhook, and PagerDuty at job and task level.

Quick reference

Implementing CI/CD

  • Databricks Asset Bundles are defined in databricks.yml with bundle name, targets (dev/staging/prod), workspace paths, and resources such as jobs and pipelines.

  • Bundle lifecycle commands: databricks bundle validate, databricks bundle deploy -t , databricks bundle run -t , databricks bundle destroy.

  • Bundles support variables (${var.name}) and interpolation (${workspace.host}, ${bundle.target}); target overrides customize values per environment.

  • The Terraform databricks provider manages jobs, clusters, notebooks, and Unity Catalog objects as code alongside or instead of bundles.

  • CI authentication uses OAuth machine-to-machine with a service principal client ID and secret, or a token stored in a .databrickscfg profile.

  • Git folders (Repos) sync notebooks with a Git remote; jobs can also reference a Git provider source directly without copying files into the workspace.

  • Secrets live in secret scopes (Databricks-backed or Azure Key Vault-backed) and are read with dbutils.secrets.get(scope, key) rather than hardcoded.

  • Testing pattern: pytest unit tests against a local SparkSession for transformations, plus integration tests that deploy the bundle to a dev target.

Quick reference

Troubleshooting, Monitoring, and Optimization

  • Spark UI Stages tab exposes shuffle read/write, spill, and task duration distributions; a maximum task far above the median indicates data skew.

  • Adaptive Query Execution (spark.sql.adaptive.enabled=true) coalesces shuffle partitions, converts sort-merge joins to broadcast joins, and splits skewed partitions.

  • Broadcast hint /*+ BROADCAST(small_table) */ avoids shuffling the large side; the default auto-broadcast threshold is 10 MB (spark.sql.autoBroadcastJoinThreshold).

  • Query Profile in the SQL editor shows operator-level rows, bytes, and time; spills and exploding joins appear as the dominant cost nodes.

  • Target Delta file size is roughly 128 MB to 1 GB; delta.autoOptimize.optimizeWrite and OPTIMIZE reduce the small-file problem.

  • System tables for monitoring: system.billing.usage (cost), system.query.history (query text and duration), system.access.audit (access events).

  • repartition(n, col) performs a full shuffle to redistribute data; coalesce(n) reduces partition count without a shuffle and can create skew.

  • Cluster autoscaling sets min and max workers; Photon and the Delta cache accelerate repeated scans of the same Delta data.

Quick reference

Governance and Security

  • Unity Catalog hierarchy: metastore > catalog > schema > table/view/volume/function/model; privileges inherit down the hierarchy.

  • Privileges include USE CATALOG, USE SCHEMA, SELECT, MODIFY, CREATE TABLE, and ALL PRIVILEGES; the object owner has full control and can grant privileges.

  • Row filters and column masks are SQL UDFs attached to a table or column and applied at query time; users need EXECUTE on the function to read the object.

  • Storage credentials wrap an IAM role or managed identity; external locations combine a credential with a cloud path for external tables and volumes.

  • Managed tables store data under the catalog/schema managed location and are deleted with the table; external tables keep data after the table is dropped.

  • Audit events are queryable in system.access.audit; table and column lineage are in system.access.table_lineage and system.access.column_lineage.

  • Governed tags enable attribute-based access control so policies apply by tag (for example PII) rather than by individual object name.

  • Service principals with OAuth are preferred over personal access tokens for automation; cluster policies constrain which cluster configurations users may create.

Databricks Data Engineer key terms

Delta Lake

Open-source storage layer that adds a transaction log to Parquet files in cloud storage, providing ACID transactions, schema enforcement, time travel and efficient upserts on data lake tables.

Related: Delta table, OPTIMIZE, VACUUM
Delta table

A table whose data files and _delta_log transaction log live in cloud storage; the log records each commit so readers always see a consistent snapshot of the table.

Related: Delta Lake, Time travel
Time travel

Querying an earlier version of a Delta table by version number or timestamp, useful for auditing, reproducing past reports and recovering data after an incorrect write.

Related: Delta table, VACUUM
All-purpose cluster

Interactive compute that a user starts and keeps running for notebooks and ad hoc analysis; it can be shared by several users but costs more per DBU than job compute.

Related: Job cluster, SQL warehouse
SQL warehouse

Compute resource tuned for SQL and BI workloads, offered in serverless or classic sizes, used by dashboards, the SQL editor and clients connecting over JDBC or ODBC.

Related: All-purpose cluster, Lakeflow Jobs
Unity Catalog

Account-level governance layer that provides a shared metastore, three-level namespaces, centralized privileges, lineage and audit records across workspaces and clouds.

Related: Metastore, Three-level namespace, Unity Catalog privilege
Metastore

Top-level Unity Catalog container that holds catalogs and is assigned to workspaces; it defines the boundary inside which principals, storage credentials and grants are managed.

Related: Unity Catalog, External location
Three-level namespace

Naming pattern catalog.schema.table that uniquely identifies Unity Catalog objects and lets queries resolve tables unambiguously regardless of the workspace a user is working in.

Related: Unity Catalog, Metastore
External location

Unity Catalog object that pairs a cloud storage path with a storage credential, letting authorized users read and write external data without embedding secrets in code.

Related: Storage credential, Unity Catalog privilege
Storage credential

Unity Catalog object holding a cloud IAM role or managed identity that Databricks assumes to reach storage; external locations and external tables reference it.

Related: External location, Unity Catalog
Unity Catalog privilege

Permission such as USE CATALOG, SELECT or MODIFY granted on a securable object to a user, group or service principal; privileges are inherited from catalog down to table.

Related: Unity Catalog, Service principal
Row filter and column mask

Unity Catalog functions attached to a table that restrict which rows a caller sees or transform sensitive column values at query time, based on the caller's identity or group.

Related: Unity Catalog privilege, Unity Catalog
Service principal

Non-human identity used by automated jobs, pipelines and CI/CD tooling; it authenticates with OAuth client credentials or a secret and is granted privileges like a user.

Related: Unity Catalog privilege, Databricks CLI
Auto Loader

Structured Streaming source, configured as cloudFiles, that incrementally discovers and ingests newly arriving files from cloud storage, tracking processed files so each file is read once.

Related: Streaming checkpoint, Schema evolution
COPY INTO

SQL command that loads files from a storage location into a Delta table idempotently, recording which files were already loaded and skipping them on subsequent runs.

Related: Auto Loader, Delta table
Streaming checkpoint

Directory where a Structured Streaming query stores its progress, file offsets and state, allowing the query to resume from the correct position after a restart.

Related: Auto Loader, Streaming table
Schema evolution

Ingestion setting that lets a pipeline add newly appearing columns to the target automatically, or capture unexpected data in a rescued column instead of failing the load.

Related: Auto Loader, COPY INTO
Medallion architecture

Layered data design of bronze (raw ingested), silver (cleaned and conformed) and gold (business aggregates) tables, where each layer improves quality for downstream consumers.

Related: Streaming table, Lakeflow Declarative Pipelines
MERGE INTO

Delta SQL statement that inserts, updates or deletes target rows from a source based on a match condition, enabling idempotent upserts and slowly changing dimension loads.

Related: Delta table, APPLY CHANGES INTO
APPLY CHANGES INTO

Declarative pipeline command that applies a change data capture feed of inserts, updates and deletes to a target streaming table, using a sequencing column to resolve out-of-order records.

Related: Lakeflow Declarative Pipelines, Streaming table
Streaming table

Declarative pipeline object that processes each incoming record once and appends results incrementally, making it suitable for continuous ingestion and change data capture targets.

Related: Materialized view, Lakeflow Declarative Pipelines
Materialized view

Declarative pipeline object whose query result is recomputed on each refresh; used for aggregations and joins where incremental processing is not feasible.

Related: Streaming table, Lakeflow Declarative Pipelines
Lakeflow Declarative Pipelines

Framework in which you declare streaming tables, materialized views and expectations in code; the engine builds the dependency graph, manages compute and refreshes results incrementally.

Related: Streaming table, Expectations
Expectations

Data quality constraints declared inside a declarative pipeline that record metrics and can warn, drop affected rows, or fail the update when records violate the rule.

Related: Lakeflow Declarative Pipelines, Streaming table
Partitioning

Physically splitting table data into directories by column value, typically a date, so queries that filter on that column avoid reading unrelated data.

Related: Liquid clustering, OPTIMIZE
Liquid clustering

Delta feature that replaces fixed partitions and Z-order by selecting and maintaining clustering keys automatically as data is written and later optimized.

Related: Partitioning, OPTIMIZE
OPTIMIZE

Delta command that compacts many small files into fewer larger ones and can apply clustering, improving read throughput and the effectiveness of file skipping.

Related: VACUUM, Liquid clustering
VACUUM

Delta command that deletes data files no longer referenced by the transaction log and older than the retention threshold, reclaiming storage but removing older time travel history.

Related: OPTIMIZE, Time travel
Adaptive Query Execution

Spark optimization that re-plans a query at runtime using actual statistics, coalescing small shuffle partitions, splitting skewed partitions and switching join strategies.

Related: Data skew, Spark UI
Data skew

Uneven distribution of join or grouping keys so that a few tasks process most of the data, creating long stragglers that dominate a stage's duration.

Related: Adaptive Query Execution, Spark UI
Spark UI

Web interface for a running or finished Spark application showing stages, tasks, shuffle, spill and executor metrics used to diagnose slow or failing jobs.

Related: Data skew, Adaptive Query Execution
System tables

Account-level Unity Catalog tables such as system.billing.usage and system.access.audit that expose cost, audit and operational data for querying and monitoring.

Related: Unity Catalog, Spark UI
Lakeflow Jobs

Orchestration service for scheduling multi-task workflows that run notebooks, pipelines, SQL and other tasks with dependencies, triggers, retries and notifications.

Related: Job cluster, Job trigger, Repair run
Job cluster

Ephemeral compute created for a single Lakeflow job run and terminated when it ends; cheaper and isolated, and the recommended choice for scheduled production workloads.

Related: Lakeflow Jobs, All-purpose cluster
Job trigger

Rule that starts a job run: a cron schedule, arrival of files in a location, continuous processing, or a manual or API request.

Related: Lakeflow Jobs
Repair run

Rerunning only the failed and skipped tasks of a job run while reusing the results of tasks that already succeeded, saving time and compute cost.

Related: Lakeflow Jobs, Job cluster
Databricks Asset Bundles

YAML-defined package of jobs, pipelines, notebooks and permissions that can be validated and deployed with the CLI, giving version-controlled and repeatable deployments.

Related: Databricks CLI, Environment promotion
Databricks CLI

Command-line tool that authenticates to a workspace and runs bundle validate, deploy and run commands, typically invoked from CI/CD pipeline scripts.

Related: Databricks Asset Bundles, Service principal
Git folder

Workspace feature that clones a remote Git repository so notebooks and files are version controlled and developers can work on branches before merging changes.

Related: Databricks Asset Bundles
Environment promotion

Deploying the same bundle artifact from a development workspace to staging and production, using target-specific variables, permissions and service principals.

Related: Databricks Asset Bundles, Service principal

Databricks Data Engineer study plan

8 weeks · About 7 hours per week

Week 1

Data Ingestion and Loading

  • Configure Auto Loader to ingest JSON and CSV files from cloud storage into a bronze table with schema inference and schema evolution enabled

  • Implement a COPY INTO batch load with idempotent re-runs and validate that rerunning the same files produces no duplicate rows

  • Capture malformed records using the rescued data column and route them to a quarantine table

  • Choose between managed and external Unity Catalog tables for landing vs. curated layers and justify the choice in writing

Your checkpoint

A bronze ingestion pipeline (Auto Loader plus one COPY INTO path) that loads a sample file set twice, with a row-count report proving zero duplicates and a quarantine table listing rescued records

Week 2

Data Ingestion and Loading & Data Transformation and Modeling

  • Build a streaming pipeline that reads bronze change data and applies CDC merges into a silver table using AUTO CDC or APPLY CHANGES

  • Add data-quality expectations to the silver layer and demonstrate what happens on warn vs. drop vs. fail

  • Handle late-arriving and out-of-order records with watermarking and sequence columns

  • Parameterize the pipeline so the same code targets dev and prod catalogs without edits

Your checkpoint

An end-to-end bronze-to-silver streaming pipeline with CDC applied, expectation metrics captured, and a short document showing correct SCD behavior for at least one update and one delete

Week 3

Data Transformation and Modeling

  • Deduplicate source records using window functions or dropDuplicates and document the tie-breaking rule used

  • Model a gold star schema with one fact table and at least two conformed dimensions at a declared grain

  • Implement SCD Type 2 history for one dimension and query it as-of a past date

  • Lay out the medallion layers with clear naming, ownership, and column-comment conventions

Your checkpoint

A gold star schema (fact plus two dimensions, one versioned as SCD2) with a grain-validation query proving uniqueness of the fact primary key

Week 4

Data Transformation and Modeling & Troubleshooting, Monitoring, and Optimization

  • Read EXPLAIN and query profiles to identify shuffles, skewed joins, and full scans in your pipelines

  • Apply partition pruning, file compaction, and clustering keys to the largest tables you built

  • Replace an expensive join or UDF with a broadcast join or built-in function and measure the effect

  • Set up a repeatable benchmark harness that times the same query before and after each change

Your checkpoint

A tuning report for one pipeline showing before/after runtimes and bytes read for at least three optimizations, each tied to an EXPLAIN or profile excerpt

Week 5

Working with Lakeflow Jobs

  • Author a multi-task job with explicit task dependencies, shared job parameters, and per-task cluster or serverless compute choices

  • Configure retries, timeouts, and email or webhook notifications on failure

  • Schedule the job with a trigger and use run repair to re-run only failed tasks after a fix

  • Pass values between tasks using task values and verify them in downstream results

Your checkpoint

A five-task orchestrated job definition with parameters, retry policy, and alerts, plus a screenshot-free run log showing a repaired run that skipped already-successful tasks

Week 6

Governance and Security

  • Grant catalog, schema, and table privileges to two distinct groups and verify least-privilege access by testing both identities

  • Implement a column mask and a row filter on a PII table and confirm a restricted user sees only permitted data

  • Create a storage credential and external location, then restrict where external tables can be written

  • Review audit logs and lineage for one table to confirm who read it and where its data came from

Your checkpoint

An access-control matrix implemented in Unity Catalog with a test table proving masked and filtered results for a restricted principal and full results for an admin

Week 7

Implementing CI/CD & Working with Lakeflow Jobs

  • Package pipelines and jobs as a Databricks Asset Bundle with dev and prod targets

  • Store environment-specific values in bundle variables and secrets rather than hard-coded strings

  • Run bundle validation and deployment from a CI pipeline triggered on pull request

  • Add a lightweight data test that runs in CI and fails the build on broken transformations

Your checkpoint

A Git repository containing a bundle that deploys the job and pipeline to two targets via CI, with a pipeline run that fails and then passes after a deliberate fix

Week 8

Databricks Intelligence Platform & Troubleshooting, Monitoring, and Optimization & Data Ingestion and Loading

  • Compare serverless, job cluster, and SQL warehouse compute options and pick the right fit for three given workloads

  • Size clusters using the metrics you collected and identify idle or over-provisioned resources

  • Use system tables and query history to find the slowest and most expensive statements from weeks 1-7

  • Complete a timed full-length practice exam and log every missed question by domain

  • Write a remediation list that re-drills only the two weakest domains before test day

Your checkpoint

A one-page operating runbook covering compute selection, monitoring queries, and cost checks, plus a scored practice exam with a written remediation list for the two weakest domains

Milestones

Week 2: Bronze and silver layers are running end to end, and you can explain without notes how Auto Loader checkpoints and CDC merges prevent duplicates

Week 4: A gold star schema exists and you can demonstrate at least three measured performance improvements with numbers attached

Week 6: An orchestrated multi-task job is scheduled and repaired, and a restricted user is provably blocked from raw PII

Week 8: Bundle-based CI/CD deploys to two targets and your practice exam score is at or above 80 percent with no domain below 70 percent

Databricks Data Engineer exam day

Your checklist

Government-issued photo ID

Use the exact name that appears on your certification registration. A mismatch between ID and registration is the single most common reason candidates are turned away or their result is voided, so check the spelling before test day.

Second form of ID (if testing in person)

Some test centers require a second ID or a signature-bearing card. Confirm the test-center requirement with the provider when you schedule - do not assume one ID is enough.

Your registration confirmation and appointment details

Save the confirmation email offline (screenshot or PDF) so you can still access the appointment time, order number, and support phone number if your network or email is unavailable.

Working webcam, microphone, and speakers

Remote-proctored exams typically require a live camera pan of your workspace, a microphone for the proctor conversation, and audio. Run the provider's system check 24-48 hours ahead so a driver problem is not discovered on test morning.

Stable wired-or-strong wireless internet and a backup option

Have a fallback: a phone hotspot or a second machine. If the connection drops mid-exam, the proctor session may be paused or terminated - confirm the provider's reconnect policy before you start.

A cleared, private room with a closed door

Remove papers, books, second monitors, phones, smart watches, and anything with writing on it from the desk and walls within camera view. Proctors commonly ask for a 360-degree room scan.

Closed all other applications and browser tabs

Only the exam browser should be running. Background apps, chat clients, and notification pop-ups can be flagged as suspicious activity or can steal focus from the exam window.

Notified anyone sharing your space

Tell housemates or family that you cannot be interrupted for the full appointment window plus check-in time; voices or a door opening can trigger a proctor intervention.

Tested the exact browser and device you will use

The provider usually publishes supported browsers and OS versions. Verify yours passes the compatibility check rather than assuming a default browser works - confirm specifics with the provider.

Reviewed the exam's allowed-materials policy

Many certification exams prohibit notes, a second screen, or any documentation lookup. Confirm whether a scratch pad or on-screen notepad is provided, since policies differ by exam and delivery mode.

Planned your timing strategy for the question count

Know the total number of questions and minutes so you can set a per-question budget, then leave 10-15% of the clock for flagged items. The current question count and time limit for your specific exam version should be confirmed with the provider.

Charged laptop and a power cable plugged in

A battery warning or shutdown mid-exam is treated as an interruption. Run on mains power even if you believe the battery will last.

Plan the day

72 hours before

Confirm every logistics item end to end: appointment time and time zone, delivery mode (remote proctored vs. test center), ID requirements, and the allowed-materials policy. Resolve any mismatch now, when rescheduling is still possible.

72 hours before

Run the provider's system test on the exact machine and network you will use. Update the browser, install any required secure-browser component, and confirm camera and microphone permissions are granted.

72 hours before

Do a final diagnostic pass rather than new learning: re-read your own notes on weak areas (for a data engineering exam these are typically Delta Lake table operations, streaming vs. batch semantics, orchestration and job scheduling, ingestion patterns, and access-control/governance concepts), and stop studying material that keeps scoring well.

48 hours before

Take one timed practice set under exam-like conditions - no pausing, no documentation. Use the result only to calibrate pacing and flag topics, not to cram new material.

24 hours before

Stop heavy studying. Light review of a one-page summary of formulas, default behaviors, and command syntax is fine; a full mock exam the night before usually costs more in fatigue than it returns.

24 hours before

Re-confirm the appointment (time zone conversions are a frequent source of no-shows), note the provider's support number, and check that the room you will use is still available and quiet at that hour.

24 hours before

Prepare the physical setup: clear the desk, charge devices, plug in power, and lay out your ID. Sleep a full night.

Morning of

Eat and hydrate before the appointment. Check-in for remote proctoring often takes 10-30 minutes beyond the scheduled start, so build in a buffer and do not schedule anything immediately after.

Morning of

Close all other applications, silence the phone and put it out of reach (not merely face-down on the desk), and log in early to the proctoring platform.

During the test - first pass

Answer the questions you know quickly and flag anything that requires long reasoning. Do not let one scenario question consume several minutes; the exam is scored on total correct answers, not on solving the hardest item.

During the test - second pass

Return to flagged items and eliminate options on partial knowledge. For code-behavior and SQL-result questions, reason about what the statement actually returns row by row rather than pattern-matching to a familiar snippet.

During the test - last 10 minutes

Ensure no question is left blank if the format penalizes unanswered items, then submit deliberately. Confirm what the provider's scoring rules are for unanswered questions before test day, as this varies.

Rules to check

  • Bring the ID that matches your registration exactly. If your legal name differs from the name on the registration, resolve it with the provider before test day - a proctor may refuse entry or invalidate the attempt.

  • Follow the proctor's instructions during check-in, including any room scan or desk-clear request. Refusing or delaying a proctor instruction is typically treated as a conduct violation.

  • Do not use unauthorized materials, notes, second screens, phones, or any AI/assistant tool during the exam, and do not communicate with anyone but the proctor. Attempted unauthorized assistance generally results in a voided score and possible certification ban - the exact sanction is set by the provider's policy, so read it in advance.

  • Do not screenshot, record, or reproduce exam content afterwards. Certification agreements almost universally prohibit disclosing question content, and this is enforceable independent of your score.

  • Understand the break policy before you begin: many proctored certification exams allow no scheduled breaks, and leaving camera view may end the session. If you need accommodations, request them through the provider well before the appointment.

  • Know your score report mechanics - whether results appear immediately on screen, arrive by email, or are held for review, and what the passing threshold is. Confirm both with the provider rather than relying on secondhand reports, since exam versions and thresholds change.

  • If you experience a technical failure, stop and contact the provider's support line immediately rather than continuing through a degraded session; most providers have a documented incident process, but only if it is reported at the time.

  • Arrive or log in early. Late arrival can forfeit your seat or appointment slot, and the exact grace period is provider- and delivery-mode-specific - confirm it when you book.

Databricks Data Engineer FAQ

How many Databricks Data Engineer practice questions are available?

The practice set has 200 Databricks Data Engineer questions. Each one includes the choices, the correct answer, and an explanation, and the set is free to read with no signup.

How many Databricks Data Engineer flashcards are available?

The flashcard set has 32 Databricks Data Engineer cards. Each card shows the term and its definition.

Which topics does the Databricks Data Engineer study material cover?

The Databricks Data Engineer study material has 7 sections, including Databricks Intelligence Platform, Data Ingestion and Loading, and Data Transformation and Modeling.

What format facts does the catalog list for the Databricks Data Engineer exam?

The catalog lists a duration of 90 minutes and these item types: multiple choice and proctored for the Databricks Data Engineer exam.