Databricks Certified Associate Developer for Apache Spark 3.5 - Python 問題集 : Associate-Developer-Apache-Spark-3.5

A data scientist wants each record in the DataFrame to contain:
The first attempt at the code does read the text files but each record contains a single line. This code is shown below:

The entire contents of a file
The full file path
The issue: reading line-by-line rather than full text per file.
Code:
corpus = spark.read.text("/datasets/raw_txt/*") \
.select('*', '_metadata.file_path')
Which change will ensure one record per file?
Options:

• A. Add the option lineSep=", " to the text() function
• B. Add the option lineSep='\n' to the text() function
• C. Add the option wholetext=False to the text() function
• D. Add the option wholetext=True to the text() function

21 of 55.
What is the behavior of the function date_sub(start, days) if a negative value is passed into the days parameter?

• A. An error message of an invalid parameter will be returned.
• B. The number of days specified will be removed from the start date.
• C. The number of days specified will be added to the start date.
• D. The same start date will be returned.

13 of 55.
A developer needs to produce a Python dictionary using data stored in a small Parquet table, which looks like this:
region_id
region_name
10
North
12
East
14
West
The resulting Python dictionary must contain a mapping of region_id to region_name, containing the smallest 3 region_id values.
Which code fragment meets the requirements?

• A. regions_dict = regions.select("region_id", "region_name").take(3)
• B. regions_dict = dict(regions.select("region_id", "region_name").rdd.collect())
• C. regions_dict = dict(regions.take(3))
• D. regions_dict = dict(regions.orderBy("region_id").limit(3).rdd.map(lambda x: (x.region_id, x.region_name)).collect())

5 of 55.
What is the relationship between jobs, stages, and tasks during execution in Apache Spark?

• A. A stage contains multiple jobs, and each job contains multiple tasks.
• B. A job contains multiple stages, and each stage contains multiple tasks.
• C. A job contains multiple tasks, and each task contains multiple stages.
• D. A stage contains multiple tasks, and each task contains multiple jobs.

A developer is running Spark SQL queries and notices underutilization of resources. Executors are idle, and the number of tasks per stage is low.
What should the developer do to improve cluster utilization?

• A. Enable dynamic resource allocation to scale resources as needed
• B. Increase the size of the dataset to create more partitions
• C. Reduce the value of spark.sql.shuffle.partitions
• D. Increase the value of spark.sql.shuffle.partitions