Databricks Certified Associate Developer for Apache Spark 3.0 : Associate-Developer-Apache-Spark

Which of the following code blocks returns all unique values across all values in columns value and productId in DataFrame transactionsDf in a one-column DataFrame?

• A. transactionsDf.agg({'value': 'collect_set', 'productId': 'collect_set'})
• B. tranactionsDf.select('value').join(transactionsDf.select('productId'), col('value')==col('productId'),
  'outer')
• C. transactionsDf.select(col('value'), col('productId')).agg({'*': 'count'})
• D. transactionsDf.select('value', 'productId').distinct()
• E. transactionsDf.select('value').union(transactionsDf.select('productId')).distinct()

Which of the following code blocks displays the 10 rows with the smallest values of column value in DataFrame transactionsDf in a nicely formatted way?

• A. transactionsDf.sort(col("value").asc()).print(10)
• B. transactionsDf.sort(col("value")).show(10)
• C. transactionsDf.sort(col("value").desc()).head()
• D. transactionsDf.orderBy("value").asc().show(10)
• E. transactionsDf.sort(asc(value)).show(10)

Which of the following describes the role of the cluster manager?

• A. The cluster manager schedules tasks on the cluster in client mode.
• B. The cluster manager allocates resources to Spark applications and maintains the executor processes in remote mode.
• C. The cluster manager allocates resources to the DataFrame manager.
• D. The cluster manager schedules tasks on the cluster in local mode.
• E. The cluster manager allocates resources to Spark applications and maintains the executor processes in client mode.

Which of the following code blocks returns a DataFrame that has all columns of DataFrame transactionsDf and an additional column predErrorSquared which is the squared value of column predError in DataFrame transactionsDf?

• A. transactionsDf.withColumn("predError", pow(col("predErrorSquared"), 2))
• B. transactionsDf.withColumn("predErrorSquared", "predError"**2)
• C. transactionsDf.withColumn("predErrorSquared", pow(col("predError"), lit(2)))
• D. transactionsDf.withColumnRenamed("predErrorSquared", pow(predError, 2))
• E. transactionsDf.withColumn("predErrorSquared", pow(predError, lit(2)))

Which of the following code blocks reads in the two-partition parquet file stored at filePath, making sure all columns are included exactly once even though each partition has a different schema?
Schema of first partition:
1.root
2. |-- transactionId: integer (nullable = true)
3. |-- predError: integer (nullable = true)
4. |-- value: integer (nullable = true)
5. |-- storeId: integer (nullable = true)
6. |-- productId: integer (nullable = true)
7. |-- f: integer (nullable = true)
Schema of second partition:
1.root
2. |-- transactionId: integer (nullable = true)
3. |-- predError: integer (nullable = true)
4. |-- value: integer (nullable = true)
5. |-- storeId: integer (nullable = true)
6. |-- rollId: integer (nullable = true)
7. |-- f: integer (nullable = true)
8. |-- tax_id: integer (nullable = false)

• A. spark.read.parquet(filePath)
• B. 1.nx = 0
  2.for file in dbutils.fs.ls(filePath):
  3. if not file.name.endswith(".parquet"):
  4. continue
  5. df_temp = spark.read.parquet(file.path)
  6. if nx == 0:
  7. df = df_temp
  8. else:
  9. df = df.join(df_temp, how="outer")
  10. nx = nx+1
  11.df
• C. spark.read.parquet(filePath, mergeSchema='y')
• D. spark.read.option("mergeSchema", "true").parquet(filePath)
• E. 1.nx = 0
  2.for file in dbutils.fs.ls(filePath):
  3. if not file.name.endswith(".parquet"):
  4. continue
  5. df_temp = spark.read.parquet(file.path)
  6. if nx == 0:
  7. df = df_temp
  8. else:
  9. df = df.union(df_temp)
  10. nx = nx+1
  11.df