Associate-Developer-Apache-Spark 無料問題集「Databricks Certified Associate Developer for Apache Spark 3.0」

（A）The cluster manager receives input from the driver through the SparkContext.

（B）The cluster manager does not exist in standalone mode.

（C）Each cluster manager works on a single partition of data.

（D）The cluster manager transforms jobs into DAGs.

（E）In client mode, the cluster manager runs on the edge node.

（A）itemsDf.write.option('destination', 'memory').save()

（B）itemsDf.store()

（C）itemsDf.cache(StorageLevel.MEMORY_AND_DISK)

（D）itemsDf.persist(StorageLevel.MEMORY_ONLY)

（E）itemsDf.cache()

（A）In cluster mode, the Spark driver is not co-located with the cluster manager, while it is co-located in client mode.

（B）In cluster mode, the driver resides on a worker node, while it resides on an edge node in client mode.

（C）In cluster mode, executor processes run on worker nodes, while they run on gateway nodes in client mode.

（D）In cluster mode, a gateway machine hosts the driver, while it is co-located with the executor in client mode.

（E）In cluster mode, the cluster manager resides on a worker node, while it resides on an edge node in client mode.

（A）Stages with narrow dependencies can be grouped into one task.

（B）Within one task, the slots are the unit of work done for each partition of the data.

（C）Tasks are the smallest element in the execution hierarchy.

（D）Tasks are the second-smallest element in the execution hierarchy.

（E）Tasks with wide dependencies can be grouped into one stage.

（A）from pyspark import StorageLevel
transactionsDf.persist(StorageLevel.MEMORY_ONLY)

（B）from pyspark import StorageLevel
transactionsDf.cache(StorageLevel.MEMORY_ONLY)

（C）transactionsDf.cache()

（D）transactionsDf.clear_persist()

（E）transactionsDf.storage_level('MEMORY_ONLY')

（F）transactionsDf.persist()

（A）DataFrame.rdd.map()

（B）DataFrame.collect()

（C）DataFrame.select()

（D）DataFrame.coalesce()

（E）DataFrame.count()

（A）1.itemsDfSchema = StructType([
2. StructField("itemId", IntegerType()),
3. StructField("attributes", ArrayType([StringType()])),
4. StructField("supplier", StringType())])
5.
6.itemsDf = spark.read(schema=itemsDfSchema).parquet(filePath)

（B）1.itemsDf = spark.read.schema('itemId integer, attributes <string>, supplier string').parquet(filePath)

（C）1.itemsDfSchema = StructType([
2. StructField("itemId", IntegerType()),
3. StructField("attributes", ArrayType(StringType())),
4. StructField("supplier", StringType())])
5.
6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath)

（D）1.itemsDfSchema = StructType([
2. StructField("itemId", IntegerType()),
3. StructField("attributes", StringType()),
4. StructField("supplier", StringType())])
5.
6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath)

（E）1.itemsDfSchema = StructType([
2. StructField("itemId", IntegerType),
3. StructField("attributes", ArrayType(StringType)),
4. StructField("supplier", StringType)])
5.
6.itemsDf = spark.read.schema(itemsDfSchema).parquet(filePath)

（A）Shuffles belong to a class known as "full transformations".

（B）In a shuffle, Spark writes data to disk.

（C）A shuffle is one of many actions in Spark.

（D）Shuffles involve only single partitions.

（E）Operations involving shuffles are never evaluated lazily.

（A）The avg("value") should be specified as a second argument to agg() instead of being appended to it.

（B）All column names should be wrapped in col() operators.

（C）agg should be replaced by groupBy.

（D）Instead of avg("value"), avg(col("value")) should be used.

（E）"storeId" and "value" should be swapped.

（A）transactionsDf.agg("count", "mean", "stddev", "25%", "50%", "75%", "min")

（B）transactionsDf.agg("count", "mean", "stddev", "25%", "50%", "75%", "min").show()

（C）transactionsDf.summary()

（D）transactionsDf.summary("count", "mean", "stddev", "25%", "50%", "75%", "max").show()

（E）transactionsDf.summary().show()