pyspark.BarrierTaskContext¶

class pyspark.BarrierTaskContext[source]¶

A TaskContext with extra contextual info and tooling for tasks in a barrier stage. Use BarrierTaskContext.get() to obtain the barrier context for a running barrier task.

New in version 2.4.0.

Notes

This API is experimental

Methods

`allGather`([message])	This function blocks until all tasks in the same stage have reached this routine.
`attemptNumber`()	”
`barrier`()	Sets a global barrier and waits until all tasks in this stage hit this barrier.
`get`()	Return the currently active `BarrierTaskContext`.
`getLocalProperty`(key)	Get a local property set upstream in the driver, or None if it is missing.
`getTaskInfos`()	Returns `BarrierTaskInfo` for all tasks in this barrier stage, ordered by partition ID.
`partitionId`()	The ID of the RDD partition that is computed by this task.
`resources`()	Resources allocated to the task.
`stageId`()	The ID of the stage that this task belong to.
`taskAttemptId`()	An ID that is unique to this task attempt (within the same SparkContext, no two task attempts will share the same attempt ID).

pyspark.RDDBarrier pyspark.BarrierTaskInfo