In our application, we performed read and count operation on files and DataFrame. If the application executes Spark SQL queries then the SQL tab displays information, such as the duration, Spark jobs, and physical and logical plans for the queries. Number of cores = 3 as I gave master as local with 3 threads The Executors tab provides not only resource information like amount of memory, disk, and cores used by each executor but also performance information. The Storage Memory column shows the amount of memory used and reserved for caching data. The Executors tab displays summary information about the executors that were created for the application, including memory and disk usage and task and shuffle information. The Environment tab displays the values for the different environment and configuration variables, including JVM, Spark, and system properties. I had explained the description part in the coming part. 1.4 Descriptionĭescription links the complete details of the associated SparkJob like Spark Job Status, DAG Visualization, Completed Stages Data is partitioned into two files by default. In our case, Spark job0 and Spark job1 have individual single stages but when it comes to Spark job 3 we can see two stages that are because of the partition of data. 1.3 Number of StagesĮach Wide Transformation results in a separate Number of Stages. So if we look at the fig it clearly shows 3 Spark jobs result of 3 actions. In our above application, we have performed 3 Spark jobs (0,1,2) As I was running in a local machine, I tried using Standalone mode 1.2 Number of Spark Jobs:Īlways keep in mind, the number of Spark jobs is equal to the number of actions in the application and each Spark job should have at least one Stage.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |