Spark UI

Setting up the Spark History Server on an Amazon EC2 instance involves configuring the Spark History Server to track and display Spark application logs. Here’s a step-by-step guide:

Prerequisites

1. EC2 Instance: A running EC2 instance with sufficient resources (e.g., memory, storage) and configured with a public IP or DNS.
2. Spark Installed: Spark should be installed on the EC2 instance.
3. Java Installed: Java is required to run Spark.
4. S3 Bucket (Optional): For storing Spark event logs, an S3 bucket can be used instead of the local filesystem.
5. IAM Role (Optional): If using S3, attach an IAM role to the EC2 instance with permissions to access the S3 bucket.

Spark History Server Setup on EC2

Step 1: Create and Launch an EC2 Instance

1. Go to the AWS Management Console and launch a new EC2 instance:
   • Choose an Amazon Machine Image (AMI), such as Amazon Linux 2 or Ubuntu.
   • Select an instance type (e.g., t2.medium for testing or larger for production).
   • Configure key details like storage, security groups, and IAM roles.
2. Security Group Configuration:
   • Add rules to allow traffic:
     • TCP port 22: For SSH access.
     • TCP port 18080: For accessing the Spark History Server UI.
   • Restrict the source to your IP or 0.0.0.0/0 for public access.

Step 2: Install Prerequisites

Install Java:

sudo yum install java-1.8.0 -y  # Amazon Linux 2

Install Spark: Download and extract Apache Spark:

wget https://dlcdn.apache.org/spark/spark-<version>/spark-<version>-bin-hadoop<version>.tgz
tar -xvf spark-<version>-bin-hadoop<version>.tgz
sudo mv spark-<version>-bin-hadoop<version> /opt/spark

Set Up Environment Variables:

export SPARK_HOME=/opt/spark
export PATH=$SPARK_HOME/bin:$PATH

Step 3: Configure Spark Event Logging

1. Edit the spark-defaults.conf file:bashCopy codenano /opt/spark/conf/spark-defaults.conf
2. Add the following configurations:propertiesCopy codespark.eventLog.enabled true spark.eventLog.dir file:///tmp/spark-events spark.history.fs.logDirectory file:///tmp/spark-events
3. If using Amazon S3 for logs:propertiesCopy codespark.eventLog.dir s3a://<bucket-name>/spark-events spark.history.fs.logDirectory s3a://<bucket-name>/spark-events
4. Ensure the IAM role attached to the EC2 instance has S3 read/write permissions.

Step 4: Start the Spark History Server

1. Start the server:bashCopy code/opt/spark/sbin/start-history-server.sh
2. Verify the server status:bashCopy codetail -f /opt/spark/logs/spark--org.apache.spark.deploy.history.HistoryServer-*.out
3. Access the Spark History Server:vbnetCopy codehttp://<EC2-PUBLIC-IP>:18080