Running Apache Hadoop on GCP (Page: 2)

After creation, it is very important to check how cluster is going, to know it’s health status, it’s load, used and available resources etc.

1 – Go back to left menu to find there Clusters option within Dataproc section.

2 – Click on the cluster name to go to see its status

1 – In this window you will go to MONITORING, where you can check the overall load of the YARN, HDFS, CPU, Disk and Network.

2 – VM INSTANCES will give you the view to all nodes in your cluster

3 – Click in the name of the master node.

1 – In the master node details it is interesting to check it’s network address. In the section Network tags

2 – Find and copy into clipboard the External IP - this will be your access point. When you will start with more valuable data, I would recommend you to use SSH-tunell but this will be described later on.

Accessing YARM manager on Hadoop node

1 – Open another tab in your browser and go to previously copied IP address with port 8088 - you will go to YARM manager. If you see this picture then it’s mean that you’ve set up everything correctly. You can check few metrics related to the jobs on this cluster

2 – This is the name of your user, Doctor Who

HDFS NameNode service

1 – With your IP in clipboard, open new browser tab and go to port number 9870. This application will give you a lot of information about HDFS

2 – In the Overview tab you can see the summary of your system.

3 – Check that all your nodes are alive

4 – Go to Datanodes tab to check information about utilization of every node resources

5 – Go to Datanodes Volume Falure to see any problems with your nodes – ideally this section should be empty

1 – Tab Utilities give your access to some utilities for managing HDFS

2 – Click on Browse file system and you will go to file manager

3 – Choose directory for browsing

4 – You can see the content of this directory and you can brows further down

Go to Page: 1; 2;

Published: 2021-11-24 06:10:16