My Coding > Operational systems > Google Cloud Platform > Running Apache Hadoop on GCP

Running Apache Hadoop on GCP (Page: 2)

Go to Page:

  1. Setting up Hadoop Cluster;
  2. Browsing your cluster;

After creation, it is very important to check how cluster is going, to know it’s health status, it’s load, used and available resources etc.

Check the cluster status
Check the cluster status
Where to find cluster status
Original image: 897 x 534

1 – Go back to left menu to find there Clusters option within Dataproc section.

2 – Click on the cluster name to go to see its status

Cluster details
Cluster details
Go into cluster details
Original image: 924 x 704

1 – In this window you will go to MONITORING, where you can check the overall load of the YARN, HDFS, CPU, Disk and Network.

2 – VM INSTANCES will give you the view to all nodes in your cluster

3 – Click in the name of the master node.

Master node
Master node
Master node details
Original image: 1281 x 585

1 – In the master node details it is interesting to check it’s network address. In the section Network tags

2 – Find and copy into clipboard the External IP - this will be your access point. When you will start with more valuable data, I would recommend you to use SSH-tunell but this will be described later on.

Accessing YARM manager on Hadoop node

YARN manager
YARN manager
YARM manager on Hadoop node
Original image: 1291 x 532

1 – Open another tab in your browser and go to previously copied IP address with port 8088 - you will go to YARM manager. If you see this picture then it’s mean that you’ve set up everything correctly. You can check few metrics related to the jobs on this cluster

2 – This is the name of your user, Doctor Who

HDFS NameNode service

HDFS NameNode service
HDFS NameNode service
Browsing HDFS NameNode service
Original image: 884 x 1023

1 – With your IP in clipboard, open new browser tab and go to port number 9870. This application will give you a lot of information about HDFS

2 – In the Overview tab you can see the summary of your system.

3 – Check that all your nodes are alive

4 – Go to Datanodes tab to check information about utilization of every node resources

5 – Go to Datanodes Volume Falure to see any problems with your nodes – ideally this section should be empty

HDFS File system
HDFS File system
Browsing HDFS file system
Original image: 1195 x 544

1 – Tab Utilities give your access to some utilities for managing HDFS

2 – Click on Browse file system and you will go to file manager

3 – Choose directory for browsing

4 – You can see the content of this directory and you can brows further down

Go to Page: 1; 2;


Published: 2021-11-24 06:10:16

Last 10 artitles


9 popular artitles
© 2020 MyCoding.uk -My blog about coding and further learning. This blog was writen with pure Perl and front-end output was performed with TemplateToolkit.