HDFS Failover guide With Cloudera
Kerberized HDFS and NFS Failover with Superna Eyeglass
- Overview
- Kerberized HDFS
- Test setup:
- Required SPN for Cloudera CDH Kerberized HDFS
- Access Zone Failover:
- Kerberized NFS
- Test Setup:
- Required SPN for Kerberized NFS
- Access Zone Failover:
Overview
It is possible to failover HDFS with Superna Eyeglass. The test setup and post failover instructions are outlined below for HDFS and NFS typical configurations. NOTE: in a release coming soon, the ability to manage NFS and HDFS SPN's will be supported to fully automate all steps of failover. See the feature page here https://www.supernaeyeglass.com/feature-descriptions This solution will simplify and automate key steps required for HDFS failover and ensures Big Data solutions can failover seamlessly.
Kerberized HDFS
Test setup:
Cloudera CDH VM:
Guest OS: Ubuntu Server version Ubuntu 16.04.2 LTS
Cloudera CDH
Version 5.16.1 (Cloudera Express)
PowerScale Cluster
OneFS 8.0.0.7
Active Directory
Windows Server 2012 R2
Required SPN for Cloudera CDH Kerberized HDFS
SPN | Name | Rule |
hdfs/clustername.fqdn | Clustername that is joined to AD | Hdfs authentication to AD |
hdfs/namenode.smartconnectname.fqdn | NN FQDN used | Hdfs authentication to aD per smartconnect zone |
HTTP/namenode.smartconnectname.fqdn | NN FQDN used | WebHDFS authentication to AD per Smartconnect Zone |
Access Zone Failover:
Follow Eyeglass Access Zone Failover configuration as per normal
Create required SPN as per the above table on Production Cluster. Example:
isi auth ads spn create ad1.test HTTP/rnsm04-c07-z01.ad1.test
isi auth ads spn create ad1.test hdfs/rnsm04-c07-z01.ad1.test
isi auth ads spn create ad1.test hdfs/rnsm04-c07.ad1.test
Delete the following SPNs on Production cluster
isi auth ads spn delete ad1.test HTTP/rnsm04-c07-z01.ad1.test
isi auth ads spn delete ad1.test hdfs/rnsm04-c07-z01.ad1.test
isi auth ads spn delete ad1.test hdfs/rnsm04-c07.ad1.test
Create the following SPNs on DR Cluster
isi auth ads spn create ad1.test HTTP/rnsm04-c07-z01.ad1.test
isi auth ads spn create ad1.test hdfs/rnsm04-c07-z01.ad1.test
isi auth ads spn create ad1.test hdfs/rnsm04-c07.ad1.test
Verify that HDFS is able to access data successfully (test with Cloudera CDH, do not need to reboot cloudera cdh machine after failover, able to access data successfully)
Kerberized NFS
Test Setup:
NFS Client - Linux
Centos 7.6
PowerScale Cluster
OneFS 8.0.0.7
Active Directory
Windows Server 2012 R2
Required SPN for Kerberized NFS
SPN | Name |
nfs/smartconnectzonemame.fqdn | Smartconnect zone name of the pool for this kerberized NFS |
Access Zone Failover:
Follow Eyeglass Access Zone Failover configuration as per normal
Create required SPN as per the above table on Production Cluster. Example:
isi auth ads spn create ad1.test nfs/rnsm04-c07-z01.ad1.test
After Access Zone Failover
Delete the following SPNs on Production cluster
isi auth ads spn delete ad1.test nfs/rnsm04-c07-z01.ad1.test
Create the following SPNs on DR Cluster
isi auth ads spn create ad1.test nfs/rnsm04-c07-z01.ad1.test
Need to reboot NFS client machine, before able to access data from DR successfully (Due to cached kerberized ticket)