Eyeglass Solutions Publication

HDFS Failover guide With Cloudera

Home

Kerberized HDFS and NFS Failover with Superna Eyeglass


Overview

It is possible to failover HDFS with Superna Eyeglass.  The test setup and post failover instructions are outlined below for HDFS and NFS typical configurations.   NOTE: in a release coming soon, the ability to manage NFS and HDFS SPN's will be supported to fully automate all steps of failover.  See the feature page here https://www.supernaeyeglass.com/feature-descriptions  This solution will simplify and automate key steps required for HDFS failover and ensures Big Data solutions can failover seamlessly.


Kerberized HDFS

Test setup:

Cloudera CDH VM:

  • Guest OS: Ubuntu Server version Ubuntu 16.04.2 LTS


Cloudera CDH

  • Version 5.16.1 (Cloudera Express)


PowerScale Cluster

  • OneFS 8.0.0.7


Active Directory

  • Windows Server 2012 R2


Required SPN for Cloudera CDH Kerberized HDFS


SPN

Name

Rule

hdfs/clustername.fqdn

Clustername that is joined to AD

Hdfs authentication to AD

hdfs/namenode.smartconnectname.fqdn

NN FQDN used

Hdfs authentication to aD per smartconnect zone

HTTP/namenode.smartconnectname.fqdn

NN FQDN used

WebHDFS authentication to AD per Smartconnect Zone


Access Zone Failover:

Follow Eyeglass Access Zone Failover configuration as per normal

  • Create required SPN as per the above table on Production Cluster. Example:

isi auth ads spn create ad1.test HTTP/rnsm04-c07-z01.ad1.test

isi auth ads spn create ad1.test hdfs/rnsm04-c07-z01.ad1.test

isi auth ads spn create ad1.test hdfs/rnsm04-c07.ad1.test


After Access Zone Failover

  • Delete the following SPNs on Production cluster

isi auth ads spn delete ad1.test HTTP/rnsm04-c07-z01.ad1.test

isi auth ads spn delete ad1.test hdfs/rnsm04-c07-z01.ad1.test

isi auth ads spn delete ad1.test hdfs/rnsm04-c07.ad1.test

  • Create the following SPNs on DR Cluster

isi auth ads spn create ad1.test HTTP/rnsm04-c07-z01.ad1.test

isi auth ads spn create ad1.test hdfs/rnsm04-c07-z01.ad1.test

isi auth ads spn create ad1.test hdfs/rnsm04-c07.ad1.test

  • Verify that HDFS is able to access data successfully (test with Cloudera CDH, do not need to reboot cloudera cdh machine after failover, able to access data successfully)




Kerberized NFS

Test Setup:

NFS Client - Linux

  • Centos 7.6


PowerScale Cluster

  • OneFS 8.0.0.7


Active Directory

  • Windows Server 2012 R2



Required SPN for Kerberized NFS


SPN

Name

nfs/smartconnectzonemame.fqdn

Smartconnect zone name of the pool for this kerberized NFS


Access Zone Failover:

  • Follow Eyeglass Access Zone Failover configuration as per normal

  • Create required SPN as per the above table on Production Cluster. Example:

isi auth ads spn create ad1.test nfs/rnsm04-c07-z01.ad1.test


After Access Zone Failover

  • Delete the following SPNs on Production cluster

isi auth ads spn delete ad1.test nfs/rnsm04-c07-z01.ad1.test

  • Create the following SPNs on DR Cluster

isi auth ads spn create ad1.test nfs/rnsm04-c07-z01.ad1.test

  • Need to reboot NFS client machine, before able to access data from DR successfully (Due to cached kerberized ticket)





© Superna Inc