[hadoop][基本原理]zookeeper场景使用

代码:https://github.com/xufeng79x/ZkClientTest

1. 简介

  zookeeper的特性决定他适用到某些场景非常合适,比如典型的应用场景:

  1.集群管理(Group Membership)

  2.统一命名服务(Name Service)

  3.配置管理(Configuration Management)

  4.共享锁(Locks)

  5.队列管理

2.集群管理                                                                                                                                                                                                                                  

   在hadoop中主备节点的概念大家都应该不默认,比如HBase中,我们可以启动多个master节点,但是在某时刻只有一个主master,当这个主master节点退出后

其他的背master将会去争取成为主master。一般流程为:

  如上,一开始的时候多个master都会去zookeeper服务上创建相同的临时路径,其中某个master会节点会成功创建,那么这个节点就是主master了,其他创建不成功的将会观察这个临时路径。

当主master由于宕机或者网络原因失去和zookeeper的连接(session)那么这个临时节点就会被zookeeper删除,随后其他备master将会感知到,再去争抢这个临时路径的创建权。

  如下代码利用多线程来模拟多个master的场景:

public class Master implements Runnable {
    // master节点路径
    private String masterPath = null;
    
    // 当前master的信息
    private MasterInfo myinfo = null;
    
    // 当前主master的信息
    private MasterInfo activeMasterInfo = null;
    
    // 控制当前master的运行时间
    private long runningTime = 0;
    
    // 监控handler
    private IZkDataListener masterPathListener = null;
    
    // 当前session
    private ZkClient zc = new ZkClient("xufeng-1:2181,xufeng-2:2181,xufeng-3:2181", 10000, 10000, new SerializableSerializer());
    public Master(MasterInfo info, long runningTime, String masterPath)
    {
        this.myinfo = info;
        this.runningTime = runningTime;
        this.masterPath = masterPath;
        this.masterPathListener = new IZkDataListener() {
            // 当节点被删除的时候触发此方法
            public void handleDataDeleted(String dataPath) throws Exception {
                attackMaster();
            }
            
            public void handleDataChange(String dataPath, Object data) throws Exception {
                // do nothing
                
            }
        };
        
        // 订阅节点
        zc.subscribeDataChanges(masterPath, masterPathListener);
    }

    public void run() {
        attackMaster();
        try {
            Thread.sleep(runningTime);
        } catch (InterruptedException e) {
            // do nothing
        }
    }
    
    // 去争抢这个节点创建(注册)
    private void attackMaster()
    {
        try {
            // 注册节点
            zc.create(masterPath, myinfo, CreateMode.EPHEMERAL);
            // 如果当前注册成功了,那么他就是主master
            activeMasterInfo = myinfo;
            System.out.println("the active master is : " + activeMasterInfo);
        }
        catch (ZkNodeExistsException e)
        {
            // 当节点已经被其他master注册了
            activeMasterInfo = zc.readData(masterPath);
            // 当无法读取到节点信息则认为其他master可能宕机了,再去抢注
            if (null == activeMasterInfo)
            {
                attackMaster();
            }
            else{
                System.out.println(activeMasterInfo + " has become active! " + myinfo +  " wait for next time to be active!");
            }
        }
        catch (Exception e)
        {
            // 当发生其他错误的时候,不去例会
        }
        
    }
}

  当我们建立多个线程去启动的时候,多个master就会去抢注/master节点:

the active master is : MasterInfo [id=C, name=masterC]
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=B, name=masterB] wait for next time to be active!
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=E, name=masterE] wait for next time to be active!
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!

  这个时候我们模拟网络问题,手动地去删除/master节点多次的时候,各个master感知到节点删除会再次抢注:

the active master is : MasterInfo [id=C, name=masterC]
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=B, name=masterB] wait for next time to be active!
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=E, name=masterE] wait for next time to be active!
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
MasterInfo [id=C, name=masterC] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!
the active master is : MasterInfo [id=B, name=masterB]
MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!
MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=E, name=masterE] wait for next time to be active!
MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=C, name=masterC] wait for next time to be active!
MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
MasterInfo [id=B, name=masterB] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!
the active master is : MasterInfo [id=E, name=masterE]
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=C, name=masterC] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=B, name=masterB] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!

  我们可以看到每一次抢注成功的master都不一样,这样如果是由于网络问题而不是当前主master真的宕机了,那么会造成不必要的主备切换。所以说我们还可以进行如下的程序优化:  

  如上图,当由于网络原因原来的主master其实并没有宕机,那么为了减小由于主备切换带来的集群抖动,可以让其他备master延迟一定时间去争抢,而当前的主master则马上去争抢。所以说即使

成为了主master也要观察这个临时路径。

  所以在程序中我们可以在抢注的时候判断如果当前master并不是之前的主master,则延迟一定时间去抢注,使得当前主master能够再次成功的抢的节点,具体优化代码如下:

            // 当节点被删除的时候触发此方法
            public void handleDataDeleted(String dataPath) throws Exception {
                if (null != activeMasterInfo && activeMasterInfo.equals(myinfo))
                {
                    // 如果当前master就是主master的时候,直接去抢注
                    attackMaster();
                }
                else
                {
                    // 如果不是则延迟5秒去抢注,给原先的主master一个机会
                    delayExector.schedule(new Runnable() {
                        
                        public void run() {
                            attackMaster();
                            
                        }
                    }, 5, TimeUnit.SECONDS);
                }
                
            }

  手动的去删除/master节点,结果:masterE始终很稳定的抢注到了/master节点。

the active master is : MasterInfo [id=E, name=masterE]
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=C, name=masterC] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=B, name=masterB] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!
the active master is : MasterInfo [id=E, name=masterE]
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=B, name=masterB] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=C, name=masterC] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
the active master is : MasterInfo [id=E, name=masterE]
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=F, name=masterF] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=H, name=masterH] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=C, name=masterC] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=A, name=masterA] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=D, name=masterD] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=B, name=masterB] wait for next time to be active!
MasterInfo [id=E, name=masterE] has become active! MasterInfo [id=G, name=masterG] wait for next time to be active!

小结:

  除了以上抢注某个临时节点的方式去进行主备切换实现外,我们也可以让每一个master在某个永久节点下各自注册自己的临时节点(CreateMode.EPHEMERAL_SEQUENTIAL

方式,当观察到这个永久节点下znode有变动的时候,查看自己是不是后缀最小的一个,是,则将变成主master。

  除了主备切换场景外,集群管理中的节点发现,任务分发,也同样可以有zookeeper来处理,这种灵活的使用方式可以解决很多分布式场景的问题。

3.其他                                                                                                                                                                                                                               

// 略

原文地址:https://www.cnblogs.com/ios123/p/5735034.html