HamaWhite原创,转载请注明出处。欢迎大家增加Giraph
技术交流群: 228591158
本文目的:在改动GPS源代码后,具体描写叙述怎样编译和分发到各Worker节点上。
以下以Graph Coloring 算法为例进行解说。本文基于GPS的前三篇文章。
1. 首先在Master(test150)上改动源代码。
Graph Coloring算法源代码路径:gps.examples.coloring包,主要改动ColoringVertex.java类。
该算法在Selection(MIS_1)阶段是依照顶点的出度大小概率性的选择UNDECIDED状态的顶点,源代码例如以下:
if (ColoringVertexType.NOT_IN_SET == value.type || ColoringVertexType.IN_SET == value.type) { return; } double probability = getNeighborsSize() > 0 ? 1.0 / ((double) 2*value.numRemainingNeighbors) : 1; if (Math.random() <= probability) { value.type = ColoringVertexType.SELECTED_AS_POSSIBLE_IN_SET; if (value.numRemainingNeighbors > 0) { ColoringMessage newSelectedAsPossibleMessage = ColoringMessage .newNeighborSelectedAsPossibleMessage(getId()); for (int neighborId : getNeighborIds()) { if (neighborId >= 0) { sendMessage(neighborId, newSelectedAsPossibleMessage); } } } }
以下改动为:选择全部的UNDECIDED状态的顶点,而非概率性选择。
改动后源代码例如以下:
if (ColoringVertexType.NOT_IN_SET == value.type || ColoringVertexType.IN_SET == value.type) { return; } //double probability = getNeighborsSize() > 0 ?1.0 / // ((double) 2*value.numRemainingNeighbors) : 1; //if (Math.random() <= probability) { // value.type = ColoringVertexType.SELECTED_AS_POSSIBLE_IN_SET; if (value.numRemainingNeighbors > 0) { ColoringMessage newSelectedAsPossibleMessage = ColoringMessage .newNeighborSelectedAsPossibleMessage(getId()); for (int neighborId : getNeighborIds()) { if (neighborId >= 0) { sendMessage(neighborId, newSelectedAsPossibleMessage); } } } //}
2. 由于之前已编译过GPS源代码,而且把Jar包等文件已分发到集群中各个Worker节点上,所以此步骤是删除之前的文件。删除Master节点trunk文件夹下的:gps-0.0.1-slave.tar.gz 、gps_node_runner.jar和classes文件夹,删除Worker节点trunk文件夹下的 conf 、gps-0.0.1-slave.tar.gz 、gps_node_runner.jar 、libs 、scripts。logs文件夹保留。
3. 參考 GPS-Graph Processing System集群安装笔记(一),又一次编译和分发Jar包等文件。
以下附上我的脚本。因中间使用了我自己的脚本,故不可直接使用,可是能够參考。脚本所在文件夹:/home/gougou/GPS/trunk。
cd /home/gougou/GPS/trunk # delete master files rm -rf gps_node_runner.jar rm -rf classes rm -rf gps-0.0.1-slave.tar.gz # delete worker files. the Shell writed by myself. cd /home/gougou/ShellUtils ./deleteDirectory.sh /home/gougou/GPS/trunk/conf ./deleteDirectory.sh /home/gougou/GPS/trunk/gps-0.0.1-slave.tar.gz ./deleteDirectory.sh /home/gougou/GPS/trunk/gps_node_runner.jar ./deleteDirectory.sh /home/gougou/GPS/trunk/libs ./deleteDirectory.sh /home/gougou/GPS/trunk/scripts/ # compile GPS source code cd /home/gougou/GPS/trunk cd local-master-scripts # generate gps_node_runner.jar and classes under trunk directory ./make_gps_node_runner_jar.sh # generate gps-0.0.1-slave.tar.gz under trunk directory ./make_gps_tar_gz.sh cd ../master-scripts cp slaves temp cp slaves-12 slaves ./copy_and_untar_gps_tar_to_slaves.sh 12 mv temp slaves4. 在trunk/master-scripts文件夹下,执行Graph Coloring 算法,命令例如以下:
./start_gps_nodes.sh 2 GC-Test5-1 "-ifs /user/gougou/GC-Test5/gc-5.txt -hcf /home/gougou/hadoop-1.0.3/conf/core-site.xml -jc gps.examples.coloring.JobConfiguration -mcfg /machine-configs/test_machine_config_2.cfg -log4jconfig /home/gougou/GPS/trunk/conf/log4j.config"
完!