Hadoop集群完全分布式模式环境部署(3)

1) 格式化一个新的分布式文件系统

$ cd /home/hadoop/hadoop-1.2.1 $ bin/hadoop namenode -format
成功情况下系统输出:
Warning: $HADOOP_HOME is deprecated.
 
13/09/10 16:39:31 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = Master/192.168.137.100
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: Java = 1.6.0_45
************************************************************/
13/09/10 16:39:32 INFO util.GSet: Computing capacity for map BlocksMap
13/09/10 16:39:32 INFO util.GSet: VM type = 64-bit
13/09/10 16:39:32 INFO util.GSet: 2.0% max memory = 1013645312
13/09/10 16:39:32 INFO util.GSet: capacity = 2^21 = 2097152 entries
13/09/10 16:39:32 INFO util.GSet: recommended=2097152, actual=2097152
13/09/10 16:39:32 INFO namenode.FSNamesystem: fsOwner=hadoop
13/09/10 16:39:32 INFO namenode.FSNamesystem: supergroup=supergroup
13/09/10 16:39:32 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/09/10 16:39:32 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/09/10 16:39:32 INFO namenode.FSNamesystem: isAccessTokenEnabled=falseaccessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/09/10 16:39:32 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
13/09/10 16:39:32 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/09/10 16:39:33 INFO common.Storage: Image file /home/hadoop/name1/current/fsimage of size 112 bytes saved in 0 seconds.
13/09/10 16:39:33 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/home/hadoop/name1/current/edits
13/09/10 16:39:33 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/home/hadoop/name1/current/edits
13/09/10 16:39:33 INFO common.Storage: Storage directory /home/hadoop/name1 has been successfully formatted.
13/09/10 16:39:33 INFO common.Storage: Image file /home/hadoop/name2/current/fsimage of size 112 bytes saved in 0 seconds.
13/09/10 16:39:33 INFO namenode.FSEditLog: closing edit log: position=4, editlog= /home/hadoop/name2/current/edits
13/09/10 16:39:33 INFO namenode.FSEditLog: close success: truncate to 4, editlog= /home/hadoop/name2/current/edits
13/09/10 16:39:33 INFO common.Storage: Storage directory /home/hadoop/name2 has been successfully formatted.
13/09/10 16:39:33 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Master/192.168.137.100
************************************************************/
查看输出保证分布式文件系统格式化成功执行完后可以到master机器上看到/home/hadoop//name1和/home/hadoop/name2两个目录。在主节点master上面启动hadoop,主节点会启动所有从节点的hadoop。
2) 启动所有节点
启动方式1:

$ bin/start-all.sh
(同时启动HDFS和Map/Reduce)系统输出:
$ bin/start-all.sh
Warning: $HADOOP_HOME is deprecated.
 
starting namenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-namenode-Master.out
Slave3: starting datanode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-Slave3.out
Slave2: starting datanode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-Slave2.out
Slave1: starting datanode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-Slave1.out
The authenticity of host 'master (192.168.137.100)' can't be established.
RSA key fingerprint is d5:18:cb:5f:92:66:74:c7:30:30:bb:36:bf:4c:ed:e9.
Are you sure you want to continue connecting (yes/no)? yes
Master: Warning: Permanently added 'master,192.168.137.100' (RSA) to the list of known hosts.
hadoop@master's password:
Master: starting secondarynamenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode-Master.out
starting jobtracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-jobtracker-Master.out
Slave2: starting tasktracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-Slave2.out
Slave3: starting tasktracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-Slave3.out
Slave1: starting tasktracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-Slave1.out
执行完后可以到master(Master)和slave(Slave1,Slave2,Slave3)机器上看到/home/hadoop/hadoopfs/data1和/home/hadoop/data2两个目录。
启动方式2:

启动Hadoop集群需要启动HDFS集群和Map/Reduce集群。
在分配的NameNode上,运行下面的命令启动HDFS:
$ bin/start-dfs.sh

(单独启动HDFS集群)
bin/start-dfs.sh脚本会参照NameNode上${HADOOP_CONF_DIR}/slaves文件的内容,在所有列出的slave上启动DataNode守护进程。
在分配的JobTracker上,运行下面的命令启动Map/Reduce:$bin/start-mapred.sh
(单独启动Map/Reduce)
bin/start-mapred.sh脚本会参照JobTracker上${HADOOP_CONF_DIR}/slaves文件的内容,在所有列出的slave上启动TaskTracker守护进程。

3) 关闭所有节点

从主节点master关闭hadoop,主节点会关闭所有从节点的hadoop。

$ bin/stop-all.sh
Hadoop守护进程的日志写入到 ${HADOOP_LOG_DIR} 目录 (默认是 ${HADOOP_HOME}/logs).
${HADOOP_HOME}就是安装路径.

6、测试

1) 用jps检验各后台进程是否成功启动

--在master节点查看后台进程
$ /usr/local/jdk1.6.0_45/bin/jps
3180 Jps
2419 SecondaryNameNode
2236 NameNode
2499 JobTracker
--在slave节点查看后台进程
$ /usr/local/jdk1.6.0_45/bin/jps
2631 Jps
2277 DataNode
2365 TaskTracker
2) 通过用浏览器和http访问NameNode和JobTracker的网络接口:

Hadoop集群完全分布式模式环境部署

NameNode – :50070/dfshealth.jsp

Hadoop集群完全分布式模式环境部署

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/7d442a3edc22a2aa39babefd0183aa37.html