需要的配置文件

centos7的鏡像

centos-7.9.2009-isos-x86_64安裝包下載_開源鏡像站-阿里云 (aliyun.com)

(資料圖片僅供參考)

java8

Java Downloads | Oracle

hadoop3.3.5

Index of /dist/hadoop/common/hadoop-3.3.5 (apache.org)

步驟

首先第一步在本地下載好vmware和centos7的鏡像

之后的選項根據(jù)自己的實際需求選擇

創(chuàng)建完之后

將這個路徑換為你自己鏡像的路徑

然后就可以進入了，完成一些初始的設定之后

開始配置java8環(huán)境

下載完后

打開命令窗口，并進入管理員模式

mkdir /usr/local/java8

tar zxvf jdk-8u371-linux-x64.tar.gz -C /usr/local/java8/

cd /usr/local/java8/jdk1.8.0_371/

vi /etc/profile

## JDK8export JAVA_HOME=/usr/local/java8/jdk1.8.0_371export JRE_HOME=${JAVA_HOEM}/jreexport CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:$CLASSPATHexport JAVA_PATH=${JAVA_HOME}/bin:${JRE_HOME}/binexport PATH=$PATH:${JAVA_PATH}

source /etc/profile

在執(zhí)行完vi /ect/profile命令完后，按i鍵進入編輯模式，在末尾添加上jdk8的環(huán)境設置，然后按esc退出，再按shift+:，然后輸入wq按回車保存，最后使用source命令刷新配置

出現(xiàn)java版本說明配置成功了

配置hadoop3.3.5偽分布式

wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.5/hadoop-arm64-3.3.5.tar.gz

tar zxvf hadoop-3.3.5.tar.gz -C /usr/local/

cd /usr/local/hadoop-3.3.5/

vi /etc/profile

## Hadoop3.3.5

export HADOOP_HOME=/usr/local/hadoop-3.3.5

export PATH=$PATH:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin

source /etc/profile

vi etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/local/java8/jdk1.8.0_371

export HADOOP_PID_DIR=${HADOOP_HOME}/pids

在hadoop-env.sh末尾添加上面兩句話

然后是對hadoop文件的配置

vi etc/hadoop/core-site.xml

#把configuration換位如下配置

fs.defaultFS hdfs://192.168.15.130:9000#注意這里是你自己的IP地址不要照抄 io.file.buffer.size 131072

vi etc/hadoop/hdfs-site.xml

dfs.replication 1 dfs.namenode.name.dir /mnt/data01/hadoop dfs.blocksize 268435456 dfs.namenode.handler.count 100

dfs.datanode.data.dir /mnt/data01/hdfs_dndata

配置hosts

hostnamectl set-hostname billsaifu #可以任意取主機名稱bashecho "192.168.15.130 billsaifu" >> /etc/hosts#你自己的主機名和IP地址

這里可以查看你的IP地址

創(chuàng)建免密

useradd xwpsu - xwp -c "ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsa"su - xwp -c "cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys"su - xwp -c "chmod 0600 ~/.ssh/authorized_keys"chown -R xwp:xwp /usr/local/hadoop-3.3.5mkdir /usr/local/hadoop-3.3.5/pidschown -R xwp:xwp /usr/local/hadoop-3.3.5/pidsmkdir -p /mnt/data01/hadoopmkdir -p /mnt/data01/hdfs_dndatamkdir -p /mnt/data01/yarn_nmdatachown -R xwp:xwp /mnt/data01/hadoopchown -R xwp:xwp /mnt/data01/hdfs_dndatachown -R xwp:xwp /mnt/data01/yarn_nmdata

啟動HDFS

su - xwpcd /usr/local/hadoop-3.3.5bin/hdfs namenode -formatsbin/start-dfs.sh

創(chuàng)建執(zhí)行mapreduce任務所需的HDFS目錄

hdfs dfs -mkdir /user

hdfs dfs -mkdir /user/xwp

執(zhí)行mapreduce任務

hdfs dfs -mkdir inputhdfs dfs -put etc/hadoop/*.xml inputhadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar grep input output "dfs[a-z.]+"hdfs dfs -get output outputcat output/

啟動YARN

vi etc/hadoop/mapred-site.xml

mapreduce.framework.name yarn mapreduce.map.memory.mb 1536 mapreduce.map.java.opts -Xmx1024M mapreduce.reduce.memory.mb 3072 mapreduce.reduce.java.opts -Xmx2560M mapreduce.task.io.sort.mb 512 mapreduce.task.io.sort.factor 100 mapreduce.application.classpath $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*

mapreduce.jobhistory.webapp.address http://192.168.15.130:19888#注意換成自己的IP地址

vi etc/hadoop/yarn-site.xml

yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.env-whitelist JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME yarn.nodemanager.local-dirs /mnt/data01/yarn_nmdata

sbin/start-yarn.sh#開啟yarn

bin/mapred --daemon start historyserver #開啟JobhistoryServer

輸入 hadoop version

出現(xiàn)對應的版本說明配置成功了

利用jps查看是否啟動成功

未啟動

[xwp@billsaifu hadoop-3.3.5]$ jps8129 Jps

成功啟動

[xwp@billsaifu hadoop-3.3.5]$ ./sbin/start-dfs.shStarting namenodes on [billsaifu]billsaifu: Warning: Permanently added "billsaifu,192.168.15.130" (ECDSA) to the list of known hosts.Starting datanodeslocalhost: Warning: Permanently added "localhost" (ECDSA) to the list of known hosts.Starting secondary namenodes [billsaifu][xwp@billsaifu hadoop-3.3.5]$ jps9108 SecondaryNameNode9259 Jps8622 NameNode

登錄相應的頁面

失敗是這樣的