以下命令行脚本 宿主机 --- (dp) jansora@jansora-PC:~/docker/bigdata$ ls 容器 ---
准备工作
1. 安装Docker
2. 配置基础镜像 Ubuntu 20.04
拉取Ubuntu 镜像 [[docker pull ubuntu:20.04]]
3. 创建数据卷
执行 [[docker volume create bigdata]] 创建数据卷
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
(dp) jansora@jansora-PC:~/docker/bigdata$ docker volume create bigdata
bigdata
(dp) jansora@jansora-PC:~/docker/bigdata$ docker volume inspect bigdata
[
{
"CreatedAt": "2020-06-05T09:54:28+08:00",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/bigdata/_data",
"Name": "bigdata",
"Options": {},
"Scope": "local"
}
]4. 创建网络
[[docker network create -d bridge --subnet=192.168.0.0/24 --gateway=192.168.0.100 --ip-range=192.168.0.0/24 bigdata]]
查看
Bash
1
2
3
4
(dp) jansora@jansora-PC:~/docker/bigdata$ docker network ls
NETWORK ID NAME DRIVER SCOPE
6ba66f53ceba bigdata bridge local
45592e17a68e bridge bridge local5. 启动容器
另开一个终端, 启动基础容器
[[docker run -it --network bigdata --mount source=bigdata,target=/data ubuntu:20.04]]
安装依赖组件
安装依赖前,我们先加速镜像源(可做可不做)
1. 配置 qinghua 镜像源加速(可选)
- 在宿主机
/var/lib/docker/volumes/bigdata/_data目录下创建sources.list,并写入以下内容
纯文本
1
2
3
4
5
6
7
8
9
10
11
12
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-updates main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-updates main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-backports main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-backports main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-security main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-security main restricted universe multiverse
# 预发布软件源,不建议启用
# deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-proposed main restricted universe multiverse
# deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ focal-proposed main restricted universe multiverse- 安装依赖
apt update && apt install ca-certificates -y - 在容器终端内执行 [[cp /data/sources.list /etc/apt/]]
- 更新软件包列表 [[apt update]]
安装依赖软件
容器内终端执行[[apt install vim net-tools lrzsz]]
配置 jdk8
1.容器内终端执行[[apt install openjdk-8-jdk -y]]
Bash
1
2
3
4
5
root@4b2705f587d8:/# java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1ubuntu1-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
root@4b2705f587d8:/# apt install openjdk-8-jdk -y2.配置JAVA_HOME . 容器内终端打开
vim $HOME/.bashrc , 文件底部写入export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
3. 刷新 source $HOME/.bashrc && echo $JAVA_HOME配置 ssh
输入
Asia 和 Shanghai
1 安装 apt install ssh
2 启动 service ssh start
3 配置 ssh local 互信
3.1、ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
3.2、cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
3.3、chmod 0600 ~/.ssh/authorized_keys
3.4、[[ssh localhost]]保存以上改动到 bigdata:base 镜像
4b2705f587d8 为 容器 id,不同容器可能会不同 宿主机内执行docker commit 4b2705f587d8 jansora/bigdata:base
更换容器
- 保存好后退出当前容器(容器内执行 [[ctrl + d]])
- 启动新的容器 [[docker run -it --network bigdata --mount source=bigdata,target=/data jansora/bigdata:base]]
配置 Hadoop
下载安装Hadoop
- 宿主机下载
hadoop-2.10.0[[wget https://mirrors.bfsu.edu.cn/apache/hadoop/common/hadoop-2.10.0/hadoop-2.10.0.tar.gz]] - 解压:
tar xvf hadoop-2.10.0.tar.gz - 移动到数据卷 [[sudo mv hadoop-2.10.0 /var/lib/docker/volumes/bigdata/_data]]
- 在容器内配置
HADOOP_HOME. 容器内终端打开vim $HOME/.bashrc, 文件底部写入export HADOOP_HOME=/data/hadoop-2.10.0 - 配置 sh 启动脚本确保可以root启动 在
start-dfs.sh中追加:
Bash
1
2
3
4
HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root 在
start-yarn.sh中Bash
1
2
3
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root配置 hadoop etc/hadoop/hadoop-env.sh:
追加
Bash
1
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/配置 etc/hadoop/core-site.xml:
创建文件夹 [[mkdir -p ${HADOOP_HOME}/data/temp]]
HTML / XML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:19230</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>${HADOOP_HOME}/data/temp</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>4096</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>10080</value>
</property>
</configuration>配置 etc/hadoop/mapred-site.xml:
HTML / XML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.job.ubertask.enable</name>
<value>true</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:19238</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19239</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>
配置 etc/hadoop/yarn-site.xml:
HTML / XML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:18040</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:18030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:18025</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:18141</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:18088</value>
</property>
</configuration>配置 etc/hadoop/hdfs-site.xml
HTML / XML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:19231</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>master:19232</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///hdfs/data/namenodeDatas</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///hdfs/data/datanodeDatas</value>
</property>
<property>
<name>dfs.namenode.edits.dir</name>
<value>file:///hdfs/data/dfs/nn/edits</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file:///hdfs/data/dfs/snn/name</value>
</property>
<property>
<name>dfs.namenode.checkpoint.edits.dir</name>
<value>file:///hdfs/data/dfs/nn/snn/edits</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<!--指定block块的的大小-->
<property>
<name>dfs.blocksize</name>
<value>134217728</value>
</property>
<!-- -->
<property>
<name>dfs.namenode.handler.count</name>
<value>100</value>
</property>
<!--block的副本数-->
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
配置 etc/hadoop/slaves
配置机器集群, 需要注意的是,这里的每行都是一个容器,docker启动的时候需要指定 --name
集群主机启动的时候需要指定
docker run -it --name master --......
集群从机1启动的时候需要指定 docker run -it --name slave1 --......
集群从机2启动的时候需要指定 docker run -it --name slave2 --......
... 需要启动多少从机就指定多少, 与 slaves 对应Bash
1
2
3
4
5
master
slave1
slave2
slave3
......启动
./start-all.sh验证安装状态
Bash
1
2
3
4
5
6
7
8
root@master:/data/hadoop-2.10.0/etc/hadoop# hadoop fs -mkdir /input
root@master:/data/hadoop-2.10.0/etc/hadoop# echo 'test' > test.txt
root@master:/data/hadoop-2.10.0/etc/hadoop# hadoop fs -put test.txt /input/
root@master:/data/hadoop-2.10.0/etc/hadoop# hadoop fs -ls /input
Found 1 items
-rw-r--r-- 3 root supergroup 5 2020-06-10 15:20 /input/test.txt
root@master:/data/hadoop-2.10.0/etc/hadoop# hadoop fs -cat /input/test.txt
test保存以上改动到 bigdata:base-hadoop 镜像
4b2705f587d8 为 容器 id,不同容器可能会不同 宿主机内执行docker commit 4b2705f587d8 jansora/bigdata:base-hadoop
更换容器
- 保存好后退出当前容器(容器内执行 [[ctrl + d]])
- 启动新的容器 [[docker run -it --network bigdata --mount source=bigdata,target=/data --publish 19231:19231 --publish 19232:19232 --publish 19233:19233 --publish 19234:19234 --publish 19235:19235 --publish 19236:19236 --publish 19237:19237 --publish 19238:19238 --publish 19239:19239 --publish 19240:19240 --publish 19241:19241 jansora/bigdata:base-hadoop]]
问题追踪
配置Hive
下载hive
配置环境变量
hadoop 与以上保持一致 配置环境变量 [[vim ~/.bashrc]]
Bash
1
2
3
4
5
6
7
8
9
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/
export HADOOP_HOME=/data/hadoop-2.10.0
export HIVE_HOME=/data/apache-hive-2.3.7-bin
export PATH=$HADOOP_HOME/bin:$PATH
export PATH=$HADOOP_HOME/sbin:$PATH
export PATH=$HIVE_HOME/bin:$PATH
service ssh start配置 metascore
- 使用默认的 derby 来初始化 [[schematool -dbType derby -initSchema]]
derby 会把原信息保存到执行命令的当前目录下的metastore_db目录, 因此后续启动hive 需要从当前目录执行.
- 使用默认的 mysql 来初始化
配置 MySQL驱动
这里需要配置一个外部的MySQL, 如何安装MySQL此处不解释了 假定 mysql 端口号=1688 用户名=root 密码=123456 且已开启远程访问 2.1 下载mysql驱动 选择mysql 驱动 https://downloads.mysql.com/archives/c-j/ , 最新版本即可
- 解压后放到
$HIVE_HOME/lib
Bash
1
2
3
root@master:/data/apache-hive-2.3.7-bin/lib# ls -l
total 2304
-rw-r--r-- 1 root root 2356711 Dec 4 2019 mysql-connector-java-8.0.19.jar2.2 配置
hive-site.xml
$HIVE_HOME/conf 新建 hive-site.xml , 更多配置项可参考 $HIVE_HOME/conf/hive-default.xml.templateHTML / XML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.31.231:1688/hive?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
<description>password to use against metastore database</description>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateTables</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateColumns</name>
<value>true</value>
</property>
<!-- 设置 hive仓库的HDFS上的位置 -->
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/hive</value>
<description>location of default database for the warehouse</description>
</property>
<!--资源临时文件存放位置 -->
<property>
<name>hive.downloaded.resources.dir</name>
<value>/tmp/hive/resources</value>
<description>Temporary local directory for added resources in the remote file system.</description>
</property>
<!-- Hive在0.9版本之前需要设置hive.exec.dynamic.partition为true, Hive在0.9版本之后默认为true -->
<property>
<name>hive.exec.dynamic.partition</name>
<value>true</value>
</property>
<property>
<name>hive.exec.dynamic.partition.mode</name>
<value>nonstrict</value>
</property>
<property>
<name>hive.server2.thrift.bind.host</name>
<value>127.0.0.1</value>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
<property>
<name>hive.server2.thrift.http.port</name>
<value>10001</value>
</property>
<property>
<name>hive.server2.thrift.http.path</name>
<value>cliservice</value>
</property>
<!-- HiveServer2的WEB UI -->
<property>
<name>hive.server2.webui.host</name>
<value>127.0.0.1</value>
</property>
<property>
<name>hive.server2.webui.port</name>
<value>19238</value>
</property>
<property>
<name>hive.scratch.dir.permission</name>
<value>755</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
</property>
<!-- property>
<name>hive.server2.authentication</name>
<value>NOSASL</value>
</property -->
<property>
<name>hive.auto.convert.join</name>
<value>false</value>
</property>
</configuration>
启动Hive
以命令行启动 [[hive]]
以后台服务启动 [[nohup hive --service hiveserver2 liap metascore &]]
Bash
1
2
3
4
5
root@master:/# hive --service --help
Available Services:
beeline cleardanglingscratchdir cli hbaseimport hbaseschematool help
hiveburninclient hiveserver2 hplsql jar lineage llap llapdump llapstatus
metastore metatool orcfiledump rcfilecat schemaTool version Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
root@master:/data/apache-hive-2.3.7-bin/bin# hive
hive> show databases;
OK
default
Time taken: 3.171 seconds, Fetched: 1 row(s)
hive> create database abc;
OK
Time taken: 2.634 seconds
hive> show databases;
OK
abc
default
保存以上改动到 bigdata:hadoop-hive 镜像
4b2705f587d8 为 容器 id,不同容器可能会不同 宿主机内执行docker commit 4b2705f587d8 jansora/bigdata:hadoop-hive
更换容器
- 保存好后退出当前容器(容器内执行 [[ctrl + d]])
- 启动新的容器 [[docker run -it --network bigdata --mount source=bigdata,target=/data --publish 19231:19231 --publish 19232:19232 --publish 19233:19233 --publish 19234:19234 --publish 19235:19235 --publish 19236:19236 --publish 19237:19237 --publish 19238:19238 --publish 19239:19239 --publish 19240:19240 --publish 19241:19241 jansora/bigdata:hadoop-hive]]
安装 spark
下载 spark
配置环境变量
在.bashrc添加如下配置:
Bash
1
export SPARK_HOME=/data/spark-2.4.5-bin-without-hadoop配置生效
source ~/.bashrc配置 slave 主从节点
参考
$HADOOP_HOME/conf/slaves. 配一样即可配置 $SPARK_HOME/conf/spark-env.sh
Bash
1
2
3
4
export SPARK_DIST_CLASSPATH=$(/data/hadoop-2.10.0/bin/hadoop classpath)
export HADOOP_CONF_DIR=/data/hadoop-2.10.0
export SPARK_MASTER_IP=master
export SPARK_MASTER_WEBUI_PORT=19241启动 spark
[[cd $SPARK_HOME/sbin && ./start-all.sh]]
保存以上改动到 bigdata:hadoop-hive-spark 镜像
4b2705f587d8 为 容器 id,不同容器可能会不同 宿主机内执行docker commit 4b2705f587d8 jansora/bigdata:hadoop-hive-spark
更换容器
- 保存好后退出当前容器(容器内执行 [[ctrl + d]])
- 启动新的容器 [[docker run -it --network bigdata --mount source=bigdata,target=/data --publish 19231:19231 --publish 19232:19232 --publish 19233:19233 --publish 19234:19234 --publish 19235:19235 --publish 19236:19236 --publish 19237:19237 --publish 19238:19238 --publish 19239:19239 --publish 19240:19240 --publish 19241:19241 jansora/bigdata:hadoop-hive-spark]]
安装 hbase
下载 hbase
配置环境变量
在.bashrc添加如下配置:
Bash
1
2
export HBASE_HOME=/data/hbase-2.2.5
export PATH=$HBASE_HOME/bin:$PATH配置 hbase-site.xml
在原有内容下加入以下内容, 启用web端口
HTML / XML
1
2
3
4
<property>
<name>hbase.master.info.port</name>
<value>19241</value>
</property>启动hbase
[[start-hbase.sh]]
取消文件挂载
- 拷贝
data: [[cp -r /data /data.bak]] - 宿主机保存镜像: 宿主机内执行
docker commit 4b2705f587d8 jansora/bigdata:hadoop-hive-spark-hbase-with-dependencies - 退出容器, 进入容器
jansora/bigdata:hadoop-hive-spark-hbase-with-dependencies4.拷贝data: [[mv /data.bak /data]] - 宿主机保存镜像: 宿主机内执行
docker commit 4b2705f587d8 jansora/bigdata:hadoop-hive-spark-hbase-with-dependencies - push 到 docker 容器
docker push jansora/bigdata:hadoop-hive-spark-hbase-with-dependencies

评论栏
分享你的想法,或参与下面的讨论。
参与讨论
支持 Markdown、代码和图片
友善交流,让讨论更有价值。
更多留言
加载中…