當(dāng)前位置：首頁 > news >正文

上海閔行做網(wǎng)站seo關(guān)鍵詞推廣優(yōu)化

news 2025/7/3 5:12:30

上海閔行做網(wǎng)站,seo關(guān)鍵詞推廣優(yōu)化,深圳商城網(wǎng)站設(shè)計多少錢,wordpress 大數(shù)據(jù)備份目錄一、基礎(chǔ)環(huán)境準(zhǔn)備 1.1 組件下載地址 1.2 前置準(zhǔn)備工作二、 DolphinScheduler集群部署 2.1 解壓安裝包 2.2 配置數(shù)據(jù)庫 2.3 準(zhǔn)備 DolphinScheduler 啟動環(huán)境 2.3.1 配置用戶免密及權(quán)限 2.3.2 配置機器 SSH 免密登陸 2.3.3 啟動 zookeeper集群 2.3.4 修改instal…

一、基礎(chǔ)環(huán)境準(zhǔn)備

1.1 組件下載地址

1.2 前置準(zhǔn)備工作

二、?DolphinScheduler集群部署

2.1 解壓安裝包

2.2 配置數(shù)據(jù)庫

2.3 準(zhǔn)備 DolphinScheduler 啟動環(huán)境

2.3.1 配置用戶免密及權(quán)限

2.3.2 配置機器 SSH 免密登陸

2.3.3 啟動 zookeeper集群

2.3.4 修改install_env.sh 文件

2.3.5 修改dolphinscheduler_env.sh 文件

2.3.6 初始化數(shù)據(jù)庫

2.3.7 修改application.yaml文件

2.3.8 啟動 DolphinScheduler

2.3.9 登錄 DolphinScheduler

2.3.10 DolphinScheduler概覽頁

一、基礎(chǔ)環(huán)境準(zhǔn)備

1.1 組件下載地址

DolphinScheduler-3.2.0官網(wǎng)下載地址：

https://dolphinscheduler.apache.org/zh-cn/download/3.2.0https://dolphinscheduler.apache.org/zh-cn/download/3.2.0

官網(wǎng)安裝文檔：https://dolphinscheduler.apache.org/zh-cn/docs/3.2.0/guide/installation/pseudo-clusterhttps://dolphinscheduler.apache.org/zh-cn/docs/3.2.0/guide/installation/pseudo-cluster

1.2 前置準(zhǔn)備工作

JDK：下載JDK?(1.8+)，安裝并配置?JAVA_HOME?環(huán)境變量，并將其下的?bin?目錄追加到?PATH?環(huán)境變量中。

[root@bigdata102 logs]# vim /etc/profile.d/my_env.sh

數(shù)據(jù)庫：PostgreSQL?(8.2.15+) 或者?MySQL?(5.7+)，兩者任選其一即可，如 MySQL 則需要 JDBC Driver 8.0.16 及以上的版本。（這塊后面會詳細(xì)介紹）

[root@bigdata102 ~]$ mysql -h 192.168.10.102 -u root -P3306 -p
#密碼123456

注冊中心：ZooKeeper?(3.8.0+)

?ps:集群zk版本用的是3.5.7暫時沒出問題

二、?DolphinScheduler集群部署

? ? ? ? ? 部署規(guī)劃：

組件名稱	bigdata102	bigdata103	bigdata104
DolphinScheduler3.2.0	MasterServer
	WorkerServer	WorkerServer	WorkerServer
	AlertServer
	ApiApplicationServer
Zookeeper-3.5.7	zk	zk	zk
Mysql-5.7.16	mysql
	mysql-connector-j-8.3.0.jar	mysql-connector-j-8.3.0.jar	mysql-connector-j-8.3.0.jar

? ps: 為了書寫方便，下文中的DolphinScheduler 直接簡稱為ds

2.1 解壓安裝包

? ? 將ds安裝包上傳到bigdata102節(jié)點的/opt/software目錄下，并直接解壓到當(dāng)前目錄（ps:此處的解壓目錄不是最終的安裝目錄）

[root@bigdata102 software]# tar -zxvf apache-dolphinscheduler-3.2.0-bin.tar.gz

2.2 配置數(shù)據(jù)庫

? ?ds元數(shù)據(jù)存儲在關(guān)系型數(shù)據(jù)庫中，故需創(chuàng)建相應(yīng)的數(shù)據(jù)庫和用戶。

mysql -h 192.168.10.102 -u root -P3306 -p
//創(chuàng)建數(shù)據(jù)庫
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
//創(chuàng)建用戶
//修改 {user} 和 {password} 為你希望的用戶名和密碼
mysql> CREATE USER '{user}'@'%' IDENTIFIED BY '{password}';
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%';
mysql> CREATE USER '{user}'@'localhost' IDENTIFIED BY '{password}';
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost';
mysql> FLUSH PRIVILEGES;

若出現(xiàn)以下錯誤信息，表明新建用戶的密碼過于簡單。
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
可提高密碼復(fù)雜度或者執(zhí)行以下命令降低MySQL密碼強度級別。

mysql> set global validate_password_policy=0;
mysql> set global validate_password_length=4;

賦予用戶相應(yīng)權(quán)限

mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'%';
mysql> flush privileges;

ps:非常關(guān)鍵的一步：

如果使用 MySQL作為ds的元數(shù)據(jù)庫，還需要將mysql-connector-java 驅(qū)動 ( 8.0.16 及以上的版本) 移動到ds的每個模塊的 libs 目錄下，其中包括 api-server/libs ,?alert-server/libs , master-server/libs ,worker-server/libs,?tools/libs

? 本集群中：是將?mysql-connector-j-8.3.0.jar 移動到上述五個libs目錄下

2.3 準(zhǔn)備 DolphinScheduler 啟動環(huán)境

2.3.1 配置用戶免密及權(quán)限

? ? ? ? ? 創(chuàng)建部署用戶，并且一定要配置?sudo?免密。以創(chuàng)建 dolphinscheduler 用戶為例。

# 創(chuàng)建用戶需使用 root 登錄
useradd dolphinscheduler# 添加密碼
echo "dolphinscheduler" | passwd --stdin dolphinscheduler# 配置 sudo 免密
sed -i '$adolphinscheduler ?ALL=(ALL) ?NOPASSWD: NOPASSWD: ALL' /etc/sudoers
sed -i 's/Defaults ? ?requirett/#Defaults ? ?requirett/g' /etc/sudoers# 修改目錄權(quán)限，使得部署用戶對二進制包解壓后的 apache-dolphinscheduler-*-bin 目錄有操作權(quán)限
chown -R dolphinscheduler:dolphinscheduler apache-dolphinscheduler-*-bin
chmod -R 755 apache-dolphinscheduler-*-bin

2.3.2 配置機器 SSH 免密登陸

? ? ? ? 由于ds是集群部署模式，安裝的時候，由單節(jié)點向其他節(jié)點發(fā)送資源，各節(jié)點間需要能 SSH 免密登陸。

免密登陸配置過程：

生成公鑰和私鑰

?[dolphinscheduler@bigdata102 .ssh]$ ssh-keygen -t rsa
然后敲（三個回車），就會生成兩個文件id_rsa（私鑰）、id_rsa.pub（公鑰）

將公鑰拷貝到要免密登錄的目標(biāo)機器上

[dolphinscheduler@bigdata102 .ssh]$ ssh-copy-id bigdata102

[dolphinscheduler@bigdata102 .ssh]$ ssh-copy-id bigdata103

[dolphinscheduler@bigdata102 .ssh]$ ssh-copy-id bigdata104

? ?免密配置完成后，用命令 ssh localhost 判斷是否成功，如果不需要輸入密碼就能ssh登陸，則證明配置成功。

2.3.3 啟動 zookeeper集群

? ? ? 不贅述，zk集群部署流程請自行百度。

2.3.4 修改install_env.sh 文件

? ? ?vim bin/env/install_env.sh

# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
ips=${ips:-"bigdata102,bigdata103,bigdata104"}# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort=${sshPort:-"22"}# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
masters=${masters:-"bigdata102"}# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers=${workers:-"bigdata102:default,bigdata103:default,bigdata104:default"}# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
alertServer=${alertServer:-"bigdata102"}# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers=${apiServers:-"bigdata102"}# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd). Do not add quotes to it if you using related path.
installPath=${installPath:-"/opt/module/dolphinscheduler"}# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser=${deployUser:-"dolphinscheduler"}# The root of zookeeper, for now DolphinScheduler default registry server is zookeeper.
# It will delete ${zkRoot} in the zookeeper when you run install.sh, so please keep it same as registry.zookeeper.namespace in yml files.
# Similarly, if you want to modify the value, please modify registry.zookeeper.namespace in yml files as well.
zkRoot=${zkRoot:-"/dolphinscheduler"}

2.3.5 修改dolphinscheduler_env.sh 文件

# applicationId auto collection related configuration, the following configurations are unnecessary if setting appId.collect=log
#export HADOOP_CLASSPATH=`hadoop classpath`:${DOLPHINSCHEDULER_HOME}/tools/libs/*
#export SPARK_DIST_CLASSPATH=$HADOOP_CLASSPATH:$SPARK_DIST_CLASS_PATH
#export HADOOP_CLIENT_OPTS="-javaagent:${DOLPHINSCHEDULER_HOME}/tools/libs/aspectjweaver-1.9.7.jar":$HADOOP_CLIENT_OPTS
#export SPARK_SUBMIT_OPTS="-javaagent:${DOLPHINSCHEDULER_HOME}/tools/libs/aspectjweaver-1.9.7.jar":$SPARK_SUBMIT_OPTS
#export FLINK_ENV_JAVA_OPTS="-javaagent:${DOLPHINSCHEDULER_HOME}/tools/libs/aspectjweaver-1.9.7.jar":$FLINK_ENV_JAVA_OPTS
# JAVA_HOME, will use it to start DolphinScheduler server
export JAVA_HOME=${JAVA_HOME:-/opt/module/jdk1.8.0_212}# Database related configuration, set database type, username and password# Tasks related configurations, need to change the configuration if you use the related tasks.
export HADOOP_HOME=${HADOOP_HOME:-/opt/module/hadoop-3.1.3}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/module/hadoop-3.1.3/etc/hadoop}
export SPARK_HOME=${SPARK_HOME:-/opt/module/spark}
#export SPARK_HOME1=${SPARK_HOME1:-/opt/soft/spark1}
#export SPARK_HOME2=${SPARK_HOME2:-/opt/soft/spark2}
#export PYTHON_HOME=${PYTHON_HOME:-/opt/soft/python}
export JAVA_HOME=${JAVA_HOME:-/opt/module/jdk1.8.0_212}
export HIVE_HOME=${HIVE_HOME:-/opt/module/hive}
#export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink}
#export DATAX_HOME=${DATAX_HOME:-/opt/datax/datax}export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$PATHexport SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
# Database related configuration, set database type, username and password
export DATABASE=${DATABASE:-mysql}
export SPRING_PROFILES_ACTIVE=${DATABASE}
# export SPRING_DATASOURCE_DRIVER_CLASS_NAME=com.mysql.jdbc.Driver
export SPRING_DATASOURCE_URL="jdbc:mysql://192.168.10.102:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false"
export SPRING_DATASOURCE_USERNAME="root"
export SPRING_DATASOURCE_PASSWORD="123456"
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-bigdata102:2181,bigdata103:2181,bigdata104:2181}
export REGISTRY_ZOOKEEPER_BLOCK_UNTIL_CONNECTED=${REGISTRY_ZOOKEEPER_BLOCK_UNTIL_CONNECTED:30s}

?ps: 這里有個坑：SPRING_DATASOURCE_URL參數(shù)最開始配置為：

export SPRING_DATASOURCE_URL="jdbc:mysql://192.168.10.102:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8"

在【2.3.6 初始化數(shù)據(jù)庫】的時候，程序報Unsupported record version Unknown-0.0

排查后發(fā)現(xiàn)是：jdbc連接開啟了ssl協(xié)議，所以我在上述?SPRING_DATASOURCE_URL參數(shù)的最后面，添加了&useSSL=false，此bug解決了。

2.3.6 初始化數(shù)據(jù)庫

? ? ? ? 經(jīng)過上述步驟，已經(jīng)為 ds創(chuàng)建了元數(shù)據(jù)數(shù)據(jù)庫，通過Shell腳本一鍵初始化數(shù)據(jù)庫（自動建表）

2.3.7 修改application.yaml文件

? ? ? ?需要修改以下5個文件：

master-server/conf/application.yaml
api-server/conf/application.yaml
worker-server/conf/application.yaml
alert-server/conf/application.yaml
tools/conf/application.yaml

? ? ? 每個文件修改的部分相同，如截圖：主要是參數(shù)driver-class-name 和參數(shù)url

  datasource:driver-class-name: com.mysql.cj.jdbc.Driverurl: jdbc:mysql://192.168.10.102:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=falseusername: rootpassword: 123456hikari:connection-test-query: select 1minimum-idle: 5auto-commit: truevalidation-timeout: 3000pool-name: DolphinSchedulermaximum-pool-size: 50connection-timeout: 30000idle-timeout: 600000leak-detection-threshold: 0initialization-fail-timeout: 1

spring:config:activate:on-profile: mysqldatasource:driver-class-name: com.mysql.cj.jdbc.Driverurl: jdbc:mysql://192.168.10.102:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=falseusername: rootpassword: 123456quartz:properties:org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.StdJDBCDelegate