harborV1.10.1 共享存储库预演测试

harborV1.10.1 共享存储库预演测试

三月 05, 2020

测试环境说明:
harbor1
IP:10.XX.XX.X1
role:harbor、redis、mysql、postgreSQL
harbor2
IP:10.XX.XX.X2
role:harbor、nginx、nfs
功能
redis: harbor_session
mysql: harbor_db
posrgreSQL:clair_db
nfs: 共享存储盘(测试使用)
——————————————
环境需求与文档:
https://docs.docker.com/install/linux/docker-ce/centos/
[harbor ~]# docker –version
Docker version 19.03.6, build 369ce74a3c

python –version
Python 2.7.5

安装docker-compose
https://github.com/docker/compose/releases
docker-compose version 1.25.4, build 8d51620a

离线安装包
https://github.com/goharbor/harbor/releases/download/v1.10.1/harbor-offline-installer-v1.10.1.tgz
harbor-offline-installer-v1.10.1.tgz

v1.10.1初始化postgresql
https://github.com/goharbor/harbor/blob/v1.10.1/make/photon/db/initial-registry.sql

安装redis
https://redis.io/download

注意事项:
docker默认使用https,需要有证书进行配置

这是一个主从复制的案例:
https://www.jianshu.com/p/7374f7481d18
这个一个k8s集成文档
https://my.oschina.net/u/2306127/blog/1819645
参考文档
环境准备:https://www.cnblogs.com/breezey/p/9444231.html

selinux防火墙关闭
host主机解析
搭建nfs
部署外部redis、mysql
环境:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[harbor]# getenforce
Permissive
[harbor ~ ]# vim /etc/sysconfig/selinux
[harbor]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[harbor ~ ]# firewall-cmd --state
not running
[harbor ~ ]# python --version
Python 2.7.5
[harbor ~ ]# docker --version
Docker version 19.03.3, build a872fc2f86

安装nfs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[harbor2 ~]# yum -y install nfs-utils
[harbor2 ~]# mkdir /data/images
[harbor2 ~]# vim /etc/exports
[harbor2 ~]# cat /etc/exports
/data/images 10.XX.XX.X2/24(rw,no_root_squash)
[harbor2]# systemctl start nfs
[harbor2]# systemctl status nfs
● nfs-server.service - NFS server and services
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
Active: active (exited) since 二 2020-02-18 16:33:33 CST; 3s ago
Process: 6650 ExecStartPost=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi (code=exited, status=0/SUCCESS)
Process: 6633 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
Process: 6631 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
Main PID: 6633 (code=exited, status=0/SUCCESS)
Tasks: 0
Memory: 0B
CGroup: /system.slice/nfs-server.service

2月 18 16:33:33 master3 systemd[1]: Starting NFS server and services...
2月 18 16:33:33 master3 systemd[1]: Started NFS server and services.

挂载nfs

1
2
3
4
5
[harbor1 ~]# yum -y install nfs-utils
[harbor1 ~]# mkdir /data/images
[harbor1 ~]# mount 10.XX.XX.X2:/data/images /data/images
[root@master2 ~]# df -Th |grep /data/images
10.XX.XX.X2:/myimages nfs4 148G 15G 128G 10% /data/myimages

安装mysql客户端

1
2
3
yum -y install mysql
[harbor]# which mysql
/usr/bin/mysql

docker-compose启动redis和mysql

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
[harbor ~]# cat docker-compose.yml
version: '3'
services:
# mysql-server:
# hostname: mysql-server
# restart: always
# container_name: mysql-server
# image: mysql:5.7
# volumes:
# - /database/mysql:/var/lib/mysql
# command: --character-set-server=utf8
# ports:
# - '3306:3306'
# environment:
# MYSQL_ROOT_PASSWORD: your_passwd
redis:
hostname: redis-server
container_name: redis-server
command: redis-server --requirepass your_passwd(reids一定要加密码,不然就会被挖矿)
restart: always
image: redis:3
volumes:
- /database/redis:/data
ports:
- '6379:6379'
postgres:
hostname: postgres
restart: always
container_name: postgres-server
image: postgres
volumes:
- /database/postgres:/data
ports:
- '5432:5432'
environment:
POSTGRES_PASSWORD: your_passwd

[root@master2 ~]# docker-compose up -d
Creating network "root_default" with the default driver
Creating redis-server ... done
Creating postgres-server ... done
Creating mysql-server ... done
[root@master2 ~]# docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------------
mysql-server docker-entrypoint.sh --cha ... Up 0.0.0.0:3306->3306/tcp, 33060/tcp
postgres-server docker-entrypoint.sh postgres Up 0.0.0.0:5432->5432/tcp
redis-server docker-entrypoint.sh redis ... Up 0.0.0.0:6379->6379/tcp

Mysql导入表结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@master2 ~]# mysql -h10.XX.XX.X1 -uroot -p
mysql> CREATE DATABASE registry DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
mysql> use registry
Database changed
mysql> CREATE TABLE schema_migrations(version bigint not null primary key, dirty boolean not null);
Query OK, 0 rows affected (0.05 sec)
mysql> show tables;
+--------------------+
| Tables_in_registry |
+--------------------+
| schema_migrations |
+--------------------+
1 row in set (0.00 sec)

postgresql导入表结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
yum install postgresql-server -y
[root@master2 ~]# psql -h 10.XX.XX.X1 -p 5432 -U postgres
用户 postgres 的口令: kY#82oYVueDMDA
psql (9.2.24, 服务器 12.2 (Debian 12.2-1.pgdg100+1))
警告:psql 版本9.2, 服务器版本12.0.
一些psql功能可能无法工作.
输入 "help" 来获取帮助信息.

postgres=# \i /root/initial-registry.sql
CREATE DATABASE
psql (9.2.24, 服务器 12.2 (Debian 12.2-1.pgdg100+1))
警告:psql 版本9.2, 服务器版本12.0.
一些psql功能可能无法工作.
您现在已经连线到数据库 "registry",用户 "postgres".
CREATE TABLE

设置http访问

1
2
3
4
5
6
7
8
9
[harbor1 ~]# find / -name docker.service
# vim /etc/systemd/system/docker.service
# ExecStart=/usr/bin/dockerd \
$DOCKER_OPTS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$INSECURE_REGISTRY --insecure-registry 10.XX.XX.X1
# systemctl daemon-reload
# systemctl restart docker

解压启动harbor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[harbor1 ~]# tar -zxvf harbor-offline-installer-v1.10.1.tgz
harbor/harbor.v1.10.1.tar.gz

harbor/prepare
harbor/LICENSE
harbor/install.sh
harbor/common.sh
harbor/harbor.yml

[harbor1 ~]# vim harbor.yml
修改hostname、数据库密码、使用了外置数据库打开外置数据库的注释修改IP和密码
[harbor1 ~]# ./prepare
执行prepare检测启动环境
[harbor1 ~]# ./install.sh
启动harbor

登录、拉取、上传测试:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
[root@master2 harbor]# docker login 10.XX.XX.X1
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

[harbor1 harbor]# docker tag admin/test:v1 10.XX.XX.X1/library/admin/test
[harbor1 harbor]# docker push 10.XX.XX.X1/library/admin/test

[root@master2 ~]# docker push 10.XX.XX.X1/library/postgres:v2
The push refers to repository [10.XX.XX.X1/library/postgres]
0f4c9531c043: Pushed
f2296b808d85: Pushed
428585e93601: Pushed
0b14078a7e74: Pushed
2eb0b93970c9: Pushed
9b95b8eade9b: Pushed
8abf7e1726c6: Pushed
ad5d7aba6303: Pushed
7a1725b13885: Pushed
1bb65a17f346: Pushed
a872403d70e5: Pushed
9129eada97a4: Pushed
efb2aa6f2c78: Pushed
488dfecc21b1: Pushed
v2: digest: sha256:40ebbae0ce4d29eacb5a0ad3ae450f31f089124acdc2cc68b177872f716d0454 size: 3245


[harbor harbor]# docker pull 10.XX.XX.X2/library/admin/test:v1
v1: Pulling from library/admin/test
bc51dd8edc1b: Pull complete
d2b355dbb6c6: Pull complete
d237363a1a91: Pull complete
ff4b9d2fde66: Pull complete
646492d166e7: Pull complete
50eeac6fd5fb: Pull complete
502963de6da8: Pull complete
d7263f7627b9: Pull complete
d234d8f1a205: Pull complete
1b2c24e5275c: Pull complete
3f7c6dd9a7ae: Pull complete
d6d6977a74b3: Pull complete
acf1093f8b78: Pull complete
77e1ac8c247a: Pull complete
Digest: sha256:40ebbae0ce4d29eacb5a0ad3ae450f31f089124acdc2cc68b177872f716d0454
Status: Downloaded newer image for 10.XX.XX.X2/library/admin/test:v1
10.XX.XX.X2/library/admin/test:v1
[harbor harbor]# docker images |grep test
10.XX.XX.X2/library/admin/test v1 0d2531ee3abd 5 days ago 397MB

[harbor ~]# docker tag goharbor/prepare:v1.10.1 10.XX.XX.X2/library/admin/test:v2
[harbor ~]# docker push 10.XX.XX.X2/library/admin/test:v2
The push refers to repository [10.XX.XX.X2/library/admin/test]
bf5ac9b9c61c: Pushed
06e0f1585c01: Pushed
3710d94e58dd: Pushed
5c5501748347: Pushed
f59c6315bf8a: Pushed
fb3507ff707e: Pushed
93e0577272a9: Pushed
v2: digest: sha256:e025a09df19c99e0afb4b116cbd698d5de27fb39e26314934460c7bae7c21afc size: 1787

Harbor预演测试过程
目前测试进度:

  1. 搭建 v1.10.1(harbor最新版本),单节点使用内置postgresql,和本地磁盘存储,测试成功
  2. 搭建 v1.10.1 使用外置mysql、postgresql、redis(全部docker-compose启动)harbor服务启动后不能登录,发现该版本不支持外置mysql(查阅官方文档https://github.com/goharbor/harbor/blob/master/docs/1.10/install-config/configure-yml-file.md显示目前该版本外置数据库仅支持postgresql),服务启动后redis auth认证有问题
  3. 搭建 v1.10.1 更改外置数据库为postgresql,redis使用手动搭建方式进行单点测试,搭建、测试拉取、推送镜像成功
  4. 版本 v1.10.0 和v1.8.0 经过查阅github文档、启动测试,该版本不支持外置mysql,目前仅支持postpresql
  5. 搭建 v1.10.1 外接数据库 postpres、redis ,2个harbor节点构成集群共享存储,搭建、测试拉取、推送镜像成功

还需继续研究项:

  1. 研究https认证
  2. 研究nginx keeplive 分流
  3. 版本 v1.5.4(发于2018.1.29版本)支持外置mysql,尝试搭建v1.5.4进行测试和调研
  4. 继续研究最新版harbor集群postgresql容灾方式

常用版本列举对比 https://github.com/goharbor/harbor/releases:
版本:v1.10
发版时间:2019.12.13
新功能对比:
外部数据库仅支持postgresql
支持第三方漏洞扫描程序
新增项目配置规则
管理员和用户增加功能,新增了“受限访客”用户
增强OIDC功能,复制功能增强
问题:
v1.10.0 job中有正在运行的任务,当redis重启后,任务被冻结,需点击重试恢复,后续补丁修复
v1.10.1 修复了1.9.3版本拉取镜像等https://github.com/goharbor/harbor/issues?q=is%3Aissue+label%3Atarget%2F1.10.1+is%3Aclosed
优缺点:
目前最新版,修复了旧版本的bug,增加了新的功能
统一了配置文件

版本:v1.8
发版时间:2019.5.21
新功能对比:
外部数据库仅支持postgresql
支持OIDC,使用外部验证验证身份
扩展了镜像复制功能
新增机器人账户
新增非harbor存储库之间互相复制镜像
问题:
复制策略标签过滤器丢失
不支持清单
某些情况下redis持久化数据巨大问题
优缺点:
查看官方文档和网上博客反馈,存在一些bug
配置文件也是统一的修改配置文件启动方式简单

版本:v1.5
发版时间:2018.10.26
新功能对比:
外部数据库支持mysql
升级Clair
NVD迁移至AWS
添加了镜像标签过滤器
添加LDAP组
用户可以为镜像添加标签
新增仓库只读模式
问题:
1.5.0 存在镜像标签问题,镜像复制签名问题
1.5.4 修复了前面几个版本的bug,不支持在线安装,支持离线安装
https://github.com/goharbor/harbor/releases/tag/v1.5.4
优缺点:
1.5.4支持外置mysql
1.5之后的版本,添加了新的功能,mysql换为postgresql,官方文档显示是将多个mysql迁移为统一的postgresql方便管理