NFS存储配置

由于我们是用NFS作为数据存储的,所以需要先配置下nfs

1
/data *(rw,no_root_squash,no_all_squash,async)/usr/local/mysql *(rw,no_root_squash,no_all_squash,sync)/usr/local/redis *(rw,no_root_squash,no_all_squash,sync)

/data 就是cloudreve的数据存储目录 (此目录使用 async异步 来提高性能 适用于文件实时同步要求较低的情况)

/usr/local/mysql 是mysql的pod数据存储的目录

/usr/local/redis 是redis的数据和配置存储目录

刷新下配置

1
exportfs -r

配置文件了,还有挺多不完善的

cloudreve.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
#MySQL

apiVersion: v1
kind: PersistentVolume
metadata:
name: mysql-pv
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
storageClassName: nfs
nfs:
path: /usr/local/mysql
server: 192.168.57.61

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pvc
namespace: cloudreve
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: nfs


---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: mysql
name: mysql
namespace: cloudreve
spec:
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
containers:
- image: mysql:8.0
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-root-password
key: password
ports:
- containerPort: 3306
volumeMounts:
- name: mysqlvolume
mountPath: /var/lib/mysql
volumes:
- name: mysqlvolume
persistentVolumeClaim:
claimName: mysql-pvc
---
#定义mysql的Service
apiVersion: v1
kind: Service
metadata:
labels:
app: mysql
name: svc-mysql
namespace: cloudreve
spec:
selector:
app: mysql
type: NodePort
ports:
- port: 3306
protocol: TCP
targetPort: 3306
nodePort: 30006


---
#redis
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-nfs-pv
namespace: cloudreve
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: nfs-redis
nfs:
path: /usr/local/redis
server: 192.168.57.61

# 创建pvc
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: redis-nfs-pvc
namespace: cloudreve
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: nfs-redis

# 部署redis
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
namespace: cloudreve
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:5.0
volumeMounts:
- name: redis-nfs
mountPath: "/data"
command:
- "redis-server"
volumes:
- name: redis-nfs
persistentVolumeClaim:
claimName: redis-nfs-pvc
---
apiVersion: v1
kind: Service
metadata:
name: svc-redis
namespace: cloudreve
spec:
type: NodePort
ports:
- port: 6379
protocol: TCP
targetPort: 6379
nodePort: 30079
selector:
app: redis

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cloudreve-deployment
namespace: cloudreve
spec:
replicas: 1
selector:
matchLabels:
app: cloudreve
template:
metadata:
labels:
app: cloudreve
spec:
containers:
- name: cloudreve
image: cloudreve:latest
imagePullPolicy: Never
volumeMounts:
- name: data
mountPath: /app/data
- name: config-volume
mountPath: /app/conf.ini
subPath: conf.ini
volumes:
- name: config-volume
configMap:
name: cloudreve-config
- name: data
persistentVolumeClaim:
claimName: cloudreve-pvc

---
apiVersion: v1
kind: PersistentVolume
metadata:
name: cloudreve-pv
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
nfs:
server: 192.168.57.61
path: /data
mountOptions:
- vers=3
- nolock
- proto=tcp
- noatime
- nodiratime
- noexec
- hard
- async
- rsize=524288
- wsize=524288

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cloudreve-pvc
namespace: cloudreve
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi


---
apiVersion: v1
kind: Service
metadata:
name: cloudreve-service
namespace: cloudreve
labels:
app: cloudreve
spec:
type: NodePort
ports:
- port: 9000
name: cloudreve
protocol: TCP
targetPort: 5212
nodePort: 30000
selector:
app: cloudreve
---
#这里需要实现使用集群的dns访问其他pod
apiVersion: v1
kind: ConfigMap
metadata:
name: cloudreve-config
namespace: cloudreve
data:
conf.ini: |
[System]
Debug = false
Mode = master
Listen = :5212
SessionSecret = NWHeTEjapYr1WTr3utGgy6sZ31EUG10NUtOFZx0ORZDNF8LYaYRRW5wYWNOmenx8
HashIDSalt = fsoe4LGVhQFscdBp8TmRZRyfaE5EPU33I7Xf3rWQexO6nxw29D8k4Bqac21TfDkg
[CORS]
AllowOrigins = *
AllowMethods = *
AllowHeaders = *
AllowCredentials = true
[Database]
Type = mysql
Port = 30006
User = root
Password = 123456
Host = 192.168.57.76
Name = cloudreve
Charset = utf8mb4
DBFile = cloudreve.db
GracePeriod = 60
UnixSocket = false
[Redis]
Server = 192.168.57.76:30079
Password =
DB = 0

由于集群DNS出现了一点问题导致redis和mysql连接我暂时先用外部暴露的ip+端口来连接
   mountPath: /app/data 是将NFS挂到 /app/data 作为数据目录来存储网盘的文件

因为要使用configmap挂conf.ini 但是挂载到 /app目录后会将目录内的所有文件覆盖找不到启动文件导致镜像无法启动

所以用了

        - name: config-volume
          mountPath: /app/conf.ini
          subPath: conf.ini

这样就不会覆盖 /app下的文件了

mysql设置下密码

1
kubectl create secret generic mysql-root-password --from-literal=password=123456 -n cloudreve

密码是 123456

连接下数据库

1
mysql -h 192.168.57.76 -P30006 -uroot  -p123456

创建一个名字为cloudreve的数据库

1
CREATE DATABASE IF NOT EXISTS cloudreve DEFAULT CHARSET utf8mb4 COLLATE utf8mb4_general_ci;

网盘程序打包镜像

这里使用本地的镜像,先编译 cloudreve:latest镜像

Dockerfile

1
2
3
4
5
6
7
8
FROM ubuntu:latest

COPY cloudreve /app/cloudreve
COPY conf.ini /app/conf.ini
RUN chmod +x /app/cloudreve
WORKDIR /app

CMD ["./cloudreve"]
1
docker build -t cloudreve:latest .

部署配置

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@master cloudreve]# kubectl apply -f cloudreve.yaml 
persistentvolume/mysql-pv unchanged
persistentvolumeclaim/mysql-pvc unchanged
deployment.apps/mysql unchanged
service/svc-mysql unchanged
persistentvolume/redis-nfs-pv unchanged
persistentvolumeclaim/redis-nfs-pvc unchanged
deployment.apps/redis unchanged
deployment.apps/cloudreve-deployment unchanged
persistentvolume/cloudreve-pv unchanged
persistentvolumeclaim/cloudreve-pvc unchanged
service/cloudreve-service unchanged
configmap/cloudreve-config unchanged

查看cloudreve默认账号密码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[root@master ~]# kubectl get pod -n cloudreve 
NAME READY STATUS RESTARTS AGE
cloudreve-deployment-7c7f887cc8-mhzbm 1/1 Running 6 (21m ago) 31m
mysql-8474cf8649-dxz2q 1/1 Running 2 (24m ago) 31m
redis-84bfc7c68c-khnqk 1/1 Running 2 (24m ago) 31m
[root@master cloudreve]# kubectl logs cloudreve-deployment-7c7f887cc8-mhzbm -n cloudreve

___ _ _
/ __\ | ___ _ _ __| |_ __ _____ _____
/ / | |/ _ \| | | |/ _ | '__/ _ \ \ / / _ \
/ /___| | (_) | |_| | (_| | | | __/\ V / __/
\____/|_|\___/ \__,_|\__,_|_| \___| \_/ \___|
V3.8.3 Commit #88409cc Pro=false
================================================
[Info] 2023-12-05 12:09:55 Initializing database connection...
[Info] 2023-12-05 12:09:55 Start initializing database schema...
[Info] 2023-12-05 12:09:55 Admin user name: admin@cloudreve.org
[Info] 2023-12-05 12:09:55 Admin password: oizCkg10
[Info] 2023-12-05 12:09:56 Start executing database script "UpgradeTo3.4.0".
[Info] 2023-12-05 12:09:56 Finish initializing database schema.
[Info] 2023-12-05 12:09:56 Initialize task queue with WorkerNum = 10
[Info] 2023-12-05 12:09:56 Initialize crontab jobs...
[Info] 2023-12-05 12:09:56 Current running mode: Master.
[Info] 2023-12-05 12:09:56 Listening to ":5212"
[GIN] 2023/12/05 - 12:09:56 | 200 | 701.824µs | 192.168.57.76 | GET "/login?redirect=%2Flogin"
[GIN] 2023/12/05 - 12:09:56 | 200 | 350.788µs | 192.168.57.76 | GET "/static/css/6.57254ef2.chunk.css"
[GIN] 2023/12/05 - 12:09:56 | 200 | 18.047561ms | 192.168.57.76 | GET "/static/js/main.b93ef58f.chunk.js"
[GIN] 2023/12/05 - 12:09:56 | 200 | 237.995µs | 192.168.57.76 | GET "/login?redirect=%2Flogin"
[GIN] 2023/12/05 - 12:09:56 | 200 | 34.672902ms | 192.168.57.76 | GET "/static/js/6.df035d4b.chunk.js"
[GIN] 2023/12/05 - 12:09:56 | 200 | 213.24µs | 192.168.57.76 | GET "/static/css/6.57254ef2.chunk.css"
[GIN] 2023/12/05 - 12:09:56 | 200 | 12.010573ms | 192.168.57.76 | GET "/static/js/main.b93ef58f.chunk.js"
[GIN] 2023/12/05 - 12:09:56 | 200 | 44.951791ms | 192.168.57.76 | GET "/static/js/6.df035d4b.chunk.js"
[GIN] 2023/12/05 - 12:09:57 | 200 | 1.320966ms | 192.168.57.76 | GET "/api/v3/site/config"
[GIN] 2023/12/05 - 12:09:57 | 200 | 1.030957ms | 192.168.57.76 | GET "/static/img/favicon.ico"
[Warning] 2023-12-05 12:10:02 更新检查失败, Get "https://api.github.com/repos/cloudreve/cloudreve/releases": tls: failed to verify certificate: x509: certificate signed by unknown authority

这里附上cloudreve忘记管理员密码重置的方法

1.查看cloudreve的pod名字

1
[root@master cloudreve]# kubectl get pod -n cloudreve NAME                                    READY   STATUS    RESTARTS   AGEcloudreve-deployment-5d9bfcdd9f-khzms   1/1     Running   0          5m5smysql-8474cf8649-xfbhk                  1/1     Running   0          10mredis-84bfc7c68c-4p7q4                  1/1     Running   0          10m

2.进入pod

1
[root@master cloudreve]# kubectl exec -it cloudreve-deployment-5d9bfcdd9f-khzms -n cloudreve bashkubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.root@cloudreve-deployment-5d9bfcdd9f-khzms:/app# lscloudreve  conf.ini  dataroot@cloudreve-deployment-5d9bfcdd9f-khzms:/app# 

 3.执行命令

1
./cloudreve-main --database-script ResetAdminPassword

优化

1.经测试打开上传前预支用户存储文件存储速度会快一些,可以根据自己实际情况判断要不要打开

上传前预支用户存储

2.NFS传输优化

1
vim /etc/nfsmount.conf 

找到这几个参数,将其改成如下配置

1
# Maximum Read Size (in Bytes)Rsize=1024k## Maximum Write Size (in Bytes)Wsize=1024k## Maximum Server Block Size (in Bytes)Bsize=1024k

3.关于k8s的挂载优化这一块我参考了网上一些技术论坛优化NFS速度的帖子

nfs挂载的优化
timeo:  如果超时,客户端等待的时间,以十分之一秒计算
retrans: 超时尝试的次数。
bg:    后台挂载,很有用
hard:   如果server端没有响应,那么客户端一直尝试挂载
wsize:  写块大小
rsize:  读块大小
intr:   可以中断不成功的挂载
noatime: 不更新文件的inode访问时间,可以提高速度
async:  异步读写

4.nfsd的个数

通过查看/proc/net/rpc/nfsd文件的th行,第一个是nfsd的个数,后十个是线程是用的时间数,第二个到第四个值如果很大,那么就需要增加nfsd的个数。

这里标注下参考文章:NFS优化 - Linux文档专区-Chinaunix