GPFS 5.0 구성(Configuration)
GPFS/config2021. 2. 21. 09:08
반응형
| * Written by Jung Byung Wook 구성환경 - 2NODE
- Hostname : gpfs1, gpfs2
- IP Addr : 10.10.10.10, 10.10.10.11 - Data disk : 5G * 3EA
- Tie breakers disk : 5G * 3EA
|
0. 사전 준비단계
1) 노드 구성 계획
- 몇 개의 노드로 구성할 것인지?
2) Inter-Connect(Heartbeat) Interface
- 1G Interface로도 충분하나, 10G 구성도 무방.
- Heartbeat Interface의 etherchannel은 Active / Standby로 구성하는 것이 일반적.
3) GPFS 공유파일시스템(MMFS)의 용량과 각 NSD disk(LUN)은 몇개로 가져갈 것인지?
- GPFS 파일시스템의 증설단위는 1 NSD(LUN)이며, 사용용량에 맞게 NSD를 구성해야한다.
- NSD disk(LUN)의 갯수는 각 사이트 정책/관행에 따라 갯수가 정해지나, 너무 적은 NSD는 성능저하를 줄수 있다.
4) Remote GPFS Cluster를 사용할 것인지?
- 일반적인 사이트에서는 사용하지 않음.
5) Communication protocol(통신방식)은 어떤 것을 사용할 것인지?
- SSH / RSH 방식이 있다.
- RSH의 경우, 보안취약점이 존재하여, SSH 암호화방식을 통한 Communication을 권장한다.
1. 상호 노드간 패스워드 없이 통신할 수 있는 연결구성 및 디스크 속성 설정
- 권장구성방식인 SSH를 통해, GPFS를 구성하기 위한 방식이며, 이 스텝에서는 각 노드가 Password 없이 접근할 수 있게, 노드 상호간의 공개키(Public key)를 공유한다.
1) /etc/hosts에 상호 노드 등록
- gpfs1 node : 10.10.10.10
- gpfs2 node : 10.10.10.11
2) 각 노드에서 공개키 교환
- gpfs1 node
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
|
[gpfs1:root] /.ssh] ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (//.ssh/id_rsa): Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in //.ssh/id_rsa.
Your public key has been saved in //.ssh/id_rsa.pub.
The key fingerprint is:
3a:9c:92:c2:d1:64:6d:52:49:dd:08:02:01:82:3d:16 root@gpfs1
The key's randomart image is:
+--[ RSA 2048]----+
|+oE+..++ o |
|o + +. o . |
| . .+ o |
| + o |
| . . S |
| . . o o |
| o o = |
| . . . |
| |
+-----------------+
[gpfs1:root] /.ssh] cp id_rsa.pub authorized_keys
[gpfs1:root] /.ssh] ssh gpfs2 'cat /.ssh/authorized_keys' >> /.ssh/authorized_keys
root@gpfs2's password:
[gpfs1:root] /.ssh] cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDD8R2ZcvjYAos8rDVKFOQ++bzfMAGjOTTRcHhlGLJrvgNzHHm/Ws50QaTDHk8/xKtRUoS+fqf3epBTPCWe3VHa+m4WzDlKLGqLRgM3AuHpYxN/d2ePaeiHV8uxjMG+Cm6WKyTPDbLzmpXYmom50C58oaYz227eJMBLlgUrO+yIVA3/k5rORk4fCEgAReaJyartkEgrkrw/+i48zl/eMj8ZFMx13aIHIzAHAnrTpkFlyIoSHqNDskYpgS7DfxbA64/RBKOqf4h6BCYcxmpCpVrtKCI4SXo6K8M/N2slqrDlVbP4uB4f0mm2y0Fm5q+QZW+QmA/gEI4FDAAAe2tDYjyN root@gpfs1
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN+ePBOlrkT6WPvtsdYoxuMbFcpHmEE5Ao9Rm65eni42ugHZnnMPS9VXyCuZm7FIlm6YAguHJ1mr+q3c+czDjdRzeRzfqAIk/mOdq/LmWlc2J5QsEqISdb+b9vzrv+3kH8c/FqljpO9rGM7Jau78JlWS3Pjx4nunmSs3BLaf6+MTldRW9w21zCH2u4d0D2f9zjks347ounmlNcttaERBwcV44x0Me+aLRfwpfUjhi9krflX1JSiKoGvP2uJdRHoXCXx09i7wOQHO6AWAMGGTEkxOA9PLbGnr9TtpymDvRLy9Ve4EioMLr+RDPeHhXIJDybwDz2CcQJeT1/juQRNWdb root@gpfs2
[gpfs1:root] /.ssh]
[gpfs1:root] /.ssh] ssh gpfs2
Last unsuccessful login: Fri Feb 19 11:25:55 KORST 2021 on /dev/vty0 from lpar8
Last login: Mon Feb 22 12:05:57 KORST 2021 on ssh from gpfs1
*******************************************************************************
* *
* *
* Welcome to AIX Version 7.1! *
* *
* *
* Please see the README file in /usr/lpp/bos for information pertinent to *
* this release of the AIX Operating System. *
* *
* *
*******************************************************************************
[YOU HAVE NEW MAIL]
[gpfs2:root] /]
|
cs |
- gpfs2 node
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
|
[gpfs2:root] /.ssh] ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (//.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in //.ssh/id_rsa.
Your public key has been saved in //.ssh/id_rsa.pub.
The key fingerprint is:
de:fe:c7:fe:99:85:0d:92:43:26:80:74:9e:73:33:08 root@gpfs2
The key's randomart image is:
+--[ RSA 2048]----+
| .E.o |
| .+ + |
| = = o |
| o * . |
| S + . |
| . . o + |
| . . .. o|
| . o +|
| ...o.+.|
+-----------------+
[gpfs2:root] /.ssh] cp id_rsa.pub authorized_keys
[gpfs2:root] /.ssh] ssh gpfs1 'cat /.ssh/authorized_keys' >> /.ssh/authorized_keys
The authenticity of host 'gpfs1 (10.10.10.10)' can't be established.
ECDSA key fingerprint is e4:fc:57:46:17:6a:e8:5e:70:5a:c4:34:b9:fc:60:1a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'gpfs1,10.10.10.10' (ECDSA) to the list of known hosts.
[gpfs2:root] /.ssh] cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN+ePBOlrkT6WPvtsdYoxuMbFcpHmEE5Ao9Rm65eni42ugHZnnMPS9VXyCuZm7FIlm6YAguHJ1mr+q3c+czDjdRzeRzfqAIk/mOdq/LmWlc2J5QsEqISdb+b9vzrv+3kH8c/FqljpO9rGM7Jau78JlWS3Pjx4nunmSs3BLaf6+MTldRW9w21zCH2u4d0D2f9zjks347ounmlNcttaERBwcV44x0Me+aLRfwpfUjhi9krflX1JSiKoGvP2uJdRHoXCXx09i7wOQHO6AWAMGGTEkxOA9PLbGnr9TtpymDvRLy9Ve4EioMLr+RDPeHhXIJDybwDz2CcQJeT1/juQRNWdb root@gpfs2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDD8R2ZcvjYAos8rDVKFOQ++bzfMAGjOTTRcHhlGLJrvgNzHHm/Ws50QaTDHk8/xKtRUoS+fqf3epBTPCWe3VHa+m4WzDlKLGqLRgM3AuHpYxN/d2ePaeiHV8uxjMG+Cm6WKyTPDbLzmpXYmom50C58oaYz227eJMBLlgUrO+yIVA3/k5rORk4fCEgAReaJyartkEgrkrw/+i48zl/eMj8ZFMx13aIHIzAHAnrTpkFlyIoSHqNDskYpgS7DfxbA64/RBKOqf4h6BCYcxmpCpVrtKCI4SXo6K8M/N2slqrDlVbP4uB4f0mm2y0Fm5q+QZW+QmA/gEI4FDAAAe2tDYjyN root@gpfs1
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN+ePBOlrkT6WPvtsdYoxuMbFcpHmEE5Ao9Rm65eni42ugHZnnMPS9VXyCuZm7FIlm6YAguHJ1mr+q3c+czDjdRzeRzfqAIk/mOdq/LmWlc2J5QsEqISdb+b9vzrv+3kH8c/FqljpO9rGM7Jau78JlWS3Pjx4nunmSs3BLaf6+MTldRW9w21zCH2u4d0D2f9zjks347ounmlNcttaERBwcV44x0Me+aLRfwpfUjhi9krflX1JSiKoGvP2uJdRHoXCXx09i7wOQHO6AWAMGGTEkxOA9PLbGnr9TtpymDvRLy9Ve4EioMLr+RDPeHhXIJDybwDz2CcQJeT1/juQRNWdb root@gpfs2
[gpfs2:root] /.ssh]
[gpfs2:root] /.ssh] ssh gpfs1
Last login: Mon Feb 22 12:06:49 KORST 2021 on ssh from gpfs2
*******************************************************************************
* *
* *
* Welcome to AIX Version 7.1! *
* *
* *
* Please see the README file in /usr/lpp/bos for information pertinent to *
* this release of the AIX Operating System. *
* *
* *
*******************************************************************************
[YOU HAVE NEW MAIL]
[gpfs1:root] /]
|
cs |
3) 디스크 속성
- Reservation policy = no_reserve 권장
- queue_depth = 6이상 (Disk의 queue full이 발생하면, 더 증설요구됨.)
- algorithm = round_robin 권장 (연결된 모든 패스에 I/O 분산처리됨)
|
1
2
3
4
5
6
7
8
9
10
11
|
[gpfs1:root] /home/gpfs] lsattr -El hdisk1
PCM PCM/friend/vscsi Path Control Module False
algorithm round_robin Algorithm True
hcheck_cmd test_unit_rdy Health Check Command True+
hcheck_interval 0 Health Check Interval True+
hcheck_mode nonactive Health Check Mode True+
max_transfer 0x40000 Maximum TRANSFER Size True
pvid 00c48b555f9e08c00000000000000000 Physical volume identifier False
queue_depth 16 Queue DEPTH True
reserve_policy no_reserve Reserve Policy True+
|
2. GPFS 구성
- GPFS 설치 및 패스 등록
- 테스트 환경 설치 경로 : /home/gpfs/
1) 설치 (모든 노드에서 GPFS 설치)
- 설치
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
[gpfs1:root] /home/gpfs] ls -lrt
total 126576
-rw-r----- 1 root system 64803443 Feb 22 05:39 Spectrum_Scale_Standard-5.0.3.0-ppc64-AIX-install.tgz
[gpfs1:root] /home/gpfs]
[gpfs1:root] /home/gpfs] gunzip -d *
[gpfs1:root] /home/gpfs] ls -lrt
total 249992
-rw-r----- 1 root system 127989760 Feb 22 05:39 Spectrum_Scale_Standard-5.0.3.0-ppc64-AIX-install.tar
[gpfs1:root] /home/gpfs] tar xvf Spectrum_Scale_Standard-5.0.3.0-ppc64-AIX-install.tar
x gpfs.base, 79464448 bytes, 155204 media blocks.
x gpfs.compression, 79872 bytes, 156 media blocks.
x gpfs.docs.data, 1835008 bytes, 3584 media blocks.
x gpfs.ext, 7836672 bytes, 15306 media blocks.
x gpfs.gskit, 38384640 bytes, 74970 media blocks.
x gpfs.msg.en_US, 371712 bytes, 726 media blocks.
x gpfs.license.std, 8192 bytes, 16 media blocks.
[gpfs1:root] /home/gpfs]
[gpfs1:root] /home/gpfs] installp -aYd . all >> 현재 디렉토리에 있는 모든 파일셋 설치
|
- 설치된 GPFS Fileset 확인
|
1
2
3
4
5
6
7
8
9
|
[gpfs1:root] /home/gpfs] lslpp -l |grep -i gpfs
gpfs.base 5.0.3.0 COMMITTED GPFS File Manager
gpfs.compression 5.0.3.0 COMMITTED GPFS Compresson Libraries
gpfs.ext 5.0.3.0 COMMITTED GPFS Extended Features
gpfs.gskit 8.0.50.86 COMMITTED GPFS GSKit Cryptography
gpfs.license.std 5.0.3.0 COMMITTED IBM Spectrum Scale Standard
gpfs.msg.en_US 5.0.3.0 COMMITTED GPFS Server Messages - U.S.
gpfs.base 5.0.3.0 COMMITTED GPFS File Manager
gpfs.docs.data 5.0.3.0 COMMITTED GPFS Server Manpages and
|
2) 패스 등록(모든 노드에서 패스등록)
- 모든 경로에서 GPFS Command를 수행할 수 있게, GPFS 명령어 디렉토리는 패스(PATH) 지정해준다.
- /usr/lpp/mmfs/bin 을 PATH 환경변수에 등록해준다.
- vi /.profile
PATH=$PATH:/usr/bin:/usr/dt/bin:/etc:/usr/ucb:$HOME/bin:/usr/bin/X11:/sbin:/usr/lpp/mmfs/bin:.
3. GPFS [Specturm Scale] 구성
- 구성스탭 : 노드 정의 > NSD(디스크 정의) > 클러스터 생성 > 파일시스템 생성 > 마운트 GPFS
- GPFS 구성을 위한 Config 파일들은 생성 명령어를 수행할 한쪽 노드에서만 생성하면 된다.
1) 노드 정의
- GPFS Node로 동작할 노드에 대한 정의를 Description 파일형태로 만든다.
- Quorum-manager의 역할 : 클러스터 노드의 Cluster alive condition(생존조건)을 관리하는 매니저.
|
1
2
3
|
[gpfs1:root] vi /home/gpfs/node.list
gpfs1:quorum-manager
gpfs2:quorum-manager
|
2) NSD 정의
- GPFS 파일시스템 용도 및 Tie Breakers disk로 사용될 디스크를 정의.
- 테스트 환경에서는 hdisk1~3 파일시스템용, hdisk4~6은 Tie 디스크 용도로 구성함.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
|
[gpfs1:root] /home/gpfs] vi disk.list
%nsd:
device=/dev/hdisk1
nsd=NSD001
usage=dataAndMetadata
pool=system
%nsd:
device=/dev/hdisk2
nsd=NSD002
usage=dataAndMetadata
pool=system
%nsd:
device=/dev/hdisk3
nsd=NSD003
usage=dataAndMetadata
pool=system
[gpfs1:root] /home/gpfs] vi tie.list
%nsd:
device=/dev/hdisk4
nsd=TIE_NSD001
usage=descOnly
pool=system
%nsd:
device=/dev/hdisk5
nsd=TIE_NSD002
usage=descOnly
pool=system
%nsd:
device=/dev/hdisk6
nsd=NSD003
usage=descOnly
pool=system
[gpfs1:root] /home/gpfs]
|
3) GPFS Cluster 생성
- --ccr-enable : 구성 및 파일시스템 정보 저장.
- -r : 접속 communication protocol
- -R : Copy communication protocol
- -C : Cluster name > "-C" 옵션을 주지 않으면, Primary node의 hostname으로 클러스터 이름 지정됨.
- -U : Domain name > remote cluster환경에서 사용되는 domain name.
- -A : Autoload > AIX 재기동시, 자동 기동옵션
|
1
2
3
4
5
6
7
8
9
|
[gpfs1:root] /home/gpfs] mmcrcluster -N ./node.list --ccr-enable -r /usr/bin/ssh -R /usr/bin/scp -C GPFS.SPECTRUM -U SCALE -A
mmcrcluster: Performing preliminary node verification ...
mmcrcluster: Processing quorum and other critical nodes ...
mmcrcluster: Finalizing the cluster data structures ...
mmcrcluster: Command successfully completed
mmcrcluster: 6027-1254 Warning: Not all nodes have proper GPFS license designations.
Use the mmchlicense command to designate licenses as needed.
mmcrcluster: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
|
4) GPFS 라이센스 동의 및 노드 상태 확인
- IBM Spectrum scale server licensing agreement 동의.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
[gpfs1:root] /home/gpfs] mmchlicense server --accept -N gpfs1,gpfs2
The following nodes will be designated as possessing server licenses:
gpfs1
gpfs2
mmchlicense: Command successfully completed
mmchlicense: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
[gpfs1:root] /home/gpfs] mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
-------------------------------------------------------------------------------------
1 gpfs1 0 0 2 down quorum node
2 gpfs2 0 0 2 down quorum node
[gpfs1:root] /home/gpfs]
|
5) GPFS Startup 및 노드 상태 확인
- startup 명령어 후, 노드 상태 변화 : down > arbitrating > active
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
[gpfs1:root] /home/gpfs] mmstartup -a
Mon Feb 22 13:38:46 KORST 2021: 6027-1642 mmstartup: Starting GPFS ...
[gpfs1:root] /home/gpfs]
[gpfs1:root] /home/gpfs] mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
-------------------------------------------------------------------------------------
1 gpfs1 0 0 2 down quorum node
2 gpfs2 0 0 2 down quorum node
[gpfs1:root] /home/gpfs] mmgetstate -aL
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
-------------------------------------------------------------------------------------
1 gpfs1 2 0 2 arbitrating quorum node
2 gpfs2 2 0 2 arbitrating quorum node
[gpfs1:root] /home/gpfs] mmgetstate -aL
mmgetstate
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
-------------------------------------------------------------------------------------
1 gpfs1 2 2 2 active quorum node
2 gpfs2 2 2 2 active quorum node
[gpfs1:root] /home/gpfs]
|
6) NSD 및 파일시스템 생성
- data용도의 NSD Description : /home/gpfs/disk.list
- tie용도의 NSD Description : /home/gpfs/tie.list -F /home/gpfs/
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
[gpfs1:root] /home/gpfs] mmcrnsd -F /home/gpfs/disk.list
mmcrnsd: Processing disk hdisk1
mmcrnsd: Processing disk hdisk2
mmcrnsd: Processing disk hdisk3
mmcrnsd: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
[gpfs1:root] /home/gpfs] mmcrnsd -F /home/gpfs/tie.list
mmcrnsd: Processing disk hdisk4
mmcrnsd: Processing disk hdisk5
mmcrnsd: Processing disk hdisk6
mmcrnsd: 6027-1680 Disk name NSD003 is already registered for use by GPFS.
mmcrnsd: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
[gpfs1:root] /home/gpfs]
[gpfs1:root] /home/gpfs] mmcrfs testFS -F /home/gpfs/disk.list -T /TEST -A yes -m1 -M2 -r1 -R2 -v yes
GPFS: 6027-531 The following disks of testFS will be formatted on node gpfs1:
NSD001: size 5120 MB
NSD002: size 5120 MB
NSD003: size 5120 MB
GPFS: 6027-540 Formatting file system ...
GPFS: 6027-535 Disks up to size 77.49 GB can be added to storage pool system.
Creating Inode File
Creating Allocation Maps
Creating Log Files
62 % complete on Mon Feb 22 13:49:02 2021
100 % complete on Mon Feb 22 13:49:04 2021
Clearing Inode Allocation Map
Clearing Block Allocation Map
Formatting Allocation Map for storage pool system
GPFS: 6027-572 Completed creation of file system /dev/testFS.
mmcrfs: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
|
7) Tie breakers disk 지정
- GPFS 4.x 부터 온라인상으로 Tie breakers disk를 지정할 수 있다.
|
1
2
3
4
5
6
7
8
9
10
11
12
|
[gpfs1:root] /home/gpfs] lspv
hdisk0 00c48b55df919dc1 rootvg active
hdisk1 00c48b555f9e08c0 NSD001
hdisk2 00c48b555f9e09b0 NSD002
hdisk3 00c48b555f9e0aaa NSD003
hdisk4 00c48b55ecc3ae3f TIE_NSD001
hdisk5 00c48b55ecc47bd3 TIE_NSD002
hdisk6 00c48b55ecc5310c TIE_NSD003
[gpfs1:root] /home/gpfs] mmchconfig tiebreakerDisks="TIE_NSD001;TIE_NSD002;TIE_NSD003"
mmchconfig: Command successfully completed
mmchconfig: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
|
8) GPFS 파일시스템 마운트
- mmmount all -a : GPFS 전체 노드에 모든 GPFS 파일시스템 마운트
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
[gpfs1:root] /home/gpfs] mmmount all -a
Mon Feb 22 13:57:54 KORST 2021: 6027-1623 mmmount: Mounting file systems ...
[gpfs1:root] /home/gpfs] df -gP
Filesystem GB blocks Used Available Capacity Mounted on
/dev/hd4 1.00 0.37 0.63 37% /
/dev/hd2 5.00 3.16 1.84 64% /usr
/dev/hd9var 0.41 0.26 0.15 64% /var
/dev/hd3 1.00 0.00 1.00 1% /tmp
/dev/hd1 10.00 0.46 9.54 5% /home
/proc - - - - /proc
/dev/livedump 0.25 0.00 0.25 1% /var/adm/ras/livedump
/dev/testFS 15.00 1.49 13.51 10% /TEST
[gpfs1:root] /home/gpfs]
[gpfs1:root] /home/gpfs] lsfs |grep -i mmfs
/dev/testFS - /TEST mmfs -- rw,mtime,relatime,dev=testFS no no
|
반응형