'GPFS' 카테고리의 글 목록

GPFS 5.0 구성(Configuration)

GPFS/config2021. 2. 21. 09:08

 
* Written by Jung Byung Wook

구성환경 
 - 2NODE
- Hostname : gpfs1, gpfs2
- IP Addr : 10.10.10.10, 10.10.10.11
 - Data disk : 5G * 3EA
 - Tie breakers disk : 5G * 3EA

0. 사전 준비단계

1) 노드 구성 계획

몇 개의 노드로 구성할 것인지?

2) Inter-Connect(Heartbeat) Interface

1G Interface로도 충분하나, 10G 구성도 무방.
Heartbeat Interface의 etherchannel은 Active / Standby로 구성하는 것이 일반적.

3) GPFS 공유파일시스템(MMFS)의 용량과 각 NSD disk(LUN)은 몇개로 가져갈 것인지?

GPFS 파일시스템의 증설단위는 1 NSD(LUN)이며, 사용용량에 맞게 NSD를 구성해야한다.
NSD disk(LUN)의 갯수는 각 사이트 정책/관행에 따라 갯수가 정해지나, 너무 적은 NSD는 성능저하를 줄수 있다.

4) Remote GPFS Cluster를 사용할 것인지?

일반적인 사이트에서는 사용하지 않음.

5) Communication protocol(통신방식)은 어떤 것을 사용할 것인지?

SSH / RSH 방식이 있다.
RSH의 경우, 보안취약점이 존재하여, SSH 암호화방식을 통한 Communication을 권장한다.

1. 상호 노드간 패스워드 없이 통신할 수 있는 연결구성 및 디스크 속성 설정

권장구성방식인 SSH를 통해, GPFS를 구성하기 위한 방식이며, 이 스텝에서는 각 노드가 Password 없이 접근할 수 있게, 노드 상호간의 공개키(Public key)를 공유한다.

1) /etc/hosts에 상호 노드 등록

gpfs1 node : 10.10.10.10
gpfs2 node : 10.10.10.11

2) 각 노드에서 공개키 교환

gpfs1 node

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

[gpfs1:root] /.ssh] ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (//.ssh/id_rsa): Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in //.ssh/id_rsa.
Your public key has been saved in //.ssh/id_rsa.pub.
The key fingerprint is:
3a:9c:92:c2:d1:64:6d:52:49:dd:08:02:01:82:3d:16 root@gpfs1
The key's randomart image is:
+--[ RSA 2048]----+
|+oE+..++ o       |
|o +  +. o .      |
| . .+ o          |
|   + o           |
|  . .   S        |
| . . o o         |
|  o o =          |
|   . . .         |
|                 |
+-----------------+
[gpfs1:root] /.ssh] cp id_rsa.pub authorized_keys
[gpfs1:root] /.ssh] ssh gpfs2 'cat /.ssh/authorized_keys' >> /.ssh/authorized_keys 
root@gpfs2's password: 
[gpfs1:root] /.ssh] cat authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDD8R2ZcvjYAos8rDVKFOQ++bzfMAGjOTTRcHhlGLJrvgNzHHm/Ws50QaTDHk8/xKtRUoS+fqf3epBTPCWe3VHa+m4WzDlKLGqLRgM3AuHpYxN/d2ePaeiHV8uxjMG+Cm6WKyTPDbLzmpXYmom50C58oaYz227eJMBLlgUrO+yIVA3/k5rORk4fCEgAReaJyartkEgrkrw/+i48zl/eMj8ZFMx13aIHIzAHAnrTpkFlyIoSHqNDskYpgS7DfxbA64/RBKOqf4h6BCYcxmpCpVrtKCI4SXo6K8M/N2slqrDlVbP4uB4f0mm2y0Fm5q+QZW+QmA/gEI4FDAAAe2tDYjyN root@gpfs1
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN+ePBOlrkT6WPvtsdYoxuMbFcpHmEE5Ao9Rm65eni42ugHZnnMPS9VXyCuZm7FIlm6YAguHJ1mr+q3c+czDjdRzeRzfqAIk/mOdq/LmWlc2J5QsEqISdb+b9vzrv+3kH8c/FqljpO9rGM7Jau78JlWS3Pjx4nunmSs3BLaf6+MTldRW9w21zCH2u4d0D2f9zjks347ounmlNcttaERBwcV44x0Me+aLRfwpfUjhi9krflX1JSiKoGvP2uJdRHoXCXx09i7wOQHO6AWAMGGTEkxOA9PLbGnr9TtpymDvRLy9Ve4EioMLr+RDPeHhXIJDybwDz2CcQJeT1/juQRNWdb root@gpfs2
[gpfs1:root] /.ssh] 
[gpfs1:root] /.ssh] ssh gpfs2
Last unsuccessful login: Fri Feb 19 11:25:55 KORST 2021 on /dev/vty0 from lpar8
Last login: Mon Feb 22 12:05:57 KORST 2021 on ssh from gpfs1
*******************************************************************************
*                                                                             *
*                                                                             *
*  Welcome to AIX Version 7.1!                                                *
*                                                                             *
*                                                                             *
*  Please see the README file in /usr/lpp/bos for information pertinent to    *
*  this release of the AIX Operating System.                                  *
*                                                                             *
*                                                                             *
*******************************************************************************
[YOU HAVE NEW MAIL]
[gpfs2:root] /] 
ipter

cs

gpfs2 node

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

[gpfs2:root] /.ssh] ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (//.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in //.ssh/id_rsa.
Your public key has been saved in //.ssh/id_rsa.pub.
The key fingerprint is:
de:fe:c7:fe:99:85:0d:92:43:26:80:74:9e:73:33:08 root@gpfs2
The key's randomart image is:
+--[ RSA 2048]----+
|     .E.o        |
|      .+ +       |
|        = = o    |
|         o * .   |
|        S   + .  |
|       . .   o + |
|        . .  .. o|
|         .    o +|
|          ...o.+.|
+-----------------+
[gpfs2:root] /.ssh] cp id_rsa.pub authorized_keys
[gpfs2:root] /.ssh] ssh gpfs1 'cat /.ssh/authorized_keys' >> /.ssh/authorized_keys
The authenticity of host 'gpfs1 (10.10.10.10)' can't be established.
ECDSA key fingerprint is e4:fc:57:46:17:6a:e8:5e:70:5a:c4:34:b9:fc:60:1a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'gpfs1,10.10.10.10' (ECDSA) to the list of known hosts.
[gpfs2:root] /.ssh] cat authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN+ePBOlrkT6WPvtsdYoxuMbFcpHmEE5Ao9Rm65eni42ugHZnnMPS9VXyCuZm7FIlm6YAguHJ1mr+q3c+czDjdRzeRzfqAIk/mOdq/LmWlc2J5QsEqISdb+b9vzrv+3kH8c/FqljpO9rGM7Jau78JlWS3Pjx4nunmSs3BLaf6+MTldRW9w21zCH2u4d0D2f9zjks347ounmlNcttaERBwcV44x0Me+aLRfwpfUjhi9krflX1JSiKoGvP2uJdRHoXCXx09i7wOQHO6AWAMGGTEkxOA9PLbGnr9TtpymDvRLy9Ve4EioMLr+RDPeHhXIJDybwDz2CcQJeT1/juQRNWdb root@gpfs2
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDD8R2ZcvjYAos8rDVKFOQ++bzfMAGjOTTRcHhlGLJrvgNzHHm/Ws50QaTDHk8/xKtRUoS+fqf3epBTPCWe3VHa+m4WzDlKLGqLRgM3AuHpYxN/d2ePaeiHV8uxjMG+Cm6WKyTPDbLzmpXYmom50C58oaYz227eJMBLlgUrO+yIVA3/k5rORk4fCEgAReaJyartkEgrkrw/+i48zl/eMj8ZFMx13aIHIzAHAnrTpkFlyIoSHqNDskYpgS7DfxbA64/RBKOqf4h6BCYcxmpCpVrtKCI4SXo6K8M/N2slqrDlVbP4uB4f0mm2y0Fm5q+QZW+QmA/gEI4FDAAAe2tDYjyN root@gpfs1
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN+ePBOlrkT6WPvtsdYoxuMbFcpHmEE5Ao9Rm65eni42ugHZnnMPS9VXyCuZm7FIlm6YAguHJ1mr+q3c+czDjdRzeRzfqAIk/mOdq/LmWlc2J5QsEqISdb+b9vzrv+3kH8c/FqljpO9rGM7Jau78JlWS3Pjx4nunmSs3BLaf6+MTldRW9w21zCH2u4d0D2f9zjks347ounmlNcttaERBwcV44x0Me+aLRfwpfUjhi9krflX1JSiKoGvP2uJdRHoXCXx09i7wOQHO6AWAMGGTEkxOA9PLbGnr9TtpymDvRLy9Ve4EioMLr+RDPeHhXIJDybwDz2CcQJeT1/juQRNWdb root@gpfs2
[gpfs2:root] /.ssh]     
[gpfs2:root] /.ssh] ssh gpfs1
Last login: Mon Feb 22 12:06:49 KORST 2021 on ssh from gpfs2
*******************************************************************************
*                                                                             *
*                                                                             *
*  Welcome to AIX Version 7.1!                                                *
*                                                                             *
*                                                                             *
*  Please see the README file in /usr/lpp/bos for information pertinent to    *
*  this release of the AIX Operating System.                                  *
*                                                                             *
*                                                                             *
*******************************************************************************
[YOU HAVE NEW MAIL]
[gpfs1:root] /] 
Colored by Color Scripter

cs

3) 디스크 속성

Reservation policy = no_reserve 권장
queue_depth = 6이상 (Disk의 queue full이 발생하면, 더 증설요구됨.)
algorithm = round_robin 권장 (연결된 모든 패스에 I/O 분산처리됨)

1
2
3
4
5
6
7
8
9
10
11

[gpfs1:root] /home/gpfs] lsattr -El hdisk1
PCM             PCM/friend/vscsi                 Path Control Module        False
algorithm       round_robin                      Algorithm                  True
hcheck_cmd      test_unit_rdy                    Health Check Command       True+
hcheck_interval 0                                Health Check Interval      True+
hcheck_mode     nonactive                        Health Check Mode          True+
max_transfer    0x40000                          Maximum TRANSFER Size      True
pvid            00c48b555f9e08c00000000000000000 Physical volume identifier False
queue_depth     16                                Queue DEPTH                True
reserve_policy  no_reserve                       Reserve Policy             True+
 
 

 

2. GPFS 구성

GPFS 설치 및 패스 등록
테스트 환경 설치 경로 : /home/gpfs/

1) 설치 (모든 노드에서 GPFS 설치)

설치

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

[gpfs1:root] /home/gpfs] ls -lrt
total 126576
-rw-r-----    1 root     system     64803443 Feb 22 05:39 Spectrum_Scale_Standard-5.0.3.0-ppc64-AIX-install.tgz
[gpfs1:root] /home/gpfs]
[gpfs1:root] /home/gpfs] gunzip -d *
[gpfs1:root] /home/gpfs] ls -lrt
total 249992
-rw-r-----    1 root     system    127989760 Feb 22 05:39 Spectrum_Scale_Standard-5.0.3.0-ppc64-AIX-install.tar
[gpfs1:root] /home/gpfs] tar xvf Spectrum_Scale_Standard-5.0.3.0-ppc64-AIX-install.tar 
x gpfs.base, 79464448 bytes, 155204 media blocks.
x gpfs.compression, 79872 bytes, 156 media blocks.
x gpfs.docs.data, 1835008 bytes, 3584 media blocks.
x gpfs.ext, 7836672 bytes, 15306 media blocks.
x gpfs.gskit, 38384640 bytes, 74970 media blocks.
x gpfs.msg.en_US, 371712 bytes, 726 media blocks.
x gpfs.license.std, 8192 bytes, 16 media blocks.
[gpfs1:root] /home/gpfs] 
[gpfs1:root] /home/gpfs] installp -aYd . all >> 현재 디렉토리에 있는 모든 파일셋 설치
 

 

설치된 GPFS Fileset 확인

1
2
3
4
5
6
7
8
9

[gpfs1:root] /home/gpfs] lslpp -l |grep -i gpfs
  gpfs.base                  5.0.3.0  COMMITTED  GPFS File Manager
  gpfs.compression           5.0.3.0  COMMITTED  GPFS Compresson Libraries
  gpfs.ext                   5.0.3.0  COMMITTED  GPFS Extended Features
  gpfs.gskit               8.0.50.86  COMMITTED  GPFS GSKit Cryptography
  gpfs.license.std           5.0.3.0  COMMITTED  IBM Spectrum Scale Standard
  gpfs.msg.en_US             5.0.3.0  COMMITTED  GPFS Server Messages - U.S.
  gpfs.base                  5.0.3.0  COMMITTED  GPFS File Manager
  gpfs.docs.data             5.0.3.0  COMMITTED  GPFS Server Manpages and

 

2) 패스 등록(모든 노드에서 패스등록)

모든 경로에서 GPFS Command를 수행할 수 있게, GPFS 명령어 디렉토리는 패스(PATH) 지정해준다.
/usr/lpp/mmfs/bin 을 PATH 환경변수에 등록해준다.
vi /.profile

PATH=$PATH:/usr/bin:/usr/dt/bin:/etc:/usr/ucb:$HOME/bin:/usr/bin/X11:/sbin:/usr/lpp/mmfs/bin:.

3. GPFS [Specturm Scale] 구성

구성스탭 : 노드 정의 > NSD(디스크 정의) > 클러스터 생성 > 파일시스템 생성 > 마운트 GPFS
GPFS 구성을 위한 Config 파일들은 생성 명령어를 수행할 한쪽 노드에서만 생성하면 된다.

1) 노드 정의

GPFS Node로 동작할 노드에 대한 정의를 Description 파일형태로 만든다.
Quorum-manager의 역할 : 클러스터 노드의 Cluster alive condition(생존조건)을 관리하는 매니저.

1
2
3

[gpfs1:root] vi /home/gpfs/node.list
gpfs1:quorum-manager
gpfs2:quorum-manager

 

2) NSD 정의

GPFS 파일시스템 용도 및 Tie Breakers disk로 사용될 디스크를 정의.
테스트 환경에서는 hdisk1~3 파일시스템용, hdisk4~6은 Tie 디스크 용도로 구성함.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

[gpfs1:root] /home/gpfs] vi disk.list
%nsd:
device=/dev/hdisk1
nsd=NSD001
usage=dataAndMetadata
pool=system
%nsd:
device=/dev/hdisk2
nsd=NSD002
usage=dataAndMetadata
pool=system
%nsd:
device=/dev/hdisk3
nsd=NSD003
usage=dataAndMetadata
pool=system
 
[gpfs1:root] /home/gpfs] vi tie.list
%nsd:
device=/dev/hdisk4
nsd=TIE_NSD001
usage=descOnly
pool=system
%nsd:
device=/dev/hdisk5
nsd=TIE_NSD002
usage=descOnly
pool=system
%nsd:
device=/dev/hdisk6
nsd=NSD003
usage=descOnly
pool=system
[gpfs1:root] /home/gpfs]

 

3) GPFS Cluster 생성

--ccr-enable : 구성 및 파일시스템 정보 저장.
-r : 접속 communication protocol
-R : Copy communication protocol
-C : Cluster name > "-C" 옵션을 주지 않으면, Primary node의 hostname으로 클러스터 이름 지정됨.
-U : Domain name > remote cluster환경에서 사용되는 domain name.
-A : Autoload > AIX 재기동시, 자동 기동옵션

1
2
3
4
5
6
7
8
9

[gpfs1:root] /home/gpfs] mmcrcluster -N ./node.list --ccr-enable -r /usr/bin/ssh -R /usr/bin/scp -C GPFS.SPECTRUM -U SCALE -A
mmcrcluster: Performing preliminary node verification ...
mmcrcluster: Processing quorum and other critical nodes ...
mmcrcluster: Finalizing the cluster data structures ...
mmcrcluster: Command successfully completed
mmcrcluster: 6027-1254 Warning: Not all nodes have proper GPFS license designations.
    Use the mmchlicense command to designate licenses as needed.
mmcrcluster: 6027-1371 Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
 

 

4) GPFS 라이센스 동의 및 노드 상태 확인

IBM Spectrum scale server licensing agreement 동의.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

[gpfs1:root] /home/gpfs] mmchlicense server --accept -N gpfs1,gpfs2       
 
The following nodes will be designated as possessing server licenses:
        gpfs1
        gpfs2
mmchlicense: Command successfully completed
mmchlicense: 6027-1371 Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[gpfs1:root] /home/gpfs] mmgetstate -aL
 
 Node number  Node name       Quorum  Nodes up  Total nodes  GPFS state   Remarks    
-------------------------------------------------------------------------------------
       1      gpfs1              0        0          2       down         quorum node
       2      gpfs2              0        0          2       down         quorum node
[gpfs1:root] /home/gpfs] 
 

 

5) GPFS Startup 및 노드 상태 확인

startup 명령어 후, 노드 상태 변화 : down > arbitrating > active

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

[gpfs1:root] /home/gpfs] mmstartup -a
 
Mon Feb 22 13:38:46 KORST 2021: 6027-1642 mmstartup: Starting GPFS ...
[gpfs1:root] /home/gpfs] 
[gpfs1:root] /home/gpfs] mmgetstate -aL
 
 Node number  Node name       Quorum  Nodes up  Total nodes  GPFS state   Remarks    
-------------------------------------------------------------------------------------
       1      gpfs1              0        0          2       down         quorum node
       2      gpfs2              0        0          2       down         quorum node
 
[gpfs1:root] /home/gpfs] mmgetstate -aL
 Node number  Node name       Quorum  Nodes up  Total nodes  GPFS state   Remarks    
-------------------------------------------------------------------------------------
       1      gpfs1              2        0          2       arbitrating  quorum node
       2      gpfs2              2        0          2       arbitrating  quorum node
[gpfs1:root] /home/gpfs] mmgetstate -aL
mmgetstate
 
 Node number  Node name       Quorum  Nodes up  Total nodes  GPFS state   Remarks    
-------------------------------------------------------------------------------------
       1      gpfs1              2        2          2       active       quorum node
       2      gpfs2              2        2          2       active       quorum node
[gpfs1:root] /home/gpfs] 
 

 

6) NSD 및 파일시스템 생성

data용도의 NSD Description : /home/gpfs/disk.list
tie용도의 NSD Description : /home/gpfs/tie.list -F /home/gpfs/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

[gpfs1:root] /home/gpfs] mmcrnsd -F /home/gpfs/disk.list
mmcrnsd: Processing disk hdisk1
mmcrnsd: Processing disk hdisk2
mmcrnsd: Processing disk hdisk3
mmcrnsd: 6027-1371 Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[gpfs1:root] /home/gpfs] mmcrnsd -F /home/gpfs/tie.list 
mmcrnsd: Processing disk hdisk4
mmcrnsd: Processing disk hdisk5
mmcrnsd: Processing disk hdisk6
mmcrnsd: 6027-1680 Disk name NSD003 is already registered for use by GPFS.
mmcrnsd: 6027-1371 Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[gpfs1:root] /home/gpfs] 
[gpfs1:root] /home/gpfs] mmcrfs testFS -F /home/gpfs/disk.list -T /TEST -A yes -m1 -M2 -r1 -R2 -v yes
 
GPFS: 6027-531 The following disks of testFS will be formatted on node gpfs1:
    NSD001: size 5120 MB
    NSD002: size 5120 MB
    NSD003: size 5120 MB
GPFS: 6027-540 Formatting file system ...
GPFS: 6027-535 Disks up to size 77.49 GB can be added to storage pool system.
Creating Inode File
Creating Allocation Maps
Creating Log Files
  62 % complete on Mon Feb 22 13:49:02 2021
 100 % complete on Mon Feb 22 13:49:04 2021
Clearing Inode Allocation Map
Clearing Block Allocation Map
Formatting Allocation Map for storage pool system
GPFS: 6027-572 Completed creation of file system /dev/testFS.
mmcrfs: 6027-1371 Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
 

 

7) Tie breakers disk 지정

GPFS 4.x 부터 온라인상으로 Tie breakers disk를 지정할 수 있다.

1
2
3
4
5
6
7
8
9
10
11
12

[gpfs1:root] /home/gpfs] lspv
hdisk0          00c48b55df919dc1                    rootvg          active      
hdisk1          00c48b555f9e08c0                    NSD001                      
hdisk2          00c48b555f9e09b0                    NSD002                      
hdisk3          00c48b555f9e0aaa                    NSD003                      
hdisk4          00c48b55ecc3ae3f                    TIE_NSD001                  
hdisk5          00c48b55ecc47bd3                    TIE_NSD002                  
hdisk6          00c48b55ecc5310c                    TIE_NSD003   
[gpfs1:root] /home/gpfs] mmchconfig tiebreakerDisks="TIE_NSD001;TIE_NSD002;TIE_NSD003"
mmchconfig: Command successfully completed
mmchconfig: 6027-1371 Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
 

 

 
 8) GPFS 파일시스템 마운트
mmmount all -a : GPFS 전체 노드에 모든 GPFS 파일시스템 마운트

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

[gpfs1:root] /home/gpfs] mmmount all -a 
Mon Feb 22 13:57:54 KORST 2021: 6027-1623 mmmount: Mounting file systems ...
 
[gpfs1:root] /home/gpfs] df -gP
Filesystem    GB blocks      Used Available Capacity Mounted on
/dev/hd4           1.00      0.37      0.63      37% /
/dev/hd2           5.00      3.16      1.84      64% /usr
/dev/hd9var        0.41      0.26      0.15      64% /var
/dev/hd3           1.00      0.00      1.00       1% /tmp
/dev/hd1          10.00      0.46      9.54       5% /home
/proc                 -         -         -       - /proc
/dev/livedump      0.25      0.00      0.25       1% /var/adm/ras/livedump
/dev/testFS       15.00      1.49     13.51      10% /TEST
[gpfs1:root] /home/gpfs] 
[gpfs1:root] /home/gpfs] lsfs |grep -i mmfs
/dev/testFS     -          /TEST                  mmfs  --      rw,mtime,relatime,dev=testFS no   no 
 
 

 

저작자표시

Posted by 정병욱

‹ Prev 1 Next ›

GPFS 5.0 구성(Configuration)

0. 사전 준비단계

1. 상호 노드간 패스워드 없이 통신할 수 있는 연결구성 및 디스크 속성 설정

2. GPFS 구성

3. GPFS [Specturm Scale] 구성

티스토리툴바