hp小型機(jī) mc集群介紹

1、,高可用性系統(tǒng)介紹(MC/ServiceGuard),HP 小型機(jī)培訓(xùn),,HA (High Availability)定義,A system is highly available if a single component or resource failure interrupts the system for only a brief time,What cause a system to go down,planned reas

2、ons: reconfigure the kernal apply patchs perform hardware and software upgrades perform full system backups perform system maintenance,unplanned reasons: hardware failures: CPU, Memory, Di

3、sk drives , LAN Card, Cable,Disk Controller cards etc system panics application errors power failures user errors,% of Failures,Hardware,,High Availability Terms,Downtime: any amount of ti

4、me when the application is unavailable (planned or unplanned) planned: customer plans to bring down the system unplanned: due to an unplanned event or outage,High available

5、: A system that can be recover quickly from all or most resource failures. The application may become unavailable, but only for a short period of time. DownTime: 5 min 50 min

6、 8.8 hours 12 hours 24 hours 3.6 days 7.2 days 10.8 days Availability: 99.999% 99.99% 99.9% 99.86% 99.73% 99.0% 98% 97%,Outage: an occurrence that renders

7、 an application unavailable when it is expected to be available (Hardware,software,user,environmental Problem),Availability: The time that application is can be used during times when when it is

8、 expected to be useble. Availability igored planned or scheduled downtime and is expressed as a igored planned or scheduled downtime and is expressed as a percentag

9、e,Fault tolerant: These system protect against hardware failures by providing totally redundant hardware in a single system,Standard reliability: A system that relies only on basic har

10、dware;there are no additional precautions taken to protect against an outage. (97-98%),,SPOF(Single Points of Failure),,,,,SPOF Solution,CPU

11、Memory Cluster,Disk Mirror and RAID,Interface Cards Mirror and PV Links,LAN, NICs

12、 Redundant LANs and LANIC,Power UPS,SPU,LAN,Power,CPU,Memory,NIC,Disk,SCSI Controller,,,root,root mirror,,High Availability Solution,,Continuously Available Systems

13、 future HP products,Highly Available System MC/ServiceGuard MC/LockManager

14、 OnLine JFS Process Resource Manager ClusterView,Protected Data

15、 MirrorDisk/UX HP DiskArray/EMC DiskArray JFS,Reliable system

16、 HP9000 systems HP peripherals HP-UX,Cluster(群集),c

17、luster is a networked group of nodes (hosts) which monitor each other in order to ensure that interruptions to the availability of application running on these nodes are kept small .,,,Pkg A,,,Pkg B,,,,,,,,,,,,,,,,,,,r

18、oot,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated Heatbeat LAN,Primary Lan :Heatbeat/Data,,,Standby LAN :Heatbeat/Data,,Standby LAN :Heatbeat/Data,,Node 1,Node 2,,,,cli

19、ent,,,Pkg A,,,,Pkg B,,,,,,,,,,,,,,,,,,,root,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated Heatbeat LAN,Primary Lan :Heatbeat/Data,,,Standby LAN :Heatbeat/Data,,Standby

20、 LAN :Heatbeat/Data,,Node 1,Node 2,,,Sample cluster (two-nodes),,cmcld,Package概念,Package: an application along with its programs and resources (volume group, target node, Network address, control Script

21、and services) Floating IP: application IP address(attach to host NIC). Client connect to host through the floating IP Original node: adoptive node : a pac

22、kage can have several adoptive nodes,LVM,PV links: dual links(hardware paths) to the same disk such that if one link fails, LVM automaticlly rerouteds the I/O to an alternate path MC/SG VG: if a VG

23、 is a part of an MC/SG, only one node will be allowed to access the VG at a time Exclusive Mode Activation: in general, you must provide at least one volume group for each package,Sample cluster (8 nodes cluster)

24、 (Max 16 nodes),,,,,,,,,,,,,,,,,,,,,,,,,,,,,,WAN,,,client,DiskArray,standby,EMC symmetrix,HP XP256,Cluster reformation,,System B,,Pkg 3,System C,Pkg 4,,,clusterReformation,,,System A leave,S

25、ystem A join,,clusterReformation,Lock Disk概念,The cluster lock is a disk located in a volume group shared by all nodes in the cluster,required for 2-nodes clusteroptional for 3 or 4 nodes clusternot supported for 5 nod

26、e or more cluster,,,Pkg A,,,,Pkg B,,,,,,,,,,,root,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated Heatbeat LAN,Node 1,Node 2,,,,Lock Disk,X,,,,,,,model 10, mode20, model3

27、0,FC60等DiskArray 需要單獨(dú)另配一塊鎖盤AutoRaid12H:其中的一個(gè)物理卷可用作鎖盤 不需單獨(dú)另配一塊鎖盤,,MC處理的失效類型,Node(host) failover : SPU (CPU, Memory, disk I/O, Power)LA

28、N failover: LAN Card, LAN link,,Pkg A(float IP_A),,,,Pkg B(float IP_B),,,,,,,,,,,,,,,,,,,root,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated Heatbeat

29、 LAN,Primary Lan :Heatbeat/Data,,,Standby LAN :Heatbeat/Data,,Standby LAN :Heatbeat/Data,,Node 1,Node 2,Pkg A(float IP_A),X,,,,Client,Application Switch Demo(SPU Failure),Pkg A client,,,,,Pkg A,,,,Pkg B,,,,,,,,,,,,,

30、,,,,,,root,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated Heatbeat LAN,Primary Lan :Heatbeat/Data,,,Standby LAN :Heatbeat/Data,,Standby LAN :Heatbeat/Data,,Node 1,Node 2

31、,Pkg A(float IP),X,,,,Client,Application Switch Demo(SPU Failure),Pkg A client,,,,Pkg A,,,,Pkg B,,,,,,,,,,,,,,,,,,,root,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated H

32、eatbeat LAN,Primary Lan :Heatbeat/Data,,,Standby LAN :Heatbeat/Data,,Standby LAN :Heatbeat/Data,,Node 1,Node 2,Pkg A,,,,,,,,Application Switch Demo(LAN Failure),Client,X,,應(yīng)用切換時(shí)間,activate_volume_group,,,Pkg A,,,,Pkg

33、B,,,,,,,,,,,root,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated Heatbeat LAN,Node 1,Node 2,Pkg A,X,,,,umount_fs,remove_ip_address,customer_defined_halt_cmds,halt_services

34、,deactivate_volume_group,check_and_mount,add_ip_address,customer_defined_run_cmds,start_services,MC管理命令(1): Cluster startup,1. Automatic->/etc/rc.config.d/cmcluster AUTOSTART_CMCLD=1 2. Manual: cmr

35、uncl 3. Single-node: cmruncl -n hostname,MC管理命令(2): cluster view:,CLUSTER STATUS cluster1 up NODE STATUS STATE systemA

36、 up running PACKAGE STATUS STATE PKG_SWITCH NODE pkg_A up running enabled systemA pkg_B

37、 up running enabled systemB NODE STATUS STATE systemB up running,cmviewcl,MC管理命令(3): cluster stop:

38、,cmhaltcl [-f] forcely close database and applicationcmviewcl CLUSTER STATUS cluster1 down,MC管理命令(4): node stop & join,node stop: cmhaltn

39、ode [-f] -n systemBCLUSTER STATUS cluster1 upNODE STATUS STATE systemA up running

40、 PACKAGE STATUS STATE PKG_SWITCH NODE pkg_A up running enabled systemA pkg_B up running enabled

41、 systemA NODE STATUS STATE systemB down haltednode start : cmrunnode systemBCLUSTER STATUS

42、 cluster1 up NODE STATUS STATE systemA up running PACKAGE STATUS STATE PKG_SW

43、ITCH NODE pkg_A up running enabled systemA pkg_B up running enabled systemA NODE

44、 STATUS STATE systemB up running,MC管理命令(5): package stop,PACKAGE STATUS STATE PKG_SWITCH NODE pkg_A

45、 up running enabled systemA pkg_B up running enabled systemBcmhaltpkg pkg_BPACKAGE STATUS STATE PKG_SWIT

46、CH NODE pkg_A up running enabled systemA pkg_B down unowned disabled unowned,MC管理命令(6): package status chang

47、e & start,PACKAGE STATUS STATE PKG_SWITCH NODE pkg_A up running enabled systemA pkg_B down unowned

48、 disabled unownedcmrunpkg -n systemB pkg_B --------> not successfulcmrunnode systemBcmrunpkg -n systemA pkg_B PACKAGE STATUS STATE PKG_SWITCH NODE

49、 pkg_A up running enabled systemA pkg_B up running disabled systemAcmmodpkg -e pkg_B PACKAGE

50、 STATUS STATE PKG_SWITCH NODE pkg_A up running enabled systemA pkg_B up running enabled sys

51、temA,,MC測(cè)試方法,MC/ServiceGuard軟件安裝: swlist －> B3935BA B.11.00MC/ServiceGuard運(yùn)行: cmruncl cmviewcl手工切換包: cmhaltpkg pkg_name cmrunpkg pkg_name手工停止節(jié)點(diǎn): cmhaltnode -f [node_na

52、me]操作系統(tǒng)故障: shutdown -r -y 0,注意事項(xiàng)：電源連接,,,,,,,,,,,,,,,,,,,,,,,,,,,,,N,L,G,,,,,,,,,,,,,,,,,,UPS,N,N,專用地線,輸入端,G,L,G,電源箱,G,N,L,G,N,L,G:：地線N：零線L：火線,,,,220v,< 1.0 v,電阻小于1歐姆,,,,,,L,,,,15A,15A,15A,零線與地線不能接在一起地線要求直接接地,,,,

53、,,Standby LAN Card,注意事項(xiàng)：心跳線網(wǎng)絡(luò)連接(switch),PrimaryLAN Card,,,,,Pkg A,,,,Pkg B,,,,,,,,root,root,HeartBeat LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Node 1,Node 2,,,,,,,,,,,,,,Pkg A,,,,Pkg B,,,,,,,,root,root,HeartBeat LAN

54、 Cards,Pkg A Disks,Pkg B Disks,,,,,Node 1,Node 2,,,,,,,,,,,,,,,,12345678,12345678,,,,,,,,,1---32---6,Directconnect,SPOF,,注意事項(xiàng),1.應(yīng)用穩(wěn)定: MC不能保護(hù)應(yīng)用程序本身的缺陷、OS的bug等等。 應(yīng)用在單機(jī)上穩(wěn)定運(yùn)行后再配置MC系統(tǒng) 2.

55、數(shù)據(jù)可靠性: MC不能保證數(shù)據(jù)的可用性。 采用適合的磁盤技術(shù)保護(hù)數(shù)據(jù)。3.應(yīng)用系統(tǒng)整體可靠性: MC只保證主機(jī)系統(tǒng)的高可靠性。 整個(gè)應(yīng)用系統(tǒng)的可靠性需要考慮到各方面的單點(diǎn)故障SPOF 如

56、采用可靠性的網(wǎng)絡(luò)，中間件產(chǎn)品，客戶端程序等。4.主機(jī)處理能力：考慮MC系統(tǒng)切換后，一臺(tái)主機(jī)運(yùn)行多個(gè)應(yīng)用的處理能力。5. 應(yīng)用設(shè)計(jì)考慮：分解應(yīng)用均衡負(fù)載(active/active模式 －>避免active/standby模式) 一個(gè)應(yīng)用一個(gè)卷組 (根據(jù)應(yīng)用劃分磁盤陣列的空間) 客戶端程序用 flo

57、ating IP 進(jìn)行連接，不要用固定的主機(jī)地址。 數(shù)據(jù)一致性：保證MC卷組對(duì)各節(jié)點(diǎn)同步。 (vgexport vgimport命令) 不要改變MC配置文件 ： /.rhosts /etc/hosts /etc/cmcluster/

58、cmclnodelist /etc/cmcluster/* 網(wǎng)絡(luò)服務(wù),,MC系統(tǒng)切換后的措施,假設(shè)2節(jié)點(diǎn)Cluster ,主機(jī)名為host1 、host2， 主機(jī)host1出現(xiàn)故障：確認(rèn)應(yīng)用切換并且可用： 在主機(jī)host2上執(zhí)行： cmviewcl [pkg_name的狀態(tài)應(yīng)為running] ps -ef

59、| grep ora ping float_IP查找故障： log文件 : /var/adm/syslog/syslog.log /etc/cmcluster/pkg??/control.sh.log修復(fù)： HP響應(yīng)中心：記錄主機(jī)序列號(hào) (010)656

60、43888 接好Modem及電話線，Key=>Service狀態(tài) 恢復(fù)應(yīng)用：主機(jī)host1修復(fù)啟動(dòng)后，cmrunnode host1恢復(fù)應(yīng)用在原主機(jī)運(yùn)行： cmhaltpkg pkg_name [此命令將中斷應(yīng)用] cmmodpkg -e -n host1 -n host2 pkg_name

61、 cmrunpkg -n host1 pkg_name cmviewcl,MC/ServiceGuard與MC/LockManager的區(qū)別,ServiceGuard LockManager Multiple applications ea

62、ch running exclusively on one nodealmost any application oracle OPS(Oracle Parallel Server) ONLYRaw volumes, HFS,JFS OPS DB: raw vol

63、umes ONLY Applications reconnects to the same IP addressEach application has its own all node accesses the same OPS disk volume groupsdisk volume groupsapplic

64、ation scaling dependent upon potential increase in application scaling dependent onperformance of single SPU database partitioningapplication is not available