1、1FortiGate IITroubleshootingFortiGate 5.2.12目标识别网络常规行为监控非正常行为如流量突发或非典型性协议Troubleshoot物理和逻辑网络接口理解会话表使用“diagnose debug flow” 来对流量流向进行排错对资源使用问题进行排错, 如当防病毒和IPS打开时高CPU或高内存占用测试没有保存到flash的OS3在任何问题发生之前定义正常行为(基线):CPU 使用率Memory 使用率流量等级流量如何走向(流量)使用了哪些协议和TCP/UDP 端口流量模式和分布Why?如果你知道什么是正常流量, 识别非正常流量会更容易NowBaseline
2、 (Average)Normal RangeAbnormal4网络图为何需要网络图? 没有网络图，解释和分析复杂网络是困难且耗时的物理图包含所有物理网络接口, 连线和端口对 Layer 1/2/3 的问题很有效逻辑图包含路由器, 逻辑设备(VDOMs)和UTM对Layer 3+的问题很有效2019:db8:b108port2192.168.1.0/24port4172.16.1.0/27port110.0.0.0/8port35监控数据流 & 资源使用情况获取正常的网络数据 在发生问题之前不正常的行为非常难发现 除非知道什么是正常的CPU使用率RAM使用率允许通过的应用入和出的带宽工具SNMP
3、Alert emailLogging / SyslogFortiAnalyzer或者第三方SIEM(system information & event management)Dashboard / get system statusNormalTraffic spikes6SNMPAllowed source of queries7通过SNMP获取事件通知 trapDestination触发FortiGate t发送SNMP消息的事件8# get sys statusVersion: FortiGate-VM64 v5.2.0,build0589,140613 (GA)Virus-DB: 2
4、2.00856(2019-09-24 05:33)Extended DB: 1.00000(2019-10-17 15:46)IPS-DB: 5.00549(2019-09-23 00:49)IPS-ETDB: 0.00000(2019-01-01 00:00)Serial-Number: FGVM040000025212Botnet DB: 1.00736(2019-08-24 10:18)License Status: ValidVM Resources: 1 CPU/4 allowed, 969 MB RAM/6144 MB allowedBIOS version: 04000002Lo
5、g hard disk: AvailableHostname: STUDENTOperation Mode: NATCurrent virtual domain: rootMax number of virtual domains: 10Virtual domains status: 1 in NAT mode, 0 in TP modeVirtual domain configuration: disableFIPS-CC mode: disableCurrent HA mode: standaloneBranch point: 589Release Version Information:
6、 GAFortiOS x86-64: YesSystem time: Thu Oct 9 00:26:54 2019# get sys perf statCPU states: 2% user 15% system 0% nice 83% idleCPU0 states: 2% user 15% system 0% nice 83% idleMemory states: 44% usedAverage network usage: 542 kbps in 1 minute, 1050 kbps in 10 minutes, 512 kbps in 30 minutesAverage sessi
7、ons: 7 sessions in 1 minute, 5 sessions in 10 minutes, 5 sessions in 30 minutesAverage session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutesVirus caught: 0 total in 1 minuteIPS attacks blocked: 0 total in 1 minu
8、teUptime: 0 days, 0 hours, 19 minutes系统信息 & 资源使用情况9# diagnose firewall statistic showgetting traffic statistics.Browsing: 328 packets, 132562 bytesDNS: 797 packets, 127917 bytesE-Mail: 0 packets, 0 bytesFTP: 0 packets, 0 bytesGaming: 0 packets, 0 bytesIM: 0 packets, 0 bytesNewsgroups: 0 packets, 0 b
9、ytesP2P: 0 packets, 0 bytesStreaming: 0 packets, 0 bytesTFTP: 0 packets, 0 bytesVoIP: 0 packets, 0 bytesGeneric TCP: 1098554 packets, 817573554 bytesGeneric UDP: 1490 packets, 210976 bytesGeneric ICMP: 0 packets, 0 bytesGeneric IP: 6 packets, 192 bytes# diagnose hardware deviceinfo nic port1Name: po
10、rt1Driver: e1000Version: 5.1.13k2 NAPIFW version: N/ABus: 00:11.0Memory: 0 xfeb80000 - 0 xfeba0000Base address: 0 x1400Interrupt: 18Hwaddr: 00:0c:29:95:8c:faPermanent Hwaddr:00:0c:29:95:8c:faState: upLink: upMtu: 1500Supported: auto 10half 10full 100half 100full 1000fullAdvertised: auto 10half 10ful
11、l 100half 100full 1000fullSpeed: 1000fullAuto: enabledRx packets: 136154Rx bytes: 10901815Rx compressed: 0Rx dropped: 0Rx errors: 0 Rx Length err: 0 Rx Buf overflow: 0 Rx Crc err: 0 Rx Frame err: 0 Rx Fifo overrun: 0 Rx Missed packets: 0Tx packets: 1611Tx bytes: 257565.Multicasts: 0Collisions: 0带宽利用
12、率，系统崩溃和错误10其他工具CLI get system status get system performance status diagnose sys top diagnose sys top-summary diagnose hardware sysinfo memory diagnose hardware sysinfo shm diagnose netlink device list diagnose hardware deviceinfo nic port1 diagnose firewall statistics show .DashboardSNMP trapsAlert
13、emailLogs11# diagnose hardware deviceinfo nic port1Description :FortiASIC NP6 AdapterDriver Name :FortiASIC Unified NPU DriverName :np6_2PCI Slot :8d:00.0irq :58Board :FGT3700DSN :NP6KR44613000276Major ID :2Minor ID :0lif id :0lif oid :156netdev oid :156netdev flags :1203Current_HWaddr :08:5b:0e:4a:
14、2e:e4Permanent_HWaddr:08:5b:0e:4a:2e:e4phy name :np6_2_0bank_id :255phy_addr :0 x20lane :0sw_port :51sw_np_port (cat)vid_phy6 :0 x000 x000 x0b0 x000 x000 x00vid_fwd6 :0 x000 x000 x000 x000 x000 x00oid_fwd6 :0 x000 x000 x000 xcc0 x000 x00= Link Status =Admin :upnetdev status :downautonego_setting:1li
15、nk_setting :1link_speed :40000link_duplex :1Speed :0Duplex :Fulllink_status :Downrx_link_status :0int_phy_link :0local_fault :0local_warning :0remote_fault :0= Counters =Rx Pkts :0Rx Bytes :0Tx Pkts :0Tx Bytes :0Host Rx Pkts :0Host Rx Bytes :0Host Rx dropped :0Host Tx Pkts :4Host Tx Bytes :198Host T
16、x dropped :0sw_rx_pkts :0sw_rx_bytes :0sw_tx_pkts :0sw_tx_bytes :0sw_np_rx_pkts :4sw_np_rx_bytes :272sw_np_tx_pkts :0sw_np_tx_bytes :0物理层/数据链路层的Troubleshooting12网络层的Troubleshooting：路由# execute ping-options ?data-size 定义数据包的大小,以bytes为单位df-bit 在IP头里设置 DF 位interval 两个ping直接的间隔时间,以秒为单位pattern 十六进制格式, e.
17、g. 00ffaabbrepeat-count 重复ping多少次source auto | timeout 定义多少秒后timeouttos IP的服务类型ttl 存活时间 time-to-live.validate-reply 有效的reply数据.view-settings 查看ping的当前设置# execute ping # execute traceroute | 13网络层的Troubleshooting：会话清空之前的过滤条件# diagnose sys session filter clear设置过滤条件# diagnose sys session filter ?dport
18、 destination portdst destination IP addresspolicy policy idsport source portsrc source ip address列出所有匹配过滤条件的会话# diagnose sys session list清空所有匹配过滤条件的会话# diagnose sys session clear14会话表：TCPsession info: proto=6 proto_state=65 duration=3 expire=9 timeout=3600 flags=00000000 sockflag=00000000 sockport=4
19、43 av_idx=9 use=5origin-shaper=guarantee-100kbps prio=2 guarantee 12800Bps max 134217728Bps traffic 13895Bpsreply-shaper=guarantee-100kbps prio=2 guarantee 12800Bps max 134217728Bps traffic 13895Bpsper_ip_shaper=ha_id=0 policy_dir=0 tunnel=/state=redir local may_dirty ndr npu nlb os rs statistic(byt
20、es/packets/allow_err): org=864/8/1 reply=2384/7/1 tuples=3orgin-sink: org pre-post, reply pre-post dev=7-6/6-7 gwy=172.17.87.3/10.1.10.1hook=post dir=org act=snat 192.168.1.110:57999-74.201.86.29:443(172.17.87.16:57999)hook=pre dir=reply act=dnat 74.201.86.29:443-172.17.87.16:57999(192.168.1.110:579
21、99)hook=post dir=reply act=noop 74.201.86.29:443-192.168.1.110:57999(0.0.0.0:0)pos/(before,after) 0/(0,0), 0/(0,0)misc=0 policy_id=1 id_policy_id=0 auth_info=0 chk_client_info=0 vd=0serial=0008b037 tos=ff/ff ips_view=1 app_list=2000 app=24534dd_type=0 dd_mode=0per_ip_bandwidth meter: addr=192.168.1.
22、110, bps=4872npu_state=00000000npu info: flag=0 x00/0 x00, offload=0/0, ips_offload=0/0, epid=0/0, ipid=0/0, vlan=0/0连连接状接状态态流量整形流量整形数据包数据包统计统计剩余剩余TTLNAT硬件加速硬件加速协议协议目的端口目的端口会会话处话处理理15会话表：协议proto=6服务代码在IP头中常见的代码1 = ICMP6 = TCP17 = UDP132 = SCTPIPv4 Header16传输层的Troubleshooting: TCP状态proto_state=05总是两位
23、数字第一位数字 = 客户端的会话状态（没有代理则是0）第二位数字 = 服务器端的会话状态StateValueExpiry Timer (default)NONE010 sESTABLISHED13600 sSYN_SENT2120 sSYN & SYN/ACK360 sFIN_WAIT4120 sTIME_WAIT5120 sCLOSE610 sCLOSE_WAIT7120 sLAST_ACK830 sLISTEN9120 s17传输层的Troubleshooting: UDP & ICMP 状态虽然UDP 是一个无状态协议, FortiGate 仍会有两个不同的 “proto_state”
24、值:State Value未看到UDP回应 00看到UDP回应01ICMP 无状态proto_state 一直标记为0018传输层的Troubleshooting: SCTP状态StateValueExpiry Timer (default)SCTP_S_NONE060 sSCTP_S_ESTABLISHED13600 sSCTP_S_CLOSED210 sSCTP_S_COOKIE_WAIT35 sSCTP_S_COOKIE_ECHOED410 sSCTP_S_SHUTDOWN_SENT530 sSCTP_S_SHUTDOWN_RECD630 sSCTP_S_SHUTDOWN_ACK_SEN
25、T73 sSCTP_S_MAX8n/a19会话表：会话处理标识state=log shape may_dirty 并不通用如果会话被卸载到ASIC芯片上，则不一定代表是现在的状态hardware accelerationFlagMeaninglogSession is being loggedlocalSession is to/from local stackextSession is created by a firewall session helpermay_dirtySession is created by traffic hitting a policy. ndrSession
26、will be checked by IPS signaturendsSession will be checked by IPS anomalybrSession is being bridged (TP mode)npuSession is possible to be offloaded to NPUwccpSession is handled by WCCPnpdSession cannot be offloaded to NPUdirtyNext packet in original direction will be revalidated against policyredirS
27、ession is being processed by an application layer proxyauthedSession was successfully authenticatedauthSession is requires (or required) authenticationsrc-visSession is being scanned for device detection purposes20会话表: 连接自动删除会话超时expire=89 timeout=3600不活跃的会话当两个值都为0时TCP连接被拆除FIN, FIN/ACK, ACKTCP连接超时tcp
28、-halfclose-timer: FIN WAIT and CLOSE WAITtcp-half-open-timer: SYN SENT and SYN & SYN/ACKtcp-timewait-timer: TIME WAITudp-idle-timer21高级抓包选项#diag sniffer packet 当抓取了这个数目的报文时自动停止抓包 修改时间戳的格式a 绝对UTC时间l 当地时间22高级抓包选项：输出 # diag sniff packet any icmp 4 interfaces=any filters=icmp 2.101199 wan2 in 192.168.1.
29、110 - 4.2.2.2: icmp: echo request 2.101400 wan1 out 172.17.87.16 - 4.2.2.2: icmp: echo request 2.123325 wan1 in 4.2.2.2 - 172.17.87.16: icmp: echo reply 2.123500 wan2 out 4.2.2.2 - 192.168.1.110: icmp: echo reply 4 packets received by filter 0 packets dropped by kernel # diag sniff packet any icmp 4
30、 3 l interfaces=any filters=icmp 2019-11-14 10:28:19.769989 wan2 in 192.168.1.110 - 4.2.2.2: icmp: echo request 2019-11-14 10:28:19.770143 wan1 out 172.17.87.16 - 4.2.2.2: icmp: echo request 2019-11-14 10:28:19.792325 wan1 in 4.2.2.2 - 172.17.87.16: icmp: echo reply 3 packets received by filter 0 pa
31、ckets dropped by kernel报文数量时间戳23诊断系统反应过慢高高CPU使用率使用率高内存使用率高内存使用率上一个开启的功能是什么？每次开启一个功能快速诊断CPU使用率有多高，为什么？# get system performance status# diagnose sys top 124高CPU占用率的Troubleshooting: get sys perf stat# get system performance statusCPU states: 4% user 13% system 0% nice 83% idleCPU0 states: 3% user 13% s
32、ystem 0% nice 84% idleCPU1 states: 5% user 13% system 0% nice 82% idleCPU2 states: 2% user 13% system 0% nice 85% idleCPU3 states: 6% user 13% system 0% nice 81% idleMemory states: 19% usedAverage network usage: 12740 kbps in 1 minute, 3573 kbps in 10 minutes, 1077 kbps in 30 minutesAverage sessions
33、: 118 sessions in 1 minute, 11 sessions in 10 minutes, 40 sessions in 30 minutesAverage session setup rate: 11 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 1 sessions per second in last 30 minutesVirus caught: 3 total in 1 minuteIPS attacks blocked: 64 total in 1 m
34、inuteUptime: 60 days, 9 hours, 58 minutesCPU使用率网络使用率内存使用率25高CPU使用率：临时bypass一些检查进程n可以暂时bypass一些检查进程# diagnose test application ipsmonitor 5全局bypass更容易,之后可以调整策略来确认问题无需检查的任务继续运行nCPU使用率在bypass这些进程后是否降低了？n恢复某个检查进程：# diagnose test application ipsmonitor 526内存诊断进程的内存使用率# get system performance status# diag
35、 sys top-summaryFortiOS的内存使用率，不是某一个进程的# diagnose hardware sysinfo mem# diagnose hardware sysinfo slab27# diagnose sys top-summary CPU | 38.4% Mem | 54.0% 1009M/1841M Processes: 20 (running=1 sleeping=86) PID RSS CPU% MEM% FDS TIME+ NAME * 72 32M 34.2 1.7 11 00:03.39 httpclid x5 95 11M 1.9 0.6 20 53:
36、07.83 cw_wtpd 40 23M 1.2 1.3 24 03:02.60 httpsd x5 1173 27M 0.0 1.5 10 00:02.82 pyfcgid x4 36 10M 0.0 0.5 88 00:47.75 zebos_launcher x12 37 9M 0.0 0.5 9 00:00.23 uploadd 38 15M 0.0 0.8 41 01:52.19 miglogd 39 9M 0.0 0.5 5 00:01.41 kmiglogd 46 25M 0.0 1.4 821 01:47.98 proxyd x6 47 10M 0.0 0.5 7 00:00.
37、12 wad_diskd 51 12M 0.0 0.7 16 00:02.72 scanunitd x3 53 61M 0.0 3.3 16 00:15.14 ipsmonitor x2 57 9M 0.0 0.5 7 00:00.13 merged_daemons 69 13M 0.0 0.7 18 00:34.20 urlfilter在diag sys top 中RAM复杂交叉的进程会使条目众多交叉的进程中共享数据# diagnose sys topRun Time: 11 days, 3 hours and 29 minutes0U, 0S, 10I; 500T, 345F, 78KF
38、thttp 48 S 0.0 4.4 httpsd 74 S 0.0 3.4 httpsd 54 S 0.0 3.4 cmdbsvr 23 S 0.0 3.4 httpsd 18618 S 0.0 2.9 httpsd 18645 S 0.0 2.9 httpsd 18643 S 0.0 2.9 httpsd 18646 S 0.0 2.8 httpsd 39 S 0.0 2.8 ipsengine 251 S 0.0 2.7 fgfmd 69 S 0.0 2.4 newcli 103 S 0.0 2.4 newcli 18655 R 0.0 2.4 newcli 18651 S 0.0 2.
39、4 miglogd 38 S 0.0 2.2 scanunitd 7798 S 0.0 0.6 scanunitd 5676 S 0.0 0.6 scanunitd 7797 S 0.0 0.6 updated 60 S 0.0 2.1 fdsmgmtd 61 S 0.0 2.0diag sys top-summary vs. diag sys top28R运行新的OSSave as Default firmware/Run image without saving:D/R新的OS包含了新的功能& 行为发生了一些改变对关键应用会不会产生影响?在升级之前，临时启用一个新的OS(不保存到disk)
40、 来进行测试，这样更安全在lab中测试或在维护窗口运行重启会恢复到先前的image 和配置查看Release Notes也可被用来加载排错images29测试硬件特定的排错images 可用来对硬件排错基础测试高级HQIP测试(RMA)硬盘测试Flash测试在boot loader中，通过TFTP，加载HQIP，而不是FortiOS软件操作指南：https:/support.fortinet/Download/HQIPImages.aspx30临时加载一个image只能只能 通过console进行(或者一个终端server)Press any key to display configurat
41、ion menu. . G: Get firmware image from TFTP server. F: Format boot device. Q: Quit menu and continue to boot with default firmware. H: Display this list of options. Enter G,F,Q,or H ( Press G here.) Enter TFTP server address 192.168.1.168: xxx.xxx.xxx.xxxEnter local address 192.168.1.188: xxx.xxx.xxx.xxxEnter firmware image file name image.out: xxxxxxxxxxxxxxx MAC:00:09:0f:0a:1a:7c # Total 10643362 bytes data downloaded. Verifying the integrity of the firmware image. Total 28000kB unzipped. Save as Default firmware/Run image without saving:D/R谢谢
