サポート #54
システムダウン対応
| ステータス: | 新規 | 開始日: | 2014/03/23 | |
|---|---|---|---|---|
| 優先度: | 通常 | 期日: | ||
| 担当者: | - | 進捗 %: | 0% | |
| カテゴリ: | - | 作業時間の記録: | - | |
| 対象バージョン: | - |
説明
2014/3/23 4時頃からサーバアクセスできない症状発生。
http、sshともにアクセス不可。
3/23 13時頃
sshアクセスできるようになったが、garoll.netにアクセスするとinternal server error。
syslogを見ると
Out of memory: Kill process 11032 (httpd) score 6 or sacrifice child
[root@localhost webroot]# sar
00時00分01秒 CPU %user %nice %system %iowait %steal %idle
03時30分03秒 all 4.40 0.03 0.93 23.41 0.24 71.00
03時40分03秒 all 6.04 0.04 1.01 16.00 0.20 76.71
03時50分02秒 all 6.77 0.04 1.00 19.81 0.22 72.16
04時00分03秒 all 5.64 0.04 1.02 20.47 0.21 72.63
04時10分03秒 all 4.42 0.04 1.20 42.31 0.38 51.64
04時20分04秒 all 5.47 0.03 1.14 47.69 0.38 45.29
04時30分09秒 all 6.38 0.03 1.08 64.57 0.27 27.67
04時41分30秒 all 4.89 0.02 0.93 84.83 0.22 9.11
04時50分18秒 all 0.77 0.05 0.47 98.63 0.09 0.00
05時00分26秒 all 0.41 0.00 0.42 99.07 0.10 0.00
07時50分52秒 all 0.08 0.00 0.26 99.59 0.07 0.00
08時04分57秒 all 0.02 0.00 0.26 99.66 0.06 0.00
08時12分27秒 all 0.02 0.00 0.46 99.44 0.07 0.00
08時58分23秒 all 0.04 0.00 0.53 99.37 0.06 0.00
09時05分27秒 all 0.23 0.00 0.94 98.67 0.16 0.00
09時10分16秒 all 0.58 0.00 1.13 97.95 0.35 0.00
11時43分00秒 all 0.18 0.00 0.84 98.86 0.11 0.00
12時49分07秒 all 0.36 0.00 27.37 72.06 0.20 0.00
13時02分06秒 all 0.52 0.00 0.26 99.14 0.07 0.00
13時10分48秒 all 0.16 0.00 0.29 99.44 0.11 0.00
13時31分23秒 all 0.58 0.00 0.73 98.52 0.16 0.00
13時40分01秒 all 2.55 0.08 1.63 81.43 0.33 13.97
13時50分01秒 all 2.79 0.04 0.22 1.07 0.06 95.82
14時00分01秒 all 1.33 0.04 0.16 0.51 0.03 97.94
4:30あたりからiowaitが急激にあがってる
cakephpのdebugレベルを上げてアクセスすると
Host 'hostname' is blocked because of many connection errors.
Unblock with 'mysqladmin flush-hosts'
DBサーバにはいってmysqladmin flush-hosts実行したらアクセスできるようになった。
◯原因調査
apacheのアクセスログを見ると
AhrefsBotとかいう怪しいbotからアクセスが頻発しているので拒否。
[.htaccess]
order allow,deny
allow from all
deny from 5.10.83.
履歴
#1 山本 義治 がほぼ11年前に更新
参考
http://yotuba10.com/trouble-20140214/
[root@www14012uf admin]# ps aux | grep [h]ttpd | grep [a]pache | awk 'BEGIN{x=0}{x+=$6}END{ print x/NR }'
29061.3
Apacheの最大子プロセス数 : P =
Apacheに割り当て可能なメモリ量 / Apacheの子プロセスあたりの平均メモリ使用量
P = 2,000,000 / 29061.3 = 68.82 = 65
<IfModule prefork.c>
StartServers P×5/256
MinSpareServers P×5/256
MaxSpareServers P×10/256
ServerLimit P
MaxClients P
MaxRequestsPerChild 1000
</IfModule>
変更前
KeepAlive Off
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
変更後
KeepAlive Off
StartServers 8
MinSpareServers 5
MaxSpareServers 10
ServerLimit 128
MaxClients 65
MaxRequestsPerChild 1000
変更前
~$ ab -c 100 -n 1000 http://garoll.net/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking garoll.net (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software: Apache/2.2.15
Server Hostname: garoll.net
Server Port: 80
Document Path: /
Document Length: 31462 bytes
Concurrency Level: 100
Time taken for tests: 128.944 seconds
Complete requests: 1000
Failed requests: 923
(Connect: 0, Receive: 0, Length: 923, Exceptions: 0)
Write errors: 0
Total transferred: 31787254 bytes
HTML transferred: 31591669 bytes
Requests per second: 7.76 [#/sec] (mean)
Time per request: 12894.352 [ms] (mean)
Time per request: 128.944 [ms] (mean, across all concurrent requests)
Transfer rate: 240.74 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 23 760 1291.4 47 7705
Processing: 2376 11473 3405.4 12033 20124
Waiting: 2258 11171 3343.6 11665 19983
Total: 2409 12232 3729.1 12759 21426
Percentage of the requests served within a certain time (ms)
50% 12759
66% 13931
75% 14742
80% 15115
90% 16101
95% 17953
98% 19658
99% 20347
100% 21426 (longest request)
変更後
~$ ab -c 100 -n 1000 http://garoll.net/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking garoll.net (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
Server Software: Apache/2.2.15
Server Hostname: garoll.net
Server Port: 80
Document Path: /
Document Length: 31379 bytes
Concurrency Level: 100
Time taken for tests: 120.705 seconds
Complete requests: 1000
Failed requests: 39
(Connect: 0, Receive: 0, Length: 39, Exceptions: 0)
Write errors: 0
Total transferred: 31627354 bytes
HTML transferred: 31431964 bytes
Requests per second: 8.28 [#/sec] (mean)
Time per request: 12070.518 [ms] (mean)
Time per request: 120.705 [ms] (mean, across all concurrent requests)
Transfer rate: 255.88 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 22 1855 5723.2 60 69998
Processing: 2118 9770 2521.0 10069 17779
Waiting: 2069 9622 2497.0 9882 17542
Total: 2178 11625 6185.0 11148 79601
Percentage of the requests served within a certain time (ms)
50% 11148
66% 12087
75% 12581
80% 12987
90% 15205
95% 16575
98% 27311
99% 44344
100% 79601 (longest request)