やっ太郎ブログ

2012年11月22日木曜日

Openstack Swift recon太郎

Swiftにreconっていう機能があって、loadとかmemoryとかの情報がとれるみたい。これを使って監視とかしたらいんかな。英語よめねからよくわかんね・・・

#proxyサーバからうつぞ
#load averageみる

root@swift-proxy01:~# curl http://xx.xx.xx.xx:6000/recon/load
{"5m": 2.3199999999999998, "15m": 2.27, "processes": 5311, "tasks": "2/364", "1m": 2.4700000000000002}

#memoryの情報チェックするぞ

curl http://xx.xx.xx.xx:6000/recon/mem
{"WritebackTmp": "0 kB", "SwapTotal": "1998840 kB", "Active(anon)": "337664 kB", "SwapFree": "1998840 kB", "DirectMap4k": "6756 kB", "KernelStack": "2936 kB", "MemFree": "128024 kB", "HugePages_Rsvd": "0", "Committed_AS": "4068192 kB", "Active(file)": "4061696 kB", "NFS_Unstable": "0 kB", "VmallocChunk": "34359370072 kB", "Writeback": "0 kB", "Inactive(file)": "15142944 kB", "MemTotal": "24731684 kB", "VmallocUsed": "331016 kB", "HugePages_Free": "0", "AnonPages": "368188 kB", "Active": "4399360 kB", "Inactive(anon)": "30712 kB", "CommitLimit": "14364680 kB", "Hugepagesize": "2048 kB", "Cached": "19014360 kB", "SwapCached": "0 kB", "VmallocTotal": "34359738367 kB", "Shmem": "252 kB", "Mapped": "9884 kB", "SUnreclaim": "811208 kB", "Unevictable": "64 kB", "SReclaimable": "3741636 kB", "Mlocked": "64 kB", "DirectMap2M": "25149440 kB", "HugePages_Surp": "0", "Bounce": "0 kB", "Inactive": "15173656 kB", "PageTables": "5080 kB", "HardwareCorrupted": "0 kB", "HugePages_Total": "0", "Slab": "4552844 kB", "Buffers": "190532 kB", "Dirty": "456 kB"}

#Diskの情報もとれるぞ

curl http://xx.xx.xx.xx:6000/recon/diskusage
[{"device": "sdb1", "avail": 198553079808, "mounted": true, "used": 57811726336, "size": 256364806144}]

#objectのreplicationタイムもとれる(´；ω；｀)
curl http://xx.xx.xx.xx:6000/recon/replication
{"object_replication_time": 21.035725466410319}

#asyncの情報もとれるんだけど、nullかえってきてる・・・
#ここは調べときます。

curl http://10.200.32.41:6000/recon/async
{"async_pending": null}

#swift clientからもたたけるんだな

swift-recon object -r --zone 1
===============================================================================
--> Starting reconnaissance on 1 hosts
===============================================================================
[2012-11-16 14:31:19] Checking on replication
[replication_time] low: 21, high: 21, avg: 21.0, total: 21, Failed: 0.0%, no_result: 0, reported: 1
===============================================================================

Openstack Swift ACL太郎

SwiftのACLの方法でおじゃる。

#containerの作成
swift -A http://PROXY_VIP:8080/auth/v1.0 -U yattarou:yattarou -K yattarou post -r '.r:*' Container_Name
 
#indexの作成
swift -A http://PROXY_VIP:8080/auth/v1.0 -U yattarou:yattarou -K yattarou post -m 'web-index:index.html' Container_Name
 
#list設定
swift -A http://PROXY_VIP:8080/auth/v1.0 -U yattarou:yattarou -K yattarou post -m 'web-listings: true' Container_Name
 
#ACLの確認
swift -A http://PROXY_VIP:8080/auth/v1.0 -U yattarou:yattarou -K yattarou stat Container_Name
 
  Account: AUTH_system
Container: Container_Name
  Objects: 1
    Bytes: 12
 Read ACL: .r:*
Write ACL: 
  Sync To: 
 Sync Key: 
Meta Web-Listings: true
Meta Web-Index: index.html

Openstack swift rebalance太郎

SwiftでのRebalance手順まとめたお(*´ω｀*)(*´ω｀*)

レプリカの一台が復旧できない状態に鳴った場合を想定して、
nodeの切り離し、別のnodeの追加まとめた。

#zone確認
root@swift-proxy01:/etc/swift# swift-ring-builder object.builder
object.builder, build version 3
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  zone      ip address  port      name weight partitions balance meta
0     1    xx.xxx.xx.41  6000      sdb1   1.00     262144    0.00
1     2    xx.xxx.xx.43  6000      sdb1   1.00     262144    0.00
2     3    xx.xxx.xx.44  6000      sdb1   1.00     262144    0.00

#ringからのremoce
root@swift-proxy01:/etc/swift# swift-ring-builder object.builder remove z1-xx.xxx.xx.41/sdb1
d0z1-xx.xxx.xx.41:6000/sdb1_"" marked for removal and will be removed next rebalance.

#ringへの追加
root@swift-proxy01:/etc/swift# swift-ring-builder object.builder add z1-xx.xxx.xx.45:6000/sdb1 1
Device z1-xx.xxx.xx:6000/sdb1_"" with 1.0 weight got id 3

#rebalance
root@swift-proxy01:/etc/swift# swift-ring-builder object.builder rebalance
Reassigned 262144 (100.00%) partitions. Balance is now 0.00.

#zone確認
#xx.xxx.xx.41がはずれ、45が追加されていることがわかる。
root@swift-proxy01:/etc/swift# swift-ring-builder object.builder
object.builder, build version 6
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  zone      ip address  port      name weight partitions balance meta
1     2    xx.xxx.xx.43  6000      sdb1   1.00     262144    0.00
2     3    xx.xxx.xx.44  6000      sdb1   1.00     262144    0.00
3     1    xx.xxx.xx.45  6000      sdb1   1.00     262144    0.00
root@swift-proxy01:/etc/swift#

#この時点ではrebalanceは動かないため、proxyサーバに、builderファイル、ringファイルを展開。container object accountにはringファイルを展開
#その後、swift-init all reload"で再読み込みすればrebalanceが開始される。

2012年11月9日金曜日

2012年10月26日金曜日

Openstack Swift構築太郎

openstack swiftの構築手順。 OSはubuntu 10.04 swiftのverは1.7.0(folsum) 構成は、 LB配下にproxy:2 account:3 container:3 object:3 公式ドキュメント参考にして、計11台構成で作ってみたお


#まずはproxyサーバから。
#必要なモジュールの準備

apt-get install python-software-properties
add-apt-repository ppa:swift-core/release
apt-get update
apt-get install curl gcc git-core memcached python-configobj python-coverage python-dev python-nose python-setuptools python-simplejson python-xattr sqlite3 xfsprogs python-webob python-eventlet python-greenlet python-pastedeploy python-netifaces

#ユーザつくる
/usr/sbin/groupadd -g XXXXX yatta_swift
/usr/sbin/useradd -u XXXXX -g yatta_swift yatta_swift


#ディレクトリつくる
mkdir -p /etc/swift

#hashの設定ファイルつくる
cat >/etc/swift/swift.conf <> /etc/fstab
mkdir -p /srv/node/sdb1
mount -a
chown -R yatta_swift: /srv/*

#recon用のディレクトリきっとく
mkdir -p /var/cache/swift
chown -R yatta_swift: /var/cache/swift/

#proxy serverの設定 2台ともやるお

vim /etc/swift/proxy-server.conf
[DEFAULT]
#cert_file = /etc/swift/cert.crt
#key_file = /etc/swift/cert.key
bind_port = 8080
workers = 8
user = yatta_swift
log_facility = LOG_LOCAL1
[pipeline:main]
pipeline = healthcheck cache tempauth staticweb proxy-server

[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true

[filter:tempauth]
use = egg:swift#tempauth
##proxyのinternval vipを設定
user_system_root = testpass .admin https:XXX.XXX.XXX.XXX:8080/
reseller_prefix = /yatta_images/
user_test_tester = testing .admin
user_test2_tester2 = testing2 .admin
user_test_tester3 = testing3

[filter:healthcheck]
use = egg:swift#healthcheck

[filter:cache]
use = egg:swift#memcache
##proxyserverのipを追加
memcache_servers = XXX.XXX.XXX.XXX:11211,XXX.XXX.XXX.XXX:11211

#静的配信もするので、staticwebモジュール追加
[filter:staticweb]
use = egg:swift#staticweb



#ログ周りの設定
vim /etc/rsyslog.d/10-swift.conf 
# Uncomment the following to have a log containing all logs together
#local1.*   /var/log/swift/all.log

# Uncomment the following to have hourly proxy logs for stats processing
#$template HourlyProxyLog,"/var/log/swift/hourly/%$YEAR%%$MONTH%%$DAY%%$HOUR%"
#local1.*;local1.!notice ?HourlyProxyLog

local1.*;local1.!notice /var/log/swift/proxy.log
local1.notice           /var/log/swift/proxy.error
local1.*                ~

vim /etc/rsyslog.conf
$PrivDropToGroup adm

mkdir -p /var/log/swift/hourly
chown -R yatta_swift /var/log/swift
chmod -R g+w /var/log/swift
service rsyslog restart

ここから、container、account、objectの設定するおその前に、swiftの各node間の同期はrynscでやるんでproxyをのぞく各nodeで下記を設定

vim /etc/rsyncd.conf 
uid = yatta_swift
gid = yatta_swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = xxx.xxx.xx.xxx 

[account]
max connections = 100
path = /srv/node/
read only = false
lock file = /var/lock/account.lock

#rysncの設定ファイル編集
vim /etc/default/rsync 
RSYNC_ENABLE=true

#rsync再起動
/etc/init.d/rsync restart

#accountの設定の場合 ※各nodeでやる
vim  /etc/swift/account-server.conf 
[DEFAULT]
#containerのip
bind_ip = XXX.XXX.XXX.XXX
workers = 100
log_facility = LOG_LOCAL1
mount_check = false
disable_fallocate = true
recon_cache_path = /var/cache/swift
user = yatta_swift

[pipeline:main]
pipeline = recon account-server

[app:account-server]
use = egg:swift#account

[filter:recon]
use = egg:swift#recon

[account-replicator]

[account-auditor]

[account-reaper]

#containerの場合
vim /etc/swift/container-server.conf 
[DEFAULT]
bind_ip = xxx.xxx.xxx.xxx
mount_check = false
disable_fallocate = true
workers = 100
log_facility = LOG_LOCAL1
devices = /srv/node
user = yatta_swift
recon_cache_path = /var/cache/swift

[pipeline:main]
pipeline = recon container-server

[app:container-server]
use = egg:swift#container

[container-replicator]

[filter:recon]
use = egg:swift#recon

[container-updater]

[container-auditor]

[container-sync]

#objectの場合
/etc/swift/object-server.conf 
[DEFAULT]
bind_ip = xxx.xxx.xxx.xxx
workers = 100
log_facility = LOG_LOCAL1
devices = /srv/node
user = yatta_swift
mount_check = false
disable_fallocate = true
recon_cache_path = /var/cache/swift

[pipeline:main]
pipeline = recon object-server

[app:object-server]
use = egg:swift#object

[filter:recon]
use = egg:swift#recon

[object-replicator]

[object-updater]

[object-auditor]

swift-clientいれとく

python-swiftclient-1.2.0.tar.gzを本家からDL
tar zxvf python-swiftclient-1.2.0.tar.gz
python setup.py install

swift起動させてみる

#swift起動させる
#proxyの場合
swift-init proxy start

#container object accountの場合
#※ほんとは個別であげるべきだとおもうけど　とりま。
swift-init all start

#proxyでringファイル生成(複数ある場合は一台だけで)
swift-ring-builder account.builder create 18 3 1
swift-ring-builder container.builder create 18 3 1
swift-ring-builder object.builder create 18 3 1

swift-ring-builder account.builder add z1-xxx.xxx.xxx.xxx:6002/sdb1 1
swift-ring-builder account.builder add z2-xxx.xxx.xxx.xxx:6002/sdb1 1
swift-ring-builder account.builder add z3-xxx.xxx.xxx.xxx:6002/sdb1 1
swift-ring-builder account.builder rebalance
 
swift-ring-builder container.builder add z1-xxx.xxx.xxx.xxx:6001/sdb1 1
swift-ring-builder container.builder add z2-xxx.xxx.xxx.xxx:6001/sdb1 1
swift-ring-builder container.builder add z3-xxx.xxx.xxx.xxx:6001/sdb1 1
swift-ring-builder container.builder rebalance
 
swift-ring-builder object.builder add z1-xxx.xxx.xxx.xxx:6000/sdb1 1
swift-ring-builder object.builder add z2-xxx.xxx.xxx.xxx:6000/sdb1 1
swift-ring-builder object.builder add z3-xxx.xxx.xxx.xxx:6000/sdb1 1
swift-ring-builder object.builder rebalance

#builderファイルと*.ring.gzが生成されるので、builderは各proxyに*.ring.gzは全nodeの/etc/swift配下に。

ちょっといじってみるお

#各nodeの状況確認@proxyで

#swift-ring-builder container.builder 
container.builder, build version 6
262144 partitions, 3 replicas, 3 zones, 3 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  zone      ip address     port      name weight partitions balance meta
             0     1    xxx.xxx.xxx.xxx  6001      sdb1   1.00     262144    0.00 
             1     2    xxx.xxx.xxx.xxx  6001      sdb1   1.00     262144    0.00 
             3     3    xxx.xxx.xxx.xxx  6001      sdb1   1.00     262144    0.00 

#nodeのadd remove rebalance
#node外す
swift-ring-builder object.builder remove z1-xxx.xxx.xxx.xxx/sdb1

#node入れる(重み代えるのもこれでできる)
swift-ring-builder object.builder add z1-xxx.xxx.xxx.xxx/sdb1 1

#reblance ちなみにrebalanceは一度やると一時間はできない　↑の設定だと。
swift-ring-builder object.builder

swift-ring-builder object.builder rebalance
Reassigned 262144 (100.00%) partitions. Balance is now 726.36.
-------------------------------------------------------------------------------
NOTE: Balance of 726.36 indicates you should push this 
      ring, wait at least 1 hours, and rebalance/repush.
-------------------------------------------------------------------------------

とりあえずいったんここまでで。ほかにも結構いろいろ検証しているので、あとでのせます。

2012年6月12日火曜日

Megacli64でディスク交換太郎

Megacli64を使ったディスク交換の手順だお。
ちなみに今回は、Medium Errorがたくさんでてるディスクの予防交換。

OSはCent5.4
RAIDカードは、LSI Megaraid 9280 FW:12.12.0-0090
Megacli 8.02.16で。

#まずはHDDの状態を確認。
/opt/MegaRAID/MegaCli/MegaCli64 -PDList -a0

Enclosure Device ID: 252
Slot Number: 0
Drive's postion: DiskGroup: 0, Span: 0, Arm: 0
Enclosure position: 0
Device Id: 16
WWN:
Sequence Number: 2
Media Error Count: 67　　　　　← errorはいてる
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 1.819 TB [0xe8e088b0 Sectors]
Non Coerced Size: 1.818 TB [0xe8d088b0 Sectors]
Coerced Size: 1.818 TB [0xe8d00000 Sectors]
Firmware state: Online, Spun Up
Is Commissioned Spare : NO
Device Firmware Level: A5C0
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x4433221100000000
Connected Port Number: 0(path0)
Inquiry Data:       MN1220F32HYULDHitachi HDS723020BLA642                 MN6OA5C0

・・・省略

#対象のディスクを確認(ランプ点灯)
/opt/MegaRAID/MegaCli/MegaCli64 -Pdlocate start physdrv[252:0] -a0

#対象のディスクを確認(ランプ消灯)
/opt/MegaRAID/MegaCli/MegaCli64 -Pdlocate stop physdrv[252:0] -a0


#対象ディスクのオフライン
/opt/MegaRAID/MegaCli/MegaCli64 -PDOffline -PhysDrv[252:0] -a0

Adapter: 0: EnclId-252 SlotId-0 state changed to OffLine.

Exit Code: 0x00


#status確認
/opt/MegaRAID/MegaCli/MegaCli64 -PDList -a0 | less

Adapter #0

Enclosure Device ID: 252
Slot Number: 0
Drive's postion: DiskGroup: 0, Span: 0, Arm: 0
Enclosure position: 0
Device Id: 16
WWN:
Sequence Number: 3
Media Error Count: 67
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 1.819 TB [0xe8e088b0 Sectors]
Non Coerced Size: 1.818 TB [0xe8d088b0 Sectors]
Coerced Size: 1.818 TB [0xe8d00000 Sectors]
Firmware state: Offline  ←　offlineになってる。

・・・省略

#対象のディスクにmissing markつける
/opt/MegaRAID/MegaCli/MegaCli64 -PDMarkMissing -PhysDrv[252:0] -a0

EnclId-252 SlotId-0 is marked Missing.

Exit Code: 0x00

#missing markのついたdiskの確認
/opt/MegaRAID/MegaCli/MegaCli64 -PDGetMissing -aALL

    Adapter 0 - Missing Physical drives

    No.   Array   Row   Size Expected
    0     0       0     1907200 MB

#削除前の準備
/opt/MegaRAID/MegaCli/MegaCli64 -PDPrpRmv -PhysDrv[252:0] -a0


Prepare for removal Success

Exit Code: 0x00

#ここでHDDを交換。


#Missing markの状態変更
/opt/MegaRAID/MegaCli/MegaCli64 -PDReplaceMissing -PhysDrv[252:0] -Array0 -row0 -a0

#Rebuildスタート
/opt/MegaRAID/MegaCli/MegaCli64 -PDRbld -Start -PhysDrv[252:0] -a0

#状態確認
/opt/MegaRAID/MegaCli/MegaCli64 -PDList -a0 | less
^M
Adapter #0

Enclosure Device ID: 252
Slot Number: 0
Drive's postion: DiskGroup: 0, Span: 0, Arm: 0
Enclosure position: 0
Device Id: 16
WWN:
Sequence Number: 12
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SATA
Raw Size: 1.819 TB [0xe8e088b0 Sectors]
Non Coerced Size: 1.818 TB [0xe8d088b0 Sectors]
Coerced Size: 1.818 TB [0xe8d00000 Sectors]
Firmware state: Rebuild   ← rebuildのステータスになった

・・・省略

#Rebuildの状況確認

/opt/MegaRAID/MegaCli/MegaCli64 -PDRbld -showprog -physdrv[252:0] -a0

Rebuild Progress on Device at Enclosure 252, Slot 0 Completed 4% in 11 Minutes.

Exit Code: 0x00

こんな感じでできる。ただ、PDmarkMissingが対応していない場合があるので注意。

2012年4月12日木曜日

やっと構築までこぎつけたので、一旦メモで。

SPDYとはGoogleが策定を進めている通信プロトコルらしく
HTTPよりはやいお！！ってことらしい。

gmail、googlecalenderとか、Twitterとかが実装しているらしい太郎。

なんかみたところ、通常のHTTPの50%近くの高速化を目標としているらしい。
今後の主流となること間違いないという予想太郎。

ただ、Firefox11系と、Choromeの最新版？でしか対応してないので
大規模サービスの画像配信とかで使うにはちょっと難しいのかな・・・

まあ、apacheのモジュール公開されてたんで構築してみた。
環境は、Ubuntu 11.04


#まずは事前準備
apt-get install subversion curl g++ apache2 patch binutils julius make

#任意のディレクトリで

mkdir mod_spdy
mkdir temp
cd temp

#depot_tools持ってくる
svn co http://src.chromium.org/svn/trunk/tools/depot_tools

#PATH通す
export PATH="$PATH":`pwd`/depot_tools
echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/yattarou/temp/depot_tools

#SPDYのコード落としてくる ちなみにrootで実行してる
gclient config "http://mod-spdy.googlecode.com/svn/trunk/src"
Running depot tools as root is sad.
#失敗したｗ

#gclientソース見てみる
#base_dirんとこでうまくいってないっぽいので、パワープレイ。
#パス直書きにしてやったw
 vim /home/yattarou/temp/depot_tools/gclient

#!/usr/bin/env bash
# Copyright (c) 2009 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

base_dir=$(dirname "$0")

"$base_dir"/update_depot_tools

PYTHONDONTWRITEBYTECODE=1 exec python "$base_dir/gclient.py" "$@"

#もっかいチャレンジ
/home/yattarou/temp/depot_tools/gclient config "http://mod-spdy.googlecode.com/svn/trunk/src"
Running depot tools as root is sad.

#失敗した
#update_depot_toolsのコードみる
#rootの権限チェック、コメントアウトした。
vim /home/yattarou/temp/depot_tools/update_depot_tools

#!/usr/bin/env bash
# Copyright (c) 2011 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

# This script will try to sync the bootstrap directories and then defer control.

if [ "$USER" == "root" ];
then
  echo Running depot tools as root is sad.
  exit
fi

#もっかいやってみる
/home/yattarou/temp/depot_tools/gclient config "http://mod-spdy.googlecode.com/svn/trunk/src"

#いけたぽいので
/home/udagawa/temp/depot_tools/gclient sync --force
1>________ running 'svn checkout http://mod-spdy.googlecode.com/svn/trunk/src /home/udagawa/mod_spdy/src --non-interactive --force --ignore-externals' in '/home/yattarou/mod_spdy'
・・・・
いっぱいおとしてくる。

26>Checked out revision 933.
Syncing projects:  96% (26/27), done.ols/gyp

________ running '/usr/bin/python src/build/gyp_chromium' in '/home/yattarou/mod_spdy'
Updating projects from gyp files...

#落としてきたら


cd src/
./build_modssl_with_npn.sh

#.soをコピー
cp -rp mod_ssl.so /usr/lib/apache2/modules/
a2enmod ssl

#おれおれ証明書作成
apt-get install ssl-cert

make-ssl-cert /usr/share/ssl-cert/ssleay.cnf /tmp/selfsigned.crt
mkdir /etc/apache2/ssl
mv /tmp/selfsigned.crt /etc/apache2/ssl/

#証明書の設定編集
vim /etc/apache2/sites-available/default-ssl

        SSLCertificateFile    /etc/apache2/ssl/selfsigned.crt
#コメントアウト
        #SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

a2ensite default-ssl

#apache再起動
/etc/init.d/apache2 restart

#SPDYのBUILD
make BUILDTYPE=Release
・・・・
#いっぱいでる

cp out/Release/libmod_spdy.so /usr/lib/apache2/modules/mod_spdy.so
echo "LoadModule spdy_module /usr/lib/apache2/modules/mod_spdy.so" | tee /etc/apache2/mods-available/spdy.load
echo "SpdyEnabled on" | tee /etc/apache2/mods-available/spdy.conf

a2enmod spdy
Enabling module spdy.
Run '/etc/init.d/apache2 restart' to activate new configuration!

#apache再起動
/etc/init.d/apache2 restart


#これでおｋ
#下のでいろいろ見れる
chrome://net-internals/#spdy

#あと、SPDY用のベンチマークツール
#windowsで確認するとき、chrome.exe --enable-benchmarkingしないと起動しないので注意
http://dev.chromium.org/developers/design-documents/extensions/how-the-extension-system-works/chrome-benchmarking-extension

すこし、確認したところ、まあまま早い感じがする。
検証結果は、あとでちゃんと載せます。

登録: 投稿 (Atom)