waf比较(lua-nginx-module,modsecurity,naxsi)

ngx_lua_waf是一个基于lua-nginx-module(openresty)的web应用防火墙

https://github.com/loveshell/ngx_lua_waf

nginx配合modsecurity实现WAF功能

http://www.52os.net/articles/nginx-use-modsecurity-module-as-waf.html

NAXSI is an open-source, high performance, low rules maintenance WAF for NGINX

https://github.com/nbs-system/naxsi

中、小企业如何自建免费的云WAF

https://zhuanlan.zhihu.com/p/22068364

X-WAF是一款适用中、小企业的云WAF系统,让中、小企业也可以非常方便地拥有自己的免费云WAF。

https://waf.xsec.io/docs

基于openresty的Web应用安全防护系统(WAF)
http://git.oschina.net/miracleqi/OpenWAF

kjournald占I/O高,如何优化

nfsd 42687 kjournald 31201 rsync 21846 两分钟内的磁盘块读写数量排行

1、用dumpe2fs看一下Journal size是多大,把Journal size的值改大一些。

2、调整一下文件系统的jounal模式,默认为ordered ,改成writeback会提高一些效率。

Despite writing some data more than once, ext3 is often faster (higher throughput) than ext2 because ext3’s journaling optimizes hard drive head motion. You can choose from three journaling modes to optimize speed, optionally choosing to trade off some data integrity.

  • One mode, data=writeback, limits the data integrity guarantees, allowing old data to show up in files after a crash, for a potential increase in speed under some circumstances. (This mode, which is the default journaling mode for most journaling file systems, essentially provides the more limited data integrity guarantees of the ext2 file system and merely avoids the long file system check at boot time.)

  • The second mode, data=ordered (the default mode), guarantees that the data is consistent with the file system; recently-written files will never show up with garbage contents after a crash.

  • The last mode, data=journal, requires a larger journal for reasonable speed in most cases and therefore takes longer to recover in case of unclean shutdown, but is sometimes faster for certain database operations.

The default mode is recommended for general-purpose computing needs. To change the mode, add the data=something option to the mount options for that file system in the /etc/fstab file, as documented in the mount man page (man mount).

查看linux服务器硬盘IO读写负载

用top命令查看

top – 16:15:05 up 6 days,  6:25,  2 users,  load average: 1.45, 1.77, 2.14
Tasks: 147 total,   1 running, 146 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2% us,  0.2% sy,  0.0% ni, 86.9% id, 12.6% wa,  0.0% hi,  0.0% si
Mem:   4037872k total,  4003648k used,    34224k free,     5512k buffers
Swap:  7164948k total,   629192k used,  6535756k free,  3511184k cached

查看12.6% wa

IO等待所占用的CPU时间的百分比,高过30%时IO压力高

iostat -x 1 10

avg-cpu:  %user   %nice    %sys %iowait   %idle
0.00       0.00     0.25    33.46    66.29

Device:    rrqm/s  wrqm/s   r/s    w/s     rsec/s   wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda          0.00    0.00      0.00   0.00    0.00    0.00         0.00     0.00     0.00           0.00    0.00    0.00   0.00
sdb          0.00   1122  17.00  9.00  192.00 9216.00    96.00  4608.00   123.79   137.23 1033.43  13.17 100.10
sdc          0.00    0.00     0.00   0.00     0.00     0.00      0.00     0.00     0.00             0.00    0.00      0.00   0.00

查看%util 100.10 %idle 66.29

如果 %util 接近 100%,说明产生的I/O请求太多,I/O系统已经满负荷,该磁盘可能存在瓶颈。

idle小于70% IO压力就较大了,一般读取速度有较多的wait.

vmstat

同时可以结合vmstat 查看查看b参数(等待资源的进程数)

vmstat -1

再通过如下脚本查看高峰的进程io情况

#!/bin/sh
/etc/init.d/syslog stop
echo 1 > /proc/sys/vm/block_dump
sleep 60
dmesg | awk '/(READ|WRITE|dirtied)/ {process[$1]++} END {for (x in process) \
print process[x],x}' |sort -nr |awk '{print $2 " " $1}' | \
head -n 10
echo 0 > /proc/sys/vm/block_dump
/etc/init.d/syslog start

使用rvm安装多个ruby版本

Step 1: Upgrade Packages

# yum update

Step 2: Installing Recommended Packages

# yum install gcc-c++ patch readline readline-devel zlib zlib-devel
# yum install libyaml-devel libffi-devel openssl-devel make
# yum install bzip2 autoconf automake libtool bison iconv-devel

Step 3: Install RVM ( Ruby Version Manager )

# curl -L get.rvm.io | bash -s stable

Step 4: Setup RVM Environment

# source /etc/profile.d/rvm.sh

Step 5: Install Required Ruby Version

# rvm install 1.9.3

Step 6: Setup Default Ruby Version

# rvm use 1.9.3 --default 

Step 7: Check Current Ruby Version

# ruby --version

配置nginx使用letsencrypt免费https证书

https://letsencrypt.org/

生成证书

certbot certonly --webroot \
-w /data/web/c4ys/frontend/web/ -d c4ys.com -d www.c4ys.com \
-w /data/web/c4ys/mobile/web/ -d m.c4ys.com

修改nginx配置

listen       443 ssl;
ssl                  on;
ssl_certificate /etc/letsencrypt/live/www.c4ys.com/fullchain.pem;     
ssl_certificate_key /etc/letsencrypt/live/www.c4ys.com/privkey.pem;
ssl_session_cache    shared:SSL:1m;
ssl_session_timeout  5m;
ssl_ciphers  HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers  on;
error_page 497  https://$host$uri?$args;

自动更新证书

0 3 */10 * * certbot renew –quiet
0 3 */10 * * service nginx restart

inotify-tools + rsync 实现文件自动同步备份

公司图片服务器以前以nfs挂载在各台php-fpm服务器下,最近因为部分fpm服务器迁移到aliyun,远程挂载效率低下,为了本地不修改代码,又可以达到不低,所以想到了inotify + rsync。

inotify-tools介绍:https://github.com/rvoicilas/inotify-tools/wiki

#!/bin/sh

# get the current path
CURPATH=`pwd`

inotifywait -mr --timefmt '%d/%m/%y %H:%M' --format '%T %w %f' \
-e close_write /tmp/test | while read date time dir file; do

       FILECHANGE=${dir}${file}
       # convert absolute path to relative
       FILECHANGEREL=`echo "$FILECHANGE" | sed 's_'$CURPATH'/__'`

       rsync --progress --relative -vrae 'ssh -p 22'  $FILECHANGEREL usernam@example.com:/backup/root/dir && \
       echo "At ${time} on ${date}, file $FILECHANGE was backed up via rsync"
done

yii实现redis读写分离

/**
 * FileName:RedisCluster
 * 配置说明
 * 配置为1主多从 或者  1个独立的服务器
 * 写往主的里面写
 * 读是从从的里面读
 * 'class'=>'RedisCache',
 * 'servers'=>array(
 *                array(
 *                   'host'=>'IP1',
 *                   'port'=>'6380',
 *                    'master'=>true //主机
 *                ),
 *                array(
 *                    'host'=>'IP2',
 *                    'port'=>'6381'
 *               )
 *
 * 单独
 * 'class'=>'RedisCache',
 * 'servers'=>array(
 *                array(
 *                   'host'=>'IP1',
 *                   'port'=>'6380',
 *                ),
 
 * @author   zhonghailin
 * @version  v1.0
 * @since    v1.0
 * @Date     2015-3-10 下午4:57:02
 */
class ERedisCache extends CCache{
        
    // 是否使用 M/S 的读写集群方案
    private $_isUseCluster = false;
        
    // Slave 句柄标记
    private $_sn = 0;
        
    // 服务器连接句柄
    private $_linkHandle = array(
        'master'=>null,// 只支持一台 Master
        'slave'=>array(),// 可以有多台 Slave
    );
    //配置文件
    private $_servers=array();
    /**
     * 构造函数
     *
     * @param boolean $isUseCluster 是否采用 M/S 方案
     */
    /*
     * 初始化
     */
    public function init()
    {
        $this->keyPrefix = "";
        parent::init();
        if(is_array($this->_servers)){
            foreach($this->_servers as $servers){
                    if($servers->master){
                        $this->_isUseCluster = true;//如果设置了主服务器那么就是1主多从
                        $this->connect($servers,true);
                    }else{
                        $this->connect($servers);
                    }
            }
        }
    }
     
    /**
     * @return array list of memcache server configurations. Each element is a {@link CMemCacheServerConfiguration}.
     */
    public function getServers()
    {
        return $this->_servers;
    }
     
    /**
     * @param array $config list of memcache server configurations. Each element must be an array
     * with the following keys: host, port, persistent, weight, timeout, retryInterval, status.
     * @see http://www.php.net/manual/en/function.Memcache-addServer.php
     */
    public function setServers($config)
    {
        foreach($config as $c)
            $this->_servers[]=new CRedisServerConfiguration($c);
    }
    /**
     * 连接服务器,注意:这里使用长连接,提高效率,但不会自动关闭
     *
     * @param array $config Redis服务器配置
     * @param boolean $isMaster 当前添加的服务器是否为 Master 服务器
     * @return boolean
     */
    public function connect($config, $isMaster=false){
        // default port
        if(!isset($config->port)){
            $config->port = 6379;
        }
        // 设置 Master 连接
        if($isMaster){
            $this->_linkHandle['master'] = new Redis();
            $ret = $this->_linkHandle['master']->pconnect($config->host,$config->port);
        }else{
            // 多个 Slave 连接
            $this->_linkHandle['slave'][$this->_sn] = new Redis();
            $ret = $this->_linkHandle['slave'][$this->_sn]->pconnect($config->host,$config->port);
            ++$this->_sn;
        }
        return $ret;
    }
        
    /**
     * 关闭连接
     *
     * @param int $flag 关闭选择 0:关闭 Master 1:关闭 Slave 2:关闭所有
     * @return boolean
     */
    public function close($flag=2){
        switch($flag){
            // 关闭 Master
            case 0:
                $this->getRedis()->close();
            break;
            // 关闭 Slave
            case 1:
                for($i=0; $i<$this->_sn; ++$i){
                    $this->_linkHandle['slave'][$i]->close();
                }
            break;
            // 关闭所有
            case 2:
                if(!empty( $this->_linkHandle['master'])){
                    $this->_linkHandle['master']->close();
                }
                for($i=0; $i<$this->_sn; ++$i){
                    $this->_linkHandle['slave'][$i]->close();
                }
            break;
        }
        return true;
    }
        
    /**
     * 得到 Redis 原始对象可以有更多的操作
     *
     * @param boolean $isMaster 返回服务器的类型 true:返回Master false:返回Slave
     * @param boolean $slaveOne 返回的Slave选择 true:负载均衡随机返回一个Slave选择 false:返回所有的Slave选择
     * @return redis object
     */
    public function getRedis($isMaster=true,$slaveOne=true){
        // 只返回 Master
        if($isMaster){
            return $this->_linkHandle['master'];
        }else{
            return $slaveOne ? $this->_getSlaveRedis() : $this->_linkHandle['slave'];
        }
    }
        
    /**
     * 写缓存
     *
     * @param string $key 组存KEY
     * @param string $value 缓存值
     * @param int $expire 过期时间, 0:表示无过期时间
     */
    public function setValue($key, $value, $expire=0){
        // 永不超时
        if($expire == 0){
            if($this->_isUseCluster){
                $ret = $this->getRedis()->set($key, $value);
            }else{
                $ret = $this->getRedis(false)->set($key, $value);
            }
        }else{
            if($this->_isUseCluster){
                 $ret = $this->getRedis()->setex($key, $expire, $value);
            }else{
                 $ret = $this->getRedis(false)->setex($key, $expire, $value);
            }
        }
        return $ret;
    }
        
    /**
     * 读缓存
     *
     * @param string $key 缓存KEY,支持一次取多个 $key = array('key1','key2')
     * @return string || boolean  失败返回 false, 成功返回字符串
     */
    public function getValue($key){
        // 是否一次取多个值
        $func = is_array($key) ? 'mGet' : 'get';
        // 使用了 M/S
        return $this->_getSlaveRedis()->{$func}($key);
    }
        
    /**
     * 条件形式设置缓存,如果 key 不存时就设置,存在时设置失败
     *
     * @param string $key 缓存KEY
     * @param string $value 缓存值
     * @return boolean
     */
    public function setnx($key, $value){
        if($this->_isUseCluster){
            return $this->getRedis()->setnx($key, $value);
        }else{
            return $this->getRedis(false)->setnx($key, $value);
        }
         
    }
        
    /**
     * 删除缓存
     *
     * @param string || array $key 缓存KEY,支持单个健:"key1" 或多个健:array('key1','key2')
     * @return int 删除的健的数量
     */
    public function remove($key){
        if($this->_isUseCluster){
             return $this->getRedis()->delete($key);
        }else{
             return $this->getRedis(false)->delete($key);
        }
    }
     
    /* (non-PHPdoc)
     * @see BaseCache::delete()
     */
    public function deleteValue($key){
        if($this->_isUseCluster){
             return $this->getRedis()->delete($key);
        }else{
             return $this->getRedis(false)->delete($key);
        }
    }
    /**
     * 值加加操作,类似 ++$i ,如果 key 不存在时自动设置为 0 后进行加加操作
     *
     * @param string $key 缓存KEY
     * @param int $default 操作时的默认值
     * @return int 操作后的值
     */
    public function incr($key,$default=1){
        if($default == 1){
            return $this->getRedis()->incr($key);
        }else{
            return $this->getRedis()->incrBy($key, $default);
        }
    }
        
    /**
     * 值减减操作,类似 --$i ,如果 key 不存在时自动设置为 0 后进行减减操作
     *
     * @param string $key 缓存KEY
     * @param int $default 操作时的默认值
     * @return int 操作后的值
     */
    public function decr($key,$default=1){
        if($default == 1){
            return $this->getRedis()->decr($key);
        }else{
            return $this->getRedis()->decrBy($key, $default);
        }
    }
        
    /**
     * 添空当前数据库
     *
     * @return boolean
     */
    public function clear(){
        if(!empty( $this->_linkHandle['master'])){
            $this->_linkHandle['master']->flushDB();
        }
        for($i=0; $i<$this->_sn; ++$i){
            $this->_linkHandle['slave'][$i]->flushDB();
        }
        return true;
    }
        
    /* =================== 以下私有方法 =================== */
        
    /**
     * 随机 HASH 得到 Redis Slave 服务器句柄
     *
     * @return redis object
     */
    private function _getSlaveRedis(){
        // 就一台 Slave 机直接返回
        if($this->_sn <= 1){
            return $this->_linkHandle['slave'][0];
        }
        // 随机 Hash 得到 Slave 的句柄
        $hash = $this->_hashId(mt_rand(), $this->_sn);
        return $this->_linkHandle['slave'][$hash];
    }
        
    /**
     * 根据ID得到 hash 后 0~m-1 之间的值
     *
     * @param string $id
     * @param int $m
     * @return int
     */
    private function _hashId($id,$m=10)
    {
        //把字符串K转换为 0~m-1 之间的一个值作为对应记录的散列地址
        $k = md5($id);
        $l = strlen($k);
        $b = bin2hex($k);
        $h = 0;
        for($i=0;$i<$l;$i++)
        {
            //相加模式HASH
            $h += substr($b,$i*2,2);
        }
        $hash = ($h*1)%$m;
        return $hash;
    }
        
}// End Class
 
 
 
class CRedisServerConfiguration extends CComponent
{
    /**
     * @var string memcache server hostname or IP address
     */
    public $host;
    /**
     * @var integer memcache server port
     */
    public $port=11211;
     
    public $master = false;
    /**
     * Constructor.
     * @param array $config list of memcache server configurations.
     * @throws CException if the configuration is not an array
     */
    public function __construct($config)
    {
        if(is_array($config))
        {
            foreach($config as $key=>$value)
                $this->$key=$value;
            if($this->host===null)
                throw new CException(Yii::t('yii','CMemCache server configuration must have "host" value.'));
        }
        else
            throw new CException(Yii::t('yii','CMemCache server configuration must be an array.'));
    }
}

php取得cpu使用

function getCpuUsage()
{
    $data = getrusage();
    $cpuu = ($data['ru_utime.tv_sec'] + $data['ru_utime.tv_usec'] / 1e6);
    $cpus = ($data['ru_stime.tv_sec'] + $data['ru_stime.tv_usec'] / 1e6);
    $renderedtime = round(microtime(1) - PROFILE_START, 6);
    return [$renderedtime > $cpuu ? $cpuu : 0, $renderedtime > $cpus ? $cpus : 0,];
}