Skip to content


11月26日在淘宝aDev技术沙龙的《BigPipe后端技术实现分享》

PPT下载:PPT ,   ODP ,  PDF
在线(被转换过,排版上有些小问题):
View more presentations from slawdan.

Posted in 未分类.

Tagged with , , , .


Hadoop & HBase安装笔记

目标

在两台机器上安装Hadoop和HBase作为实验,两台机器分别作为主从。

背景环境

  • 对Hadoop,HBase有一定了解
  • CentOS 5
  • JDK 1.6.0_25
  • Hadoop 0.20.205.0
  • HBase 0.90.4

安装Hadoop

主要参考:官方集群建立文档 http://hadoop.apache.org/common/docs/current/cluster_setup.html

次要参考:

拓扑结构:

  • A,B两台机器,A为NameNode + JobTracker + DataNode + TaskTracker, B仅为DataNode + TaskTracker

基本步骤:

按官方文档走即可,注意调整各种配置。在A上配置完毕以后,直接scp hadoop安装目录(包括conf)到B的对应目录,然后从A bin/start-all 启动。主从配置完全取决于 conf/masters 和 conf/slaves。

碰到的各种问题:

  1. 坑爹的公司服务器的SSH配置: 老实用rsa/dsa就好了……实在不行单开一个sshd服务也行。
    1. ssh参数: -i PrivKey文件地址 -p 端口号
    2. scp参数:-i PrivKey文件地址 -P 端口号
    3. sshd_config里确认两台机器的IP地址以及localhost都被允许:
      1. AllowUsers hadoop_user@127.0.0.1 hadoop_user@A hadoop_user@B
      2. AllowUsers支持IP通配符 “*, ?”
    4. service sshd reload 重启sshd
  2. cfengine对各种配置文件的保护:
    1. chattr +i 锁定修改
    2. 锁定以后最好再检查一遍防止正好碰上cfengine同步
    3. 需要重新修改的时候用chattr -i 解锁,解锁之前最好先备份一下……
  3. 如果在root用户下启动,可能会碰到 -jvm wrong parameter blahblah,可以修改 bin/hadoop, 找到 if [[ $EUID -eq 0 ]]; then,改掉子句里的参数列表,跟 else 子句相同即可。主要原因是root用户的 EUID 为0;以及jvm不同版本的启动参数不太相同导致。
  4. 一定要下载 0.20.205.0 以后版本,合并了可以和 HBase 兼容的代码分支 0.20-append。 启动的时候会报很多Warning $HADOOP_HOME deprecated blahblah,不需理会。
  5. 如果用 bin/start-all 偷懒方式启动,有可能因为配置错误什么的导致某些服务起不来,但不会有任何错误提示。最好使用 netstat -nlp | grep java 看下端口号50030, 50010, 50070, 还有JobTracker的自定义端口是否已经被监听了,如果没有,可以用对应的启动脚本单独启动,这样会输出具体的错误。
  6. 启动之前一定要修改 ulimit 的 -n 51200 和 -u unlimited 以上。否则会出现各种Fetch Failed。

安装HBase

前提:

必须已经正确安装了Hadoop,但可以不启动JobTracker,HBase有自己的MapRed

主要参考:

安装步骤:

建议先启动Standalone玩儿一下,然后再往Hadoop里配置,依照官方文档即可。

拓扑结构:

B为master,A和B均为regionserver

碰到的各种问题:

  1. 官方文档里说明了HBase与Hadoop的适配版本,主要是 0.20-append 分支是否合并进hadoop主干的问题。hadoop 0.20.205.0 已经合并,所以一定要用 0.20.205.0 以后版本的hadoop。
  2. 启动之前一定要修改 ulimit -n 51200 和 ulimit -u 51200 以上。
  3. 官方文档里提到的需要替换 lib/hadoop-core-….jar 的问题一定要做。否则启动时会出现 EOFException。由于实际版本号不同,所以直接把原jar移走,新jar放入即可。
  4. 0.20.205.0 需要同时把 hadoop/lib 里的commons-configuration-1.6.jar 也考到 hbase/lib里。否则启动master时会出现master.HMaster exception of “NoClassDefFoundError” (参考 http://javoft.net/2011/09/hbase-hmaster-%E6%97%A0%E6%B3%95%E5%90%AF%E5%8A%A8-call-to-failed-on-local-exception/)
  5. 这两个jar最好设置与原来的jar相同的属性: rwsrwsrwt:
    1. 设置方法: chmod a+x, chmod ug+s, chmod o+t
  6. bin/start-hbase 会自动启动一个zookeeper。当然可以自行配置zookeeper。
  7. bin/stop-hbase 貌似只会停止zookeeper和master, 在 B(master)上会遗留 regionserver,可以kill 进程或者 bin/hbase-daemon.sh stop regionserver
  8. 同样使用 netstat -nlp | grep java 检查端口号,HBase相关服务端口均以 600开头。

Happy Starting

TODO:

  • ZooKeeper
  • 各种监视脚本
  • Hadoop NameNode单点
  • HBase master单点
  • 修改集群需要停机……
  • 测试各种文件大小空间占用
  • 各种文件大小和负载下的Benchmark
  • 在超过5台服务器上配置集群。貌似由于现在只有2台机器共4块硬盘,而默认dfs.replication为3,所以造成硬盘随机读写,简单benchmark时的IO吞吐低的离谱(w:30M/s, r: 150M/s)……

Posted in 学到的.

Tagged with , .


PHP 处理int64的一些问题和解决方法

PHP内部只有signed int 32,在处理unsigned int 32的时候,通常直接转float就可以正常操作。而处理int64时会受到操作系统字长限制,比如:

1
2
3
4
5
//32bit php:
echo PHP_INT_MAX; // 2^31 - 1
 
//64bit php:
echo PHP_INT_MAX; // 2^63 - 1

由于这个限制,处理比2^31更大的数字的时候必定会发生溢出导致得到一个负数(符号位被设置为1)。通常解决方法是,转成float。这在大多数情况下可以解决问题。但,最近碰到一个问题,某一个数据是使用 unsigned int 64 存储的,读取以后在php里被表示为了科学技术法:

1
2
3
4
5
6
7
8
9
$a = 12345678901234567890;
 
echo $a;
 
//1.2345678901235E+19
 
echo $a + 1;
 
//1.2345678901235E+19

而且,被隐掉的数字完全无法参与后续运算!

后来在64bit的php下同样如此,唯一好的一点是,64bit的php本身的PHP_INT_MAX 比较大,可以满足9223372036854775807 以下的int数据处理。但可笑的是:

1
echo floatval(PHP_INT_MAX); //  9.2233720368548E+18

……居然又发生了截断……

恩,处理办法是: php.ini里有一个precision的设置,默认是14,影响float的精度设置。由于正常的unsigned int64的位数是20位,所以,把这个数字设置到20以上,就可以保证int64的正常处理。这个处理同样会应影响json_decode和其他一些函数:

1
2
3
4
5
6
ini_set('precision', 20);
 
var_export(json_decode('{"a":1234567890123456789}'));
//stdClass::__set_state(array(
//'a' => 1234567890123456789,
//))

但很奇怪的是:

1
2
3
4
var_export(json_decode('{"a":12345678901234567890}'));
//stdClass::__set_state(array(
//'a' => 12345678901234567168,
//))

可以看到a的值已经发生变化,估计是signed int64也开始发生溢出,暂时未找到连这个也能搞定的完美解决办法……
如果不需要int64参与数学运算,想办法转成string吧……

Posted in 学到的.


《How to Use Linux epoll with Python》中文翻译《如何在python中使用linux epoll》

How To Use Linux epoll with Python

如何在Python中使用 Linux epoll

Contents 内容

Introduction  介绍

As of version 2.6, Python includes an API for accessing the Linux epoll library. This article uses Python 3 examples to briefly demonstrate the API. Questions and feedback are welcome.

从2.6版本开始,python 包含了能够直接使用linux epoll库的api。本文使用python 3的示例来大略演示一下这个epoll api。 欢迎问题和反馈(feedback)

Blocking Socket Programming Examples 阻塞socket编程的示例

Example 1 is a simple Python 3.0 server that listens on port 8080 for an HTTP request message, prints it to the console, and sends an HTTP response message back to the client.

示例1是一个python 3.0 写的简单服务器,监听8080端口,它能够接收一个http请求消息,打印到console,然后再发送给客户端一个http响应消息。

  • Line 9: Create the server socket.
  • 行9: 创建服务socket
  • Line 10: Permits the bind() in line 11 even if another program was recently listening on the same port. Otherwise this program could not run until a minute or two after the previous program using that port had finished.
  • 行10:允许行11的 bind() 执行,即使另外一个程序曾刚刚监听过同一端口。如果不这样做,在监听了同一端口的前一个程序结束运行后一两分钟之前,这个程序将不能运行。
  • Line 11: Bind the server socket to port 8080 of all available IPv4 addresses on this machine.
  • 行11:绑定服务socket到本机所有可用的IPv4地址的8080端口。
  • Line 12: Tell the server socket to start accepting incoming connections from clients.
  • 行12: 通知服务socket开始接受从客户端来的连接。
  • Line 14: The program will stop here until a connection is received. When this happens, the server socket will create a new socket on this machine that is used to talk to the client. This new socket is represented by the clientconnection object returned from the accept() call. The address object indicates the IP address and port number at the other end of the connection.
  • 行14: 程序会在这里等待,直到收到一个客户端连接。这时,服务socket会在本机创建一个新的socket来和客户端通信。调用accpet()会返回一个描述了新socket的clientconnection对象。address对象指示了客户端在此连接中使用的ip地址和端口号。
  • Lines 15-17: Assemble the data being transmitted by the client until a complete HTTP request has been transmitted. The HTTP protocol is described at HTTP Made Easy.
  • 行15-17: 在完整的http请求传输完成之前组装从客户端传输来的数据。http协议的简要描述在这里有个简单版可看: HTTP Made Easy
  • Line 18: Print the request to the console, in order to verify correct operation.
  • 行18: 将收到的http请求打印到console以验证操作是否正确。
  • Line 19: Send the response to the client.
  • 行19:将响应发送给客户端。
  • Lines 20-22: Close the connection to the client as well as the listening server socket.
  • 行20-22:关闭对客户端的连接,同时关闭正监听端口的服务socket。

The official HOWTO has a more detailed description of socket programming with Python.

Python官方的 HOWTO 对怎样使用socket进行编程描述的更为详细。

Example 1 (All examples use Python 3) 示例1 (所有示例均使用Python 3)

?View Code PYTHON
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
 1  import socket
 2
 3  EOL1 = b'\n\n'
 4  EOL2 = b'\n\r\n'
 5  response  = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
 6  response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
 7  response += b'Hello, world!'
 8
 9  serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
10  serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
11  serversocket.bind(('0.0.0.0', 8080))
12  serversocket.listen(1)
13
14  connectiontoclient, address = serversocket.accept()
15  request = b''
16  while EOL1 not in request and EOL2 not in request:
17     request += connectiontoclient.recv(1024)
18  print(request.decode())
19  connectiontoclient.send(response)
20  connectiontoclient.close()
21
22  serversocket.close()

Example 2 adds a loop in line 15 to repeatedly processes client connections until interrupted by the user (e.g. with a keyboard interrupt). This illustrates more clearly that the server socket is never used to exchange data with the client. Rather, it accepts a connection from a client, and then creates a new socket on the server machine that is used to communicate with the client.

示例2 在行15添加了一个循环来多次处理客户端连接,直至用户终止(如按Ctrl+C)。这个例子更清晰的阐明服务socket永远不应被用来与client交换数据。当然,它会接受一个客户端连接,然后在本机创建一个新socket来跟客户端通信。

The finally statement block in lines 23-24 ensures that the listening server socket is always closed, even if an exception occurs.

行23-24最终的语句保证负责监听的服务socket能够被关闭,即使有异常发生。

Example 2  示例2

?View Code PYTHON
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
 1  import socket
 2
 3  EOL1 = b'\n\n'
 4  EOL2 = b'\n\r\n'
 5  response  = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
 6  response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
 7  response += b'Hello, world!'
 8
 9  serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
10  serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
11  serversocket.bind(('0.0.0.0', 8080))
12  serversocket.listen(1)
13
14  try:
15     while True:
16        connectiontoclient, address = serversocket.accept()
17        request = b''
18        while EOL1 not in request and EOL2 not in request:
19            request += connectiontoclient.recv(1024)
20        print('-'*40 + '\n' + request.decode()[:-2])
21        connectiontoclient.send(response)
22        connectiontoclient.close()
23  finally:
24     serversocket.close()

Benefits of Asynchronous Sockets and Linux epoll 使用异步socket和linux epoll的好处

The sockets shown in Example 2 are called blocking sockets, because the Python program stops running until an event occurs. The accept() call in line 16 blocks until a connection has been received from a client. The recv() call in line 19 blocks until data has been received from the client (or until there is no more data to receive). The send() call in line 21 blocks until all of the data being returned to the client has been queued by Linux in preparation for transmission.

示例2中的socket用法被称为阻塞socket,因为在(IO)事件发生之前,这个python程序实际上是不会继续运行的。行16的accpet() 调用会保持阻塞状态,直到收到一个客户端连接。行19的recv()调用会保持阻塞状态,直到收到客户端发来的数据(或者直到数据已经发送完毕)。行21的send()调用同样会阻塞,直到要返回给客户端的数据都已经被linux推入准备传输的队列。

When a program uses blocking sockets it often uses one thread (or even a dedicated process) to carry out the communication on each of those sockets. The main program thread will contain the listening server socket which accepts incoming connections from clients. It will accept these connections one at a time, passing the newly created socket off to a separate thread which will then interact with the client. Because each of these threads only communicates with one client, it is ok if it is blocked from proceeding at certain points. This blockage does not prohibit any of the other threads from carrying out their respective tasks.

当一个程序使用阻塞socket,通常会使用一个线程(甚至是一个专门的进程)去处理一个socket请求。主程序线程则包含了负责接受客户端连接的端口监听服务socket。它一次接受一个连接,然后将新创建的socket传递给独立的请求处理线程。由于每个线程只跟一个客户端通信,所以在特定的点阻塞也没什么关系。这种(不同线程里的)阻塞不会妨碍其它线程做自己该做的事情。

The use of blocking sockets with multiple threads results in straightforward code, but comes with a number of drawbacks. It can be difficult to ensure the threads cooperate appropriately when sharing resources. And this style of programming can be less efficient on computers with only one CPU.

在多线程下使用阻塞socket可以得到比较直观的代码,但是会带来一些问题。

The C10K Problem discusses some of the alternatives for handling multiple concurrent sockets. One is the use of asynchronous sockets. These sockets don’t block until some event occurs. Instead, the program performs an action on an asynchronous socket and is immediately notified as to whether that action succeeded or failed. This information allows the program to decide how to proceed. Since asynchronous sockets are non-blocking, there is no need for multiple threads of execution. All work may be done in a single thread. This single-threaded approach comes with its own challenges, but can be a good choice for many programs. It can also be combined with the multi-threaded approach: asynchronous sockets using a single thread can be used for the networking component of a server, and threads can be used to access other blocking resources, e.g. databases.

C10K问题他哦阿伦了处理多并发socket的一些办法。其中一种是使用异步socket。这种socket不会阻塞直到某些触发event。程序对异步socket进行操作后,会立即知道操作成功与否。这允许程序自行决定如何继续。由于异步socket是非阻塞的,所有就没有必要开启多线程处理。所有工作都可一在同一个线程内完成。这个单线程方法有它自己的一些复杂的地方,但仍应该能满足大多数程序的要求。当然,也可以和多线程方法共同使用: 异步socket使用一个线程来处理服务的网络组件,另外的线程用来处理访问类似数据库之类的阻塞资源。

Linux 2.6 has a number of mechanisms for managing asynchronous sockets, three of which are exposed by the Python API’s select, poll and epoll.  epoll and poll are better than select because the Python program does not have to inspect each socket for events of interest. Instead it can rely on the operating system to tell it which sockets may have these events. And epoll is better than poll because it does not require the operating system to inspect all sockets for events of interest each time it is queried by the Python program. Rather Linux tracks these events as they occur, and returns a list when queried by Python. So epoll is a more efficient and scalable mechanism for large numbers (thousands) of concurrent socket connections, as shown in these graphs.

Linux 2.6 下有多种机制可以管理异步socket,其中三个可以在Python API中使用: select, poll, epoll。epoll和poll比select好是因为Python程序不用去盯着每一个socket的指定event,而可以等待操作系统通知哪个socket触发了event。另外,epoll比poll好是因为它在每次被Python程序询问时,不需要操作系统去检查每一个socket的指定event。当events触发时,Linux将它们收集起来,在Python程序询问时返回一个列表。所以epoll在高并发(数千/数万)socket连接时,是更加有效和可伸缩的管理机制。

(未完标记)

Asynchronous Socket Programming Examples with epoll

Programs using epoll often perform actions in this sequence:

  1. Create an epoll object
  2. Tell the epoll object to monitor specific events on specific sockets
  3. Ask the epoll object which sockets may have had the specified event since the last query
  4. Perform some action on those sockets
  5. Tell the epoll object to modify the list of sockets and/or events to monitor
  6. Repeat steps 3 through 5 until finished
  7. Destroy the epoll object

Example 3 duplicates the functionality of Example 2 while using asynchronous sockets. The program is more complex because a single thread is interleaving the communication with multiple clients.

  • Line 1: The select module contains the epoll functionality.
  • Line 13: Since sockets are blocking by default, this is necessary to use non-blocking (asynchronous) mode.
  • Line 15: Create an epoll object.
  • Line 16: Register interest in read events on the server socket. A read event will occur any time the server socket accepts a socket connection.
  • Line 19: The connection dictionary maps file descriptors (integers) to their corresponding network connection objects.
  • Line 21: Query the epoll object to find out if any events of interest may have occurred. The parameter “1″ signifies that we are willing to wait up to one second for such an event to occur. If any events of interest occurred prior to this query, the query will return immediately with a list of those events.
  • Line 22: Events are returned as a sequence of (fileno, event code) tuples. fileno is a synonym for file descriptor and is always an integer.
  • Line 23: If a read event occurred on the socket server, then a new socket connection may have been created.
  • Line 25: Set new socket to non-blocking mode.
  • Line 26: Register interest in read (EPOLLIN) events for the new socket.
  • Line 31: If a read event occurred then read new data sent from the client.
  • Line 33: Once the complete request has been received, then unregister interest in read events and register interest in write (EPOLLOUT) events. Write events will occur when it is possible to send response data back to the client.
  • Line 34: Print the complete request, demonstrating that although communication with clients is interleaved this data can be assembled and processed as a whole message.
  • Line 35: If a write event occurred on a client socket, it’s able to accept new data to send to the client.
  • Lines 36-38: Send the response data a bit at a time until the complete response has been delivered to the operating system for transmission.
  • Line 39: Once the complete response has been sent, disable interest in further read or write events.
  • Line 40: A socket shutdown is optional if a connection is closed explicitly. This example program uses it in order to cause the client to shutdown first. The shutdown call informs the client socket that no more data should be sent or received and will cause a well-behaved client to close the socket connection from it’s end.
  • Line 41: The HUP (hang-up) event indicates that the client socket has been disconnected (i.e. closed), so this end is closed as well. There is no need to register interest in HUP events. They are always indicated on sockets that are registered with the epoll object.
  • Line 42: Unregister interest in this socket connection.
  • Line 43: Close the socket connection.
  • Lines 18-45: The try-catch block is included because the example program will most likely be interrupted by a KeyboardInterrupt exception
  • Lines 46-48: Open socket connections don’t need to be closed since Python will close them when the program terminates. They’re included as a matter of good form.

Example 3

 1  import socket, select
 2
 3  EOL1 = b'\n\n'
 4  EOL2 = b'\n\r\n'
 5  response  = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
 6  response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
 7  response += b'Hello, world!'
 8
 9  serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
10  serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
11  serversocket.bind(('0.0.0.0', 8080))
12  serversocket.listen(1)
13  serversocket.setblocking(0)
14
15  epoll = select.epoll()
16  epoll.register(serversocket.fileno(), select.EPOLLIN)
17
18  try:
19     connections = {}; requests = {}; responses = {}
20     while True:
21        events = epoll.poll(1)
22        for fileno, event in events:
23           if fileno == serversocket.fileno():
24              connection, address = serversocket.accept()
25              connection.setblocking(0)
26              epoll.register(connection.fileno(), select.EPOLLIN)
27              connections[connection.fileno()] = connection
28              requests[connection.fileno()] = b''
29              responses[connection.fileno()] = response
30           elif event & select.EPOLLIN:
31              requests[fileno] += connections[fileno].recv(1024)
32              if EOL1 in requests[fileno] or EOL2 in requests[fileno]:
33                 epoll.modify(fileno, select.EPOLLOUT)
34                 print('-'*40 + '\n' + requests[fileno].decode()[:-2])
35           elif event & select.EPOLLOUT:
36              byteswritten = connections[fileno].send(responses[fileno])
37              responses[fileno] = responses[fileno][byteswritten:]
38              if len(responses[fileno]) == 0:
39                 epoll.modify(fileno, 0)
40                 connections[fileno].shutdown(socket.SHUT_RDWR)
41           elif event & select.EPOLLHUP:
42              epoll.unregister(fileno)
43              connections[fileno].close()
44              del connections[fileno]
45  finally:
46     epoll.unregister(serversocket.fileno())
47     epoll.close()
48     serversocket.close()

epoll has two modes of operation, called edge-triggered and level-triggered. In the edge-triggered mode of operation a call to epoll.poll() will return an event on a socket only once after the read or write event occurred on that socket. The calling program must process all of the data associated with that event without further notifications on subsequent calls to epoll.poll(). When the data from a particular event is exhausted, additional attempts to operate on the socket will cause an exception. Conversely, in the level-triggered mode of operation, repeated calls to epoll.poll() will result in repeated notifications of the event of interest, until all data associated with that event has been processed. No exceptions normally occur in level-triggered mode.

For example, suppose a server socket has been registered with an epoll object for read events. In edge-triggered mode the program would need to accept() new socket connections until a socket.error exception occurs. Whereas in the level-triggered mode of operation a single accept() call can be made and then the epoll object can be queried again for new events on the server socket indicating that additional calls to accept() should be made.

Example 3 used level-triggered mode, which is the default mode of operation. Example 4 demonstrates how to use edge-triggered mode. In Example 4, lines 25, 36 and 45 introduce loops that run until an exception occurs (or all data is otherwise known to be handled). Lines 32, 38 and 48 catch the expected socket exceptions. Finally, lines 16, 28, 41 and 51 add the EPOLLET mask which is used to set edge-triggered mode.

Example 4

 1  import socket, select
 2
 3  EOL1 = b'\n\n'
 4  EOL2 = b'\n\r\n'
 5  response  = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
 6  response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
 7  response += b'Hello, world!'
 8
 9  serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
10  serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
11  serversocket.bind(('0.0.0.0', 8080))
12  serversocket.listen(1)
13  serversocket.setblocking(0)
14
15  epoll = select.epoll()
16  epoll.register(serversocket.fileno(), select.EPOLLIN | select.EPOLLET)
17
18  try:
19     connections = {}; requests = {}; responses = {}
20     while True:
21        events = epoll.poll(1)
22        for fileno, event in events:
23           if fileno == serversocket.fileno():
24              try:
25                 while True:
26                    connection, address = serversocket.accept()
27                    connection.setblocking(0)
28                    epoll.register(connection.fileno(), select.EPOLLIN | select.EPOLLET)
29                    connections[connection.fileno()] = connection
30                    requests[connection.fileno()] = b''
31                    responses[connection.fileno()] = response
32              except socket.error:
33                 pass
34           elif event & select.EPOLLIN:
35              try:
36                 while True:
37                    requests[fileno] += connections[fileno].recv(1024)
38              except socket.error:
39                 pass
40              if EOL1 in requests[fileno] or EOL2 in requests[fileno]:
41                 epoll.modify(fileno, select.EPOLLOUT | select.EPOLLET)
42                 print('-'*40 + '\n' + requests[fileno].decode()[:-2])
43           elif event & select.EPOLLOUT:
44              try:
45                 while len(responses[fileno]) > 0:
46                    byteswritten = connections[fileno].send(responses[fileno])
47                    responses[fileno] = responses[fileno][byteswritten:]
48              except socket.error:
49                 pass
50              if len(responses[fileno]) == 0:
51                 epoll.modify(fileno, select.EPOLLET)
52                 connections[fileno].shutdown(socket.SHUT_RDWR)
53           elif event & select.EPOLLHUP:
54              epoll.unregister(fileno)
55              connections[fileno].close()
56              del connections[fileno]
57  finally:
58     epoll.unregister(serversocket.fileno())
59     epoll.close()
60     serversocket.close()

Since they’re similar, level-triggered mode is often used when porting an application that was using the select or poll mechanisms, while edge-triggered mode may be used when the programmer doesn’t need or want as much assistance from the operating system in managing event state.

In addition to these two modes of operation, sockets may also be registered with the epoll object using the EPOLLONESHOT event mask. When this option is used, the registered event is only valid for one call to epoll.poll(), after which time it is automatically removed from the list of registered sockets being monitored.

Performance Considerations

Listen Backlog Queue Size

In Examples 1-4, line 12 has shown a call to the serversocket.listen() method. The parameter for this method is the listen backlog queue size. It tells the operating system how many TCP/IP connections to accept and place on the backlog queue before they are accepted by the Python program. Each time the Python program calls accept() on the server socket, one of the connections is removed from the queue and that slot can be used for another incoming connection. If the queue is full, new incoming connections are silently ignored causing unnecessary delays on the client side of the network connection. A production server usually handles tens or hundreds of simultaneous connections, so a value of 1 will usually be inadequate. For example, when using ab to perform load testing against these sample programs with 100 concurrent HTTP 1.0 clients, any backlog value less than 50 would often produce performance degradation.

TCP Options

The TCP_CORK option can be used to “bottle up” messages until they are ready to send. This option, illustrated in lines 34 and 40 of Examples 5, might be a good option to use for an HTTP server using HTTP/1.1 pipelining.

Example 5

 1  import socket, select
 2
 3  EOL1 = b'\n\n'
 4  EOL2 = b'\n\r\n'
 5  response  = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
 6  response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
 7  response += b'Hello, world!'
 8
 9  serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
10  serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
11  serversocket.bind(('0.0.0.0', 8080))
12  serversocket.listen(1)
13  serversocket.setblocking(0)
14
15  epoll = select.epoll()
16  epoll.register(serversocket.fileno(), select.EPOLLIN)
17
18 try:
19     connections = {}; requests = {}; responses = {}
20     while True:
21        events = epoll.poll(1)
22        for fileno, event in events:
23           if fileno == serversocket.fileno():
24              connection, address = serversocket.accept()
25              connection.setblocking(0)
26              epoll.register(connection.fileno(), select.EPOLLIN)
27              connections[connection.fileno()] = connection
28              requests[connection.fileno()] = b''
29              responses[connection.fileno()] = response
30           elif event & select.EPOLLIN:
31              requests[fileno] += connections[fileno].recv(1024)
32              if EOL1 in requests[fileno] or EOL2 in requests[fileno]:
33                 epoll.modify(fileno, select.EPOLLOUT)
34                 connections[fileno].setsockopt(socket.IPPROTO_TCP, socket.TCP_CORK, 1)
35                 print('-'*40 + '\n' + requests[fileno].decode()[:-2])
36           elif event & select.EPOLLOUT:
37              byteswritten = connections[fileno].send(responses[fileno])
38              responses[fileno] = responses[fileno][byteswritten:]
39              if len(responses[fileno]) == 0:
40                 connections[fileno].setsockopt(socket.IPPROTO_TCP, socket.TCP_CORK, 0)
41                 epoll.modify(fileno, 0)
42                 connections[fileno].shutdown(socket.SHUT_RDWR)
43           elif event & select.EPOLLHUP:
44              epoll.unregister(fileno)
45              connections[fileno].close()
46              del connections[fileno]
47  finally:
48     epoll.unregister(serversocket.fileno())
49     epoll.close()
50     serversocket.close()

On the other hand, the TCP_NODELAY option can be used to tell the operating system that any data passed to socket.send() should immediately be sent to the client without being buffered by the operating system. This option, illustrated in line 14 of Example 6, might be a good option to use for an SSH client or other “real-time” application.

Example 6

 1  import socket, select
 2
 3  EOL1 = b'\n\n'
 4  EOL2 = b'\n\r\n'
 5  response  = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
 6  response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
 7  response += b'Hello, world!'
 8
 9  serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
10  serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
11  serversocket.bind(('0.0.0.0', 8080))
12  serversocket.listen(1)
13  serversocket.setblocking(0)
14  serversocket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
15
16  epoll = select.epoll()
17  epoll.register(serversocket.fileno(), select.EPOLLIN)
18
19 try:
20     connections = {}; requests = {}; responses = {}
21     while True:
22        events = epoll.poll(1)
23        for fileno, event in events:
24           if fileno == serversocket.fileno():
25              connection, address = serversocket.accept()
26              connection.setblocking(0)
27              epoll.register(connection.fileno(), select.EPOLLIN)
28              connections[connection.fileno()] = connection
29              requests[connection.fileno()] = b''
30              responses[connection.fileno()] = response
31           elif event & select.EPOLLIN:
32              requests[fileno] += connections[fileno].recv(1024)
33              if EOL1 in requests[fileno] or EOL2 in requests[fileno]:
34                 epoll.modify(fileno, select.EPOLLOUT)
35                 print('-'*40 + '\n' + requests[fileno].decode()[:-2])
36           elif event & select.EPOLLOUT:
37              byteswritten = connections[fileno].send(responses[fileno])
38              responses[fileno] = responses[fileno][byteswritten:]
39              if len(responses[fileno]) == 0:
40                 epoll.modify(fileno, 0)
41                 connections[fileno].shutdown(socket.SHUT_RDWR)
42           elif event & select.EPOLLHUP:
43              epoll.unregister(fileno)
44              connections[fileno].close()
45              del connections[fileno]
46  finally:
47     epoll.unregister(serversocket.fileno())
48     epoll.close()
49     serversocket.close()

Source Code

The examples on this page are in the public domain and available for download.

Posted in 翻译的.


在公司《验证码开发和经验分享》讲座的PDF

幻灯片见这里: 验证码开发和经验分享(PDF)

课后反馈居然得了4.5分(5分满分),嘿嘿,个人以为第一次这么紧张的情况下这已经相当不错了~

Posted in 偶尔那么一激灵.

Tagged with , .


Zombie进程笔记

From wikipedia

  1. Zombie不是Orphan,zombie的父进程有可能仍然活着。
  2. Zombie所占资源已经释放,但仍然保留了Process table 的entry,所以还占着进程号和一个process记录
  3. Zombie出现一般是父程序有Bug,或者负载过高导致,当然,也可能就是父进程想要的结果(将SIGCHLD的handler设置为 SIG_IGN, 可以保证下一个子进程的pid与之前的不同)。
  4. 如果父进程还活着,那么可以给父进程一个 kill SIGCHLD,由父进程来处理子进程的退出问题;或者直接kill SIGTERM 父进程,使其变成orphan,被init收养后会定期(很快)的清理掉。
  5. 一般情况下,Zombie不会对系统有什么影响,但当系统overload的时候可能还是需要手动清除一下……
  6. zombie在某些情况下,会被标记为 defunct

Posted in 学到的.

Tagged with .


Okay, 终于回来了……

其实有个博客还是蛮好的~

从09年7月的一次宕机导致数据丢失之后,博客一直没有重新搭起来,因为丢了几篇当时刚写的文章,很是伤心哪……话说 rapidxen的vps还的确是得自己做好备份,血的教训摆在这里……

在现在的主机上,因为PHP是5.3.1,大堆的ereg deprecated 阿……用 E_ALL &~E_DEPRECATED & ~E_USER_DEPRECATED 竟然关不掉,难道是跟我没编译ereg库导致的?
终于升级到了3.0.1,主题也升级了一下,插件没升级-_-,不管怎样,好了不少了……

Posted in 乱弹.


IE 6/ IE 7+ / FF 通吃的防止页面缓存的Http头

1
2
3
4
5
6
7
header('Pragma: no-cache');// 为兼容http 1.0
header('Expires: Mon, 6 Aug 1984 00:00:00 GMT'); // 设置过期时间到一个很早之前……
header('Cache-Control: no-cache, no-store, must-revalidate, max-age=0');
//设置no-cache, 不缓存
//设置 no-store, 让 ie6和 ff 不缓存
//设置must-revalidate,强迫浏览器每次重新验证
//max-age=0…… 只是为了备份,反正都写这么多了,不差这一个了(理论上效果和Expires 头相似)……

Posted in 好东东, 学到的.

Tagged with , , .


关于 count(*) 和 count(col) 的种种

MysqlPerformance的原文

这个博客是聚宝盆,连评论也是啊~

1. MyISAM 下,count(*) 在大多数情况下比 count(col) 快。因为count(*) 能用任何一个索引辅助扫描

2. MyISAM 下,count(col) 的速度取决于 col 的类型以及索引情况。 col 如果是 not null的,要比null的快,因为不需要去全表扫描null值。如果索引中包含col 字段,则扫描将使用索引,可以大大提高速度。针对count(col)进行优化时,需要explain以查看一下count(col)是否使用到了索引。

3. MyISAM 下,加了where子句(range) 的情况下,count(col)的索引需要以col和where子句中的范围限制字段做多字段联合索引,至于多字段索引中的字段顺序,博客中只提到了where field在前 col field在后的情况,不知道反过来是否也有同样效果。有一点需要注意,因为多字段索引的数据量要比单键的索引大一倍以上,索引扫描时间对单键索引时间会成线性增加。

4. MyISAM 下, count(*) == count(0) (内部解析都是相同的),count(1) != count(0) ,不过性能应该差别不大。

5. INNODB下,row count不会被缓存(MyISAM会,原因注:INNODB是row-level lock而MyISAM是table-level lock)。所以INNODB有并发控制(而MyISAM实际上无并发),且INNODB理论上不同的事务中获取到的count(*)是不同的,每做一次count(*)都需要全表扫描一次,很慢~~在INNODB下,count(col)在大多数情况下比count(*)快很多。

6. INNODB下,row count虽然不会被缓存,但count(col)会自动使用较小(应该是索引文件长度吧~~)的索引,总的来说性能还是不错的。加了range以后同样。

番外 :

1.  看来INNODB有接近SQL Server的本事。MyISAM很多时候需要数据库设计者根据经验去手动设计优化,而INNODB大部分时间都是自动的去优化,可以获得接近于MyISAM的性能。

2.  博文评论中提到了在InnoDb使用trigger或者每次Insert/Update后执行一次count然后存入一个“缓存”表的做法。针对这个有一个技巧可以优化:update random of them and run select sum(count) to get the value(没看懂……貌似跟PHP的掷骰子GC机制很相似)

3.  据一大牛同事说,不论是MyISAM还是InnoDb在高并发的情况下,count(*)都是不准的。MyISAM的row count在高并发下更新有些问题。所以如果要精确统计总数的话,使用count(pk)更准确;而应付分页的总数,count(*)也足够了。

Posted in 好东东, 学到的.

Tagged with , , , , .


SVN update 小技巧一则

如果你的库只是一个很大的库中的一个很小的部分,如果你的库里有很多你不需要的东西,还tm超级占磁盘空间(该死的svn repo是plain不压缩存储的……),然后你还需要时不时的看看总库里有什么更新,那么可以看看下面这个命令:

svn update –depth immediates

该参数全部解释如下:

1
2
  --depth ARG              : limit operation by depth ARG ('empty', 'files',
                            'immediates', or 'infinity')

怎么用呢?费点儿事啊:

从第一级目录开始,用  svn up –depth immediates,找到需要更新的子目录,继续执行这个命令,直到需要完全更新的目录,执行 svn up –depth infinity

这样,你费了点儿事,但是只在根目录更新一次,就只更新所有自己关注的目录,其他无关的一概不管!!!

如果你用TortoiseSVN,则应该用 Update to Revision… => 在Update Depth中选择相应的选项即可达到与以上命令行相同的效果。

清爽,开心~

—————————————

备注:

1. svn update 还有一个参数–set-depth,貌似是用来直接设置某个路径的默认更新depth的,以后再研究。反正我一直是用乌龟svn的

2. 即使指定了–depth immediates,svn client也一样会检索所有的数据,在客户端进行过滤,所以,第一次做的时候速度依然很慢……残念啊~

Posted in 偶尔那么一激灵, 好东东, 曾经的拦路虎.

Tagged with .