Monday, April 14

Add memory info to python logging: several options?

Our program is using a lot of memory. Too much of memory, maybe. So I want to write memory of current process in logging information.

Getting the memory size of current process is the easy part (in Linux):
I can think of several ways to add this into logging system: Adding a handler (derive the handler from either StreamHandler or FileHandler), or change the Formatter. Finally I decided to add a filter:
class MemFilter(logging.Filter):
    def filter(self, record):
        record.mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        return True
logging.basicConfig(format='[%(levelname)s][MEM:%(mem)s] [%(asctime)s]:%(message)s')

Case closed.

Wednesday, April 2

Display current time for each command, in Python Interactive Interface

I like making the prompt sign (ps) into showing current time, so that I know exactly what time I executed a command, and what time it is completed. I made that for DOS command, Linux terminal, and MySQL terminal. Here I will show you how to do that in Python Interactive Interface (Python Interpreter).

Here is what it is after setting:

The first thing, setup a $PYTHONSTARTUP environment variable to point to a python file location. Exactly how to do that depends on your operating system. Now matter how it is done, the file it points to will be executed when "python" (Python Interactive Interface) is called.
In my Ubuntu, execute
Next step, in my home folder "~", I created the file as:
# filename:
# after setting $PYTHONSTARTUP=~/, this file will be executed before python interactive interface is open
import sys, time
sys.ps1 = time.strftime("%A %H:%M:%S") + ">>>"
Pretty simple, then the prompt sign (I guess that is the full name of "ps) will be current time, as "Wednesday 16:49:51" showing in the above screenshot.

Friday, March 28


《南都之夜》(曲:许石 词:郑志峰)
我爱我的妹妹啊 害我空悲哀
彼当时在公园内 按怎你甘知
看见月亮渐渐光 有话想要问
请妹妹你想看觅 甘苦你甘知
我爱我的妹妹啊 相招来[辶日]迌
黄昏时在爱河边 想起彼当时
双人对天有立誓 我无娶你无嫁
亲像风雨浇好花 何时再相会
双人对天有立誓 我无娶你无嫁
亲像风雨浇好花 何时再相会




Saturday, March 22

并非敌人Not the enemy

译者前言:有感于国内正在收紧网络言论的控制,罔顾网民(人民)的隐私/权利,我把2600去年夏天的编辑弁言Not the enemy的一段翻译一下。黑客只是好奇于手册中没有记载的漏洞,热爱追求自由的一群人。他们不是敌人。


-- 2014/03/22 注:这是2010年4月敲下来的,不知为何却没有发出去。

Labels: , ,

Friday, March 21

A bug, definitely a bug, in MySQL

First of all, I have a very complicate query to insert from select group by on duplicate update. So I need to add "select * from () " from the group by subquery, then this bug is shown: The subquery result is different from "select * from (subquery)"!

The subquery is also very complicate, but I can simplify it as following:
create table test(
firstdate datetime default null,
lastdate datetime default null
I need to find the last date of this table, but either of the 2 fields can be null. So the subquery I came up with is:
select GREATEST( IFNULL(MAX(firstdate), 0), IFNULL(MAX(lastdate), 0)) from test;
That works perfectly. You can insert some dates for testing:
insert into test(firstdate) values('2014-03-08');
insert into test(firstdate, lastdate) values(' 2012-02-01', '2013-03-08');
Then the query will have expected result:
'2014-03-08 00:00:00'
Here is the wired part: Adding select * from () will have different result!
select * from (
        select GREATEST( IFNULL(MAX(firstdate), 0), IFNULL(MAX(lastdate), 0)) from test) t
Change the "0" in subquery to a datetime string such as '0000-00-00 00:00:00':
select GREATEST( IFNULL(MAX(firstdate), '0000-00-00 00:00:00'), IFNULL(MAX(lastdate), '0000-00-00 00:00:00')) from test;
Then select * from (subquery) will have the expected result.

Looks like that because of the "0" value, the IFNULL changed the result column somehow into a string? Even if all the firstdate and lastdate are filled (there is no null value in the table), the result is still the same.
But if the result column property is modified, why the subquery itself works fine?

MySQL 5.5.

Friday, March 14

Using python to execute a .sql file for mysql

The task is to execute one import.sql file to import several data files into mysql. The trick is that it is known some files will be missing, so the execution of the import.sql will have error, but we want to ignore the error, import whatever exists in the harddrive.

There are two ways to do that:

1, Execute mysql command line, feed in the file as pipe input:
    f = open("import.sql")
    process = Popen(['mysql', '--force'], stdout=PIPE, stdin=f, stderr=PIPE)
    out, err = process.communicate()
The trick is to use '--force' argument of mysql to ignore the error. This is equivalent to:
mysql --force < import.sql
The err will have the error output for missing data file, then you can write that into log file.
2, Execute mysql  command line client, then execute "source import.sql":
    from subprocess import Popen, PIPE
    process = Popen('mysql', stdout=PIPE, stdin=PIPE, stderr=PIPE)
    out, err = process.communicate(input="source import.sql")
Last time I checked, this way was not working very well. It always failed at the missing data file step.
Reference: : "--force is intended to be used in Batch Mode only."


Thursday, March 13

python: replacing strings in file

In perl, this one-liner is easy:
perl -p -i -e 's/replace this/using that/g' filename
"-i" means to write the modified text back to the same file.

When it comes to python, to make the files in line, you will need fileinput to specify inplace=1 .
import fileinput
for line in fileinput.FileInput(filename, inplace=1):
    line=line.replace("replace this","using that")
    print line
Somehow the program generate 2 line breaks in my Windows machine, for each line: \r\n becomes \r\n\r\n.

The other way is to open the same file twice, one for reading, and one for writing:
s = open(filename).read()
s = s.replace('replace this', 'using that')
f = open(filename, 'w')
This is "python way" because s is operated as a set, making the program easy to read, with high run-time efficiency.


Friday, February 21

The end of an era: Windows XP is dead (next month)

On April 8, 2014, Microsoft will stop supporting Windows XP. That means, the resource that monitoring the security issues and creating fixes for this 12-year-old operation system will be allocated somewhere else, inside Microsoft, Symantec, McAfee, and every software company.

Windows XP is a great operating system. No need to argue about that. But the fact is that it was released 12 years ago, and there are so many things changed during this period: The heads of all countries are changed already, including Chávez,  Castro, Bush, Putin... Wait, Putin is still the president of Russia, after all these years? That's weird.

Anyway, what I want to say is that technology has been changing A LOT in the last 12 years, so Windows XP is not suitable for this changing world, fundamentally. Everyday there are a lot of resource being applied to secured this outdated operating system, in such a situation that people know how to solve the problem correctly, and the problem is already fixed beautifully in the next version, but we still need to fix this old version in a distorted way, fixing this specific problem without creating new problem in this broken system. It is not fun at all. It is not Microsoft's fault: Nobody can make an operating system that can be used perfectly for 12 years.

Every Windows XP user should upgrade to new operating system: Windows 7, 8, or Linux. Don't ever use XP again after the dead line. Every XP will become an easy target after Microsoft (and other software companies) stop patching it.

test after removing Google+ comment

Friday, February 14





后座的三岁的儿子也奶声奶气地跟着大叫:“家里没牛奶啦~~ 好惨啊!!”

Monday, January 20

The Famous Track 1!

丫丫在客厅用Kindle 放音乐:“Country road, take me home, to the place, I belong...”。走过来问我:"Daddy, do you like the song?"

"Sure, that is my favorite. The name is 'Country road, take me home'."

"The name is Track 1. The famous Track 1!"