Friday, May 9

One tip about logging.debug() in python

The idea of logging.debug( output string ) is that, if the logging system was set as higher level, the output string will be ignored, not being output to the logging system.

One problem, if the logging.debug() is calling some functions to format the output string, that functions ARE executed, no matter what the logging system is.

For example, I used this sentence to find out the size of the critical data object in the environment:
logging.debug("  size of dataprovider1 is: %s", sys.getsizeof(cPickle.dumps(dataprovider1)))
It works fine in debug environment; then when switching to production environment, even though the logging.level was set to INFO, this sentence was not being output to the log file, this function was still being executed, and an object of 90G size was created, and crashed the production system immediately.

Solution: use the isEnableFor() function of the logging.
if logging.root.isEnabledFor(logging.DEBUG):
    logging.debug("  size of dataprovider1 is: %s", sys.getsizeof(cPickle.dumps(dataprovider1)))

Monday, April 14

Add memory info to python logging: several options?

Our program is using a lot of memory. Too much of memory, maybe. So I want to write memory of current process in logging information.

Getting the memory size of current process is the easy part (in Linux):
resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
I can think of several ways to add this into logging system: Adding a handler (derive the handler from either StreamHandler or FileHandler), or change the Formatter. Finally I decided to add a filter:
class MemFilter(logging.Filter):
    def filter(self, record):
        record.mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
        return True
logging.basicConfig(format='[%(levelname)s][MEM:%(mem)s] [%(asctime)s]:%(message)s')
logging.root.addFilter(MemFilter())

Case closed.

Wednesday, April 2

Display current time for each command, in Python Interactive Interface

I like making the prompt sign (ps) into showing current time, so that I know exactly what time I executed a command, and what time it is completed. I made that for DOS command, Linux terminal, and MySQL terminal. Here I will show you how to do that in Python Interactive Interface (Python Interpreter).

Here is what it is after setting:



The first thing, setup a $PYTHONSTARTUP environment variable to point to a python file location. Exactly how to do that depends on your operating system. Now matter how it is done, the file it points to will be executed when "python" (Python Interactive Interface) is called.
In my Ubuntu, execute
export PYTHONSTARTUP=~/.pythonstartup.py
Next step, in my home folder "~", I created the file .pythonstartup.py as:
# filename: .pythonstartup.py
# after setting $PYTHONSTARTUP=~/.pythonstartup.py, this file will be executed before python interactive interface is open
import sys, time
sys.ps1 = time.strftime("%A %H:%M:%S") + ">>>"
Pretty simple, then the prompt sign (I guess that is the full name of "ps) will be current time, as "Wednesday 16:49:51" showing in the above screenshot.

PS: the way for showing the same information in linux command line (Ubuntu) is to modify the PS1:
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\] [\D{%a} \t]\$ '
(add this in .bashrc)

For mysql client tool, add this line under [mysql] of the ~\.my.cnf:
prompt=\\u@\\d [\\w \\R:\\m:\\s]>



Friday, March 28

海角七号:看见月亮渐渐光

海角七号里,老头用威逼利诱,走关系进了乐队,好快活地在这个摇滚舞台弹着古琴唱:
看见月色渐渐光
有话对你讲
妹妹你啊想一想
...
根据百度,这首歌是《南都之夜》
《南都之夜》(曲:许石 词:郑志峰)
我爱我的妹妹啊 害我空悲哀
彼当时在公园内 按怎你甘知
看见月亮渐渐光 有话想要问
请妹妹你想看觅 甘苦你甘知
我爱我的妹妹啊 相招来[辶日]迌
黄昏时在爱河边 想起彼当时
双人对天有立誓 我无娶你无嫁
亲像风雨浇好花 何时再相会
双人对天有立誓 我无娶你无嫁
亲像风雨浇好花 何时再相会
这首《南都之夜》创作于1945年,而就是在1945年发生了一件台湾历史上很重要的一件事,就是日本战败,台湾光复,台湾从此结束了50年的日据时期,回归祖国,而这首歌正是创作在台湾光复后不久。
关于这首歌呢,其实还有一段历史,1945年国民党接收台湾时﹐全面禁止日本歌﹐但是由于日本统治时期限制台湾歌(包括其他台湾的文化)﹐故写歌在当时是无法维生的﹐所以很多艺人都纷纷改行或是转唱日本歌曲,以至于禁止日本歌后台湾同胞每日听来听去的都是些军歌或进行曲﹐后来忽然有首歌曲就在大街小巷流行起来﹐这首歌呢就是《南都之夜》。
在1959年的电影《空中小姐》里,同样的曲子,被唱成了《台湾小调》:



当然,现在更让这首曲子广为流传的是李克勤唱的《旧欢如梦》:

当年相恋意中人,大家性情近;早种爱根极亲密,心心相印互信任。
月底花间相偎依,共喜有缘份;恩爱百般愿比翼,痴心一缕共订盟。
喜逢知己倍精神,内心快乐无憾;朝晚眷恋共欢聚,天天相见互慰问。
立心栽花花不香,仲反惹仇恨;只怨爱海起风波,一朝生变断爱盟。
恩情于今化烟云,未许再续情份;空有爱丝万千丈,可惜都已尽化恨。
枉抛相思枉痴恋,恨卿心太忍;只有叹息旧欢似梦,早经消散莫再寻。
早经消散莫再寻……


喇叭和上海舞步,都是我的最爱。

Saturday, March 22

并非敌人Not the enemy

译者前言:有感于国内正在收紧网络言论的控制,罔顾网民(人民)的隐私/权利,我把2600去年夏天的编辑弁言Not the enemy的一段翻译一下。黑客只是好奇于手册中没有记载的漏洞,热爱追求自由的一群人。他们不是敌人。

黑客们发现问题,告诉大众。他们是奥巴马所宣誓要支持的开放环境的最典范例子。他们不是从公司阴谋中取利的坏人,也没有满世界送垃圾邮件,或用病毒和蠕虫制造混乱。许多年来,媒体造成这样的形象:任何跟网络或计算机有关的坏事,都是黑客做的。讽刺意味的是,这使得那些做了那些坏事的人很骄傲地称自己为黑客。其实他们并不是。只要瞥一下本刊物(2600)上这些持续的对话,任何一个外行都能看出真正的黑客对于做坏事是很严肃的。如果给每个用键盘来做坏事的人这个标签,


-- 2014/03/22 注:这是2010年4月敲下来的,不知为何却没有发出去。

Labels: , ,

Friday, March 21

A bug, definitely a bug, in MySQL

First of all, I have a very complicate query to insert from select group by on duplicate update. So I need to add "select * from () " from the group by subquery, then this bug is shown: The subquery result is different from "select * from (subquery)"!

The subquery is also very complicate, but I can simplify it as following:
create table test(
firstdate datetime default null,
lastdate datetime default null
);
I need to find the last date of this table, but either of the 2 fields can be null. So the subquery I came up with is:
select GREATEST( IFNULL(MAX(firstdate), 0), IFNULL(MAX(lastdate), 0)) from test;
That works perfectly. You can insert some dates for testing:
insert into test(firstdate) values('2014-03-08');
insert into test(firstdate, lastdate) values(' 2012-02-01', '2013-03-08');
Then the query will have expected result:
'2014-03-08 00:00:00'
Here is the wired part: Adding select * from () will have different result!
select * from (
        select GREATEST( IFNULL(MAX(firstdate), 0), IFNULL(MAX(lastdate), 0)) from test) t
'2014-0'
Work-around:
Change the "0" in subquery to a datetime string such as '0000-00-00 00:00:00':
select GREATEST( IFNULL(MAX(firstdate), '0000-00-00 00:00:00'), IFNULL(MAX(lastdate), '0000-00-00 00:00:00')) from test;
Then select * from (subquery) will have the expected result.

Analysis:
Looks like that because of the "0" value, the IFNULL changed the result column somehow into a string? Even if all the firstdate and lastdate are filled (there is no null value in the table), the result is still the same.
But if the result column property is modified, why the subquery itself works fine?

MySQL 5.5.

Friday, March 14

Using python to execute a .sql file for mysql

The task is to execute one import.sql file to import several data files into mysql. The trick is that it is known some files will be missing, so the execution of the import.sql will have error, but we want to ignore the error, import whatever exists in the harddrive.

There are two ways to do that:

1, Execute mysql command line, feed in the file as pipe input:
    f = open("import.sql")
    process = Popen(['mysql', '--force'], stdout=PIPE, stdin=f, stderr=PIPE)
    out, err = process.communicate()
The trick is to use '--force' argument of mysql to ignore the error. This is equivalent to:
mysql --force < import.sql
The err will have the error output for missing data file, then you can write that into log file.
2, Execute mysql  command line client, then execute "source import.sql":
    from subprocess import Popen, PIPE
    process = Popen('mysql', stdout=PIPE, stdin=PIPE, stderr=PIPE)
    out, err = process.communicate(input="source import.sql")
Last time I checked, this way was not working very well. It always failed at the missing data file step.
Reference: http://bugs.mysql.com/bug.php?id=533 : "--force is intended to be used in Batch Mode only."

Labels:

Thursday, March 13

python: replacing strings in file

In perl, this one-liner is easy:
perl -p -i -e 's/replace this/using that/g' filename
"-i" means to write the modified text back to the same file.

When it comes to python, to make the files in line, you will need fileinput to specify inplace=1 .
import fileinput
for line in fileinput.FileInput(filename, inplace=1):
    line=line.replace("replace this","using that")
    print line
Somehow the program generate 2 line breaks in my Windows machine, for each line: \r\n becomes \r\n\r\n.

The other way is to open the same file twice, one for reading, and one for writing:
s = open(filename).read()
s = s.replace('replace this', 'using that')
f = open(filename, 'w')
f.write(s)
f.close()
This is "python way" because s is operated as a set, making the program easy to read, with high run-time efficiency.


Labels:

Friday, February 21

The end of an era: Windows XP is dead (next month)

On April 8, 2014, Microsoft will stop supporting Windows XP. That means, the resource that monitoring the security issues and creating fixes for this 12-year-old operation system will be allocated somewhere else, inside Microsoft, Symantec, McAfee, and every software company.

Windows XP is a great operating system. No need to argue about that. But the fact is that it was released 12 years ago, and there are so many things changed during this period: The heads of all countries are changed already, including Chávez,  Castro, Bush, Putin... Wait, Putin is still the president of Russia, after all these years? That's weird.

Anyway, what I want to say is that technology has been changing A LOT in the last 12 years, so Windows XP is not suitable for this changing world, fundamentally. Everyday there are a lot of resource being applied to secured this outdated operating system, in such a situation that people know how to solve the problem correctly, and the problem is already fixed beautifully in the next version, but we still need to fix this old version in a distorted way, fixing this specific problem without creating new problem in this broken system. It is not fun at all. It is not Microsoft's fault: Nobody can make an operating system that can be used perfectly for 12 years.

Every Windows XP user should upgrade to new operating system: Windows 7, 8, or Linux. Don't ever use XP again after the dead line. Every XP will become an easy target after Microsoft (and other software companies) stop patching it.



test after removing Google+ comment
 

Friday, February 14

我太二了,我

我太二了,我。

昨天回家的路上,经过Vons时妻子说:“家里没牛奶了,过去买两桶吧。”

我就用周星驰在唐府前捏着小强的腔调大声说:“家里没牛奶啦!好惨啊!!”

后座的三岁的儿子也奶声奶气地跟着大叫:“家里没牛奶啦~~ 好惨啊!!”