Wednesday, September 16

Setting GoDaddy hosted site to use Python3/Flask

I planned to actually use my hosted site of fadshop.net in GoDaddy, so last time I explored using C# in Windows hosted site (Using Visual Studio to build a website, and publish it in the GoDaddy hosting site). 

Yes, it was working, but my patience in Windows is thin. Now I switched the site back to Linux.

Again, the software in the GoDaddy hosted server is outdated. Last time, in the Microsoft hosted site, the server only has ASP.NET, doesn't have ASP.NET Core which was published since 2016. Now, the python version is 2.6.6, which was released in 2010. 

So the first thing I did was to find a way to run Python 3 for my website that I planned to build.

After reading dozens of pages about this topic, I found the method was to compile and install python3 under your own folder as non-root user. At first I thought that was a hack, until I found a blog page of GoDaddy:  How to install and configure Python on a hosted server , using the same method, then I was sure that is the right/correct/plausible way of getting things done.

My main reference pages are:
https://towardsdatascience.com/installing-python-3-and-flask-on-godaddy-1635fe6f24bc
https://www.godaddy.com/garage/how-to-install-and-configure-python-on-a-hosted-server/
https://www.reddit.com/r/linuxquestions/comments/c5wxh0/help_with_error_on_install_of_python37_from/

All these pages ( and other Googled pages) gave similar solutions but all of them failed during my practice. So I have to consult dozens of other pages (including C compiler parameter setting information)  and add my knowledge into this practice to make it work. Hopefully this is also helpful for you.


First of all, this is a clean GoDaddy hosted "Economy Linux Hosting with cPanel". I think it is CentOS 6 or 7. You can either turn on "SSH access" in your hosting server page, or get to the Terminal in the cPanel.

I have been using Linux for a long time, and usually I am the root/admin of the server, and the software can be easily installed using yum or apt-get. Now with a non-root account, things are very different. The idea is to install all software/library into one folder under this non-root account. In here I am using  "~/.python/" folder,  and it is also /home/zwsjyo79b9/.python, $HOME/.python in the following scripts:

Step 1, set path, library path environment variables:
 
export PATH=$PATH:~/.python/bin
export LIBRARY_PATH=$LIBRARY_PATH:~/.python/lib:~/.python/lib64
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/.python/lib:~/.python/lib64
export C_INCLUDE_PATH=$C_INCLUDE_PATH:~/.python/include
export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:~/.python/include
export PKG_CONFIG_PATH=~/.python/lib/pkgconfig:$PKG_CONFIG_PATH
export LD_RUN_PATH=~/.python/lib:~/.python/lib64
You can put this into your ~/.bash_profile for future use. The lib64 in this page is my invention and the key difference between this post and other people's pages. Some of the software created library files in this folder. It might be a new thing, so the other pages didn't mention it, but without it, the python compilation will fail miserably.


Step 2, before compiling python3, we need to have 3 things ready:
1. openssl package. Without it, the pip of the new python3 will not work. You will get this message when running pip:
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
A python without pip is useless. I blame python for letting itself compiled at all without openssl. It wasted me a lot of time on that.
2, libffi. Without it, the python package will failed with this message when installing some package like jasonpickle or some other normal operation:
ImportError: No module named '_ctypes'
Again, I blame python for letting itself compiled without this package.
3, sqlite. I will use sqlite in my python program, and I can't just do "pip install pysqlite" at that time. Although this GoDaddy server already has sqlite running, it is required to compile and install it (with libraries) for the compilation of python3 to include sqlite module.

So now I create one folder ~/tobeinstall/ , and google for these 3 packages (openssl, libffi, sqlite) and downloaded their source code packages. At the time of this post, they are retrieved by:
wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
wget https://github.com/libffi/libffi/releases/download/v3.3/libffi-3.3.tar.gz
wget https://www.sqlite.org/2020/sqlite-autoconf-3330000.tar.gz
then "tar xzvf xxxx" to unzip them into corresponding folders.
Execute these commands to compile and install them:


cd openssl
CFLAGS=-I/home/zwsjyo79b9/.python/include LDFLAGS=-L/home/zwsjyo79b9/.python/lib ./config --prefix=$HOME/.python
make
make install

cd ../libffi
CFLAGS=-I/home/zwsjyo79b9/.python/include LDFLAGS=-L/home/zwsjyo79b9/.python/lib ./configure --prefix=$HOME/.python
make
make install


cd ../sqlite3
CFLAGS=-I/home/zwsjyo79b9/.python/include LDFLAGS=-L/home/zwsjyo79b9/.python/lib ./configure --prefix=$HOME/.python
make
make install

The idea is to use --prefix=$HOME/.python to specify the installation folder. 
Note: The libffi will install some library files in the ~/.python/lib64 folder. This is not mentioned in other discussions about compiling python3.


Step 3, with all these packages installed, we can start getting python source code:
wget https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tgz
and untar it. Before compilation, we need to modify one file in the unzipped folder:
vi Modules/Setup


Uncomment the lines about SSL, and give the correct path to SSL

Now with this specific command, you can start building:
CFLAGS=-I/home/zwsjyo79b9/.python/include LDFLAGS="-L/home/zwsjyo79b9/.python/lib -L/home/zwsjyo79b9sc/.python/lib64" ./configure --prefix=$HOME/.python --enable-loadable-sqlite-extensions  --with-openssl=/home/zwsjyo79b9/.python 

The configuration process will take several minutes to complete. Make sure you see:

checking for openssl/ssl.h in /home/zwsjyo79b9/.python... yes
checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes
That is the proof that the openssl is correctly installed, and that is the guarantee that pip will perform correctly.

Now it is time to compile it:
make
Check the output like this:


Make sure "sqlite" and "ctypes" are not in the "The necessary bits to build these optional modules were not found" group.

Now you can do "make install" to complete the installation. With the "export PATH=$PATH:~/.python/bin"  you put in the beginning of this tutorial, you should be able to execute "python3" now.


Now we get to the part to actually use it for your application. First of all, my suggestion is to use a subdomain to play with this, keep to main site as default. To do that, you need to create a subdomain in your cPanel. For example, when I create a "deepparser" subdomain, the system will automatically set the default file path as "/home/zwsjyo79b9/public_html/deepparser.fadshop.net".  The rest of this tutorial are mostly copied from the second part of  Installing Python 3 and Flask on GoDaddy with some modifications:

public_html/deepparser.fadshop.net/
├── .htaccess
└── cgi-bin/
├── app.cgi
├── app.py
├── templates/
│ └── home.html
└── venv/



Create a cgi-bin folder in this subdomain, then execute:
pip3 install virtualenv
virtualenv -p /home/zwsjyo79b9/.python/bin/python3 venv    # make sure the created virtual env is using the lately installed python3 environment
source venv/bin/activate
pip install Flask                    # if you have other package to install, now it is the good time
deactivate

Now create the cgi-bin/app.cgi file with this content:

#!/home/zwsjyo79b9/.python/bin/python3
import os
import sys
sys.path.insert(0,
                '/home/zwsjyo79b9/public_html/deepparser.fadshop.net/cgi-bin/venv/lib/python3.8/site-packages')
from wsgiref.handlers import CGIHandler
from app import app
class ProxyFix(object):
    def __init__(self, app):
        self.app = app
    
    def __call__(self, environ, start_response):
        environ['SERVER_NAME'] = ""
        environ['SERVER_PORT'] = "80"
        environ['REQUEST_METHOD'] = "GET"
        environ['SCRIPT_NAME'] = ""
        environ['QUERY_STRING'] = ""
        environ['SERVER_PROTOCOL'] = "HTTP/1.1"
        return self.app(environ, start_response)
if __name__ == '__main__':
    app.wsgi_app = ProxyFix(app.wsgi_app)
    CGIHandler().run(app)
app.wsgi_app = ProxyFix(app.wsgi_app)
CGIHandler.run(app)
#Note: The origin page has error in the 5th line about sys.path.insert(). Information in this page is correct.
And create a simple Flask application in cgi-bin/app.py:
from flask import Flask, render_templateapp = Flask(__name__)@app.route('/')
def home():
return render_template('home.html')

Now create a cgi-bin/templates/home.html  and type anything inside. We are just trying to prove the content in this file will be served when visitor visit the subdomain "http://deepparser.fadshop.net/"

Now comes to the last file, .htaccess in the subdomain folder. For example, in my server, this file is /home/zwsjyo79b9/public_html/deepparser.fadshop.net/.htaccess
The content:

Options +ExecCGI
AddHandler cgi-script .py
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ /home/USERNAME/public_html/python/cgi-bin/app.cgi/$1 [L]
Now we have all the files, but one more thing to do: Set all the files as 755, executable, but not writable for other users. 
chmod 755 *
This is required for the system to work.

If you have to understand the logic, at first the .htaccess in the main folder of the subdomain has a ReWriteRule to redirect all request (link) into the cgi-bin/app.cgi , then the app.cgi call the newly installed python3 with the library of the virtual environment (cgi-bin/venv/lib/python3.8/site-packages) to handle the request, and pass the request to app.py, which is a standard Flask application.

Now the server (this subdomain) is ready. Fire up your browser to visit it:
 














Wednesday, September 9

从世界末日到烟花扬州

 今天天象很奇特,中午时分,外面惨黄一片,很像以前暴雨之前飞沙走石的颜色,其实一点风都没有。听说是山火的烟雾升上高空,把太阳都遮住了。总之,一副世界末日的惨象。


说起“世界末日”,我就跟老婆吹嘘:“你知道‘世界末日’怎么翻译?”

“Then end of the world?”

“不是,在西方文化里,应该翻译成 Judgement Day.”

“为什么?”

圣经经常说起,有一天,神会回到我们人间,到时谁该上天堂,谁该下地狱,就会被一一裁决。圣经里还用了整整一本书《启示录》(Book of Revelation)来描写这一天的各种异象,各种灾殃。


总之,需要对西方文化有足够的了解,才能理解从“世界末日”到“Judgement Day”的翻译。与之可喻美的一个翻译是将“烟花三月下扬州”的三月翻译成April。 啊?词典上,三月不应该是March吗?这时候你就需要理解古代农历的三月是现代的春光明媚的四月份,而不是春寒料峭的三月。必须翻译成April,才能让人享受明媚春光、花团锦秀的扬州时光。


说完文化和翻译,循例放一首应景的《直到世界末日》,歌手齐秦:

Tuesday, September 8

学西班牙语

 这段时间来,我一直用 duolingo.com 自学西班牙语。


这天和妻子出去跑步,在街头经过两个路人,他们正在聊天,用的正是西班牙语!我很激动,跟妻子说:

“我学了这么久的西班牙语,终于有了用场!我听出来了,刚才他们聊天用的是西班牙语!”


妻子也替我高兴:“很高兴你的学习派上了用场!他们说了什么?”

我按耐不住自己的激动,说:“他们说的是西班牙语!”


妻子进一步问:“你听见了什么?”

我很耐心地又说一遍:“我听出来了,他们说的是西班牙语!”


Monday, August 31

关于回车符、换行符

 

打字机(这个组里用过原始机械打字机的人不多)上有两个机械装置,一个是回到句首,另一个是换行,真正使用时这两个机械装置是联合在一起的,成为回到句首并换行,就是 ENTER 键的作用。这就是我们所说的"回车".

 

回到句首在英文里是 Carriage Return, 就是让那个附带着纸的转盘回到原始位置,打字口对着行首

换行在英文里是"Line Feed",让纸往上走1行,打字口就对着下一行。有时也可以翻译成进纸

所以有时你能看到 CRLF,就是指这两个操作。



在电脑发明之后,国际标准设计了 ASCII表,比如 65来表示A66表示B;其中还用13来表示换行 Line Feed, 10来表示回到句首 Carriage Return。在Word, textpad上,只要发现文件里这字节是1013在一起,就从下一行开头显示下一个字节。

这样的不能直接显示,但是有意义的字符我们叫做控制字符

Unix/Linux/Mac里,设计人员偷懒,因为这两个字节总是连在一起的,那么就用其中一个就足以表示换行(并回到句首)了,所以他们只用 ASCII 10 Carriage Return 来做这件事情。Linux是爽了,但是跟Windows交换文件时就经常出现混乱,所以还特地写了两个程序来替换回车符,一个叫做 dos2unix,另一个叫做 unix2dos (很没有创新感)。



总说“ASCII 10 Carriage Return ” “ASCII 13 Line Feed ” 很麻烦,写在程序上也乱,所以我们给了这两个字符别称,第一个叫做 '\n', 第二个叫做 '\r' 。在C语言里,单引号表征字符,比如 char c='\n';  大家一看就知道这是一个ascii 10 的控制字符,在显示时应该走到下一行;而如果用 string s="\n"; 这就是两个字符,分别是 ascii=92 '\' ascii=110'n' python语言里没有单引号和双引号的区别,所以它规定:如果 \ 后面带着有意义的字母,比如 \r, \n, \t, \f ,它就认为这是控制字符,就把 "\" 和后面的字母 (这两个字符)记录成一个控制字符的ascii码。 如果你要拒绝这个转义,在代码里你要写两个斜杠。例子:





>>> print("a\nb")

a

b

>>> print("a\\nb")

a\nb



能看到 "a\nb" 和 "axb"一样都是3个字节。

希望从这个例子里你能看出:我们必须分清楚,当我们说 "\n"的时候,是表示一个字节的控制字符?还是表示能够看得见得两个字符!



Saturday, August 22

中国央行的“数字货币"

 中国央行发布的“数字货币”也叫“数字人民币”,专业词是DC/EP,其实就是一个银行账号的电子支付手段而已。

换一句话说,你银行账号里的钱直接连接到手机上的专用APP里,收取、支付时,钱直接反应到你的银行账号中。以前大家也有用银行卡来交易,当然很少小额交易,但是概念是一致的,现在的“革命性的数字货币”就是用手机来实现那个银行卡的功能,增加小额支付的方式,如此而已。


除此之外,大家一般说起的“数字货币”是说比特币式的加密货币(Crypto-currency),因为比特币在前段时间(2017年底)升值到两万美金一枚而引起众人瞩目。它的特点是:发布之后,每一段时间发行的数量是确定的,没有政府可以改变。与之相对应的是各国家货币,随时可以由国家央行按照当前经济的运转状况而增发(或减少)。比特币的价值之一就是它的这个确定性和稀缺性,不会因货币滥发而贬值。当然,比特币是不是有价值,会不会因为其他原因而贬值,这是另外一回事。它就像贝壳一样,古代把贝壳当作货币(《盐铁论》中说:“夏后以玄贝,周人以紫石,后世(春秋战国)或金钱刀布。”),连“货”字都是以“贝”为字根,它就有了价值,当大家换以紫石、金银为货币之后,贝壳就一钱不值了。


简单介绍一下,比特币从2009年发布,以数字性、去中心化为特点,慢慢发展了一段时间,从“一个币买一块比萨饼”发展到2017年底“一个币换两万美金”,全球关注,它所采用的区块链技术被深深研究,众多追随货币(copy-cat)也不断发行。这类货币的特点是把账本完全公布出来,全球监督,因此只能一次性写入,无法改动。从这个角度(人人都能看到账本)上说,它们叫做“加密货币(crypto-currency)”是完全错的!在中国也经常说是“数字货币”,但是请一定把这种数字货币跟最近中国央行发行的数字人民币区别开来!


Thursday, August 6

Using Apple Magic mouse in Windows

The Apple Magic mouse is a bluetooth mouse, and it is compatible with most (if not all) windows computer. It can be easily paired with, then you can start using the left click and right click.

Except the scrolling functionality.

The scrolling functionality of the Magic mouse is so different from the traditional Windows mouse, that I can't really describe how it is, if you haven't use it yet. But it is so unique, that every Mac user can't forget. And yes, that is pretty much the reason why I want to use the Magic mouse in this Windows laptop.

Now I need to find the driver of it.

There are some apple community discussion  suggested to get the Boot Camp Support software and install the mouse driver, but the page gave a five year old package. I got to brigadier to get the source code running and download some Book Camp packages and finally found the driver I need.

Now I uploaded the package into AppleWirelessMouse.zip  so you can download, unzip, right click the AppleWirelessMouse.inf to install the driver, then the scrolling functionality is in effect immediately.

Disclaimer: The  Boot Camp Support software is supposed to be used to install Windows in a Mac device. I went all the way through to exact the Magic Mouse driver from it, hopefully it is not violating the copyright of the software.

Labels: ,

Saturday, August 1

一首黑人劳动歌曲 Po Lazuras

首先,有这么一首黑人的劳动歌曲,是说可怜的 Lazurus被警察追踪的事。

现在能查到 Bob Dylan在50年代写了同一个名字的歌,在Youtube上也能找到,但说实话,不好听。



坊间还流传着一个老录音拷贝,大概也是50年代的,节奏、韵律更黑人,更劳动一些,所以在2000年一个电影制作过程中要找这段素材时,辗转买到了版权并重新制作,大获成功,长期登上排行榜,并获得格莱美音乐奖!

下面这段是在电影中的音乐片段:


其实这也就是这电影的开头。

回到音乐来,得奖之后,他们反复查找当时的资料,包括监狱记录等等,才确认了这是密西西比一些监狱犯人的录音,甚至找到了一个带头领唱的犯人James Carter,现在在芝加哥生活。飞到那里,当面给了他一张支票,他已经完全不记得40年前曾经唱过这歌,曾经录音。

当他听说歌曲热卖,比 Michael Jackson 的记录还强时,他很高兴,说:“你告诉Michael, 我会慢下来,让他赶上。”

这首歌现在的署名是“James Carter和他的狱友们”,因为其他狱友的名字再也找寻不到了。


歌词 (source: Musixmatch):

Lyrics
James Carter & The Prisoners
Po LazarusPo LazarusJames Carter & The Prisoners(O Brother, Where Art Thou?)Well, the high sheriffHe told his deputyWant you go out and bring me LazarusWell, the high sheriffTold his deputyI want you go out and bring me LazarusBring him dead or alive,Lawd, LawdBring him dead or aliveWell the deputy he told the high sheriffI ain't gonna mess with LazarusWell the deputy he told the high sheriffSays I ain't gonna mess with LazarusWell he's a dangerous manLawd, LawdHe's a dangerous manWell then the high sheriff, he found LazarusHe was hidin' in the chill of a mountainWell the high sheriff, found LazarusHe was hidin' in the chill of the mountainWith his head hung downLawd, LawdWith his head hung downWell then the high sheriff, he told LazarusHe says Lazarus I come to arrest youWell the high sheriff, told LazarusSays Lazarus I come to arrest youAnd bring ya dead or aliveLawd, LawdBring you dead or aliveWell then Lazarus, he told the high sheriffSays I never been arrestedWell Lazarus, told the high sheriffSays I never been arrestedBy no one manLawd, LawdBy no one manAnd then the high sheriff, he shot LazarusWell, he shot him mighty big numberWell the high sheriff, shot LazarusWell he shot him with a mighty big numberWith a forty fiveLawd, LawdWith a forty fiveWell then they take old LazarusYes they laid him on the commissary gharryWell they taken poor LazarusAnd the laid him on the commissary gharryHe said my wounded sideLawd, LawdMy wounded side

Labels: ,