Wednesday, September 16

Setting GoDaddy hosted site to use Python3/Flask

I planned to actually use my hosted site of fadshop.net in GoDaddy, so last time I explored using C# in Windows hosted site (Using Visual Studio to build a website, and publish it in the GoDaddy hosting site). 

Yes, it was working, but my patience in Windows is thin. Now I switched the site back to Linux.

Again, the software in the GoDaddy hosted server is outdated. Last time, in the Microsoft hosted site, the server only has ASP.NET, doesn't have ASP.NET Core which was published since 2016. Now, the python version is 2.6.6, which was released in 2010. 

So the first thing I did was to find a way to run Python 3 for my website that I planned to build.

After reading dozens of pages about this topic, I found the method was to compile and install python3 under your own folder as non-root user. At first I thought that was a hack, until I found a blog page of GoDaddy:  How to install and configure Python on a hosted server , using the same method, then I was sure that is the right/correct/plausible way of getting things done.

My main reference pages are:
https://towardsdatascience.com/installing-python-3-and-flask-on-godaddy-1635fe6f24bc
https://www.godaddy.com/garage/how-to-install-and-configure-python-on-a-hosted-server/
https://www.reddit.com/r/linuxquestions/comments/c5wxh0/help_with_error_on_install_of_python37_from/

All these pages ( and other Googled pages) gave similar solutions but all of them failed during my practice. So I have to consult dozens of other pages (including C compiler parameter setting information)  and add my knowledge into this practice to make it work. Hopefully this is also helpful for you.


First of all, this is a clean GoDaddy hosted "Economy Linux Hosting with cPanel". I think it is CentOS 6 or 7. You can either turn on "SSH access" in your hosting server page, or get to the Terminal in the cPanel.

I have been using Linux for a long time, and usually I am the root/admin of the server, and the software can be easily installed using yum or apt-get. Now with a non-root account, things are very different. The idea is to install all software/library into one folder under this non-root account. In here I am using  "~/.python/" folder,  and it is also /home/zwsjyo79b9/.python, $HOME/.python in the following scripts:

Step 1, set path, library path environment variables:
 
export PATH=$PATH:~/.python/bin
export LIBRARY_PATH=$LIBRARY_PATH:~/.python/lib:~/.python/lib64
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/.python/lib:~/.python/lib64
export C_INCLUDE_PATH=$C_INCLUDE_PATH:~/.python/include
export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:~/.python/include
export PKG_CONFIG_PATH=~/.python/lib/pkgconfig:$PKG_CONFIG_PATH
export LD_RUN_PATH=~/.python/lib:~/.python/lib64
You can put this into your ~/.bash_profile for future use. The lib64 in this page is my invention and the key difference between this post and other people's pages. Some of the software created library files in this folder. It might be a new thing, so the other pages didn't mention it, but without it, the python compilation will fail miserably.


Step 2, before compiling python3, we need to have 3 things ready:
1. openssl package. Without it, the pip of the new python3 will not work. You will get this message when running pip:
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
A python without pip is useless. I blame python for letting itself compiled at all without openssl. It wasted me a lot of time on that.
2, libffi. Without it, the python package will failed with this message when installing some package like jasonpickle or some other normal operation:
ImportError: No module named '_ctypes'
Again, I blame python for letting itself compiled without this package.
3, sqlite. I will use sqlite in my python program, and I can't just do "pip install pysqlite" at that time. Although this GoDaddy server already has sqlite running, it is required to compile and install it (with libraries) for the compilation of python3 to include sqlite module.

So now I create one folder ~/tobeinstall/ , and google for these 3 packages (openssl, libffi, sqlite) and downloaded their source code packages. At the time of this post, they are retrieved by:
wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
wget https://github.com/libffi/libffi/releases/download/v3.3/libffi-3.3.tar.gz
wget https://www.sqlite.org/2020/sqlite-autoconf-3330000.tar.gz
then "tar xzvf xxxx" to unzip them into corresponding folders.
Execute these commands to compile and install them:


cd openssl
CFLAGS=-I/home/zwsjyo79b9/.python/include LDFLAGS=-L/home/zwsjyo79b9/.python/lib ./config --prefix=$HOME/.python
make
make install

cd ../libffi
CFLAGS=-I/home/zwsjyo79b9/.python/include LDFLAGS=-L/home/zwsjyo79b9/.python/lib ./configure --prefix=$HOME/.python
make
make install


cd ../sqlite3
CFLAGS=-I/home/zwsjyo79b9/.python/include LDFLAGS=-L/home/zwsjyo79b9/.python/lib ./configure --prefix=$HOME/.python
make
make install

The idea is to use --prefix=$HOME/.python to specify the installation folder. 
Note: The libffi will install some library files in the ~/.python/lib64 folder. This is not mentioned in other discussions about compiling python3.


Step 3, with all these packages installed, we can start getting python source code:
wget https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tgz
and untar it. Before compilation, we need to modify one file in the unzipped folder:
vi Modules/Setup


Uncomment the lines about SSL, and give the correct path to SSL

Now with this specific command, you can start building:
CFLAGS=-I/home/zwsjyo79b9/.python/include LDFLAGS="-L/home/zwsjyo79b9/.python/lib -L/home/zwsjyo79b9sc/.python/lib64" ./configure --prefix=$HOME/.python --enable-loadable-sqlite-extensions  --with-openssl=/home/zwsjyo79b9/.python 

The configuration process will take several minutes to complete. Make sure you see:

checking for openssl/ssl.h in /home/zwsjyo79b9/.python... yes
checking whether compiling and linking against OpenSSL works... yes
checking for X509_VERIFY_PARAM_set1_host in libssl... yes
That is the proof that the openssl is correctly installed, and that is the guarantee that pip will perform correctly.

Now it is time to compile it:
make
Check the output like this:


Make sure "sqlite" and "ctypes" are not in the "The necessary bits to build these optional modules were not found" group.

Now you can do "make install" to complete the installation. With the "export PATH=$PATH:~/.python/bin"  you put in the beginning of this tutorial, you should be able to execute "python3" now.


Now we get to the part to actually use it for your application. First of all, my suggestion is to use a subdomain to play with this, keep to main site as default. To do that, you need to create a subdomain in your cPanel. For example, when I create a "deepparser" subdomain, the system will automatically set the default file path as "/home/zwsjyo79b9/public_html/deepparser.fadshop.net".  The rest of this tutorial are mostly copied from the second part of  Installing Python 3 and Flask on GoDaddy with some modifications:

public_html/deepparser.fadshop.net/
├── .htaccess
└── cgi-bin/
├── app.cgi
├── app.py
├── templates/
│ └── home.html
└── venv/



Create a cgi-bin folder in this subdomain, then execute:
pip3 install virtualenv
virtualenv -p /home/zwsjyo79b9/.python/bin/python3 venv    # make sure the created virtual env is using the lately installed python3 environment
source venv/bin/activate
pip install Flask                    # if you have other package to install, now it is the good time
deactivate

Now create the cgi-bin/app.cgi file with this content:

#!/home/zwsjyo79b9/.python/bin/python3
import os
import sys
sys.path.insert(0,
                '/home/zwsjyo79b9/public_html/deepparser.fadshop.net/cgi-bin/venv/lib/python3.8/site-packages')
from wsgiref.handlers import CGIHandler
from app import app
class ProxyFix(object):
    def __init__(self, app):
        self.app = app
    
    def __call__(self, environ, start_response):
        environ['SERVER_NAME'] = ""
        environ['SERVER_PORT'] = "80"
        environ['REQUEST_METHOD'] = "GET"
        environ['SCRIPT_NAME'] = ""
        environ['QUERY_STRING'] = ""
        environ['SERVER_PROTOCOL'] = "HTTP/1.1"
        return self.app(environ, start_response)
if __name__ == '__main__':
    app.wsgi_app = ProxyFix(app.wsgi_app)
    CGIHandler().run(app)
app.wsgi_app = ProxyFix(app.wsgi_app)
CGIHandler.run(app)
#Note: The origin page has error in the 5th line about sys.path.insert(). Information in this page is correct.
And create a simple Flask application in cgi-bin/app.py:
from flask import Flask, render_templateapp = Flask(__name__)@app.route('/')
def home():
return render_template('home.html')

Now create a cgi-bin/templates/home.html  and type anything inside. We are just trying to prove the content in this file will be served when visitor visit the subdomain "http://deepparser.fadshop.net/"

Now comes to the last file, .htaccess in the subdomain folder. For example, in my server, this file is /home/zwsjyo79b9/public_html/deepparser.fadshop.net/.htaccess
The content:

Options +ExecCGI
AddHandler cgi-script .py
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ /home/USERNAME/public_html/python/cgi-bin/app.cgi/$1 [L]
Now we have all the files, but one more thing to do: Set all the files as 755, executable, but not writable for other users. 
chmod 755 *
This is required for the system to work.

If you have to understand the logic, at first the .htaccess in the main folder of the subdomain has a ReWriteRule to redirect all request (link) into the cgi-bin/app.cgi , then the app.cgi call the newly installed python3 with the library of the virtual environment (cgi-bin/venv/lib/python3.8/site-packages) to handle the request, and pass the request to app.py, which is a standard Flask application.

Now the server (this subdomain) is ready. Fire up your browser to visit it:
 














Hey, thanks for this guide. Although for some reason, I keep getting this error "Failed to build these modules: _ctypes" after running make for the Python step. This error only shows up with the latest version of Python (3.9.2), works when I use Python-3.8.5
 
omg, you are a life saver! i've tried to solve this ctypes error for months, and rebuild the python with your guide and it has been solved! god bless you!
 
Glad it helped! Not sure what's wrong for 3.9.2...
 
This was really helpful! The only issue I had was I still could not get _ctypes successfully installed. I realized it was because I was using find and replace for your paths and there was the following inconsistency. Which could be helpful for anyone else going over this:

"CFLAGS=-I/home/zwsjyo79b9/.python/include LDFLAGS="-L/home/zwsjyo79b9/.python/lib -L/home/zwsjyo79b9sc/.python/lib64" ./configure --prefix=$HOME/.python --enable-loadable-sqlite-extensions --with-openssl=/home/zwsjyo79b9/.python"

The second LDFLAGS path is home/zwsjyo79b9sc instead of home/zwsjyo79b9 like the others.
 
It's not working for me, after downloading the python file and changing the path of the SSL, I get the "NO":
checking for openssl/ssl.h in /home/zwsjyo79b9/.python... no
checking whether compiling and linking against OpenSSL works... no

I checked inside /home/zwsjyo79b9/.python/openssl and there's no ssl.h there... any help?
 
Great guide!! Thanks so much for this info... I got everything to work except at the end when I'm doing the make for python 3.8.5, it's complaining that it can't find libssl. Wondering if I didn't set the correct path in the Modules/Setup script? My file looks different than the screenshot you provided regarding uncommenting the SSL bits. Any help is appreciated!!
 
Very complete and well written - every step is important! A couple items I didn't read correctly (my fault) were to (1) also uncomment (derp!) the SSL lines in Modules/Setup, and (2) I neglected to set LD_LIBRARY_PATH in my .bash_profile. That was the first step in your tutorial, but the need for it doesn't come up until the end of the python make, which came up the next day at my next login (temp env vars lost of course).

Now I can use pip to get some missing modules needed by my app on the server, which is what I was doing yesterday before I ran into all of this. Glad to see I'm not alone, but sorry to see so many of us in this boat. :)
 
Just want to thank you for a fantastic and very well written guide! Using your guide I was just able to install openssl Version 3.0.1 14 Dec 2021 and Python Version 3.11.0a5 on Godaddy shared hosting. Python compiled perfectly and ctypes, ssl, and sqlite3 all work in Python...finally! Can't tell you how many different attempts I made to get everything working before I came across your guide.

Though I didn't require Flask, the first part of your guide was all I needed to get everything working for my application. Thanks again! For anyone considering Python 3.11, you don't need to edit the Setup file in modules to get SSL working. Perl is required for running some Python tests so depending on your platform you might need to upgrade Perl or do a local Perl install into your home directory.
 
God bless you! I almost lost hope to make it work.
 
This comment has been removed by the author.
 
I'm glad my article could help you in installing python on Flask. :)
 

Wednesday, September 9

从世界末日到烟花扬州

 今天天象很奇特,中午时分,外面惨黄一片,很像以前暴雨之前飞沙走石的颜色,其实一点风都没有。听说是山火的烟雾升上高空,把太阳都遮住了。总之,一副世界末日的惨象。


说起“世界末日”,我就跟老婆吹嘘:“你知道‘世界末日’怎么翻译?”

“Then end of the world?”

“不是,在西方文化里,应该翻译成 Judgement Day.”

“为什么?”

圣经经常说起,有一天,神会回到我们人间,到时谁该上天堂,谁该下地狱,就会被一一裁决。圣经里还用了整整一本书《启示录》(Book of Revelation)来描写这一天的各种异象,各种灾殃。

这一天是这个世界的最后一天,然后所有人就转换场地了。

施瓦辛格的《终结者》2 就是以“Judgement Day”为名字,说在原本历史上将要出现的核弹毁灭,而它与主角将要改变这段历史,改变人类可怜的命运。


总之,需要对西方文化有足够的了解,才能理解从“世界末日”到“Judgement Day”的翻译。与之可喻美的一个翻译是将“烟花三月下扬州”的三月翻译成April。 啊?词典上,三月不应该是March吗?这时候你就需要理解古代农历的三月是现代的春光明媚的四月份,而不是春寒料峭的三月。必须翻译成April,才能让人享受明媚春光、花团锦秀的扬州时光。


说完文化和翻译,循例放一首应景的《直到世界末日》,歌手齐秦:


这首歌的原版是 "A Spaceman Came Travelling"

Tuesday, September 8

学西班牙语

 这段时间来,我一直用 duolingo.com 自学西班牙语。


这天和妻子出去跑步,在街头经过两个路人,他们正在聊天,用的正是西班牙语!我很激动,跟妻子说:

“我学了这么久的西班牙语,终于有了用场!我听出来了,刚才他们聊天用的是西班牙语!”


妻子也替我高兴:“很高兴你的学习派上了用场!他们说了什么?”

我按耐不住自己的激动,说:“他们说的是西班牙语!”


妻子进一步问:“你听见了什么?”

我很耐心地又说一遍:“我听出来了,他们说的是西班牙语!”