Enabling Python 2.7 on a PaaS with the OpenShift DIY App Type

Preface

I have been using OpenShift for a few months now and have to say that I still love it. I've sampled most of the other PaaS solutions and feel most comfortable with OpenShift. One of the features I enjoy most in the service is the "do-it-yourself" application. With this, you can run just about anything in a "minimal-hassle", scalable environment.

I created a quickstart a few weeks ago that had Django running on Python 2.7 in "DIY" mode. This will use some of that, but be a more "step-by-step" approach for newcomers. This example will use the Flask micro-framework as well as the uWSGI application container for a true, production ready environment.

Diving In

The first thing you will need to do is get an OpenShift account (it's free).

Client Tools

Install the OpenShift client tools:

gem install rhc

** See here if you have trouble getting the tools installed.

Application Setup

Once you have the toolchain setup, we will create the application environment in OpenShift. For this example, we will name our application "py27".

rhc app create py27 diy-0.1

You will see something similar to:

Creating application: py27 in ehazlett
Now your new domain name is being propagated worldwide (this might take a minute)...
Confirming application 'py27' is available:  Success!

py27 published:  http://py27-ehazlett.rhcloud.com/
git url:  ssh://uuid1234567890@py27-ehazlett.rhcloud.com/~/git/py27.git/
Disclaimer: This is an experimental cartridge that provides a way to try unsupported languages, frameworks, and middleware on Openshift.

You can view the bootstrapped application at http://py27-[namepsace].rhcloud.com

Python 2.7

Now that the application environment is created, we will build and install the latest Python 2.7.x release. To do that, we will first need to get the application SSH credentials.

Run the following to show your application config (enter your account password when prompted):

rhc app show -a py27

You should see something similar to this:

Application Info
================
py27
    Framework: diy-0.1
     Creation: 2012-05-15T22:54:09-04:00
         UUID: 1qaz2wsx3edc4rfv
      Git URL: ssh://1qaz2wsx3edc4rfv@py27-ehazlett.rhcloud.com/~/git/py27.git/
   Public URL: http://py27-ehazlett.rhcloud.com/

 Embedded:
      None

Log into your OpenShift application using the SSH credentials from above (in the Git URL line):

ssh 1qaz2wsx3edc4rfv@py27-[username].rhcloud.com

We will build everything in the OpenShift application tmp directory.

Navigate into the "tmp" directory:

cd $OPENSHIFT_TMP_DIR

Download the latest Python 2.7.x release:

wget http://python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2

Extract:

tar jxf Python-2.7.3.tar.bz2

Configure (to put the custom Python into the OpenShift runtime dir):

cd Python-2.7.3

./configure --prefix=$OPENSHIFT_DATA_DIR

Make and install:

make install

You can now check that Python was successfully installed:

$OPENSHIFT_DATA_DIR/bin/python -V

You should get

Python 2.7.3

Supporting Tools

We now have Python installed, but we can't do much. Part of the common Python toolchain is PIP for Python package management.

Setuptools

Change into the OpenShift "tmp" directory:

cd $OPENSHIFT_TMP_DIR

Download and install setuptools:

wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz

tar zxf setuptools-0.6c11.tar.gz

cd setuptools-0.6c11

Install:

$OPENSHIFT_DATA_DIR/bin/python setup.py install

PIP

Change into the OpenShift "tmp" directory:

cd $OPENSHIFT_TMP_DIR

Download and install setuptools:

wget http://pypi.python.org/packages/source/p/pip/pip-1.1.tar.gz

tar zxf pip-1.1.tar.gz

cd pip-1.1

Install:

$OPENSHIFT_DATA_DIR/bin/python setup.py install

Application

Okay, so now we have all of the parts needed to run the app. We now need to create the Flask application and configure OpenShift so that it knows how to start and stop it.

First we will create the Flask application. Logout of your OpenShift environment and navigate into the Git repository that was created for your app.

Create a file named application.py in the repo, for example - in the diy folder:

    ??? README
    ??? diy
    ?   ??? index.html
    ?   ??? testrubyserver.rb
    ?   ??? application.py
    ??? misc

BTW, the index.html and testrubyserver.rb are not longer necessary and can be removed anytime.

Edit application.py with the following:

from flask import Flask
import platform
application = Flask(__name__)
 
@application.route("/")
def index():
    return 'Hello from Flask'
 
@application.route("/info")
def info():
    return platform.python_version()

Create a requirements.txt in the root application directory with the following:

uWSGI==1.2.3
Flask==0.8

Directory structure should look like:

??? README
??? diy
?   ??? application.py
?   ??? index.html
?   ??? testrubyserver.rb
??? misc
??? requirements.txt

Hooks

We will now create the hooks that are needed for deployment as well as for starting and stopping the application. In your application Git repository, there is a hidden directory called ".openshift" that contains stubs for the action hooks. We will be editing these.

Edit .openshift/action_hooks/build to have the following:

$OPENSHIFT_DATA_DIR/bin/pip install --use-mirrors -r $OPENSHIFT_REPO_DIR/requirements.txt

Edit .openshift/action_hooks/start to have the following:

$OPENSHIFT_DATA_DIR/bin/uwsgi -s $OPENSHIFT_DIY_IP:$OPENSHIFT_DIY_PORT --socket-protocol http --pp $OPENSHIFT_REPO_DIR/diy --module application -d $OPENSHIFT_DIY_LOG_DIR/app.log --pidfile $OPENSHIFT_TMP_DIR/uwsgi.pid

Edit .openshift/action_hooks/stop to the following:

kill `cat $OPENSHIFT_TMP_DIR/uwsgi.pid`

Deploy

Commit all of the changes to the repository:

git add .

git commit -am "initial commit"

Deploy to OpenShift:

git push

You should now be able to browse and see your application running:

curl http://py27-[namespace].rhcloud.com

Should return:

Hello from Flask

And to verify that your application is using Python 2.7:

curl http://py27-[namespace].rhcloud.com/info

Should return:

2.7.3
Tags:

I like where your headed..... Since you have used OpenShift for months...

How would make my app/ acct act like a private PasS?

i notice a not ideal case, when a user copy the hook script, there is one line cmd, but not correct formated.also there is not have line number, as normal user, we can't identify the cmd's right purpose. please correct it.thanks.

The formatting is not proper, anyone who is new in pythone can easily get confuse with such a editor. There is no proper user friendliness, so difficult to understand.

I followed exactly and it is showing me following error

 

 

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Apache/2.2.15 (Red Hat) Server at APPNAME-USERNAME.rhcloud.com Port 80

Hi @surya;

We went ahead and modified the start hook in the example above cause it was trying to bind to 127.0.0.1 (localhost) rather than $OPENSHIFT_INTERNAL_IP.

Try it again and let us know. If you do run into the 503 error, please be sure post a snippet of the errors in the log as well.

Thanks;
~Nam

Updated a bit for the application.py and start hook.

It should be working well now.

The python/setuptools/pip/uWSGI installation process can be scripted in a pre-build hook.  Just update these 2 hooks in your git repo and git commit/push it across:
1) .openshift/action_hooks/pre-build
2) .opensnift/action_hooks/stop

--- stop ---
#!/bin/bash
#  The logic to stop your application should be put in this script.
if [  -f $OPENSHIFT_TMP_DIR/uwsgi.pid ]; then
kill `cat $OPENSHIFT_TMP_DIR/uwsgi.pid`
fi
------------

--- pre_build ---
#!/bin/bash
# This is a simple script and will be executed on your CI system if
# available.  Otherwise it will execute while your application is stopped
# before the build step.  This script gets executed directly, so it
# could be python, php, ruby, etc.
if [ ! -d $OPENSHIFT_DATA_DIR/bin ]; then
# install python 2.7.3 on builder gear
cd $OPENSHIFT_TMP_DIR
wget http://python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2
tar jxf Python-2.7.3.tar.bz2
cd Python-2.7.3
./configure --prefix=$OPENSHIFT_DATA_DIR
make install

export PATH=$OPENSHIFT_DATA_DIR/bin:$PATH

# install setuptools and pip
cd $OPENSHIFT_TMP_DIR
wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.ta...
tar zxf setuptools-0.6c11.tar.gz
cd setuptools-0.6c11
python setup.py install
cd $OPENSHIFT_TMP_DIR
wget http://pypi.python.org/packages/source/p/pip/pip-1.1.tar.gz
tar zxf pip-1.1.tar.gz
cd pip-1.1
python setup.py install

# install uWSGI
cd $OPENSHIFT_TMP_DIR
pip install uwsgi

# cleanup
cd $OPENSHIFT_TMP_DIR
rm -rf Python-2.7.3.tar.bz2
rm -rf setuptools-0.6c11.tar.gz
rm -rf pip-1.1.tar.gz
fi
------------

The benefits of using the pre-build hook logic is that it enables Jenkins integration OOTB.  Special thanks to @surya and @tjr9898 for their contributions.

@NamDuong: your scripts are very useful, thanks. Just one fix: pre_build is missing a fi at the end (in fact it throws an unexpected end of file error).

Hi

I have tried the exact steps , however, I got this message when I reached the step: make install :

/usr/bin/install: cannot create regular file `/bin/python2.7': Permission denied

Any help would be appreciated

Sorry for the inconvenience. Can you try again? This blog needed to refresh from using $OPENSHIFT_RUNTIME_DIR to $OPENSHIFT_DATA_DIR as the runtime dir env variable is no longer in use.

Thanks;
~Nam

After Openshift upgrade ,it doesn't work.

Can you let us know which part doesn't work? E.g., what command you're trying and what the error message is? If it's the $OPENSHIFT_RUNTIME_DIR related, just set it to $OPENSHIFT_HOMEDIR/diy-0.1/runtime/

If you're trying the install again, I've updated to use $OPENSHIFT_DATA_DIR instead.

HTHs;
~Nam

thanks!It works now after installed again.

Thanks for the confirmation. Sorry for the inconvenience.

Updates to this blog 11/12/2012:
Updated all references of $OPENSHIFT_RUNTIME_DIR to $OPENSHIFT_DATA_DIR.
Updated 1 reference of $OPENSHIFT_LOG_DIR in the start hook to use $OPENSHIFT_DIY_LOG_DIR

This is due to the typeless gears changes made in relationship to changes in {application name} directory to {cartridge ID} directory. Logs and runtime directories are now tied to {cartridge ID}.

@Nam Duong,
I've noticed that $OPENSHIFT_TMP_DIR's value is /tmp/, so when you concat it with /uwsgi.pid in the .openshift/action_hooks/stop script you'll get /tmp//uwsgi.pid (which is not correct, right?). Wouldn't it be better that environment variables that represent directories didn't have that trailing slash? Why is it like that?

Also, after the aforementioned update my diy app wasn't working correctly anymore. It was throwing a 503 error. I tried updating the scripts (replacing OPENSHIFT_RUNTIME_DIR with OPENSHIFT_DATA_DIR): no luck, the same 503 error (a Service Temporarily Unavailable message was showing). I deleted the application and reinstalled the whole thing but, again, with the same results and no success.

I'm trying again, in case I missed something in the last attempt.

Thanks in advance, N.

All $OPENSHIFT_*_DIR env variables end with a trailing / to denote a directory following convention. The stop hook should work just fine. If precision is needed, you can concatenate the 2 strings instead (in python where paths get finicky).

With the 503, you can check your logs to see what errors are thrown. That should give hints to what's failing. Some errors will not get triggered on your local apache, in which case, you can ssh onto the gear and run curl -vvv http://$OPENSHIFT_INTERNAL_IP:$OPENSHIFT_INTERNAL_PORT

HTHs;
~Nam

That explains a lot.

By the way I solved all the issues (although I'm not sure about the original reasons of the 503 error). The last problem I had was a uWsgi one. Basically I copy-pasted the application.py source from this topic's OP, but didn't notice some indentation errors; after fixing them the uWsgi error was still present.

Thus I tried rhc app stop -a <myapp>: the output was SUCCESS, but the app wasn't really stopped. Anway using rhc app force-stop -a <myapp> and then rhc app start -a <myapp> finally made the app running fine.

Thanks for your time and your help!

Nice workaround! Keep the feedback flowing!!
Thanks;
~Nam

I am following the blog and I think I have set it all up correctly. Status shows started but I am getting 503.

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Apache/2.2.15 (Red Hat) Server at dupoudit-innovate.rhcloud.com Port 443

Check out this FAQ. Hope it helps in tracking down the root cause.

Thanks;
~Nam

I have seen the logs and the request is still handled by webrick. As I have deleted the ruby file, its not working. So, why would the action_hook/start not kick in and let wsgi handle the request?

when I tried to restart the app, it gives me

Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support. Reference ID: eea601c117a34a35bfc2b76ea967fdd7

Node execution failure could mean many different things. Can you open a new thread and post the output to "rhc app restart -a {appName} -d"?
-d will print out the debug details. Usually, it means there's a possible resource allocation issue (threads, memory, etc) which is code and/or environment related, but we'll need those details for better guidance. You can also force-stop your app as a temporary workaround so that you can start it up again, but it might eventually get back into this bad state.

@Nam

The stop hook script you provided has one tiny issue:

#!/bin/bash
#  The logic to stop your application should be put in this script.
if [  -f $OPENSHIFT_TMP_DIR/uswgi.pid ]; then
kill `cat $OPENSHIFT_TMP_DIR/uwsgi.pid`
fi

In the if-statement it should be  ../uWsgi.pid not  ../uSwgi.pid

Regards,

Nice typo catch!! Thanks! I've updated the post.

No worrries. Glad to offer any small piece of help to this service, I hope some access always remains free. One additional note while I am here for anyone wanting to try this:

If one uses the pre-build script Nam provided OR at the command line installs uwsgi using  pip install uwsgi be sure to change the requirements.txt so that it reads:

uwsgi=>1.3.2

As I think what happens is that pip installs the latest stable release registered in the python package directory and that version is something like 1.9.x. It will hang reading the requirements.txt because it will look for only 1.3.2 and not be able to find it or install it, at least I believe that was my issue initially.

Also, when deploying to the remote openshift server doesn't the application.run() line in the application.py module attempt to bind to localhost, is it necessary as the flask documentation suggests to change that line to read at a minimum to application.run(host='0.0.0.0')?

Also I had experienced some issues related to the uwsgi process not getting killed on "ctl_app stop" and then upon "app restart" the subsequent "git push" the app would fail to stop as the pid in the tmp file uwsgi.pid doesn't match the currently running process pid. Sometimes this prevented commit updates to the git repository. If anyone else experienced that issue, how did you resolve it? My usual strategy was to ssh into the gear, kill the pid, and repush my commits.

Hello,

$OPENSHIFT_DATA_DIR/bin/uwsgi -s $OPENSHIFT_INTERNAL_IP:$OPENSHIFT_INTERNAL_PORT --socket-protocol http --pp $OPENSHIFT_REPO_DIR/diy --module application -d $OPENSHIFT_DIY_LOG_DIR/app.log --pidfile $OPENSHIFT_TMP_DIR/uwsgi.pid

The trouble is the last line: "bind(): Permission denied [core/socket.c line 755]". How can I set the permissions required by uwsgi?

DEBUG: Connecting to https://openshift.redhat.com/broker/rest/api

DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains

DEBUG: code 200 702 ms

DEBUG: Getting all domains

DEBUG: Request GET https://openshift.redhat.com/broker/rest/domains/mynamespace/applications/myapp

DEBUG: code 200 657 ms

DEBUG: Restarting application myapplication

DEBUG: Request POST https://openshift.redhat.com/broker/rest/domains/mynamespace/applications/myapp/events

DEBUG: code 500 1723 ms

Unable to complete the requested operation due to: Failed to correctly execute

all parallel operations - --DEBUG--

Failed to execute: 'control restart' for

/var/lib/openshift/51d9ab1250044606d600002c/diy

Restarting DIY cart

Stopping DIY cart

Starting DIY cart

bind(): Permission denied [core/socket.c line 755]

Looks like this blog wasn't updated to the latest v2 cartridge changes. Note the environment variable changes:

$OPENSHIFT_DATA_DIR/bin/uwsgi -s $OPENSHIFT_DIY_IP:$OPENSHIFT_DIY_PORT --socket-protocol http --pp $OPENSHIFT_REPO_DIR/diy --module application -d $OPENSHIFT_DIY_LOG_DIR/app.log --pidfile $OPENSHIFT_TMP_DIR/uwsgi.pid

as per https://www.openshift.com/page/openshift-environment-variables

I've made the changes to the blog as well.

It was very kind of you. Thank you.

Hey @ndoung, Could you please let me know how can I enable html files on my server to be downloaded? I did all the steps, the python is working fine, however the php/html files are not reachable from my browser...

More from this author