Cross compiling Python for Android

Sep
2011
05

Android, Programming, Python

24 comments

So, one day you wake up and decide that you don’t have enough problems in your life and that you’d like to have more. That’s when you go with the obvious choice: You’ll try to cross compile the Python runtime for use in Android.

These instructions and the required patch apply to Python v2.7.2 (at least that’s the version I know works). The patch was adapted by me, and it’s based on the Py4A patch. I also got ideas and guidance from the Pygame For Android project, specially their build scripts, which if you are interested in this kind of self inflicted pain I suggest you have a look at them. All testing and development was done on Ubuntu Natty 64 bits version.

The first thing we have to do is create a host version of the Python runtime, as it is required to cross compile Python. This is easily done by extracting the Python source, and running the usual configure/make/make install.

Once you have that, after extracting the Python source to some other place and applying the patch (patch -p0 python-2.7.2.android.diff), you need to set up the Android NDK, and set up some environment variables (valid at least for Android NDK R6):

export ANDROID_NDK=[PATH WHERE THE ANDROID NDK IS]
export PATH="$ANDROID_NDK/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/:$ANDROID_NDK:$ANDROID_NDK/tools:/usr/local/bin:/usr/bin:/bin"
export ARCH="armeabi"
export CFLAGS="-DANDROID -mandroid -fomit-frame-pointer --sysroot $ANDROID_NDK/platforms/android-5/arch-arm"
export CXXFLAGS = "$CFLAGS"
export CC="arm-linux-androideabi-gcc $CFLAGS"
export CXX="arm-linux-androideabi-g++ $CXXFLAGS"
export AR="arm-linux-androideabi-ar"
export RANLIB="arm-linux-androideabi-ranlib"
export STRIP="arm-linux-androideabi-strip --strip-unneeded"
export MAKE="make -j4 HOSTPYTHON=[PATH TO HOST PYTHON] HOSTPGEN=[PATH TO HOST PGEN] CROSS_COMPILE=arm-eabi- CROSS_COMPILE_TARGET=yes"

In the final variable “MAKE” up above you have to complete the appropriate full path to the python executable and pgen executable that you generated when you compiled Python for the host (I don’t go into the details of compiling Python for the host as it is rather simple and there’s plenty of information on the net about this, as I mentioned it shouldn’t be harder than doing configure/make/make install).

I configured the cross compilation with:

./configure LDFLAGS="-Wl,--allow-shlib-undefined" CFLAGS="-mandroid -fomit-frame-pointer --sysroot $ANDROID_NDK/platforms/android-5/arch-arm" HOSTPYTHON=[HOST PYTHON PATH] HOSTPGEN=[HOST PGEN PATH] --host=arm-eabi --build=i686-pc-linux-gnu --enable-shared --prefix="[WHERE YOU WANT TO PUT THE GENERATED PYTHON STUFF]"

After this I had to make a small correction to the generated Makefile.

sed -i "s|^INSTSONAME=\(.*.so\).*|INSTSONAME=\\1|g" Makefile

Now you are ready to compile:

$MAKE

With a little bit of luck that should be it, I promised pain but it didn’t seem like much trouble, right? Well, I just gave you my hard earned patch that makes the trick!

Actually using the compiled library is a different matter entirely, but to give you a hint of where to go I suggest that you take a look at my previous article on how to embed and freeze modules and packages in Python, which is what I did. Once you do that, maybe mix it with a bit of SDL (of special interest is the Android project skeleton that they use), and you’ll have a fully working Python environment that you can build apps on!

Stay tuned for instructions on how to accomplish something similar (but with a WAY bigger patch) for cross compiling to Windows via Mingw32

UPDATE: Thanks to Anthony Prieur who let me know of a couple of typos in the instructions (already fixed) and that the patch has an indentation issue in setup.py, which is trivial to fix if you need the file (I don’t have any use for that file in the application I’m developing).
UPDATE2: Patch now supports cross compiling from OS X in addition to Linux.

  • http://twitter.com/olivierbrousse Olivier Brousse

    Thank for this trick!!!
    It gave me a nice python binary and the associated lib. I have a question regarding your sed command that is not working as it is put here in the tutorial, what is the purpose of this command?I also have a question on how can I check that the binary is actually working?Then can you give links to articles you are talking about as to “take a look”

    Best regards,
    Olivier

  • http://ignifuga.org/ Gabriel Jacobo

    The sed command changes the generated library from libpython2.7.so.1 to libpython2.7.so which is what I needed for it to be accepted when I use this library with the NDK builder in a later stage in my project. It’s not mandatory, YMMV.
    I don’t use the python binary directly, as I just mentioned I use the library to link against other files using the Android NDK, but if you want to test it I suppose you can copy it to the device or simulator using ADB and try running it in a shell.

  • phani burra

    Hi Gabriel,

    I have tried the above steps and ended up in the following error.

    ./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c
    ./Grammar/Grammar: line 18: single_input:: command not found
    ./Grammar/Grammar: line 18: simple_stmt: command not found
    ./Grammar/Grammar: line 18: compound_stmt: command not found
    ./Grammar/Grammar: line 19: syntax error near unexpected token `NEWLINE’
    ./Grammar/Grammar: line 19: `file_input: (NEWLINE | stmt)* ENDMARKER’
    make: *** [Parser/pgen.stamp] Error 2Also the patch has changes to configure.in and Makefile.in. So should we run autoreconf before Make. And if we do that the patch changes to configure will be erased…. Please give your piece of advice.ThanksPhani

    • http://ignifuga.org/ Gabriel Jacobo

      You don’t need to run autoreconf, just follow the instructions as provided. I don’t know about the Grammar errors you are getting, they didn’t happen to me, you should start fresh following the instructions exactly and if it still happens let me know.

      • phani burra

        The one major difference is I am crosscompiling for an x86 Android platform.

        Followed the exact steps and ended up below:

        i686-android-linux-gcc -DANDROID -mandroid -fomit-frame-pointer –sysroot /home/phani/Downloads/ndk/platforms/android-9/arch-x86 -c -fno-strict-aliasing -mandroid -fomit-frame-pointer –sysroot /home/phani/Downloads/ndk/platforms/android-9/arch-x86 -DNDEBUG -g  -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include  -fPIC -DPy_BUILD_CORE -o Python/asdl.o Python/asdl.c

        ./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c
        make: execvp: ./Grammar/Grammar: Permission denied
        make: *** [Parser/pgen.stamp] Error 127

        so did below steps:

        chmod 777 ./Grammar/Grammar
        make clean
        make 

        Ended up with errors:

        i686-android-linux-gcc -DANDROID -mandroid -fomit-frame-pointer –sysroot /home/phani/Downloads/ndk/platforms/android-9/arch-x86 -c -fno-strict-aliasing -mandroid -fomit-frame-pointer –sysroot /home/phani/Downloads/ndk/platforms/android-9/arch-x86 -DNDEBUG -g  -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include  -fPIC -DPy_BUILD_CORE -o Python/asdl.o Python/asdl.c
        ./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c
        ./Grammar/Grammar: 18: single_input:: not found
        ./Grammar/Grammar: 18: simple_stmt: not found
        ./Grammar/Grammar: 18: compound_stmt: not found
        ./Grammar/Grammar: 19: Syntax error: word unexpected (expecting “)”)
        make: *** [Parser/pgen.stamp] Error 2

        • phani burra

          Ok I tried same exact steps for  arm as mentioned above, i am still facing the above issue. Looks like there are some update to python 2.7.2 due to which the cross compile is broken now.

          • phani burra

            Hi Gabriel,

            My bad. I was not setting the environment variables correctly earlier. When I did a clean procedure again, I ended up in below error.
            i686-android-linux-gcc -DANDROID -mandroid -fomit-frame-pointer –sysroot /home/phani/Downloads/ndk/platforms/android-9/arch-x86 -Wl,–allow-shlib-undefined -Xlinker -export-dynamic -o python                         Modules/python.o                         libpython2.7.a -ldl    -lm  File “./setup.py”, line 316    self.announce(‘*** WARNING: renaming “%s” since importing it failed: %s’ % (ext.name, why), level=3)       ^IndentationError: expected an indented blockmake: *** [sharedmods] Error 1

            I am not a python guy. And I have seen your patch. the above error comes as part of your modifications. Let me know if you have any advice.

            Thanks
            phani

          • http://ignifuga.org/ Gabriel Jacobo

            At that point you’ve successfully cross compiled Python, I didn’t bother fixing setup.py as I didn’t have any use for it, but I believe (from what I recall) that it was an identation problem easily fixed.

  • rajashekhar v

    hi, 

    Thanks for the this article. I followed as suggested in this article to cross compile python2.7.2 for Android. 
    My Build Environment
    ————————————–
    My target board is TI’s BLAZE — board.
    Android OS is “Froyo”.
    My host machine is Ubuntu 10.04 [64 bit]I built hostpython and followed as per the article. Went on successfully.I did the below configuration :./configure LDFLAGS=”-Wl,–allow-shlib-undefined” CFLAGS=”-mandroid -fomit-frame-pointer –sysroot $ANDROID_NDK/platforms/android-5/arch-arm” HOSTPYTHON=/home/mverma/newpython1/Python-2.7.2/python HOSTPGEN=/home/mverma/newpython1/Python-2.7.2/Parser/pgen –host=arm-eabi –build=i686-pc-linux-gnu –enable-shared –prefix=/home/mverma/androidpythonIt went on  successfully.When I did the $MAKE, I’m getting below error message:========================================================checking for suffix of executables… 
    checking whether we are cross compiling… configure: error: in `/home/mverma/newpython/Python-2.7.2/build/temp.linux-x86_64-2.7/libffi’:
    configure: error: cannot run C compiled programs.
    If you meant to cross compile, use `–host’.
    See `config.log’ for more details.
    Failed to configure _ctypes module

    Python build finished, but the necessary bits to build these modules were not found:
    _bsddb             _sqlite3           _ssl            
    _tkinter           bsddb185           bz2             
    dbm                dl                 gdbm            
    imageop            readline           sunaudiodev     
    To find the necessary bits, look in setup.py in detect_modules() for the module’s name.

    Failed to build these modules:
    _curses            _curses_panel      _locale         
    _multiprocessing   crypt              nis             
    ossaudiodev        spwd                               ==============================================================While configuring, I provided the path where the built python should come., but nothing is there in that directory !!! [--prefix=/home/mverma/androidpython]As log as said it has built python….and as there no folders in the  –prefix folder, Manually, 1. What all directories I need to copy to target ?2. What environment variables I need to set in the target ?3. Do I need to copy any shared libraries, please let me know.4. Any fix or trick how to fix the above problem?Please help me……Regards,Rajashekhar

    • http://ignifuga.org/ Gabriel Jacobo

      Did you run “make install” after make? The file you need is a libpython2.7.a or similar, you can link that against your files, but as it is mentioned in the article, this is not trivial. I haven’t tested it, but some people have mentioned to me that you can also use the python interpreter as is, copying the entire install dir (Python module library included of course) over to the Android device. If you want an idea of how to compile a fully static Python app that runs on Android, you can take a look at the source of my game engine (ignifuga.org), specifically the Schafer utility, which takes what’s mentioned in this article and does something useful with it.

      • rajashekhar v

        hi Gabriel, 

        Thanks for the reply.. I did the make install to get the host python and host pgen.
        But I didn’t do the make install after $MAKE. Do I need to do ??
        If I need to do make install after the $MAKE do I need to send any parameters or arguments for it…if so please let me know.

        Also you have informed to copy the entire install dir — to the target.
        Is this is the install dir you are mentioning [--prefix="[WHERE YOU WANT TO PUT THE GENERATED PYTHON STUFF]“]. In my case the install dir [/home/mverma/androidpython] But, in this install dir there is nothing….

        Please let me know…..

        Regards,
        Rajashekhar

        • rajashekhar v

          Hi Gabriel, 

          I did the $Make Install….then I got the generated python stuff in my install directory …………..that’s cool. I copied the entire generated content into my target board and configured the environment variables “PYTHONHOME” & “PYTHONPATH”

          Also I copied the generated libs libpython2.7.a. libpython2.7.so.1.0 and libpython2.7.so  to the target’s system/lib. 

          Now python is working on my target board :) In the sense I can use python executable for doing testing on my target board….

          I have one question, i’m not getting the python prompt on my target device,,,, any idea /trick how can I get this too any my target …

          Really thankful for the article, it was very very usefull..

          Once again thanks,,,,

          Regards,
          Rajashekhar

          • rajashekhar v

            hi Gabriel, 

            Please help me out…
            I cross compiled the python 2.7 for android as per this article.

            The problem is I’m unable to import the standard modules likes random, time in my  python program. 
            I have set the the PYTHONHOME & PYTHONPATH environment variables as below..

            PYTHONHOME = /system/lib/python2.7
            PYTHONPATH =  /system/lib/python2.7

            On my target if I give python -V, I’m getting the version as Python2.7.2. But if I have program like this, I’m unable to proceed….Please please help me 

            =====================================
            import os,randomdef pick_test():    valid = True    testsexecuted = open(‘/pythonforandroid/bin/testsexecuted’,'a’)    testtoexecute = open(‘/pythonforandroid/bin/testtoexecute’,'w’)    rand_num = random.randint(1,22)    filetoread = open(‘/pythonforandroid/bin/OneScript’,'r’)    for line in filetoread.readlines():        content = str(line)        content = content.rstrip()        #print content        if (valid):            if (content == str(rand_num)):                #contenttoput = contenttoput + content                valid = False                continue            else:                continue        else :            #contenttoput = contenttoput + content            #print contenttoput            if (content != “end” ):                testtoexecute.write(content + ‘n’)                #print contenttoput                continue            else :                break    filetoread.close()    testtoexecute.close()    f = open(‘/pythonforandroid/bin/testtoexecute’,'r’).read()    testsexecuted.write(f)    testsexecuted.close()    #print contenttoputdef maintest():    num_test = 0    while(num_test <= 5):        pick_test()        num_test = num_test + 1  maintest()
            ===================================== 

            Regards,
            Rajashekhar

          • http://ignifuga.org/ Gabriel Jacobo

            If you pay close attention at the compilation process some modules (most of those that require some C counterpart), fail when statically compiled. That’s probably the problem you are facing, but I don’t have a general solution, you’ll have to see if random has a C counterpart (like _random), and then why random isn’t compiling. I don’t know about the interactive console either, someone mentioned to me there was something else that required patching to get it working, but I don’t recall what it was.

  • Pingback: Run Python apps, Scripts on Android - Phone Fair

  • http://www.overvprojects.nl Alexander Overvoorde

    I tried this, but the final compilation step with $MAKE failed with undefined reference errors for pgen:

    /home/overv/code/android/Python-2.7.2-android/Parser/pgen.c:52: undefined reference to `PyObject_Realloc’
    /home/overv/code/android/Python-2.7.2-android/Parser/pgen.c:55: undefined reference to `Py_FatalError’

    How can I solve this?

    • http://www.overvprojects.nl Alexander Overvoorde

      Never mind, wasn’t using the host pgen.

  • http://www.facebook.com/abus00 Adi Busu

    There is a slight issue: when you call “./configure”, you overwrite the CFLAGS so that -DANDROID is lost :) .. other than that, very good job, congrats!

  • Robert Zero

    Hi, Gabriel!

    Thanks for the post.

    I tried to run your script but the patch file was not found on the link you provided inside the script.

    Even whether I change the link to a valid one (python.android.diff) it start applying the patch but takes forever and never ends.

    Skipping the patching, at the end almost finishing (without the patch) it complains about C compiler not being able to create the executables files and finally, to the final blow, it says something ’bout not finding the install file.

    Any ideas?

    Thanks in advance

    • http://ignifuga.org/ Gabriel Jacobo

      Applying the patch should be a very quick operation, I don’t know which system you use to develop, but if you have the right patch utility, downloaded the right Python source code (2.7.2), it should work pretty much as advertised.
      As to what you can do with Python on Android, you can do something like ignifuga.org (it won’t be easy though). As is, the patch only lets you run Python on Android, how you copy it there and how you glue it to the display and input devices is a different subject.

      • Robert Zero

        Hi!

        Thanks for the answer.

        I’m not that deep in python! hahaha

        I’m mostly willing to learn it using my GT-P3110 while I’m on the road (yes, I travel a lote. :( ).

        I only want to use the basics of python, like: read [terminal] user input (as in “input” or “raw_input”).

        Deal with hashs (dictionaries) and so forth.

        I did use the source ball of the 2.7.2 version, it even create the folder “android…python” (don’t recall very well)

        I could even run ./CONFIGURE, MAKE in there.

        I moved the folder to my Android and when I tried to run the python executable, it said it had no permission ‘tho it had full access.

        • Robert Zero

          Can you please help me with that? I really need it up and running on my Android. :-)

  • Robert Zero

    Hi, Danilo!

    Thanks for the post.

    When I try to apply the patch (python.android.diff) it starts applying but takes forever and never ends.

    Skipping the patching, at the end almost finishing (without the patch) it complains about C compiler not being able to create the executables files and finally, to the final blow, it says something ’bout not finding the install file.

    Any ideas?

    Thanks in advance!

    Another question: Doing the procedure above, will I be able to use Python more-or-less like in my PC/MAC?

    Like, I recently used “QPython” but it has some issues: It doesn’t allow input (as of raw_input() or input()) plus the console provided by it has quite a few customizing options. It doesn’t even let you alter the font size or color.

    Will I be able to import common libraries such as input, print, etc..?

    My goal is to use Python in my Samsung Galaxy Tab 2 7.0 for learning purposes.

    I only want to shorten the pain of this process if it’s not going to help me in any way.

    Ty

  • Phartat Misassa

    I’m sorry but I am a total noob, I can’t figure out how to apply the patch.

    I keep getting this when I try doing this “patch < ../python.android.diff"

    "can't find file to patch at input line 4
    Perhaps you used the wrong -p or –strip option?
    The text leading up to this was:
    ————————–
    |diff -r 38897e3d21f0 Lib/ctypes/__init__.py
    |— Lib/ctypes/__init__.py Tue Feb 19 13:31:31 2013 -0800
    |+++ Lib/ctypes/__init__.py Tue Feb 19 16:25:30 2013 -0800
    ————————–
    File to patch:"

    I also want to put myself through the pain of cross compiling other
    interpreters, even if I have to make some additions to my bionic/libc
    that I would "cherry pick" from a libc for armhf.

    What is the correct way to apply this patch?

    nevermind: "other place and applying the patch (patch -p0 python-2.7.2.android.diff), you"