Duck-Typing (or Duct Taping) Dynamic Objects in C#

I've been delving into DLR and dynamic types as of late; in particular, how to enforce a static "contract" (an interface per se) on a dynamic type. During my perusal of blogs and articles, I came across an article by Tobi that delved into what I was trying to do, but not in a generic fashion. The interesting bit of it is that Tobi referenced another open source project called the Castle Project. This project, among other things, has some utility methods that allow developers to create class proxies and implement interfaces at runtime.The proxied objects could then have their method and property calls specially handled by an IInterceptor implementation. So, after reviewing of the code and a little bit of hacking, I came up with runtime duck typing solution:

Python Code:

import clr 

class Dragon: 
    def Quack(self): 
        print "ROAR!" 

dragon = Dragon()

C# Code:

public interface Duck
{
    void Quack();
}

public class Cow
{
    public void Quack()
    {
        Console.WriteLine("Moo?");
    }
}

class Program
{
    static void Main(string[] args)
    {
        ScriptEngine pythonEngine = Python.CreateEngine();
        ScriptRuntime runtime = pythonEngine.Runtime;
        ScriptScope scope = runtime.ExecuteFile("hello.py");
        IDynamicObject dynamicObject = (IDynamicObject)scope.GetVariable("dragon");

        // I can't figure out how to do this without the runtime? 
        // Using GetMetaObject and building an expression tree?
        Duck dragon = DuctTapeHelper.DuctTape<Duck>(dynamicObject, runtime);
        Duck cow = DuctTapeHelper.DuctTape<Duck>(new Cow());

        // Duck Type!
        dragon.Quack();
        cow.Quack();
    }
}

Notice that neither "Cow" or "Dragon" implement the "Duck" interface, but at runtime, an interface proxy is generated to allow them to become Ducks!

Incidentally, another potential way to do this, without the dependency on the Castle Project utilities, is to use reflection to generate a IronRuby/IronPython script that implements an interface proxy.

Here's the full source code to my version of the DuctTape project. I am packaging the various projects from DLR and Castle into it as well. The code is nowhere near complete; it is just a functional prototype to make sure the concepts actually work! I feel that it could be done better through expression trees, but I couldn't figure out how to use an Expression tree to call into a dynamic method on IDynamicObject. Please let me know if you have more insight into this, as I am still a DLR noob.

 

Completely Unrelated:

Does anyone know of a good way to embed code into a blog? As you can see, my line breaks become all messed up with whatever tool I use (I use Live Writer with Paste from Visual Studio or Source Code Snippet).

Solved:

Blogger has some silly "Convert line breaks" option that is defaulted to Yes. As a result, it completely mangles all line breaks if you have a <pre> tag in your post body. Turn that off, and all is well.

Microsoft's DLR and Mono bring Python and Ruby to Android

With my recent work on porting Mono to Android, I began needing a way to create Java classes and invoke their various methods from C#. I had read briefly about the Dynamic Language Runtime and upcoming C# support for dynamic types; so that naturally led to me considering implementing a JNI-P/Invoke interop layer that allows access to Java/Dalvik code through C# and vice versa. However, I wasn't sure if the DLR would even run on Mono, but I was pleasantly surprised. Support for the DLR was added to Mono around a year and a half ago!

After copying the necessary DLLs over to the phone, I was able to run the simple ipy.exe console application that comes with the DLR source code. The neat thing I discovered while perusing through the documentation is that IronRuby and IronPython are actually compiled into CIL byte code and thus eventually native code! Anyhow, as you can see, IronPython is working dandily on the phone. [1]

The Dynamic Language Runtime is probably the coolest new toy in the .NET arsenal since lambda expressions. Couple the DLR with the new dynamic keyword in C# 4.0, and you can invoke dynamic languages from C# as easily as you can access .NET classes from IronPython or IronRuby![0]

Now to get back to implementing Java interop...

 

[0] The dynamic keyword basically provides compile time parsing of an expression tree, but the expression isn't actually validated until runtime.

[1] Though Mono + DLR is clocking in at a hefty 13mb in files. But, applications will soon be installable onto the SD card so the storage constraints will be a moot point.

JNI in Android (and a foreword of why JNI Sucks)

Quick Preface: You do NOT need root to create and use a shared native library (.so) in Android from your Java application.

I had never tried Java Native Interface until today. And I must say, it completely sucks. Not only is the header generation/implementation nontrivial and tedious, JNI is inherently broken in that it is not architecture/platform nonspecific. So I proudly present to you my list of why JNI sucks.

• You can't access arbitrary functions in arbitrary DLLs. JNI requires that you write a glue C/C++ layer to do whatever you want to do natively.
• On Android, your glue layer must contain a JNI_OnLoad function that explicitly register every method that needs to be made available to the JVM. The JVM can't simply check the export tables and intelligently import it.
• Since developers must write a glue layer, you must also compile and package that glue layer per platform you are targeting. This design is absurd because platforms can have different CPU architectures and different Operating Systems, and still support similar calls in the similar libraries. For example:
  • OpenGL ES (libGLES_CM). On Windows Mobile it is called libGLES_CM.dll. On Linux/Android it is called libGLES_CM.so. These libraries have the same "undecorated" library name, and also contain the same functions with the same signatures. Unfortunately due to how how terrible an implementation JNI is, the same Java application would not work on every platform.
  • Standard System Libraries on Linux or Windows (libc, libdl, libm, ole32, kernel32, etc) - Once again, need to compile libraries per architecture (ARM, x86, x64, MIPS, etc) to use the common/base functions that would be available on the platform (GetWindowEx, statfs, etc).
• Write once, run anywhere. (Given that you are willing to do multiple cross compiles and embed multiple native binaries into a single package and unpack the appropriate one at runtime.)

For amusement's sake, let's look at what a C# PInvoke looks like versus JNI on Android. Suppose we want to clear the screen in OpenGL:

C#:

Just declare the method signature and the DLL. Simple! Call that like any other method in C#. (Note that the managed executable generated by this code works on both Android or Windows Mobile. Yes, I tested it.)

[DllImportAttribute("libGLES_CM")]
public static extern void glClearColor(float red, float green, float blue, float alpha);

JNI? Let's start out by declaring it.

public static native void glClearColor(float red, float green, float blue, float alpha);

Not done yet! Now the fun begins. Drop to a command prompt and go to the bin directory of your project: javah com.koushikdutta.JNITest.JNITestAcitivity. Now we have a generated header file, com_koushikdutta_jnitest_JNITestActivity.h, that looks like this:

/* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class com_koushikdutta_jnitest_JNITestActivity */

#ifndef _Included_com_koushikdutta_jnitest_JNITestActivity
#define _Included_com_koushikdutta_jnitest_JNITestActivity
#ifdef __cplusplus
extern "C" {
#endif
/*
 * Class:     com_koushikdutta_jnitest_JNITestActivity
 * Method:    glClearColor
 * Signature: (FFFF)V
 */
JNIEXPORT void JNICALL Java_com_koushikdutta_jnitest_JNITestActivity_glClearColor
  (JNIEnv *, jclass, jfloat, jfloat, jfloat, jfloat);

#ifdef __cplusplus
}
#endif
#endif

Ok, now let's implement the C code:

#include <GLES/gl.h>

JNIEXPORT void JNICALL Java_com_koushikdutta_jnitest_JNITestActivity_glClearColor
  (JNIEnv * env, jclass clazz, jfloat r, jfloat g, jfloat b, jfloat a)
{
    glClearColor(r, g, b, a);
}

You probably thought you were done, didn't you? Sun thinks otherwise. You need to register the function in C code as well. Don't screw up when copy pasting the cryptic method signature strings!

static JNINativeMethod sMethods[] = {
     /* name, signature, funcPtr */

    {"glClearColor", "(FFFF)V", (void*)Java_com_koushikdutta_jnitest_JNITestActivity_glClearColor},
};

extern "C" jint JNI_OnLoad(JavaVM* vm, void* reserved)
{
    JNIEnv* env = NULL;
    jint result = -1;

    if (vm->GetEnv((void**) &env, JNI_VERSION_1_4) != JNI_OK) {
        return result;
    }

    jniRegisterNativeMethods(env, "com/koushikdutta/JNITest/JNITestActivity", sMethods, 1);
    return JNI_VERSION_1_4;
}

To be completely fair, jniRegisterNativeMethods is actually a convenience function that is found in libnativehelper.so on Android; I imported that library for the sake of... convenience. Its implementation is as follows (JNIHelp.c in the source):

int jniRegisterNativeMethods(JNIEnv* env, const char* className,
    const JNINativeMethod* gMethods, int numMethods)
{
    jclass clazz;

    LOGV("Registering %s natives\n", className);
    clazz = (*env)->FindClass(env, className);
    if (clazz == NULL) {
        LOGE("Native registration unable to find class '%s'\n", className);
        return -1;
    }
    if ((*env)->RegisterNatives(env, clazz, gMethods, numMethods) < 0) {
        LOGE("RegisterNatives failed for '%s'\n", className);
        return -1;
    }
    return 0;
}

You need to load the library in Java code in your class's static constructor.

System.load("/data/data/com.koushikdutta.JNITest/libjnitest.so");

On Android, you must use System.load and provide a full path instead of using System.loadLibrary. This is because loadLibrary only checks /system/lib; it does not search LD_LIBRARY_PATH or your current working directory. Unfortunately, applications can't write to the /system/lib directory (without root).

.NET's P/Invoke is so much easier/better that a third party company (God forbid Sun actually do something to better the Java language) implemented something quite similar for Java, named, *drumroll please*, J/Invoke. Too bad Android doesn't have that. That would be nice.

Also, on Android, you can't do a normal ARM/GCC cross compile to make shared libraries (I've discussed this in a previous blog post). The resultant .so files will not work with Android's linker due to different linker scripts. To create shared libraries for Android, I would highly recommend just creating a subproject in the Android Source code tree and building the tree. Alternatively you can use the handy script found on this page.

Also, remember that the shared library will need to be contained in your APK file and extracted to the file system so it can be accessed. For an example of how that can be done, check out the source code that does something very similar in the Android Mono port.

Incidentally, this learning experience came about as a result of figuring out how to run Mono side by side with Java/Dalvik in the same process.

Windows Mobile Unified Sensor API heads to CodePlex

At the behest of the development community and colleagues, the Windows Moble Unified Sensor API is now hosted on CodePlex. I will continue blogging about the changes on my website of course, but all the releases and source code must now be retrieved from CodePlex! (CodePlex is pretty awesome by the way... it has support for multiple source control bindings, and the web client repository browser is fantastic. (I use CodePlex for Windows Mobile stuff, and Google Code for Android. It makes sense in my head. [0])

In the following weeks, I intend to do the following:

• Reexamine the implementation (and possibly even the current interfaces) so it is easier for manufacturers to build devices that can leverage the dizzying array of sensor enabled software that already utilize the Sensor API.
• Clean up the HTC implementation.
• Add support for capacitive touch panel to the HTC implementation with the new reversed engineered headers provided by a reader of this blog.
• Create a few tutorials.
• Add a FAQ to the CodePlex wiki.

If anyone else is interested in contributing to the Sensor API project, please let me know!

 

[0] The decision on whether to put the Android Mono port on CodePlex or Google Code was a tough one. :)

Compiling Mono under the Android Build Evironment

I mentioned previously that I got Mono compiling under the Android build environment. I have also submitted all the changes I made to Mono back to their team. In addition, all external changes are hosted on Google code. The external changes were filling holes in Bionic, setting up a config.h that works on Android, and converting the Makefile to an Android.mk. Click here to go to the AndroidMono repository.

The build instructions are straightforward, and assumes you have a Release 1.0 repository (not Master or Cupcake) already set up:

• In a shell, go to your mydroid/external directory and type:
• svn co http://androidmono.googlecode.com/svn/trunk/ androidmono
• cd androidmono
• svn co svn://anonsvn.mono-project.com/source/trunk/mono mono
• Apply the androidmono.patch file to the mono directory make the necessary changes to the mono project. (This won't be necessary once they accept my patch)
• Run make in your mydroid directory.

Now you should have a mono and libmono.so! The build process does not create the managed assemblies. That needs to still be done the old fashioned configure/make way. The resultant mono binary will look for the managed assembles in /data/data/com.koushikdutta.mono/assets/lib/mono/2.0/(mscorlib.dll).

Mono and Bionic Together At Last

I finally got Mono building in the Android build environment (I.e., a project located at mydroid\external\mono). The great part about this is that I am now doing cross compiles to build the project, as opposed to an ARM native build on the emulator; those were painfully slow. Converting Makefiles to Android.mk files was pretty tedious, but that didn't compare to the significant differences between glibc and Bionic (Android's libc implementation). When Google said Bionic was "lightweight", they weren't joking. For example, Bionic does not implement div. Seriously? (However, the cupcake branch seems to have it now)

Regardless, I have Hello World working linked to Bionic now, which is quite exciting. I also tried out some PInvokes on libGLES_CM.so (OpenGL ES), and the initialization function eglGetDisplay was working just fine. So things are looking good, and this project should be able to really start taking off now.

Edit:

Mono on Bionic is around 50% faster... check my original Mono vs Dalvik post for updated performance comparisons. I am guessing the speed increase is due to Bionic being compiled for the Thumb instruction set as opposed to ARM. Or maybe because Google is using some CPU/architecture specific compiler optimizations in their build process.

Android's Linker

"Next task: create a JNI lib that can access Mono... should be fairly easy." Famous last words. I guess I didn't think through the problem of Android using a non-standard linker. Not only is it named and located differently (/system/bin/linker as opposed to ld-linux.so.3), the format of the dynamic libraries it loads is also different as well. So, Android's linker can not load Linux .so files, and vice versa.

This basically means that Mono with its current build configuration can not interact with the Android system at all, other than through RPC. And while it certainly is possible[0] to do that (and Android uses RPC extensively), I feel that would be a clumsy solution, as my goal is to get Mono running side by side with Java/Dalvik, and allow seamless interaction with runtime through usage of Dynamic Language Runtime (I'm not sure if Mono supports the "dynamic" keyword/type yet, but when it does... this will be quite amazing).

There are two possible solutions to this:

• Recompile Mono and the glibc dependencies using the linker script for Android.
• Compile Mono within the Android build environment so it uses Bionic instead of libc.

Using Crosstool, I tried to build a new toolchain tailored for Android's linker. I quickly found that whoever set up the toolchain/glibc build process must be a sadist. I wasn't too hesitant to give up on this approach anyway: this would result in two copies of libc being loaded (if that is even possible).

Although getting Mono fully integrated to work with Android's libraries will take considerably longer, I can rest knowing it is the proper solution.

[0] Set up an RPC server for OpenGL ES or skia? Ugh...

Mono for Android - First Update

I mentioned in my previous post that Mono was displaying assertions and exceptions on run when not being under the Debian environment. I have resolved these issues by tweaking a few variables on configure, and one by patching mono/metadata/security.c.

I am also now building Mono from the trunk (instead of 2.0.1) and using eglib instead of glib. Here's what I run to build Mono (still done on the emulator, under Debian):

export 'CFLAGS=-DARM_FPU_NONE -DPLATFORM_ANDROID -Wl,-rpath,.,-rpath,/data/data/com.koushikdutta.mono/assets/lib,--dynamic-linker,/data/data/com.koushikdutta.mono/assets/lib/ld-linux.so.3'

export CPPFLAGS=$CFLAGS

export 'LDFLAGS=-Wl,-rpath,.,-rpath,/data/data/com.koushikdutta.mono/assets/lib,--dynamic-linker,/data/data/com.koushikdutta.mono/assets/lib/ld-linux.so.3'

./configure --disable-mcs-build --with-tls=pthread --disable-shared-handles --with-static_mono=no --with-glib=embedded --prefix=/data/data/com.koushikdutta.mono/assets --exec-prefix=/data/data/com.koushikdutta.mono/assets

make

However, "make" fails while building docs. That is ignorable though. I actually comment out docs in my Makefile, as I haven't looked into how to disable it during configure.

With this setup, the base mono installation directory is /data/data/com.koushikdutta.mono/assets (instead of /usr/local). The binaries, libraries, and assemblies should be placed in their usual relative locations: ie, mono binary should be in bin, the libraries in lib, etc.

Now, if you were to run "./mono test.exe" from a terminal, you will see a lengthy assertion. This is because Mono is failing to find the user's home directory, because Android does not set that environment variable. And thus Mono doesn't have a valid location for the ~/.wapi/ directory. The ".wapi" directory is used by Mono for some transient files and communication (from what I understand). For example, if you were to run Mono under Terminal Emulator, HOME needs to be a location the Terminal emulator process and uid has full read/write permissions: /data/data/com.android.term:

export HOME=/data/data/com.android.term

./mono test.exe

I have updated the APK on the Market with my latest changes. I also have the APK available for direct download.

Next task: create a JNI lib that can access Mono... should be fairly easy. (Calling into the lib will automatically set the HOME directory for you as well)

Mono for Android - Now Available on the Market

After another day of messing around compiling Mono, I finally got it working on Android out of the box. The final hurdle was to tweak a few environment variables at build time to pass the "--dynamic-linker /data/data/com.koushikdutta.mono/ld-linux.so" option into the linker. This way the mono binary will actually find the dynamic linker at a valid location in the Android file system rather than fail with a "not found" error.

The application on the market contains the Mono runtime, which will be unpackaged on first run. It will allow you to run simple command line applications from a tool like Terminal Emulator. Here's some more Mono on Android action, sans Debian. The video shows the simple installation followed by a demonstration of the "Hello World" program that is included with the package.

Note that the application is downloaded, installed, and executed, all without root permissions. There are still some outstanding issues though. If you watched carefully, you will see that I redirected stderr to /dev/null. This is because Mono not running under Debian reports a couple of assertions and errors; once upon starting up and again upon exiting. But they don't really seem to be affect anything from as far as I can tell. It is something I'll need to look into. But first... calling into Mono from Java/Dalvik.

Building Mono for Android

<Edit>I have figured out how to get Mono building in the Android build environment fairly easily. Click here for the updated instructions.</Edit>

As I mentioned in previous posts, I managed to get Mono compiled and running under Debian running side by side with Android. I built Mono "natively" by setting up the Debian environment in the emulator. For the sake of having all the instructions in one place on how to set up a native build environment under Debian, I will be consolidating and tweaking the directions from Saurik's post on how to set up Debian on the G1.

Update 1: If you want an installable Mono APK, which is trimmed down to bare minimum dependencies (3MB) and works on non rooted phones, click here or get it off Market. I have updated the instructions below to reflect the configuration and steps I took to build Mono to create this APK.

Update 0: If you just want a premade SD card image for your phone, skip to the very end of this post about deploying Mono to a real phone and follow the directions from there.

Setting up a Native Build Environment under Debian

If you follow the instructions to build Mono for ARM, you will fail miserably. Doing native builds on a G1 is not an option: the hardware is slow, runs out of memory, and just becomes unstable and unresponsive. Ultimately the watchdog timer kicks in and restarts the phone. However, the emulator does not have this issue. It runs faster than a G1, and you can tweak the qemu options to allocate extra RAM. So, the emulator makes a perfect "native" build environment!

As I mentioned, Mono will run under Debian. The Debian installation will be run off the SD card, contained in an ext2 image file. We will set up the ext2 image in the emulator, build Mono, and then transfer the image to a real phone.

Go to a directory somewhere and create a workspace for the installation:

mkdir monoimage

cd monoimage

Set up the SD card image to be used by the emulator by using the mksdcard command that comes with the Android SDK:

# use whatever size you wish

mksdcard 800MB sdcard.img

Build the Debian image as per the instructions in Saurik's post:

sudo apt-get install debootstrap

# use a size smaller than your SD card, must be less than 4GB as that is the max file size for FAT

dd if=/dev/zero of=debian.img seek=749999999 bs=1 count=1

mkefs -F debian.img

sudo mount -o loop debian.img debian

sudo debootstrap --verbose --arch armel --foreign lenny debian http://ftp.de.debian.org/debian

sudo umount debian

Now create the Debian kit on the SD card:

mkdir sdcard

sudo mount -o loop sdcard.img sdcard

sudo mkdir sdcard/kit

sudo cp debian.img sdcard/kit

rm debian.img # don't need this anymore!

# this script sets up the debian mount and gets you a bash prompt

wget http://koushikdutta.blurryfox.com/MonoWalkthrough/bash.sh

sudo cp bash.sh sdcard/kit

# this unmounts the debian image

wget http://koushikdutta.blurryfox.com/MonoWalkthrough/umount.sh

sudo cp umount.sh sdcard/kit

# busybox. handy tools...

wget http://koushikdutta.blurryfox.com/MonoWalkthrough/busybox

sudo cp busybox sdcard/kit

sudo umount sdcard

Time to start the emulator with SD card image we created. However, before we can can do that, we need to make a quick change. The emulator does not support ext2 out of the box, but that can be fixed by recompiling the kernel. I have a prebuilt emulator kernel available for download. (For further do it yourself instructions on how to compile the Android kernel manually, check out Motz's instructions. After following his instructions, you can use "make menuconfig" to tweak the kernel build options and then use "make" to build it.)

# get the custom emulator kernel

wget http://koushikdutta.blurryfox.com/MonoWalkthrough/zImageExt2

Ok, we're set now! Start her up!

# the last 2 arguments tell qemu to provide 512MB of virtual RAM to the emulator

emulator -sdcard sdcard.img -kernel zImageExt2 -qemu -m 512

Now, let's get into a shell on the emulator to finalize the Debian installation:

# This will get you to a "sh" instance on the emulator. Let's get a real bash prompt next.

adb shell

# Ignore any errors you may have with this command!

#You should see "I have no name!@localhost:/#" as your bash prompt after executing this.

. /sdcard/kit/bash.sh

/debootstrap/debootstrap --second-stage

echo 'deb http://ftp.de.debian.org/debian lenny main' > /etc/apt/sources.list

echo 'nameserver 4.2.2.2' > /etc/resolv.conf

# Let's restart the prompt to get rid of that "I have no name!" problem.

exit

# Ignore any errors again. You should get a "localhost:/#" bash prompt now.

. /sdcard/kit/bash.sh

 

Building Mono

Whew! You now have a full Debian distribution on the emulator. Now that we're set up, let's get the ARM portion of Mono built. But first, let's grab all the updates, dependencies, and tools we will need to build it.

apt-get update

apt-get install bison libglib2.0-0 libglib2.0-dev bzip2 gcc make

cd ~

# The following URL is the trunk snapshot at the time of writing this entry.

# Go to http://mono.ximian.com/monobuild/snapshot/sources-trunk/ to get the latest snapshot.

wget http://mono.ximian.com/monobuild/snapshot/snapshot_sources/mono/mono-122990.tar.bz2

tar xvf mono-12290.tar.bz2

cd ~/mono-122990

wget http://anonsvn.mono-project.com/viewvc/trunk/mono/eglib.tar.gz?view=tar

# wget mangles the file name...

tar xvf eglib.tar.gz?view=tar

And now we can finally begin the native build process. First let's set up some environment variables.

# CPPFLAGS=-DARM_FPU_VFP is used because the G1 supports VFP.

# Also, tell the linker where the runtime linker is found, and the path to the libraries.

export 'CFLAGS=-DARM_FPU_NONE -DPLATFORM_ANDROID -Wl,-rpath,.,-rpath,/data/data/com.koushikdutta.mono/assets/lib,--dynamic-linker,/data/data/com.koushikdutta.mono/assets/lib/ld-linux.so.3'

export CPPFLAGS=$CFLAGS

# We need to set the linker flags specifically as well...

export 'LDFLAGS=-Wl,-rpath,.,-rpath,/data/data/com.koushikdutta.mono/assets/lib,--dynamic-linker,/data/data/com.koushikdutta.mono/assets/lib/ld-linux.so.3'

# --disable-mcs-build configures it not to build the managed assemblies.

# Managed assemblies will be built on the host.

# --with-tls-=pthread changes how thread local storage is implemented.

# The default __thread does not work on Android.

# --disable-shared-handles is used because the G1 kernel does not have SYSVIPC enabled.

# --with-static-mono=no forces mono to dynamically link to libmono to save space.

# --with-glib=embedded embeds a trimmer version of glib into Mono.

# --prefix and --exec-prefix inform Mono of where it's binaries, libraries, and assemblies are found.

./configure --disable-mcs-build --with-tls=pthread --disable-shared-handles --with-static_mono=no --with-glib=embedded --prefix=/data/data/com.koushikdutta.mono/assets --exec-prefix=/data/data/com.koushikdutta.mono/assets

# This will take a couple hours... make a pot of coffee or 6.

make

Note that we have only built the Mono runtime at this point. We need to build the managed code portion now. First let's exit the emulator cleanly and unmount the ext2 image cleanly:

# Exit bash

exit

# Unmount the ext2 image

. /sdcard/kit/umount.sh

# Exit adb shell

exit

You can now close the emulator and remount the SD card on your host PC. You should be in the monoimage directory you created earlier.

sudo mount -o loop sdcard.img sdcard

sudo mount -o loop sdcard/kit/debian.img debian

Get the Mono sources again and build them on the host:

wget http://ftp.novell.com/pub/mono/sources/mono/mono-2.0.1.tar.bz2

tar xvf mono-2.0.1.tar.bz2

cd mono-2.0.1

./configure --with-glib=embedded --prefix=/data/data/com.koushikdutta.mono/assets --exec-prefix=/data/data/com.koushikdutta.mono/assets

make

So, both portions of Mono have been built. Let's install the managed portion directly to the ext2 image.

sudo make install DESTDIR=/full/path/to/monoimage/data/data/com.koushikdutta.mono/assets

Unmount the images:

sudo umount debian

sudo umount sdcard

Restart the emulator to finalize the Mono installation (this will overwrite the PC runtime that was installed):

emulator -sdcard sdcard.img -kernel zImageExt2 -qemu -m 512

. /sdcard/kit/bash.sh

cd ~/mono-2.0.1

make install

Mono should now be installed and functional! Let's test it:

cd ~

wget http://koushikdutta.blurryfox.com/MonoWalkthrough/hello.cs

# gmcs IS a managed executable; and obviously quite a complex one at that.

# If it succeeds. Then Mono is working... quite well.

gmcs hello.cs

# Hello World!

mono hello.exe

Before we transfer the Debian image to a real phone, let's shut down gracefully again:

# Exit bash

exit

# Unmount the ext2 image

. /sdcard/kit/umount.sh

# Exit adb shell

exit

Putting Android onto a Real Phone

If you just want a a premade SD card image that contains a working Debian install and Mono runtime, click here. Unzip the sdcard.zip and follow the instructions below to transfer the debian.img (which is contained in the sdcard.img) to your phone.

Your phone must be rooted and have the latest version of modified RC30 (1.3+) . Connect your phone and copy the kit directory from the SD card image to your real SD card. You should preferably do this by mounting your phone's SD card.

# Mount the SD card image first so we can access the kit.

sudo mount -o loop sdcard.img sdcard

# It is actually much faster to mount your phone's SD card as a removable disk on your

# computer and copy the kit directory over from the sdcard.img to the real SD card.

# It is also less likely to be corrupted as well. Don't use adb push unless you really want to.

adb push sdcard/kit /sdcard/kit

Now, go to a bash prompt on your phone and you will be able to run Mono!

adb shell

. /sdcard/kit/bash.sh

cd ~

mono hello.exe

What Now?

Well, now Mono is running on Android, side by side with Debian. However, attempting to invoke it from outside of a chroot /bin/bash will result in a "not found" error. This is because Mono is not configured to use Android's linker (/system/bin/linker), and looks for it at "/lib/linux-ld.so.3". You can fix that by doing the unionfs trick that Saurik posted about in his blog.

However, this process still requires the phone to have root, and have a Debian installation. For this to really be viable, Mono would need to be compiled statically, with all its dependencies. And I have no idea how to do that, yet. If someone does, please let me know!

Dalvik vs Mono

<Edit 2>I managed to get Mono natively on top of Android's libc, Bionic! This resulted in another 50% or so speed improvement; I am guessing this is because Bionic is compiled for the Thumb instruction set, which is supposedly faster. The post has been updated once again with the new results</Edit>

<Edit 1>I've updated the performance scores with Mono as pulled from the trunk (I was using 2.0.1) and using eglib. It's gotten faster and uses even less memory.</Edit>

<Edit 0>The original article was only about Dalvik vs Mono, and begged the question, why didn't Google leverage Mono's existing open source technology? However, I found that Sun has an ARM version of their Java Runtime environment available as a 90 day trial. So I also ran the tests against that as well. Sun's ARM JRE is around as fast Mono, but at a greater memory cost. However, the Sun Java Runtime Environment is not open source, unlike Mono. So, it is not a viable runtime for the Open Handset Alliance's platform.</Edit>

Google has had a little egg on its face recently. They wrote up a 40 page comic touting the awesomeness Chrome V8's performance, only to be thoroughly trounced by TraceMonkey and Squirrelfish Extreme in comparative benchmarks (It's ok guys, I still prefer Chrome as my browser though).

So after that embarrassing showing, I was naturally a little skeptical about the supposed benefits that Dalvik provided for mobile devices. To better understand Dalvik's goals and inner workings, I watched an hour long presentation starring its creator, Dan Bornstein. The two line summary is that Dalvik is designed to minimize memory usage by maximizing shared memory. The memory that Dalvik is sharing are the common framework dex files and application dex files (dex is the byte code the Dalvik interpreter runs).

The first thing that bugged me about this design, is that sharing the code segments of dex files would be completely unnecessary if the applications were purely native. In Linux, the code segments of libraries are shared by all processes anyways. So, realistically, there is no benefit in doing this. In fact, Mono's managed assemblies also reap these same benefits of multiple processes sharing the same code segment in memory.

The second thing that bugged me about this presentation was Dan starts out talking about how battery life is not scaling with Moore's law, which is certainly true. But if the battery is the primary constraint on the device, why is Dalvik so concerned with minimizing memory usage? I am by no means a VM design guru, as I'm sure he is, but I can say the following with certainty:

• Total memory usage has absolutely no impact on battery life. The chips are being powered regardless of how much of their memory is being used. Increasing the total memory available on a device will also only cause marginal increase in battery drain. Memory is not something that taxes the battery compared to other components of the system.
• Battery life is primarily affected by how much you tax the processor and the other hardware components of the device: especially the use of 3G/EDGE and WiFi radios.
• Interpreting byte code will tax the processor and thus the battery much more than native/JIT code.
• Modern (Dream/iPhone comparable) hardware running Windows Mobile is rarely memory constrained, and they don't have a fancy memory sharing runtime. Memory constraints (in my mobile experience) become an issue on Windows Mobile when several applications are running at the same time. And this problem can be solved at the application framework level; such as how the Android Application life cycle is implemented. If all applications can suspend and restore at the system's whim, then memory consumption is trivialized. However, the application framework is not tied to the Dalvik runtime. (I.e., it can be ported to work with native code, Mono/.NET, JVM, whatever)
• Generally in applications, the code's memory footprint is trivial compared to the application data memory footprint (images, text, video, etc). Dalvik is overly concerned with optimizing the memory size of dex files and sharing memory. Dan's presentation did a comparison between the Browser's Java .class files versus the Dalvik .dex files (the .dex file is around 250k, around half the size of the .class files). My reaction to that is whoopity-shit. What happens when you start up the Browser? You head to your favorite webpage, it loads up a half dozen images which decompress to a raw R5G6B5 format, which then clocks in at several megabytes. That really trivializes the few hundred kilobytes that Dalvik is trying to save.

This leads me to believe that Google committed a classic performance optimization mistake: they are optimizing an aspect of the system that is trivial in the grand scheme. To poke a little nerdy fun at a portion of Dan's presentation, it is akin to tweaking your for loops to iterate downwards for better performance. And all the while the loops are being used perform an inefficient selection sort.

Regardless, all speculations and theories aside, let's let real world scenarios speak for itself. The T-Mobile G1, aka HTC Dream, has terrible battery life when compared to its siblings of the Windows Mobile variety. (I own or have owned a Dream, Touch, Touch Cruise, and Touch Diamond in the past year)

 

Runtime Memory Usage

My first test was to create a simple hello world program for both runtimes. Hello World would be printed to the screen, and then the thread would sleep for 30 seconds, allowing me to peek at the process' memory usage.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     14500 43.0  3.8  14788  3816 pts/0    Rl+  01:35   0:01 ./mono MonoTests.exe ...
root      1605 49.0  4.8  34940  4812 pts/2    Sl+  20:10   0:00 /system/bin/dalvikvm ...
root      4464 64.0  6.7 180388  6740 pts/1    Sl+  18:21   0:02 java -jar JavaTests ...

Ok, so this surprised me a bit. Mono needs around half the memory to start up? Using pmap on the dalvikvm process shows that it is referencing a lot more "base" system libraries than Mono. I suppose at the end of the day, it doesn't matter, because on Linux, libraries are loaded and shared between processes. I also took a pmap snapshot of Mono and Java for those interested (Sun's ARM JRE is quite bloated...).

 

Performance Comparisons

I'll be the first to admit, these comparisons aren't fair at all. No interpreter will ever run as fast as native code. But, I'll test it anyway. These tests purposely steer clear of the calling into underlying libraries. The goal is to benchmark the memory usage and performance of the runtimes themselves by way of very simple applications. Click here to view the code for the Java and the C# tests.

Selection Sort Test

This test creates a reverse sorted array of integers between 0 and 1000 and sorts them into increasing order (and does it 10 times, excluding the results of the first). Lower numbers are better. Results:

	Time (ms)
Dalvik	4668
Mono	411
Java SE for Embedded	895

 

Class (and Structure) method call Test

This test creates instantiates an array of 10000 FibContainer instances. FibContainer is a simple class:

C#

Java

class FibContainer {     int myValue;     public int Value     {         get         {             return myValue;         }         set         {             myValue = value;         }     }     public void Compute(FibContainer previous, FibContainer beforePrevious)     {         Value = previous.Value + beforePrevious.Value;     } }   struct FibContainerStruct {     int myValue;     public int Value     {         get         {             return myValue;         }         set         {             myValue = value;         }     }     public void Compute(FibContainerStruct previous, FibContainerStruct beforePrevious)     {         Value = previous.Value + beforePrevious.Value;     } }

class FibContainer {     private int mValue;     public int getValue()     {         return mValue;     }     public void setValue(int value)     {         mValue = value;     }     public void Compute(FibContainer previous, FibContainer beforePrevious)     {         setValue(previous.getValue() + beforePrevious.getValue());     } }

It then iterates over the array and calculates and stores the Fibonacci series. The test notes 3 things: total memory in use by the runtime after allocating the array, the time to allocate the array, and the time to calculate the Fibonacci series (the method calls are intentional). Note that I also performed this same test on Mono with a feature not available in the Java language: I used a struct instead of a class. Smaller numbers are better in all cases. This test was run 50 times (the first excluded):

	Memory (bytes)	Allocation Time (ms)	Calculation Time (ms)
Dalvik Class	9817240	5388	3013
Mono Class	7376896	933	167
Mono Struct	2007040	12	107
Java Class	10438176	319	211
C++ Class	1960000	N/A	N/A

Equivalent C++ code would allocate the amount of memory shown above. So, as you can see, Mono has around 33% less overhead when allocating classes. It is also around 8% faster at doing those allocations, and the calculation time completely blows Dalvik out of the water. And by way of intelligent usage of structs in Mono, you can leverage near bare metal memory usage. (Not to mention that arrays of structs containing blittable types are themselves blittable. This is very friendly for processor/memory caches. It also provides for easier interaction with native calls, such sending an array of vertices to OpenGL. But I digress.)

Long story short: from my initial, limited, and naive testing, Mono is faster and uses less memory than Dalvik. And it is not even designed to run on mobile devices. So it begs the question, why didn't Google just convert the .class files to CIL and use the Mono runtime? That way they wouldn't have alienated Java developers, would have access to open source Java libraries they so covet, wooed .NET developers, and wouldn't have needed to invent their own sub-par runtime!

I also want to test out the performance of the two Garbage Collectors as well as native function invocation (P/Invoke and JNI), but I'm hungry and will do that in a later post. Until next time friends!

Mono on Android - With Gratuitous Shaky Cam Action

I thought I had Mono working; however, that was just on the emulator. The phone was failing due some triple combo exception (StackOverflow, OutofMemory, NullReference) that was ultimately related to my temporary hack/kludge. Getting it working on a phone took a couple more tweaks and fixes provided by Miguel de Icaza and Mark Chricton. The following video shows me natively compiling a simple hello world application using gmcs and then running it. Performance comparisons coming soon (for real this time)! (Note: The initial load times on mono and gmcs are abysmal since the Debian install is running off the SD card. That shouldn't affect execution time once everything is loaded into memory.)