My Brain Hurts: Hacking

Showing posts with label Hacking. Show all posts

Manually Updating your rooted XOOM

What I just roughly tested.

Download the update.zip: http://www.mediafire.com/?hxmkxdafhkahi41
adb push <zip name> /sdcard/update.zip
adb shell
su
cd /cache
mkdir recovery #it may already exist
cd recovery
echo '--update_package=CACHE:/recovery/update.zip' > command
cat /sdcard/update.zip > update.zip
reboot recovery

Please do not repost this right into your blog. Last time you guys did that, my instructions had an error, which got corrected. On my site at least. But not in all the copy-pasted locations.

Motorola XOOM Rooted

Since it’s another Google experience device, and ships with fastboot support (albeit, limited), it really does come rooted out of the box. Just needed to figure out the board kernel base, and compile up a new kernel.

Unfortunately the kernel was not available in the Android repositories. At first, I tried using the Harmony kernel, since they are both tegra 2 250 chips. That turned out to be major fail. As soon as I was about to give up, I noticed that AOSP had updated their tegra kernel repository with some new tasty branches for stingray. Kudos to these guys for being so on the ball! I was able to compile that up and get a working recovery to obtain root, and then get Superuser on the device.

I also built up a recovery, but due to a nonfunctional SD card slot (until they release a firmware update that enables the slot), nothing really works. That will come later.

Here are the instructions to root your device (this assumes you have adb and fastboot installed on your computer):

# Download the XOOM root zip.
# Unzip the package.
# Put your junk in the box.
adb reboot bootloader (skip the next 3 steps if you have already unlocked via fastboot)
fastboot oem unlock
# wait for reboot
adb reboot bootloader
fastboot flash boot rootboot.img
fastboot reboot
# wait for reboot
adb remount
adb push su /system/bin/
adb shell ln –s /system/bin/su /system/xbin/su
adb shell chmod 4755 /system/bin/su
adb push Superuser.apk /system/app/

Yep, that should do it.

As I mentioned, I have a working recovery, but will not be releasing it until Google or I get the SD card working.

ROM Manager support will come as soon as that happens. But feel free to buy a Premium copy in advance. Winking smile

And hit me up on Twitter @koush!

PS. Stock boot image for XOOM.

Android.dll

Picture above represents the results of 3 sleepless nights. No, I didn’t rewrite the Android framework. I generated a bunch of proxy classes to access them from Mono though. Rough idea of how it works:

Java application reflects through Android framework SDK jar and outputs an XML file that represents the object model.
C# application generates thousands of code files of said object model. (This is similar to jni4net, but that doesn’t work on anything but Windows).
Tag all the methods with the [MethodImpl(MethodImplOptions.InternalCall)] attribute.
Implement the method calls at runtime with a combination of dynamically created expression tree delegates and mono_add_internal_call.

The class contents actually have no code, so the total dll size is pretty small (800k). Here’s what the LinearLayout “implementation” looks like:

namespace android.widget 
{ 
    public class LinearLayout : android.view.ViewGroup
    { 
        public class LayoutParams : android.view.ViewGroup.MarginLayoutParams
        { 
            [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
            public virtual extern java.lang.String debug(java.lang.String arg0); 
            [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
            public extern LayoutParams(android.content.Context arg0, android.util.AttributeSet arg1); 
            [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
            public extern LayoutParams(int arg0, int arg1); 
            [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
            public extern LayoutParams(int arg0, int arg1, float arg2); 
            [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
            public extern LayoutParams(android.view.ViewGroup.LayoutParams arg0); 
            [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
            public extern LayoutParams(android.view.ViewGroup.MarginLayoutParams arg0); 
            [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
            internal extern LayoutParams(); 
        } 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public virtual extern void setGravity(int arg0); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        protected override extern void onLayout(bool arg0, int arg1, int arg2, int arg3, int arg4); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public override extern int getBaseline(); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        protected override extern void onMeasure(int arg0, int arg1); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        protected override extern bool checkLayoutParams(android.view.ViewGroup.LayoutParams arg0); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public virtual extern new android.widget.LinearLayout.LayoutParams generateLayoutParams(android.util.AttributeSet arg0); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        protected virtual extern new android.widget.LinearLayout.LayoutParams generateLayoutParams(android.view.ViewGroup.LayoutParams arg0); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        protected virtual extern new android.widget.LinearLayout.LayoutParams generateDefaultLayoutParams(); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public virtual extern bool isBaselineAligned(); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public virtual extern void setBaselineAligned(bool arg0); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public virtual extern int getBaselineAlignedChildIndex(); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public virtual extern void setBaselineAlignedChildIndex(int arg0); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public virtual extern float getWeightSum(); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public virtual extern void setWeightSum(float arg0); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public virtual extern void setOrientation(int arg0); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public virtual extern int getOrientation(); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public virtual extern void setHorizontalGravity(int arg0); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public virtual extern void setVerticalGravity(int arg0); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public extern LinearLayout(android.content.Context arg0, android.util.AttributeSet arg1); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        public extern LinearLayout(android.content.Context arg0); 
        [global::System.Runtime.CompilerServices.MethodImpl(global::System.Runtime.CompilerServices.MethodImplOptions.InternalCall)] 
        internal extern LinearLayout(); 
    } 
}

A year of on and off hacking coming to fruition soon…

Blacklisted by Google's Geolocation Service

Several months ago, I described how one could use Google Gears to easily add Geolocation and GPS to their Windows Mobile applications. Basically, this is a trick that would work with any embedded browser by adding and scripting a DOM element, or by using my unsavory window.location hack when that is not possible (as is in the case of Pocket IE).

I was recently trying to reuse this code, and I noticed that the Geolocation sample page I had hosted was not working on Windows Mobile anymore; the service seemed to be returning an invalid response. I navigated to that same URL in Google Chrome and Firefox and it worked fine. I then tried navigating to the Gears Geolocation sample in Pocket IE, and that worked as well. I verified that I still had the most up to date code hosted on my domain. Still no luck, as the code had not changed.

So out of curiosity, I hosted the geolocation page on a separate domain and navigated to it within Pocket IE. 'Lo and behold, it starts working. So mentally, I constructed this chart in my mind:

	Pocket IE	Google Chrome	Firefox
code.google.com	Works	Works	Works
koushikdutta.blurryfox.com	Broken!	Works	Works
supersecretdomain.com	Works	Works	Works

My conclusion? Google has blacklisted my domain! And, to be honest, I'm not really sure why. I'm using their sample code on my domain to get geolocation data, but using the results outside of the context of the browser.

Congratulations Google, you have earned the first honorary YouBastard tag on my blog!

Edit:

Google Gears EULA. I haven't violated it as far as I can see, but they do reserve the right to cut off access to the service at their whim.

Android's Linker

"Next task: create a JNI lib that can access Mono... should be fairly easy." Famous last words. I guess I didn't think through the problem of Android using a non-standard linker. Not only is it named and located differently (/system/bin/linker as opposed to ld-linux.so.3), the format of the dynamic libraries it loads is also different as well. So, Android's linker can not load Linux .so files, and vice versa.

This basically means that Mono with its current build configuration can not interact with the Android system at all, other than through RPC. And while it certainly is possible[0] to do that (and Android uses RPC extensively), I feel that would be a clumsy solution, as my goal is to get Mono running side by side with Java/Dalvik, and allow seamless interaction with runtime through usage of Dynamic Language Runtime (I'm not sure if Mono supports the "dynamic" keyword/type yet, but when it does... this will be quite amazing).

There are two possible solutions to this:

Recompile Mono and the glibc dependencies using the linker script for Android.
Compile Mono within the Android build environment so it uses Bionic instead of libc.

Using Crosstool, I tried to build a new toolchain tailored for Android's linker. I quickly found that whoever set up the toolchain/glibc build process must be a sadist. I wasn't too hesitant to give up on this approach anyway: this would result in two copies of libc being loaded (if that is even possible).

Although getting Mono fully integrated to work with Android's libraries will take considerably longer, I can rest knowing it is the proper solution.

[0] Set up an RPC server for OpenGL ES or skia? Ugh...

Mono for Android - Now Available on the Market

After another day of messing around compiling Mono, I finally got it working on Android out of the box. The final hurdle was to tweak a few environment variables at build time to pass the "--dynamic-linker /data/data/com.koushikdutta.mono/ld-linux.so" option into the linker. This way the mono binary will actually find the dynamic linker at a valid location in the Android file system rather than fail with a "not found" error.

The application on the market contains the Mono runtime, which will be unpackaged on first run. It will allow you to run simple command line applications from a tool like Terminal Emulator. Here's some more Mono on Android action, sans Debian. The video shows the simple installation followed by a demonstration of the "Hello World" program that is included with the package.

Note that the application is downloaded, installed, and executed, all without root permissions. There are still some outstanding issues though. If you watched carefully, you will see that I redirected stderr to /dev/null. This is because Mono not running under Debian reports a couple of assertions and errors; once upon starting up and again upon exiting. But they don't really seem to be affect anything from as far as I can tell. It is something I'll need to look into. But first... calling into Mono from Java/Dalvik.

Building Mono for Android

<Edit>I have figured out how to get Mono building in the Android build environment fairly easily. Click here for the updated instructions.</Edit>

As I mentioned in previous posts, I managed to get Mono compiled and running under Debian running side by side with Android. I built Mono "natively" by setting up the Debian environment in the emulator. For the sake of having all the instructions in one place on how to set up a native build environment under Debian, I will be consolidating and tweaking the directions from Saurik's post on how to set up Debian on the G1.

Update 1: If you want an installable Mono APK, which is trimmed down to bare minimum dependencies (3MB) and works on non rooted phones, click here or get it off Market. I have updated the instructions below to reflect the configuration and steps I took to build Mono to create this APK.

Update 0: If you just want a premade SD card image for your phone, skip to the very end of this post about deploying Mono to a real phone and follow the directions from there.

Setting up a Native Build Environment under Debian

If you follow the instructions to build Mono for ARM, you will fail miserably. Doing native builds on a G1 is not an option: the hardware is slow, runs out of memory, and just becomes unstable and unresponsive. Ultimately the watchdog timer kicks in and restarts the phone. However, the emulator does not have this issue. It runs faster than a G1, and you can tweak the qemu options to allocate extra RAM. So, the emulator makes a perfect "native" build environment!

As I mentioned, Mono will run under Debian. The Debian installation will be run off the SD card, contained in an ext2 image file. We will set up the ext2 image in the emulator, build Mono, and then transfer the image to a real phone.

Go to a directory somewhere and create a workspace for the installation:

mkdir monoimage

cd monoimage

Set up the SD card image to be used by the emulator by using the mksdcard command that comes with the Android SDK:

# use whatever size you wish

mksdcard 800MB sdcard.img

Build the Debian image as per the instructions in Saurik's post:

sudo apt-get install debootstrap

# use a size smaller than your SD card, must be less than 4GB as that is the max file size for FAT

dd if=/dev/zero of=debian.img seek=749999999 bs=1 count=1

mkefs -F debian.img

sudo mount -o loop debian.img debian

sudo debootstrap --verbose --arch armel --foreign lenny debian http://ftp.de.debian.org/debian

sudo umount debian

Now create the Debian kit on the SD card:

mkdir sdcard

sudo mount -o loop sdcard.img sdcard

sudo mkdir sdcard/kit

sudo cp debian.img sdcard/kit

rm debian.img # don't need this anymore!

# this script sets up the debian mount and gets you a bash prompt

wget http://koushikdutta.blurryfox.com/MonoWalkthrough/bash.sh

sudo cp bash.sh sdcard/kit

# this unmounts the debian image

wget http://koushikdutta.blurryfox.com/MonoWalkthrough/umount.sh

sudo cp umount.sh sdcard/kit

# busybox. handy tools...

wget http://koushikdutta.blurryfox.com/MonoWalkthrough/busybox

sudo cp busybox sdcard/kit

sudo umount sdcard

Time to start the emulator with SD card image we created. However, before we can can do that, we need to make a quick change. The emulator does not support ext2 out of the box, but that can be fixed by recompiling the kernel. I have a prebuilt emulator kernel available for download. (For further do it yourself instructions on how to compile the Android kernel manually, check out Motz's instructions. After following his instructions, you can use "make menuconfig" to tweak the kernel build options and then use "make" to build it.)

# get the custom emulator kernel

wget http://koushikdutta.blurryfox.com/MonoWalkthrough/zImageExt2

Ok, we're set now! Start her up!

# the last 2 arguments tell qemu to provide 512MB of virtual RAM to the emulator

emulator -sdcard sdcard.img -kernel zImageExt2 -qemu -m 512

Now, let's get into a shell on the emulator to finalize the Debian installation:

# This will get you to a "sh" instance on the emulator. Let's get a real bash prompt next.

adb shell

# Ignore any errors you may have with this command!

#You should see "I have no name!@localhost:/#" as your bash prompt after executing this.

. /sdcard/kit/bash.sh

/debootstrap/debootstrap --second-stage

echo 'deb http://ftp.de.debian.org/debian lenny main' > /etc/apt/sources.list

echo 'nameserver 4.2.2.2' > /etc/resolv.conf

# Let's restart the prompt to get rid of that "I have no name!" problem.

exit

# Ignore any errors again. You should get a "localhost:/#" bash prompt now.

. /sdcard/kit/bash.sh

Building Mono

Whew! You now have a full Debian distribution on the emulator. Now that we're set up, let's get the ARM portion of Mono built. But first, let's grab all the updates, dependencies, and tools we will need to build it.

apt-get update

apt-get install bison libglib2.0-0 libglib2.0-dev bzip2 gcc make

cd ~

# The following URL is the trunk snapshot at the time of writing this entry.

# Go to http://mono.ximian.com/monobuild/snapshot/sources-trunk/ to get the latest snapshot.

wget http://mono.ximian.com/monobuild/snapshot/snapshot_sources/mono/mono-122990.tar.bz2

tar xvf mono-12290.tar.bz2

cd ~/mono-122990

wget http://anonsvn.mono-project.com/viewvc/trunk/mono/eglib.tar.gz?view=tar

# wget mangles the file name...

tar xvf eglib.tar.gz?view=tar

And now we can finally begin the native build process. First let's set up some environment variables.

# CPPFLAGS=-DARM_FPU_VFP is used because the G1 supports VFP.

# Also, tell the linker where the runtime linker is found, and the path to the libraries.

export 'CFLAGS=-DARM_FPU_NONE -DPLATFORM_ANDROID -Wl,-rpath,.,-rpath,/data/data/com.koushikdutta.mono/assets/lib,--dynamic-linker,/data/data/com.koushikdutta.mono/assets/lib/ld-linux.so.3'

export CPPFLAGS=$CFLAGS

# We need to set the linker flags specifically as well...

export 'LDFLAGS=-Wl,-rpath,.,-rpath,/data/data/com.koushikdutta.mono/assets/lib,--dynamic-linker,/data/data/com.koushikdutta.mono/assets/lib/ld-linux.so.3'

# --disable-mcs-build configures it not to build the managed assemblies.

# Managed assemblies will be built on the host.

# --with-tls-=pthread changes how thread local storage is implemented.

# The default __thread does not work on Android.

# --disable-shared-handles is used because the G1 kernel does not have SYSVIPC enabled.

# --with-static-mono=no forces mono to dynamically link to libmono to save space.

# --with-glib=embedded embeds a trimmer version of glib into Mono.

# --prefix and --exec-prefix inform Mono of where it's binaries, libraries, and assemblies are found.

./configure --disable-mcs-build --with-tls=pthread --disable-shared-handles --with-static_mono=no --with-glib=embedded --prefix=/data/data/com.koushikdutta.mono/assets --exec-prefix=/data/data/com.koushikdutta.mono/assets

# This will take a couple hours... make a pot of coffee or 6.

make

Note that we have only built the Mono runtime at this point. We need to build the managed code portion now. First let's exit the emulator cleanly and unmount the ext2 image cleanly:

# Exit bash

exit

# Unmount the ext2 image

. /sdcard/kit/umount.sh

# Exit adb shell

exit

You can now close the emulator and remount the SD card on your host PC. You should be in the monoimage directory you created earlier.

sudo mount -o loop sdcard.img sdcard

sudo mount -o loop sdcard/kit/debian.img debian

Get the Mono sources again and build them on the host:

wget http://ftp.novell.com/pub/mono/sources/mono/mono-2.0.1.tar.bz2

tar xvf mono-2.0.1.tar.bz2

cd mono-2.0.1

./configure --with-glib=embedded --prefix=/data/data/com.koushikdutta.mono/assets --exec-prefix=/data/data/com.koushikdutta.mono/assets

make

So, both portions of Mono have been built. Let's install the managed portion directly to the ext2 image.

sudo make install DESTDIR=/full/path/to/monoimage/data/data/com.koushikdutta.mono/assets

Unmount the images:

sudo umount debian

sudo umount sdcard

Restart the emulator to finalize the Mono installation (this will overwrite the PC runtime that was installed):

emulator -sdcard sdcard.img -kernel zImageExt2 -qemu -m 512

. /sdcard/kit/bash.sh

cd ~/mono-2.0.1

make install

Mono should now be installed and functional! Let's test it:

cd ~

wget http://koushikdutta.blurryfox.com/MonoWalkthrough/hello.cs

# gmcs IS a managed executable; and obviously quite a complex one at that.

# If it succeeds. Then Mono is working... quite well.

gmcs hello.cs

# Hello World!

mono hello.exe

Before we transfer the Debian image to a real phone, let's shut down gracefully again:

# Exit bash

exit

# Unmount the ext2 image

. /sdcard/kit/umount.sh

# Exit adb shell

exit

Putting Android onto a Real Phone

If you just want a a premade SD card image that contains a working Debian install and Mono runtime, click here. Unzip the sdcard.zip and follow the instructions below to transfer the debian.img (which is contained in the sdcard.img) to your phone.

Your phone must be rooted and have the latest version of modified RC30 (1.3+) . Connect your phone and copy the kit directory from the SD card image to your real SD card. You should preferably do this by mounting your phone's SD card.

# Mount the SD card image first so we can access the kit.

sudo mount -o loop sdcard.img sdcard

# It is actually much faster to mount your phone's SD card as a removable disk on your

# computer and copy the kit directory over from the sdcard.img to the real SD card.

# It is also less likely to be corrupted as well. Don't use adb push unless you really want to.

adb push sdcard/kit /sdcard/kit

Now, go to a bash prompt on your phone and you will be able to run Mono!

adb shell

. /sdcard/kit/bash.sh

cd ~

mono hello.exe

What Now?

Well, now Mono is running on Android, side by side with Debian. However, attempting to invoke it from outside of a chroot /bin/bash will result in a "not found" error. This is because Mono is not configured to use Android's linker (/system/bin/linker), and looks for it at "/lib/linux-ld.so.3". You can fix that by doing the unionfs trick that Saurik posted about in his blog.

However, this process still requires the phone to have root, and have a Debian installation. For this to really be viable, Mono would need to be compiled statically, with all its dependencies. And I have no idea how to do that, yet. If someone does, please let me know!

Dalvik vs Mono

<Edit 2>I managed to get Mono natively on top of Android's libc, Bionic! This resulted in another 50% or so speed improvement; I am guessing this is because Bionic is compiled for the Thumb instruction set, which is supposedly faster. The post has been updated once again with the new results</Edit>

<Edit 1>I've updated the performance scores with Mono as pulled from the trunk (I was using 2.0.1) and using eglib. It's gotten faster and uses even less memory.</Edit>

<Edit 0>The original article was only about Dalvik vs Mono, and begged the question, why didn't Google leverage Mono's existing open source technology? However, I found that Sun has an ARM version of their Java Runtime environment available as a 90 day trial. So I also ran the tests against that as well. Sun's ARM JRE is around as fast Mono, but at a greater memory cost. However, the Sun Java Runtime Environment is not open source, unlike Mono. So, it is not a viable runtime for the Open Handset Alliance's platform.</Edit>

Google has had a little egg on its face recently. They wrote up a 40 page comic touting the awesomeness Chrome V8's performance, only to be thoroughly trounced by TraceMonkey and Squirrelfish Extreme in comparative benchmarks (It's ok guys, I still prefer Chrome as my browser though).

So after that embarrassing showing, I was naturally a little skeptical about the supposed benefits that Dalvik provided for mobile devices. To better understand Dalvik's goals and inner workings, I watched an hour long presentation starring its creator, Dan Bornstein. The two line summary is that Dalvik is designed to minimize memory usage by maximizing shared memory. The memory that Dalvik is sharing are the common framework dex files and application dex files (dex is the byte code the Dalvik interpreter runs).

The first thing that bugged me about this design, is that sharing the code segments of dex files would be completely unnecessary if the applications were purely native. In Linux, the code segments of libraries are shared by all processes anyways. So, realistically, there is no benefit in doing this. In fact, Mono's managed assemblies also reap these same benefits of multiple processes sharing the same code segment in memory.

The second thing that bugged me about this presentation was Dan starts out talking about how battery life is not scaling with Moore's law, which is certainly true. But if the battery is the primary constraint on the device, why is Dalvik so concerned with minimizing memory usage? I am by no means a VM design guru, as I'm sure he is, but I can say the following with certainty:

Total memory usage has absolutely no impact on battery life. The chips are being powered regardless of how much of their memory is being used. Increasing the total memory available on a device will also only cause marginal increase in battery drain. Memory is not something that taxes the battery compared to other components of the system.
Battery life is primarily affected by how much you tax the processor and the other hardware components of the device: especially the use of 3G/EDGE and WiFi radios.
Interpreting byte code will tax the processor and thus the battery much more than native/JIT code.
Modern (Dream/iPhone comparable) hardware running Windows Mobile is rarely memory constrained, and they don't have a fancy memory sharing runtime. Memory constraints (in my mobile experience) become an issue on Windows Mobile when several applications are running at the same time. And this problem can be solved at the application framework level; such as how the Android Application life cycle is implemented. If all applications can suspend and restore at the system's whim, then memory consumption is trivialized. However, the application framework is not tied to the Dalvik runtime. (I.e., it can be ported to work with native code, Mono/.NET, JVM, whatever)
Generally in applications, the code's memory footprint is trivial compared to the application data memory footprint (images, text, video, etc). Dalvik is overly concerned with optimizing the memory size of dex files and sharing memory. Dan's presentation did a comparison between the Browser's Java .class files versus the Dalvik .dex files (the .dex file is around 250k, around half the size of the .class files). My reaction to that is whoopity-shit. What happens when you start up the Browser? You head to your favorite webpage, it loads up a half dozen images which decompress to a raw R5G6B5 format, which then clocks in at several megabytes. That really trivializes the few hundred kilobytes that Dalvik is trying to save.

This leads me to believe that Google committed a classic performance optimization mistake: they are optimizing an aspect of the system that is trivial in the grand scheme. To poke a little nerdy fun at a portion of Dan's presentation, it is akin to tweaking your for loops to iterate downwards for better performance. And all the while the loops are being used perform an inefficient selection sort.

Regardless, all speculations and theories aside, let's let real world scenarios speak for itself. The T-Mobile G1, aka HTC Dream, has terrible battery life when compared to its siblings of the Windows Mobile variety. (I own or have owned a Dream, Touch, Touch Cruise, and Touch Diamond in the past year)

Runtime Memory Usage

My first test was to create a simple hello world program for both runtimes. Hello World would be printed to the screen, and then the thread would sleep for 30 seconds, allowing me to peek at the process' memory usage.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     14500 43.0 3.8 14788 3816 pts/0    Rl+ 01:35   0:01 ./mono MonoTests.exe ...
root      1605 49.0 4.8 34940 4812 pts/2    Sl+ 20:10   0:00 /system/bin/dalvikvm ...
root      4464 64.0 6.7 180388 6740 pts/1    Sl+ 18:21   0:02 java -jar JavaTests ...

Ok, so this surprised me a bit. Mono needs around half the memory to start up? Using pmap on the dalvikvm process shows that it is referencing a lot more "base" system libraries than Mono. I suppose at the end of the day, it doesn't matter, because on Linux, libraries are loaded and shared between processes. I also took a pmap snapshot of Mono and Java for those interested (Sun's ARM JRE is quite bloated...).

Performance Comparisons

I'll be the first to admit, these comparisons aren't fair at all. No interpreter will ever run as fast as native code. But, I'll test it anyway. These tests purposely steer clear of the calling into underlying libraries. The goal is to benchmark the memory usage and performance of the runtimes themselves by way of very simple applications. Click here to view the code for the Java and the C# tests.

Selection Sort Test

This test creates a reverse sorted array of integers between 0 and 1000 and sorts them into increasing order (and does it 10 times, excluding the results of the first). Lower numbers are better. Results:

	Time (ms)
Dalvik	4668
Mono	411
Java SE for Embedded	895

Class (and Structure) method call Test

This test creates instantiates an array of 10000 FibContainer instances. FibContainer is a simple class:

Java

class FibContainer
{
    int myValue;
    public int Value
    {
        get
        {
            return myValue;
        }
        set
        {
            myValue = value;
        }
    }

    public void Compute(FibContainer previous, FibContainer beforePrevious)
    {
        Value = previous.Value + beforePrevious.Value;
    }
}

struct FibContainerStruct
{
    int myValue;
    public int Value
    {
        get
        {
            return myValue;
        }
        set
        {
            myValue = value;
        }
    }

    public void Compute(FibContainerStruct previous, FibContainerStruct beforePrevious)
    {
        Value = previous.Value + beforePrevious.Value;
    }
}

class FibContainer
{
private int mValue;

    public int getValue()
    {
        return mValue;
    }

    public void setValue(int value)
    {
        mValue = value;
    }

    public void Compute(FibContainer previous, FibContainer beforePrevious)
    {
        setValue(previous.getValue() + beforePrevious.getValue());
    }
}

It then iterates over the array and calculates and stores the Fibonacci series. The test notes 3 things: total memory in use by the runtime after allocating the array, the time to allocate the array, and the time to calculate the Fibonacci series (the method calls are intentional). Note that I also performed this same test on Mono with a feature not available in the Java language: I used a struct instead of a class. Smaller numbers are better in all cases. This test was run 50 times (the first excluded):

	Memory (bytes)	Allocation Time (ms)	Calculation Time (ms)
Dalvik Class	9817240	5388	3013
Mono Class	7376896	933	167
Mono Struct	2007040	12	107
Java Class	10438176	319	211
C++ Class	1960000	N/A	N/A

Equivalent C++ code would allocate the amount of memory shown above. So, as you can see, Mono has around 33% less overhead when allocating classes. It is also around 8% faster at doing those allocations, and the calculation time completely blows Dalvik out of the water. And by way of intelligent usage of structs in Mono, you can leverage near bare metal memory usage. (Not to mention that arrays of structs containing blittable types are themselves blittable. This is very friendly for processor/memory caches. It also provides for easier interaction with native calls, such sending an array of vertices to OpenGL. But I digress.)

Long story short: from my initial, limited, and naive testing, Mono is faster and uses less memory than Dalvik. And it is not even designed to run on mobile devices. So it begs the question, why didn't Google just convert the .class files to CIL and use the Mono runtime? That way they wouldn't have alienated Java developers, would have access to open source Java libraries they so covet, wooed .NET developers, and wouldn't have needed to invent their own sub-par runtime!

I also want to test out the performance of the two Garbage Collectors as well as native function invocation (P/Invoke and JNI), but I'm hungry and will do that in a later post. Until next time friends!

Mono on Android - With Gratuitous Shaky Cam Action

I thought I had Mono working; however, that was just on the emulator. The phone was failing due some triple combo exception (StackOverflow, OutofMemory, NullReference) that was ultimately related to my temporary hack/kludge. Getting it working on a phone took a couple more tweaks and fixes provided by Miguel de Icaza and Mark Chricton. The following video shows me natively compiling a simple hello world application using gmcs and then running it. Performance comparisons coming soon (for real this time)! (Note: The initial load times on mono and gmcs are abysmal since the Debian install is running off the SD card. That shouldn't affect execution time once everything is loaded into memory.)

Setting up a Native Build Environment for Android

My initial attempts at cross compiling Mono were on Scratchbox. But Scratchbox was having technical difficulties building Mono from the trunk (not sure why), did not have a working apt-get, the Mono binary would not actually run when it did build, etc. What I really wanted was to do real native builds. However, the G1/Dream has limited RAM (96MB total I think). As a result, compilation took forever, and often the watchdog daemon would kick in and restart the phone due to unresponsiveness. So that is not an ideal build environment. So my solution was to get Debian running on the Android emulator.

I figured it would be straightforward enough, but I ran into a snag mounting the ext2 image containing the Debian boot strap. That was solved by recompiling the Android Kernel with ext2 support added:

Follow these directions to prepare to build the kernel.
Before building the kernel, run make menuconfig, and add ext2.
Now make.

That will give you a an image with ext2 compiled into the kernel. For the lazy, click here for the compiled kernel image.

When you start the emulator from the command line, use the following flags:

emulator -kernel zImage -sdcard sdcard.img -qemu -m 512 # The '-qemu -m 512' options tells qemu to start with 512MB of virtual RAM!

Now install Debian with the same instructions Saurik's post. Note that you will not need to insmod the ext2.ko module. Always remember to unmount the ext2 partition before closing the emulator. Otherwise you may end up getting some nondescript NFS error; probably a result of file system corruption. However, this is fixable by mounting the ext2 file in Linux and running the fsck tool.

Mono on Android - Success At Last!

Last week, I blogged about my attempts to get Mono running on Android. I was running into a nasty Illegal Instruction issue, which I didn't look into further until today. And after implementing a quick hack/fix I got it working!

After running strace on Mono, I was able to determine that the Illegal Instruction was occuring in mini-arm.c in the mono_arch_flush_icache function. I wasn't able to decipher whether the inline assembly was correct or not; but I got the jist of what it was trying to do: it was calling cacheflush, but failing with the Illegal Instruction in there. I looked at the mini-x86.c and saw that the equivalent mono_arch_flush_icache was empty; the comment indicating it was "not needed". I don't know if this is some vestigial code or what, but I figured it would be relatively benign to not flush the instruction cache and just see what happens. And the grand results:

root@localhost:~/Desktop# mono test.exe

Hello World

This would not be possible without the help of my co-worker, Kevin Eustice, and my abhorrence towards Java.

Expect to see some performance comparisons between Mono vs Dalvik shortly! Though I don't expect it to even be a fair competition, because Dalvik does not JIT. Lame.

Quick and Dirty Converter Tool to Create Android Splash Images

An Android splash image is just a 320x480 bitmap in R5G6B5 format. But there is a lack of decent tool that will easily create such a file for you. I wrote one up really quickly to fill that gap. This will perform all the resizing, pixel format conversion, and centering for you. You can also preview the image. Pretty handy.

To flash an image:

Start your phone up in Fastboot mode (you must have the Engineering bootloader). You can do this by holding camera and pressing power. Then connect your phone to your PC. Make sure that it is detected as HTC Dream. If you only see "High Composite Device", you will need to install the Android USB Driver. Then press back until it says "Fastboot".
Convert an image using the tool.
Type the following command: fastboot flash splash1 mysplash.rgb565
Reboot your phone with the command: fastboot reboot

You should now see your new splash screen!

Download for the source code and binaries for the converter.

Fastboot for Windows.

Fastboot for Linux.

Fastboot for Mac OSX.

Building the Android source and Deploying it to your G1 or Dev Phone

I have no idea why this is not well documented anywhere... I figured most of this out by scouring posts and through trial and error.

This process will create a build of the Android source tree with the Maps application included. Note that this does not have Gmail, YouTube, Market, or IM. These are third party applications to which source is not included. I tried to patch them in, but at the time of writing, their versions were too old to work with the new framework/build. As such, I would not say this is a really usable build by any means.

Installation prerequisites:

You must have a rooted (or a dev) phone.

Installation of the provided Android image files (see the end of the post for how to build them manually):

Install the JesusFreke's RC30 V1.3+. This will give you a root shell root access in recovery mode. A root adb shell is required for the next step.
Install the engineering boot loader.
Download the prebuilt cupcake installation images to your computer. Extract them to a folder somewhere. (Note: I will probably not be keeping this image up to date, so you are better of learning how to build from source. See the do-it-yourself instructions at the end.)
Get Fastboot for your computer.
Start your phone up into fastboot mode. You can do this by shutting your phone off, and holding camera and power. You will see Androids on skateboards.
Connect your phone to your computer. Press the back button on your phone until your screen says FASTBOOT.
Go to a terminal on your computer and navigate to your extracted (or built) files and type:
- chmod a+x fastboot
- (If you are building from source, make sure the built /out/host/platform/bin directory is in your PATH. Additionally, type "fastboot" rather than "./fastboot")
- ./fastboot flash system system.img
- ./fastboot flash boot boot.img
- ./fastboot flash userdata userdata.img
- ./fastboot reboot
- If your phone hangs for a long time while flashing or rebooting, unplug and reconnect your phone. I run into this issue, and that is how I work around it. This may be an issue with the phone being connected to a VM.

Congratulations! You are done!

Reverting Back to the G1/Dev Image

Reflash the modified RC30 image from your SD card. If you start up now, your phone will not start up (it will stay hung at the flashing robot). No big deal though.
Start up in recovery mode (turn on the phone using home and power). Press Alt-L to show the menu, and press Alt-W to wipe the phone.

Do it Yourself Instructions For Building from Source

Follow these instructions to get the Android source.
Follow these instructions to target Dream hardware and generate the images: http://source.android.com/documentat...ding-for-dream

At the time of writing, the Android Dream build was broken. I needed to do the following to make it work:

Several (relatively minor) changes in the Dream audio driver code to fix compilation issues.
Copied libOmxCore.so to mydroid/out/target/product/dream/system/lib (this was a missing step in the Building for Dream documentation, and something that should be in the HTC provided script)

Patching in Maps

Copy Maps.apk from a Android Dev Phone 1 into mydroid/out/target/product/dream/system/app
Copy com.google.android.maps.jar from a from a Android Dev Phone 1 into mydroid/out/target/product/dream/system/framework
Delete mydroid\out\target\product\dream\obj\PACKAGING\systemimage_unopt_intermediates\system.img
Delete mydroid\out\target\product\dream\system.img
make

Mono on Android

It's been a little quiet on my blog lately... too quiet. It's cause I've been bashing my head against a wall trying to get Mono on Android (well, Debian running side by side with Android really). So far, I've gotten the Mono runtime cross compiled for Android, but attempting to run anything results in the following:

localhost:~/Desktop/junk# mono
Usage is: mono [options] program [program-options]

Development:
    --aot                  Compiles the assembly to native code
    --debug[=<options>]    Enable debugging support, use --help-debug for details
    --profile[=profiler]   Runs in profiling mode with the specified profiler module
    --trace[=EXPR]         Enable tracing, use --help-trace for details
    --help-devel           Shows more options available to developers

Runtime:
    --config FILE          Loads FILE as the Mono config
    --verbose, -v          Increases the verbosity level
    --help, -h             Show usage information
    --version, -V          Show version information
    --runtime=VERSION      Use the VERSION runtime, instead of autodetecting
    --optimize=OPT         Turns on or off a specific optimization
                           Use --list-opt to get a list of optimizations
    --security[=mode]      Turns on the unsupported security manager (off by default)
                           mode is one of cas, core-clr, verifiable or validil

localhost:~/Desktop/junk# mono test.exe
Illegal instruction

Note that mono works fine, but attempting to run a program fails. So yeah, I'm at a loss. I'm guessing there is a bug in the ARM JIT somewhere. Or maybe my make process went horribly wrong due to user error somewhere. I'll try getting Mono from the trunk and building and installing. Apparently there are some more ARM related fixes in there.

I managed to get this far by:

Install Debian on Android.

Cross Compile Mono and its dependency chain using Scratchbox (there are a lot, which I'll detail in a later post). I also did some of the compilation on the G1 itself rather than Scratchbox.

Android Market Bug?

Several users of the Shell application are reporting that Shell is somehow reinstalling itself after uninstall: this includes people who do not have rooted G1s. I frankly have no idea how this is even possible, other than it being a bug in the Android Market application: Android Market is the only Android application that has permissions to install other applications. Suffice to say, I'm looking into the issue.

Fixing the "setuid su" security hole on Modified Android RC30

If you have a rooted Android phone, you are probably using a variation of JesusFreke's RC30 image. With JF's image, there are two ways you can get root:

For remote root, you can adb shell into the phone
For local root, run the root setuid su from Terminal Emulator or pTerminal.

The problem with the latter version is that any application can run su and get full unfettered access to everything on the phone. This leaves the door open to malicious applications. However, the answer is not simply removing the su file, as then there is no way to perform superuser tasks from within legitimate applications.

To that end, I wrote the Superuser application that fixes the security hole, and also allows you or any application to get root when properly authorized. I've also written a Shell application to demonstrate how an application can request authorization.

How Superuser sets itself up:

The standard RC30 install will have a setuid /system/bin/su. (if you deleted disabled it, reenable it for setup)
Install the Superuser Java application to your phone and run it.
Superuser will create a copy of su named superuser.
Superuser will chown superuser to user "root" and group app_gid (where app_gid is the group id of the Superuser application as determined by Android)
The superuser binary will also be chmod 4750 superuser, so that the Superuser Java application, and only that application, can execute it as root.
Finally, Superuser will chmod 4700 su to close the security hole.

How other applications get root access with the user's permissions:

Any application can fire an intent to request access to the locked down /system/bin/su.
When that happens, Superuser will catch the intent and ask the user if it should grant that application root permissions.
If allowed, Superuser will chown 0:app_gid /system/bin/su (where app_gid is the gid of the requesting application).
That application can then use /system/bin/su as normal.

Notes:

/system/bin/su will get reset to chown 0:0 after 10 seconds, so the requesting application must start the instance of su up within that grace period. This is a bit of a kludge, but I'm a Linux newbie and don't know a better/cleaner way to do it.
Since this is an unmodified version of su, 3rd party applications don't have to worry about piping in passwords and such.
This is using the standard Linux level permissions and restrictions, as well as the Android framework permissions present at the Java level.

Full source code to Superuser and Shell are at the bottom of this post.

So, here's what it the user experience looks like when running Shell and requesting root access:

First, we start up shell and try to run su. Note that running su is not allowed (as the uid of the process did not change). Shell needs to request permission to access su first:

This menu button fires the intent to request permission to su:

Superuser receives the request and asks the user if they wish to grant Shell superuser permissions:

Once granted, the user can properly execute su (as indicated by the root id in the prompt):

Superuser and Shell APK install files (will be on the market too soon).

Superuser and Shell Source Code.

Android Telnet Client

After starting telnetd on your G1, naturally the next thing you want to do is to telnet into it. Android did not have a telnet application, so I quickly whipped up a raw telnet client. Download it from the Android Market!

Here's a direct download to the Telnet.apk.

Root Access on the TMobile G1 and why it works

In my previous post, I received a comment that included a link explaining why this all works.

To translate: anything the user types on the keyboard is actually being fed into a console shell and potentially executed. So the easiest way to get root on your phone is to simply reboot it, and when it finishes rebooting, type "telnetd" and press enter. Your phone will start doing a contact lookup in your UI, but at the same time it is typing into a root shell.

This is basically a huge oops on Google's part! Looks like some debug related settings that they never took out.

So really, pTerminal is not necessary at all in this hack, but it it accidentally exposed it!

Note:

There is an easier way to do it: press enter (this runs/clears whatever may be pending on the console), type telnetd, press enter. Although this may not work if the console is blocked by another operation. That's why a reboot is recommended: to forcibly exit the pending process.

How get root access on the T-Mobile G1

Instructions:

Turn on your phone's WiFi. This gives your phone an IP you can reach it at freely.
Get to a command prompt on your device by using the PTerminal application from the Android Market. (adb shell does not seem to work with these instructions, telnetd does not start up)
cd system
cd bin
telnetd
netstat (get your phone's IP)
telnet into your phone from your computer

You now have root! I verified this by remounting the /system partition as read/write and changing files around:

mount -oremount,rw /dev/block/mtdblock3 /system

That remounts the /system directory as read/write instead of read only. This can only be done as root.

WARNING:

As I suspected, changes to the /system directory can not be reverted by a hard reset; the /system directory is actually a YAFFS2 file system (system.img) that is supposed to be read only, and thus is not part of the recovery image. Don't play with it unless you know what you are doing. I moved some files around in /system and broke my Browser, so I performed a hard reset, but the Browser was still broken after a hard reset. Luckily I had tracked my changes and was able to revert what I had done manually.

Notes:

Incidentally, in the /system/bin directory there is a flash_image executable that changes the recovery.img used when you hard reset the device. I've noticed that I can run this without root access from a standard adb shell. Maybe we never needed to root the device after all... I think we can flash it without root access... I'm too scared to mess with that beyond that (I don't want to brick my phone!).
There is no cp command, so you must use the dd command. To get a file from your computer to your phone, copy it to the SD card, and then use the root shell to copy it from the SD card to wherever you need.
I've noticed that you must start up telnetd very quickly upon starting pterminal. If you wait too long, telnetd will not stay resident; it exits immediately. It looks as if there is some timing and security related issue here. I've also written a quick telnet client so I can set telnet into root on my phone, from my phone. This works fine. However, I have not been able to successfully launch telnetd from my own application (using Runtime.exec). I'm not sure what pTerminal is doing differently here.
Kudos to SplasPood of Xda-developers for noticing this exploit.

My Brain Hurts