Instruction coverage for an Android app
Before continuing our app detailed analysis let’s wrap up our previous achievements. First, we created DebloatApp — a super small Android app for our website that weighs just 20 KB on Google Play. Then, we added push notifications, and what’s important, we demonstrated how much additional code a single Firebase dependency contributed to our app. Today, we dive deeper into the app internals.
Our app is still made from a single java class and a single activity layout. As we have seen, the app size increased dramatically when we enabled background notifications. Honestly, it’s complicated to figure out compiled app code. We just got used to trust our instruments to do the right thing.
DebloatApp is made of a single-class activity
The Debloat.App project is one of a few that allow to reduce the app code observation surface through highlighting actually executed code. We solely based our technology on ACVTool — an instruction coverage measurement tool for Android apps. This tool was initially developed and extensively tested at the University of Luxembourg. Some technical details can be found in the paper “Fine-grained Code Coverage Measurement in Automated Black-box Android Testing” by Pilgun et al. These days we maintain ACVTool within our DebloatApp project.
When working with ACVTool we explicitly use instruction coverage term instead of just code coverage because we focus on a decompiled representation of original bytecode. This is a human readable representation called smali. It directly matches to the final bytecode, allows us to read and edit actually executable instructions and smoothly repackage pretty much any Android app (beware of legal restrictions though).
MainActivity.java in smali representation. Executed code is highlighted with ACVTool
To begin with app testing we to follow this protocol:
prepare an app with ACVTool
run your end-to-end tests
generate instruction coverage report
In the end we obtain the instruction coverage report that shows coverage in a similar to JaCoCo way except that we see smali representation and its coverage over whole app codebase including 3rd-party libraries.
We creatively approached testing of DebloatApp combining various states of the app and device. In the end we could only achieve almost 8% instruction coverage.
Total app coverage
From the report we immediately see that our main activity class is fully covered. However, we also see additional R classes that were never run. Many other packages and classes are barely executed, too! In total 83% of classes did not run any code.
Coverage of main package classes
All of the above gives the idea how much we can optimise. Actually, R8 and ProGuard tools are great at shrinking and obfuscating apps. However, those tools under-approximate (leave too much) executable code because they rely only on static analysis techniques. There is still plenty of room for app size improvement.
Thus, we realised that 92% of app code was useless in our sample due to firebase/androidx dependency bloat. Another time we may take a look how much code left in already shrunk apps.