The State of MUTF-8 in Python

CPython rejected adding a MUTF-8 parser to the standard library. This is unfortunate since there are many times you might want to use MUTF-8 in Python. You might be trying to:

• Read NBT, the file format use by Minecraft
• Read a JVM ClassFile
• Read a DEX file, the binary format used by Android
• Query SAP HANA (although this is really CESU-8)
• Any object saved with Java Object Serialization

This has resulted in some significant replication of work, with many projects re-implementing MUTF-8 parsing:

• androguard/core/mutf8.py
• Krakatau/classfileformat/mutf8.py
• javaobj/modifiedutf8.py
• py2jdbc/mutf8.py

So I've plucked the MUTF-8 decoder/encoder out of my Lawu project as the mutf8 package and added an optional C version. Hopefully, this version will not only be correct, but fast as well. The fallback Python version is a tad bit faster than others, and the C version is significantly faster than existing implementations:

Using our C version in the popular tool androguard reduced the total time to parse the Facebook.apk by 20%. Due to the sheer number of strings in a DEX or JVM ClassFile, any improvement in decoding can have a large impact.

With precompiled binaries for most platforms, and a pure-python fallback, handling MUTF-8 is now a simple pip install mutf8.