Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# AGENTS

This repository contains ToCode, a Python-only binary exporter. ToCode takes one binary or IDA database path and writes one source-like project directory for reverse-engineering agents.
This repository contains ToCode, a Python-only binary exporter. ToCode takes one binary, IDA database, or Android APK path and writes one source-like project directory for reverse-engineering agents.

## Scope

Expand All @@ -15,7 +15,8 @@ This repository contains ToCode, a Python-only binary exporter. ToCode takes one
- `src/tocode/cli.py`: command-line entry point for `tocode`.
- `src/tocode/__init__.py`: package version and `export_from_binaryview()`, the in-UI Binary Ninja library entry.
- `src/tocode/analysis.py`: backend-neutral binary inventory and call graph normalization.
- `src/tocode/backends/`: IDA Domain, radare2, angr, and Binary Ninja (`binja.py`) session adapters.
- `src/tocode/backends/`: IDA Domain, radare2, angr, and Binary Ninja (`binja.py`) session adapters, plus the ASC/droidasc APK backend (`asc.py`: APK set discovery, DEX inventory, worker-side class decompilation).
- `src/tocode/apk.py`: APK export pipeline (extraction, decompile pool, resource decoding, metadata); `apk_metadata.py`: manifest parsing and Android JSON documents; `apk_native.py`: extraction and background native export of every `.so`.
- `src/tocode/exporter.py`: project writer, function rendering, worker-session rendering, generated export `AGENTS.md`.
- `src/tocode/metadata.py`: JSON metadata and triage documents.
- `src/tocode/cluster.py`: call-graph clustering.
Expand Down Expand Up @@ -59,6 +60,17 @@ When `--tree` is passed, the export also contains:
- `src/tree/**/*.c`
- `function-index-tree.json`

APK input (`.apk`, `.apks`, `.xapk`) uses the ASC backend and writes instead:

- `src/raw/<package>/**/*.java` (one file per class, folders are Java packages; no clusters, no summaries)
- `AndroidManifest.xml`, `manifest.json`, `classes.json`, `package-graph.json`, `native-libs.json`
- `functions.json`, `function-index.json`, `strings.json`, `imports.json`, `exports.json`, `sections.json`, `reachable.json`, `triage.json`, `project.json`, `export-manifest.json` (same shapes as the native export where applicable)
- `lib/<abi>/*.so` (every native library, always extracted) and `native/<abi>/<lib>/` (a full nested ToCode export per library, all ABIs, unless `--no-native`)
- `data/apk/**` (all other entries verbatim), `data/res/**/*.xml`, `data/resources.json`
- `tocode.log`, generated `AGENTS.md` and `CLAUDE.md`

`base.apk` merges sibling `split_*.apk` files unless `--no-splits`; bundles are unpacked and merged. `--backend` selects the native backend for the `.so` exports (`binja` is rejected).

## Development

- Prefer `uv` for local commands.
Expand All @@ -74,6 +86,8 @@ When `--tree` is passed, the export also contains:
- IDA Domain is the preferred backend when available.
- radare2/r2pipe is a fallback backend.
- angr is the optional pure-Python fallback backend (`[angr]` extra).
- ASC (`droidasc`, PyPI) is the APK/DEX backend and a core runtime dependency. It pulls androguard. `backends/asc.py` is the only module that imports it; it pre-imports the modules ASC would otherwise stub in `sys.modules` and silences androguard's loguru logging. Class decompilation is not thread-safe, so it runs in spawned worker processes (recycled in rounds; do not use `max_tasks_per_child`, it deadlocks spawn pools on some CPython builds). The big per-method/per-class/per-string JSON documents are streamed row by row (`apk_metadata.write_json_rows`), never built as one object.
- APK native libraries are exported by `apk_native.py` on a background thread, each `export_binary` in its own spawned process so a backend OOM-kill or crash only fails that library (`native-libs.json` status). The thread waits for `TOCODE_APK_NATIVE_MIN_FREE_MB` (default 1024) of free memory before starting each library so it does not starve the DEX pool. `TOCODE_WORKER_TMP_DIR` also places the unpacked `.apks` bundle.
- Binary Ninja is an opt-in backend (`--backend binja`, never auto-selected). The
`binaryninja` module is supplied by the Binary Ninja install (in-UI) or the
remote VM, so it is not a pip dependency. `rpyc`, the client used for the
Expand Down
44 changes: 40 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ToCode

ToCode exports a binary or IDA database into a source-like project tree: raw recovered C, matching assembly, function summaries, section data, optional IDA database, and metadata that coding agents can read directly.
ToCode exports a binary, IDA database, or Android APK into a source-like project tree: raw recovered C (or Java for APKs), matching assembly, function summaries, section data, optional IDA database, and metadata that coding agents can read directly.

## Why

Expand All @@ -23,11 +23,22 @@ sample_decompiler/
src/raw/**/*.c
src/raw/**/*.asm
src/raw/**/*.summary
src/raw/<package>/**/*.java # Only for APK files (replaces the .c/.asm/.summary tree)
include/*.h
include/*.types.h
data/*.bin
data/variables.json
data/variables_interesting.json
data/apk/<apk>/** # Only for APK files: every non-code APK entry, verbatim
data/res/<apk>/**/*.xml # Only for APK files: binary XML resources decoded to text
data/resources.json # Only for APK files: decoded resources.arsc
lib/<abi>/*.so # Only for APK files: extracted native libraries
native/<abi>/<lib>/ # Only for APK files: full nested ToCode export per .so
AndroidManifest.xml # Only for APK files
manifest.json # Only for APK files
classes.json # Only for APK files
package-graph.json # Only for APK files (replaces cluster-graph.json)
native-libs.json # Only for APK files
function-index.json
functions.json
types.json
Expand All @@ -47,11 +58,19 @@ sample_decompiler/
| Path | Description |
| --- | --- |
| `src/raw` | Decompiled C-like output, assembly, and summaries. Grouped by call-graph cluster, or by the original source file/directory when the binary has debug info (DWARF). |
| `include` | Generated headers, including `*.types.h` with the structs/enums/typedefs recovered from the binary. |
| `src/raw/<package>/**/*.java` | **Only for APK files.** Decompiled Java (ASC + androguard DAD), one file per class, folders follow Java packages; inner classes are `Outer$Inner.java`. No clustering, no `.summary` files. |
| `include` | Generated headers, including `*.types.h` with the structs/enums/typedefs recovered from the binary. Not written for APK files. |
| `data` | Raw section dumps and variable metadata. |
| `data/apk`, `data/res`, `data/resources.json` | **Only for APK files.** Every non-code entry of each APK in the set verbatim, `res/**/*.xml` decoded from binary XML, and the decoded `resources.arsc` tables. |
| `lib/<abi>/*.so` | **Only for APK files.** Every native library found in the APK set (all ABIs), always extracted. |
| `native/<abi>/<lib>/` | **Only for APK files.** A complete nested ToCode export for each native library (own `AGENTS.md` with an Origin section naming the APK, `src/raw/*.c`, `functions.json`, `exports.json`, ...). Skipped with `--no-native`. |
| `AndroidManifest.xml` / `manifest.json` | **Only for APK files.** Decoded manifest and its parsed form: package, versions, SDKs, permissions, components with intent filters and exported state, application attributes, split manifests. |
| `classes.json` | **Only for APK files.** Every class with superclass, interfaces, access flags, fields, methods, source file, and Java file/line ranges. |
| `package-graph.json` | **Only for APK files.** Inter-package call graph (the APK counterpart of `cluster-graph.json`). |
| `native-libs.json` | **Only for APK files.** ABI, hash, source APK, export directory, and decompilation status of every native library. |
| `types.json` | Catalog of types recovered from the binary's debug info or type library, with C declarations. |
| `*.json` | Functions (with recovered types and original source decl file/line), sections, strings, imports, exports, relocations, reachability, clusters, triage, project metadata, and export manifest. |
| `tocode.log` | Export log with checkpoint, resume, and per-function render history. |
| `*.json` | Functions (with recovered types and original source decl file/line), sections, strings, imports, exports, relocations, reachability, clusters, triage, project metadata, and export manifest. For APK files the same documents describe DEX methods, strings, framework imports, exported components/JNI methods, and reachability from manifest components; `relocations.json`, `cluster-graph.json`, and `types.json` are not written. |
| `tocode.log` | Export log with checkpoint, resume, and per-function render history. For APK files it also carries the native library export status. |
| `AGENTS.md` / `CLAUDE.md` | Instructions for agents analyzing the exported binary. |
| `src/tree` | Optional scanner-friendly C output when `--tree` is used. |

Expand All @@ -76,6 +95,23 @@ Three backends are supported, selected with `--backend` (default `auto`, which p

Other disassemblers may be added in the future.

### Android APKs

`tocode app.apk` (also `.apks`/`.xapk` bundles) uses [ASC](https://github.com/MG1937/ASC) (`droidasc`, a core dependency) for the DEX side and the regular native backends for every shared object in the package:

- `src/raw/<package>/<Class>.java`: one decompiled Java file per class, folders follow Java packages (no clustering, no summaries).
- `AndroidManifest.xml` + `manifest.json`: decoded and parsed manifest (permissions, components with intent filters and exported state, application attributes, split manifests).
- `classes.json`, `functions.json`, `function-index.json`, `strings.json`, `imports.json`, `exports.json` (exported components + JNI methods), `reachable.json` (from manifest components), `package-graph.json`, `sections.json`, `triage.json`.
- `lib/<abi>/*.so`: every native library extracted; `native/<abi>/<lib>/`: a complete nested ToCode export per library (all ABIs), produced on a background thread while the DEX side decompiles. `native-libs.json` records the status of each (each library runs in its own process; one failing or being OOM-killed never fails the APK export, and the native thread waits for `TOCODE_APK_NATIVE_MIN_FREE_MB`, default 1024 MB, of free memory before each library). Pass `--no-native` to skip the native decompilation (libraries are still extracted). `--backend` picks the native backend (`auto`/`ida`/`r2`/`angr`; `binja` is not supported for APKs).
- `data/apk/**`: every other APK entry verbatim; `data/res/**/*.xml` and `data/resources.json`: decoded binary XML and `resources.arsc`.

`base.apk` automatically merges sibling `split_*.apk` files (config and ABI splits) into the same project; `--no-splits` exports it alone. The default output directory is `<manifest package>_decompiler`.

```bash
tocode base.apk # DEX + all splits + native libs (IDA/r2/angr)
tocode app.apks --no-native -j 4 # bundle, DEX/Android side only
```


### Using

Expand Down
6 changes: 6 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ description = "Export compiled binaries as source-like projects for reverse engi
readme = "README.md"
requires-python = ">=3.10"
dependencies = [
"droidasc==0.1.1.post1",
"ida-domain==0.5.0",
"idapro==0.0.9",
"r2pipe==1.9.8",
Expand Down Expand Up @@ -59,6 +60,11 @@ ignore_missing_imports = true
module = ["binaryninja.*"]
ignore_missing_imports = true

# droidasc (ASC, the APK backend) and androguard ship no type stubs.
[[tool.mypy.overrides]]
module = ["droidasc.*", "androguard.*"]
ignore_missing_imports = true

[tool.setuptools]
package-dir = { "" = "src" }

Expand Down
Loading
Loading