Repository / docs/EXPORT_HASHES.md
File hashes and export preflight
TgWitness / Documentation · Evidence and verification
Account Export supports download, hashes_only and exclude media actions with either regular API or Takeout. Use the existing export settings dialog, or send an accepted configuration to POST /api/account-export/start. chat_ids: [] selects all visible chats of the chosen types; provide IDs to restrict the scope.
media_types selects eligible types. media_action is their default action; media_actions overrides individual types. An optional media_rules[kind].action is used when there is no corresponding media_actions override. For example, {"photo":"download","video":"hashes_only"} downloads selected photos and records selected video hashes. Unchecked types stay excluded regardless of overrides. Profile photos remain separately opt-in. Selected additional account JSON is not message media and is still downloaded when include_other_data is true.
Animated MP4s use the animation rule; video stickers use the sticker rule, regardless of Telegram attribute order. Account Export calls round video messages video_message; the restricted capture configuration uses video_note for the same media category.
max_media_file_mib bounds file downloads. oversized_media: "hashes_only" requests hashes for larger eligible files instead. Hash collection does not download their bytes. Takeout's server-side file eligibility ceiling is increased when hashes are requested, while the local payload writer still enforces the chosen download limit. Unsupported or unavailable hashes do not cause a silent fallback to downloading the file.
max_total_gib bounds unique saved media bytes, not the final ZIP, report or hash JSON. Repeated references reuse the bounded attachment cache and do not consume this budget repeatedly. unique_media_bytes reports actual distinct stored file bytes; the legacy bytes_saved counter still sums message references so old archives remain verifiable. Cache eviction can require another request; identical saved content still occupies one file.
budget_exhausted: "hashes_only" (the default) switches remaining eligible downloads to server hashes when the media budget or packaging reserve would be exceeded. "stop_files" records the missing files instead. Hash acquisition failures remain explicit. Successfully downloaded files and complete hash manifests are signed in a private result cache before their message page is sealed, so a stop in the middle of a page does not discard completed attachments.
With messages_first: true (the default), all selected history and account metadata pages are saved and signed before file or hash requests start. Expiring Telegram file references stay in an encrypted private queue. Resume verifies the frozen pages and continues from them without rereading edited or deleted messages. Attachment processing seals at most ten enriched messages at a time to bound memory when hash manifests are large. A paused local job can either resume or Finish with saved data; finishing creates an archive and explicitly records unavailable history and attachments.
export_format accepts tgwitness, telegram_json, telegram_html and telegram_both. The compatible views add result.json and/or export_results.html with offline chat pages. Core chat/message fields follow the Telegram Desktop JSON schema; TgWitness renders them from its signed journal. Dates use UTC, messages are ordered by ID, and media paths refer to the same deduplicated files. tgwitness_media and hashes/index.jsonl retain hash-only outcomes. Signed JSONL remains authoritative, including source fields not mapped into the compatible view.
On this page
What the hashes mean#
Telegram's upload.getFileHashes supplies SHA-256 hashes for ranges. Each fileHash has an offset, length and hash. These are different from a file ID, access_hash, or a checksum of metadata.
The export records the document or exact photo size variant, expected length, ordered ranges and whether they cover the entire file. Telegram's final range may extend beyond EOF; comparison hashes only the bytes that exist. A different photo variant can legitimately have different bytes. No access hash or expiring file reference is included in the public manifest.
Hash requests use the media's data centre. Repeated representations reuse the result. DC migration and expired references are handled without switching the configured transport. Unavailable/partial coverage is recorded and makes the selected acquisition incomplete; it cannot be counted as a complete hash-only attachment.
Compare a downloaded file#
First verify the archive with an independently trusted signing-key fingerprint. hashes/index.jsonl maps chat/message/attachment references to individual JSON manifests in hashes/. The same manifests are bound into signed message pages; archive verification checks the index and standalone manifest against those pages. The HTML transcript links to each complete hash manifest.
Run:
python -m tgwitness.file_hashes path/to/hash-manifest.json path/to/local-file.jpg
The result distinguishes match, different, missing, different_size_or_representation and incomplete_hash_coverage. This command reads the local file, not Telegram. A match confirms that the checked local byte ranges agree with the recorded Telegram file hashes.
Check settings before starting#
Use Check settings and size in the existing dialog, or POST /api/account-export/preflight with the same options. ?sample=false checks configuration and local disk without reading Telegram.
The bounded sample reads at most 1,000 visible dialogs and the latest 25 messages from at most 12 matching chats, with a 20-second deadline. It does not start Takeout, download attachments, send messages, join chats or change export settings. Selection and media policy are applied to the sample. It reports known selected download bytes, duplicate references, unknown sizes and estimated hash-manifest size. Optional account metadata/avatars are not included in that estimate.
fit: "unknown" is intentional even when the sample fits: the sample measures recent messages, and the size of older history is measured during the export, particularly with date or sender filters. Hash JSON estimates assume 128 KiB ranges and 160 bytes per range plus per-file overhead, based on observed responses; actual ranges can differ. No fixed duration is promised because rate limits, approval, connectivity and report generation vary. Clearly excessive sampled bytes produce specific budget/disk warnings. Critically low free disk and an empty selection block starting.
Takeout approval is separate from QR/phone/2FA login. When Telegram requests approval, use an already authorized device to allow the export, then Check approval before the deadline if desired. The same export and authorization resume. Rate-limit waits are labelled separately.
Restricted GitHub capture and single-chat export#
The existing restricted capture form now exposes the same three media actions, per-type overrides/download limits, oversize behavior, avatar choice and messages-first processing. Its default UI choice is hashes only, with avatars off. It remains English by default. No new application page is added.
The restricted server accepts a selection configuration at POST /api/preflight or POST /api/collect. Preflight reads at most 25 recent messages from the selected chat in 20 seconds. It reports actual free disk, remaining capture time, known sampled payload bytes, estimated unique hash JSON and the separate delivery limits. It does not request hashes, download files or change the selection. Settings and chat become fixed when collection starts. The chosen policy is recorded in signed sealed.selection, observation and history records.
For full-account or selected-chat Account Export, use the same endpoints with the account selection configuration: mode: "account" and account_options. This mode supports regular API and Takeout, dates, file policies, optional account sections and compatible output formats. The form defaults to all accessible chats and hashes only. The event-journal mode remains a separate single-chat regular API choice. Account preflight uses the bounded multi-chat sample described above.
Takeout approval is separate from successful phone, QR and 2FA authentication. Review Telegram's request in an existing session; newly logged-in clients may require an older session or a wait. The UI displays Telegram's actual delay and Check export approval resumes the same job. If approval exceeds the runner's lifetime, finish with an honest partial result and start another run after approval; fresh-session GitHub captures do not transfer Telegram authorizations across runs.
The signed account-export.zip is included in the outer evidence archive and bound to a sealed.account_export journal event. Verification checks both archives, their signing identity and the binding. At the runner's deadline or storage limit, already signed history and completed attachments are finalized; unread history and unfinished attachments are recorded as missing before the fresh authorization is revoked.
With messages_first: true, both attachment workers wait for the history read to finish. A paused history keeps them waiting. Complete hash acquisitions produce media.hashed records and one hashes/<digest>.json per unique manifest. The HTML report labels them File hashes; content not downloaded, names the source message, byte coverage and range count, and links the JSON. Incomplete hashes produce failed attachment records. The archive verifier checks complete range coverage and binds each hash file to its signed journal record. The fresh authorization is revoked at finish; failure to confirm revocation remains explicit.
The separate single-chat exporter also honors the shared capture policy, including hashes and disabled avatars. Configure it with the capture object from the example through POST /api/capture, then use the existing chat export. Hash records are bound into signed chat.json; their standalone manifests are checked by verify_chat_export. The existing regular recording form does not yet expose all these new policy fields; use Account Export's form or the restricted form when configuring hashes interactively.
A single-chat export that fails or is cancelled keeps the messages it has read and the attachments saved so far in the workspace's partial-exports/<chat id>/, under a statement signed with the collector key (tgwitness-chat-partial-v1: data hash, message count, ID range, sessions and stop reason). While history is read, the statement is refreshed every 5,000 messages and once more when reading ends, before attachments are downloaded, so a process crash leaves at most the unsigned tail to read again. The next export of the same chat by the same account verifies the statement, reads messages newer than the saved range, carries the saved messages over and continues below the lowest saved number; acquisition.sessions and the limitations name every session. Saved data that fails verification or belongs to another account is moved aside unchanged and the export starts from the newest message; the first startup more than seven days later deletes the moved-aside copy. After a failed or cancelled export the panel shows the saved range with Resume export and Start over; Start over deletes the saved data and reads the chat again from the newest message. If Telegram ends the account's authorization during the export, the collector also builds a signed archive of the saved range, marked partial and naming the coverage.gap record of the loss; an Account Export finished with saved data names the same record in account_loss. A successful export removes the saved working data.
While history is read, the same export writes a live archive next to its saved messages (tgwitness-live-chat-export-v1). Its first member, live/export.json, is a signed start statement naming the export, chat, account and the hashes of README.txt, verify.py and the verification kit. Messages follow in pages of at most 2,000 records or 8 MiB, closed at the latest ten seconds after their first message: live/pages/NNNNNN.jsonl holds the records exactly as read, newest first, live/pages/NNNNNN.html shows them to a reader, and live/seals/NNNNNN.json signs both hashes, the counts and number range and the SHA-256 of the previous seal or of the start statement. After every page manifest.json and manifest.sig are rewritten at the end of the ZIP with the state (reading, complete or stopped and why), the totals and the head seal, and the file is synced, so the archive is a complete ZIP after each seal. Download messages read so far copies the file up to its last seal. A process stopped during a write is restored at the next start from a .sealed record kept beside the archive; given a damaged file, verify.py reads the members one by one, checks the seal chain up to the last intact seal and reports the bytes after it. Records in the live archive have no attachment fields yet; files, rechecks and reports belong to the finished export, which removes the live archive together with the other working data.
Attachments already saved for the same Telegram file (document or photo ID, size variant and byte size) are reused instead of downloaded again. Live recording writes media.saved with bytes_reused and downloaded_bytes: 0; chat exports mark the message with file_reused. Reused bytes are hashed again before use.
Limits and release status#
| Path | Scope / transport | Hash policy | Preflight | Messages before attachments |
|---|---|---|---|---|
| Account Export | All / selected chats; regular API or Takeout | UI and API | UI and API; bounded multi-chat sample | Signed history first by default; resumable encrypted attachment queue |
| Single-chat export | One chat; regular API | Shared capture API | No dedicated form | History first, then a separate files stage with up to three downloads at a time |
| Restricted GitHub Account Export | All / selected chats; fresh regular API or Takeout session | Existing form and API | Bounded multi-chat sample | Required; partial finalization at deadline or storage limit |
| Restricted GitHub event journal | One chat; fresh regular API session | Existing form and API | Selected-chat sample | Configurable; enabled by default in the form |
The sealed lifetime includes login and approval. The supervisor allows at most 120 minutes of capture, with a 220-minute workflow limit covering build, sealing, attestation, encryption, upload and cleanup. The container has a 2 GiB RAM limit. A single evidence ZIP attempt has a 4 GiB uncompressed ceiling. Larger results use the multipart path below. Free disk is measured at runtime; no fixed runner disk amount is assumed. Account acquisition reserves space for its archive and, on a disposable runner, its outer encrypted copy. This is a conservative reserve.
When formatted ZIP generation exceeds that limit or runs out of disk, the restricted collector removes the failed temporary ZIP and tries compact packaging once. It retains the full signed journal, coverage, every preserved file, server-hash manifests, checkpoints and verification material. It omits duplicated CSV, formatted JSON/HTML/PDF reports, indicators and viewer files. sealed.packaging records the reason; the signed manifest and PACKAGING.json identify the compact result, and index.html explains how to read and verify it. A compact ZIP uses the same encrypted handoff and unchanged archive verifier. Permission failures and unrelated I/O errors do not trigger this fallback.
If the compact result exceeds the single-ZIP ceiling or its temporary copy cannot fit, the collector streams the compact ZIP directly into encrypted parts, retaining the journal and every preserved file. Successful ZIPs larger than 128 MiB also use parts. Each encrypted file contains at most 128 MiB of plaintext. evidence.parts.json signs the ordered names, sizes, plaintext hashes, encrypted hashes and whole-archive hash; GitHub attests this manifest. delivery.tgwsealed contains the manifest and attestation, while delivery-000001.tgwsealed, etc. contain the archive parts. Only encrypted files are uploaded.
After opening delivery.tgwsealed and extracting delivery.zip, keep the encrypted parts beside evidence.parts.json and run trusted installed tools:
python -m tgwitness.sealed_parts evidence.parts.json --key sealed-key.json --out evidence.zip
python -m tgwitness.sealed_verify evidence.parts.json --bundle attestation.jsonl --repo OWNER/FORK --max-gib 64
Assembly rejects missing, reordered or changed parts and never overwrites an existing output. The verification ceiling is explicit; raise it only for an expected larger archive. The browser can decrypt individual numbered parts without loading the entire archive. If disk is exhausted even after the reserve and compact fallback, the run fails explicitly. There is no cross-run sealed authorization transfer or automatic continuation on another runner.
Account Export now paginates HTML chats. Each page has at most 250 message records and normally at most 512 KiB of source JSON; a single larger record remains intact. These bounds apply to the input records. The first page retains chats/ID.html, subsequent pages use chats/ID-2.html, etc. Previous/next links, continuous record numbers and exact page citations work offline. All pages are signed through the archive manifest; authoritative JSONL, receipts and acquisition options remain unchanged. Pagination improves opening large chats, but does not cap the total size of reports or paginate standalone sharing copies.
The compatible renderer is tested against synthetic UTF-16 text/entities, chat IDs, message ordering, media policies, partial results and signed archive verification. It has not been compared byte-for-byte against a live Telegram Desktop export. Historical acquisition checks matched live range hashes against previously downloaded Telegram media bytes of the recorded representations; file IDs alone never establish byte identity.
Updating code in a fork alone does not update the pinned collector. Releases require immutable source/workflow commits and a matching trusted-policy entry; do not weaken verification to accept arbitrary fork code. Existing archives remain verifiable; new archives include the updated standalone verifier.