warc2zip examples

Small, self-contained WARC files for trying out and testing warc2zip.

Files

The file names follow Common Crawl's repackage convention, <FLAVOUR>-REPACKAGE-<CRAWL-ID>.warc.gz, the same scheme used for e.g. EOT-2024-REPACKAGE-CC-MAIN-2024-42-GOV-000000-001.warc.gz in the End of Term Archive.

Using them with warc2zip

Installation, usage and ready-made commands for each of these files are documented in the warc2zip repository — see in particular WARC examples for testing. warc2zip can read the files straight from the bucket URLs above; no download is required.

The files are ordinary .warc.gz files and also work with any other WARC tooling.

File details

500_RECORDS-REPACKAGE-CC-MAIN-2026-30.warc.gz

The first 500 captures of a regular Common Crawl segment WARC (CC-MAIN-20260618163205-20260618193205-00999.warc.gz), kept exactly as the crawler wrote them: each capture consists of a request, a response and a metadata record, preceded by the original warcinfo record. This is the one to use for exercising request and metadata record handling.

Record counts: 1 warcinfo, 501 request, 500 response, 500 metadata.

warcinfo:

isPartOf: CC-MAIN-2026-25
publisher: Common Crawl
description: Wide crawl of the web for June 2026
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-233
software: Apache Nutch 1.21 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.7-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/

Note that the retained warcinfo record reports isPartOf: CC-MAIN-2026-25, the crawl the source segment belongs to.

HOMEPAGES-REPACKAGE-CC-MAIN-2026-21.warc.gz

Homepage captures (site roots such as https://example.com/, occasionally /index.html and the like) extracted from CC-MAIN-2026-21 with cdx_toolkit. Only response records are included; the matching request and metadata records are not.

Record counts: 1 warcinfo, 35,700 response.

warcinfo:

software: pypi_cdx_toolkit/0.9.40.dev89+g53a7ef76c
isPartOf: CC-MAIN-2026-21
description: Repackage of CC-MAIN-2026-21 containing only response records of homepages
format: WARC file version 1.0
creator: Common Crawl Foundation <https://commoncrawl.org>
operator: Malte Ostendorff <mailto:malte@commoncrawl.org>

IS_US_FEDERAL-REPACKAGE-CC-MAIN-2025-13.warc.gz

Captures from hosts classified as US federal government, extracted from CC-MAIN-2025-13 with cdx_toolkit as part of the End of Term Archive effort. Only response records are included.

Record counts: 1 warcinfo, 35,700 response.

warcinfo:

software: pypi_cdx_toolkit/0.9.40.dev91+ga04800ea0
isPartOf: CC-MAIN-2025-13
description: Repackage of CC-MAIN-2025-13 containing only response records of US federal government hosts
format: WARC file version 1.0
creator: Common Crawl Foundation <https://commoncrawl.org>
operator: Malte Ostendorff <mailto:malte@commoncrawl.org>

Things to know

• WARC-Filename still carries the original file names. The two cdx_toolkit repackages were renamed after they were written, so their warcinfo records say WARC-Filename: homepages_CC-MAIN-2026-21.warc.gz and WARC-Filename: is_us_federal_CC-MAIN-2025-13.warc.gz; the 500-record file names its source segment, CC-MAIN-20260618163205-20260618193205-00999.warc.gz. Tools that derive names from WARC-Filename will use those values.
• The content is a redistribution of Common Crawl data and is subject to the Common Crawl Terms of Use.