scale-img2dataset

This repository provisions and configures Hetzner Cloud downloader fleets for large img2dataset runs. The current primary path is the PDF Ray cluster inventory, which is also used for the CC full-image-URL Ray/S3 downloader flow.

The current PDF Ray deployment:

• attaches an existing Hetzner cpx42 server as the Ray head
• creates 60 Hetzner cpx32 Ray workers in hel1
• disables public IPv6 on hosts and prefers IPv4 locally
• installs a local Knot Resolver cache with one resolver instance per host
• raises file descriptor and TCP limits for high downloader concurrency
• installs ray[default]==2.56.0, img2dataset==1.47.0, pyarrow, boto3, and Hugging Face transfer packages
• uploads downloader output to S3 bucket scale-img2dataset-temp-96639723 under runs/<run-id>/
• exposes Prometheus, node exporter, Ray metrics, and a provisioned Grafana dashboard for cluster monitoring

Local Environment

Put credentials at the repository root:

./hz_token
./hf_token
./wandb_token
./scale-img2dataset_accessKeys.csv

Enter the toolchain shell:

nix develop path:$PWD
scripts/bootstrap-python

The devShell exports HCLOUD_TOKEN from ./hz_token, HF_TOKEN from ./hf_token, and WANDB_API_KEY from ./wandb_token when those files exist.

Provision the Ray Cluster

Review infra/pdf-ray/terraform.tfvars.example, then apply:

cd infra/pdf-ray
terraform init
terraform apply

Important current Terraform defaults:

head_existing_server_name=img2dataset-bench-cpx42-20260714
worker_prefix=cc-full-worker
worker_count=60
initial_worker_count=60
worker_server_type=cpx32
allowed_grafana_cidrs=[]

Set allowed_grafana_cidrs to the CIDR ranges that may reach Grafana on TCP port 3000. Terraform writes the Ansible inventory to infra/ansible/pdf-ray-inventory.ini.

Configure Hosts

Apply the main host configuration:

cd ../ansible
ansible-playbook -i pdf-ray-inventory.ini playbook.yml

Apply monitoring:

ansible-playbook -i pdf-ray-inventory.ini monitoring.yml

Monitoring defaults:

node_exporter_port=9100
ray_metrics_export_port=8080
prometheus_port=9090
grafana_port=3000
prometheus_retention_time=14d
img2dataset_stats_interval_seconds=15
img2dataset_stats_window_seconds=180

Grafana admin credentials are generated on the head and stored at /etc/grafana/admin_password.

Run the Ray img2dataset Flow

The active Ray img2dataset settings live in infra/ansible/group_vars/pdf_ray_cluster/vars.yml and infra/ansible/group_vars/pdf_ray_workers/vars.yml.

Current run defaults:

img2dataset_run_id=cc-full-image-urls-ray-s3-20260723T095907Z
laion_metadata_repo=xinyangli/cc-full-image-urls
laion_metadata_mode=hf_text_gz_batches
img2dataset_total_rows=189232647623
img2dataset_input_format=txt.gz
img2dataset_output_format=webdataset
img2dataset_url_col=url
img2dataset_caption_col=
img2dataset_processes_count=4
img2dataset_thread_count=88
img2dataset_number_sample_per_shard=10000
img2dataset_timeout=2
img2dataset_retries=0
img2dataset_normalize_urls=true
img2dataset_max_download_bytes=67108864

Ray scheduling defaults:

img2dataset_ray_enabled=true
img2dataset_ray_max_in_flight=<number of pdf_ray_workers>
img2dataset_ray_task_cpus=<worker vCPU count>
img2dataset_ray_task_retries=1
img2dataset_ray_task_timeout_seconds=1800
img2dataset_ray_urls_per_task=250000
img2dataset_ray_metadata_chunks_per_fill=64
img2dataset_ray_metadata_max_in_flight=4
img2dataset_ray_skip_s3_completed=true
img2dataset_ray_cleanup_after_upload=true
img2dataset_ray_enable_wandb=false

Start the Ray head and workers through the configured systemd units from the playbook, then start the named Ray run on the head. Logs are available through the corresponding img2dataset-ray-* systemd units and Ray dashboard/metrics.

S3 Output

S3 upload is enabled by default. Current upload settings:

s3_output_bucket=scale-img2dataset-temp-96639723
s3_output_region=us-east-1
s3_output_prefix=runs
s3_output_file_workers=1
s3_output_transfer_workers=4
s3_output_upload_batch_files=256
s3_output_upload_batch_bytes=214748364800
s3_output_stable_seconds=60
s3_output_delete_after_upload=true

The Ray drain writes completion markers under runs/<run-id>/completion-markers/ and supports child task markers for bundled worker results.

Legacy Single-Worker Flow

The older single-worker Hugging Face dataset flow still exists under infra/terraform and the non-Ray Ansible groups. Use it only when you intentionally want that deployment shape; the README defaults above describe the current Ray/S3 cluster path.