Impulse manifests: You can now build out meta-manifests for Impulse!
See ./tests/impulse_manifest/manifest.yaml for details.
You can use that manifest with uv run lpad add ./tests/impulse_manifest/manifest.yaml
Impulse is an easy-to-deploy all-in-one solution for extremely high-throuput embarrassingly parallel data extraction built for digitization, indexing, and distribution through IIIF manifests.
Running on Northwestern University's Quest supercomputing cluster, we observed hundreds of pages per hour. (This is while limiting ourselves to 50 concurrent jobs!) This could easily hit thousands of pages per hour.
This code is very easily extensible to new models/paradigms. Just edit the
impulse.tasks.DocumentExtraction class!
This project would not be possible without the work of other amazing open- source projects.