Digital scans of analogue photographic film typically contain artefacts such as dust and scratches. Automated removal of these is an important part of preservation and dissemination of photographs of historical and cultural importance.
While state-of-the-art deep learning models have shown impressive results in general image inpainting and denoising, film artefact removal is an understudied problem. It has particularly challenging requirements, due to the complex nature of analogue damage, the high resolution of film scans, and potential ambiguities in the restoration. There are no publicly available high-quality datasets of real-world analogue film damage for training and evaluation, making quantitative studies impossible.
We address the lack of ground-truth data for evaluation by collecting a dataset of 4K damaged analogue film scans paired with manually-restored versions produced by a human expert, allowing quantitative evaluation of restoration performance. We construct a larger synthetic dataset of damaged images with paired clean versions using a statistical model of artefact shape and occurrence learnt from real, heavily-damaged images. We carefully validate the realism of the simulated damage via a human perceptual study, showing that even expert users find our synthetic damage indistinguishable from real. In addition, we demonstrate that training with our synthetically damaged dataset leads to improved artefact segmentation performance when compared to previously proposed synthetic analogue damage.
Finally, we use these datasets to train and analyse the performance of eight state-of-the-art image restoration methods on high-resolution scans. We compare both methods which directly perform the restoration task on scans with artefacts, and methods which require a damage mask to be provided for the inpainting of artefacts.
@article{ivanova23analogue,title={Simulating analogue film damage to analyse and improve artefact restoration on high-resolution scans},author={Ivanova, Daniela and Williamson, John and Henderson, Paul},journal={Computer Graphics Forum (Proceedings of Eurographics 2023)},year={2023},volume={42},number={2},doi={10.1111/cgf.14749},}
DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathology
Marco Aversa, Gabriel Nobis, Miriam Hägele, Kai Standvoss, Mihaela Chirica, Roderick Murray-Smith, Ahmed Alaa, Lukas Ruff, Daniela Ivanova, Wojciech Samek, and
3 more authors
Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2023
We present DiffInfinite, a hierarchical diffusion model that generates arbitrarily large histological images while preserving long-range correlation structural information. Our approach first generates synthetic segmentation masks, subsequently used as conditions for the high-fidelity generative diffusion process. The proposed sampling method can be scaled up to any desired image size while only requiring small patches for fast training. Moreover, it can be parallelized more efficiently than previous large-content generation methods while avoiding tiling artefacts. The training leverages classifier-free guidance to augment a small, sparsely annotated dataset with unlabelled data. Our method alleviates unique challenges in histopathological imaging practice: large-scale information, costly manual annotation, and protective data handling. The biological plausibility of DiffInfinite data is validated in a survey by ten experienced pathologists as well as a downstream segmentation task. Furthermore, the model scores strongly on anti-copying metrics which is beneficial for the protection of patient data.
@article{aversa2023diffinfinite,title={DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathology},author={Aversa, Marco and Nobis, Gabriel and Hägele, Miriam and Standvoss, Kai and Chirica, Mihaela and Murray-Smith, Roderick and Alaa, Ahmed and Ruff, Lukas and Ivanova, Daniela and Samek, Wojciech and Klauschen, Frederick and Sanguinetti, Bruno and Oala, Luis},journal={Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS)},year={2023},doi={10.48550/arXiv.2306.13384},}
2022
Perceptual Loss based Approach for Analogue Film Restoration
Analogue film restoration, both for still photographs and motion picture emulsions,
is a slow and laborious manual process. Artifacts such as dust and scratches are random in shape,
size, and location; additionally, the ovserall degree of damage varies between different frames.
We address this less popular case of image restoration by training a U-Net model with a modified perceptual loss function.
Along with the novel perceptual loss function used for training, we propose a more rigorous quantitative model evaluation
approach which measures the overall degree of improvement in perceptual quality over our test set.
@article{ivanova2022,author={Ivanova, Daniela and Siebert, Jan and Williamson, John},title={Perceptual Loss based Approach for Analogue Film Restoration},journal={Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications},year={2022},volume={4},pages={126-135},publisher={SciTePress},organization={INSTICC},doi={10.5220/0010829300003124},isbn={978-989-758-555-5},}
