Abstract
Accurate environment maps are a key component to modelling real-world outdoor scenes. They enable captivating visual arts, immersive virtual reality and a wide range of scientific and engineering applications. To alleviate the burden of physical-capture, physically-simulation and volumetric rendering, sky-models have been proposed as fast, flexible, and cost-saving alternatives. In recent years, sky-models have been extended through deep learning to be more comprehensive and inclusive of cloud formations, but recent work has demonstrated these models fall short in faithfully recreating accurate and photorealistic natural skies. Particularly at higher resolutions, DNN sky-models struggle to accurately model the 14EV+ class-imbalanced solar region, resulting in poor visual quality and scenes illuminated with skewed light transmission, shadows and tones. In this work, we propose AllSky, an all-weather sky-model capable of learning the exposure range of Full Dynamic Range (FDR) physically captured outdoor imagery. Our model demonstrates conditional multi-exposure image environment map generation with intuitive user-positioning of solar and cloud formations, extending the current state-of-the-art to user-controlled texturing of atmospheric formations. Through our evaluation, we demonstrate AllSky is interchangeable with FDR physically captured outdoor imagery or parametric sky-models, and illuminates scenes with unprecedented accuracy, photorealism, lighting directionality (shadows), and tones in Image Based Lighting (IBL).
Related Work
This work is part of the broader Deep Sky initiative, which explores neural approaches to photorealistic sky modeling with user control over environmental parameters.
The Problem
Early sky-models modelled only luminance (grayscale intensity) for engineering and scientific applications 1,2. With the advent of the digital age, Nishita et al.3 proposed the first colour sky-model, enabling the generation of extraterrestrial views of the Earth for space flight simulators, and with it spurred a wide range of digital applications including Image-Based Lighting (IBL) techniques to render synthetic objects into real and virtual scenes.
With the emergence of deep learning, interest shifted to all-encompassing sky-models capable of generating weathered skies (e.g. sun, blue sky and clouds!). Early works included SkyNet4, CloudNet5, SkyGAN6, and LM-GAN7 which demonstrated the potential of deep learning to model complex sky appearances. However, we found these models struggled to generate the full dynamic range of outdoor scenes and cloud formations, leading to poor visual quality and scenes illuminated with skewed light transmission, shadows and tones. More specifically, we identify the following key limitations of current sky-models:
1 - Exposure Range
3D Intensity
3D surface of skydome illumination, coloured from \(\textcolor{blue}{low}\) to \(\textcolor{red}{high}\) intensity. Disk above 3D surface illustrates skydome segmentation. The sun (\(\textcolor{red}{red}\) spike) is a small subset of pixels whose intensities belittle the remainder of the skydome.
Exposure Range
Each column illustrates the impact of an incremental clipping of exposure with exposure equalization to the 15EV ground truth. Though the environment-maps are visually unaltered, renderings have altered tones, shadows and light transmission (\(\textcolor{blue}{sun transmission}\); black glass orb).
When looking at the Exposure Value (EV) of HDRI given \(EV=log_2(|I|_{max} - |I|_{min} + 1)\), where \(|I|\) is grayscale intensity, we found that HDR literature generally does not provide sufficient detail to determine supported exposure ranges. Literature focuses on exposure ranges \(\leq6EV\) when, as shown in Fig. 3D Intensity , real-world outdoor scenes are \(\geq14EV+\).
Why is the sun important? Why does exposure range matter?
Because the sun can represent over 60% of the sky’s illumination! As demonstrated by Fig. Exposure Range, incrementally clipping the EV of an HDRI while equalizing exposure to the FDR 15EV ground truth results in visually indiscernible alterations to the environment-maps (top row), but significant alteration to illumination in IBL scenes through softer tones, shadows, and light transmission (bottom row).
2 - High Variability
From an extraterrestrial point-of-view (POV), the sun is a \(0.5^{\circ}\) angular-diameter disk with near-constant illumination but, from a terrestrial POV, its size and radiant intensity are attenuated by a stochastic atmosphere. As illustrated in Fig. Variability , there is no intuitive relationship between solar-elevation, cloud cover, dynamic range and/or illumination. Similarly, there are 27 categories of cloud formations with variable textures and altitude-specific classification.
No FDR datasets with labelled cloud-formations exist and no input modality (before Icarus) has been demonstrated to condition solar radiant intensity and/or cloud textures. As a result, to DNN sky-models, skies are essentially nondeterministic systems with intractable variability. This has resulted in two things:
- Current DNN sky-models unconditionally generating skies with little diversity.
- Supervised losses (e.g. L1, LPIPS) guide per an invalid assumption of exactness in reconstruction, resulting in blurred cloud formations and diminished solar intensity.
3 - Overfitting and redundant input modalities
The conditioning of sky-models is challenging task given the high variability discussed in the previous section. Previous works have attempted to condition DNN sky-models on input modalities such as masks (e.g. for solar and cloud positioning), clear-sky augmentation, and/or textual descriptions. In our experimentation, we found that all of these input modalities were deeply flawed, with models overfitting to detailed masks (Fig. Crude Label, with performance significantly dropping when tested against Fig. ‘Hand-drawn’ Label), augmentation models completely ignoring clear-sky inputs, and textual conditioning being sporadic and imprecise to a laughable degree.
4 - Tonemapping is a flawed solution
Sky-models make heavy use of tone-mapping operators I = \(T_m(I')\) to compress HDRI to a displayable LDR range. Though blue sky and cloud formations can generally be displayed with ease, the sun can be \(10,000\times\) brighter and require aggressive compression to be displayed. We investigate a range of tone-mapping operators, including: Power-Law (\(T_\gamma\)), logarithmic (\(T_{log_n}\)), \(\mu\)-law (\(T_\mu\)), \(\mu\)-lawLog\(_2\) (\(T_{\mu \log_2}\)) and a few variations thereof. Each operator is a bijection (1-to-1 mapping) allowing for the recovery of the original image via \(I = {T_{m}}^{-1}(I')\). We express error (\(\delta\)) in LDR compressed space and error (\(\Delta\)) inHDR space with the relationship: \[\Delta I = | I - {T_m}^{-1}({T_m}(I)-\delta)|\]
As shown in Fig. Tonemapping, these operators introduce a non-linearity between error in LDR compressed space (left) and error in uncompressed HDR space (right). Most pronounced with sun, this means a small LDR error in the sun’s intensity results in a huge error in HDR space. Though this doesn’t really compromise visual quality, it has a big impact on applications such as rendering where it results in varying scene illumination, shadows, and light transmission.
5 - Resolution
So they images are small, who cares right? Wrong. This is a big problem, as exposure range and illumination are directly tied to the resolution. As shown in Fig. Resize Impact, interpolation results in a significant reduction in exposure range, with the sun’s intensity being reduced by a factor of 4 with every halving of resolution. As a directly result, we found most DNN sky-models simply collapse when trained at a higher resolution, and those that don’t produce blurry smeared skies.
As result, existing DNN Sky-Models work only implicitly…
Most HDR works omit daytime outdoor imagery in their results and, if included, demonstrates saturated clouds and solar features. Many that appear to work at low resolutions simply collapse at higher resolutions (LM-GAN 7) or loose all photorealism (SkyGAN 6). Despite advances in HDR image generation using DNNs, sky-models struggle to photorealistically reproduce the physical accuracy and illumination of real-world skies. These limitations hinder their effectiveness in downstream applications such as rendering, where accurate lighting is crucial.
So what’s the solution?
Through our model (AllSky), we demonstrate a state-of-the-art model and use it to study the paradigm of sky-modelling. Though we ultimately don’t propose solutions, we demonstrate the paradigm of sky-modelling is tangential to HDR literature and that a new approach is necessary to model the exposure range, variability and complexity of real-world outdoor scenes. We hope this work inspires the community to explore new approaches to sky-modelling, as we did with Icarus, that can overcome these limitations and enable more accurate and photorealistic representations of the sky for a wide range of applications.
Citation
@inproceedings{Maquignaz2026AllSky,
title={Towards Physically-Based Sky-Modeling For Image Based Lighting},
author={Maquignaz, Ian J.},
booktitle={Thirteenth International Conference on 3D Vision (3DV)},
year={2026},
url={https://openreview.net/forum?id=m6PGDcc5W0}
}
Poster
References
Parry Moon. Proposed standard solar-radiation curves for engineering use. Journal of the Franklin Institute, 230(5):583–617, 1940. ↩︎
R. Perez, R. Seals, and J. Michalsky. All-weather model for sky luminance distribution—preliminary configuration and validation. Solar Energy, 50(3):235–245, 1993. ↩︎
Tomoyuki Nishita, Takao Sirai, Katsumi Tadamura, and Eihachiro Nakamae. Display of the earth taking into account atmospheric scattering. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques, pages 175–182, 1993. ↩︎
Yannick Hold-Geoffroy, Akshaya Athawale, and Jean-François Lalonde. Deep sky modeling for single image outdoor lighting estimation. CVPR, 2019. ↩︎
Pinar Satilmis, Demetris Marnerides, Kurt Debattista, and Thomas Bashford-Rogers. Deep synthesis of cloud lighting. IEEE Computer Graphics and Applications, 2022. ↩︎
Mirbauer, M., Rittig, T., Iser, T., Křivánek, J. and Šikudová, E. (2024), SkyGAN: Realistic Cloud Imagery for Image-based Lighting. Computer Graphics Forum, 43: e14990. ↩︎ ↩︎
Lucas Valença, Ian J. Maquignaz, Hadi Moazen, Rishikesh Madan, Yannick Hold-Geoffroy, and Jean-François Lalonde. LM-GAN: A photorealistic all-weather parametric sky model, 2023 ↩︎ ↩︎