Project Deep Sky explores DNN approaches to generating photorealistic and physically accurate sky models for artistic, scientific and engineering communities.
Sky models are essential for:
Our work develops DNN approaches to capture the complexity of real-world skies—including sun positioning, atmospheric conditions, and exposure range—enabling artists, researchers, and engineers to control generation through an intuitive set of parameters and/or modalities.
Though conventional Low Dynamic Range (LDR) imagery is suitable for some applications, High Dynamic Range (HDR) imagery is integral to sky-models and the capture of outdoor scenes. HDR images capture a greater range of illumination and, in the particular case of outdoor lighting, can capture the estimated 22 f-stops of exposure necessary for the highlights and shadows of an average real-world outdoor scene. To distinguish the proportionality of scene exposure captured by an HDRI, we define the following:
Low Dynamic Range (LDR) Imagery: Display-referenced images with compressed dynamic range which can be clipped and displayed in 8-bit colour.
High Dynamic Range (HDR) Imagery: Scene-referenced measures of illumination with uncompressed dynamic range and precision greater than LDR 8-bit colour for later display as LDR. This includes imagery captured by conventional cameras in 12-bit RAW.
Extended Dynamic Range (EDR) Imagery: HDR images captured using techniques such as LDR-bracketing for greater exposure range than a singular image from a conventional camera.
Full Dynamic Range (FDR) Imagery / Physically-Captured Imagery: HDR images that fully-capture the exposure range of a reference scene without saturation of the exposure range.
We demonstrate the importance of FDR imagery for sky-modeling and the limitations of LDR and HDR paradigms through AllSky. Through Icarus, we demonstrate modelling of the Full Dynamic Range (FDR) of skies, generating photorealistic and physically accurate weathered skies and sun. Project Deep Sky targets interchangeability with physically-captured imagery, facilitating direct use in existing rendering (IBL) pipelines.
Stay tuned for updates on Helios, the next phase of Project Deep Sky research initiative exploring advanced neural approaches to sky modeling.
This work is part of the broader Deep Sky initiative, which explores neural approaches to photorealistic sky modeling with user control over environmental parameters.
[Read More][arXiv PrePrint]
![Scenes rendered per \(512^2\) environment maps. [Centre] Our DNN sky-model (Fig.\(\textcolor{red}{b}\), per \(\textcolor{red}{R}\textcolor{green}{G}\textcolor{blue}{B}\)-Icarus with Robertson fusion) recreates the illumination, tones and light transmission of real-world Full Dynamic Range imagery (\(\textit{FDR}\) ground truth in Fig.\(\textcolor{red}{a}\)). Icarus accurately models solar illumination for unprecedented lighting directionality (shadows, Fig.\(\textcolor{red}{c}\)). [Border] Icarus enables intuitive user-control over positioning and styling of solar and atmospheric formations.](/deep_sky/icarus/icarus_header.png)
Scenes rendered per \(512^2\) environment maps. [Centre] Our DNN sky-model (Fig.\(\textcolor{red}{b}\), per \(\textcolor{red}{R}\textcolor{green}{G}\textcolor{blue}{B}\)-Icarus with Robertson fusion) recreates the illumination, tones and light transmission of real-world Full Dynamic Range imagery (\(\textit{FDR}\) ground truth in Fig.\(\textcolor{red}{a}\)). Icarus accurately models solar illumination for unprecedented lighting directionality (shadows, Fig.\(\textcolor{red}{c}\)). [Border] Icarus enables intuitive user-control over positioning and styling of solar and atmospheric formations.
[Read More]
Accurate environment maps are a key component to modelling real-world outdoor scenes. They enable captivating visual arts, immersive virtual reality and a wide range of scientific and engineering applications. To alleviate the burden of physical-capture, physically-simulation and volumetric rendering, sky-models have been proposed as fast, flexible, and cost-saving alternatives. In recent years, sky-models have been extended through deep learning to be more comprehensive and inclusive of cloud formations, but recent work has demonstrated these models fall short in faithfully recreating accurate and photorealistic natural skies. Particularly at higher resolutions, DNN sky-models struggle to accurately model the 14EV+ class-imbalanced solar region, resulting in poor visual quality and scenes illuminated with skewed light transmission, shadows and tones. In this work, we propose AllSky, an all-weather sky-model capable of learning the exposure range of Full Dynamic Range (FDR) physically captured outdoor imagery. Our model demonstrates conditional multi-exposure image environment map generation with intuitive user-positioning of solar and cloud formations, extending the current state-of-the-art to user-controlled texturing of atmospheric formations. Through our evaluation, we demonstrate AllSky is interchangeable with FDR physically captured outdoor imagery or parametric sky-models, and illuminates scenes with unprecedented accuracy, photorealism, lighting directionality (shadows), and tones in Image Based Lighting (IBL).
[Read More]