Takero Yoshida, Yuikazu Ito, Yoshihiro Fujiwara, Shinji Tsuchida, Daisuke Sugiyama, Daisuke Matsuoka
Japan Agency for Marine-Earth Science and Technology (JAMSTEC) has made available the JAMSTEC Earth Deep-sea Image (J-EDI), a deep-sea video and image archive (https://www.godac.jamstec.go.jp/jedi/e/index.html). This archive serves as a valuable resource for researchers and scholars interested in deep-sea imagery. The dataset comprises images and videos of deep-sea phenomena, predominantly of marine organisms, but also of the seafloor and physical processes. In this study, we propose J-EDI QA, a benchmark for understanding images of deep-sea organisms using a multimodal large language model (LLM). The benchmark is comprised of 100 images, accompanied by questions and answers with four options by JAMSTEC researchers for each image. The QA pairs are provided in Japanese, and the benchmark assesses the ability to understand deep-sea species in Japanese. In the evaluation presented in this paper, OpenAI o1 achieved a 50% correct response rate. This result indicates that even with the capabilities of state-of-the-art models as of December 2024, deep-sea species comprehension is not yet at an expert level. Further advances in deep-sea species-specific LLMs are therefore required.
Yuki Yasuda, Ryo Onishi, Yuichi Hirokawa, Dmitry Kolomenskiy, Daisuke Sugiyama
The present paper proposes a super-resolution (SR) model based on a convolutional neural network and applies it to the near-surface temperature in urban areas. The SR model incorporates a skip connection, a channel attention mechanism, and separated feature extractors for the inputs of temperature, building height, downward shortwave radiation, and horizontal velocity. We train the SR model with sets of low-resolution (LR) and high-resolution (HR) images from building-resolving large-eddy simulations (LESs) in a city, where the horizontal resolutions of LR and HR are 20 and 5 m, respectively. The generalization capability of the SR model is confirmed with LESs in another city. The estimated HR temperature fields are more accurate than those of the bicubic interpolation and image SR model that takes only the temperature as its input. Except for the temperature input, the building height is the most important to reconstruct the HR temperature and enables the SR model to reduce errors in temperature near building boundaries. The SR model considers the appropriate boundary for each building from its height information. The analysis of attention weights indicates that the importance of the building height increases as the downward shortwave radiation becomes larger. The contrast between sun and shade is strengthened with the increase in solar radiation, which may affect the temperature distribution. The short inference time suggests the potential of the proposed SR model to facilitate a real-time HR prediction in metropolitan areas by combining it with an LR building-resolving LES model.
Ryo Onishi, Daisuke Sugiyama, Keigo Matsuda
We propose a super-resolution (SR) simulation system that consists of a physics-based meteorological simulation and an SR method based on a deep convolutional neural network (CNN). The CNN is trained using pairs of high-resolution (HR) and low-resolution (LR) images created from meteorological simulation results for different resolutions so that it can map LR simulation images to HR ones. The proposed SR simulation system, which performs LR simulations, can provide HR prediction results in much shorter operating cycles than those required for corresponding HR simulation prediction system. We apply the SR simulation system to urban micrometeorology, which is strongly affected by buildings and human activity. Urban micrometeorology simulations that need to resolve urban buildings are computationally costly and thus cannot be used for operational real-time predictions even when run on supercomputers. We performed HR micrometeorology simulations on a supercomputer to obtain datasets for training the CNN in the SR method. It is shown that the proposed SR method can be used with a spatial scaling factor of 4 and that it outperforms conventional interpolation methods by a large margin. It is also shown that the proposed SR simulation system has the potential to be used for operational urban micrometeorology predictions.