Current Issue Cover

祁彬斌,朱学芳(南京大学信息管理学院多媒体信息处理研究所, 南京 210023)

摘 要
目的 针对当前文物资源由传统的实体文物向虚拟展示和数字文物进行扩展的趋势,如何提供一种多模态的信息呈现方式就显得尤为重要。通过将力触觉技术引入3维文物展示领域,提出一种基于多模感知的3维文物交互式呈现的算法框架。在对文物的基本特征进行视、听、触觉多通道分析的基础上,依据用户与文物模型的接触状态对多通道信息进行计算和整合。方法 在力触觉计算渲染方面,基于嵌入深度构建弹簧系统模拟轮廓形状的接触过程,引入动摩擦和静摩擦因数来反映表面摩擦力这一材质特征,通过法线贴图来实现文物表面纹理的触觉处理;针对交互的环境由2维平面拓展至立体空间,结合力触觉设备将操作时的行为和状态映射为虚拟环境中的操作代理,借助操作代理构建"旋转"和"选择-移动-释放"两种基本的操作范式来实现用户意图;最后,物理引擎的引入将物体的基本运动规律集成至虚拟场景,提升场景交互的真实感.结果 使用Phantom Omni手控器搭建面向馆藏文物的多模感知实验系统,抽取志愿者对实验系统进行测评。实验结果表明:运用本文方法,用户可从视觉、听觉、触觉多个通道对数字文物的整体和细节信息进行感知,且交互的整体过程简单、自然、有效。结论 本文提出的基于多模感知的数字文物交互式呈现方法,可有效实现对各类数字遗产特别是3维文物的多模重现,在保证较高实时性的同时拥有良好的可用性和情感体验效果。
Multi-mode interactive method for digital cultural relics based on haptics

Qi Binbin,Zhu Xuefang(Institute of Multimedia Information Processing School of Information Management, Nanjing University, Nanjing 210023, China)

Objective Cultural heritage is a powerful witness to the cultural inheritance and development of a country or region. In view of the current trend in which cultural relics extend from traditional physical relics to virtual exhibitions and digital relics, providing a multi-mode presentation method of digital cultural relics is necessary. By introducing computer haptics into the field of cultural relic exhibitions, we propose an algorithm framework for interactive presentation based on the multi-mode perception of 3D artifacts. The proposed framework is based on multi-channel analysis of tactual, auditory, and visual characteristics of cultural relics. Considering the contact states between users and these digital 3D models of relics, the multi-channel information is identified, calculated, integrated, and processed. Eventually, information is passed to the visual, audio, and haptic devices to complete the multi-mode perception. Method In this study, the impedance control mode is used as the basic driving mechanism of the haptic computing rendering. The basic idea is that a user inputs the position and direction information, and the force and torque are then calculated and fed back to the user. The calculation and rendering for different characteristics of these artifacts, such as contour shapes, physical materials, and surface textures, are also implemented in stages. For the contour shape of these artifacts, the spring system based on the embedding depth is constructed to simulate the contact process, and the stiffness coefficient of the spring is related with the cultural object. To reflect these material characteristics of the surface friction, the friction during the contact of cultural relics is simulated by introducing the dynamic and static friction factors. With the movement of the operation proxy on the surface of cultural relics, the finite state machine is alternately updated between static and dynamic friction. Although discrete triangular patch groups can identify the complex form, deviations remain because the existing models are usually represented by triangular patches. To address this problem, the normal mapping is used to perform the haptic processing of the surface texture. Compared with the traditional 2D interaction, 3D space provides more freedom and more abundant interactive tasks. At this time, through the comparison or abstraction of certain mechanisms existing in the real world, the behaviors and states of users' operation can be mapped to a proxy in a virtual environment. Through the operation proxy, we further build unified "rotation" and "select-move-release" interaction models to realize user intention. The "rotation" operation allows users to freely change their perspective according to their own needs, providing multi-view observation and appreciation of cultural relic models. During 3D interaction, the "select-move-release" operation helps users to freely manipulate the artifact model in a virtual environment. This method can also enhance the overall cognition of the artifact model. Finally, to further enhance the realistic characteristics of the virtual environment, ODE physics engine is introduced to cultural relic interaction. The reasonable introduction of physics engine integrates the basic motion law of objects into the virtual environment to enhance the realism of physical movement and scene interaction. Result With Phantom Omni haptic device as basis, multi-mode perceptual experimental systems for cultural artifacts are built, and 15 volunteers (eight males and seven females) are selected to evaluate the system. Experimental results show that the method can enable human operators to perceive the overall and detailed information of digital relics through multiple channels of vision, hearing, and touch. In addition, the entire interaction is simple, natural, and effective. On the basis of the user experience theory of Rolls-Royce, we evaluate the experiential effects from three dimensions, namely, usability, sensory experience, and emotional experience. Among these dimensions, usability and emotional experience scores are higher, whereas the sensory experience score is relatively lower. With regard to these set problems, users' acceptance and satisfaction degree are relatively high, satisfying the curiosity of people and the attraction of the presentation manner. However, users' ratings in realistic details of cultural models and the naturalness of interaction between the human and artifacts are relatively low. In general, the interactive mode proposed in this study is more natural than the mouse interaction, but it still maintains a certain distance from the human-object interaction under the real environment. Conclusion This study presents an interactive presentation method of digital artifacts based on multi-modal perception that can effectively achieve the multi-modal reproduction of various types of digital heritage, especially 3D artifacts. While ensuring high real-time performance, the method has good usability and emotional experience. In our future work, we will continue to explore the new surface haptic-rendering algorithm of the cultural relics. The method of haptic interaction for multi-point grasping of virtual artifacts needs to be further studied. Multi-point interaction can provide higher fidelity and richer operation experience than the existing interaction between single point and digital relics.