A principal endeavor of computer graphics research has been the pursuit of photorealism, The principal reason that image-based modeling and rendering is interesting is that these representations do not need to be as complete as traditional computer graphics models in order to transcend many of the limitations of photographs. This thesis restricts itself on the topic of image-based rendering techniques and physically based modeling. Firstly, a novel method of depth image based rendering is proposed by applying texture mapping onto some planes in virtual environment, called virtual planes. In this way, the viewpoint can be either static or moving around, including crossing the plane of the source images, which excel in the viewing range. In addition, both hardware acceleration can be also incorporated to increase the power of conventional texture mapping in image based rendering. In Chapter 3, we present a representation of image-based models by obtaining textured planes reconstructed from calibrated depth images. These textured planes are textured and partially transparent planes into which the input images are mapped using perspective projection. The Hough transform is used to find the position of these textured planes, and optic flow measures are used to determine their textures. Differing from IBMR working well on the static image synthesis, physically-based modeling and simulation attempts to map a natural phenomena to a computer simulation program. In this research topic, the sketch-based design is one of the most challenging and active problems. In in Chapter 4, we present an interactive modelling system for generating free-form surfaces using a 2D sketch interface. Our system could identify the 3D surfaces automatically, and then we apply Delaunay triangulation on these surfaces. Finally, the shape of the triangular surface mesh that follows the 3D profile curves is computed using harmonic interpolation by solving Laplacian equations. We present experimental results on various kinds of drawings by the interactive modeler. In addition, we can use the physically based method to simulate the natural phenomenon. In chapter 5, we present a method for the efficient simulation of a large number of autumn leaves, realistically showing their weathering process in autumn time, with various deformed shapes and texture patterns. In modeling various shapes of leaves for weathering, we introduce a botanically and physically-based method for simulating variations of leaf shape by using double layered model (DLM). The double layers, defined as mass springs, affect each other and determine together the deformed shape of leaves.