3D modeling from stereo pairs and multiple tilt series IVEM micrographs of thick specimens is one of our major developments. Pairs of stereo images are displayed and viewed stereoscopically to an observer wearing liquid crystal glasses synchronized with the display. Our current programs allow for reconstruction, in 3D, of branched networks, such as are commonly found in the preparations of several of our users. Examples are the T-system of skeletal muscle, and the fibrin networks in fibrin clots. We currently have two versions of these programs. In one version, the manual version, the cell biologist traces in 3D structures of interest in the images. In the second version, the reconstructions are created entirely automatically by the computer program, with some editing possible at the end, when necessary. In the manual version of these programs, the user locates nodes in the network as a first step. Then he/she indicates which pairs of nodes are connected by a link. The computer tabulates the 3D coordinates of the nodes, and builds a """"""""link list"""""""" of connections between nodes. These data are then used to display the reconstructed network in 3D. Initially, the reconstruction is created by the operator using a single stereo pair of images, usually the pair centered on 0 degrees of tilt, i.e. looking along the optic axis of the microscope, perpendicular to the plane of the specimen. We know, on theoretical grounds and from experience, that the accuracy of positioning elements of the reconstruction in relation to the image of the specimen is accurate to a single pixel in the plane of the images (X and Y), but can be several times as bad along the viewing direction (Z). A similar anisotropy of resolution appears in tomography, when only a limited range of viewing angles is incorporated into the reconstruction calculation. In our case with a single stereo pair, the range of tilt angles as very limited, being just two images and the tilt between these being limited by our power to comprehend stereo visually. However, with the latest version of our programs, we can incorporate many stereo views of the same objects, taken from a wide range of viewing angles (using our high tilt holder) into the reconstruction process and thereby dramatically improve the accuracy of the reconstruction along the original viewing direction. Stated in simple terms, we have stereo pairs of the object taken """"""""from the side"""""""" not just along the original viewing direction. When we look at these side views, along with the original reconstruction made looking along the optic axis, we clearly see the errors referred to above, and our programs allow us to correct the reconstruction by moving its elements until they coincide properly with the structure as seen in this side view. After a series of several such rotations and corrections, a reconstruction with improved accuracy in the third dimension is achieved. In collaboration with Stanley Dunn and Junqing Huang, a graduate student, we are automating this process. Initially, Ms. Huang used images of Golgi stained skeletal muscle, where the transverse tubular network is densely stained relative to the rest of the muscle cells. Intensity thresholding was used to segment the stained tubular network from background, and skeletonization applied to the network, with use of various algorithms to complete missing parts of the network. The resulting skeletal network then was searched, identifying nodes as branch points in the skeletonized network. The most difficult part of the process of completely automating the reconstruction from stereo pairs process is establishing correspondence between identical nodes in the two images. Ms. Huang has explored two ways of doing this, first finding that a process of geometric hashing works well. Typically more than 90% of the nodes in Golgi images are found correctly in both images and the correspondence established. A second process, sieve processing, which more closely mimics the way a human solves this problem, is having even better success in early trials. The remaining few percent of unidentified or incorrectly identified correspondences can be edited by hand. Overall this is a very large saving in time and effort compared to the completely manual approach. The future of this development will be greater ease of reconstruction, and automation of the alignment process for structures other than networks using knowledge-driven feature recognition by the computer.
Showing the most recent 10 out of 38 publications
