IEEE Rev
Biomed Eng. Author manuscript; available in PMC 2011 Jul 7. Published in final edited form as: PMCID: PMC3131209 NIHMSID: NIHMS301360 Many important eye diseases as well as systemic diseases manifest themselves in the retina. While a number of other anatomical structures contribute to the process of vision,
this review focuses on retinal imaging and image analysis. Following a brief overview of the most prevalent causes of blindness in the industrialized world that includes age-related macular degeneration, diabetic retinopathy, and glaucoma, the review is devoted to retinal imaging and image analysis methods and their clinical implications. Methods for 2-D fundus imaging and techniques for 3-D optical coherence tomography (OCT) imaging are reviewed. Special attention is given to quantitative
techniques for analysis of fundus photographs with a focus on clinically relevant assessment of retinal vasculature, identification of retinal lesions, assessment of optic nerve head (ONH) shape, building retinal atlases, and to automated methods for population screening for retinal diseases. A separate section is devoted to 3-D analysis of OCT images, describing methods for segmentation and analysis of retinal layers, retinal vasculature, and 2-D/3-D detection of symptomatic exudate-associated
derangements, as well as to OCT-based analysis of ONH morphology and shape. Throughout the paper, aspects of image acquisition, image analysis, and clinical relevance are treated together considering their mutually interlinked relationships. Index Terms: Computer-aided diagnosis, fundus photography, optical coherence tomography (OCT), population screening, retina The retina is a layered
tissue lining the interior of the eye that enables the conversion of incoming light into a neural signal that is suitable for further processing in the visual cortex of the brain. It is thus an extension of the brain. The ability to image the retina and develop techniques for analyzing the images is of great interest. As its function requires the retina to see the outside world, the involved ocular structures have to be optically transparent for image formation. Thus, with proper techniques, the
retina is visible from the outside, making the retinal tissue, and thereby brain tissue, accessible for imaging noninvasively (Fig. 1). Because the retina’s function makes it a highly metabolically active tissue with a double blood supply, the retina allows direct noninvasive observation of the circulation. First known image of human retina as drawn by Van Trigt in 1853 [1]. Thus, because of its architecture—dictated by its function—both diseases of the eye, as well as diseases that affect the circulation and the brain can manifest themselves in the retina. These include ocular diseases, such as macular degeneration and glaucoma, the first and third most important causes of blindness in the developed world. A number of systemic diseases also affect the retina. Complications of such systemic diseases include diabetic retinopathy from diabetes, the second most common cause of blindness in the developed world, hypertensive retinopathy from cardiovascular disease, and multiple sclerosis. Thus, on the one hand, the retina is vulnerable to organ-specific and systemic diseases, while on the other hand, imaging the retina allows diseases of the eye proper, as well as complications of diabetes, hypertension and other cardiovascular diseases, to be detected, diagnosed and managed. This review focuses on quantitative approaches to retinal image analysis. Principles of 2-D and 3-D retinal imaging are outlined first. Special emphasis is given to fundus and optical coherence tomography (OCT) image analysis and its use to provide comprehensive descriptions of retinal morphology and function. The described methods cover the developments of the past decade and were selected with respect to their potential for screening-motivated computer-aided detection of retinal abnormalities as well as for translational clinical applications including improved retinal disease diagnoses and image-guided retinal therapy. As such, the methods presented are expected to influence routine clinical patient care in the years to come. A. Eye AnatomyThis review focuses on the retina, nevertheless, a brief review of gross eye anatomy is in place (Fig. 2). The visible parts of the eye include the transparent cornea, the normally white sclera, the colored (blue, green, brown or a mixture of these) iris, and an opening in the iris, the normally black pupil. A ray of light, after passing through the cornea, which partially focuses the image, passes through the anterior chamber, the pupil, the lens, which focuses the image further, the vitreous and is then focused on the retina. The retina itself is supported by its retinal pigment epithelium, which is normally opaque, the choroid and the sclera. The blood supply of the retina is primarily (~65%) through the choroid and secondarily (~35%) through the retinal vasculature which lies on top of the retina. It is useful to divide the retina and choroid into the following layers:  Illustration of eye anatomy and retinal layers [2], [3]. (a) Cross-sectional view of eye and its major structures. Retina is a thin transparent tissue that lines the back of the eye and is comprised of a number of layers, as illustrated in enlarged portion. (b) Schematic drawing of cellular layers of retina. (a) Two-dimensional illustration of eye anatomy. (b) Schematic of retinal layers. Illustrations from Kolb [3] used with kind permission of Sigma Xi, The Scientific Research Society, Research Triangle Park, NC.
Most of the retinal layers can be seen on optical coherence tomography (OCT) images (Section V). However, imaging of the capillary choroid and choroid plexus, though available in a research setting, cannot yet be done with commercially available devices. B. Retinal Manifestations of Eye and Systemic DiseaseMany important diseases manifest themselves in the retina and originate either in the eye, the brain, or the cardiovascular system. A brief overview of the most prevalent diseases that can be studied via eye imaging and image analysis follows. 1) DiabetesDiabetes mellitus, according to the current definition from the World Health Organization [4], is typically diagnosed if a patient has a fasting plasma glucose over 7.0 mmol/l. Its causes are not fully understood, but genetic background, obesity, and sedentary lifestyle all confer increased risk of developing diabetes. Treatment is primarily through diet changes, administration of insulin and/or anti-hyperglycemic drugs. Hyperglycemia, the presence of elevated blood glucose, is known to damage small and large blood vessels, as well as nerve cells, and thereby damages the kidneys, heart, brain and eyes, and results in a retinal complication of diabetes called diabetic retinopathy. 2) Diabetic RetinopathyDiabetic retinopathy (DR) is a complication of diabetes mellitus and the second most common cause of blindness and visual loss in the U.S., and the most important cause in the working age population. The number of patients with diabetes in the U.S. is increasing rapidly and in 2007 reached 23.5 million [5]–[7]. There is abundant evidence that blindness and visual loss in these patients can be prevented through annual screening and early diagnosis [8]. In the eye, hyperglycemia damages the retinal vessel walls, which can lead to:
The primary cause of visual loss in people with diabetes is DME, which is more common in type 2 diabetes. The breakdown of the blood-retinal barrier causes leakage of dilated hyperpermeable capillaries and microaneurysms into intracellular and extracellular retinal tissue with subsequent fluid accumulation [9], [10]. Clinically significant macular edema (CSME) occurs if there is thickening of the retina involving the center of the retina (macula) or the area within 500 m of the center, if there are hard exudates at or within 500 m of the center with thickening of adjacent retina, or if there is a zone of retinal thickening one optic disc area or larger in size, any part of which is within one disc diameter of the center of the retina [9]. This definition of CSME generally refers to the threshold level at which laser photocoagulation treatment is considered. While visual loss occurs when macular edema involves the visual center, lesser degrees of DME may cause visual deterioration. It is clear that DME affects macular structure in both the short and long term. The leaking exudate in DME initially enters the cytoplasm of Müller’s cells (radial glial cells of the retina), preferentially in the outer retina, though fluid accumulation has been found to extend through most macular layers in more advanced stages of DME [11]. Cysts (retinal extracellular fluid) occur predominantly in the outer retina. Over time, cysts tend to fuse and extend from the outer into the inner retina. In these cases, atrophy or apoptosis of the remaining retinal tissue occurs [11], [12]. Serous detachment may occur in ~20% of DME cases and does not seem to correlate with visual acuity. Hard exudates can occur and tend to be located at the level of the outer plexiform layer. Patients with longstanding DME with impaired visual acuity show decreased directional sensitivity of photoreceptors and decreased visual pigment density [13]. The management of diabetes primarily involves lowering of blood sugar, through diet, lifestyle changes and anti-diabetic drugs. If DR is present, management of CSME and proliferative DR through laser photocoagulation, administration of anti-vascular growth factors, and of steroids have been shown in large randomized clinical trials to prevent blindness and further visual loss [9], [14]–[16]. 3) Age-Related Macular DegenerationAge-related macular degeneration (AMD) is the most common cause of visual loss in the U.S. and is a growing public health problem. Currently, almost 7.3 million Americans (6.12% of Americans aged 40 years and older) have some form of AMD, and AMD is the cause of blindness for 54% of all legally blind Americans [17]. Severe AMD reduces the likelihood of employment by 61% and salary by 39%, while mild AMD reduces these by 44% and 32%, respectively. The estimated annual cost burden from AMD in the U.S. has been estimated as $30 billion [18]. The prevalence of AMD is expected to double over the next 25 years [5]. The two major forms are dry and wet AMD, of which dry AMD typically leads to gradual loss of visual acuity. Wet AMD, also called choroidal neovascularization (CNV), is the most visually threatening form, characterized by ingrowth of a choroidal vascular structure into the macula accompanied by increased vascular permeability. The increase in vascular permeability leads to abnormal fluid collection within or below the retina that causes visual dysfunction when it involves the center of the macula. The natural course of CNV is rapidly deteriorating acuity, scarring of the pigment epithelium, and permanent visual loss or blindness. Progression of dry AMD can be slowed in many patients through dietary supplements [19], while visual loss from wet AMD is treated with intravitreal administration of anti-vascular growth factor [20], [21]. 4) GlaucomaGlaucoma is the third leading cause of blindness in the U.S., characterized by gradual damage to the optic nerve and resultant visual field loss [22]. Early diagnosis and optimal treatment have been shown to minimize the risk of visual loss due to glaucoma [23]. Glaucoma is primarily a neuropathy, not a retinopathy, and acts on the retina by damaging ganglion cells and their axons. The hallmark of glaucoma is cupping of the optic disc, which is the visible manifestation of the optic nerve head (ONH) 3-D structure. The optic disc can be imaged two-dimensionally either through indirect stereo biomicroscopy or with stereo color fundus photography. The ratio of the optic disc cup and neuroretinal rim surface areas in these images, called cup-to-disc ratio, is an important structural indicator for assessing the presence and progression of glaucoma. Glaucoma is typically treated with ocular pressure lowering drops, and in refractory cases through surgery. 5) Cardiovascular DiseaseCardiovascular disease manifests itself in the retina in a number of ways. Hypertension and atherosclerosis cause changes in the ratio between the diameter of retinal arteries and veins, known as the A/V ratio. A decrease in the A/V ratio, i.e., thinning of the arteries and widening of the veins, is associated with an increased risk of stroke and myocardial infarction [24], [25]. Hypertension can also invoke direct retinal ischemia, which causes retinal infarcts visible as cotton wool spots and choroidal infarcts visible as deep retinal white spots. In addition, systemic vascular disease can cause arterial and venous occlusions, known as central and branch arterial occlusions (CRAO, BRAO) and central and branch venous occlusions (CRVA, BRVO). C. History of Retinal ImagingSomewhat paradoxically, the optical properties of the eye that allow image formation prevent direct inspection of the retina. In other words, the very nature of the imaging transform resulting in a focused image on the retinal surface disallows depiction of the retina when attempting to form a focused retinal image from the outside via usage of the inverse transform. The red reflex, when a blurred reflection of the retina makes the pupil appear red if light is shined into the eye at the appropriate angle, was known for centuries. However, special techniques are needed to obtain a focused image of the retina. The first attempt to image the retina in a cat was completed by the French physician Jean Mery, who showed that if a live cat is immersed in water, its retinal vessels are visible from the outside. The impracticality of such an approach for humans lead to the invention of the principles of the ophthalmoscope in 1823 by Czech scientist Jan Evangelista Purkyně (frequently spelled Purkinje) and its reinvention in 1845 by Charles Babbage [26], [27]. Note that Babbage also originated the concept of a programmable computer and thus the link between computation [28] and retinal imaging is not a new one. Finally, the ophthalmoscope was reinvented again and reported by von Helmholtz in 1851 [29]. Thus, inspection and evaluation of the retina became routine for ophthalmologists, and the first images of the retina (Fig. 1) were published by the Dutch ophthalmologist van Trigt in 1853 [1]. Earlier sketches by Purkyně provided drawings of his own retinal vasculature [30] (Fig. 3). Early drawing of retinal vasculature including outlines of ONH and fovea published by Purkyne in 1823 [30]. Because of the prevalence of infectious diseases at the time and because the ophthalmoscope required the physician to come close to the face of the patient, it was attractive to image the eye photographically. The first useful photographic images of the retina, showing blood vessels, were obtained in 1891 by the German ophthalmologist Gerloff [31]. In 1910, Gullstrand developed the fundus camera, a concept still used to image the retina today [32]; he later received the Nobel Prize for this invention. Because of its safety and cost-effectiveness at documenting retinal abnormalities, fundus imaging has remained the primary method of retinal imaging. The next important development was the invention of fluorescein angiographic imaging, where a fundus camera with additional narrow band filters is used to image a fluorescent dye injected into the bloodstream that binds to leukocytes [33]. It remains widely used because it allows an understanding of the functional state of the retinal circulation. Concerns about safety and cost-effectiveness are leading it to be slowly replaced by tomographic imaging methods for its primary applications, namely image-guided treatment of macular edema and the “wet form” of macular degeneration. A major limitation of fundus photography is that it obtains a 2-D representation of the 3-D semi-transparent retinal tissues projected onto the imaging plane. The initial approach to depict the 3-D shape of the retina was stereo fundus photography, as first described by Allen in 1964 [34], where multi-angle images of the retina are combined by the human observer into a 3-D shape. Subsequently, confocal scanning laser ophthalmoscopy was developed, using the confocal aperture to obtain multiple images of the retina at different confocal depths, yielding estimates of 3-D shape. However, the optics of the eye limit the depth resolution of confocal imaging to approximately 100 μm which is poor when compared with the typical 300–500 μm thickness of the whole retina [35]. Tomographic imaging of the retina became commonplace with the development of super-luminescent diodes, femtosecond lasers and the application of optical coherence tomography (OCT) to retinal imaging [36], which allows truly 3-D optical sectioning of the retina [37]. D. History of Retinal Image ProcessingMatsui et al. were the first to publish a method for retinal image analysis, primarily focused on vessel segmentation [38]. Their approach was based on mathematical morphology and they used digitized slides of fluorescein angiograms of the retina. In the following years, there were several attempts to segment other anatomical structures in the normal eye, all based on digitized slides. The first method to detect and segment abnormal structures was reported in 1984, when Baudoin et al. described an image analysis method for detecting microaneurysms, a characteristic lesion of diabetic retinopathy [39]. Their approach was also based on digitized angiographic images. The work of Baudoin et al. detected microaneurysms using a “top-hat” transform, a step-type digital image filter [40]. The field dramatically changed in the 1990s with the development of digital retinal imaging and the expansion of digital filter-based image analysis techniques. These developments resulted in a rapidly increasing number of publications that is continuing to expand. Closely related to retinal image analysis, the first multicenter, randomized clinical trials in the history of ophthalmology, the Diabetic Retinopathy Study and especially the Early Treatment of Diabetic Retinopathy Study, showed the relevance of the thickness of retinal structures. II. Current Status of Retinal ImagingRetinal imaging has developed rapidly during the last 160 years and is a now a mainstay of the clinical care and management of patients with retinal as well as systemic diseases. Fundus photography is widely used for population-based, large scale detection of diabetic retinopathy, glaucoma, and age-related macular degeneration. Optical coherence tomography (OCT) and fluorescein angiography are widely used in the diagnosis and management of patients with diabetic retinopathy, macular degeneration, and inflammatory retinal diseases. OCT is also widely used in preparation for and follow-up in vitreo-retinal surgery. A. Fundus ImagingWe define fundus imaging as the process whereby a 2-D representation of the 3-D retinal semi-transparent tissues projected onto the imaging plane is obtained using reflected light. Thus, any process which results in a 2-D image, where the image intensities represent the amount of a reflected quantity of light, is fundus imaging. Consequently, OCT imaging (Section II-B) is not fundus imaging, while the following modalities/techniques all belong to the broad category of fundus imaging:
1) Technical Challenges in Fundus ImagingSince the retina is normally not illuminated internally, external illumination projected into the eye as well as the light reflected by the retina must traverse the pupillary plane. Thus the size of the pupil, the small opening in the iris usually between 2 and 8 mm in diameter, has always been the primary technical challenge in fundus imaging [32]. Fundus imaging is complicated by the fact that the illumination and imaging beams cannot overlap because that results in corneal and lenticular reflections diminishing or eliminating image contrast. Consequently, separate paths are used in the pupillary plane, resulting in optical apertures on the order of only a few millimeters. Because the resulting imaging setup is technically challenging, fundus imaging historically involved relatively expensive equipment and highly trained ophthalmic photographers. Over the last ten years or so, there has been a major effort to make fundus imaging more accessible, resulting in less dependence on such experience and expertise due to the following three most important developments:
Though standard fundus imaging is widely used, it is not suitable for retinal tomography, because of the mixed backscatter caused by the semi-transparent retinal layers. Consequently, the backscatter’s origin is decoupled from the specific retinal depth location. B. Optical Coherence Tomography ImagingThe principle of Optical Coherence Tomography (OCT) is the estimation of the depth at which a specific backscatter originated by measuring its time of flight. Backscatters are typically caused by differences in refractive index in transitions from one tissue to another. The backscatter from deeper tissues can be differentiated from backscatter originating at more superficial tissues because it takes longer for the light to arrive at the sensor. As the total retinal thickness is between 300–500 μm, the differences in time of flight are very small and can only be measured through interferometry [36]. OCT employs low-coherent light interferometry, also called white light interferometry—though the wavelengths used for OCT are usually slightly longer than visible light. Low-coherent light autocorrelates only for a short amount of time, or equivalently, for only a small number of wavelengths, while autocorrelation function values are essentially zero beyond that. Low-coherent illumination can be thought of as a train of highly autocorrelated overlapping “bursts” of light—each burst labeled by its unique autocorrelogram. While we use the term “burst” to make the description more intuitive, it is important to understand that the low-coherent light is actually continuous and not pulsed. To determine the time delay of the low-coherent light that has backscattered from the retina, and thus the depth at which the backscatter occurred, the bursts are identified by their autocorrelation function. By splitting low coherent light optically, sending one reference beam to reflect from a mirror at a specific distance, the other to reflect from the tissues in the retina, non-zero cross-correlation (or interference) between the two reflected beams occurs only when their flight times are equal, being zero everywhere else because of the low coherence property. It is the envelope, or energy, of the non-zero interferogram that is converted into an intensity that represents the amount of backscatter from the image location I=krIs+ ksIs+2(krIs)·(ksIs)·Re{γ(τ)} (1) where kr and ks equal the splitting ratio for the reference and sample arm, respectively, Is is the source intensity, and γ(τ) the complex degree of coherence, equal to γ(τ)=exp[−(πΔvτ2ln2)2]·exp(−i2π v0τ) (2) with v0 being the center wavelength of Is and Δv the width of Is in the frequency domain. Therefore, the amount of light coherence as measured (1) is inversely proportional to the depth that can be resolved: the less coherent the light, the narrower the autocorrelogram, and thus the narrower the peak in the cross-correlation with the reference arm. Wavelengths longer than visible light penetrate deeper into retinal and choroidal tissue, and thus OCT technology is dependent on broadband near infrared or even infrared light sources, such as super-luminescent LEDs. Commonly, a beam splitter is used to split the light into the two beams: a beam that reflects of the retinal tissue represents the sample arm and a beam that is reflected from the reference mirror is called the reference arm (Fig. 4). The interferogram energy between the reference and sample arms is converted into image intensities with a photo sensor, CCD, or a CMOS sensor. A depth scan, typically known as an A-scan using ultrasound terminology, with different intensities representing the backscatter at different depths, is created from the interferogram intensities. For 3-D imaging—same as in scanning laser ophthalmoscopy—the illuminating beam is moved across the retina, typically using galvanic mirrors that change the position in two directions (along x and y axes), resulting in a tomographic image with an A-scan for each x and y location. Schematic diagram of OCT, with emphasis on splitting of the light, overlapping train of labeled bursts based on their autocorrelogram, and their interference after being reflected from retinal tissue as well as from the reference mirror (assuming the time delays of both paths are equal). Different approaches have been used to maximize the number of A-scans that can be obtained per unit of time, because reducing motion artifacts, patient comfort, and achieving high image resolution are all dependent on achieving short A-scan imaging intervals. Three main OCT principles were developed to create an A-scan for the desired tissue depth range.
1) Three-Dimensional OCT ImagingOriginally, OCT imaging was highly limited by the amount of time it took to image an A-scan. Therefore the correlogram intensities were displayed to the user in the form of 2-D slices, called B-scans, with the A-scans in the B-scan obtained from a single sweep of the scanning beam over the retina in a linear or circular fashion. Increased A-scanning speed resulted in the possibility of increased numbers of B-scans per examination. The ability to image the retina two-dimensionally and three-dimensionally then depends on the number of A-scans that can be acquired over time. Because of motion artifacts such as saccades, safety requirements limiting the amount of light that can be projected onto the retina, and patient comfort, 1–3 seconds per image or volume is essentially the limit of acceptance. Thus, the commercially available time-domain OCT, which allowed collecting of up to 400 A-scans per second, has not yet been suitable for 3-D imaging. With spectral-domain OCT, tens of thousands of A-scans can be acquired each second, and thus true 3-D imaging is routinely possible. Consequently, 3-D OCT is now in wide clinical use and has become the standard of care. With swept-source OCT, hundreds of thousands of A-scans can be obtained every second, promising additional increase in image resolution when acquiring 3-D image volumes. 2) Resolution, Isotropic ImagingThe transverse resolution of OCT scans (i.e., in the x, y directions) depends on the speed and quality of the galvanic scanning mirrors and is typically 20–40 μm. The resolution of the A-scans along the z direction depends on the coherence of the light source and is currently 4–8 μm in commercially available scanners. As explained above, obtaining 2-D or 3-D OCT images is largely a matter of the scanning speed versus the amount of time available for scanning. A crude 3-D volume was initially created by juxtaposing several B-scan slices spaced widely apart. With the advent of faster scanners, the spaces between B-scans were minimized and currently some commercially available scanners are capable of acquiring close-to-isotropic 3-D volumes. Isotropic (isometric) means that the size of each imaged element, or voxel, is the same in all three dimensions. Current commercially available OCT devices routinely offer voxel sizes of 30 × 30 × 2 μm, achieving isometricity in the x − y plane only (note the difference between resolution and voxel size). Another way of explaining isotropicity (in the x − y plane) is that the spacing between the acquired B-scans (in what we call the y-dimension) is the same as the distance between each A-scan in the B-scans (in what we call the x-dimension). Available SD-OCT scanners are never truly isotropic, because the retinal tissue in each A-scan is sampled at much smaller intervals in depth than are the distances between A- and/or B-scans. The resolution in depth (or what we call the z-dimension) is currently always higher than the resolution in the x − y plane. The primary advantage of x − y isotropic imaging when quantifying properties of the retina is that fewer assumptions have to be made about the tissue in-between the measured samples, thus potentially leading to more accurate indices of retinal morphology. C. Areas of Active Research in Retinal ImagingRetinal imaging is rapidly evolving and newly completed research results are quickly translated into clinical use. Much of the active research is beyond the scope of this review but several active directions deserve at least basic coverage. 1) Portable, Cost-Effective Fundus ImagingFor early detection and screening, the optimal place for placing fundus cameras is where the patients are: medicine and family care clinics, rapid clinics in shopping malls, etc., Though the transition from film based to digital fundus imaging has revolutionized the art of fundus imaging and made telemedicine applications feasible, the current cameras are still too bulky, expensive and may be difficult to use for nontrained staff in places lacking ophthalmic imaging expertise. Several groups are attempting to create more cost-effective and easier-to-use handheld fundus cameras, employing a variety of technical approaches [42], [43]. 2) Functional ImagingFor the patient as well as for the clinician, the outcome of disease management is mainly concerned with the resulting organ function, not its structure. In ophthalmology, current functional testing is mostly subjective and patient dependent, such as assessing visual acuity and utilizing perimetry, which are all psychophysical metrics. Among more recently developed “objective” techniques, oxymetry is a hyper-spectral imaging technique, in which multispectral reflectance is used to estimate the concentration of oxygenated and deoxygenated hemoglobin in the retinal tissue [44]. The principle allowing to sense such differences is simple: deoxygenated hemoglobin reflects longer wavelengths better than does oxygenated hemoglobin. Nevertheless, measuring absolute oxygenation levels with reflected light is difficult because of the large variety in retinal reflection across individuals and the variability caused by the imaging process. The retinal reflectance can be modeled by a system of equations, and this system is typically under-constrained if this variability is not accounted for adequately. Increasingly sophisticated reflectance models have been developed to correct for the underlying variability, with some reported success [45]. Near-infrared fundus reflectance in response to visual stimuli is another way to determine the retinal function in vivo and has been successful in cats. Initial progress was demonstrated in humans [46]. 3) Adaptive OpticsThrough the evolutionary processes, the human eye and retina are well matched for optimal visual performance. The optical properties of the normal eye result in a point spread function width approximately the size of a photoreceptor. It is therefore impossible to image individual cells or cell structure using standard fundus cameras because of aberrations in the human optical system. Adaptive optics uses mechanically activated mirrors to correct the wavefront aberrations of the light reflected from the retina and thus has allowed individual photoreceptors to be imaged in vivo [47]. Imaging other cells, especially the clinically highly important ganglion cells, has so far been unsuccessful. 4) Longer Wavelength OCT ImagingThree-dimensional OCT imaging is now the clinical standard of care for several eye diseases. However, the wavelengths around 840 μ m used in currently available devices only allow imaging of the retina. Deeper structures, such as the choroidal vessels, which are important in AMD and uveitis, and the lamina cribrosa—a deep structure in the optic nerve relevant for glaucomatous damage—are not visualized. Because longer wavelengths penetrate deeper into the tissue, a major research effort has been invested to developing low-coherence swept-source lasers with center wavelengths of 1000–1300 μ m. Prototypes of these devices are already able to resolve detail in the choroid and lamina cribrosa [48]. III. Disease-Specific Analysis of Retinal ImagesThe everyday cost associated with eye care providers’ decisions and the ever-increasing numbers of retinal images to be reviewed are the major motivations for the adoption of image analysis in ophthalmology. Clearly, since clinicians are costly experts, they need to optimize the time devoted to each patient, whether their cost is born by patients, third party insurers, or society as a whole. As presented in the following sections, the development of new imaging technology invariably results in rapidly increasing amounts of data collected as part of any specific retinal imaging exam. The amount of information provided by the current generation of scanners and cameras is already exceeding the limit of clinicians’ ability to fully utilize it. When factoring in that clinicians are subjective, and their decisions suffer from the inter- and intra-observer variability, the need for reliable computerized approaches to retinal image analysis is more than obvious, if for no other reason, than to increase the precision with which patients are managed. An additional important reason for incorporating automated analyses of retinal images in patient management is the potential societal benefit of increasing clinician productivity in a routine population screening setting. While the patient management decision making and population screening scenarios are somewhat different and specific, they both require quantitative retinal image analysis to be rapidly translated to everyday use. A. Early Detection of Retinal Disease From Fundus PhotographyThe most obvious example of a retinal screening application is retinal disease detection, in which the patient’s retinas are imaged in a telemedicine remote manner. This scenario requires utilizing easy-to-use, relatively low-cost fundus cameras, automated analyses of the images, and focused reporting of the results. This screening application has spread rapidly over the last few years, with the exception of the automated analysis behavior, and is one of the most successful examples of telemedicine [49]. While screening programs exist for detection of glaucoma, age-related macular degeneration, and retinopathy of prematurity, the main screening application focuses on early detection of diabetic retinopathy. Early detection of diabetic retinopathy (Section I-B2) via population screening associated with timely treatment have been shown to prevent visual loss and blindness in patients with retinal complications of diabetes [50], [51]. Almost 50% of people with diabetes in the U.S. currently do not undergo any form of regular documented dilated eye exam, in spite of guidelines published by the American Diabetes Association, the American Academy of Ophthalmology and the American Optometric Association [52]. In the U.K., a smaller proportion or approximately 20% of diabetics are not regularly evaluated, as a result of an aggressive effort to increase screening for people with diabetes. Blindness and visual loss can be prevented through early detection and timely management. There is widespread consensus that regular early detection of diabetic retinopathy via screening is necessary and cost-effective in people with diabetes [8], [53]–[55]. Remote digital imaging and ophthalmologist expert reading have been shown to be comparable or superior to an office visit for assessing DR [56], [57] and have been suggested as an approach to make the dilated eye exam available to un- and under-served populations that do not receive regular exams by eye care providers. If all of these underserved populations were to be provided with digital imaging, the annual number of retinal images requiring evaluation would exceed 32 million in the U.S. alone (approximately 40% of people with diabetes with at least two photographs per eye) [57], [58]. In the next decade, projections for the U.S. are that the average age will increase, the number of people with diabetes in each age category will increase, and there will be an under-supply of qualified eye care providers, at least in the near term. Several European countries have successfully instigated in their health care systems early detection programs for diabetic retinopathy using digital photography with reading of the images by human experts. In the U.K., 1.7 million people with diabetes were screened for diabetic retinopathy in 2007–2008. In the Netherlands, over 30 000 people with diabetes were screened since 2001 in the same period, through an early detection project called EyeCheck [59]. The United States Department of Veterans Affairs (VA) has deployed a successful photo screening program in the VA medical centers, through which more than 120 000 patients were screened in 2008. While the remote imaging followed by human expert diagnosis approach was shown successful for a limited number of participants, the current challenge is to make the early detection more accessible by reducing the cost and manpower required, while maintaining or improving DR detection performance. This challenge can be met by utilizing computer-assisted or fully automated methods for detection of DR in retinal images, as described in Section IV. B. Early Detection of Systemic Disease From Fundus PhotographyIn addition to detecting diabetic retinopathy and age-related macular degeneration, it also deserves mention that fundus photography allows cardiovascular risk factors to be determined. Such metrics are primarily based on measurement of retinal vessel properties, such as the arterial to venous diameter ratio, or A-V ratio, and indicate the risk for stroke, hypertension or myocardial infarct [60], [61]. C. Three-Dimensional OCT and Retinal Diseases—Image Guided TherapyWith the introduction of 3-D OCT imaging, the wealth of new information about the retinal morphology enabled its usage for close monitoring of retinal disease status and guidance of retinal therapies. The most obvious example of successful image-guided management in ophthalmology is its use in diabetic macular edema (DME). DME is a form of diabetic retinopathy in which visual loss occurs through leaking of fluid in the macula—the central part of the retina. The original research entitled Early Treatment in Diabetic Retinopathy Study [9], [62]–[64] demonstrated that early treatment of DME’s thickened areas of the retina with focal laser can prevent further visual loss. More recently, novel treatment using anti-VEGF agents (anti-vascular endothelial growth factor), such as ranibizumab combined with focal laser, has shown to be beneficial for treatment of DME. Currently, OCT imaging is widely used to determine the extent and amount of retinal thickening. We expect that detailed analyses of retinal layer morphology and texture from OCT similar to those described in Section V will allow direct image-based treatment to be guided by computer-supported or automated quantitative analysis of OCT and subsequently optimized allowing personalized approach to retinal disease treatment to become a reality. Another highly relevant example of a blinding disease that will benefit from image guided therapy is choroidal neovascularization—the wet form of age related macular degeneration (Section I-B3). With the advent of the anti-VEGF agents ranibizumab and bevacizumab, it has become clear that outer retinal and subretinal fluid is the main indicator of a need for anti-VEGF retreatment [21], [65]–[67]. Several studies are underway to determine whether OCT-based quantification of fluid parameters and affected retinal tissue can help improve the management of patients with anti-VEGF agents. As described above, glaucoma is characterized by gradual damage to the optic nerve and resultant visual field loss (Section I-B4) [22]. Early diagnosis and optimal treatment have been shown to minimize the risk of visual loss due to glaucoma [23]. As shown below, 3-D analysis of the optic nerve head can be used for glaucoma management decisions. However, it has been previously shown that manual planimetry is time consuming with substantial inter-observer variability [68]. Methods for automated assessment of the cup-to-disc ratio and for NCO-based ONH analyses from fundus photography and from 3-D OCT imaging are presented as follows. Their adoption for use in routine clinical care is highly desirable. IV. Fundus Image AnalysisAs discussed previously in Section II-A, fundus imaging is the most established way of retinal imaging. Until recently, fundus image analysis was the only source of quantitative indices reflecting retinal morphology. Subjects that lend themselves for fundus image analysis include:
As this paper went to press, over 700 papers have been published on these subjects in fundus image analysis, and discussing each one is beyond the scope of this review. Therefore, we have focused only on those fundamental tasks and related approaches to fundus image analysis that are actively researched by a large number of groups: retinal vessel detection (Section IV-A), retinal lesion detection (Section IV-B), construction of fundus-imaging-based retinal atlases (Section IV-C), and analysis of the optic nerve head morphology from fundus photographs (Section IV-E), in more detail. Registration of fundus images and change detection will be discussed in Section VI-A. In addition, individual methods have been combined into disease-detection systems, particularly for diabetic retinopathy [69]–[71]. A. Detection of Retinal VesselsAutomated segmentation of retinal vessels is a generally well understood problem, aided by the public availability of the annotated STARE [72] and DRIVE datasets [73] (Fig. 5) with hundreds of papers published [74], [75]. Pixel feature classification and identification of elongated structures has been highly successful in the detection of large and medium vessels [73], [76]. Automated vessel analysis. From left to right: fundus image; retinal specialist annotation; vesselness map from Staal algorithm [76]; vesselness map from direct pixel classification [73]. Though not by design, the similarities among the different approaches to vessel detection are often not obvious at first, because of different terms used for the same concepts. For example, template matching, kernel convolution, detector correlation all describe the same concept explained in more detail in the following, though implementation details may vary. 1) Pixel Feature ClassificationPixel feature classification is a machine learning technique that assigns one or more classes to the pixels in an image. Pixel classification uses multiple pixel features: numeric properties of a pixel and its surroundings. Pixel feature classification is typically performed using a supervised approach. Originally, pixel intensity was used as a single feature. More recently, n-dimensional multifeature vectors are utilized including pixel contrast with the surrounding region, its proximity to an edge, and similarity. Two distinct stages are required for a supervised learning/classification algorithm to function: 1) a training stage, in which the algorithm “statistically learns” to correctly classify pixels from known classifications, and 2) a testing or classification stage in which the algorithm classifies previously unseen images. For proper assessment of supervised classification method functionality, training data and performance testing data sets must be completely disjoint [77]. The n-dimensional multifeature vectors are calculated for each pixel, frequently utilizing local convolutions with multiple Gaussian derivative, Gabor, or other wavelet kernels [78]. The image is thus transformed into an n-dimensional feature space and pixels are classified according to their position in feature space. The resulting hard (categorical) or soft (probabilistic) classification is then used to either assign labels to each pixel (for example vessel or nonvessel in the case of hard classification), or to construct class-specific likelihood maps (e.g., a vesselness map for soft classification). For example, an image Ii(x, y) can be transformed into the Gaussian derivative space Ii(x, y, σk) by convolution with Gaussian derivative kernels as follows: Ii(x,y,σk)=I 0(x,y)★Gnα(σk) (3) where ★ represents convolution, σk ∈ [0, ∞)is the relative scale, and G is the Gaussian derivative kernel of order n ∈ {0, 1, 2} with orientation α ∈ [0, …, 2π]. The number of potential features in the multifeature vector that can be associated with each pixel is essentially infinite. One or more subsets of this infinite set can be considered optimal for classifying the image according to some reference standard. Hundreds of features for a pixel can be calculated in the training stage to cast as wide a net as possible, with algorithmic feature selection steps used to determine the most distinguishing set of features. Extensions of this approach include different approaches to subsequently classify groups of neighboring pixels by utilizing group properties in some manner, for example cluster feature classification, where the size, shape and average intensity of the cluster may be used. 2) Vessel Diameter and Vessel Tree AnalysisBecause retinal vessel diameter and especially the relative diameters of arteries and veins are known to signal the risk of systemic diseases including stroke, accurate determination of retinal vessel diameters, as well as differentiation of veins and arteries have become more important, several semi-automated and automated approaches have now been published [24], [25], [79]. Other active areas of research include separation of arteries and veins, detection of small vessels with diameters of less than a pixel, and analysis of the complete vessel trees using graphs. B. Detection of Retinal LesionsIn this section, we will primarily focus on detection of lesions in diabetic retinopathy. It has the longest history as a research subject in retinal image analysis. Fig. 6 shows examples of a fundus photograph with the typical lesions automatically detected. Many approaches used the following principle (Fig. 7): A transform of some kind is used for detecting candidate lesions, after which a mathematical morphology template is utilized to characterize the candidates. This approach or a modification thereof is in use in many algorithms for detecting DR and AMD [80]. Additional enhancements include the contributions of Spencer, Cree, Frame, and co-workers [81], [82]. They added preprocessing steps, such as shade-correction and matched filter post-processing to this basic framework, to improve performance. Algorithms of this kind function by detecting candidate microaneurysms of various shapes, based on their response to specific image filters. A supervised classifier is typically developed to separate the valid microaneurysms from spurious or false responses. However, these algorithms were originally developed to detect the high-contrast signatures of microaneurysms in fluorescein angiogram images. The next important development resulted from applying a modified version of the top-hat algorithm to red-free fundus photographs rather than angiogram images, as was first described by Hipwell et al. [83]. They tested their algorithm on a large set of >3500 images and found a sensitivity/specificity operating point of 0.85/0.76. Once this step had been taken, development accelerated. The approach was further refined by broadening the candidate detection transform, originally developed by Baudoin to detect candidate pixels, to a multifilter filter-bank approach [73], [84]. The filter responses are used to identify pixel candidates using a classification scheme. Mathematical morphology and additional classification steps are applied to these candidates to decide whether they indeed represent microaneurysms and hemorrhages. A similar approach was also successful in detecting other types of DR lesions, including exudates or cotton-wool spots, as well as drusen in AMD [85]. Automated analysis of fundus photographs. (a) Fundus photograph showing several lesions typical of diabetic retinopathy. (b) Detection of red lesions (RL)—microaneurysms and hemorrhages. (c) Detection of bright lesions (BL)—lipoprotein exudates. (d) Detection of neovascularization (NVD) of the optic disc. (e) All automatically detected lesions shown.
Typical steps necessary for analysis of fundus images, in this case for early diabetic retinopathy. Top row from left to right: original image; detection of fovea and optic disc superimposed as yellow circles on the vesselness map; automatically detected red lesions indicated in shades of green, bright lesions in shades of blue. Bottom row: details of red and bright lesion detection steps shown in a small region of the image including pixel classification identifying suspect pixels, clustering of suspect pixels, and classification of clusters as lesions. 1) Detection of Red LesionsSmall red retinal lesions, namely microaneurysms and small retinal hemorrhages, are typical for diabetic retinopathy, hypertensive retinopathy, and other retinal disorders such as idiopathic juxtafoveal teleangiectasia. The primary importance of small red lesions is that they are the leading indicators of diabetic retinopathy. Because they are difficult to differentiate for clinicians on standard fundus images from nonmydriatic cameras, hemorrhages and microaneurysms are usually detected together and associated with a single combined label. Larger red lesions, primarily large hemorrhages and retinal neovascularizations are still problematic and are discussed in Section IV-B3. Historically, red lesion detection algorithms focused on detection of normal anatomical objects, especially the vessels, because they can locally mimic red lesions. Subsequently, a combination of one or more filtering operations combined with mathematical morphology is employed to detect red lesion suspects. In some cases, suspect red lesion are further classified in individual lesion types and refined algorithms are capable of detecting specific retinal structures and abnormalities as shown in Figs. 7–9. Red lesion detection. (a) Thresholded probability map. (b) Remaining objects after connected component analysis and removal of large vasculature. (c) Shape and size of extracted objects in panel (b) does not correspond well with actual shape and size of objects in original image. Final region growing procedure is used to grow back actual objects in original image which are shown here. In (b) and (c), the same red lesions as in Fig. 8(a) are indicated with a circle. Initially, red lesions were detected in fluoroscein angiograms because their contrast against the background is much higher than that of microaneurysms in color fundus photography images [81], [82], [86]. Hemorrhages mask out fluorescence and present as dark spots in the angiograms. These methods employed a mathematical morphology technique that eliminated the vasculature from a fundus image but left possible microaneurysm candidates untouched as first described in 1984 [39]. Later, this method was extended to high-resolution red-free fundus photographs by Hipwell et al. [83]. Instead of using morphology operations, a neural network was used, for example by Gardner et al. [87]. In their work, images are divided into 20 × 20 pixel grids and the grids are individually classified. Sinthanayothin et al. [88] applied a recursive region growing procedure to segment both the vessels and red lesions in a fundus image. A neural network was used to detect the vessels exclusively, and the remaining objects were labeled as microaneurysms. Niemeijer et al. [84] presented a hybrid scheme that used both the top-hat based method as well as a supervised pixel classification based method to detect the microaneurysm candidates in color fundus photographs. This method allowed for the detection of larger red lesions (i.e., hemorrhages) in addition to the microaneurysms using the same system. A large set of additional features, including color, was added to those described in [82] and [86]. Using the features in a supervised classifier distinguished between real and spurious candidate lesions. These algorithms can usually deal with overlapping microaneurysms because they give multiple candidate responses. Other recent algorithms only detect microaneurysms and forego a phase of detecting normal retinal structures like the optic disc, fovea and retinal vessels, which can act as confounders for abnormal lesions. Instead, the recent approaches find the microaneurysms directly [89] using template matching in wavelet-subbands. In this approach, the optimal adapted wavelet transform is found using a lifting scheme framework. By applying a threshold on the matching result of the wavelet template, the microaneurysms are labeled. This approach has meanwhile been extended to explicitly account for false negatives and false positives [69]. Because it avoids detection of the normal structures, such algorithms can be very fast, on the order of less than a second per image. 2) Detection of Bright LesionsOften, bright lesions, defined as lesions brighter than the retinal background, can be found in the presence of retinal and systemic disease. Drusen are the hallmark of age-related macular degeneration, cotton wool spots are typical for diabetic retinopathy and hypertensive retinopathy, while lipoprotein exudates are most frequently seen in diabetic retinopathy, but also in Coats’ disease and other retinal disorders. To complicate the analysis, flash artifacts can be present as false positives for bright lesions. If the lipoprotein exudates would only appear in combination with red lesions, they would only be useful for grading diabetic retinopathy. The exudates can, however, in some cases appear as isolated signs of diabetic retinopathy in the absence of any other lesion. Therefore, their importance is strengthened and several computer-based systems to detect exudates have been proposed [80], [85], [87], [88], [90]. Because the different types of bright lesions have different diagnostic importance and patient management implications, algorithms should be capable not only of detecting bright lesions, but also of differentiating among the bright lesion types. One example algorithm capable of detection and differentiation of bright lesions was reported in [85]. The algorithm is based on an earlier red lesion algorithm [84] and includes the following main steps.
Fig. 10 illustrates these steps. Compared to retinal experts, the algorithm performed with an area under the ROC curve of Az = 0.95 for detecting bright lesions. The performance Az = 0.94, 0.85, 0.88 of was achieved for the detection and differentiation of exudates, cotton-wool spots, and drusen, respectively. Bright lesion detection algorithm steps performed to detect and differentiate “bright lesions.” From left to right: exudates, cotton-wool spots, and drusen. From top to bottom: relevant regions in the retinal color image (all at same scale); a posteriori probability maps after first classification step; pixel clusters labeled as probable bright lesions (potential lesions); bottom row shows final labeling of objects as true bright lesions, overlaid on original image. 3) Detection of Rare, or Irregular, Lesions and AbnormalitiesPerformance of a system that has been developed for screening should not be evaluated based solely on its sensitivity and specificity for detection of that disease. Such metrics do not accurately reflect the complete performance in a screening setup. Rare, irregular, or atypical lesions often do not occur frequently enough in standard datasets to affect sensitivity and specificity but can have huge health and safety implications. To maximize screening relevance, the system must therefore have a mechanism to detect rare, atypical, or irregular abnormalities, for example in DR detection algorithms [70]. For proper performance assessment, the types of potential false negatives—lesions that can be expected or shown to be incorrectly missed by the automated system—must be determined. While detection of red lesions and bright lesions is widely covered in the literature, detection of rare or irregular lesions, such as hemorrhages, neovascularizations, geographic atrophy, scars and ocular neoplasms has received much less attention, despite the fact that they all can occur in combination with diabetic retinopathy and other retinal diseases as well as in isolation. For example, presence of such lesions in isolated forms and without any co-occurrence of small red lesions are rare in DR [59] and thus missing these does not affect standard metrics of performance such as ROC curves to a measurable degree, except if these are properly weighted as corresponding to serious lesions. One suitable approach for detecting such lesions is to use a retinal atlas, where the image is routinely compared to a generic normal retina (Section IV-C). After building a retinal atlas by registering the fundus images according to a disc, fovea and a vessel-based coordinate system, image properties at each atlas location from a previously unseen image can be compared to the atlas-based image properties. Consequently, locations can be identified as abnormal if groups of pixels have values outside the normal atlas range. C. Retinal AtlasCompared to other anatomic structures (e.g., the brain, heart, or lungs), the retina has a relatively small number of key anatomic structures (landmarks) visible using fundus camera imaging. Additionally, the expected shape, size, and color variations across a population is expected to be high. While there have been a few reports [91] on estimating retinal anatomic structure using a single retinal image, we are not aware of any published work demonstrating the construction of a statistical retinal atlas using data from a large number of subjects. The choice of atlas landmarks in retinal images may vary depending on the view of interest. Regardless, the atlas should represent most retinal image properties in a concise and intuitive way. Three landmarks can be used as the retinal atlas key features; the optic disc center, the fovea, and the main vessel arch defined as the location of the largest vein–artery pairs. The disc and fovea provide landmark points, while the arch is a more complicated two-part curved structure that can be represented by its central axis. The atlas coordinate system then defines an intrinsic, anatomically meaningful framework within which anatomic size, shape, color, and other characteristics can be objectively measured and compared. Choosing either the disc center or fovea alone to define the atlas coordinate system would allow each image from the population to be translated so a pinpoint alignment can be achieved. Choosing both disc and fovea allows corrections for translation, scale, and rotational differences across the population. However, nonlinear shape variations across the population would not be considered—which can be accomplished when the vascular arch information is utilized. The end of the arches can be defined as the first major bifurcations of the arch branches. The arch shape and orientation vary from individual to individual and influence the structure of the remaining vessel network. Establishing an atlas coordinate system that incorporates the disc, fovea and arches allows for translation, rotation, scaling, and nonlinear shape variations to be accommodated across a population. An isotropic coordinate system for the atlas is desirable so images can refer to the atlas independent of spatial pixel location by a linear one-to-one mapping. The radial-distortion-correction (RADIC) model [92] attempts to register images in a distortion-free coordinate system using a planar-to-spherical transformation, so the registered image is isotropic under a perfect registration, or quasi-isotropic allowing low registration error. As shown in Fig. 11, the fundus curvature can be represented in the registration result using the quadratic model, while the RADIC-based registration unfolds the curvature to put the registered image in an isotropic coordinate system. An isotropic atlas makes it independent of spatial location to map correspondences between the atlas and test image. The intensities in overlapping area are determined by a distance-weighted blending scheme [93]. Registration of fundus image pair using (a) quadratic model and (b) RADIC model. Vessel center lines are overlaid for visual assessment of registration accuracy. This registration is performed to disk-centered and macula-centered images to provide an increased anatomic field of view. 1) Landmark StatisticsRetinal images in clinical practice are acquired under diverse fundus camera settings subjected to saccadic eye movement; and with variable focal center, zooming, tilting, etc. Thus, atlas landmarks from training data need to be aligned to derive any meaningful statistical properties from the atlas. Since the projective distortion within an image is corrected during the pairwise registration, the inter-image variations in the registered images appear as the difference in the rigid coordinate transformation parameters of translation, scale and rotation. For each pair of landmarks pdi and pfi, where i = 0, 1, …, N − 1 and N = 500, pinning all pdi landmarks to an arbitrary point (μd) clears the translation. The centroid of the point cloud formed by pfi landmarks is evaluated to get the fovea atlas location (μf) so every pfi can be aligned to μf using the similarity transformation to remove the inter-image variations in scale and rotation. The steps of rigid coordinate alignment for each parameter are illustrated in Fig. 12. Consequently, an aligned pixel position p′ is determined using p as Registration of anatomic structures according to increasing complexity of registration transform—500 retinal vessel images are overlaid and marked with one foveal point landmark each (red spots). Rigid coordinate alignment by (a) translation, (b) translation and scale, and (c) translation, scale, and rotation. where si, θi, and Ti are the differences in scale, rotation and translation measured between point pairs of and (pdi, pfi), and (μd, μf), the rotation matrix ℛ(θ) is defined as [cos(thetas;) sin(thetas;); − sin(thetas;) cos(thetas;)]. 2) Coordinate MappingThe atlas landmarks serve as the reference set so each color fundus image can be mapped to the coordinate system defined by the landmarks. As the last step of atlas generation, color fundus images are warped to the atlas coordinate system so that the arch of each image is aligned to the atlas vascular arch. A thin-plate-spline (TPS) [94] is used in this method for mapping retinal images to the atlas coordinate system. Rigid coordinate alignment as described above is done for each fundus images to register the disc center and the fovea. The seven control points required for TPS are determined by sampling points from equidistant locations in radial directions centered at the disc center. Consequently, 16 (1 at disc center, 1 at fovea, and 2 × 7 on vascular arch) control points are used to calculate the TPS. Usually, the sampling uses smoothed trace lines utilizing third order polynomial curve fitting because naive traces of vascular arch lines could have locally high tortuosity, which may cause large geometric distortions by TPS. Fig. 13 illustrates retinal image mapping process by TPS—the vessel main arch that runs along the naive trace (yellow line) is mapped onto the atlas vessel arch (green) line. Atlas coordinate mapping by TPS: (a) before and (b) after mapping. Naive main arch traces obtained by Dijkstra’s line-detection algorithm are drawn as yellow lines that undergo polynomial curve fitting to result in blue lines. Atlas landmarks (disc center, fovea, and vascular arch) are drawn in green, and equidistant radial sampling points marked with dots. 3) Using Retinal Atlas to Detect Grossly Abnormal RetinaBy creating a retinal atlas using this method, the atlas can be used as a reference to quantitatively assess the level of deviation from normality. An analyzed image can be compared with the retinal atlas directly in the atlas coordinate space. The normality can thus be defined in several ways depending on the application purpose—using local or global chromatic distribution, degree of vessel tortuosity, presence of pathological features, presence of artifacts, etc. Fig. 14 shows an example application driven by a retinal atlas, the region where imaging artifacts are present are highlighted. The atlas was created from 1000 color fundus images (two fields per left eye, from 500 subjects without retinal parthology or imaging artifacts). Example application of employing retinal atlas to detect imaging artifacts. (a), (c) Color fundus images with artifacts. (b), (d) Euclidean distance maps in atlas space using atlas coordinate system. Note that distances are evaluated within atlas image. Consequently, field of view of distance map is not identical to that of fundus image. Other uses for a retinal atlas include image quality detection and disease severity assessment. Retinal atlases can also be employed in content-based image retrieval leading to abnormality detection in retinal images [95]. D. Assessing Performance of Fundus Image Analysis AlgorithmsFundus lesion detection algorithms are primarily intended to perform automatically and autonomously. In other words, some retinal images may never be seen by a human expert. Consequently, high demands must be placed on the fundus lesion detection system since the performed diagnostic decisions may have vision-threatening consequences. Lesion detection systems are most commonly employed for diabetic retinopathy screening. In all such systems, a high level of confidence in the agreement between the system and expert human readers is required. In reality, the agreement between an automatic system and an expert reader may be affected by many influences—system performance may become impaired due to the algorithmic limitations, the imaging protocol, properties of the camera used to acquire the fundus images, and a number of other causes. For example, an imaging protocol that does not allow small lesions to be depicted and thus detected will lead to an artificially overestimated system performance if such small lesions might have been detected with an improved camera or better imaging protocol. Such a system then appears to perform better than it truly does if human experts and the algorithm both overlook true lesions. The performance of a lesion detection system can be measured by its sensitivity, a number between 0 and 1, which is the number of true positives divided by the sum of the total number of (incorrectly missed) false negatives plus the number of (correctly identified) true positives [77]. System specificity, also a number between 0 and 1, is determined as the number of true negatives divided by the sum of the total number of false positives (incorrectly identified as disease) and true negatives. Sensitivity and specificity assessment both require ground truth, which is represented by location-specific discrete values (0 or 1) of disease present/absent for each subject in the evaluation set. The location-specific output of an algorithm can also be represented by a discrete number (0 or 1). However, the output of the assessment algorithm is often a continuous value determining the likelihood p of local disease presence, with an associated probability value between 0 and 1. Consequently, the algorithm can be made more specific or more sensitive by setting an operating threshold on this probability value p. The resulting sensitivity/specificity pairs are plotted in a graph, yielding a receiver operator characteristics (ROC) curve [77], [96]. The area under the ROC curve (AUC, represented by its value Az) is determined by setting a number of different thresholds for p. Sensitivity and specificity pairs of the algorithm are obtained at each of these thresholds. The ground truth is of course kept unchanged. The algorithm behavior represented by this ROC curve can thus be reduced to a single number. The maximum value of AUC is 1, denoting a perfect diagnostic performance, with both the sensitivity and specificity being 1 (100% performance). While the AUC assessment of performance is highly relevant and covers the most important aspects of lesion detection behavior, this approach has a number of limitations, including its dependence on the quality of annotated datasets [69], [70] and on the underestimation of missing rare, but sight- or life-threatening abnormalities, as discussed in Section IV-B3. 1) Performance Comparison of Diabetic Retinopathy Detection Systems to That of Retinal SpecialistsSeveral groups have studied the performance of detection algorithms in a real world setting, i.e., when the systems are used on populations of patients with diabetes, not previously known to have diabetic retinopathy. The main goal of such a system is to decide whether the patient should be evaluated by a human expert or can return for followup, only involving automated analysis of retinal images [70], [71]. As mentioned previously, performance of the algorithm that placed first at the 2009 Retinopathy Online Challenge competition [97] was compared to that of a large computer-aided early DR detection project EyeCheck [59]. In this comparison, fundus photographic sets from 17 877 patient visits of 17 877 people with diabetes who had not previously been diagnosed with DR consisting of two fundus images from each eye were used for performance comparison. The fundus photographic set from each visit was analyzed by a single retinal expert and 792 of the 17 877 sets were classified as containing more than minimal DR (threshold for patient referral). The two algorithmic lesion detectors were applied separately to the dataset and compared by standard statistical measures. The area under the ROC curve was the main performance characteristic. The results showed that the agreement between the two computerized lesion detectors was high. Retinal exams containing more than minimal DR were detected with an AUC of Az = 0.84 by the Eyecheck algorithm and an AUC of Az = 0.82 for the ROC-2009 winner. This difference in AUC was not statistically significant (z-score of 1.91). If the detection output of these two algorithms were combined (at least one detection constituted a hit), the detection AUC increased to Az = 0.86, a value identical to the theoretically expected maximum [69]. At 90% sensitivity, the specificity of the EyeCheck algorithm was 47.7%. The specificity of the ROC-2009 winner algorithm was 43.6%. By comparison with interobserver variability of the employed experts, the study concluded that DR detection algorithms appear to be mature and further improvements in detection performance cannot be differentiated from the current best clinical practice because the performance of competitive algorithms has now reached the human intrareader variability limit [69]. Additional validation studies on larger, well-defined, but more diverse populations of patients with diabetes are urgently needed, anticipating cost-effective early detection of DR in millions of people with diabetes to triage those patients who need further care at a time when they have early rather than advanced DR, and such trials are currently underway in the U.S., U.K., and the Netherlands, though the results have not yet been disclosed. 2) Multilevel Approach to Lesion Detection: From Pixel to PatientAs outlined above, the retinal lesion detection algorithms operate at a broad range of levels according to the utilization of the detection algorithm outputs. Such a utility level is limited at one end by the finite resolution of the imaging device and at the other end by the feasibility of imaging that can be employed over a finite time (i.e., number of repeated image acquisitions on the same subject). At the lowest level, algorithms classify individual pixels, followed by groups of pixels (possibly representing lesions), areas (organs or organ structures) in images, and at even higher level, complete images, multiple images may form a subject-level exam, and finally—at the highest level—multifaceted analyses of individual subjects are attempted. At each such level, the probability of abnormality detection is frequently determined while relying on findings at previous lower levels. At the highest level the system may be diagnosing a single patient based on the fused information from all the lower-level contributions. Clearly, answering the ultimate question how to effectively fuse all such information is nontrivial. This subject was studied by Niemeijer et al. [98], and their approach involved application of multiple unsupervised and supervised analysis approaches that were compared in terms of performance at the patient level. A compound computer-aided retinal diagnostic system was developed that takes into account abnormalities of multiple types and at multiple levels, as well as the estimated confidence in individual analysis outcomes. A reliable analysis scheme was proposed based on a supervised fusion scheme for combining the output of the different components, and its performance evaluated on 60 000 images from 15 000 patients. The choice of the fusion system was identified as significantly influencing the overall system performance with simple fusion methods achieving classification performance associated AUC = 0.82 while the supervised fusion system reached an AUC = 0.89 [98]. 3) Role of Publicly Available and Comparative DatabasesTo drive the development of progressively better fundus image analysis methods, research groups have established publicly available, annotated image databases in various fields. Fundus imaging examples are represented by the STARE [72], DRIVE [73], REVIEW [99] and MESSIDOR databases [100], with large numbers of annotated retinal fundus images, with expert annotations for vessel segmentation, vessel width measurements, and diabetic retinopathy detection, as well as competitions such as the Retinopathy Online Challenge [97], some of which will be discussed in the following. A major inspiration for these online image databases and online competitions was the Middlebury Stereo Vision competition [101], [102]. 4) DRIVE—(Digital Retinal Images for Vessel Evaluation)The DRIVE database was established to enable comparative studies on segmentation of retinal blood vessels in retinal fundus images. It contains 40 fundus images from subjects with diabetes, both with and without retinopathy, as well as retinal vessel segmentations performed by two human observers. In one of the available images, high-contrast choroidal regions were also segmented because these can be easily confused with retinal vessels. Starting in 2005, researchers have been invited to test their algorithms on this database and share their results with other researchers through the DRIVE website [103]. At the same web location, results of various methods can be found and compared. An early comparative analysis of the performance of vessel segmentation algorithms was reported in [73] and by now, over 100 papers have been published using the DRIVE database as a benchmark. Currently, retinal vessel segmentation research is primarily focusing on improved segmentation of small vessels, as well as on segmenting vessels in images with substantial abnormalities. ROC—Retinopathy Online ChallengeThe DRIVE database was a great success, allowing comparisons of algorithms on a comparative dataset. In retinal image analysis, it represented a substantial improvement over method evaluations on unknown datasets. However, different groups of researchers tend to use different metrics to compare the algorithm performance, making truly meaningful comparisons difficult or impossible. Additionally, even when using the same evaluation measures, implementation specifics of the performance metrics may influence final results. Consequently, until the advent of the Retinopathy Online Challenge ROC competition in 2009, comparing the performance of retinal image analysis algorithms was difficult [97]. A logical next step was therefore to provide publicly available annotated datasets for use in the context of online, standardized evaluations asynchronous competitions. In an asynchronous competition, a subset of images is made available with annotations, while the remainder of the images are available with annotations withheld. This allows researchers to optimize their algorithm performance on the population from which the images were drawn (assuming the subset with annotated images is representative of the entire population), but they are unable to test–retest on the evaluation images, because those annotations are withheld. All results are subsequently evaluated using the same evaluation software and research groups are allowed to submit results continuously over time. Nevertheless, some groups may be tempted to artificially influence the performance outcome for example by using human readers to assist with the performance of their algorithm, or iteratively improving the performance by submitting multiple results serially and using the obtained performance differences to tune-up their algorithms. More recently, the concept of synchronous competitions was born, for which a deadline is given for submitting analysis results with competition results announced at a single moment in time. The most well-known example of such an approach is the Netflix competition [104]. These kinds of joint evaluations on a common dataset have the potential to steer future research by showing the failure modes of certain techniques and guide the practical application of techniques in the clinical practice, especially if appropriate reward mechanisms are available (again, the highly successful Netflix competition may serve as a motivational example). The first Retinopathy Online Challenge competition [105] focused on detection of microaneurysms and was organized in 2009. Twenty-six groups participated in the competition out of which six groups submitted their results on time, as published in [97]. One group decided to drop out of the competition after the results were announced, and the remainder allowed their performance to be discussed publicly [89], [106]–[108]. The results from each of the methods in this competition are summarized in Table I. Table ISensitivities of Different Methods in 2009 ROC Competition (Table II in [97]. All Microaneurysms From Test Set Were Included)
ROC-2009 DatasetsA set of 100 digital color fundus photographs were selected from a large dataset of over 150 000 images, acquired during diabetic retinopathy screening [59]. The inclusion criteria were that the screening program ophthalmologist had marked the image as containing microaneurysms and did not mark it as ungradable. Since multiple screening sites utilizing different fundus camera types were involved in the screening program, the images in the ROC-2009 set are quite heterogeneous. Three different sizes of field of view (FOV) are present in the dataset, each corresponding to different image resolution. The images were captured using either Topcon NW 100, Topcon NW 200, or Canon CR5-45NM camera, resulting in two differently shaped FOV’s. All images were made available in JPEG format with standard image compression levels set in the camera. Four retinal experts annotated all microaneurysms as well as all “don’t care” lesions in the 100 images. For the training set, a logical OR was used to combine the lesion locations annotated by the four experts—thus ensuring that the reference dataset was highly sensitive to lesions, as it required only one retinal expert to identify a lesion. The annotations were exported as a file in XML format that contained the center locations for all microaneurysms and all “don’t care” lesions in each image of the set. E. Optic Nerve Head Analysis From Fundus ImagesGlaucoma is the third leading cause of blindness in the U.S., characterized by gradual damage to the optic nerve and resultant visual field loss. Early diagnosis and optimal treatment have been shown to minimize the risk of visual loss due to glaucoma. The hallmark of glaucomatous progression is cupping of the optic nerve head. One way of determining the amount of cupping is planimetry by experienced glaucoma specialists from stereo color photographs of the optic nerve head. Pixel feature classification is an attractive technique to detect the cup and rim in stereo color photographs, as shown in the following paragraphs. 1) Pattern Recognition Approach to ONH SegmentationStereo photographs of the optic disc (cropped from 4096 × 4096 pixel resolution to 512 × 512, centered at the optic nerve head) were used to form a reference standard via manual planimetry of the stereo pairs, labeling each image pixel as one of cup, rim, and background in a majority-win manner according to the labeling of three expert glaucoma specialists (Fig. 15). In the case of a draw, the pixel was assigned to the background class [110]. Annotations of optic disc stereo pair by three expert glaucoma specialists. Note substantial inter-observer variability. ONH rim is shown in grayish and cup in whitish overlay on left image of stereo pair. Rightmost panel D shows a reference standard that was created from expert analyses A, B, C by majority voting with white color representing cup, gray color denoting rim, and black color corresponding to background. As mentioned in Section IV-A, pixel intensity and simple edge operators can be used to generate local image features. Considering spatial context, a Gaussian filter bank can be used to generate a large number of (sometimes correlated) features, from which an optimal feature combination can be identified using a formal feature selection process. Gaussian filter bank features are designed to be sensitive to edges and textures at different scales and orientations and are usually applied to gray-level image data (Fig. 16). To incorporate the color information that is inherently present in the fundus photographs, additional color-sensitive features can result from a Gaussian steerable filter bank applied to the hue, saturation, and brightness image representations together with assessing signal variance in the R, G, and B channels in small neighborhoods. Such features are obviously different from the color opponency model of primate color vision. Color opponency intensities, for dark-bright, red-green and blue-yellow opponency, that are substantially closer to primate color vision color opponency processing can be computed from each color image as follows [111]: where Lr, Lg, Lb are the red, green and blue channels of the analyzed image of the stereo pair. Ldb, Lrg, Lby are then each convolved with the respective Gaussian derivatives to obtain the Gaussian derivative-related features in the opponency space. Color opponency steerable Gaussian filter bank kernel examples. First row, from left to right shows dark-bright opponency kernels for 0th order, first-order 0° to local gradient, first-order 90° to local gradient, second-order 0° to local gradient, second-order 60° to local gradient, and second-order 120° to local gradient, at a scale of 32 pixels. Second row, same for scale of 64 pixels, and third row for scale of 128 pixels. Next three rows show identical information for blue-yellow opponency kernels, and last three rows show red-green kernels. Smaller scales not shown because they are difficult to depict. These kernel images represent responses of each of feature detectors to an impulse function. Note that true kernel colors are shown. When employing the above-described Gaussian steerable filter bank in dark-bright, red-green, and blue-yellow center-surround color opponency images, a performance improvement was achieved [110]. After feature selection and training, the selected image features were classified by k-NN and support-vector machine classifiers, yielding probability maps of pixel labeling of cup, rim, or background. The approach that was presented in detail in [110] achieved three-class segmentation (labeling) correctness of 88%. Fig. 17 shows how the gradings by glaucoma specialists, glaucoma fellows, and the algorithm visually compare to each other. Correlations of the calculated cup to disc ratios with the reference standard was r = 0.93 (95% CI, 0.89–0.96, n = 58). Classification of stereo pairs (left two columns) by glaucoma specialists (third column), three glaucoma fellows (columns 4–6), and automated pixel feature classification (right-most column). Rows from top to bottom: Small, medium, large disc excavation, and excavation with inferior notching. The ONH segmentation performance clearly indicated that pixel feature classification with biologically inspired features derived from color fundus images is a good starting point for the classification of the optic nerve head and likely also other 3-D structures of the retina. Importantly, it was shown that features benefiting from the understanding to the physiologic vision process outperform standard pixel features when segmenting the optic nerve head. 2) Three-Dimensional Reconstruction of ONH Shape From Stereo Color Fundus PhotographsClinicians have used stereo color fundus images of the optic disc for many decades. Three-dimensional shape of the optic nerve head provides the visible manifestation of optic nerve damage. To allow objective and quantitative analysis of the ONH, several groups have developed techniques to estimate its 3-D shape using stereo fundus images [112]. Estimation of 3-D shape from stereo images has been performed for decades [113]. By measuring the relative position differences or disparity of one or more corresponding patches or regions in the two stereo images, the underlying shape can be estimated [114]. The problem of identification of correspondences is usually solved by making a number of assumptions, such as that of the Lambertian reflectance, which assumes that image intensities of the same 3-D point are identical regardless of the view angle variations. Another commonly used assumption is that the 3-D surface is fronto-parallel to both image planes within a local neighborhood. Problems arise when these assumptions are violated. Consequently, the same 3-D point or patch may not have the exact same appearance due to depth variation relative to its surroundings and due to the different view angles. Repetitive patterns or lack of distinct features produce matching ambiguities. In contrast to computational processes deriving shape from stereo images of natural scenes, images of the eye often have low contrast and high noise caused by safety-based limitations on the amount of illumination that can be used. The images thus consist of low contrast regions without clear boundaries. Large imaging apertures can gather more light but simultaneously suffer from a small depth of focus. As a consequence, depth variations may cause different degrees of blur due to de-focusing of the camera. Similarly, an increase of exposure time may incur blur due to the motion of the eye. The shape estimate of the optic disc is formed from two sets of densely corresponding pixels (x1, y1) and (x2, y2) from the stereo fundus images I1 and I2, taken from two different view angles while simultaneously imaging the same eye (Fig. 18). The two image planes are horizontally displaced in a typical camera configuration. The horizontal disparity of a pair of correspondences d = x2 − x1 is inversely proportional to the depth of the associated 3-D point. Given the correspondences between the image point pairs, the disparity map D = d(x, y) is defined as a 2-D matrix satisfying Two examples of 3-D ONH surface reconstruction obtained from a stereo fundus pair and from 3-D OCT scan shown in two rows. From left to right (both rows): left and right fundus image centered at the optic disc. Shape estimate of optic nerve head surface represented as grayscale depth maps derived from OCT scan. Reference (left) image shown to correspond to OCT scan view. Shape estimate of optic nerve surface represented as grayscale depth maps derived from stereo fundus pair analysis. Reference (left) image shown to correspond to output from stereo fundus pair reconstruction. I1(x, y1) = I2(x + d(x, y1), y2), for (x, y1) ∈ I1 (8) where image I1 is usually referred to as the reference image I2 and image the matching image. The disparity map D contains depth information about the observed structure. If the disparity map D(x, y) of a pair of fundus images is found by dense-matching and plotted as a 3-D surface, the shape of the optic disc is recovered. For stereo fundus images with slowly varying texture, it is easier to associate a pair of globally matching regions since more prominent landmarks can be utilized, such as blood vessels and the optic disc. On the other hand, given a limited number of candidate correspondences and the deformations in order to achieve correct matches between such landmarks, detailed local information is frequently sufficient and more reliable to discern subtle differences among these candidates. This motivates a scale-space-based framework described as follows. Scale space consists of x − y image evolutions with the scale s representing a third dimension. Given a pair of images I1(x, y, sk) and I2(x, y, sk), a disparity map D(x, y, sk) is estimated at scale sk and then upscaled to D0(x, y, sk−1), which matches the stereo pair I1(x, y, sk−1) and I2(x, y, sk−1)at the higher scale sk−1. With constraints imposed by D0(x, y, sk−1), the disparity map evolves to the finer scale D(x, y, sk−1) while at each scale, certain features are selected as the salient ones. The scale-space-based representation of the image structure is intended to provide globally coherent solutions for the correspondence problem. Stereo images at different scales provide hierarchical features for correspondence identification while disparity maps along the scale dimension provide structure description in terms of point evolution paths and act as the regularization component. Constraints enforced by landmarks at a certain scale guide finer searches toward correct directions along those paths while the small additive noise is filtered out by a 2-D adaptive noise-removing Wiener filter, which can preserve depth edges during disparity evolution. The matching process involves assigning one label (disparity value) to each pixel in the reference image within a rather large disparity range. The scale-space-based approach essentially distributes this task to different scales so that at each scale the matching ambiguity is reduced significantly. This is important for noisy stereo fundus pairs with low texture density. The formulation is consistent with the perceptual grouping performed by the human visual system. The described approach to reconstruct ONH surface shape was evaluated on 30 pairs of stereo fundus images in comparison with ONH surface shapes derived from 3-D OCT imaging of the same subjects—OCT analysis was performed as described in Section V-D. The accuracy of each disparity map was measured by the root of mean squared (RMS) differences ERMS between the estimate d(x, y) derived from stereo fundus photographs and the depth information d*(x, y) obtained from OCT scans as ERMS=1N ∑x∑y[d(x,y)−d∗(x,y)]2 (9) where N is the total number of pixels. Both depth maps (stereo-fundus and OCT derived) were normalized to the same scale according to the depth of the cup and the OCT-derived surface served as the independent standard. In this relative manner, the average RMS error of the stereo-fundus surface reconstruction was 15.9 ± 8.8% of the cup depth when assessed over all 30 analyzed stereo fundus pairs (Fig. 19). Example of 3-D agreement between stereo-fundus-photography-derived (lower surface) and OCT-derived (upper surface, smoothed) 3-D reconstructions of ONH shape. F. Active Areas of Research in Fundus Image AnalysisMajor progress has been accomplished in many fundus image analysis tasks mentioned above. Current challenges, on which multiple research groups worldwide are actively working, include the following areas:
V. OCT Image AnalysisBecause of OCT’s relatively new presence in ophthalmic care compared to fundus photography, the use of image analysis techniques for the use of image analysis for processing OCT images has a shorter history. Nevertheless, it is a rapidly growing and important area, especially as spectral-domain OCT technology has enabled true 3-D volumetric scans of the retina to be acquired (see Fig. 20 for example scanning locations). With this ever-increasing wealth of image information, the importance of developing advanced image analysis techniques to maximize the clinically relevant information to be extracted is especially important. Nevertheless, the development of such advanced techniques can be challenging as OCT images are inherently noisy, thus often requiring the utilization of 3-D contextual information (Fig. 22). Furthermore, the structure of the retina can drastically change during disease (as illustrated in Figs. 27, 29, and 30). Here, we review some of the important image analysis areas for processing OCT images. We start with the segmentation of retinal layers (Section V-A1), one of the earliest, yet still extremely important, OCT image analysis areas. We then discuss techniques for flattening OCT images in order to correct scanning artifacts (Section V-A2). Building upon the ability to extract layers, we discuss use of thickness information in Section V-A3 and use of texture information in Section V-A4. This is followed by the segmentation of retinal vessels (Section V-B), which currently has its technical basis in many of the techniques used for segmenting vessels in fundus photography, but is beginning to take advantage of the 3-D information only available in SD-OCT. Utilizing both layer-based and texture-based properties to detect the locations of retinal lesions is described in Sections V-C and V-C1, with a 3-D-based approach for segmenting the boundaries of such lesions being described in Section V-C2. The ability to segment layers in the presence of lesions is described in Section V-C3. We finally describe approaches for segmenting structures of the optic nerve head in SD-OCT in Section V-D. Typical scanning locations (illustrated on center fundus photograph) of spectral-domain OCT scanning system: Macular volumetric scans (left, in yellow) which are centered on macula, and peripapillary volumetric scans (right, in green) which are centered on optic nerve head. Illustration of helpfulness in using 3-D contextual information in intraretinal layer segmentation process. (Top) Sequence of 2-D result on three adjacent slices within spectral-domain volume obtained using a slice-by-slice 2-D graph-based approach. Note the “jump” in segmentation result for third and fourth surfaces in middle slice. (Bottom) Sequence of 3-D result on same three adjacent slices using same graph-based approach, but with addition of 3-D contextual information. Three-dimensional contextual information prevented third and fourth surface segmentation from failing. Example of SEAD footprint detection. Panel (a) presents an x − z slice running through SEADs in SD-OCT volume. Expert standards for footprint of these SEADs and automatically generated SEAD footprint probability map, in x − y plane, are presented in panels (b) and (c), respectively. Note probability scale in panel (c). Projection of x − z slice in x − y plane is represented by a vertical line in (b) and (c). Location of SEADs visible in panel (a) are indicated by vertical lines in each panel. SEAD segmentation from 3-D OCT and SEAD development over time: top row: 0, 28, and 77 days after first imaging visit. Middle row: 0 and 42 days after first imaging visit. Bottom row: 0, 14, and 28 days after first imaging visit. Three-dimensional visualization in right column shows data from week 0. Each imaging session was associated with anti-VEGF reinjection. Automated intraretinal layer segmentation approach in presence of SEADs. (a), (b) Zeiss Cirrus OCT image data—two perpendicular slices from 3-D volume. (c), (d) Automated layer/SEAD segmentation. (e) SEAD and layers in three dimensions. A. Retinal Layer Analysis From 3-D OCT1) Retinal Layer DetectionThe segmentation of retinal layers in OCT scans has been an important goal since OCT’s availability to the ophthalmic community [36] as thickness changes in the layers are one indication of disease status (Section V-A3). An example of the visible layers in one slice from a spectral-domain OCT macular volume is given in Fig. 21. With the previous-generation time-domain scanning systems (such as the Stratus OCT by Carl Zeiss Meditec, Inc.), commercial systems offered the ability to segment and provide thickness measurements for one layer of the retina. In particular, the retinal nerve fiber layer (RNFL) thickness measurements of peripapillary circular scans were frequently used for the disease status/progression/regression assessment in glaucoma patients, while the total retinal thickness measurements were often used in the assessment of patients with macular edema, choroidal neovascularization and macular hole. While the proprietary details of commercial approaches are not known, it can be assumed that they utilized an inherently 2-D approach (i.e., if multiple 2-D slices are available in a particular scanning sequence they are segmented independently). Segmentation results of 11 retinal surfaces (ten layers). (a) X-Z image of OCT volume. (b) Segmentation results, nerve fiber layer (NFL), ganglion cell layer (GCL), inner plexiform layer (IPL), inner nuclear layer (INL), outer plexiform layer (OPL), outer nuclear layer (ONL) + inner segments (IS), outer segments (OS), and retinal pigment epithelium complex (RPE+). Stated anatomical labeling is based on observed relationships with histology although no general agreement exists among experts about precise correspondence of some layers, especially outermost layers. (c) Three-dimensional rendering of segmented surfaces (N: nasal, T: temporal). Similarly, most of the early approaches reported in the literature [115]–[121] for the segmentation of time-domain scans were two dimensions in nature; however, some of the approaches did move towards segmenting additional layers of the retina. While variations to each of the early 2-D approaches exist for the segmentation of retinal boundaries, a typical 2-D approach proceeds as follows.
Other 2-D time-domain approaches included the use of 2-D dynamic programming by Baroni et al. [122] and manually initialized deformable models for the segmentation of fluid-filled regions by Cabrera Fernández [119]. These segmentation approaches have attempted to find different numbers of boundaries of the retina. In particular, Koozekanani et al. [115], [116] found two boundaries, Baroni et al. [122] found three, Shahidi et al. found four [121], Ishikawa et al. [117], [118] found five, and Cabrera Fernández found seven retinal layer boundaries [120]. Haeker/Garvin et al. [123]–[126] reported the first true 3-D segmentation approach for the segmentation of retinal layers on OCT scans, thus taking advantage of 3-D contextual information. In addition to segmenting the layers in three dimensions, their approach was unique in that the layers could be segmented simultaneously [127]. The approach involved the construction of a graph such that the minimum-cost closed set of this graph would correspond to a set of optimal feasible surfaces. Feasibility was defined by smoothness constraints (i.e., not allowing neighboring surface points to be too far apart) and surface interaction constraints (i.e., requiring the distances between surfaces to be in a particular range). Their initially reported approaches utilized edge-based cost functions and constant feasibility constraints [123], [126], but they later extended the graph-based approach to allow for true regional information [125] and varying feasibility constraints [124]. For time-domain macular scans, they segmented six to seven surfaces (five to six layers), obtaining an accuracy and reproducibility similar to that of retinal specialists. This work was then extended to work with spectral-domain OCT volumes [128], making the utilization of 3-D contextual information more advantageous. The importance of 3-D contextual information is illustrated in Fig. 22. The originally published method was validated on repeated scans from macula-centered SD-OCT volumes of normal subjects and did suffer from relatively long processing times of hours per volume. The processing time has substantially decreased by employing a multiscale approach to a few minutes while segmenting additional layers (Fig. 21) [129]. A similar approach for segmenting the intraretinal layers in ONH-centered SD-OCT volumes was reported with an accuracy similar to that of the inter-observer variability of two human experts [130]. Based on their graph-theoretic approach, a preliminary layer thickness atlas was built from a small set of normal subjects [131] and unique layer changes were demonstrated in diabetes subjects [132], [133]. Other reported approaches for the segmentation of spectral-domain OCT volumes or slices include an interactive 3-D approach by Fuller et al. [134], the 2-D segmentation of slices from rodent images using an active contour approach by Yazdanpanah et al. [135], and the 2-D segmentation of slices from rodent images using a dynamic programming method [136] by Mishra et al. [137]. 2) OCT Image FlatteningSD-OCT volumes frequently demonstrate motion artifacts in the slow-scanning direction causing a high-frequency ripple in this direction. Furthermore, other artifacts may be present, such as the tilting due to an off-axis placement of the pupil. Thus, various approaches have been reported in an attempt to correct these artifacts. Common approaches for reducing these artifacts include 1-D and 2-D methods that use cross-correlation of either A-scans [117] or B-scans [134], [138], [139]. In some cases, a complete flattening of the volume is desired based on a surface segmentation to ensure a consistent shape for segmentation (to aid in learning shape constraints) and visualization purposes. In addition, flattening the volumes makes it possible to truncate the image substantially in the axial direction (z-direction), thereby reducing the memory and time-requirements of an intraretinal layer segmentation approach. For example, in the approach by Garvin et al. [128], flattening an image involved first segmenting the retinal pigment epithelial surface in a lower resolution, fitting a (regularized) thin-plate spline [140] to this surface, and then vertically realigning the columns of the volume to make this surface completely flat. However, in order to avoid a strong dependence on the surface segmentation, a relatively large regularization constraint was used. This had the side effect of not completely removing many of the ripples observed in the slow-scanning direction. In an effort to better reduce the ripples observed in the slow-scanning direction, Antony et al. proposed a modification of this approach [141] to separately address the distortions in each direction using a two-stage flattening approach. 3) Retinal Layer Thickness AnalysisOnce the layers are segmented and flattened, the properties of the macular tissues in each of these layers can be extracted and analyzed. Currently, the most common property analyzed is layer thickness, although more recent work has been proposed for analyzing textural properties as well (Section V-A4). For example, in glaucoma, one can observe a thinning of the retinal nerve fiber layer and ganglion cell layer [142]. In other ocular diseases, thickening of specific layers can be important (e.g., macular edema) [143]. Typically, it is useful to compare the obtained thickness values to a normative database or atlas, as is available in commercial machines for the total macular thickness and the retinal nerve fiber layer. However, a normative atlas for all the layers in three dimensions currently only exists within individual research groups [131]. Nevertheless, work has been done to demonstrate previously unknown changes in the layers in other diseases such as diabetes [132], [133]. 4) Retinal Texture AnalysisIn addition to assessing layer thickness and its variations, texture is well suited to characterize tissue properties and tissue differences. For example, textural properties may be important for assessing changes in the structural or tissue composition of layers that cannot be measured by changes in thickness alone. Using texture can also be helpful for detecting and segmenting structures involving changes in texture, such as retinal lesions (Section V-C). Texture can be determined in each of the identified layers either globally or regionally. To capture the 3-D character of the retinal tissue layers, 3-D texture analysis may include features describing intensity level distribution, run length indices, co-occurrence matrix measures, and wavelet analysis measures [144]–[146]. Three-dimensional formulations of these texture descriptors were previously developed for pulmonary parenchymal analysis [147] and can be directly employed for OCT texture analysis [148]. The intensity level distribution measures may contain the mean, variance, skewness, kurtosis, and gray level (intensity) entropy and are used to describe the occurrence frequencies of all intensity levels in a subvolume of interest. The run length measures may include the short run emphasis, long run emphasis, gray level nonuniformity, run length nonuniformity, and/or run percentage and describe the heterogeneity and tonal distributions of the intensity levels in a subvolume of interest. Frequently utilized for texture characterization, co-occurrence matrix measures like the angular second moment, correlation, contrast, entropy, inertia, and inverse difference moment describe the overall spatial relationships that the intensity tones have to each other, again calculated in a subvolume of interest. Run length and co-occurrence analyses both require quantifying voxel intensities in the OCT images. That is obvious for run length measures because the concept of uniform intensity sequences is ill-defined without quantification in the presence of noise, in particular the laser speckle. The gray-level intensities are typically quantified in equally populated bins [148]. The wavelet transform has been widely used in OCT images for denoising and despeckling [149]–[151] as well as for texture analysis [152]. Early work on 3-D wavelet analysis of OCT images was reported in [148] and was based on a computationally efficient yet flexible non-separable lifting scheme in arbitrary dimensions [153]. An adaptive implementation of this wavelet transform has previously been applied to 2-D texture retrieval [146]. For some of the textural features described above (run length, co-occurrence matrix and wavelet indices), features are computed along all main directions. In order to reduce the cardinality of the textural characterization, these values are often averaged to form directionless scalar features [145]. Additional (not strictly texture-based but still useful) features can be computed, e.g., the averages and standard deviations of layer thicknesses, globally or regionally. When assessing regional texture features, texture analysis can be performed in individual (flattened) layer-specific sub-volumes, for example with a square base domain of Sx × Sy centered at an (x, y) location. Texture characteristics can be computed for each segmented layer, several adjacent layers, or in layer combinations. Fig. 23 shows a schematic view of calculating regional texture indices. Geometry of textural characterization of macula. Local textural or thickness indices are extracted within intersection of region-defining columns (typically with a rectangular support domain in x − y plane) with each segmented intraretinal layer. Features computed in each of these intersections may be used to define an abnormality index for (x, y) line at center of the column when detecting macular lesions as described in Section V-C. B. Detection of Retinal Vessels From 3-D OCTWhile perhaps not obvious from observing a single 2-D slice (B-scan), it is possible to segment the retinal vasculature in 3-D SD-OCT volumes [154]–[156], which, among other uses, helps to enable the OCT-to-fundus and OCT-to-OCT image registration (Section VI). The absorption of light by the blood vessel walls causes vessel silhouettes to appear below the position of vessels, which thus causes the projected vessel positions to appear dark on either a full projection image of the entire volume [156] or a projection image from a segmented layer for which the contrast between the vascular silhouettes and background is highest as proposed by Niemeijer et al. [154], [155]. In particular, the work by Niemeijer et al. used the layer near the retinal pigment epithelium (RPE) to create the projection image, as illustrated in Fig. 24(a) and (b). Vessels were segmented using a k-NN pixel classification approach using Gaussian filter banks to compute features [see Fig. 24(c) and (d)]. The performance of their automated method was evaluated for both optic nerve head (ONH) centered as well as macula-centered scans. The retinal vessels were successfully identified in a set of 16 3-D OCT volumes (eight ONH and eight macula centered) with high sensitivity and specificity as determined using ROC analysis, Az = 0.96. Example of spectral 3-D OCT vessel segmentation. (a) Vessel silhouettes indicate position of vasculature. Also indicated in red are slice intersections of two surfaces that delineate subvolume in which vessels are segmented (superficial retinal layers toward vitreous are at the bottom). (b) Two-dimensional projection image extracted from projected subvolume of spectral 3-D OCT volume. (c) Automatic vessel segmentation. (d) Vessel segmentation after postprocessing—removing disconnected pieces and connecting large segments. Xu et al. reported an approach for segmenting the projected locations of the vasculature by utilizing pixel classification of A-scans [156]. The features used in the pixel classification are based on a projection image of the entire volume in combination with features of the individual A-scans. Both of these reported prior approaches focused on segmenting the vessels in the region outside the optic disc region because of difficulties in the segmentation inside this region. For example, the projected neural canal opening (NCO) can often share similar features with vessels, thus causing false positives. Thus, Hu et al. [157] proposed a modified 2-D pixel classification algorithm to segment the blood vessels in SD-OCT volumes centered at the ONH, with a special focus on better identifying vessels near the NCO. They approached this task by incorporating presegmented NCO location (Section V-D2) information into the classification process, achieving a significant improvement over the approach previously reported by Niemeijer et al. in the NCO region. Given an initial 2-D segmentation of the projected vasculature, Lee et al. presented an approach for segmenting the 3-D vasculature in the volumetric scans [158] by utilizing a graph-theoretic approach (Fig. 25). However, one of current limitations of that approach is the inability to properly resolve the depth information of crossing vessels. Example 3-D vasculature segmentation result from OCT volumetric scan [158]. C. Detection of Retinal LesionsCalculated texture and layer-based properties as described in Section V-A4 can be used to detect retinal lesions either as a 2-D footprint [148] or in three dimensions (Section V-C2). Out of many kinds of possible retinal lesions, symptomatic exudate-associated derangements (SEADs) are of utmost interest in assessing severity of age-related macular degeneration, diabetic macular edema, and other diseases. Detection of drusen, cottonwool spots, areas of pigment epithelial atrophy, pockets of fluid under epiretinal membranes, etc., may be attempted in a similar fashion. In all such applications, it is very useful to determine the normal appearance of maculae so that comparison of tissue properties with proper consideration of age, ethnicity, gender, etc., can be performed. Such a normative texture atlas can be derived from a set of N OCT volumes from normal eyes. The distribution of each texture or other layer-based feature f across these N volumes, in the neighborhood of an (x, y) line (i.e., a vertical column) within the lth layer, may be defined by the average μf, x, y, l and the standard deviation σ f, x, y, l of the N feature values (one feature value per OCT volume). This representation is convenient since the local deviation d (x, y, l) between the feature value f (x, y, l) computed for a new sample and the normality can be expressed in terms of z-score d(x,y,l)=f(x,y,l)−μf,x,y,lσf,x,y,l. (10) Because the local distribution of a feature in one layer of the macula is defined by only two parameters (mean, standard deviation), the normal appearance of maculae as defined above can be derived from a relatively small set of images. More comprehensive atlases may include feature histograms and histogram-based statistical distances, but these require larger datasets to be built. Examples of calculated texture indices and their distributions over macula in 13 normal subjects—thus forming a normative pilot atlas of macular texture properties—are given in Fig. 26. These “normal atlas” values can be used for abnormality detection as described in the following. Normal appearance of three intraretinal layers (NFL, INL and OS, see Fig. 21) in feature space optimized for SEAD footprint detection. For each feature, a map of the average (standard deviation) of feature values across macula is displayed on left (right). Inertia (b) is correlated with thickness of layer (d). Note that standard deviations of wavelet coefficients (c) and entropy (e) are almost uniform (black) across macula in normal eyes. (a) Average intensity; (b) inertia (co-occurrence matrix); (c) standard deviation wavelet coefficients (level 1); (d) layer thickness; (e) entropy (co-occurrence matrix). Deviations From Normal Atlas of Texture AppearanceA straightforward solution to detect retinal image abnormalities may be based on computing the local deviations from the normal appearance of maculae at each location (x, y) in each layer l and selecting the areas where the absolute deviation is greater than a predefined cutoff (e.g., d (x, y, l) ≥ 1.98 considering the 95% confidence level) for at least one feature. More generally, in order to build an abnormality-specific detector, a classifier can be trained, the inputs of which may be the z-scores computed for relevant features [see (10)]. The comprehensive z-scores are appropriate since an abnormality may affect several layers in the neighborhood of a given location (x, y). The classifier-determined label associated with each column may reflect the percentage of the patch covered by the target abnormality with relevant features selected by one of the many available cross-validation and/or feature selection methods [159]–[161], effectively forming a probabilistic abnormality map. A crisp binary footprint for macular lesions (say, SEADs) can thus be obtained directly from a probability map, e.g., by simple probability thresholding, yielding lesion footprints. Of course, training can result in more sophisticated ways to classify abnormalities. For example, the threshold can be determined so that it minimizes the L1 distance between the expert-defined independent standard for lesion footprints and the thresholded probability maps among all the images in the training dataset. 1) SEAD Footprint DetectionIn age-related macular degeneration, in diabetic macular edema, and in other retinal diseases, SEAD lesions are reflective of disease status and their changes of disease progression or regression. With the availability of anti-vascular endothelial growth factor (VEGF) therapy, assessment of the extent and morphology of individual SEADs is expected to contribute to patient-specific therapy. While SEADs are inherently 3-D, determining their 2-D retinal footprint (SEAD projection onto a coronal retinal plane) is highly relevant. Following the above-described analysis building blocks, SEAD footprint detection can be built on generalization of properties derived from expert-defined SEAD examples. Utilizing the differences between normal regional appearance of retinal layers as described by texture descriptors and other morphologic indices, a classifier can be trained to identify abnormal retinal appearance. As described in detail in [148], the SEAD detection starts with 3-D OCT layer segmentation (Section V-A1) resulting in ten intraretinal layers plus an additional artificial layer below the deepest intraretinal layer so that subretinal abnormalities can also be detected. Texture-based and morphologic descriptors are calculated regionally in rectangular subvolumes, the most discriminative descriptors are identified, and these descriptors are used for training a probabilistic classifier. The performance of a (set of) feature(s) is assessed by calculating the area under the receiver-operating characteristic curve of the SEAD classifier. The parameter of the ROC curve is the SEAD probability measured for OCT columns with an S × S square base from the inner validation set (per-patch AUC). Using the identified set of best features evaluated in OCT columns with an S × S square base from the inner validation set (per-patch AUC), a forward feature selection procedure is performed, in which features are sequentially selected until the AUC stops increasing. At each step, the feature maximizing the AUC increase is selected. All the feature vectors extracted from nonoverlapping S × S patches in the inner training set are used as reference samples by the k-NN classifier; their labels are derived from the expert standard. Overlapping S × S patches from the OCT volumes in the validation eye are then classified and the SEAD probability in each pixel (x, y) is defined as the average probability of all the patches containing (x, y) [84]. The cross-validation training/testing procedure is repeated for several patch sizes: S ∈ {10, 15, 20}. Once the probabilistic classifier is trained, SEAD-related probability is determined for each retinal location. In order to obtain a binary footprint for SEADs in an image input to the system, the probability map is thresholded and the footprint of the SEADs in this image defined as the set of all pixels with a probability greater than a threshold (Fig. 27). The threshold that minimizes the L1 distance between the expert standard for SEAD footprints and the thresholded probability maps among all the images in the reference dataset is selected (as outlined above). This approach was applied to determine SEAD footprints in 78 SD-OCT volumes from 23 repeatedly imaged patients with choroidal neovascularization, intra-, and sub-retinal fluid and pigment epithelial detachment (Fig. 28) [148]. The automated SEAD footprint detection method was validated against an independent standard obtained using an interactive 3-D SEAD segmentation approach. An area under the ROC curve of 0.961 ± 0.012 was obtained for the classification of vertical, cross-layer, macular columns. A study performed on 12 pairs of OCT volumes obtained from the same eye on the same day showed that the repeatability of the automated method was comparable to that of the human experts, demonstrating that useful 3-D textural information can be extracted from SD-OCT scans and—together with an anatomical atlas of normal retinas—can be used for clinically important applications. Repeatability study—two scans from same eye were acquired on same day at close temporal intervals. For each panel (a), (b), upper row shows binary SEAD footprint representing independent standard. Lower row shows SEAD footprints obtained by our automated method, gray levels represent probability of the point belonging to SEAD footprint; probability scale is provided in Fig. 27(c). These probabilities were thresholded to arrive at a binary segmentation. When varying threshold levels, obtained performance yields ROC curves discussed in text. (a) First scan and (b) second scan. 2) SEAD Segmentation in Three DimensionsComplete volumetric segmentation of SEADs from 3-D OCT is more difficult with no fully tested and robust methods existing at this time. A promising approach is based on identification of a seed point in the OCT dataset that is “inside” of a SEAD and a point “outside” of a SEAD. These seed points can be identified automatically from using a 3-D variant of the probabilistic classification approach outlined in the previous paragraphs. Once these two seed points are identified, an automated segmentation procedure that is based on regional graph-cut method [162], [163] may be employed to detect the SEAD volumetric region. The cost function utilized in a preliminary study was designed to identify darkish 3-D regions with somewhat homogeneous appearance. The desired properties of the SEAD region are automatically learned from the vicinity of the identified SEAD-region seed point. This adaptive behavior allows the same graph-cut segmentation method driven by the same cost function to reliably segment SEADs of different appearance. Fig. 29 gives an example of 3-D SEAD segmentations obtained using this approach. Note that the figure depicts the same locations in the 3-D data sets imaged several times in the course of anti-VEGF treatment. The surfaces of the segmented SEADs are represented by a 3-D mesh, which can be interactively edited to maximize SEAD segmentation accuracy in difficult or ambiguous cases. 3) Intraretinal Layer Segmentation in Presence of Retinal LesionsTo consider layer abnormalities and interruptions when segmenting multiple intraretinal layers in retinal OCT images in the presence of lesions (say, SEADs), a two-step approach is necessary in which layers are initially segmented not considering the lesion presence, then SEADs are segmented as described in Section V-C2, and the segmented SEADs are used in the subsequent step to constrain the second stage of layer segmentation. This process yields well-segmented retinal layers (Fig. 30) when SEADs occupy a single intra-retinal layer as well as in situations when the SEAD resides in several adjacent retinal layers. D. Optic Nerve Head Analysis Using 3-D OCTAs mentioned previously (Section IV-E), the optic nerve head is an important structure in the assessment of ophthalmic diseases, such as glaucoma. While stereo fundus photography provides the ability to extract some 3-D shape information of the optic nerve head (Section IV-E2), SD-OCT provides true 3-D information. Nevertheless, what 3-D structural information to be extracted from these 3-D volumes of the optic nerve head is still an active area of research. Thus, most current approaches have their basis in known 2-D parameters, such as the optic disc and cup as can be visualized/segmented from stereo fundus photography. In the following paragraphs, we describe two approaches for segmenting the disc/cup in optic-nerve head 3-D OCT image data: 1) a pixel classification approach applied to depth-columns of OCT voxels in which the reference standard is defined by manual planimetry from stereo fundus photographs [164]–[166], and 2) direct segmentation of structures (neural canal opening and cup) from 3-D OCT images using a graph-theoretic approach [167]–[170]. 1) Classification-Based Approach to ONH SegmentationThe classification-based approach starts by segmenting four intraretinal surfaces in the original spectral-domain OCT volume using a multiscale 3-D graph search-based method (Section V-A1). To obtain a consistent ONH shape, the retina in the original spectral-domain OCT volume is flattened by adjusting A-scans up and down in the z-direction using the segmented second intraretinal surface (Section V-A2). An OCT projection image is created by averaging in the z-direction the OCT subvolume between the second and fourth intraretinal surfaces. The flattened OCT volume and intraretinal surface segmentations, OCT projection image and vessel probability map from the OCT projection image (Fig. 24) [154], [155] are used as features for the classification of the optic disc cup and neuroretinal rim. The optic disc cup and neuroretinal rim are segmented by a contextual k-NN classifier incorporating neighboring A-scans. Finally, prior knowledge about the shapes of the optic disc cup and neuroretinal rim regions is incorporated through the application of convex hull-based fitting. OCT Projection ImageAn OCT projection image is necessary for creating the ONH independent standard. The retinal vasculature is visible in the projection image as described in Section V-B and [154], [155]. Feature points derived from the vasculature such as bifurcations can be used to register the fundus image with the OCT volume (Section VI-B). The projection image also serves for calculation of local features for the cup and rim classification of the OCT volume. The OCT projection image is created by averaging in the z-direction the OCT subvolume between the second and fourth intraretinal surfaces segmented in Section V-A1 (surfaces 2 and 4, see Fig. 31). These two surfaces define a layer that, due to its position in the retina and high contrast with the retinal background, contains a large number of high contrast vessel silhouettes [154], [155].
Intraretinal surface segmentation. (a) Original ONH-centered OCT volume. (b) Smoothed OCT volume. (c) Intraretinal surface segmentation result overlaid on original OCT volume. Search space for surfaces are constrained by previously segmented surfaces in multiresolution fashion. (d) Three-dimensional rendering of four segmented intraretinal surfaces. Regions of surfaces 2, 3, and 4 around the optic nerve head were ignored since intraretinal surfaces are ambiguous in these regions.
Optic Disc Cup and Neuroretinal Rim SegmentationTo segment the optic disc cup and rim, a supervised classification method assigns one of three labels (i.e., background, cup, rim) to each A-scan (voxel column) in the 3-D OCT scan (Fig. 32). Classification is based on a set of features obtained from flattened OCT volumes and intraretinal surfaces, OCT projection images and vessel probability maps [154], [155] for each voxel column in the OCT volume. Acquisition of ONH ground truth of spectral-domain OCT scan. (a) One of a pair of stereo color photographs. (b) Optic disc ground truth of (a), which is manually segmented by glaucoma expert through planimetry on one (left) of the pair of stereo fundus photographs while viewing the pair through a stereo viewer. Optic disc cup is in white, and neuroretinal rim is in gray. (c) OCT projection image. (d) Fundus photograph (panel a) registered onto OCT projection image (panel c). (e) OCT projection image overlapped with ONH ground truth. Optic disc cup is in red, and neuroretinal rim is in green. Employed features include local morphologic properties of retinal layers surrounding the ONH, including depth of the optic disc cup, thickness of individual segmented layers, distance from the center of the optic disc cup (defined as the lowest point of the top segmented intraretinal surface), probability that the voxel column is part of a vessel, OCT projection image intensity, average intensity of all voxels in the voxel column, and average voxel column intensity histogram. Following feature-space normalization, a contextual nearest neighbor classifier [171] is used. For each voxel column, the classifier determines k nearest neighbors in the feature space and assigns the most common label amongst the nearest neighbors to the query voxel column (Fig. 33) [165]. Example of optic disc cup and neuroretinal rim segmentation. (a) OCT projection image. (b) Segmentation result using contextual k-NN classifier with convex hull-based fitting. (c) OCT projection image overlapped with reference standard. Optic disc cup is in red, and neuroretinal rim is in green. (d) OCT projection image overlapped with (b). To preserve the shapes of the optic disc cup and neuroretinal rim, a local fitting method using the convex hulls of the segmentation is employed to smooth the segmentation results for both the optic disc cup and neuroretinal rim. Figs. 33 and 34 demonstrate the performance of this approach. Overall, the approach reported in [165] achieved results not significantly different (p > 0.2) from the inter-observer variability of expert-analysis of the ONH cup and rim boundaries. In a leave-one-subject-out experiment on 27 optic nerve head-centered OCT volumes (14 right eye scans and 13 left eye scans from 14 patients), the unsigned errors for the optic disc cup and neuroretinal rim were 2.52 ± 0.87 pixels (0.076 ± 0.026 mm) and 2.04 ± 0.86 pixels (0.061 ± 0.026 mm), respectively. Example of ONH segmentation performance [unsigned error for the optic disc cup = pixels (0.038 mm) and unsigned error for the neuroretinal rim = pixels (0.026 mm)]. From top to bottom, left stereo color photograph, X-Z image at center of OCT volume and 3-D rendering of top intraretinal surface mapped with left stereo color photograph. (a) Without any overlap. (b) Overlapped with result from contextual k-NN classifier with convex hull-based fitting. Optic disc cup is in red and neuroretinal rim is in green. (c) Overlapped with reference standard. (d) Overlapped with manual segmentation from second observer. 2) NCO-Approach to ONH AnalysisUsing a voxel classification approach for automatically segmenting the clinical familiar glaucomatous parameters—the ONH rim and cup—directly from the SD-OCT volumes as described [164]–[166] has the ultimate limitation that the algorithm essentially mimics the subjective assessment of 2-D parameters by human experts. Since it is not based on objective, anatomical landmarks within the 3-D volumes, the optic disc margin does not overlap with a single constant anatomic structure in volumetric OCT. This is consistent with the recent comparisons of clinical and SD-OCT optic disc margin anatomy by Strouthidis et al. [172], [173]. They found that the varying combinations of the termination of Bruch’s membrane, border tissue, or the anterior scleral canal opening may manifest as the 2-D disc margin seen on photographs, depending upon the border tissue architecture and anatomy. With the wealth of volumetric information available from SD-OCT, it is likely that better parameters can be obtained for measuring glaucomatous change that move beyond what is possible using stereo fundus photography alone. A central requirement for the detection of the ONH structural change is a longitudinally stable zero-reference plane. As described by Strouthidis et al. [172], [173], the neural canal opening (NCO)—the termination of Bruch’s membrane/retinal pigment epithelium (BM/RPE) complex can serve as a basis for a stable reference plane from which various optic nerve morphometric parameters can be derived, based on the assumption that the NCO is not likely to change substantially with glaucomatous progression [172], [173]. Thus, new efforts are directed towards segmenting structures such as the NCO directly within SD-OCT volumes. For example, Hu et al. [170] recently reported a method for segmenting the NCO directly in SD-OCT volumes. Their approach is based on first creating a projection image at the level of the BM/RPE plane and then simultaneously detecting the NCO and cup margins at the level of this plane using a graph-theoretic approach. Using the disc margin as defined by three glaucoma experts on stereo fundus photography as the reference standard (RS), they found mean unsigned and signed border differences of 2.81 ± 1.48 pixels (0.084 ± 0.044 mm) and −0.99 ± 2.02 pixels (−0.030 ± 0.061 mm, respectively. The correlations of the linear cup-to-disc (NCO) area ratio, disc (NCO) area, rim area, and cup area of the algorithm with the RS were 0.85, 0.77, 0.69, and 0.83, respectively. However, it is important to note that it was not expected that the projected NCO positions would perfectly correspond to the optic disc margin as defined on manual planimetry, as illustrated in Fig. 35. Example illustration of differences between structure-based segmentation of NCO/cup on OCT, glaucoma expert definition of optic disc margin and cup from manual planimetry, and pixel-classification-based segmentation of disc/cup on OCT. From top to bottom: raw SD-OCT and corresponding fundus image (top), structure-based (row 2), expert (on fundus photography) (row 3), and pixel-classification-based (bottom) segmentations overlapping with raw SD-OCT and corresponding fundus image. From left to right: SD-OCT central B-scan (left) and fundus image (right). Yellow arrows indicate position of NCO from algorithm (with dashed yellow line indicating projected NCO position). Blue arrows indicate clinical disc margin from RS. Green and red colors indicate each method’s projected rim and cup regions, respectively [170]. VI. Multimodality Retinal ImagingMultimodality imaging is becoming increasingly common in ophthalmology. For image information from multiple modalities to be usable in mutual context, images must be registered so that the independent information that was acquired by different methods can be concatenated and form a multimodality description vector. Thus, because of its importance in enabling multimodal analysis, retinal image registration reflects another active area of research. The several clinically used methods to image the retina were introduced above and include fundus photography, scanning laser ophthalmoscopy, fluorescence imaging, and OCT. Additional retinal imaging techniques such as hyper-spectral imaging, oxymetry, and adaptive optics SLO will bring higher resolution. To achieve a comprehensive description of retinal morphology and eventually function, diverse retinal images acquired by different or the same modalities at different time instants must be mutually registered to spatially combine all available local information. The following sections provide a brief overview of fundus photography and OCT registration approaches in both two and three dimensions. Registration of retinal images from other existing and future imaging devices can be performed in a similar or generally identical manner. A. Registration of Fundus Retinal PhotographsRegistration of fundus photographs taken either at different regions of the retina, or of the same area of the retina but at different times are useful to expand the effective field of view of a retinal image, determine what part of the retina is being viewed, or helps in analyzing changes over time [174]. We have discussed some other uses for fundus–fundus registration in Section IV-C devoted to retinal atlases. To register (2-D, planar) fundus images, most existing registration approaches utilize identification and extraction of features derived from retinal vasculature segmented separately from the individual fundus images. The choice of a specific image registration algorithm to align retinal images into a montage depends on the image characteristics and the application. Images acquired with only a small overlap may be optimally aligned using feature-based registration approaches, while images acquired with larger overlaps may be satisfactorily aligned using intensity-based approaches. Examples of feature-based registration are global-to-local matching [175], hierarchical model refinement [176], and dual-bootstrap [177]. Local intensity features [178] are particularly useful when an insufficient number of vascular features are available. Following a step of vascular skeletonization, vascular branching points can be easily used as stable landmarks for determining image-to-image correspondence [see Fig. 36(a) and (b)]. As an example, the RADIC model [179] parameters are estimated during an optimization step that uses Powell’s method [180] and is driven by the vessel center line distance (see also Section IV-C). The approach presented in [181] reported registration accuracy of 1.72 ± 0.04 pixels (25–30 μm, depending on resolution) when tested in 462 pairs of green channel fundus images. The registration accuracy was assessed as the vessel line error [see Fig. 36(c)]. The method only needed two correspondence points to be reliably identified and was therefore applicable even to cases when only a very small overlap between the retinal image pairs existed. Based on the identified vascular features, the general approach can be applied to any retinal imaging modality for which a 2-D vessel segmentation is available. Fig. 37 shows a wide-field of view retinal image constructed from eight individual fundus photographs. Example of fundus retinal image registration. (a) Detail of two fundus images with detected vessel centerlines. (b) Identified vessel landmarks. (c) Example registration result achieved on two overlapping fundus images. Retinal fundus image registration. Wide-angle fundus image is constructed by mutual registration of eight individual fundus photographs. In registering poor quality multimodal fundus image pairs, which may not have sufficient vessel-based features available, Chen et al. proposed the detection of corner points using a Harris detector followed by use of a partial intensity invariant feature descriptor (PIIFD) [182]. They reported obtaining 89.9% “acceptable” registrations (defined as registrations with a median error ≤ 1.5 pixels and a maximum error ≤ 10 pixels when compared with ground truth correspondences) when tested on 168 pairs of multimodal retinal images. B. Registration of OCT With Fundus Retinal PhotographsRegistration of 2-D fundus images with inherently 3-D OCT images requires that the dimensionality of OCT be reduced to two dimensions via z-axis projection. Building on the ability to obtain vascular segmentation from 3-D OCT projection images (Section V-B), the problem of fundus-OCT registration becomes virtually identical to that of fundus–fundus registration that was described in the previous section. Using the same general method, high-quality OCT-fundus registration can be achieved as demonstrated in Fig. 38. Fig. 39 presents the main steps of the registration process and shows the achieved registration performance. Registration of fundus images to 2-D OCT projection data. (a) Fundus camera image. (b) Two-dimensional projection (through depth dimension) of 3-D OCT data. (c) Registered and blended fundus-OCT images via application of affine transformation model with three identified vascular landmarks. Step-by-step process of registering fundus images to 2-D OCT projection data of the same subject. (a) Color fundus image. (b) Vascular segmentation in fundus image. (c) OCT projection image. (d) Vascular segmentation in OCT projection image. (e) ONH area and ONH center detected in fundus image. (f) Vascular center lines (blue) and bifurcations (red) in fundus image—bifurcations serve as prospective landmarks for which correspondence with OCT landmarks is determined in the next step. (g) ONH area and ONH center detected in OCT projection image. (h) Vascular centerlines (blue) and bifurcations (red) in OCT image—bifurcations serve as prospective landmarks for which correspondence with fundus landmarks is determined in the next step. (i) Highest reliability OCT-fundus corresponding landmarks identified in fundus image. (j) Highest reliability OCT-fundus corresponding landmarks identified in OCT image. (k) Registered OCT-fundus image—quality of registration shown in checkerboard image. (l) Registered OCT-fundus image—averaging-based blending used to construct image. C. Mutual Registration of 3-D OCT ImagesTemporal changes of retinal layers leading to assessment of disease progression or regression can be accessed from longitudinal OCT images. Similar to the cases discussed above, comparison of morphology or function over time requires that the respective OCT image data sets be registered. Since OCT is a 3-D imaging modality, such registration needs to be performed in three dimensions. For followup studies, image registration is a vital tool to enable more precise, quantitative comparison of disease status. Registration of time-domain and spectral-domain OCT images for longitudinal analysis of RNFL thickness measurement can be found in [183]. Another important aspect of OCT to OCT registration is the ability to enlarge retinal coverage by registering OCT data resulting from imaging different portions of the retina. A fully 3-D scale-invariant feature transform (SIFT)-based approach was introduced in [184]. In their work, the SIFT feature extractor locates minima and maxima in the difference of Gaussian scale space to identify salient feature points. Using calculated histograms of local gradient directions around each found extremum in three dimensions, the matching points are found by comparing the distances between feature vectors. An application of this approach to rigid registration of peripapillary (ONH-centered) and macula-centered 3-D OCT scans of the same patient for which the macular and peripapillary OCT scans had only a limited overlap was reported in [184]. The work built on a number of analysis steps introduced earlier, including segmentation of the main retinal layers and 3-D flattening of each of the two volumes to be registered (Section V-A2). Three-dimensional SIFT feature points were subsequently determined as described in [185]–[187]. Using the terminology usual for image registration when one of the registered images is called source (say the macular image) and the other target (say the peripapillary image), the feature point detection is performed in both the source and target images. After feature point extraction, those which are in corresponding positions in both images are identified. In a typical pair of two OCT scans, about 70 matching pairs can be found with a high level of certainty. Considering the already flattened 3-D OCT image pair, the major remaining deformations that need to be resolved are translation and limited rotation. Consequently, simple rigid or affine transform is appropriate to achieve the desired image registration. The transform parameters are estimated from the identified correspondence points. Fig. 40 demonstrates the functionality of such an approach to OCT-OCT registration of macular and peripapillary OCT scans [184], their 3-D registration achieved 3-D accuracy of 2.0 ± 3.3 voxels, assessed as an average voxel distance error in 1572 matched locations. Qualitative evaluation of performance demonstrated the utility of this approach to clinical-quality images. Temporal registration of longitudinally acquired OCT images from the same subjects can be obtained in an identical manner. Three-dimensional registration of macular and peripapillary OCT from the same subjects. Z-axis projection images of registered volumes are shown in left column. Representative depth-axis slices from volumes are shown on right to demonstrate registration performance in three dimensions. Location of displayed slice is indicated by a black line on registered projection images. Overlapping areas of scans are outlined by dashed rectangles to demonstrate that only relatively small regions of overlap existed. Within these rectangular patches, image data from both OCT images are shown intermittently in a checkerboard pattern to illustrate agreement of resulting registration. In projection images (same as in fundus photography), optic nerve head can be identified as a large dark region with vasculature emanating from that region while fovea can be identified as a small dark region centrally located in nonvascular region of the registered image. VII. Future of Retinal Imaging and Image AnalysisAs we have seen, translation of research in imaging and image analysis has been relatively rapid in the past and can be expected to be rapid in the future. This is partially explained by the lower capital expenditure for ophthalmic imaging devices compared to radiologic imaging devices, which can often be 10 to 100 times more expensive. It is affected by the fact that ophthalmologists manage patients directly and are also directly involved in the ordering and interpreting of images, while radiologists typically do not directly manage patients. The subtle difference in the physician–patient relationship leads to a more direct coupling between imaging innovation and clinical impact that is so well visible in ophthalmic imaging and analysis. Given that, it can be expected that translation of fundamental research findings in ophthalmology will remain rapid in the future. Realizing that the most rapid pace of medical imaging and image analysis progress in generally is seen in two scenarios: 1) when the amount of data to be analyzed and interpreted by a physician is excessively large and/or 2) when the analysis is complex and requires quantitation and thus is not well matched to the generally qualitative nature of human expert interpretation. Ophthalmologic applications and needs fall in both of these categories and the need to computerize and automate the image interpretation is correspondingly high. We expect the highest impact to be associated with the following areas.
We expect that image analysis and interpretation will be coupled to genetic and other assessment indices allowing truly personalized approaches to complex analyses of broad sets of patient-specific data. On the technological side, it will require and will thus lead to development and wide utilization of highly automated techniques for combined analysis of retinal image data in two, three and four dimensions (3-D+time), identification and quantitative assessment of temporal changes, including the assessment of local and/or systemic severity of the findings. On the patient-management side, it will therefore lead to broad utilization of semi-automated, clinician supervised management of retinal diseases, especially diabetic retinopathy, glaucoma, and choroidal neovascularization. Overall, we envision that within the next decade, the utilization of retinal imaging will go far beyond the direct needs of ophthalmic disease management. We expect that the retinal exam will become broadly used in systemic disease assessment both for patient specific care and for population studies. The unique ability of the eye to communicate the systemic status of the human body will be much more broadly utilized than today, with the obvious applications to diabetes mellitus and cardiovascular diseases likely coming first. Diseases like multiple sclerosis, Huntington’s disease, and a barrage of other brain diseases and neuropathies will likely follow at a fast pace. Retinal imaging and image analysis have developed rapidly over the past ten years, and image analysis is starting to play a crucial role in the care of patients with retinal diseases, as well as diseases that manifest in the retina. So far, image analysis has mostly operated reactively, i.e., waiting for what the newest image devices have as output and then trying to find approaches to analyze and quantify the image data. We expect that imaging device development and image analysis research will start to operate more in concert and that this paradigm is ready for prime time. We expect the image acquisition and subsequent image analysis to become closely integrated, so that image analysis successes and difficulties can directly influence device developers to focus on details that will help reliably analyze the images and vice versa. Ultimately, the presented overview of the ophthalmic imaging research and development in this field is driven by the overarching goal of preventing blindness and visual loss. We expect that this integrated development, in which a number of high-profile groups participate worldwide, will recognize the somewhat different needs of the developed as well as the developing world. Red lesion pixel feature classification. (a) Part of green color plane of a fundus image. Shown are pieces of vasculature and several red lesions. Bright lesions called exudates are also a symptom of DR. Circles mark location of some of the red lesions in the image. (b) After subtracting median filtered version of the green plane large background gradients are removed. (c) All pixels with a positive value are set to zero to eliminate bright lesions in the image. Note that exudates often partially occlude red lesions. Non-occluded parts of red lesions show up clearly in this image. An example of this is marked with a rectangle. (d) Pixel classification result produced by contrast enhancement step. Non-occluded parts of hemorrhages are visible together with the vasculature and a number of red lesions. AcknowledgmentsThis work was supported in part by the National Institutes of Health, NIBIB and NEI grants R01-EB004640, R01-EY018853, R01 EY019112, by the Research to Prevent Blindness, by the Institute for Vision Research, University of Iowa, by the Department of Veterans’ Affairs Center of Excellence for the Treatment and Prevention of Visual Loss, and by the Department of Veterans’ Affairs Merit program. When conducting multidisciplinary research like the one reviewed in this paper, many colleagues contribute to the work reported and to the way how material is presented. We would like to acknowledge the contribution of our colleagues, collaborators, and friends: B. Antony, M. Christopher, H. W. van Dijk, M. Dolejsi, J. Folk, P. Gupta, Z. Hu, C. Johnson, Y. H. Kwon, K. Lee, S. Lee, V. Mahajan, M. Niemeijer, R. H. Kardon, G. Quellec, J. M. Reinhardt, A. Rothova, S. R. Russell, T. Scheetz, L. Tang, F. D. Verbraak, J. Vinayak, A. Wahle, and X. Wu who spent numerous hours discussing ophthalmic imaging and image analysis with us over the course of many years. BiographiesMichael D. Abràmoff (M’03-SM’09) received the M.D. degree from the University of Amsterdam in 1994 and Ph.D. degree in 2001 from the University of Utrecht, The Netherlands. He is an Associate Professor of Ophthalmology and Visual Sciences at the University of Iowa, Iowa City, with joint appointments in Electrical and Computer Engineering and Biomedical Engineering. He is Associate Director of the U.S. Department of Veterans’ Affairs Center of Excellence for the Prevention and Treatment of Visual Loss. He serves on the American Academy of Ophthalmology and the Iowa Medical Society. When he is not seeing patients with retinal disease or teaching medical students, residents and fellows, he oversees a highly successful retinal image analysis research program. His focus is on automated early detection of retinal diseases, image guided therapy of retinal disease, and computational phenotype-genotype association discovery. He holds seven patents and patent applications in this field. Dr. Abràmoff is Associate Editor of IEEE Transactions on Medical Imaging. Mona K. Garvin (M’09) received the B.S.E. degree in biomedical engineering in 2003, the B.S. degree in computer science in 2003, the M.S. degree in biomedical engineering in 2004, and the Ph.D. degree in biomedical engineering in 2008, all from the University of Iowa, Iowa City. She is an Assistant Professor in the Department of Electrical and Computer Engineering, University of Iowa, Iowa City. She is also affiliated with the Iowa Institute for Biomedical Imaging and the Veterans’ Affairs Center of Excellence for the Prevention and Treatment of Visual Loss. Her research interests include medical image analysis and ophthalmic imaging. Milan Sonka (F’03) received the Ph.D. degree from the Czech Technical University in Prague, Czech Republic, in 1983. He is Professor and Chair of the Department of Electrical and Computer Engineering, Professor of Ophthalmology and Visual Sciences, and Radiation Oncology at the University of Iowa, Iowa City, and Co-director of the Iowa Institute for Biomedical Imaging. His research interests include medical imaging and knowledge-based image analysis with emphasis on cardiovascular, pulmonary, orthopedic, and ophthalmic image analysis. He is the first author of three editions of Image Processing, Analysis and Machine Vision (1993, 1998, 2008) and co-authored or co-edited 18 books/proceedings. He has published more than 90 journal papers and over 340 other publications. Dr. Sonka is an AIMBE Fellow. He is Editor in Chief of the IEEE Transactions on Medical Imaging and member of the Editorial Board of the Medical Image Analysis Journal. Contributor InformationMichael D. Abràmoff, Department of Ophthalmology and Visual Sciences, the Department of Electrical and Computer Engineering, and the Department of Biomedical Engineering, The University of Iowa, Iowa City, IA 52242 USA and the Veterans’ Administration. Mona K. Garvin, Department of Electrical and Computer Engineering, The University of Iowa, Iowa City, IA 52242 USA and the Center of Excellence for Prevention and Treatment of Visual Loss, Veterans’ Administration. Milan Sonka, Department of Electrical and Computer Engineering, the Department of Ophthalmology and Visual Sciences, and the Department of Radiation Oncology, The University of Iowa, Iowa City, IA 52242 USA. References1. Van Trigt AC. Trajecti ad Rhenum. 1853. Dissertatio ophthalmologica inauguralis de speculo oculi. [Google Scholar] 2. Kolb H. How the retina works. Amer Scientist. 2003;91(1):28–35. [Google Scholar] 3. Kolb H, Fernandez E, Nelson R, Jones BW. Webvision: Organization of the retina and visual system. 2005 [Online]. Available: http://webvision.med.utah.edu/ 6. NHIS. 2004–2006 Nat Health Interview Survey (NHIS) Nat. Center for Health Statistics, Centers for Disease Control and Prevention; 2007. [Online]. Available: www.cdc.gov/nchs/nhis.htm, Tech. Rep. [Google Scholar] 8. Klonoff DC, Schwartz DM. An economic analysis of interventions for diabetes. Diabetes Care. 2000 Mar;23(3):390–404. [PubMed] [Google Scholar] 9. Early Treatment Diabetic Retinopathy Study research group. Early photocoagulation for diabetic retinopathy. ETDRS Rep. 9. Ophthalmology. 1991;98(5 suppl):766–785. [PubMed] [Google Scholar] 10. Ciulla TA, Amador AG, Zinman B. Diabetic retinopathy and diabetic macular edema: Pathophysiology, screening, and novel therapies. Diabetes Care. 2003;26(9):2653–2664. [PubMed] [Google Scholar] 11. Otani T, Kishi S, Maruyama Y. Patterns of diabetic macular edema with optical coherence tomography. Amer J Ophthalmol. 1999;127(6):688–693. [PubMed] [Google Scholar] 12. Tso MO. Pathology of cystoid macular edema. Ophthalmology. 1982;89(8):902–915. [PubMed] [Google Scholar] 13. Lardenoye CV, Probst K, DeLint PJ, Rothova A. Photoreceptor function in eyes with edema. Invest Ophthalmol Vis Sci. 2000;41(12):4048–4053. [PubMed] [Google Scholar] 14. Diabetic Retinopathy Study Group. Photocoagulation treatment of proliferative diabetic retinopathy: Clinical application of DRS findings: DRS Rep. 8. Ophthalmology. 1981;88:583–600. [PubMed] [Google Scholar] 15. Diabetic Retinopathy Study Group. Factors influencing the development of visual loss in advanced diabetic retinopathy: DRS Rep. 9. Invest Ophthalmol Vis Sci. 1985;26:983–991. [PubMed] [Google Scholar] 16. Diabetic Retinopathy Study Group. Indications for photocoagulation treatment of diabetic retinopathy: DRS Rep. 14. Int Ophthalmol Clin. 1987;27:239–253. [PubMed] [Google Scholar] 17. Eye Diseases Prevalence Research Group. Prevalence of age-related macular degeneration in the united states. Arch Ophthalmol. 2004 Apr;122(4):564–572. [PubMed] [Google Scholar] 18. SMDS Investigators. Argon laser photocoagulation for senile macular degeneration—Results of a randomized clinical trial. Arch Ophthalmol. 1982;100:912–918. [PubMed] [Google Scholar] 19. Age-Related Eye Disease Study Research Group. A randomized, placebo-controlled, clinical trial of high-dose supplementation with vitamins C and E, beta carotene, and zinc for age-related macular degeneration and vision loss: AREDS Rep. 8. Arch Ophthalmol. 2001;119:1417–1436. [PMC free article] [PubMed] [Google Scholar] 20. Spaide RF, Laud K, Fine HF, Klancnik JM, Meyerle CB, Yannuzzi LA, Sorenson J, Slakter J, Fisher YL, Cooney MJ. Intravitreal bevacizumab treatment of choroidal neovascularization secondary to age-related macular degeneration. Retina. 2006;26:383–390. [PubMed] [Google Scholar] 21. Lalwani GA, Rosenfeld PJ, Fung AE, Dubovy SR, Michels S, Feuer W, Davis JL, Flynn HW, Jr, Esquiabro M. A variable-dosing regimen with intravitreal ranibizumab for neovascular age-related macular degeneration: Year 2 of the pronto study. Amer J Ophthalmol. 2009;148:43–58. [PubMed] [Google Scholar] 22. Tielsch JM, Katz J, Singh K, Quigley HA, Gottsch JD, Javitt J, Sommer A. A population-based evaluation of glaucoma screening: The Baltimore eye survey. Amer J Epidemiol. 1991;134(10):1102–1110. [PubMed] [Google Scholar] 23. Heijl A, Leske MC, Bengtsson B, Hyman L, Bengtsson B, Hussein M Early Manifest Glaucoma Trial Group. Reduction of intraocular pressure and glaucoma progression: Results from the early manifest glaucoma trial. Arch Ophthalmol. 2002;120(10):1268–1279. [PubMed] [Google Scholar] 24. Wong TY, Shankar A, Klein R, Klein BEK, Hubbard LD. Prospective cohort study of retinal vessel diameters and risk of hypertension. BMJ. 2004;329:799–800. [PMC free article] [PubMed] [Google Scholar] 25. Hubbard LD, Brothers RJ, King WN, Clegg LX, Klein R, Cooper LS, Sharrett AR, Davis MD, Cai J. Methods for evaluation of retinal microvascular abnormalities associated with hypertension/sclerosis in the atherosclerosis risk in communities study. Ophthalmology. 1999;106:2269–2280. [PubMed] [Google Scholar] 26. Flick CS. Centenary of Babbage’s ophthalmoscope. Optician. 1947;113:246. [PubMed] [Google Scholar] 27. Keeler CR. 150 years since Babbage’s ophthalmoscope. Arch Ophthalmol. 1997;115:1456–1457. [PubMed] [Google Scholar] 28. Halacy DS. Charles Babbage, Father of the Computer. London, U.K: Crowell-Collier Press; 1970. [Google Scholar] 29. von Helmholtz HLF. Beschreibung eines Augen-Spiegels. A Farstnerische Verlagsbuchhandlung; 1851. [Google Scholar] 30. Purkinje JE. Erstes Bandchen, Beitrage zur Kenntniss des Sehens in subjectiver Hinsicht. Prague, Austria; 1823. Beobachtungen und Versuche zur Physiologie der Sinne. [Google Scholar] 31. Gerloff O. Uber die Photographie des Augenhintergrundes. Klin Monatsblätter Augenheilkunde. 1891;29:397–403. [Google Scholar] 32. Gullstrand A. Neue methoden der reflexlosen ophthalmoskopie. Berichte Deutsche Ophthalmologische Gesellschaft. 1910;36 [Google Scholar] 33. Novotny HR, Alvis DL. A method of photographing fluorescence in circulating blood in the human retina. Circulation. 1961;24:82–86. [PubMed] [Google Scholar] 34. Allen L. Ocular fundus photography: Suggestions for achieving consistently good pictures and instructions for stereoscopic photography. Amer J Ophthalmol. 1964;57:13–28. [PubMed] [Google Scholar] 35. Webb RH, Hughes GW. Scanning laser ophthalmoscope. IEEE Trans Biomed Eng. 1981 Apr;28(4):488–492. [PubMed] [Google Scholar] 36. Huang D, Swanson EA, Lin CP, Schuman JS, Stinson WG, Chang W, Hee MR, Flotte T, Gregory K, Puliafito CA, Fujimoto JG. Optical coherence tomography. Science. 1991 Nov;254(5035):1178–1181. [PMC free article] [PubMed] [Google Scholar] 37. van Velthoven MEJ, Faber DJ, Verbraak FD, van Leeuwen TG, de Smet MD. Recent developments in optical coherence tomography for imaging the retina. Prog Retina Eye Res. 2007 Jan;26(1):57–77. [PubMed] [Google Scholar] 38. Matsui M, Tashiro T, Matsumoto K, Yamamoto S. A study on automatic and quantitative diagnosis of fundus photographs. I. Detection of contour line of retinal blood vessel images on color fundus photographs (author’s transl.) Nippon Ganka Gakkai Zasshi. 1973;77(8):907–918. [Online]. Available: PM:4594062. [PubMed] [Google Scholar] 39. Baudoin CE, Lay BJ, Klein JC. Automatic detection of microaneurysms in diabetic fluorescein angiography. Rev Epidemiol Sante Publique. 1984;32(3–4):254–261. [Online]. Available: PM:0006522738. [PubMed] [Google Scholar] 40. Sonka M, Hlavac V, Boyle R. Image Processing, Analysis, and Machine Vision. 3. Toronto, Canada: Thomson Engineering; 2008. (1st edition Chapman and Hall, London, 1993; 2nd edition PWS, Pacific Grove, CA, 1997) [Google Scholar] 41. Choma M, Sarunic M, Yang C, Izatt J. Sensitivity advantage of swept source and fourier domain optical coherence tomography. Opt Expr. 2003;11:2183–2189. [PubMed] [Google Scholar] 42. Abramoff MD, Kardon R, Vermeer K, Mensink M. A portable, patient friendly scanning laser ophthalmoscope for diabetic retinopathy imaging: Exudates and hermorrhages. ARVO Annu Meeting, Abstract. 2007 [Google Scholar] 43. Hammer DX, Ferguson RD, Ustun TE, Bigelow CD, Iftimia NV, Webb RH. Line-scanning laser ophthalmoscope. J Biomed Opt. 2006;11 [PubMed] [Google Scholar] 44. Kraats JVD, van Norren D. Directional and nondirectional spectral reflection from the human fovea. J Biomed Opt. 2008;13 [PubMed] [Google Scholar] 45. Delori FC, Pflibsen KP. Spectral reflectance of the human ocular fundus. Appl Opt. 1989;28(6):1061–1077. [PubMed] [Google Scholar] 46. Abramoff MD, Kwon YH, Ts’o D, Soliz P, Zimmerman B, Pokorny J, Kardon R. Visual stimulus-induced changes in human near-infrared fundus reflectance. Invest Ophthalmol Vis Sci. 2006;47(2):715–721. [Online]. Available: PM:16431972. [PMC free article] [PubMed] [Google Scholar] 47. Mujat M, Ferguson RD, Iftimia N, Hammer DX. Compact adaptive optics line scanning ophthalmoscope. Opt Expr. 2009;17 [PMC free article] [PubMed] [Google Scholar] 48. de Bruin DM, Burnes DL, Loewenstein J, Chen Y, Chang S, Chen TC, Esmaili DD, de Boer JF. In vivo three-dimensional imaging of neovascular age-related macular degeneration using optical frequency domain imaging at 1050 nm. Invest Ophthalmol Vis Sci. 2008;49:4545–4552. [PubMed] [Google Scholar] 50. Bresnick GH, Mukamel DB, Dickinson JC, Cole DR. A screening approach to the surveillance of patients with diabetes for the presence of vision-threatening retinopathy. Ophthalmology. 2000;107(1):19–24. [Online]. Available: PM:0010647713. [PubMed] [Google Scholar] 51. Kinyoun JL, Martin DC, Fujimoto WY, Leonetti DL. Ophthalmoscopy versus fundus photographs for detecting and grading diabetic retinopathy. Invest Ophthalmol Vis Sci. 1992;33(6):1888–1893. [Online]. Available: PM:0001582794. [PubMed] [Google Scholar] 52. American Acad. Ophthalmology Retina Panel. Preferred practice pattern: Diabetic retinopathy. 2003 [Online]. Available: www.aao.org/ppp. 53. Fong DS, Aiello L, Gardner TW, King GL, Blankenship G, Cavallerano JD, Ferris I, FL, Klein R. Diabetic retinopathy. Diabetes Care. 2003;26(1):226–229. [Online]. Available: PM:12502685. [PubMed] [Google Scholar] 54. Fong DS, Sharza M, Chen W, Paschal JF, Ariyasu RG, Lee PP. Vision loss among diabetics in a group model health maintenance organization (HMO) Amer J Ophthalmol. 2002;133(2):236–241. [Online]. Available: PM:11812428. [PubMed] [Google Scholar] 55. Wilson DF, Vinogradov SA, Grosul P, Vaccarezza MN, Kuroki A, Bennett J. Oxygen distribution and vascular injury in the mouse eye measured by phosphorescence-lifetime imaging. Appl Opt. 2005;44(25):5239–5248. [Online]. Available: PM:16149347. [PMC free article] [PubMed] [Google Scholar] 56. Lin DY, Blumenkranz MS, Brothers RJ, Grosvenor DM. The sensitivity and specificity of single-field nonmydriatic monochromatic digital fundus photography with remote image interpretation for diabetic retinopathy screening: A comparison with ophthalmoscopy and standardized mydriatic color photography. Amer J Ophthalmol. 2002;134(2):204–213. [Online]. Available: PM:12140027. [PubMed] [Google Scholar] 57. Williams GA, Scott IU, Haller JA, Maguire AM, Marcus D, McDonald HR. Single-field fundus photography for diabetic retinopathy screening: A report by the American academy of ophthalmology. Ophthalmology. 2004;111(5):1055–1062. [PubMed] [Google Scholar] 58. Lawrence MG. The accuracy of digital-video retinal imaging to screen for diabetic retinopathy: An analysis of two digital-video retinal imaging systems using standard stereoscopic seven-field photography and dilated clinical examination as reference standards. Trans Amer Ophthalmol Soc. 2004;102:321–340. [Online]. Available: PM:15747766. [PMC free article] [PubMed] [Google Scholar] 59. Abramoff MD, Suttorp-Schulten MS. Web based screening for diabetic retinopathy in a primary care population: The EyeCheck project. J Telemedicine E-Health. 2005;11:668–674. [PubMed] [Google Scholar] 61. Wong TY, Cheung N, Islam FM, Klein R, Criqui MH, Cotch MF, Carr JJ, Klein BE, Sharrett AR. Relation of retinopathy to coronary artery calcification: The multi-ethnic study of atherosclerosis. Amer J Epidemiol. 2008 Sep;167(1):51–58. [PubMed] [Google Scholar] 62. Early Treatment Diabetic Retinopathy Study research group. Photocoagulation for diabetic macular edema. Early treatment diabetic retinopathy study report number 1. Early treatment Diabetic Retinopathy Study research group. Arch Ophthalmol. 1985;103(12):1796–1806. [PubMed] [Google Scholar] 63. Kinyoun J, Barton F, Fisher M, Hubbard L, Aiello L, Ferris F. Detection of diabetic macular edema: Ophthalmoscopy versus photography—Early treatment diabetic retinopathy study (ETDRS) Rep. 5. Ophthalmology. 1989 Jun;96(6):746–750. [PubMed] [Google Scholar] 64. Early Treatment Diabetic Retinopathy Study research group. Fundus photographic risk factors for progression of diabetic retinopathy. ETDRS Rep. 12. Early treatment diabetic retinopathy study research group. Ophthalmology. 1991;98(5 Suppl):823–833. [PubMed] [Google Scholar] 65. Rosenfeld PJ, Brown DM, Heier JS, Boyer DS, Kaiser PK, Chung CY, Kim RY. Ranibizumab for neovascular age-related macular degeneration. New Engl J Med. 2006;355:1419–1431. [PubMed] [Google Scholar] 66. Brown DM, Kaiser PK, Michels M, Soubrane G, Heier JS, Kim RY, Sy JP, Schneider S ANCHOR Study Group. Ranibizumab versus verteporfin for neovascular age-related macular degeneration. New Engl J Med. 2006;355:1432–1444. [PubMed] [Google Scholar] 67. Boyer DS, Antoszyk AN, Awh CC, Bhisitkul RB, Shapiro H, Acharya NR MARINA Study Group. Subgroup analysis of the marina study of ranibizumab in neovascular age-related macular degeneration. Ophthalmology. 2007;114:246–252. [PubMed] [Google Scholar] 68. Kwon YH, Adix M, Zimmerman MB, Piette S, Greenlee EC, Alward WL, Abramoff MD. Variance owing to observer, repeat imaging, and fundus camera type on cup-to-disc ratio estimates by stereo planimetry. J Glaucoma. 2009;18:305–310. [PubMed] [Google Scholar] 69. Abramoff MD, Reinhardt JM, Russell SR, Folk JC, Mahajan VB, Niemeijer M, Quellec G. Automated early detection of diabetic retinopathy. Ophthalmology. 2010:1147–1154. [PMC free article] [PubMed] [Google Scholar] 70. Abràmoff MD, Niemeijer M, Suttorp-Schulten MSA, Viergever MA, Russell SR, van Ginneken B. Evaluation of a system for automatic detection of diabetic retinopathy from color fundus photographs in a large population of patients with diabetes. Diabetes Care. 2008 Feb;31(2):193–198. [PMC free article] [PubMed] [Google Scholar] 71. Philip S, Fleming AD, Goatman KA, Fonseca S, Mcnamee P, Scotland GS, Prescott GJ, Sharp PF, Olson JA. The efficacy of automated “disease/no disease” grading for diabetic retinopathy in a systematic screening programme. Brit J Ophthalmol. 2007 Nov;91(11):1512–1517. [PMC free article] [PubMed] [Google Scholar] 72. Hoover A, Kouznetsova V, Goldbaum M. Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response. IEEE Trans Med Imaging. 2000;19(3):203–210. [Online]. Available: PM:10875704. [PubMed] [Google Scholar] 73. Niemeijer M, Staal JS, van Ginneken B, Loog M, Abramoff MD. Comparative study of retinal vessel segmentation on a new publicly available database. Medical Imaging: Image Analysis. Proc. SPIE Medical Imaging; Bellingham, WA. SPIE; 2004. pp. 658–656. [Google Scholar] 74. Jelinek HF. Automated vessel segmentation of 35 mm colour non-mydriatic images in a community health screening project. MG&V. 2008;17(1):57–68. [Google Scholar] 75. Soares JVB, Le JJG, Cesar RM, Jelinek HF, Cree MJ. Retinal vessel segmentation using the 2-D Gabor wavelet and supervised classification. IEEE Trans Med Imaging. 2006 Oct;25(10):1214–1222. [PubMed] [Google Scholar] 76. Staal J, Abramoff MD, Niemeijer M, Viergever MA, van GB. Ridge-based vessel segmentation in color images of the retina. IEEE Trans Med Imaging. 2004 Apr;23(4):501–509. [Online]. Available: PM:15084075. [PubMed] [Google Scholar] 77. Sonka M, Fitzpatrick JM. Medical Image Processing and Analysis. Vol. 2. Wellingham, WA: Int. Soc. Optical Engineering Press; 2000. Handbook of Medical Imaging. [Google Scholar] 78. ter Haar Romeny BM. Front End Vision and Multi-Scale Image Analysis. Dordrecht: Kluwer; 2003. [Google Scholar] 79. Niemeijer M, van Ginneken B, Abramoff MD. Automatic determination of the artery vein ratio in retinal images. Medical Imaging: Image Analysis. Proc. SPIE Medical Imaging; Bellingham, WA. 2010. [Google Scholar] 80. Walter T, Klein JC, Massin P, Erginay A. A contribution of image processing to the diagnosis of diabetic retinopathy-detection of exudates in color fundus images of the human retina. IEEE Trans Med Imaging. 2002 Oct;21(10):1236–1243. [Online]. Available: PM:12585705. [PubMed] [Google Scholar] 81. Spencer T, Olson JA, McHardy KC, Sharp PF, Forrester JV. An image-processing strategy for the segmentation and quantification of microaneurysms in fluorescein angiograms of the ocular fundus. Comput Biomed Res. 1996;29(4):284–302. [Online]. Available: PM:8812075. [PubMed] [Google Scholar] 82. Frame AJ, Undrill PE, Cree MJ, Olson JC, Mchardy KC, Sharp PF, Forrester JV. A comparison of computer based classification methods applied to the detection of microaneurysms in ophthalmic fluorescein angiograms. Comput Biol Med. 1998;28:225–238. [PubMed] [Google Scholar] 83. Hipwell JH, Strachan F, Olson JA, McHardy KC, Sharp PF, Forrester JV. Automated detection of microaneurysms in digital red-free photographs: A diabetic retinopathy screening tool. Diabet Med. 2000;17(8):588–594. [PubMed] [Google Scholar] 84. Niemeijer M, van Ginneken B, Staal J, Suttorp-Schulten MSA, Abramoff MD. Automatic detection of red lesions in digital color fundus photographs. IEEE Trans Med Imaging. 2005 Apr;24(4):584–592. [PubMed] [Google Scholar] 85. Niemeijer M, van Ginneken B, Russell SR, Suttorp-Schulten MS, Abramoff MD. Automated detection and differentiation of drusen, exudates, and cotton-wool spots in digital color fundus photographs for early diagnosis of diabetic retinopathy. Invest Ophthalmol Vis Sci. 2007;45 [PMC free article] [PubMed] [Google Scholar] 86. Cree MJ, Olson JA, McHardy KC, Sharp PF, Forrester JV. A fully automated comparative microaneurysm digital detection system. Eye. 1997;11(Pt 5):622–628. [PubMed] [Google Scholar] 87. Gardner GG, Keating D, Williamson TH, Elliott AT. Automatic detection of diabetic retinopathy using an artificial neural network: A screening tool. Brit J Ophthalmol. 1996;80(11):940–944. [Online]. Available: PM:0008976718. [PMC free article] [PubMed] [Google Scholar] 88. Sinthanayothin C, Boyce JF, Williamson TH, Cook HL, Mensah E, Lal S, Usher D. Automated detection of diabetic retinopathy on digital fundus images. Diabet Med. 2002;19(2):105–112. [Online]. Available: PM:11874425. [PubMed] [Google Scholar] 89. Quellec G, Lamard M, Josselin PM, Cazuguel G, Cochener B, Roux C. Optimal wavelet transform for the detection of microaneurysms in retina photographs. IEEE Trans Med Imaging. 2008 Sep;27(9):1230–1241. [PMC free article] [PubMed] [Google Scholar] 90. Osareh A, Mirmehdi M, Thomas B, Markham R. Automated identification of diabetic retinal exudates in digital colour images. Brit J Ophthalmol. 2003;87(10):1220–1223. [Online]. Available: PM:14507751. [PMC free article] [PubMed] [Google Scholar] 91. Tobin KW, Chaum E, Govindasamy VP, Karnowski TP. Detection of anatomic structures in human retinal imagery. IEEE Trans Med Imaging. 2007 Dec;26(12):1729–1739. [PubMed] [Google Scholar] 92. Lee S, Reinhardt JM, Cattin PC, Abramoff MD. Objective and expert-independent validation of retinal image registration algorithms by a projective imaging distortion model. Med Image Anal. 2010;28(1) [PubMed] [Google Scholar] 93. Lee S, Abràmoff MD, Reinhardt JM. Retinal image mosaicing using the radial distortion correction model. Proc. SPIE Conf. Medical Imaging; San Diego, CA. Feb. 2008; [Google Scholar] 94. Rohr K, Stiehl HS, Sprengel R, Beil W, Buzug TM, Weese J, Kuhn MH. Point-based elastic registration of medical image data using approximating thin-plate splines. VBC. 1996:297–306. [Google Scholar] 95. Lamard M, Cazuguel G, Quellec G, Bekri L, Roux C, Cochener B. Content based image retrieval based on wavelet transform coefficients distribution. Proc IEEE Int Conf 29th Annu Eng Med Biol Soc EMBS. 2007 Aug;:4532–4535. [PMC free article] [PubMed] [Google Scholar] 96. Metz CE, Herman BA, Shen J-H. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Statistics Med. 1998 Dec;17(9):1033–1053. [PubMed] [Google Scholar] 97. Niemeijer M, van Ginneken B, Cree M, Mizutani A, Quellec G, Sanchez C, Zhang B, Hornero R, Lamard M, Muramatsu C, Wu X, Cazuguel G, You J, Mayo A, Li Q, Hatanaka Y, Cochener B, Roux C, Karray F, Garcia M, Fujita H, Abramoff MD. Retinopathy online challenge: Automatic detection of microaneurysms in digital color fundus photographs. IEEE Trans Med Imaging. 2010 Jan;29(1):185–195. [PubMed] [Google Scholar] 98. Niemeijer M, Abramoff MD, van Ginneken B. Information fusion for diabetic retinopathy CAD in digital color fundus photographs. IEEE Trans Med Imaging. 2009 May;28(5):775–785. [PubMed] [Google Scholar] 102. Scharstein D, Szeliski R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int J Computer Vis. 2002;47:7–42. [Google Scholar] 106. Sánchez CI, Hornero R, Mayo A, García M. Mixture model-based clustering and logistic regression for automated detection of microaneurysms in retinal images. Proc SPIE Medical Imaging 2009: Computer Aided Diagnosis. 2009 Feb;7260:72 601M-1–72 601M-8. [Google Scholar] 107. Zhang B, Wu X, You J, Li Q, Karray F. Hierarchical detection of red lesions in retinal images by multiscale correlation filtering. Proc SPIE Medical Imaging 2009: Computer Aided Diagnosis. 2009 Feb;7260:72 601L-1–72 601L-12. [Google Scholar] 108. Mizutani A, Muramatsu C, Hatanaka Y, Suemori S, Hara T, Fujita H. Automated microaneurysm detection method based on double-ring filter in retinal fundus images. Proc SPIE Medical Imaging 2009: Computer Aided Diagnosis. 2009 Febr;7260:72 601N-1–72 601N-8. [Google Scholar] 110. Abramoff MD, Alward WL, Greenlee EC, Shuba L, Kim CY, Fingert JH, Kwon YH. Automated segmentation of the optic disc from stereo color photographs using physiologically plausible features. Invest Ophthalmol Vis Sci. 2007;48:1665–1673. [PMC free article] [PubMed] [Google Scholar] 111. Calkins DJ, Tsukamoto Y, Sterling P. Microcircuitry and mosaic of a blue-yellow ganglion cell in the primate retina. J Neurosci. 1998;18(9):3373–3385. [Online]. Available: PM:9547245. [PMC free article] [PubMed] [Google Scholar] 112. Tang L, Kwon YH, Alward WLM, Greenlee EC, Lee K, Garvin MK, Abramoff MD. 3D reconstruction of the optic nerve head using stereo fundus images for computer-aided diagnosis of glaucoma. Proc SPIE. 2010;7624:76 243D–76 243D-8. [Google Scholar] 113. Poggio T, Marr D. Cooperative computation of stereo disparity. Science. 1976 [PubMed] [Google Scholar] 114. Barnard ST, Thompson WB. Disparity analysis of images. IEEE Trans Pattern Anal Mach Intell. 1980 Apr;2(4):333–340. [PubMed] [Google Scholar] 115. Koozekanani D, Boyer K, Roberts C. Retinal thickness measurements in optical coherence tomography using a Markov boundary model. Proc IEEE Conf Computer Vision Pattern Recognition. 2000 Jun;2:363–370. [Google Scholar] 116. Koozekanani D, Boyer K, Roberts C. Retinal thickness measurements from optical coherence tomography using a Markov boundary model. IEEE Trans Med Imaging. 2001 Sep;20(9):900–916. [PubMed] [Google Scholar] 117. Ishikawa H, Stein DM, Wollstein G, Beaton S, Fujimoto JG, Schuman JS. Macular segmentation with optical coherence tomography. Invest Ophthalmol Vis Sci. 2005 Jun;46(6):2012–2017. [PMC free article] [PubMed] [Google Scholar] 118. Chan A, Duker JS, Ishikawa H, Ko TH, Schuman JS, Fujimoto JG. Quantification of photoreceptor layer thickness in normal eyes using optical coherence tomography. Retina. 2006;26(6):655–660. [PMC free article] [PubMed] [Google Scholar] 119. Cabrera Fernández D. Delineating fluid-filled region boundaries in optical coherence tomography images of the retina. IEEE Trans Med Imaging. 2005 Aug;24(8):929–945. [PubMed] [Google Scholar] 120. Cabrera Fernández D, Salinas HM, Puliafito CA. Automated detection of retinal layer structures on optical coherence tomography images. Opt Expr. 2005;13(25):10 200–10 216. [PubMed] [Google Scholar] 121. Shahidi M, Wang Z, Zelkha R. Quantitative thickness measurement of retinal layers imaged by optical coherence tomography. Amer J Ophthalmol. 2005 Jun;139(6):1056–1061. [PubMed] [Google Scholar] 122. Baroni M, Fortunato P, Torre AL. Towards quantitative analysis of retinal features in optical coherence tomography. Med Eng Phys. 2007 May;29(4):432–441. [PubMed] [Google Scholar] 123. Haeker (Garvin) M, Abràmoff MD, Kardon R, Sonka M. Segmentation of the surfaces of the retinal layer from OCT images. ser Lecture Notes in Computer Science; Proc. 9th Int. Conf. Medical Image Computing Computer-Assisted Intervention (MICCAI 2006), Part I; Springer-Verlag; 2006. pp. 800–807. [PubMed] [Google Scholar] 124. Haeker (Garvin) M, Abràmoff MD, Wu X, Kardon R, Sonka M. Use of varying constraints in optimal 3-D graph search for segmentation of macular optical coherence tomography images. ser Lecture Notes in Computer Science; Proc. 10th Int. Conf. Medical Image Computing Computer-Assisted Intervention (MICCAI 2007); Springer-Verlag; 2007. pp. 244–251. [PubMed] [Google Scholar] 125. Haeker (Garvin) M, Wu X, Abràmoff MD, Kardon R, Sonka M. Incorporation of regional information in optimal 3-D graph search with application for intraretinal layer segmentation of optical coherence tomography images. Inform Process Med Imaging (IPMI) 2007;4584:607–618. [PubMed] [Google Scholar] 126. Garvin MK, Abràmoff MD, Kardon R, Russell SR, Wu X, Sonka M. Intraretinal layer segmentation of macular optical coherence tomography images using optimal 3-D graph search. IEEE Trans Med Imaging. 2008 Oct;27(10):1495–1505. [PMC free article] [PubMed] [Google Scholar] 127. Li K, Wu X, Chen DZ, Sonka M. Optimal surface segmentation in volumetric images—A graph-theoretic approach. IEEE Trans Pattern Anal Machine Intell. 2006 Jan;28(1):119–134. [PMC free article] [PubMed] [Google Scholar] 128. Garvin MK, Abràmoff MD, Wu X, Russell SR, Burns TL, Sonka M. Automated 3-D intraretinal layer segmentation of macular spectral-domain optical coherence tomography images. IEEE Trans Med Imaging. 2009 Sep;28(9):1436–1447. [PMC free article] [PubMed] [Google Scholar] 129. Lee K, Garvin MK, Russell S, Sonka M, Abràmoff MD. Automated intraretinal layer segmentation of 3-D macular OCT scans using a multiscale graph search. Invest Ophthalmol Vis Sci. 2010 May;:E-abstract 1767. [Google Scholar] 130. Antony BJ, Abràmoff MD, Lee K, Sonkova P, Gupta P, Niemeijer M, Hu Z, Garvin MK. Automated 3D segmentation of intraretinal layers from optic nerve head optical coherence tomography images. In: Molthen RC, Weaver JB, editors. Proc SPIE Medical Imaging 2010: Biomedical Applications Molecular, Structural, Functional Imaging. Vol. 7626. SPIE; 2010. p. 76260U. [Google Scholar] 131. Garvin MK, Sonka M, Kardon RH, Wu X, Kwon YH, Russell SR, Abramoff MD. Three-dimensional analysis of SD OCT: Thickness assessment of six macular layers in normal subjects. Invest Ophthalmol Vis Sci. 2008:E-Abstract 1879. [Google Scholar] 132. van Dijk HW, Kok PHB, Garvin M, Sonka M, DeVries JH, Michels RPJ, van Velthoven MEJ, Schlingemann RO, Verbraak FD, Abràmoff MD. Selective loss of inner retinal layer thickness in type 1 diabetic patients with minimal diabetic retinopathy. Invest Ophthalmol Vis Sci. 2009 Jul;50(7):3404–3409. [PMC free article] [PubMed] [Google Scholar] 133. van Dijk HW, Verbraak FD, Kok PHB, Garvin MK, Sonka M, Lee K, DeVries JH, Michels RPJ, van Velthoven MEJ, Schlingemann RO, Abràmoff MD. Decreased retinal ganglion cell layer thickness in type 1 diabetic patients. Invest Ophthalmol Vis Sci. 2010 [PMC free article] [PubMed] [Google Scholar] 134. Fuller A, Zawadzki R, Choi S, Wiley D, Werner J, Hamann B. Segmentation of three-dimensional retinal image data. IEEE Trans Visual Comput Graphics. 2007 Oct;13(10):1719–1726. [PMC free article] [PubMed] [Google Scholar] 135. Yazdanpanah A, Hamarneh G, Smith B, Sarunic M. Intra-retinal layer segmentation in optical coherence tomography using an active contour approach. In: Guang-ZhongYang, Hawkes D, Rueckert D, Noble A, Taylor C, editors. Medical Image Computing Computer-Assisted Intervention MICCAI 2009, Part II, LNCS. Vol. 5762. New York: Springer-Verlag; 2009. pp. 649–656. [Google Scholar] 136. Mishra A, Fieguth P, Clausi D. Accurate boundary localization using dynamic programming on snake. Proc Canadian Conf Computer Robot Vision. 2008:261–268. [Google Scholar] 137. Mishra A, Wong A, Bizheva K, Clausi DA. Intra-retinal layer segmentation in optical coherence tomography images. Opt Expr. 2009;17(26):23 719–23 728. [PubMed] [Google Scholar] 138. Zawadzki RJ, Fuller AR, Choi SS, Wiley DF, Hamann B, Werner JS. Correction of motion artifacts and scanning beam distortions in 3D ophthalmic optical coherence tomography imaging. Proc Ophthalmic Technologies XVII. 2007;6426:642607. [Google Scholar] 139. Ricco S, Chen M, Ishikawa H, Wollstein G, Schuman J. Correcting motion artifacts in retinal spectral domain optical coherence tomography via image registration. In: Yang G-Z, Hawkes DJ, Rueckert D, Noble JA, CJT, editors. ser Lecture Notes in Computer Science; Proc. MICCAI; New York: Springer; 2009. pp. 100–107. 0002. [PMC free article] [PubMed] [Google Scholar] 140. Donato G, Belongie S. Approximate thin plate spline mappings. In: Heyden A, Sparr G, Nielsen M, Johansen P, editors. Proc 7th Eur Conf Computer Vision (ECCV 2002), Part III, LNCS 2352. 2002. pp. 21–31. [Google Scholar] 141. Antony BJ, Tang L, Abràmoff M, Lee K, Sonka M, Garvin MK. Automated method for the flattening of optical coherence tomography images. Invest Ophthalmol Vis Sci. 2010 May;:E-abstract 1781. [Google Scholar] 142. Jaffe GJ, Caprioli J. Optical coherence tomography to detect and manage retinal disease and glaucoma. Amer J Ophthalmol. 2004;137(1):156–169. [PubMed] [Google Scholar] 143. Massin P, Girach A, Erginay A, Gaudric A. Optical coherence tomography: A key to the future management of patients with diabetic macular oedema. Acta Ophthalmol Scand. 2006 Aug;84(4):466–474. [PubMed] [Google Scholar] 144. Haralick RM. Statistical and structural approaches to texture. Proc IEEE. 1979 May;67(5):786–804. [Google Scholar] 145. Fleagle SR, Stanford W, Burns T, Skorton DJ. Feasibility of quantitative texture analysis of cardiac magnetic resonance imagery: Preliminary results. Proc SPIE Medical Imaging. 1994;2168:23–32. [Google Scholar] 146. Quellec G, Lamard M, Cazuguel G, Cochener B, Roux C. Adaptive non-separable wavelet transform via lifting and its application to content-based image retrieval. IEEE Trans Image Process. 2010 Jan;19(1):25–35. [PubMed] [Google Scholar] 147. Xu Y, Sonka M, McLennan G, Guo J, Hoffman EA. MDCT-Based 3-D texture classification of emphysema and early smoking related lung pathologies. IEEE Trans Med Imaging. 2006 Apr;25(4):464–475. [PubMed] [Google Scholar] 148. Quellec G, Lee K, Dolejsi M, Garvin MK, Abràmoff MD, Sonka M. Three-dimensional analysis of retinal layer texture: Identification of fluid-filled regions in SD-OCT of the macula. IEEE Trans Med Imaging. 2010 Jun;29(6):1321–1330. [PMC free article] [PubMed] [Google Scholar] 149. Adler DC, Ko TH, Fujimoto JG. Speckle reduction in optical coherence tomography images by use of a spatially adaptive wavelet filter. Opt Lett. 2004;29(24):2878–2880. [PubMed] [Google Scholar] 150. Puvanathasan P, Bizheva K. Speckle noise reduction algorithm for optical coherence tomography based on interval type II fuzzy set. Opt Expr. 2007;15(24):15 747–15 758. [PubMed] [Google Scholar] 151. Chitchian S, Fiddy MA, Fried NM. Denoising during optical coherence tomography of the prostate nerves via wavelet shrinkage using dual-tree complex wavelet transform. J Biomed Opt. 2009;14(1):014031. [PubMed] [Google Scholar] 152. Lingley-Papadopoulos CA, Loew MH, Zara JM. Wavelet analysis enables system-independent texture analysis of optical coherence tomography images. J Biomed Opt. 2009;14(4) [PubMed] [Google Scholar] 153. Kovacevic J, Sweldens W. Wavelet families of increasing order in arbitrary dimensions. IEEE Trans Image Proc. 2000 Mar;9(3):480–496. [PubMed] [Google Scholar] 154. Niemeijer M, Garvin MK, van Ginneken B, Sonka M, Abràmoff MD. Vessel segmentation in 3D spectral OCT scans of the retina. Proc SPIE Medical Imaging 2008: Image Processing. 2008;6914:69141R. [Google Scholar] 155. Niemeijer M, Sonka M, Garvin MK, van Ginneken B, Abràmoff MD. Automated segmentation of the retinal vasculature in 3D optical coherence tomography images. Invest Ophthalmol Vis Sci. 2008:E-Abstract 1832. [Google Scholar] 156. Xu J, Tolliver DA, Ishikawa H, Wollstein G, Schuman JS. 3D OCT retinal vessel segmentation based on boosting learning. In: Dössel O, Schlegel WC, editors. Proc World Congr Medical Physics Biomedical Engineering, IFMBE. 25/XI. Munich, Germany: Sep, 2009. pp. 179–182. [Google Scholar] 157. Hu Z, Niemeijer M, Abràmoff MD, Lee K, Garvin MK. Automated segmentation of 3-D spectral OCT retinal blood vessels by neural canal opening false positive suppression. Proc Medical Image Computing Computer-Assisted Intervention MICCAI 2010, LNCS. 2010 accepted for publication. [PubMed] [Google Scholar] 158. Lee K, Abràmoff MD, Niemeijer M, Garvin MK, Sonka M. 3D segmentation of retinal blood vessels in spectral-domain OCT volumes of the optic nerve head. Proc SPIE Medical Imaging 2010: Biomedical Applications Molecular, Structural, Functional Imaging. 2010 Feb;7626:76260V. [Google Scholar] 159. Duda RO, Hart PE. Pattern Classification and Scene Analysis. New York: Wiley; 1973. [Google Scholar] 160. Pudil P, Novovičová J, Kittler J. Floating search methods in feature selection with nonmonotinic criterion functions. Pattern Recog Lett. 1994 Nov;15(11):1119–1125. [Google Scholar] 161. Duda RO, Hart PE, Stork DG. Pattern Classification. 2. New York: Wiley; 2000. [Google Scholar] 162. Boykov YY, Jolly M-P. Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. Proc Eighth IEEE Int Conf Computer Vision (ICCV) 2001;1:105–112. [Google Scholar] 163. Boykov Y, Kolmogorov V. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans Pattern Anal Machine Intell. 2004 Sep;26(9):1124–1137. [PubMed] [Google Scholar] 164. Lee K, Niemeijer M, Garvin MK, Kwon YH, Sonka M, Abràmoff MD. 3-D segmentation of the rim and cup in spectral-domain optical coherence tomography volumes of the optic nerve head. Proc SPIE Medical Imaging 2009: Biomedical Applications Molecular, Structural, Functional Imaging. 2009:7262–7283. [Google Scholar] 165. Lee K, Niemeijer M, Garvin MK, Kwon YH, Sonka M, Abràmoff MD. Segmentation of the optic disc in 3D-OCT scans of the optic nerve head. IEEE Trans Med Imaging. 2010 Jan;29(1):159–168. [PMC free article] [PubMed] [Google Scholar] 166. Abràmoff MD, Lee K, Niemeijer M, Alward WLM, Greenlee EC, Garvin MK, Sonka M, Kwon YH. Automated segmentation of the cup and rim from spectral domain OCT of the optic nerve head. Invest Ophthalmol Vis Sci. 2009 Dec;50(12):5778–5784. [PMC free article] [PubMed] [Google Scholar] 167. Hu Z, Niemeijer M, Lee K, Abràmoff MD, Sonka M, Garvin MK. Automated segmentation of the optic disc margin in 3-D optical coherence tomography images using a graph-theoretic approach. Proc SPIE Medical Imaging 2009: Biomedical Applications Molecular, Structural, Functional Imaging. 2009;7262:72620U. [Google Scholar] 168. Hu Z, Niemeijer M, Lee K, Abràmoff MD, Sonka M, Garvin MK. Automated segmentation of the optic canal in 3D spectral-domain OCT of the optic nerve head (ONH) using retinal vessel suppression. Invest Ophthalmol Vis Sci. 2009:E-abstract 3344. [Google Scholar] 169. Kwon YH, Hu Z, Abràmoff MD, Lee K, Garvin MK. Automated segmentation of neural canal opening and optic cup in SD-OCT images. Proc Amer Glaucoma Soc 20th Annu Meeting. 2010 Mar; [Google Scholar] 170. Hu Z, Abràmoff MD, Kwon YH, Lee K, Garvin MK. Automated segmentation of neural canal opening and optic cup in 3-D spectral optical coherence tomography images of the optic nerve head. Invest Ophthalmol Vis Sci. 2010 accepted for publication. [PMC free article] [PubMed] [Google Scholar] 171. Arya S, Mount DM, Netanyahu NS, Silverman R, Wu AY. An optimal algorithm for approximate nearest neighbor searching fixed dimensions. J ACM. 1998;45:891–923. [Google Scholar] 172. Strouthidis N, Yang H, Fortune B, Downs J, Burgoyne C. Detection of the optic nerve head neural canal opening within three-dimensional histomorphometric and spectral domain optical coherence tomography data sets. Invest Ophthalmol Vis Sci. 2009;50(5):214–223. [PMC free article] [PubMed] [Google Scholar] 173. Strouthidis NG, Yang H, Reynaud JF, Grimm JL, Gardiner SK, Fortune B, Burgoyne CF. Comparison of clinical and spectral domain optical coherence tomography optic disc margin anatomy. Invest Ophthalmol Vis Sci. 2009;50(10):4709–4718. [PMC free article] [PubMed] [Google Scholar] 174. Magjarevic R, Troglio G, Nappo A, Benediktsson JA, Moser G, Serpico SB, Stefansson E. Automatic change detection of retinal images. 2009;25/11:281–284. [Online]. Available: http://dx.doi.org/10.1007/978-3-642-03891-4_75. [Google Scholar] 175. You X, Fang B, He Z, Tang Y. A global-to-local matching strategy for registering retinal fundus images. Lecture Notes Computer Science. 2005;3522:259–267. [Google Scholar] 176. Can A, Stewart CV, Roysam B, Tanenbaum HL. A feature-based robust, hierarchical algorithm for registering pairs of images of the curved human retina. IEEE Trans Pattern Anal Machine Intell. 2002 Mar;24(3):347–364. [Google Scholar] 177. Stewart C, Tasi C, Roysam B. The dual-bootstrap iterative closest point algorithm with application to retinal image registration. IEEE Trans Med Imaging. 2003 Nov;22(11):1379–1394. [PubMed] [Google Scholar] 178. Cattin PC, Bay H, Gool LV, Szekely G. Retina mosaicing using local features. Medical Image Computing Computer-Assisted Intervention (MICCAI); Copenhagen, Denmark. Oct. 2006; pp. 185–192. [PubMed] [Google Scholar] 179. Lee S, Abramoff MD, Reinhardt JM. Feature-based pairwise retinal image registration by radial distortion correction. In: Pluim JP, Reinhardt JM, editors. Proc SPIE Conf Medical Imaging. Vol. 6512. San Diego, CA: 2007. [Online]. Available: http://dx.doi.org/10.1117/12.710676. [Google Scholar] 180. Powell MJD. An efficient method for finding the minimum of a function of several variables without calculating derivatives. The Comput J. 1964;7:155–162. [Google Scholar] 181. Lee S, Abramoff MD, Reinhardt JM. Retinal atlas statistics from color fundus images. In: Dawant BM, Haynor DR, editors. Proc SPIE Conf Medical Imaging. Vol. 7623. 2010. p. 762310. [Google Scholar] 182. Chen J, Tian J, Lee N, Zheng J, Smith RT, Laine AF. A partial intensity invariant feature descriptor for multimodal retinal image registration. IEEE Trans Biomed Eng. 2010 Jul;57(7):1707–1718. [PMC free article] [PubMed] [Google Scholar] 183. Kim JS, Ishikawa H, Gabriele ML, Wollstein G, Bilonick RA, Kagemann L, Fujimoto JG, Schuman JS. Retinal nerve fiber layer thickness measurement comparability between time domain optical coherence tomography (OCT) and spectral domain OCT. Invest Ophthalmol Vis Sci. 2010;51:896–902. [PMC free article] [PubMed] [Google Scholar] 184. Niemeijer M, Garvin M, Lee K, van Ginneken B, Abramoff M, Sonka M. Registration of 3D spectral OCT volumes using 3D SIFT feature point matching. SPIE Medical Imaging. 2009;7259:72591I. [Google Scholar] 185. Cheung W, Hamarneh G. N-SIFT: N-dimensional scale invariant feature transform for matching medical images. Proc. 4th IEEE Int. Symp. Biomedical Imaging: From Nano to Macro, 2007. ISBI; 2007. pp. 720–723. [Google Scholar] 186. Lowe DG. Distinctive image features from scale-invariant key-points. Int J Comput Vis. 2004;60(2):91–110. [Google Scholar] 187. Cheung WA, Hamarneh G. Scale invariant feature transform for n-dimensional images (n-SIFT) Insight J. 2007 [Online]. Available: http://hdl.handle.net/1926/1321. What are the 3 types of perceptual constancy?Types of Perceptual Constancy: Shape, Size, and Brightness.
When we perceive familiar objects as having the same form even when the retinal image changes it is called?Constancy refers to how our perception of objects remains the same despite changes in their image on the retina.
How we perceive the increase in the size of an object on the retina?The perceived size of objects depends on a number of factors. Perhaps the most important of these is the visual angle subtended by the object on the retina. All other things being equal, the object that subtends the larger visual angle will appear larger.
What is gestalt principles of visual perception?The classic principles of the gestalt theory of visual perception include similarity, continuation, closure, proximity, figure/ground, and symmetry & order (also known as prägnanz). Others, such as “common fate,” have been added in recent years.
|
