Teaching electron microscopes to be more autonomous with AI
Jilles van Hulst defended his PhD thesis at the Department of Mechanical Engineering on 3 September.
Electron microscopes can image individual atoms, making them indispensable for breakthroughs in medicine, batteries, and microchips. During the COVID-19 pandemic, they helped researchers rapidly uncover the structure of the virus's spike protein, speeding up vaccine development. Yet these sophisticated instruments still require highly trained operators to keep them in focus. In his PhD research, Jilles van Hulst explores in collaboration with microscope manufacturer Thermo Fisher Scientific how electron microscopes can produce sharp images and steadily hold the object under study by themselves. He did this by teaching the microscopes to extract more information from just a few noisy images.
No microscope can show details smaller than the wavelength it uses, and visible light has a wavelength thousands of times larger than an atom. Electrons offer a way around this limitation: a fast electron behaves as a quantum wave far smaller than an atom. If a light wave were stretched to the length of a football field, an electron wave would span just a few sheets of paper. Electrons are also electrically charged, allowing magnetic fields to act as lenses. However, a theorem from 1936 proves that these lenses always distort the image. Correctors have existed since 1998, but they add dozens of settings that drift over time and require constant retuning.
Learning from a handful of images
Retuning these settings is difficult, and anyone who has focused an old camera by hand knows why. When you鈥檙e looking at a single blurry photo, it is impossible to tell which way to turn the focus ring. Only after turning it can you see whether the image has improved or worsened, and perfect focus is precisely the point where turning either way makes the image worse. An electron microscope faces this challenge not with one focus ring, but with dozens of settings at once. Moreover, every image takes time to acquire and can damage the specimen. Jilles van Hulst therefore trained artificial intelligence to extract the maximum amount of information from each image.
The AI learns from a simulator that can generate unlimited practice images, but a simulation never perfectly matches the real machine. The method therefore also learns the difference between simulation and reality during operation, using the few real images it collects. The symmetry of the blur around perfect focus is what uniquely identifies that point. The result is a microscope that calibrates itself in about twenty seconds, achieving twice the as accuracy of the best existing automated methods.
Standing still at the scale of atoms
Seeing atoms also demands stillness. To create a three-dimensional image, the specimen is tilted step by step while dozens of images are taken, and the object under study must remain in view. The stage carrying the specimen can move across millimeters, yet it must keep the imaged point stable to within nanometers. The actuators that move the stage are imperfect, and no sensor directly measures the specimen鈥檚 position. Van Hulst used the images already recorded by the microscope as the missing sensor. These images reveal exactly how far the specimen has shifted, while a learning-based controller removes errors it has encountered before. On an operational microscope, this approach achieved nanometer-scale accuracy, up to ten times better than existing methods.
More knowledge from less data
The common thread is a lesson from statistics: what can be learned from data depends on what is already known. Van Hulst therefore developed general mathematical methods that incorporate known structure, such as physical laws, symmetries, and models of motion, directly into the learning process. As a result, fewer measurements are needed. Together, these advances bring the self-driving microscope closer to reality. It can tune itself, keep its specimen steady, and collects data autonomously, allowing researchers to focus on scientific discoveries.
Title of PhD thesis: . Supervisors: Dr. Duarte Guerreiro Tom茅 Antunes, Prof. Maurice Heemels and Dr. Erik Franken (Thermo Fisher Scientific).