Longcreek Blog

Georeferencing Chernarus and visiting in real life 🗺️

Wed, 15 Sep 2021 10:00:00 +0000

In this post I will walk through how to

Obtain a high-resolution raster map by stitching map tiles from a tileserver with a shell script.
Georeference a fictional map to a real location by manually matching features and their corresponding real locations using QGIS.
Convert the georeferenced map to a Garmin compatible KMZ file with a Python script using GDAL for use on a handheld GPS device.

I will apply it to the fictional map of Chernarus which is based on a real area near Ústí nad Labem in Czechia. The end product is a georeferenced Garmin compatible KMZ file which you can put on your GPS handheld if you ever visit the area.

The map below shows Chernarus (left) overlayed with the real region (right) - you can move the map and drag the slider to compare:

Click here for a full screen version of the map.

Background

Chernarus is a fictional country introduced in the game Arma II, and the name of a large piece of terrain featured in the game spanning roughly 15x15 km. It was also the main location used in the DayZ mod which modified the game from a “military simulator” to a zombie survival game, which was later expanded¹ into a standalone game where new details and towns were added to the original terrain (known as Chernarus+).

The map of Chernarus is closely based on an area in nothern Bohemia in Czechia, between Ústí nad Labem and Děčín and appears very similar to the real location, as many of the real buildings are also appearing in the game. Some people in the community would occasionally post pictures comparing the game with the real location either by Google Street View or from actual visits and note how familiar the place felt even though it was their first time there.

The map can be even be georeferenced and overlayed on the real location quite accurately:

A few years ago I was in the area and decided to put the map on my GPS handheld and try to use it as reference while hiking around the area for fun.

It took a bit of effort and learning for me to get it working, and in this post I will walk through how I did it. The same process can be used for any map sufficiently similar to a real location, be it other fictional, historical or rare maps that you might have a copy of.

Download and stitch high-resolution map from a tileserver

To obtain a high-resolution map of Chernarus, we are going to download tiles from a tileserver and stitch them together into one large map. A tileserver is a common way to serve maps for the web, where a large map is rendered as many small tiles, so we don’t have to load the full map. For this project I used the iZurvive Chernarus map as a source.

The address for the tileserver is

https://maps.izurvive.com/maps/CH-Mod/1.7.9/tiles/${z}/${x}/${y}.png

In this format ${z} is the zoom level, and ${x} and ${y} is the x and y integer tile coordinates, or simply the tile number. Thus at zoom level z=6 (the highest of the server) the tile with x=4 and y=3 is found at https://maps.izurvive.com/maps/CH-Mod/1.7.9/tiles/6/4/3.png and looks like this

The whole map is split into pieces like this, and our goal is now to download all tiles for the Chernarus map and stitch them together. To download all the tiles, I used a shell script (modified from this question on GIS Stackexchange).

First we download the tiles:

# Extent of map
X1=0
X2=50
Y1=0
Y2=42

# Zoom level
Z=6

# Iterate through tiles and download
TILE_URL = https://maps.izurvive.com/maps/CH-Mod/1.7.9/tiles/${Z}/${x}/${y}.png
for x in `seq $X1 $X2`; do
    for y in `seq $Y1 $Y2`; do
        echo "Getting ${x},${y}"
        curl -s $TILE_URL -o ${Z}_${y}_${x}.png &
    done
    wait
done

echo "Rename .png to .jpg as files are wrongly named"
for i in ./${Z}_*.png ; do mv "$i" "${i/.png/.jpg}"; done

The images are actually JPEG files, despite the .png file extension, which is why they get renamed to .jpg file extension.

To stitch the images together I use montage from Imagemagick. First we iterate through x coordinates and for each x we vertically stack all the relevant tiles from Y1 to Y2 together giving us a bunch of tall images²:

for ((x=0; x<=$X2; x++)); do
    echo "Stitching x=${x}"
    montage -mode concatenate -tile "1x" $(ls ${Z}_*_${x}.jpg | sort -Vr) out_z${Z}_x${x}.jpg
done

Finally we stack all of these images horizontally

echo "Stitching Everything together"
montage -mode concatenate -tile "x1" $(ls out_z${Z}_*.jpg | sort -V) full_map.jpg

If everything worked out, we now have a full map of Chernarus in high resolution on our hands!

Georeferencing Chernarus to real life

The next step is to overlay the Chernarus map on the real life counterpart by referencing some points on the game map with the same points on the real map. These are known as Ground Control Ppoints (GCP). To do this, I am using QGIS, a free and open source application for working with GIS (Geographical Information System) and the built in Georeferencer tool. Have a look at the QGIS documentation on Georeferencing for more details.

Adding other tilemaps to QGIS

If you just installed QGIS you may want to add some more map sources, as only the standard Open Street Map tiles are provided by default. On the left panel, simply right click on “XYZ Tiles” and choose “New Connection”.

In the following dialogue you can add sources.

I like to use the OpenTopoMap for topographical maps athttps://tile.opentopomap.org/{z}/{x}/{y}.png, and Bing Aerial for satelite views at http://ecn.t3.tiles.virtualearth.net/tiles/a{q}.jpeg?g=1.

You can find more open tile servers here.

Now start a new project in QGIS and simply double click on the tilemaps you want to add them as a layer in the project.

Since we are using a web map for reference we will set the Coordinate Reference System (CRS) of the project to the Web Mercator projection or EPSG: 3857 so it renders properly. QGIS automatically sets the project CRS to be the same as the first layer added, but yo be sure you can set it by right clicking on the OpenTopoMap layer, then “Layer CRS” -> “Set project CRS from layer”. This should set the project CRS to EPS 3857 - WGS 84 / Pseudo-Mercator.

NOTE: When we later transform the Chernarus map after georeferencing, it should be transformed into EPSG: 4326 if you want to use it on your Garmin device to show up properly.

Zoom in on Ústí nad Labem (in north western Czechia, north of Prague) in preparation for the next step.

Georeferencing points

In order to load the Chernarus map into QGIS we first convert it to a PNG³, e.g. with convert from Imagemagick. Or just load it into your favourite image editor and save as PNG.

convert full_map.jpg chernarus.png

Now open the Georeferencer with Raster -> Georeferencer

Load in the Chernaus map using Open raster (Ctrl+O), and it should look like this

Now comes the fun part! We need to find points on the Chernarus map which are identifiable on the real map. Press “Add point” (or Ctrl + A) to add points, and then click somewhere on the Chernarus map that you can recognize in the real map. It will ask you to enter map coordinates - if you press “From map canvas” you can now click on the corresponding point on the real map, and the coordinates will be filled out.

This could e.g. be the T-junction in Stary Sobor (Lipová)

Or the roads meeting at Topolka dam (just north of Povrly)

Keep going until you have at least four points, preferably more - I’ve used 10-15 points for a good fit, making sure to put points near the map edges as well.

Now open Transform Settings (yellow cogwheel) to set the parameters used for the georeferencing transform. After some experimenting I have chosen transformation type “Polynomial 2” and resampling method “Cubic” for the final transform, but feel free to see what works best for you. I suggest using “Nearest Neighbour” resampling to begin with, as it is the fastest so you can do adustments more easily.

If you want to use it on a Garmin device you should set “Target SRS” to EPSG:4326 - WGS 84 as this is the standard map projection used in these devices (and for GPS in general), unless you want to fiddle with your devices projection settings.

The Georeferencer documentation describes in more detail how the transformations work, and says the following about the Polynomial transforms:

The Polynomial algorithms 2-3 use more general 2nd or 3rd degree polynomials instead of just affine transformation. This allows them to account for curvature or other systematic warping of the image, for instance photographed maps with curving edges. At least 6 (respectively 10) GCP’s are required. Angles and local scale are not preserved or treated uniformly across the image. In particular, straight lines may become curved, and there may be significant distortion introduced at the edges or far from any GCPs arising from extrapolating the data-fitted polynomials too far.

If you check “Load in QGIS when done” the georeferenced Chernarus map will automatically be added as a new layer in the project when done. The file will also be saved as a GeoTIFF, here named chernarus_modified.tif, which is our georeferenced and transformed map.

Once the transformation parameters have been set, QGIS will calculate a residual for each point. When calculating a polynomial transform, there are some mathematical constraints which means a point on the raster may not end up exactly on top of the point you referenced on the real map after the transform. The residual essentially tells you how far off each point will land from its target. Thus if one point has a much higher residual than the rest (I aim for less than 10 px in this case), you may want to correct it.

Now press “Start georeferencing” (green arrow or Ctrl + G) and wait for it to finish (takes about a minute on my laptop). If everything worked out, the Chernarus map should now be visible on top of the area around Ústí nad Labem

In order to see both the real map and Chernarus map at once, you can open the layer style panel (F7) and chose “Blending mode” “Multiply”.

While the fit will never be perfect, you should be able to see most roads fairly well aligned to the real life counterparts. Otherwise try adjusting the GCP points and transformation parameters

Add to handheld GPS (Garmin GPSMap)

If you have a Garmin GPS device you can add the georeferenced map if your device supports CustomMaps⁴. I’m using a GPSMap 64s, but the steps should be similar for other devices.

These devices don’t support GeoTIFFs directly, and we have to convert our newly made map into the Garmin CustomMap format. The format is a KMZ file, which uses raster map tiles with a KML file to keep track of their positions. This means we have to convert our image back into tiles! The difference is of course that we have adjusted the map and can specify proper coordinates for the tiles.

Installing GDAL

To convert the GeoTIFF I use the Python 3 script gdal2custommap⁵. You will need to install the GDAL library as well as the python bindings found in the GDAL package.

Linux: If you are running some variant of Debian/Ubuntu it can be installed system wide with sudo apt install python3-gdal, which will take care of the GDAL dependencies. If you are managing packages with e.g. pip you can use python -m pip install GDAL (assuming python points to python3).

Windows: Assuming you have a working Python 3 installation on your machine, you will need to install GDAL and the Python bindings manually. This can be done with binaries found at GISInternals. Have a look at the python GDAL package documentation for more details. After this, install the python GDAL package with Python Package manager of choice, e.g. Conda.

Convert to KMZ (CustomMap)

If we have a georeferenced map of Chernarus named chernarus_modified.tif we first convert it to tiles and an associated KML file with gdal2kml.py

python gdal2kml.py chernarus_modified.tif chernarus.kml

It is possible to specify some options such as JPEG quality (defaults to 75), KML draw order etc, which is detailed in the readme of the script. Next we use kml2kmz.py to zip up all the files into a single KMZ file

python kml2kmz.py chernarus.kml

Now you should have a chernarus.kmz which is ready to put on your device.

Add KMZ file to Garmin device

Connect to the device storage with e.g. USB cable and transfer chernarus.kmz onto the device under /Garmin/CustomMaps/chernarus.kmz. If your device supports an SD card you can use this to store the map, just put it under the same directory structure as the internal memory. This is recommended for transferring large files as the SD card tends to be much faster if used directly in a card reader instead of through the USB interface. Next, disconnect the device, insert the SD card if used, and reboot it.

Now you need to enable the custom map to see it. For my GPSMap 64s i do the following: In the map view, press the MENU button, Setup Map and in the bottom under Map Information/Select Map, I can choose which maps to show.

Find chernarus.kmz under Custom Maps and enable it.

You can quickly check if it works by opening it again after enabling it, which should show a preview of the Chernarus map. If it doesn’t show up, try disabling other map layers as they might cover the custom map, and make sure the CRS of the GeoTIFF is set to EPSG:4326

If you haven’t already, find the area around Ústí nad Labem and Děčín and zoom in. You have to zoom quite close before the custom map renders, for me it shows only at the 500 m scale bar and closer.

If you have other maps enabled, they might overlay or completely cover the Chernarus map. Here’s how it looks in Lipová with and without OpenTopoMap for Garmin enabled, where roads, town names and similar show as an overlay.

You can experiment with the --draw-order option for gdal2kml.py if you want to use overlays, as the KML order determines “height” of the layers, where layers with the lowest numbers are drawn on the top.

Visit to North Bohemia

The final step is of course to travel to the area in real life! Ústí can be reached by direct train from Prague in 70 minutes for 5-7 € (tickets from oneticket.cz or directly from České dráhy, the national rail service), and there are plenty of places to stay overnight in the area - I can warmly recommmend the combined hotel-brewery Na Rychtě both for eating at their restaurant or for staying a few nights in Ústí.

If you have played just a bit of Arma or Dayz in Chernaus, you will find the place quite familiar as you walk around, as the developers have included not just the road network but also several buildings in the game, as shown in this imgur album. Below are a couple of the pictures I took myself

Characteristic yellow church in Javory (“Gorka”).

View from Javorský vrch (“Altar”) to Děčín (“Berezino”).

View of GPS when entering Lipová (“Stary Sobor”).

The infamous red metal barn in Lipová (“Stary Sobor”). It seems to have been repaired when comparing to older pictures.

A farm building in Český Bukov (“Staroye”) used several places in the game.

View towards south of dam near Povrly (“Topolka Dam” north of “Elektrozavodsk”).

Summary

In this post we have obtained a high-resolution raster map from a tileserver, georeferenced it in QGIS to the real world location it is based on, and then converted it to a Garmin compatible KMZ file for viewing on a handheld GPS device.

You can also grab the Chernarus KMZ file directly to put on your device and save some time.

The methods shown here can be applied to any map which is sufficiently similar to real world locations - It could be interesting to walk around your town using a historical map to get a sense of how things used to look, and how much it changed since the map was made, for example.

DayZ standalone had quite a turbulent development history, being stuck in a paid “early alpha” pre-release stage for five years from 2013 until its official release in December 2018, but that’s a story for another time… ↩
Ideally montage can stitch all the images together in one command, but I had issues with too many open files errors on my system when trying to stack them all, which is why it is split in two sections. ↩
QGIS does support loading of JPEG raster images, but I encounter whenever I try: not a supported raster data source. (libjpeg: Wrong JPEG library version: library is 62, caller expects 80), which is why I convert to PNG. I didn’t find a solution, let me know if you can get it to work! ↩
There’s a list of supported Garmin devices on this support page in footnote 1, “Click Here for a List of Compatible Devices”. ↩
This is a fork of the Python 2 gdal2custommap scripts by Tris Forster, which I modified slightly to wok with Python 3 and iron out some bugs. ↩

Visualizing the magnetic field of nanoparticles with Python

Tue, 06 Jul 2021 10:00:00 +0000

\[\newcommand{\vr}{\mathrm{\mathbf{r}}} \newcommand{\unit}[1]{\,\mathrm{#1}} \newcommand{\vec}[1]{\mathrm{\mathbf{#1}}} \newcommand{\fourier}[1]{\mathcal{F}\left[#1 \right]} \newcommand{\vB}{\vec{B}} \newcommand{\vq}{\vec{q}} \newcommand{\Iholo}{I_\mathrm{holo}} \newcommand{\Iside}{I_\mathrm{s.b.}}\]

Note: The following is a continuation on this post on electron holography, which I suggest reading first to get a sense of what is going on.

As you have probably seen before, magnets and their magnetic fields are often visualized as field lines like in the figure below¹ which represent how the magnetic field extends outside a rectangular magnet

This is a common illustration used for magnetic fields, where magnetic field lines are drawn along lines of constant field strength – A sort of contour lines for the magnetic field strength. If we were to place a tiny compas at any position outside the magnet, the compass needle would align itself to the direction of the field lines at this position.

This is usually presented for magnets large enough hold in your hand, but it is actually possible to directly measure the magnetic field of nanoparticles in a Transmission Electron Microscope (TEM), just like this:

The TEM uses electrons as a probe to image particles, and electrons moving in a magnetic field will get affected by it. Information about the magnetic field of a sample can be saved using a technique known as Electron Holography which preserves the phase $\phi$ of the electrons in the acquired image.

Reconstructing the above image from raw holograms involve a few different image processing techniques including Fourier analysis, frequency filtering and image matching to find the translational shift between two almost identical images, as well as other image processing techniques.

In this post I walk through all the Python code used to go from a raw hologram to a magnetic phase image such as the above. You can see the full code in this Jupyter notebook, which you can download along with the holograms if you want to play along yourself.

All the data shown was acquired as part of my bachelors thesis² at DTU Nanolab (previously Center for Electron Nanoscopy) several years back and follows the procedure detailed in this book chapter³. I recently went back to revisit the subject to redo it in Python out of curiousity and decided do this writeup to share with anyone why might be interested.

Calculating phase image from a hologram

Importing microscope data

We start out with some common packages for scientific Python, namely numpy and matplotlib⁴. We use pyplot for plotting and will set some default parameters with plt.rcParams so we don’t have to do manually for each plot.

# Import numpy and fft functions directly
import numpy as np
from numpy.fft import fft2, ifft2, fftshift

# Set some default parameters for the plots
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = [16, 8]
plt.rcParams['figure.frameon'] = False
plt.rcParams['image.cmap'] = 'gray'

The raw data from the microscope is saved in dm3 files, a common format used in electron microscopy. It is usually opened with the proprietary Gatan Digital Micrograph software or a program like ImageJ which can be difficult to work with for more complex calculations, but luckily the OpenNCEM project allows us to open it in Python (look here if using Matlab).

from ncempy.io.dm import dmReader

# Load dm3 files for a particle and reference
dmdata1 = dmReader('data/h05 +30sat.dm3')
dmdata1r = dmReader('data/h06r +30sat.dm3')

# Extract image data
im1 = dmdata1['data']
im1r = dmdata1r['data']

The raw image in im1, our hologram, looks like this

Note that the numbered axes show pixel numbers instead of nanometers. We keep it this way for now as it helps illustrate the numerical calculalations we will be doing. However, it is still important to provide a scale when showing microscope images, and it is readily available from the metadata:

print('Scale from metadata: 1 px = %s %s' %(dmdata1['pixelSize'][0], dmdata1['pixelUnit'][0]))

>> Scale from metadata: 1 px = 0.0037131761 µm

So 1 px = 0.0037 µm, but: the scale is off by a factor 10 in these specific images. This is because the objective lens (the one that forms the image) is the special Lorentz lens, and the correction has not been included correctly in the metadata. We will correct for this and convert to nanometers while we’re at it:

# Redefine to nm and correct for Lorentz lens factor 10 error
pxsize = dmdata1['pixelSize'][0] * 1000/10
pixel_unit = 'nm'
print('Corrected scale: 1 px = %.4f %s' %(pxsize, pixel_unit))
print('Field of view: %.2f nm x %.2f nm' %(im1.shape[1]*pxsize, im1.shape[0]*pxsize))

>> Corrected scale: 1 px = 0.3713 nm
>> Field of view: 760.46 nm x 760.46 nm

The image is thus 760 nm from one side to the other, and the particle is about 90 nm wide.

There is a wealth of other information in the metadata such as acceleration voltage, magnification etc. which can be accessed with the fileDM method from ncempy if needed.

Fourier Transformed hologram with sidebands

We can now calculate the Fast Fourier Transform (FFT) on our hologram of the particle using fft2 from numpy. We also use fftshift which centers the main peak in the image, as it is otherwise split into the corners for numerical reasons.

# Transform image, ready for visualization
im1_fft = fftshift(fft2(im1))
im1_fft_abslog = np.log(np.abs(im1_fft))

# Adjust colormap to increase contrast
vmin_fft = np.amin(im1_fft_abslog)*3
vmax_fft = np.amax(im1_fft_abslog)*0.8

plt.imshow(im1_fft_abslog, vmax=vmax_fft, vmin=vmin_fft)
plt.xlim([600, 1448])
plt.ylim([1448, 600])

Here we see the central peak with the two “sidebands” off to the sides⁵. Each sideband contains both the amplitude of the wave which passed through the sample and the reference wave $A_i(\vr)$ and $A_r(\vr)$ respectively, as well as the electron phase $\phi(\vr)$:

\[\hat{I}_\mathrm{s.b.}(\vq) = \fourier{A_i(\vr)A_r(\vr)e^{i\phi(\vr)}}\]

Here $\vr = (x,y)$ refers to a given point (or pixel) in the image. The next step in isolating the phase image $\phi(\vr)$ is then to extract one of the sidebands.

Extracting a sideband

The FFT image contains three significant peaks, namely the central one (by far most intense) and the two sidebands (similar intensity). Each sideband is almost identical, with the important difference being that they are complex conjugates of each other - This means a phase image calculated from one sideband will correspond exactly to the phase image from the other sideband with opposite sign.

To select a sideband, we will find coordinates of the peaks in the image using peak_local_max from the scikit-image package exactly like in this example. As the central peak is always strongest we simply find the second strongest peak and check if it’s a positive or negative phase change, otherwise we take the third strongest. We write this into a function as we’ll do it a few times

from scipy.ndimage import maximum_filter
from skimage.feature import peak_local_max

def peak_coordinate(image_fft, peak_number=0):
    # Find the peaks
    im = np.abs(image_fft)
    image_max = maximum_filter(im, size=200, mode='constant')
    coordinates = peak_local_max(im, min_distance=200)

    # Get values of peaks and sort, get coordinates of second largest peak
    peaks = [image_fft[coordinate[0], coordinate[1]] for coordinate in coordinates]
    peaks_idx = np.argsort(peaks)
    coordinates_sorted = [coordinates[peak_idx] for peak_idx in peaks_idx]

    sideband_idx = coordinates_sorted[peak_number]

    # Choose sideband on the right side of the image
    if sideband_idx[1] < image_fft.shape[1]//2:
        sideband_idx[0] -= 2*(sideband_idx[0]-image_fft.shape[0]//2)
        sideband_idx[1] -= 2*(sideband_idx[1]-image_fft.shape[1]//2)

    return sideband_idx

We’ll wrap this into a function to automatically extract a sideband from a given FFT image. Note that the image is stored as a matrix where rows (up and down) are indexed before columns (left and right), meaning $x$ and $y$ must be reversed since we want a horizontal x-axis and vertical y-axis.

def extract_sideband(image_fft, width=128):
    sideband_idx = peak_coordinate(np.abs(image_fft))
    width = width//2
    x1 = sideband_idx[1]-width
    x2 = sideband_idx[1]+width
    y1 = sideband_idx[0]-width
    y2 = sideband_idx[0]+width

    sideband_fft = image_fft[y1:y2, x1:x2]

    return sideband_fft

We can now apply our new function and take a closer look at the sideband.

sideband_size = 200
sideband1_fft = extract_sideband(im1_fft, width=sideband_size)

plt.imshow(np.log(np.abs(sideband1_fft)))

The vertical and horizontal streaks are artifacts which appear because our our original image is finite at the edges. When we decompose the image into its frequency components the sharp, abrupt change at the image have to be represented by many different frequencies, giving rise to these streaks. Later you will be able to see the the result of these artifacts in the reconstructed image along the very edges of the images but this will not affect our analysis otherwise.

Phase and amplitude from sideband

We are now ready to calculate the amplitude and phase of our image!

When performing a fourier transform of an image (real values), the FFT will contain complex values. Usually when the inverse fourier transform is applied we would get back the original image containing only real values. However, by cropping out the sideband we have redefined the center of the FFT image. Now when applying the inverse fourier transform to the sideband, the resulting image will contain complex values.

This is actually intended! We can find the “usual” intensity image from the absolute value of the complex number in each pixel as $I = \sqrt{\mathrm{Re}^2+\mathrm{Im}^2}$ with np.abs (like when visualizing the FFT). The phase image is found from the angle of the complex number $\varphi=\arctan\left(\frac{\mathrm{Im}}{\mathrm{Re}}\right)$ using np.angle.

# Calculate amplitude and phase of image
sideband1 = ifft2(fftshift(sideband1_fft))
sideband1_abs = np.abs(sideband1)
sideband1_phase = np.angle(sideband1)

Here’s the amplitude image

Note how the amplitude image is very similar to the original hologram, but in reduced resolution since we cropped it out as part of the full FFT image. The phase image looks like this

We immediately notice some odd artifacts in the form of sudden jumps in value. We calculated the phase (or angle) of the complex numbers with np.angle which is an implementation of atan2, a numerical version of the $\arctan$ function which outputs an angle between $-\pi$ and $\pi$, and an angle is thus bound between these values.

This means that if a region of the phase image has a phase value extending above $\pi$ for example, the value returned by atan2 will “wrap around” and return $-\pi$ as it is does not know that it should be continuing the values of the neighbouring pixels.

Fortunately “unwrapping” phase images is a common procedure in image processing and scikit-image has it built into the unwrap_phase method:

from skimage.restoration import unwrap_phase
sideband1_phase = unwrap_phase(sideband1_phase)

Our phase image now looks like this

So far so good, we have succesfully recovered the full phase shift of the electron beam after passing through a nanoparticle! We will improve it a bit by subtracting a background reference before.

Correcting phase image with background reference

First we load in a reference background image which was acquired in an area without any sample present to get a clean background reference. Using the functions defined earlier makes quick work of extracting the sideband

im1r_fft = fftshift(fft2(im1r))
sideband1r_fft = extract_sideband(im1r_fft, width=sideband_size)

fig, (ax1, ax2, ax3) = plt.subplots(1,3)
ax1.imshow(im1r)
ax2.imshow(np.abs(np.log(im1r_fft)))
ax3.imshow(np.abs(np.log(sideband1r_fft)))

Now we calculate the phase of the reference image

sideband1r = ifft2(fftshift(sideband1r_fft))
sideband1r_phase = unwrap_phase(np.angle(sideband1r))

plt.title('Background reference phase image')
plt.imshow(sideband1r_phase)

Finally we subtract the reference image which leaves us with a much more even phase image

phi1 = sideband1_phase-sideband1r_phase

plt.title('Corrected phase image of particle')
plt.imshow(phi1)

The magnetic phase shift still not easily visible as the electric contribution dominates the phase image.

Hologram of particle with reversed magnetization

In order to isolate just the magnetic contribution $\phi_m$ to the phase shift we need an image of the exact same particle where the particle’s magnetization, and thus $\phi_m$, is reversed. Since $\phi_m$ changes sign while $\phi_e$ is unchanged it is a matter of simple subtraction

\[\phi = (\phi_e+\phi_m) - (\phi_e - \phi_m) = 2\phi_m\]

Then we just divide by 2, giving us the pure magnetic phase image $\phi_m$.

The magnetic reversal is done by applying a strong magnetic field inside the microscope in the opposite direction of the previous used to magnetize the particle in the first place, using one of the magnetic lenses.

We load in the image and subtract a reference phase image just like before:

# Load data
dmdata2 = dmReader('data/h19 -30sat.dm3')
dmdata2r = dmReader('data/h17r.dm3')

im2 = dmdata2['data']
im2r = dmdata2r['data']

# Calculate FFT and extract sideband for image
im2_fft = fftshift(fft2(im2))
sideband2_fft = extract_sideband(im2_fft, width=sideband_size)
sideband2 = ifft2(fftshift(sideband2_fft))
sideband2_phase = unwrap_phase(np.angle(sideband2))

# Calculate FFT and extract sideband for reference image
im2r_fft = fftshift(fft2(im2r))
sideband2r_fft = extract_sideband(im2r_fft, width=sideband_size)
sideband2r = ifft2(fftshift(sideband2r_fft))
sideband2r_phase = unwrap_phase(np.angle(sideband2r))

# Subtract reference phase shift
phi2 = sideband2_phase-sideband2r_phase

plt.title('Corrected phase image of particle with reversed magnetization')
plt.imshow(phi2)

This phase image is almost identical to the first, but with two important differences: $\phi_m$ is opposite of before, and the particle is in a slightly different position in the image. This means we cannot directly subtract the two images until we have found out exactly how much the particle position changed.

Matching two similar images to adjust for translation

There is a significant shift in x and y positions between the images, but only negligible rotation. This means we can use phase cross correlation to get a good estimate of the translation between images as implemented in phase_cross_correlation in scikit-image⁶. There is a decent example with details here. Knowing this shift we can correct for it and ensure the images have a common center so we can subtract them properly.

We start out by applying a bandpass with gaussian blur to remove some high- and low-frequency details which can detract from the image matching.

from skimage.filters import window, gaussian, difference_of_gaussians
from skimage.registration import phase_cross_correlation
from skimage import transform

# Bandpass with Gaussian blur
sigma1 = 1
sigma2 = 3
phi1_filt = difference_of_gaussians(phi1, sigma1, sigma2)
phi2_filt = difference_of_gaussians(phi2, sigma1, sigma2)

fig, ax = plt.subplots(1,2)
ax[0].imshow(phi1_filt)
ax[1].imshow(phi2_filt)

As noted when extracting the first sideband, there are significant artifacts along the edges of the image which can also cause issues – image edges are generally troublesome when dealing with frequency decompositions. For this reason we will construct a “window” and multiply with our images, which ensures all values go smoothly towards $0$ at the edges.

# Apply Hann window to avoid border issues
window_type = 'hann'
window_image = window(window_type, phi1_filt.shape)
phi1_windowed = phi1_filt*window_image
phi2_windowed = phi2_filt*window_image

fig, ax = plt.subplots(1,2)
ax[0].imshow(window_image)
ax[1].imshow(phi1_windowed)

With the bandpass filter and window applied to both images, we are now ready to calculate the shift in x and y

# Calculate the shift using phase cross correlation on filtered images
shifts, error, phasediff = phase_cross_correlation(phi1_filt, phi2_filt, upsample_factor=100)

# Note that matrices are indexed as rows ("y") at [0] and columns ("x") at [1]
dx = shifts[1]
dy = shifts[0]
print('Translation: (dx, dy) = (%.2f, %.2f)' % (dx, dy))

>> Translation: (dx, dy) = (15.04, 13.14)

The calculated translation is now applied as a geometrical transform to correct one of the unfiltered phase images. After this we can subtract one phase image from the other to isolate the magnetic phase shift (and remember to divide by 2).

# Translate the second unfiltered phase image for matching on top of the first
tform = transform.EuclideanTransform(translation=(dx, dy))
phi2_tf = transform.warp(phi2, tform.inverse)

# Subtract unfiltered phase images, remember division by 2!
phi_m_raw = (phi1-phi2_tf)/2

plt.title('Magnetic phase shift')
plt.imshow(phi_m_raw)

The magnetic phase shift has revealed itself! Note how there is a lighter region on one side of the particle and dark on the other. This is the magnetic contribution to the phase shift.

Magnetic field lines from phase image

We can enhance the magnetic phase image by cutting out the relevant part where the images overlap and filtering some noise away.

First we make a function to cut off the parts of the image that without overlap after subtraction

def crop_from_shifts(image, dx, dy, border=0):
    dx, dy = int(dx), int(dy)
    image = image[:, dx:-1] if dx >= 0 else image[:, 0:dx]
    image = image[dy:-1, :] if dy >= 0 else image[0:dy, :]

    border = int(border)
    if border > 0:
        image = image[border:-border, border:-border]

    return image

We can also filter the image a bit with a gaussian blur to reduce high frequency noise, since the magnetic field (and phase shift) only varies slowly across the image. We will also apply a scalebar to show the size of the features.

from matplotlib_scalebar.scalebar import ScaleBar

# Scale original pixel size to match resolution of phase image and add to scalebar
pxsize_phase = im1.shape[0]/sideband_size*pxsize
scalebar = ScaleBar(pxsize_phase, pixel_unit, length_fraction=0.25)

# Blur before cropping to minimize edge effects
phi_m_filt = gaussian(phi_m_raw, 3)

# Crop non-overlapping part and a bit of border
crop_border = 5
phi_m = crop_from_shifts(phi_m_filt, dx, dy, border=crop_border)

fig, ax = plt.subplots()
ax.axis('off')
ax.imshow(phi_m)
ax.add_artist(scalebar)
plt.title('Magnetic phase shift')

The phase image now looks like this

Contour lines representing the magnetic field lines can also be overlayed directly on top of the bright field image, letting us see the field lines emanating from the particle itself. Using a divergent colormap which shows the central values as white, while displaying values above and below that as blue and red respectively, we can enhance the readability of the phase image.

We just need to make sure we crop the bright field image in the same way so the centering is right.

# Apply a bit stronger blur to enhance contour lines
phi_m_extrafilt = crop_from_shifts(gaussian(phi_m_raw, 4), dx, dy, border=crop_border)
fig, ax = plt.subplots(1,2)

# Plot phase with divergent colormap, overlay contour lines
ax[0].imshow(phi_m, cmap='RdBu')
ax[0].contour(phi_m_extrafilt, 15, colors='k', alpha=0.5)
ax[0].axis('off')
scalebar = ScaleBar(pxsize_phase, pixel_unit, length_fraction=0.25)
ax[0].add_artist(scalebar)

# Crop BF image with same parameters as the phase image was cropped to get same center
bf_image = crop_from_shifts(sideband1_abs, dx, dy, border=crop_border)

# Overlay contours onto bright field image
ax[1].imshow(gaussian(bf_image, 1))
ax[1].contour(phi_m_extrafilt, 12, colors='white', linewidths=2, alpha=0.8)
ax[1].axis('off')
scalebar = ScaleBar(pxsize_phase, pixel_unit, length_fraction=0.25)
ax[1].add_artist(scalebar)

By plotting the contour lines we are in essence displaying the magnetic field lines as they emanate from the particle:

Now that is looking like proper magnetic field lines from a magnetic nanoparticle!

To enhance the image a bit, we will overlay the contour lines on top of the original hologram image so we get much better contrast. We do this by rescaling the magnetic phase image used for contour lines and again cropping parts of the orginal hologram to get the same center

from skimage.transform import rescale

# Increase resolution of phase image
scaling = im1.shape[0]/sideband_size
phi_m_upscaled = rescale(gaussian(phi_m, 5), scaling)

# Crop original image the same way as the sideband images were, correcting for scale
im1_crop = crop_from_shifts(im1, dx*scaling, dy*scaling, border=crop_border*scaling)
im1_filt = gaussian(im1_crop, 3)

plt.imshow(im1_filt)
plt.contour(phi_m_upscaled, 15, colors='white', linewidths=2, alpha=0.8)

Final thoughts

Congratulations for making it all the way through!

We have worked all the way from loading raw holograms taken in a Transmisison Electron Microscope into Python to extracting and visualizing a magnetic phase image by using various image processing techniques and a bit of Fourier transform magic.

There is still much that can be done with this data, such as calculations on the exact magnetization of the particle and how it compares to expected values from other sources, which I have skipped as this post is already running long. If you want to try the code yourself, make sure to download the associed Jupyter Notebook with data, where you can make your own calculations (preview of notebook).

While I have omitted many details on this complex topic, I hope it can server as an interesting appetizer on some of the things that can be done in Transmission Electron Microscopes, and how information that is usually not obtainable can be brought forward with some clever techniques.

Adapted from diagrams of the magnetic field around a bar magnet from Wikimedia Commons ↩
Supervisors: Takeshi Kasama, Marco Beleggia, Cathrine Frandsen. ↩
Takeshi Kasama, Rafal E. Dunin-Borkowski and Marco Beleggia (2011). Electron Holography of Magnetic Materials, Holography - Different Fields of Application, F. A. Ramirez (Ed.), IntechOpen, DOI: 10.5772/22366 ↩
All packages used in the notebook are: numpy, matplotlib, ncenmpy, scikit-image (imports as skimage) and matplotlib_scalebar. ↩
Remember that each pixel in the fourier transformed image im1_fft consists of complex values with a real value $\mathrm{Re}$ and imaginary value $\mathrm{Im}$. What we display as “the FFT” is the amplitude as $r = \sqrt{\mathrm{Re}^2+\mathrm{Im}^2}$ using np.abs. Further, to avoid the central peak dominating the image I also calculate the logarithm of each pixel value with np.log to enhance visibility of the smaller values in the sidebands relative to the peak, as well as adjusting vmin and vmax. ↩
Even if the image is rotated and scaled this method can still be used as demonstrated in this example – essentially the matching is performed on the FFT of the image to obtain rotation, followed by the same method as here after correcting for rotation. ↩

Visualizing the magnetic field of nanoparticles - Fundamentals

Thu, 20 May 2021 10:14:00 +0000

\[\newcommand{\vr}{\mathrm{\mathbf{r}}} \newcommand{\unit}[1]{\,\mathrm{#1}} \newcommand{\vec}[1]{\mathrm{\mathbf{#1}}} \newcommand{\fourier}[1]{\mathcal{F}\left[#1 \right]} \newcommand{\vv}{\vec{v}} \newcommand{\vB}{\vec{B}} \newcommand{\vq}{\vec{q}} \newcommand{\Iholo}{I_\mathrm{holo}} \newcommand{\Iside}{I_\mathrm{s.b.}}\]

This is the first of two posts - the second on reconstructing holograms using Python can be found here.

When learning about magnets and magnetic fields, you have probably seen visualizations similar to these¹ which represent how the magnetic field extends outside a rectangular magnet

The left represents how iron filings agglomerate when sprinkled around a magnet as they roughly align to the magnetic field, a common method to directly “see” the magnetic field around larger magnets. In the middle the magnetic field is represented with a series of magnetic field lines which are drawn along lines of constant field strength, while the rightmost image shows how an array of freely rotating magnets (e.g. compasses) would align themselves in the same magnetic field.

This is usually presented for magnets large enough hold in your hand, but it is actually possible to directly measure the magnetic field all the way down at the nano scale in a Transmission Electron Microscope or TEM, just like this:

The above image shows a magnetite nanoparticle around 90 nm wide where the contour lines represent the magnetic field around the particle as measured directly in the microscope. Iron filings and tiny compasses work great for magnets large enough to handle, but what do we do when working on the nanoscale?

Below i will explain a method known as Off-Axis Electron Holography and how it can be used to measure and visualize the magnetic field around magnetic nanoparticles in the TEM to make images like the above. Much of what follows is based on this book chapter² on electron holography of magnetic materials which I recommend reading if you are interested in the subject.

The data shown was acquired as part of my bachelors thesis³ at DTU Nanolab (previously Center for Electron Nanoscopy) several years back. I recently went back to revisit the subject and redo (and improve) in Python out of curiousity and decided do this writeup to share with anyone why might be interested.

This post is part one of two and covers some TEM fundamentals and relevant mathematics. For numerical calculations, take a look at part two where we reconstruct holograms using Python.

Very brief intro to the TEM

A Transmission Electron Microscope in conventional operation is very similar to a typical optical microscope, but instead of using light waves we use a beam of electrons. Since electron waves generated at high voltages have a much shorter wavelength than visible light, we can use it to see much smaller things. As an example, blue light has a wavelength around $400 \unit{nm}$, while an electron will have a wavelength around $0.0025 \unit{nm}$ at $200 \unit{kV}$ acceleration voltage – for reference a hydrogen atom has a diameter of around $0.01 \unit{nm}$⁴.

The diagram below shows a simplified schematic of a TEM in bright field (BF) mode, where an electron gun at the top provides a beam of coherent electrons which are condensed to a uniform plane wavefront before passing through a sample. After this the beam is focused, various apertures can be inserted, and the image is formed in the image plane on the bottom, where we place 2D detector to save the image.

Unlike in the optical microscope, the lenses are not made from glass or any physical material. Instead, we use magnetic lenses to focus the electron beam – Essentially a symmetric bunch of coils with adjustable current going through them to create a magnetic field. The reason we can manipulate the electrons with magnetic fields is due to the Lorentz force.

\[\vec{F} = q \, \vec{v}\times\vec{B}\]

With a strong field of a certain shape we can use this to focus our electron beam. This is also the reason we are able to visualise the magnetic field of a sample when using electrons, as the field around a magnetic particle deflects the electrons slightly and change their phase.

I am barely scratching the surface of the subject for the sake of brevity - have a look at the very detailed and pedagogical DoITPoMS from the University of Cambridge on the TEM if you want more details.

Electron beam as a wavefunction

Mathematically we can describe the electron beam at any point $\vr$ in the image plane as a plane wave with amplitude $A(\vr)$ and phase $\phi(\vr)$ in a complex exponential phase term

\[\psi(\vr) = A(\vr)e^{i \phi(\vr)}\]

In typical operating conditions, when recording the image on a detector the phase information is lost as the intensity distribution is the absolute square of the wavefunction, where the conjugated exponential terms cancel out

\[I(\vr) = \left| \psi(\vr) \right|^2 = \left[A(\vr)e^{i \phi(\vr)}\right]\left[A(\vr)e^{-i \phi(\vr)}\right]= A(\vr)^2\]

In other words we’re only recording the square of the amplitude of the wave, which simply tells us how many electrons reached each pixel on the sensor, regardless of their phase. This is plenty for most uses, but in our case we actually need to know the phase of the electrons.

When an electron wave passes through a sample the phase $\phi$ of the wave will experience a phase shift. For thin magnetic samples we can express the phase shift at a given point $\vr$ after passing through a sample as the sum of an electric and magnetic contribution

\[\phi(\vr) = \phi_e(\vr) + \phi_m(\vr)\]

$\phi_e$ arises from the electron beam intereacting with the electrons in the sample, while $\phi_m$ depends on the in-plane component of the magnetic field from the sample due to the Lorentz force. We note that since the force is proportional to the cross product $\vv\times\vB$, the moving charges are only affected by the components of the B-field orthogonal to the movement direction. Since the electrons are moving from the top and down into the sample they will thus only be affected by the in-plane component of the B-field. If we imagine a magnetic field pointing directly upwards from the sample, parallel to the movement of electrons there would be no phase shift, while a field perfectly orthogonal, along the sample plane, would show up fully in the phase shift.

The phase shift $\phi_m$ is thus our key to imaging the magnetic field of our particles, and we need to figure out how to record it. As luck would have it there are workarounds for this problem.

Holography in the electron microscope

In case you are wondering, holo graphy comes from Greek and can be translated as whole drawing - in microscopy and this often refers to the fact that holographic methods preserve the phase. The “whole” in cool shiny 3D holograms on old CD’s, toys, credit cards etc. refers to the fact that they capture the “whole” three dimensional scene.

The challenge for us is to find a way to make sure the phase information of our electron wave is not lost as we record it on the detector. Below is a very simplified diagram of a TEM in a setup used for a method called Off-Axis Electron Holography.

The green part of the beam which passed trough the sample we’ll call $\psi_s$, and the other half which goes unhindered through the vacuum is our reference wave, $\psi_r$.

Like before we generate a coherent plane wave of electrons, but now we only send part of it through the sample (green part). Further down, after the objective lens⁵, is the key part of the setup: the biprism. It is simply a wire going all the way through the middle of the electron beam with a plate on each side along the length of the wire outside the beam. When applying a positive charge to the beam and negative charge to the plates the electrons are repelled by the plates and attracted to the wire.

So we have essentially split the whole wavefront in two parts which we can bend towards each other causing them to overlap, and this is where the magic happens. When two coherent (in-phase) plane waves meet each other at an angle a sinusoidal interference pattern will appear as illustrated in the bottom of the diagram and can be recorded with out image sensor. The pattern is highly sensitive to small differences in phase between the two waves and will thus have information relating to $\phi_e$ and $\phi_m$ embedded in it.

So how do we extract the phase?

Mathematics of the phase retrieval

Let’s formalise a bit on what was described above.

Contributions to the phase shift

For a sample with no external charge distributions or external electric fields, the electrostatic contribution to the phase shift can be expressed in terms of a mean inner potential $V_0$ which is assumed constant throughout the sample.

\[\phi_e(x,y) = C_E V_0 t(x,y)\]

where $C_E$ is a constant dependent on the microscope acceleration voltage⁶ and $t(x,y)$ the thickness profile of the sample projected in the beam direction.

The magnetic phase shift can be expressed as

\[\phi_m(x,y) = -\frac{e}{\hbar} \int A_z(x,y,z) \,dz\]

where $A_z$ is the magnetic vector potential, specifically the component parallel to the electron beam direction. The gradient of the magnetic phase then gives us a direct relationship to the B-field form the phase shift

\[\nabla \phi_m(x,y) = \frac{e}{\hbar}\begin{bmatrix}B_y^p(x,y)\\\; -B_x^p(x,y) \end{bmatrix}\]

where $B^p$ is the total projection of the B-field components perpendicular to electron beam direction through the sample area

\[B^p(x,y) = \int B(x,y,z)\,dz\]

As such, the phase shift of the electron beam after passing through a sample can be described as

\[\phi(x,y) = \phi_e(x,y) + \phi_m(z,y)\]

Our goal now is to extract just the magnetic phase shift $\phi_m$ of the electron beam and we need to get rid of $\phi_e$ in some way.

If we can exactly reverse the magnetic field of our particle, the magnetic phase shift will also be reversed from $\phi_m$ to $-\phi_m$ while $\phi_e$ is unchanged. In the TEM this can be done by applying a strong magnetic field in one direction, record an image, then apply a strong field in the opposite direction to reverse the magnetisation of the particle, and record another image.

We can then record two images where the magnetisation of the particle is exactly reversed between them and then subtract the phase shifts. Then $\phi_e$ cancels out and leaves just the magnetic information

\[\phi_{\mathrm{rev}} = \phi_1 - \phi_2 = (\phi_e + \phi_m) - (\phi_e - \phi_m) = 2\phi_m\]

Now we just remember to divide by 2 and we have our magnetic phase shift!⁷

Preserving the phase information

Remembering that the electron beam at a given point $\vr$ in the image plane can be described as a wavefunction $\psi_i(\vr) = A_i(\vr) e^{i\phi(\vr)}$, we now introduce a plane reference wave that only passes unhindered through vacuum

\[\psi_r(\vr)=A_r(\vr)e^{i2\pi \vq_c\cdot \vr}\]

with $\vq_c$ being a two-dimenstional reciprocal space vector determining the tilt of the reference wave. This is what we do with the biprism in the off-axis holography setup, when we split the beam into two parts and make them overlap. In the overlap the wavefunction is simply a superposition of the two wavefunctions leading to the following intensity distribution

\[\Iholo(\vr) = \left|\psi_i(\vr)+\psi_r(\vr)\right|^2 = \left| A_i(\vr) e^{i\phi(\vr)} + A_r(\vr)e^{i2\pi \vq\cdot \vr}\right|^2\]

This can be evaluated as

\[I_{holo}(\vr) = A_i^2(\vr) + A_r^2(\vr) + A_i(\vr)A_r(\vr)\left(e^{i(2\pi\vq_c\cdotp\vr-\phi(\vr))}+e^{-i(2\pi\vq_c\cdotp\vr-\phi(\vr))}\right)\]

The phase information is now preserved directly in the intensity distribution on our image sensor! We have the amplitude of the wave passing through the sample $A_i(\vr)^2$ (as before), the amplitude of the reference wave $A_r(\vr)^2$, and finally an oscillating term⁸ which contains the phase of the sample beam $\phi(\vr)$.

Let’s see how it looks in practice. Below is a holographic image of a magnetite nanoparticle which is about 90 nm wide. It is actually shaped like an octahedron but as it is lying down the projection through the particle appears as a hexagon. The large round shape in the top left half of the image is a hole in a thin carbon substrate⁹, and our particle is just barely holding on to the edge of this hole.

The thin lines all across the image is a direct result of the interference of the two beams. In areas with even background it closely matches a cosine function as expected. Note that the much larger parallel lines (top left) are an artifact from diffraction fringes at the edges of the biprism wire and not directly related to the phase. Let’s look at a closeup of the particle

Notice how the lines are not straight everywhere. If we follow one from top to bottom, we see it bends at the top end of the particle, continues in a mostly straight line and then bends back close to its original position. This bending is directly due to the phase shift of the electron beam due to the particle - at the edges the thickness varies causing the bend, while the center of the particle is fairly even, giving straight lines. The phase shift is primarily from the electric interaction $\phi_e$ as it tends to be stronger than $\phi_m$.

Recovering just the phase

To separate the phase term, we turn to the trusty Fourier transform, which I wrote a bit about here. Now, as the Fourier transform is a linear operator, the transform of a sum of terms is equivalent to the sum of individually transformed terms:

\[\fourier{\Iholo} = \fourier{A_i(\vr)^2} + \fourier{A_r(\vr)^2} + \fourier{A_i(\vr)A_r(\vr)\left(e^{i(2\pi\vq_c\cdotp\vr-\phi(\vr))}+e^{-i(2\pi\vq_c\cdotp\vr-\phi(\vr))}\right)}\]

With a bit of mathematical gymnastics the last term in the expression can expressed as a convolution with a delta function $\delta(\vq\pm\vq_c)$

\[\begin{align*} \hat{I}_\mathrm{holo} =& \hat{A}_i(\vq)^2 + \hat{A}_r(\vq)^2 \\ &+ \delta(\vq+\vq_c)*\fourier{A_i(\vr)A_r(\vr)e^{i\phi(\vr)}} \\ &+ \delta(\vq-\vq_c)*\fourier{A_i(\vr)A_r(\vr)e^{-i\phi(\vr)}} \end{align*}\]

where $*$ denotes the convolution operator. The proof of the above is left as an exercise to the reader.

It is slightly complex, but here’s what you need to take away from the above: When convolving a delta function with a fourier transform, it essentially becomes “centered” at the point where the argument of the delta function is zero; $\delta(\mathbf{0})$. Thus if convolving with $\delta(\vq+\vq_c)$ the center will be at $\vq=-\vq_c$.

So now we have two amplitude terms with no phase information, $\hat{A}_i(\vq)^2$ and $\hat{A}_r(\vq)^2$ centered around $\vq=(0,0)$ in the middle of the reciprocal space in the fourier transformed image. They are more or less the regular bright field image you would get without the special technique. But now we also have two phase terms, which are the complex conjugates of each other, centered at $\vq=\vq_c$ and $\vq=-\vq_c$ respectively. These will show up at opposite sides, away from the center of the fourier transformed image.

Visually it looks like this. Below is shown the 2D Fast Fourier Transform of the hologram image

A large peak in the center with two distinct peaks away from center at opposite sides. These are the sidebands mentioned before, and the $\vq_c$ vector is the vector pointing from the center and out to these sidebands. The visible line from the center all the way to the sideband comes from the diffraction fringes from the biprism as mentioned earlier.

Note that the Fourier transform of a real valued image results in complex values, i.e. each pixel has a real part $\mathrm{Re}$ and imaginary part $\mathrm{Im}$. When showing the image here, we are seeing the absolute value, or amplitude, of this complex image.

We can now select a sideband from the full image, i.e. we pick out just the term

\[\delta(\vq+\vq_c)*\fourier{A_i(\vr)A_r(\vr)e^{i\phi(\vr)}}\]

The horizontal and vertical streaks are artifacts from the edges of the image and will not affect our analysis. The center of this new image is now at $\vq=-\vq_c$ and the convolution with the delta function becomes $1$ leaving us with

\[\hat{I}_\mathrm{s.b.}(\vq) = \fourier{A_i(\vr)A_r(\vr)e^{i\phi(\vr)}}\]

By performing an inverse Fourier transform the image is now simply

\[\Iside(\vr) = A_i(\vr)A_r(\vr)e^{i\phi(\vr)}\]

We are now liberated of the other terms and we are one important step closer to isolating $\phi(\vr)$ itself. Exciting!

Amplitude and phase of a complex number

Our sideband $\Iside(\vr)$ is now a complex function on the exponential or polar form $z = re^{i\varphi}$ where $r$ is the amplitude and $\varphi$ the phase. Mathematically we can directly read off the amplitude as $r=A_iA_r$ and the phase of our image as $\varphi=\phi$ but numerically our data is stored in another form.

The (Inverse) Fast Fourier Transform used to numerically perform fourier transforms gives us an image where complex numbers are in the cartesian form $z=a+ib$, and thus each pixel has a real value $\mathrm{Re}(z) = a$ and imaginary value $\mathrm{Im}(z) = b$. Luckily the conversion comes from basic trigonometry. The amplitude is simply the pythagorean length of the complex number

\[r = \sqrt{a^2+b^2}\]

and the phase is the angle

\[\varphi = \arctan\left(\frac{b}{a}\right)\]

That’s it, we now have everything we need to recover the phase! The reconstructed phase of our particle looks like this

\[\phi(\vr) = \phi_e(\vr) + \phi_m(\vr)\]

The above image (besides artifacts) is a direct image of the phase of the electron beam where it reached the detector in the microscope, reconstructed from the shown hologram. We would expect the magnetic field somewhere outside the particle, so where is it? The electric contribution $\phi_e$ is much larger than the magnetic $\phi_m$ so it gets a bit lost in the signal. To see it properly we still need to subtract an image of the same particle with reversed magnetic field, meaning we still have some work to do.

Summary

We’ve gone briefly through the workings of a transmission electron microscope and seen how the electron beam can be described as a wave. With a special setup in the microscope which introduces a reference wave, the interference pattern in the overlap of these waves enables us to record the phase information of the beam passing through the sample. Using some Fourier transform magic we walked through the mathematics of extracting the phase from this pattern and demonstrated it for a single particle.

The remaining work to isolate only the magnetic contribution to the phase shift is mainly computational. I continue this in the post on reconstructing holograms using Python where I walk through the code used to go from raw holograms to finished magnetic phase images.

Diagrams of the magnetic field around a bar magnet from Wikimedia Commons ↩
Takeshi Kasama, Rafal E. Dunin-Borkowski and Marco Beleggia (2011). Electron Holography of Magnetic Materials, Holography - Different Fields of Application, F. A. Ramirez (Ed.), IntechOpen, DOI: 10.5772/22366 ↩
Supervisors: Takeshi Kasama, Marco Beleggia, Cathrine Frandsen. On a personal note: Takeshi Kasama passed away a few years back at a much too young age. He did an amazing job as supervisor on my bachelor’s project. He had a knack for teaching and for making me “learn how to learn”, and I am grateful to have been under his guidance. He has definitely shaped my approach to science and physics and how I explain things to others, so if you like what you read send him a friendly thought! ↩
Strictly speaking an atom has a fuzzy undefined size, but the diameter from the Bohr model gives a useful sense of how far from the center of a hydrogen atom you are likely to find its accompanying electron. ↩
Since we are trying to image the weak magnetic field of our nanoparticles, the objective lens is now a Lorentz lens, as this does not generate a magnetic field at the sample. ↩
With acceleration voltage as $U=\frac{E}{e}$ we have $C_E = \left(\frac{2\pi}{\lambda}\right) \left(\frac{E+E_0}{E(E+2E_0)}\right)$ where $\lambda$ is the relativistic electron wavelength and $E_0$ the rest mass energy of the electron. ↩
There are also other ways of removing the electric phaseshift $\phi_e$, such as directly simulating $\phi_e$ from detailed sample information and subtracting that. It can also be useful on its own, e.g. for accurate measurements of particle thickness, but we will not consider this for now. ↩
Using some identities of complex exponentials we can also write the phase term as a cosine term $A_i(\vr)A_r(\vr) \cos\left[2\pi\vq_c\cdot\vr+\phi(\vr)\right]$. These sinusoidal interference fringes are what we directly measure in the raw hologram. ↩
Take a look at Agar scientific for some example images. ↩

Some visual intuition for 2D Fourier transforms

Thu, 10 Dec 2020 00:00:00 +0000

Improving by blurring

Let’s say we have a image of a cat and we want to blur it a bit

That’s pretty cute, but you may not need blurred cat images so often. However there are also more useful things we can do using the exact same method. This here is Constable Sinclair Tait of the Inverness-shire Constabulary

Many things can be said about Mr. Tait, which you can read a lot more about on Dave Conner’s Flickr, but he certainly hasn’t been preserved all that well. The interesting thing is that by using the exact same method as for blurring a cat we can bring some life back to our constable:

That’s pretty neat, and, some would argue, more useful than blurred cats. So how does it work?

The Fourier transform

Both the cat blurring and constable reconstruction is done by applying a low-pass filter to the images, or, in other words, by removing high frequencies.

The way we go about is by a Fourier transformation of our image. I will not go into too many details of exactly how or why it works in this post - for now all we need to know is that any¹ function or signal can be expressed as a sum of sine and cosine functions to arbitrary precision.

How does this help us blur cats? Consider this illustration by Lucas Barbosa of a 1D function being decomposed into it’s frequency components

Here a square wave function $f$ is decomposed into a range of sine and cosine functions each multiplied by a constant $a_n$ or $b_n$ (their amplitude) and each with a different frequency (here proportional to $n$) - if each of these individual sine and cosines were to be added together the result is again the original function $f$. We are essentially seeing $f$ being split up into its frequency components, and the function $\hat{f}$ shows us how large each component is.

This is essentially what the Fourier transform does - it tells us which frequencies a function or signal consists of and how strong they each are. In other words, if our signal is in “real space” we can transform it to “frequency space”.

Notation and an important property

Applying the Fourier transform can be denoted as $\mathcal{F}\left\{f\right\} = \hat{f}$, where $f$ is transformed into $\hat{f}$. Importantly we can directly recover our original function with the inverse Fourier transform, $\mathcal{F}^{-1}\{\hat{f}\} = f$. For disctrete (digital) signals such as images we typically use a numerical version called a Fast Fourier Transform. For our use in this blog post we consider them equivalent, and I will use “the FFT” in some places to refer to the result of a transform on an image.

A very useful fact about the Fourier transformation is its linearity. This means that the Fourier transform of a sum of functions can be expressed as a sum of individual Fourier transforms on each function: $\mathcal{F}\left\{f_1 + f_2 \right\} = \mathcal{F}\left\{f_1\right\} + \mathcal{F}\left\{f_2\right\}$

We are going to leverage this to our advantage.

Frequencies in an image

Let’s experiment a bit and create a little 2D image. Here $x$ and $y$ goes from $0$ to $2\pi$ (i.e. one full period of a trigonometric function) and we plot the function $f_1 = \cos (4x)$

The FFT of this simple cosine looks like this

Two dots spaced equally apart from the image center - these are the frequency components of our image, and since our image is a “pure” cosine the frequency is very well defined. The reason there are two dots instead of just one, despite having just one frequency, is beyond the scope of this post. Just know that a Fourier transform of a real valued signal gives a symmetric and complex valued result, meaning each pixel in our image is a complex number, $z=a+ib$. The two dots represent exactly the same frequency and are each others complex conjugate². When I “show the fourier transform” what I am actually showing is the absolute value of each point $\lvert z \rvert$³.

Let’s try with a higher frequency and see how $f_2 = \cos(8x)$ and its fourier transform looks

Again two dots centered around the middle, but now they are further apart. The last bit is important; higher frequencies are further away from the center in the FFT image.

We can also add the two images together, so we’re now seing $f_3 = \cos(4x) + \cos(16x)$

Then we have two sets of dots, exactly corresponding to the two frequencies we are visualising. And importantly, even though everything is muddled together in “real space”, we still se everything distinct in “frequency space”.

If we turn the picture by plotting $f_4 = \cos(4y)$ we also turn the arrangement of dots

Again we can add the pictures together, i.e. our image now shows $f_5 = \cos(4x)+\cos(8x)+ \cos(4y)$

And again we see two distinct peaks for each frequency in our image.

Filtering with a mask

Let’s freshen things up a bit and try with a whole bunch of frequencies

If you look closely you can see that our previous image with $f_5$ is actually contained in the one above. It’s quite difficult to see in real space but in the FFT the dots from before a still clearly distinct from all the new ones.

Let’s recover them!

Since all the unwanted frequencies are higher than the ones we want, we can do so with a high-pass filter. A simple way of doing this is to just set all frequencies above a certain treshold to $0$. Remember that in the FFT the frequency is proportional to the distance from the center. This means we will need a circular mask, where white pixels are $1$ and black are $0$.

Now we can apply the mask to the FFT image by multiplying them together (both real and imaginary parts). This way each pixel with a $0$ in the mask becomres $0$ in the FFT, and $1$’s in the mask simply keep their value. This results in the following

That should seem familiar. The lines crossing in the center are artifacts from discretization of the higher frequencies, but they don’t matter to omuch now. We can now do an inverse Fourier transform and recover our image!

And we’re back to how we started!

Blurring the darn cat

This brings us back to the cat. Let’s see it again along with its fourier transform⁴.

As there are not many well defined frequencies in the picture we don’t have the same well defined spots as before. Let’s do the same as we did to that previous image and cut off some higher frequencies (aka. use a low pass filter). As before we create a mask and apply it to our image

The mask is made from a Tukey window which has tapered edges to avoid edge artifacts which would appear from a sudden frequency cutoff. There are many types of window functions to use (you can even make your own!). I chose Tukey somewhat arbitrarily - we’ll save the specifics for another post.

We can increase or decrease the diameter of our filter (or window) which will in turn increase or decrease the cut off frequency and thus the sharpness of our image:

Restoring the Constable

Using what we have now (hopefully) learned, we return to Mr. Tait and his Fourier transform

The dots are back! Why is that? As we saw earlier, a well defined frequency in an image leads to a well defined point in the FFT. The dots which were used to print Mr. Taits portrait are very evenly spaced and can be apparently represented well by a limited, fairly well defined set of frequencies. This is good news for us if we want to restore his image as we then only need to block fairly specific frequencies.

We can begin with the easy way - just cut off all frequencies above a certain point which contains the dots:

Not a bad start!

But we may want to keep some information in the higher frequencies. We can basically do whatever we want with the FFT image. Instead of simply cutting off everything above a certain frequency (low-pass filter) we’ll try to just blot out the dots and keep everything else.

We create a new mask, this time much smaller than the picture and inverted compared to before

This time we have $0$’s in the center and $1$’s at the edge, so when we multiply this onto our image we can “erase” certain points. From this we create a larger mask which we can apply to the FFT of the constable

The inverse transform gives us the filtered constable

Not perfect.

You may be thinking that the improvement was not very significant compared to just the low-pass filter, and you would actually be correct.

The reason is there cannot be any details “between the dots” in the original image. In other words, as all information in the original image is encoded purely by the dots themselves, there shouldn’t be any additional information with higher frequency than those, and when we cut away higher frequencies we don’t lose image information. However there may be additional details from the digitalisation (dust, markings, etc.) which could be interesting for some applications. There are also some artifacts around the edges, which I have not accounted for, seen as vertical and horizontal streaks in the FFT, but that is maybe food for a future blog post.

Let us combine the two methods and do both a notch filter and low-pass filter

Our constable has now reached his final form:

There are many other techniques that can be used for this kind of image reconstruction, but it is at least interesting to see how much you can do with a fairly straight forward method.

The principles we have seen and the intuition gained from playing with transforming images can carry over and be of great help in many other areas of science and physics, including understanding of lenses, diffraction gratings, crystallography and many other (some of which I hope to write more about here!)

Hope you enjoyed reading!

There are exceptions, but most reasonably well-behaved functions one might normally encounter in the physical world (and the mathematics describing it) has a Fourier transform. There is a bit more on that on this Stackexchange question. ↩
complex conjugation is fancy speak for “the $i$ changes sign” in a complex number. Thus the complex conjugate of $a + ib$ is just $a - ib$. ↩
\[\lvert z\rvert = \sqrt{a^2 + (i b)^2} = \sqrt{a^2 + b^2}\]
↩
I am now displaying the logarithm of the value of each pixel now, i.e. $\log\left(\lvert z \rvert\right)$ for all pixels, as the main peak would dominate everything in the image. With the logarithm lower valued pixels are enhanced and easier to see. ↩