1

EMBO Practical Course on Image Processing for Cryo EM 30th August – 9th September 2011

Practical 1: Introduction to SPIDER/WEB and EMAN/boxer Log into a PC with the username and password provided and then on to our linux system as described in the Getting started document. You should be in your home directory (/d/emr24/u/<username> for eris, or, /d/emr25/u/<username> for hera). If you are not sure what directory you are in you can always type “pwd” and this will display the directory you are currently in. In your home directory you will need to make a new directory called prac1 (mkdir prac1) and move into that directory (cd prac1). Copy the required files for this practical to your own directory with the command: cp /d/ipcourse/EMBO-11/PRAC-1/*.spi .

You MUST copy files to your own directory before doing any work!!

You should now have four files, elsatp.spi, scan.spi, MT.spi and hh256.spi in your directory. Check by listing them (ls). (Reminder: basic UNIX commands are listed on page 3 of the Getting Started document). In order to access the software, you must run a setup script with the source command source /s/em/embo2011.csh

Introduction

The purpose of this practical is to introduce you to SPIDER, Web and the particle picking module of EMAN (Boxer), to give you some experience with the basic functions of image processing. Part 1: SPIDER and WEB Filename extensions and starting SPIDER In a UNIX window, start SPIDER by typing at the UNIX prompt: spider nar/spi The project code (nar) will be used in later practicals for SPIDER script files (and is used for the results files). The data code (spi) matches the extension of the image files, which must be the same for all data files used in a SPIDER session, except when converting to or from different file formats. These 3 letter codes are defined by the user, and are used for identification of script and image files. Note that spider commands are NOT case-sensitive, while filenames ARE.

2

When in a SPIDER session, type all filenames without their extensions. In the exercises below, enter the provided text and numbers after each prompt. SPIDER prompts always start with a period (.) and end with a colon (:). If you want to back out of a command dialogue, type * when it asks for input filename. You can always exit spider by typing EN at the command prompt. If you are really desperate, you can crash out with ctrl-C. There is extensive documentation on the SPIDER website: http://www.wadsworth.org/spider_doc/spider/docs/operations_doc.html

Now, create a simple image with the MO command (create MOdel):

.OPERATION: mo .OUTPUT FILE: circ001 .ENTER DIMENSIONS (NSAM,NROW): 128,128 .B/C/G/R/S/T/W: c .RADIUS (FLOATING POINT): 20 .OPERATION: You will see SPIDER echo the input, as well as print out additional information. Once the command has all the required arguments, it carries out the processing, and returns to the .OPERATION: prompt. View the image you have created in Web (below).

WEB

Web is the interactive viewer for SPIDER files, and is often run simultaneously with a SPIDER session, allowing the user to view the output files. In another SSH window type 'web' followed by the data extension:

web_linux spi The first time you start Web, you will need to do the Xwindows setup as described on page 3 of Getting Started, and then restart Web. Adjust the Web and SPIDER windows so that you can switch between them. When Web starts up, you will see a blank background screen, with menus at the top and a status window at the bottom.

Options listed under the main menu headings will be written as: Menu!Command. For example, when Web is first started, you may have to clear the screen with the option Commands!Clear .

To view an image, use Commands!Image, and select the file to be viewed (circ001.spi). Web displays the image, and prints out some information (image size and range of the gray levels) in the status window at the bottom.

3

The size of the displayed image may be adjusted: select Options!Image, and enter ‘-2’ in the SIZE REDUCTION entry box. Positive numbers will decrease the size by the specified factor, while negative numbers will enlarge the displayed image. Use Commands!Image again to view circ001.spi. It should appear larger this time. All subsequently displayed images will also be enlarged, until SIZE REDUCTION is changed again. Remember, Web only displays a picture - it does not change the original data file. Also remember that interpolating large images to display them can hide artefacts in those images – viewing at the original size is advisable where possible!

As you try out the SPIDER commands below, you will go back and forth between SPIDER and Web, performing image processing in SPIDER, and using Web to view the results.

3D maps

A 3D map file can be visualized in Web by viewing it slice-by-slice, or by surface rendering. View the map elsatp.spi in Web:

Commands!Clear Commands!Image select elsatp.spi, click 'OK'

You will see slices of the map volume. Try the different options for viewing slices in x, y or z. Then create a surface rendering of the map.

Commands!Clear Commands!Surface select elsatp.spi, click ‘OK’ set scale to ~4.8 set threshold to 0.04, click 'Accept'

Click the left mouse button to change the surface settings, and the right mouse button to end the surface session.

Creating another test image in SPIDER

The PT command allows you to create geometric shapes. Here is another test image you can make using PT (create PaTtern):

.OPERATION: pt .OUTPUT FILE: test001 .ENTER DIMENSIONS (NSAM,NROW): 75,75 .ENTER CODE (P,L,C,T,B): b ; to `draw` a box (rectangle in this case) .ENTER COORDINATES OF UPPER LEFT CORNER: 18,12

4

.ENTER COORDINATES OF LOWER RIGHT CORNER: 38,16 .CONTINUE? (Y/N): y .ENTER CODE (P,L,C,T,R,S): t ; to draw a triangle .ENTER COORDINATES OF FIRST POINT: 18,33 .ENTER COORDINATES OF SECOND POINT: 56,36 .ENTER COORDINATES OF THIRD POINT: 32,66 .CONTINUE? (Y/N): N In Web, view the resulting test file, test001.spi with Commands!Image. It doesn't show up in the Image file selection dialog! When you want to display a new image file, you need to press the 'Filter' button, which forces Web to re-read the contents of the current directory.

Introduction to Filtering

SPIDER has a variety of image filtering operations. The FQ NP (Filter Quick, No Padding) and related Fourier filter (FF) commands provide a set of filters to choose from. Filter your test image with FQ NP. Remember that low pass filtering removes image components at higher frequency than the set limit, defined as a radius in transform units (can be fractional units as below or in transform pixels), and high pass filtering keeps high frequency information and removes low frequency components up to the set limit. NOTE: text after a semicolon (;) is a SPIDER comment. You DO NOT have to type the comment on the line prompting for FILTER TYPE!

.OPERATION: FQ NP .INPUT FILE: test001 .OUTPUT FILE: filt001 .FILTER TYPE (1-8): 3 ; selects Gaussian low-pass filter .FILTER RADIUS: 0.1 ; This is the fraction of the box size; ; radius = 0.5 is at the edge of the box View the result of the low-pass filter. Low-pass filters tend to blur the image, while high-pass filters emphasize the edges. Experiment with the filter by adjusting the filter radius, or try different filter types. What difference does it make to use Gaussian filters instead of simple low or high pass ones? Can you explain the cause of these differences? (Before using the Fermi and Butterworth filters you should first read http://www.wadsworth.org/spider_doc/spider/docs/man/ff.html)

Geometric and Algebraic operations

These include rotating (RT), shifting (SH), or resizing (IP) the image. Apply a shift operation to an image:

.OPERATION: sh .INPUT FILE: filt001 .OUTPUT FILE: shift001 .SHIFT COMPONENTS IN X-Y: 16.7, -7.2

5

View the result, shift001.spi. For most operations, the origin of SPIDER coordinates is in the upper left corner, which is pixel 1,1 (not 0,0!). So a positive X shift moves the image to the right, a negative Y shift moves the image up. Note that the image may be shifted by non-integer values.

Images may be added (AD), subtracted (SU), multiplied (MU) or divided (MU D). Subtract the image you just shifted from the original test image:

.OPERATION: su .INPUT FILE: test001 .SUBTRACTED FILE: shift001 .OUTPUT FILE: sub001 .SUBTRACTED FILE: * SU allows you to subtract as many images as you want - enter an asterisk (*) to terminate the command. View sub001.spi in Web. The two images were subtracted pixel-by-pixel. In the resulting 'difference image', positive areas (where test001 pixels > shift001 pixels) appear light, while negative areas are dark. Pixels where the two images were equal are zero, and appear as gray. Web always stretches the contrast so that the pixels with the smallest (or most negative) value are set to black, and the pixels with highest value are white. Note the range of this image is -1.0 ... 1.0. For images with different ranges, their gray levels in Web are NOT comparable.

Correlation

Cross correlation (CC) is a way to measure the similarity of 2 images. The 2 images are multiplied together, pixel by pixel, and the sum of these multiplications is stored at a location representing the position of image1 relative to image2. Then the images are shifted by one pixel, and multiplied again, with the resultant value stored at the next pixel. This is carried out by shifting image1 across all positions of image2 (the reference image). Pixels with high values ('peaks') in the cross correlation result show where the best matches were, while the amplitude of the peaks indicates the degree of similarity.

.OPERATION: cc .INPUT FILE: shift001 .REFERENCE FILE: filt001 .OUTPUT FILE: cc001 View the cross correlation result in Web. There should be a bright spot indicating the location of the best match, or highest correlation. Select Options!Contrast. The sliders allow you to change the way the pixel intensities are displayed. (note: this sometimes doesn’t quite work interactively on the PC – but the changes take effect once you move the pop-up window, or, on older systems, the next time you display the image). Move the left slider rightward (mouse or arrow keys) until the correlation result (redisplayed) appears as nearly a single pixel. Next, set both sliders to the same intermediate value, e.g., 90. What happens to the images? They are binarized - all pixels above the threshold are white, pixels below the threshold are black. These

6

contrast numbers refer to the actual pixels in the onscreen display, not to the floating point pixel intensities in the image files. When you are done, reset the contrast back to the original values (64,127).

Registers and Register Variables

SPIDER provides variables to hold temporary values. These variables are called 'registers' and are named X11, X12, ..., X99 (X01 through X10 are reserved). You can use registers anywhere you would normally enter a number in SPIDER. Registers can be used as a form of pocket calculator. At the prompt, type:

.OPERATION: x11 = 25 .OPERATION: x12 = x11 + 3.5 Typing the register alone at the prompt returns its value: .OPERATION: x11 25.000000 .OPERATION: x12 28.500000 In recent versions of SPIDER (newer than ~v14), you can also use ‘register variables’. Their use is essentially identical to standard variables, with the following syntax: .OPERATION: [shift1] = 25 .OPERATION: [shift2] = 3.5 .OPERATION: [shiftsum] = [shift1] + [shift2] .OPERATION: [shiftsum] 28.5000000 This is more cumbersome for casual use, but invaluable in making long SPIDER procedures readable. For the rest of the examples in this practical, standard registers are used, but subsequent practicals involving prewritten procedures will use a mixture! Peak search

A peak search is commonly applied after a correlation, to find the location of the best match - a 'peak' in the pixel intensities. The PK command prints its results to the screen, but it can also store these values in registers on the command line. In the following example, registers x11 and x12 are set to the X and Y coordinates of the highest peak found. Apply a peak search to the cross correlation result to find the coordinates of the bright spot:

.OPERATION: PK X11,X12 .INPUT FILE: cc001 .ENTER # OF PEAKS, CENTER ORIGIN OVERRIDE (0/1): 1,0 ; 0 means don’t ; change origin

7

Check the values of the registers: .OPERATION: x11 17.000000 .OPERATION: x12 -7.0000000 These numbers show the position of the highest peak (best match) relative to the image center. These are integers, but there is a way to get more accurate floating point values (see PK in the Operations Manual). To shift back to the centre, we have to use the negative of these numbers. Put the negative values in other registers: .OPERATION: x21 = -x11 .OPERATION: x22 = -x12 Translational Alignment

Use the results from the peak search to align the shifted image to the original test image. For the X,Y shift components, use the negative of the numbers obtained from the peak search (i.e., -17 and 7). Since we have these numbers in registers, use registers x21 and x22 in place of the actual numbers.

.OPERATION: sh .INPUT FILE: shift001 .OUTPUT FILE: align001 .SHIFT COMPONENTS IN X-Y: x21, x22 ; instead of -17,7 Compare the 'aligned' image with the original test image by subtracting them: .OPERATION: su .INPUT FILE: test001 .SUBTRACTED FILE: align001 .OUTPUT FILE: sub002 .SUBTRACTED FILE: * How does the subtracted image sub002.spi appear? Compare to sub001.spi. The new difference image is zero (gray) everywhere, except around the edges of the shapes. This is because the edges were blurred in the filtered image, and therefore do not match the original sharp edges exactly.

Document files

Information obtained during a SPIDER session can be saved to document files (called doc files). Data is written to a doc file with the SD (save document) command, which writes one line at a time. The numbers to be stored are placed on the SD command line. The first number, (here 1), is the key. This doc file can be read by other SPIDER commands; the key tells SPIDER which line to read.

8

.OPERATION: sd 1, x11, x12, x21, x22 .DOCUMENT FILE: doc001 This writes the values of those four registers to the file doc001.spi. This file can be viewed in a text editor, or with the Unix 'cat' command (it may not be visible until the SPIDER session has ended).

The doc file has several entries:

;nar/spi 01-JUN-2002 AT 20:29:05 doc001.spi 1 4 17.000 -7.0000 -17.000 7.0000 The first line, starting with the semicolon, is a comment line with file creation information. The lines following the comment have the following format:

<key> <number of data items in this line> <data> <data> <data> <data>

Older SPIDER versions used a fixed width format, but current versions separate the data columns by blank space and supports up to 999,999 entries (lines) with no set limit on line width.

Procedures that read out values from a document file use the curiously named command UD (Unsave Document).

Note in the prewritten procedures used for later practicals, you will see UD IC and (SD IC) being used instead. This command reads the entire document file into memory and subsequent reads are very fast. For large, computationally intensive tasks, correct use of UD IC can dramatically speed up procedures.

Fourier transforms and how to view them in WEB The alignment and filtering operations use the Fourier transform (FT) of the images, even if this hidden from the user by the Spider (or Imagic) commands. It is important to understand how the data appear when represented as a function of spatial frequency instead of distance. FTs are made of complex numbers (or amplitudes and phases), and cannot therefore be simply displayed as grey-scale images. We normally extract the amplitudes for display. To create the FT amplitudes for viewing we will use SPIDER operation PW i.e. the PoWer Spectrum, which describes how the power of a signal is distributed with frequency. Note that PW operation in SPIDER furnishes the amplitude, not the squared amplitude. This makes it easier to use the distribution in displays and other operations, because of the high dynamic range. A reminder of the relationship between the FT and Power Spectrum is on the next page, using the example of the Fourier transform of a square wave.

9

phases 0° Plot of FT phases 180° Amplitudes only (square root of diffracted intensities) Intensities (power spectrum) (observed in diffraction pattern) We will use the circle you created at the start of this practical (circ001), as well as a smaller circle and a rectangle to create and look at some simple FTs. .OPERATION: MO circ002 128,128 c 5 .OPERATION: PT rect001 128,128 b 40,60 90,70 n .OPERATION: PW circ001 circamps001

10

Create amplitude files for circ002 and rect001, naming them accordingly. Now you can compare the Fourier amplitudes in WEB. Note that you will have to adjust the contrast, since the Fourier transform of an image is usually dominated by very high values at the centre and the maximum intensity needs to be reduced in order to see weaker features at higher radii. Commands!image – display circamps001 Options!contrast – set max (127) to ~60, then move the pop-up window or redisplay the image. Then display the other two amplitude spectra. Note the features of the FT amplitudes. How do they differ? Can you explain the differences, and also the features of the rectangle transform? Fourier transforms and filtering Here we will investigate the effects of filtering on a more realistic model image and on its Fourier amplitudes. Use the second image that you copied to your directory at the start (hh256.spi). Calculate its FT: .OPERATION: FT hh256 hhft Then create low pass and high pass filtered versions of the FT: .OPERATION: FF hhft hhftlow05 3 ; Gaussian low pass filter .05 ; filter radius Repeat with a radius of 0.2, naming the result hhftlow2. Then repeat the FF command, using a high pass filter (type 4), with a radius of 0.2, naming the result hhfthigh2. To view the results of these filtering operations, you must transform the filtered FTs back to real images, using the FT command: .OPERATION: FT hhftlow05 hhfiltlow05 Repeat the inverse transform operation for the other two filtered images. You can also view the filtered transforms, if you first convert them to amplitudes by running the PW command, using the filtered transforms as inputs. Give them names of the form hhpwlow05.

11

Now you can use WEB to view all the results. What are the different filters doing? In order to see the weak features on the transforms, you will need to set the contrast down from 127. Reset it to 127 when viewing the filtered images. Note: the above series of commands (FT!FF!FT) is precisely what the command FQ NP does internally. What happens if you add together the low and high pass filtered images? Try summing the low05 and the high pass filtered images, and then the low2 and high pass, and view the results in WEB. If you have time, you can also check what result is obtained by summing the FT files and then inverse transforming the sum. Be careful to note the file names so that you don’t mix things up. Use the AD command for adding images. .OPERATION: AD hhfiltlow05 hhfilthigh2 hhsum1 *

End the SPIDER session:

.OPERATION: en d note – “en d” deletes the RESULTS file, which you normally only need if the program crashes – it records all the steps in your session and can be used to trace errors when procedures crash. The results files can get very large, so you don’t want to accumulate them. They are numbered so that old ones are not overwritten – it is up to you to tidy up. To exit and keep the results file, type EN or END. Output from your session is also recorded in a LOG file, which is overwritten for each session. To clean up the files in your directory, you can write an alias: alias clean ‘\rm results.* LOG.* fort*’ (or simply type the command inside the quotes). To Exit Web click System!exit

12

Part 2: : Surveying your image and picking particles with the EMAN program “boxer” EMAN and boxer will run on most platforms, and works with menus and on-screen instructions. Again we will be using the LINUX version, so use your SSH window: 1. Type boxer (note: if this window is not visible then go to the task bar at the bottom of the screen and right click on the boxer tab and select maximise) 2. File!Read Micrograph Select scan.spi A window containing the image will appear. Boxer will read even very large micrographs, but some of the operations will become slow. Boxer reads SPIDER, MRC, IMAGIC, and some other formats. You can move around the image by clicking and dragging with the right mouse button, or you can click in the image area of the main boxer window. Images are resized with the “scale” parameter in the main boxer window (n.b. a scale of zero will display the whole image scaled to the current window size. 3. Contrast scaling. Clicking in the image with the middle mouse button brings up a second control panel.

This allows you to: a) view the histogram and rescale the image contrast in various ways (experiment with the two sliders) b) invert the image c) calculate the Fourier transform and display the power spectrum (FFT button). Note that image should be square if you want the same scale in X and Y.

13

4. Measuring distances. – Set the correct “Å/pix” sampling in the main boxer window: 5.2 Å/pix. Now click the measure button. You can now click and drag a line on the image to measure distances. 5. Manually selecting particles. Set “box” to the desired box size (in pixels) and the press the select button. Now click on particles in the image. Incorrectly selected particles can be removed by clicking the delete button, and clicking on the bad boxes in the image. Boxer saves picked particles (Boxes!Save Boxed Particles) as a stack of images. If the same file name is used, new images will be appended at the end of the existing stack. 4. Filtering. The process menu allows various filters to be applied to micrographs to help to see particles. The median filters are the most useful (note: if you filter the image and save the boxed particles after picking from the filtered image the boxed particles will also have the filter applied. You should save the particle database first (Boxes!Save Box DB), then open the original file and re-pick particles using saved database: Boxes!Open Box DB) 7. Automatic particle selection. Boxer contains a several very useful automatic particle selection tools. The simplest is the AUTOBOX function. Set the box size as before, and select three or four representative particles. Now Select “autobox” with Boxes!Autobox. A dialogue box appears with several sliders. These allow you to tune the automatic particle selection that has appeared in the centre of the image window. The most useful is the “Ptcl Location” slider – lowering the value will make the selection “more indiscriminate” (note: if you reduce this too much it cause boxer to crash). Select a level which you are happy with, and press OK. The operation will now be run on the whole micrograph. You can now save either the resulting set of coordinates or the cut particles themselves, with options from the Boxes menu. 8. Excluding parts of the image. You may find that autobox likes to select areas of high contrast, such as contamination or the edge of holes. These can be “drawn out” of the image before autobox is run. Select the draw button. You can now interactively blank out image areas with the mouse. 9. Boxing filaments/helices. For those working with filaments, boxer has the option to select particles between 2 defined endpoints. Open micrograph of microtubules decorated with Kinesin 5 MT.spi (scanned at 2.8 Å/pix), click on the Helix button, choose your box size and overlap (0 for no overlap); click once on one end of the filament and then click once on the other end. You can adjust the position of either endpoint. You can explore Rotate Helix and Align Helix options. Option Unbend Helix does not yet work properly in the current Boxer version but should work in subsequent releases. Help pages for EMAN generally: http://ncmi.bcm.tmc.edu/ncmi/homes/stevel/EMAN/doc For Boxer in particular: http://ncmi.bcm.tmc.edu/ncmi/homes/stevel/EMAN/doc/progs/boxer.html

14

y=171

x=681

y=171

x=681

We suggest you finish this practical with another Fourier transform example. The task is to compare a power spectrum of the whole MT.spi image with that of a single microtubule. Using SPIDER you can cut out one straight vertical microtubule from MT.spi. Consult the online description of the WI Spider operation (http://www.wadsworth.org/spider_doc/spider/docs/man/wi.html). View file MT.spi in Boxer and choose the long and straight area of a vertical microtubule. Determine the size of rectangle you want to cut out (in pxls) and coordinates of its left lower corner using Measure option in Boxer. Note that when viewed with Boxer X- and Y-axis of Spider files start at the left lower corner that corresponds to left upper corner in Spider manual! You can open your new rectangular image in Boxer but to see its power spectrum you need first to pad it into a square. This can be done using the PD operation in Spider. (Consult the online Spider manual for pd). If you pad your image into a 1500x1500 square - the same dimensions as the original MT.spi file - you may use the same coordinates for the top left corner you determined for windowing. Answer 'N' (no) for 'background value of the output image' and enter '0' when asked for 'background constant for padding'. Open your new file and MT.spi one after the other in Boxer and examine the image transforms: how would you explain the difference? Microtubules are helical or pseudohelical assemblies of !"-tubulin heterodimers, which have a repeating structure along their axis. The lattice gives rise to a set of diffraction spots arranged in rows (layer lines). In a power spectrum of the single microtubule you should clearly see the equator and the 4 nm layer line that corresponds to the helical arrangement of tubulin monomers. Can you find the 8 nm layer line whose reflections are enhanced due to the kinesin 5 bound to !"-tubulin? You will hear more about helical assemblies in Bridget Carragher's lecture.