Getting data home

?

Getting data home

?

FTP

Getting data home

?

FTP tapes

Getting data home

?

FTP: not securetapes: incompatible

Getting data home

?

FTP: not securetapes: incompatible

distracting

Getting data home

?

http://bl831.als.lbl.gov/~jamesh/beamline/moving_data.html

Getting data home

?

http://bl831.als.lbl.gov/~jamesh/beamline/moving_data.html

http://bcsb-web.als.lbl.gov/beamline/Sector8/DataBackup.htm

Getting data home

?

http://bl831.als.lbl.gov/~jamesh/beamline/moving_data.html

http://bcsb-web.als.lbl.gov/beamline/Sector8/DataBackup.htm

http://ucxray.berkeley.edu/~jamesh/elves/manual/tricks.html

Can you login?

ssh user@home

Universal directory-transfer

tar cf - . |

ssh user@home

“cd /dir ; tar xvf -”

Universal directory-transfer

tar cf - . |

ssh user@home

“cd /dir ; tar xvf -”

Universal directory-transfer

tar cf - . |

ssh user@home

“cd /dir ; tar xvf -”

Password:

Ctrl-z

bg

Making it faster

tar cf - . |

compress |

ssh user@home

“cd /dir ; uncompress | tar xvf -”

The easy way

http://bl831.als.lbl.gov/~jamesh/elves/scripts/sendhome

sendhome . user@home:/dir

Real Time

http://bl831.als.lbl.gov/~jamesh/elves/scripts/sendhome

sendhome first_001.img user@home:/dir

Through a firewall

tar cf - . |

ssh user@firewall

‘ssh fileserver

“cd /dir ; tar xvf -”’

The other way

ssh user@home

cd /dir

ssh user@beamline

“cd /data ; tar cf - .” |

tar xvf –

The other way, faster

ssh user@beamline

“cd /dir ; tar cf - . | compress” |

uncompress |

tar xvf –

The easy other way

ssh user@beamline

“download yourname” |

tar xvf –

What if it stops?

What if it stops?

ssh user@beamline

“download yourname frame_046.img” |

tar xvf –

What if it stops?

http://bl831.als.lbl.gov/~jamesh/elves/scripts/sendhome

sendhome frame_046.img user@home:/dir

What if I’m confused?

What if I’m confused?

4 steps

1) Generate a filename list

find . -name ‘*.img’

>! here.log

2) Generate a “done” list

ssh user@home

“cd /dir ; find .”

>! there.log

3) Make the “todo” list

diff there.log here.log |

awk ‘/^>/{print $2}’

>! 2bsent.log

4) Feed “still left” list to tar

cat 2bsent.log |

tar cTf - - |

ssh user@home

“cd /dir ; tar xvf -”

What if I can’t login?

ssh user@home warning: Connecting to home failed: Connection Refused

“SneakerNet”

“SneakerNet”

Tape

Firewire

DVD

FireWire

OS Format

Mac HFS

Windows NTFS

Linux FAT32

Realtime Firewire

1)plug in

Realtime Firewire

1)plug in

2)share it

Realtime Firewire

1)plug in

2)share it

3)smbclient

Realtime Firewire

tar cf - . |

smbclient

//laptop/share –U user%password" -N -Tx -

Realtime Firewire

find . |

tar cTf - - |

smbclient

//laptop/share –U user%password" -N -Tx -

The easy way

http://bl831.als.lbl.gov/~jamesh/beamline/backup_firewire.com.txt

backup_firewire.com .

DVD

OS Format

Mac HFS

Windows NTFS

Linux UFS

DVD

OS Format

Mac ISO

Windows ISO

Linux ISO

HOWTO make a DVD

mkisofs -r -J .

>! image.iso

HOWTO make a DVD

mkisofs -r -J .

>! image.iso

dvdrecord dev=0,0,0

–data image.iso

The easy way

http://bl831.als.lbl.gov/~jamesh/beamline/backup_local.com.txt

backup_local.com .

The easy way

http://bl831.als.lbl.gov/~jamesh/beamline/backup_local.com.txt

backup_local.com .

http://bl831.als.lbl.gov/~jamesh/beamline/autolabel.com.txt

autolabel.com

DVD data archive

DVD data archive

DVD data archive

DVD data archive

DVD data archive

DVD data archive

DVD data archive

George Meigs

Jane Tanamachi

Beamline 8.3.1 staff

Acknowledgments