[CB16] EXOTIC DATA RECOVERY & PARADAIS by Dai Shimogaito

@ CODEBLUE 2016 on Thu 20 Oct 2016

DAI SHIMOGAITOOSAKA DATA RECOVERY ( daillo,inc. )

Who is Dai Shimogaito ?

Dai Shimogaito is a Japanese ,

Data Recovery Engineer – Retrieving data from computer crash

Digital Forensic Investigator – Examining digital evidences

Cyber Security Researcher – On hidden data area in HDD

h t t p s : / / w w w . f a c e b o o k . c o m / d a i . s h i m o g a i t o

Let’s Open it and see the structure !

Inside HDD looks like this

4 Main Parts

12

3

1. DISK

12

3

This circle flat disk like a mirror is

the data recording DISK.

This part holds DATA and Firmware.

3

Read and Write HEAD is located at

the tip of the black rectangle part,

SLIDER.

2. Head Stack Assembly ( HSA, Head )

2

Main Controller and ROM are located.

Rom contains the 1st part of firmware.

Data port

Power port

RAM

3

3. PCB ( Printed Circuit Board )

Firmware is the implemented software for controlling

the movement of DISK and HSA to Read/Write data.

ROM contains the starting part of the firmware.

DISK contains the rest of the firmware.

4. Firmware

4

SA and UA

SA Service Area

Most of the firmware ( SA modules ) is stored

UA User Area

User data such as operating system, pictures,

and document files and directories are saved

Spare sectors are here

SA and SA Module

SA Service Area

Most of the firmware ( SA modules ) is stored

SA ModuleEach module has its own function as firmware

such as P-List, G-List, S.M.A.R.T. and ATA-PW.

The number of SA modules differes depending

on the design of the product

1. Power ON

2. Controller reads ROM

3. Disk spins up and Head moves to SA

4. Controller reads SA Modules

5. Ready

What happens during HDD booting


SA Module ROM SA Module SA ModuleSA Module SA Module SA ModuleSA Module SA Module SA Module

Power ON Ready

SA Module

RAM

Controller


Power ON Ready

RAM

Controller

Power ON Not Ready

RAM

Controller

The cause could be,,,,1. Head is bad for reading the SA Module

2. Disk area for the SA Module is bad

3. The content of the SA Module is bad

Operating System not found

Impossible

to access

any data

Internal Sector Location Management

Head 0

Head 1

Which Cylinder ( = Track ) ?

Which Head ( = Surface ) ?

Which Sector ?

By CHS, the physical location of a sector inside the HDD can be specified.

PBA ( Physical Block Address ) is assigned to each physical sector.

PBA 0 = CHS( 0 , 0 , 0 )

PBA 1 = CHS( 0 , 0 , 1 )

PBA 2 = CHS( 0 , 0 , 2 )

PBA 3 = CHS( 0 , 0 , 3 )

PBA 4 = CHS( 0 , 0 , 4 )

PBA 5 = CHS( 0 , 0 , 5 )

,

PBA 10 000 000 = CHS( 234 , 1 , 18 )

PBA 10 000 001 = CHS( 234 , 1 , 19 )

PBA 10 000 002 = CHS( 234 , 1 , 20 )

PBA 10 000 003 = CHS( 234 , 1 , 21 )

PBA 10 000 004 = CHS( 234 , 1 , 22 )

PBA 10 000 005 = CHS( 234 , 1 , 23 )

,

Sector is specified by

PBA instead of LBA

inside HDD

* The values are not actual information. This is an example.

LBA is mapped to PBA

Physical sector location management

in HDD is controlled by

PBA

Logical sector location management

in PC is controlled by

LBA

PBA 0 ↔ LBA 0

PBA 1 ↔ LBA 1

PBA 2 ↔ LBA 2

PBA 3 ↔ LBA 3

PBA 4 ↔ LBA 4

PBA 5 ↔ LBA 5

PBA 6 ↔ LBA 6

PBA 8 ↔ LBA 7

-

-

PBA 640768 ↔ LBA 623001

PBA 640769 ↔ LBA 623002

PBA 640771 ↔ LBA 623003

PBA 640772 ↔ LBA 623004

PBA 640773 ↔ LBA 623005

PBA 640774 ↔ LBA 623006

PBA 640782 ↔ LBA 623007

PBA 640783 ↔ LBA 623008

Firmware

Physical sectors & LBA / ! misunderstanding !

Total number of physical sectors are equal ?

Physical sector to which LBA is not mapped


LBA is mapped to all the physical sectors ?

Physical sector to which LBA is not mapped Physical sector to which LBA is mapped


NO !


Total Number of Physical Sectors differs HDD to HDD


HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

1

Primary Defects on Disk ( P-List )

Physical sector to which LBA is not mapped Physical sector to which LBA is mapped Physical sector to which factory has skipped mapping LBA

List of the location information of is called "P-List"

P-List ( Primary Defects List )

P-List is saved in SA as an SA Module

P-List is unique and essential part of firmware

HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

2

At the time of Factory Shipment


Equal Number of LBA are mapped to each HDD

so that the capacity would be the same

HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

3

Focus on LBA mapped sectors distribution

Accessible sectors are physically NOT continuous from the 1st LBA to the last LBAH

DD

-A

HD

D-B

HD

D-C

PH

ASE-0

4

Total number of LBA are equal to each HDD

Accessible sectors are physically NOT continuous from the 1st LBA to the last LBA

On the contrary, accessible sectors are logically continuous from the 1st LBA to the last LBA

HD

D-A

HD

D-B

HD

D-C

HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

4P

HA

SE-0

5

Let's see how Bad Sectors appear

At the time of Factory Shipment

Mint Condition


HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

3

Bad Sectors after Bad Sector Reallocation


Bad sector after bad sector reallocation

List of the location information of is called “G-List"

G-List ( Growth Defects List )

G-List is saved in SA as an SA Module

G-List is unique and essential part of firmware

After G-List is cleared, past data may appear.

HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

6

Bad Sectors after Bad Sector Reallocation



HD

D-A

HD

D-B

HD

D-C

HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

6P

HA

SE-0

7

Possible to access bad sectors only by E-SE


Q1.

Can we access the LBA

mapped physical sectors ?

Q2.

Can we access the Bad

Sectors, after bad sector

reallocation, to which

LBA is not mapped ?

A1.

YES

A2.

Basically

NObut Enhanced Secure Erase

can access exceptionally

for trying to erase data

PBA LBAFirmware


PBA Firmware LBA

HD

D-B

HD

D-B

PH

ASE-0

8P

HA

SE-0

7

Comparison of 3 data erase methods for HDD

Secure Erase( ATA Command / Purge )

Enhanced Secure Erase( ATA Command / Purge )

Data Erase Software( Overwrite / Clear )

&

One and Only method

which may erase the

Largest data area

Limited to the

LBA mapped areaLimited to the

LBA mapped areaor less

Physical sector to which LBA is mapped Bad sector after bad sector reallocation

Shown only the physical sectors which may be erased ( accessed ) by each method

HD

D-B

HD

D-B

HD

D-B

PH

ASE-0

8

PH

ASE-0

8

PH

ASE-0

7&08

Comparison of 3 data erase methods for HDD

Physical sector to which LBA is mapped Bad sector after bad sector reallocation

Secure Erase( ATA Command / Purge )

Enhanced Secure Erase( ATA Command / Purge )

Data Erase Software( Overwrite / Clear )

&

One and Only method

which may erase the

Largest data area

Limited to the

LBA mapped areaLimited to the

LBA mapped areaor less

All the physically existing sectors are shown

Physical sector to which LBA is not mapped Physical sector to which factory has skipped mapping LBA

HD

D-B

HD

D-B

HD

D-B

PH

ASE-0

6

PH

ASE-0

6

PH

ASE-0

6

Survey of total physical sectors in 3 HDDs

2TB SATA HDD * 3

Same model, Same capacity

( Capacity:3 907 029 168 LBA ）

HDD-A HDD-B HDD-C

Total PBA 3 931 988 368 3 933 712 984 3 933 659 976

Difference from

LBA24 959 200 26 683 816 26 630 808

Difference in

Bytes12 779 110 400 13 662 113 792 13 634 973 696

Difference in % 0.635% 0.678% 0.677%

( Total PBA) - ( Total LBA ) = Difference = Surplus Physical Sectors

Survey of total physical sectors in 3 HDDs

Surplus Physical Sectors are inaccessible,

because LBAs are not mapped to them

PBA Firmware LBA

What if there is DATA ?

Enhanced Secure Erase by Ultimate Boot CD

SN and Model are recorded with finish time

Demonstration

Let's connect HDD with a write blocker and view the LBA 0 by binary editor

Firmware defines the appearance of DATA

LBAPBA Firmware

1LBAPBA Firmware

2

Firmware 1

Physical location of MBR ( LBA 0 ) may even differ depending on the firmware

Firmware 2

LBA is NOT always mapped to the same PBA forever. It's UNSTABLE !

HD

D-B

HD

D-B

HD

D-B

PH

ASE-0

8

PH

ASE-1

4

PH

ASE-0

1

P A R A D A I S

When LBAs are mapped to the surplus physical sectors, they become accessible

despite they used to be inaccessible even by enhanced secure erase.

1. It may remain even after initializing and formatting.

2. It may remain even after OS installing / reinstalling.

3. Malware may preexist but no way to detect.

4. Inaccessible by conventional methods.

5. Any software and data may be stored.

6. There is no restriction.

7. Whatever you want.

8. Free space FOR "SOMEONE" LBAPBA Firmware

?

▼

HD

D-B

PH

ASE-1

0

3 year old HDD may look like these



HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

6

PARADAIS



HD

D-A

HD

D-B

HD

D-C

HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

6P

HA

SE-0

9

PARADAIS


Bad sector after bad sector reallocation PARADAIS

HD

D-A

HD

D-B

HD

D-C

HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

6P

HA

SE-1

0

PARADAIS



HD

D-A

HD

D-B

HD

D-C

HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

6P

HA

SE-1

1

PARADAIS



HD

D-A

HD

D-B

HD

D-C

HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

6P

HA

SE-1

2

PARADAIS



HD

D-A

HD

D-B

HD

D-C

HD

D-A

HD

D-B

HD

D-C

PH

ASE-0

6P

HA

SE-1

3

Are these physically acquired disk images ?

Could be, but not always.H

DD

-A

HD

D-B

HD

D-C

PH

ASE-0

5

These have been missed by disk imaging tools

HD

D-A

HD

D-B

HD

D-C

PARADAIS Activation

PARADAIS can be activated either by external or internal trigger.

1. External ActivationWhen a certain ATA command is sent to HDD, PARADAIS may become ready to be activated

for the next power ( boot ) session.

2. Internal ( Self ) ActivationWithout any external trigger, it may be activated. Just wait until it gets activated, someday.

This trigger works for offline PCs, therefore the activation may occur even in

air-gapped control systems.

▲Manipulating /etc/shadow

for login to Debian Linux

as root

▲

/etc/shadow

Without external operation,

unidentified partition appeared

suddenly after reboot

PROBLEMS

Consumers & Users Vendors & Makers

You should be

responsible for the

accident !

I will sue you !

We had never

expected such

incident.

Product Liability

PROBLEMS

Court Judge

Are you sure ?

Law Enforcement

Your honor,

We've examined all the

data area of the HDD.

Physically extracted

image file is a perfect

copy of the HDD.

Digital Forensics

PROBLEMS

Victim Criminal

My data is gone,,,

Hostage for RANSOM

Your data is in your HDD,

but inaccessible for you.

If you pay me ransom,

your data would be back.

PROBLEMS

None of Data Erasure software can erase all sectors.

Data Erasure

PROBLEMS

CyberTerrorism

PROBLEMS

Cyber Security

What do you wanna embed here ?

What could be embedded here ?

Solutions for PARADAIS activation

1. HDD inspection before use

The more critical the data is, the better it is to inspect the firmware of HDD before use.

Block the activation of PARADAIS even if there is unidentified data there.

To do so, firmware inspection would be useful to eliminate the activating mechanism.

Erase data on the surplus physical sectors.

To do so, first LBA mapping to the surplus physical sectors is required and then erase data.

2. Select reliable distribution channels

Who do you buy HDD from ? Why do you buy HDD from them ?

This research is goin on / Important Notice

Although I have described the mechanism of HDD and PARADAIS, it is unknown if

PARADAIS exists in all HDD products of all the manufacturers.

It could be possible that it exists only in several models that I have verified so far,

because the structure and the mechanism differ depending on the design of each

manufacturer and model.

To make it more precise and clear, it is preferable to explain on each different

product. However, it could affect the product's reliability. So I've been avoiding

mentioning the name of the products and the manufacturers so far.

I would appreciate your understanding.

このPARADAISがどのメーカーのどの製品にどの程度存在しうるのかについてはまだ調査の余地が残されており、ＨＤＤはメーカごとの設計等によっても構造が異なるため、より具体的かつ正確性を確保するためには、各製品の設計や仕様に沿った検証が本来ならば必要ですが、当研究の提言内容は、ともすれば特定のメーカや製品の信頼性に影響を及ぼし兼ねないとの考えにより、積極的に特定のメーカ名や製品名を公表することは今の時点では控えております。この点につきましてご理解を賜れますようお願い申し上げます。

2nd Part

After a Natural Disaster, HDD can look like this

DR from scratched disk had been impossible

If the surface is partly

damaged, there should be

recoverable data in the areas

which were not damaged.

Disk Burnishing Process

The 1st step of the research completed with a good result

0.02%

94%

UP !

Newspaper : Nikkei Business Daily,

26th Septempber 2013

This was a joint research with Kansai University

and Osaka Data Recovery ( daillo,inc. )

Newly developed DDRH

Survey of 12 DR cases

No. Model Failure StateDifficulty

LevelAfter Cleaning by DDRH Effect

1 ST2000DM001 Unable to boot / Abrasion Powder B Improvement in serial port output C

2 ST2000DM001 Unable to boot / Abrasion Powder B Improvement in serial port output C

3 WD10EADS-22M2B0Unable to boot / HSA Replacement /

FW ModificationD Read error partly solved B

4 SV1203NUnable to boot / HSA Replacement /

FW ModificationC Read error solved B

5 ST3000DM001Unable to boot / HSA Replacement /

FW ModificationC Improvement in serial port output C


FW ModificationB Improvement in serial port output C

7 ST2000DM001 Abrasion Powder a lot A No improvement D

8 ST1000DM003 Bootable E No change in serial port output D


FW ModificationC Read error partly solved C

10 ST31000528AS Unable to boot / FW Modification C Read error partly solved C


FW ModificationC Read error partly solved C

12 ST3000DM001 Unable to boot B Became bootable A

Difficulty Level

Disk surface totally turned into abrasion powder A

Disk Scratched Damage B

HSA Replacement and more processes required C

HSA Replacement required D

Minor Failure ( Part replacement not required ) E

Effect

Remarkable improvement A

Significant improvement *1 B

Improved C

No effect D

Became Worse E

This survey report was submitted to Osaka city

because the research and the development of

DDRH were partly funded by Osaka city subsidy

program in March 2016.

*1 More than 1000 read error sectors solved

Survey of 12 DR cases

Remarkable

improvement

8%

Significant

improvement

17%

Improved

58%

No effect

17%

Became

worse

0%

Disk surface totally turned

into abrasion powder

8%

Disk Scratched

Damage

34%HSA Replacement

and more processes

42%

HSA

Replace

8%

Minor

failure

8%

Difficulty Level of Data Recovery Cleaning Effect by DDRH

Disk surface cleaning worked for approx. 80% of the DR cases.

Ongoing Research

FIRMWARE & PARADAIS

Bad

Lubricant Layer &

Disk Surface Cleaning

Good

Thank you very much for attending this lecture !

Date post:	23-Jan-2018
Category:	Technology
Upload:	code-blue
View:	159 times
Download:	6 times

[CB16] EXOTIC DATA RECOVERY & PARADAIS by Dai Shimogaito

Technology