28
VNR SQLite Carver Veronika - Rusolut
VNR SQLite CarverVeronika - Rusolut
PСLaptopsMobile devices
Flight softwareCar multimedia systemsDrones
Most applications that people use every day
Well-knownoperatingsystems
WEB browsersFile hosting services
SQLITE IS EVERYWHERE
WHAT’S INSIDE SQLITE DATABASE
id name surname created_at updated_at
1 John Jones 2019-03-15 08:08:52 2019-03-24 08:06:10
2 Maria Mendis 2019-03-16 06:10:00 2019-03-26 02:51:12
N Joe Black 2019-03-16 08:30:03 2019-04-11 11:47:38
Database
Record(row)
Field(column)Primary key
Table
DATABASE TRANSACTION
REATE
EAD
PDATE
ELETE
Initial state
Initial state
End state
Begin
COMMIT
ROLLBACK
transaction
Completedtransaction
Failedtransaction
DATA RECOVERY LOOK INTO SQLITE
FS metadata
Allocated spaceUnallocated
spaceAllocated
spaceUnallocated
space …
clusters
Logical image/File System level
SQLiteFile level
SQLitePage level
Header
Old page 2
…
Old page N
Ro
llbac
kJo
urn
alfi
le
Header
New page 2
New page N
The newest page 2
…
Wri
te-A
he
ad L
og
file
Page 1
Page 2
...
Page N
Mai
nd
atab
ase
file
Data
Unallocted space
Page 1 Page 2Page N
MAIN SQLITE DATABASE FILELock-byte page
Freelist
Payload overflow page
B-tree page
Pointer map page
B-tree page No.1
(*.db, *.sqlite, *.sqlite3 etc.)
THE ROLLBACK JOURNAL
Page 1
Page 2
Page 3
…
Page N
Data change
Commit
Old page 2
Old page 3
Old page N
…
Page 1
New page 2
New page 3
…
New page N
Initial state
Main DB file Main DB file
Rollback journal file
Rollback journalfile header
File
sys
tem
un
allo
cate
dsp
ace
DEL
ETE
PER
SIST
TRU
NC
ATE
Rollbackjournal
file
Rollbackjournal
file
Rollbackjournal
file
Size = 0
Zeroedheader
(“-journal” file)
THE WRITE-AHEAD LOG
Page 1
Page 2
Page 3
…
Page N
Data change
Checkpoint
Page 1
New page 2
New page 3
…
New page N
New page 2
New page 3
New page NCOMMIT
…
The newest page 3
The newest page NCOMMIT
Page 1
Old page 2
Old page 3
…
Old page N
Initial state
Main DB file Main DB file Main DB file
WAL file
WAL file
WAL file header New page 2
New page 3
New page NCOMMIT
…
The newest page 3
The newest page NCOMMIT
…
WAL file header
(“-wal” file)
File
sys
tem
un
allo
cate
dsp
ace
SQLITE DATABASE PAGES
Unused page
Database file header
B-tree page header
Reserved region
Cell content/Data area
Unallocated space/Deleted cells
Cell pointer array
Freelist- Deleted data
B-tree page No.1
…
Record
Recordlength, B
ROWIDRecord header
length, BType of Field1
…Type of Field N
Data of Field 1
…Data of Field K
Record header
CELL STRUCTURE
0 0 NULL
1 ≤ N ≤ 4 N Signed integer
5 6 Signed integer
6 8 Signed integer
7 8 IEEE float
8 0 Integer 0
9 0 Integer 1
N ≥ 12 even (N-12)/2 BLOB
N ≥ 13 odd (N-13)/2 TEXT
Type of Field Data size, B Data type
DELETED CELL FROM UNALLOCATED SPACE
Сhanged data of deleted cell
Recordlength, B
ROWIDRecord header
length, BType of Field1
…Type of Field N
Data of Field 1
…Data of Field K
SQL table definition SQLite storage classes
INTEGER
TEXT
NUMERIC
BLOB
REAL
CREATE TABLE my_table (
id INTEGER PRIMARY KEY,
productID BIGINT,
address TEXT,
name VARCHAR(255),
date DATETIME,
adds NUMERIC,
image BLOB,
longitude REAL,
latitude REAL)
INTEGERType 1-6,8,9
TEXTType ≥ 13 odd
NULLType 0
BLOB
Type ≥ 12 even
REALType 7
REPRESENTATION OF SQL TABLES
SQLite data types
- Is that all?
- Yes. No. It’s complicated!
Any column in an SQLite database may store a value of any storage class …
VNR SQLite Carver
… and so we decided to create
Manual AutomaticSemi-automatic
VNR SQLite Carver
• PROS• Detailed data analysis
• CONS• Extremely time consuming• You need to convert raw
data manually• Requires very deep
knowledge of databasetheory
HEX Viewer Well-known mobileforensic solutions
• PROS• Universal flexible platform• Get results in a few simple
steps• Adjustable results• More data than other
solutions• CONS
• You may need a minimal SQLite knowledges
• PROS• One-click solutions• Easy to use interface• No SQLite knowledges
required• CONS
• Lack of flexibility• Need to wait for updates
from vendor support• Limited application support
EXISTING SOLUTIONS FOR IN-DEPTH SQLITE DATA CARVING
MORE ADVANTAGES OF VNR SQLITE CARVER
• It automatically creates carving template which can retrieve allpossible data from the dump (unallocated and allocated space )
• It works on database files, partitions and the whole dump
• It recognizes various data types and use the appropriate convertersto transform data to a readable form
• It removes all duplicates to clean your results from unnecessarygarbage
• It is a universal tool – you shouldn’t wait while developers of anyother tool will release the update to add new apps or make changesaccording to the new app version
STEP 1. SELECT SOURCE AND PRESS BUTTON
STEP 2. SELECT THE MODE TO GET CARVING TEMPLATE
STEP 3. SELECT TABLE YOU ARE INTERESTED IN AND GET CARVING TEMPLATE
STEP 4. RUN SQLITE DATA CARVING PROCESS
STEP 5. GET DATA
Processing results
GroupingSortingFiltering data
VNR SQLITE CARVER TEMPLATE ELEMENTS
CARVER TEMPLATE ELEMENT FEATURES
Set size
Set range
Text encoding
Use REGEX
Date & Time converters
Reversed byteorder
Set count
✓
✓ ✓ ✓
✓
✓ ✓
✓ ✓ ✓ ✓
✓
INTEGER
TEXT
REAL
NUMERIC
DATE
GAP
BLOB
CARVER TEMPLETE ADJUSTMENT
• Set size
• Set range
• Use REGEX
Use GAP element
Сombinetemplates
into a universal template
Get more clean data
Run data carving again
Specify the range of values
Step 1 Step 2 Step 3
UNIVERSAL TEMPLATE
+ =
1st template 2nd template Universal template
VALUE RANGE ADJUSTMENT
Specify field types
Set range
MORE CLEAN DATA
1st template results 2nd template resultsUniversal template results
Now let’s check how it works in real world
![Writing Python 2-3 compatible code · In [ ]: # Python 2 only bigint = 1L # Python 2 and 3 from builtins import int bigint = int(1) To test whether a value is an integer (of any kind):](https://static.documents.pub/doc/80x56/5fe309a1a032b02371486040/writing-python-2-3-compatible-code-in-python-2-only-bigint-1l-python.jpg)
