Porting NetBSDon

the open source LatticeMico32 CPU

Yann SionneauM-Labs

@ EHSM 2014

About me

• Yann Sionneau• Embedded software developer• Working at Sequans Communication• M-Labs contributor• @yannsionneau on twitter• Email: yann.sionneau@gmail.com

I’m going to talk about…

How to run NetBSD and EdgeBSD on theMilkymist One

Agenda

• I) The hardware part: the MMU–What is a MMU and how it works

• II) The software part–How to port NetBSD to a new CPU

Milkymist One?!

Milkymist One?!

Milkymist One?!

The Milkymist One uses an FPGA

What’s an FPGA??

• A chip

FPGA internals

Milkymist System-on-Chip

LatticeMico32 CPU

• 32 bits Harvard Architecture RISC• Big Endian• 6 stages• Fully bypassed• Optional configurable I/D caches– Direct mapped or– 2-way set associative

• Wishbone on-chip bus

LatticeMico32 , Good points

• Small• Portable (works with several FPGA vendors)• Fast (~100 MHz on Slowtanpartan 6)• Actually works• GCC/Binutils/GDB/Qemu/uCLinux/OpenWRT

support• OPEN SOURCE

LatticeMico32, Bad points

• No Memory Management Unit… yet!

LatticeMico32, Bad points

• No Memory Management Unit… yet! Done

Used in…

• Closed source commercial ASICs• Open source projects

• Can achieve 800 MHz in TSMC 90nm standard cell process

LatticeMico32 pipeline

What’s a pipeline?

• « In computing, a pipeline is a set of data processing elements connected in series, where the output of one element is the input of the next one. »

-- Pipeline (computing), Wikipedia

What’s a pipeline?

Data processing element 1

Data processing element 2

Data processing element 3

ININ INOUTOUT

OUT

What’s a pipeline?

$ cat .bash_history | grep 'cat' | wc -l 6

What’s a CPU pipeline?

Address

instruction Fetch

instruction Decode

instruction eXecute

Memory load/store

register Write Back

What’s a CPU pipeline?

A

F

D

X

M

W

Pipelined instruction executionInstr.

numberPipeline Stage

1 A

2

3

4

Clock cycle 1 2 3 4 5 6 7

Pipelined instruction executionInstr.

numberPipeline Stage

1 A F

2 A

3

4

Clock cycle 1 2 3 4 5 6 7

Pipelined instruction executionInstr.

numberPipeline Stage

1 A F D

2 A F

3 A

4

Clock cycle 1 2 3 4 5 6 7

Pipelined instruction executionInstr.

numberPipeline Stage

1 A F D X

2 A F D

3 A F

4 A

Clock cycle 1 2 3 4 5 6 7

Pipelined instruction executionInstr.

numberPipeline Stage

1 A F D X M

2 A F D X

3 A F D

4 A F

Clock cycle 1 2 3 4 5 6 7

Pipelined instruction executionInstr.

numberPipeline Stage

1 A F D X M W

2 A F D X M

3 A F D X

4 A F D

Clock cycle 1 2 3 4 5 6 7

Pipelined instruction executionInstr.

numberPipeline Stage

1 A F D X M W

2 A F D X M W

3 A F D X M

4 A F D X

Clock cycle 1 2 3 4 5 6 7

Address

instruction Fetch

instruction Decode

instruction eXecute

Memory load/store

register Write back

InstructionCache

DataCache

Main Memory

CPU Internal

Before

PHYSICAL

ADDRESS

PHYSICAL

ADDRESS

PA

PA

Address

instruction Fetch

instruction Decode

instruction eXecute

Memory load/store

register Write back

InstructionCache

DataCache

Main Memory

CPU Internal

Memory Management Unit (MMU)

Raising exception

MM

U lo

okup

Cache lookup

After

VIRTUAL ADDRESSES PHYSICAL ADDRESSES

What’s the MMU’s job?

• Translate « virtual addresses » into « physical addresses »

• Memory protection against unwanted execution of code or data write (e.g. software bug or security issue)– Memory right access management

Main Memory

Memory Management Unit (MMU)

CPU pipelineVA PA

VA : Virtual AddressPA : Physical Address

Main Memory

Memory Management Unit (MMU)

CPU pipelineVA PA

VA : Virtual AddressPA : Physical Address

How does the MMU know the VA->PA translation ?

Main Memory

Memory Management Unit (MMU)

CPU pipelineVA PA

VA : Virtual AddressPA : Physical Address

Page Table

Main Memory

Memory Management Unit (MMU)

CPU pipelineVA PA

VA : Virtual AddressPA : Physical Address

Page TableWhy « PAGE »?

Why « Page »?

• 0x00000004 -> 0x10000000• 0x00000005 -> 0x10000001• 0x00000006 -> 0x10000002Etc…

Why « Page »?

• 0x00000004 -> 0x10000000• 0x00000005 -> 0x10000001• 0x00000006 -> 0x10000002Etc…

This is WRONG!!!

Why « Page »?

• 0x00000*** -> 0x10000***• 0x00001*** -> 0x10001***• 0x00002*** -> 0x10002***Etc…

Main Memory

Memory Management Unit (MMU)

CPU pipelineVA PA

VA : Virtual AddressPA : Physical Address

Page Table

Main Memory

Memory Management Unit (MMU)

CPU pipelineVA PA

VA : Virtual AddressPA : Physical Address

Page Table

TLBTLB : Translation Lookaside Buffer

Main Memory

Memory Management Unit (MMU)

CPU pipelineVA PA

VA : Virtual AddressPA : Physical Address

Page Table

TLBOperating System

Updates the

Gets information from the

Updates the

Features?

• Page size–Only 4 kB

32 bits physical address : xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx

How many bits of an address indicate the offset within a given page?

Features?

• Page size–Only 4 kB

32 bits physical address : xxxxxxxx xxxxxxxx xxxx xxxx xxxxxxxx

Page number [31:12]

20 bits

Offset [11:0]

12 bits

Features?

• 2 TLB (Translation Lookaside Buffer)– ITLB–DTLB

• Each TLB contains 1024 entries–How many bits needed to index the TLB?

Features?

• 2 TLB (Translation Lookaside Buffer)– ITLB–DTLB

• Each TLB contains 1024 entries–How many bits needed to index the TLB?

10 bits!

Features?

• No hardware page-tree walker– i.e. TLB is software assisted

Memory Management Unit

(MMU)

Virtual address

Load or store?

Instruction or Data?

Physical address

Accessgranted/denied

Memory Management Unit

(MMU)

Virtual address

Load or store?

Instruction or Data?

Physical address

Accessgranted/denied

I don’t know!

Let’s have a look inside

Tag [10] Physical page number [20] Read-only [1] Valid [1]

0xABC 0xABC00 0 0

0x280 0xB0001 1 1

0x300 0x00001 0 1

The TLBVA = 0xA0001004

Tag [10] Physical page number [20] Read-only [1] Valid [1]

0xABC 0xABC00 0 0

0x280 0xB0001 1 1

0x300 0x00001 0 1

The TLBVA = 0xA0001 004

Page number

Offset in the page

Tag [10] Physical page number [20] Read-only [1] Valid [1]

0xABC 0xABC00 0 0

0x280 0xB0001 1 1

0x300 0x00001 0 1

The TLBVA = 0xA0001 004Page offset = 4

Tag [10] Physical page number [20] Read-only [1] Valid [1]

0xABC 0xABC00 0 0

0x280 0xB0001 1 1

0x300 0x00001 0 1

The TLBVA = 0xA0001 004Page offset = 4Virtual Page number = 0xA0001

Tag [10] Physical page number [20] Read-only [1] Valid [1]

0xABC 0xABC00 0 0

0x280 0xB0001 1 1

0x300 0x00001 0 1

The TLBVA = 0xA0001 004Page offset = 4Virtual Page number = 0xA0001

VPN = 0xA0001 1010 0000 0000 0000 0001

Tag [10] Physical page number [20] Read-only [1] Valid [1]

0xABC 0xABC00 0 0

0x280 0xB0001 1 1

0x300 0x00001 0 1

The TLBVA = 0xA0001 004Page offset = 4Virtual Page number = 0xA0001 TLB index = 1

VPN = 0xA0001 1010 0000 00 00 0000 0001

TLB index, used to select a TLB line

Tag [10] Physical page number [20] Read-only [1] Valid [1]

0xABC 0xABC00 0 0

0x280 0xB0001 1 1

0x300 0x00001 0 1

The TLBVA = 0xA0001 004Page offset = 4Virtual Page number = 0xA0001 TLB index = 1

VPN = 0xA0001 1010 0000 00 00 0000 0001

TLB index, used to select a TLB line

Tag [10] Physical page number [20] Read-only [1] Valid [1]

0xABC 0xABC00 0 0

0x280 0xB0001 1 1

0x300 0x00001 0 1

The TLBVA = 0xA0001 004Page offset = 4Virtual Page number = 0xA0001 TLB index = 1

VPN = 0xA0001 1010 0000 00 00 0000 0001

Tag = 0x280 1010 0000 00=

Tag [10] Physical page number [20] Read-only [1] Valid [1]

0xABC 0xABC00 0 0

0x280 0xB0001 1 1

0x300 0x00001 0 1

The TLBVA = 0xA0001 004Page offset = 4Virtual Page number = 0xA0001 TLB index = 1

VPN = 0xA0001 1010 0000 00 00 0000 0001

Tag = 0x280 1010 0000 00=

Physical page number = 0xB0001

Tag [10] Physical page number [20] Read-only [1] Valid [1]

0xABC 0xABC00 0 0

0x280 0xB0001 1 1

0x300 0x00001 0 1

The TLBVA = 0xA0001 004Page offset = 4Virtual Page number = 0xA0001 TLB index = 1

VPN = 0xA0001 1010 0000 00 00 0000 0001

Tag = 0x280 1010 0000 00=

Physical page number = 0xB0001Physical Address = 0xB0001004

Porting NetBSD

• 1°) NetBSD cross compilation toolchain– build.sh– Makefiles here and there– Arch-specific directories

Allows to do:$ ./build.sh -U -m lm32

tools

Porting NetBSD

• 2°) Support for built-ins in libkern– NetBSD kernel is• Not linked against libgcc• Linked against libkern

– Need to implement basic arithmetic functions emitted by gcc in object code

– Implementation in sys/lib/libkern/arch/lm32

Porting NetBSD

• 3°) Building my first kernel– Create sys/arch/lm32 and sys/arch/milkymist– Populate• sys/arch/<cpu|soc>/include• sys/arch/<cpu|soc>/conf

– Stub, stub, stub…

Allows to do:$ ./build.sh -m milkymist -U kernel=GENERIC

Porting NetBSD

• 4°) Write basic console driver for early prints

struct consdev milkymist_com_cons = { […] milkymist_com_cngetc, /* cn_getc: kernel getchar interface */ milkymist_com_cnputc, /* cn_putc: kernel putchar interface */ […]};

Porting NetBSD

• 5°) Implement exception handlers• 6°) Call milkymist_startup() C code– Initialize console driver • -> consinit() -> milkymist_uart_cnattach()• cn_tab = &milkymist_com_cons;

– Initialiaze virtual memory subsystem• Call MD pmap_bootstrap()

– Let the kernel boot• Call NetBSD MI main()

Porting NetBSD

• 7°) Implement pmap.9pmap -- machine-dependent portion of the virtual memory system– pmap_bootstrap()– pmap_init, pmap_create, pmap_destroy …– SW managed TLB? -> sys/uvm/pmap/

– used in (PowerPC Booke and LM32)

Porting NetBSD

• 8°) Implement copyin/copyout• 9°) Implement atomic operations– No atomic instruction RAS (Restartable Atomic

Sequence) CAS (Compare And Swap)– Other atomic ops built around this CAS

RAS CASint _atomic_cas_32(volatile uint32_t *val, uint32_t old, uint32_t new);_atomic_cas_32:_atomic_cas_ras_start:

lw r4, (r1+0) /* load *val into r4 */bne r4, r2, 1f /* compare r4 (*val) and old (r2) */ sw (r1+0), r3

_atomic_cas_ras_end: 1:

mv r1, r4 /* return (*val) */ret

Porting NetBSD

• 10°) Add support for interrupts– Write a function to register interrupt handlers

• 11°) Have a running system clock– Write cpu_initclocks()– Write clock irq handler• Call hardclock()

Other functions to write

• Switch context from one thread to another– cpu_switchto(9)

• Copy data and abort on page fault– kcopy(9)

• Save current context– setfault()

• Low level code to finish up fork() operation– cpu_lwp_fork(9)

Other functions to write

• Block interrupts to protect critical sections– spl(9)

• Init CPU and print copyright message– cpu_startup(9)

• Determine the root file system device– cpu_rootconf(9)

• Etc…

Porting NetBSD

• To boot user space– Create dummy ramdisk with /sbin/init– Build kernel with MFS– Insert ramdisk with mdsetimage– Boot it!

Porting NetBSD

DEMO

Thank you!

Sébastien Bourdeauducq, Michael Walle, Robert Swindells, Stefan Kristiansson, Lars-Peter Clausen, Pierre Pronchery, Radoslaw Kujawa, Youri Mouton, Matt Thomas, tech-kern@, M-Labs mailing list, and many more

Questions?

NetBSD/milkymist Memory Layout

Kernel spaceUser space

0 0xffffffff

0xc0000000

0xc8000000Ram window

User stack

Kernelstack

DDR SDRAM : 128 MB