Hank Childs, University of OregonApril 8th, 2015

CIS 330: _ _ _ _ ______ _ _____ / / / /___ (_) __ ____ _____ ____/ / / ____/ _/_/ ____/__ __ / / / / __ \/ / |/_/ / __ `/ __ \/ __ / / / _/_// / __/ /___/ /_ / /_/ / / / / /> < / /_/ / / / / /_/ / / /____/_/ / /__/_ __/_ __/ \____/_/ /_/_/_/|_| \__,_/_/ /_/\__,_/ \____/_/ \____//_/ /_/

Lecture 3:Build Systems, Memory

Announcements

• Matt’s OH: Mon 12-1, Tues 12-2• Hank’s OH: Weds 11-12, Fri 12:30-1:30• It sounds like Weds 11-12 is problematic for

many.– I am very constrained on Weds– Starting next week, I can offer an “on demand” OH

on Weds after class.

Outline

• Review: permissions• Review: project 1B• Review: basics of build• New: build• New: project 1C• New: memory allocation (maybe)

Outline

• Review: permissions• Review: project 1B• Review: basics of build• New: build• New: project 1C• New: memory allocation (maybe)

There are 9 file permission attributes

• Can user read?• Can user write?• Can user execute? • Can group read?• Can group write?• Can group execute?• Can other read?• Can other write?• Can other execute?

A bunch of bits … we could represent this with binary

User = “owner”Other = “not owner, not group”

Translating permissions to binary

Image from wikipedia

Which of these modes make sense? Which don’t?

We can have separate values (0-7) for user, group, and other

Unix command: chmod

• chmod: change file mode

• chmod 750 <filename>– User gets 7 (rwx)– Group gets 5 (rx)– Other gets 0 (no access)

Outline

• Review: permissions• Review: project 1B• Review: basics of build• New: build• New: memory allocation (maybe)

Unix scripts

• Scripts– Use an editor (vi/emacs/other) to create a file that

contains a bunch of Unix commands– Give the file execute permissions– Run it like you would any program!!

Project 1B

Project 1B

Outline

• Review: permissions• Review: project 1B• Review: basics of build• New: build• New: memory allocation (maybe)

Build: The Actors

• File types– Source code– Object code– Executable code

• Programs– Compiler– Linker

Source code(e.g., C code)

Compiler Object code Linker Executable code

Compilers, Object Code, and Linkers

• Compilers transform source code to object code– Confusing: most compilers also secretly have access

to linkers and apply the linker for you.• Object code: statements in machine code – not executable– intended to be part of a program

• Linker: turns object code into executable programs

GNU Compilers

• GNU compilers: open source– gcc: GNU compiler for C– g++: GNU compiler for C++

Our first gcc program

Invoke gcc compiler

Name of file to compile

Default name for output programs

Note: compiler is calling linker directly, object file is intermediate only and not stored to file system

Our first gcc program: named output

“-o” sets name of output

Output name is different

Output has execute permissions

Outline

• Review: permissions• Review: project 1B• Review: basics of build• New: build• New: project 1C• New: memory allocation (maybe)

Object Code Symbols

• Symbols associate names with variables and functions in object code.

• Necessary for:– debugging– large programs

Imagine a world without symbols…• I make object file (.o) that is part of an executable• It has a function called “foo”• You make an object file containing function called

“foo2” that calls “foo”• … and linker wants to make an executable with

these two object files• If there are no symbols, linker just has addresses

to work with

Symbols provide hints to linker & debugger that allow them function

Do an example with nm to show symbols

gcc flags: debug and optimization• “gcc –g”: debug symbols

– Debug symbols place information in the object files so that debuggers (gdb) can:• set breakpoints• provide context information when there is a crash

• “gcc –O2”: optimization– Add optimizations … never fails

• “gcc –O3”: provide more optimizations– Add optimizations … sometimes fails

• “gcc –O3 –g”– Debugging symbols slow down execution … and sometimes

compiler won’t do it anyways…

Large code development

Source codefile1.C

Compiler Object codefile1.o

Linker Executable code

Source codefile2.C

Compiler Object codefile2.o

Source codefile3.C

Compiler Object codefile3.o

Why could this be a good idea?

gcc flag “-c”: make object code, but don’t link

• gcc –c file1.c– makes object code file “file1.o”

Multi-file development: examplecat is a Unix commandthat prints the contentsof a file

$? is a shell construct thathas the return value of thelast executed program

Multi-file development: example

Multi-file development: example

Linker order matters for some linkers (not Macs).

Some linkers need the .o with “main” first and then extract

the symbols they need as they go.

Other linkers make multiple passes.

Libraries

• Library: collection of “implementations” (functions!) with a well defined interface

• Interface comes through “header” files.• In C, header files contain functions and

variables.– Accessed through “#include <file.h>”

Libraries• Why are libraries a good thing?• Answers:– separation

• I.e., divide and conquer– increases productivity

• I.e., simplicity• I.e., prevents tendrils between modules that shouldn’t exist

– encapsulation (hides details of the implementation)• “A little knowledge is a dangerous thing”…• Products

– I can sell you a library and don’t have to give you the source code.

Libraries• Why are libraries a bad thing?• Answers:– separation • I.e., makes connections between modules harder

– (were the library interfaces chosen correctly?)

– complexity• need to incorporate libraries into code compilation

Includes and Libraries

• gcc support for libraries– “-I”: path to headers for library– “-L”: path to library location– “-lname”: link in library libname

Library types

• Two types:– static and shared

• Static: all information is taken from library and put into final binary at link time.– library is never needed again

• Shared: at link time, library is checked for needed information.– library is loaded when program runs

More about shared and static later … for today, assume static

ar: archiver

• Makes a library– i.e., collects binary code in object files into a single

file (a library)

• Usage: ar libname.a file1.o file2.o

Making a static library

Note the ‘#’ is the comment character

(should have called this libmultiplier.a)

What’s in the file?

Typical library installations

• Convention– Header files are placed in “include” directory– Library files are placed in “lib” directory

• Many standard libraries are installed in /usr– /usr/include– /usr/lib

• Compilers automatically look in /usr/include and /usr/lib (and other places)

Installing the library

(fixing my mistake)

“mv”: unix command for renaming a file

Example: compiling with a library

• gcc support for libraries– “-I”: path to headers for library– “-L”: path to library location– “-lname”: link in library libname

Makefiles

• There is a Unix command called “make”• make takes an input file called a “Makefile”• A Makefile allows you to specify rules– “if timestamp of A, B, or C is newer than D, then

carry out this action” (to make a new version of D)• make’s functionality is broader than just

compiling things, but it is mostly used for computation

Basic idea: all details for compilation are captured in a configuration file … you just invoke “make” from a shell

Makefiles

• Reasons Makefiles are great:– Difficult to type all the compilation commands at a

prompt– Typical develop cycle requires frequent

compilation– When sharing code, an expert developer can

encapsulate the details of the compilation, and a new developer doesn’t need to know the details … just “make”

Makefile syntax

• Makefiles are set up as a series of rules• Rules have the format:

target: dependencies[tab] system command

Target: what to build. (a name you give to make.)Dependencies: what it depends on (files in the filesystem or other rules)System command: gcc …

Makefile example: multiplier lib

Fancy makefile example: multiplier lib

Configuration management tools

• Problem:– Unix platforms vary• Where is libX installed?• Is OpenGL supported?

• Idea:– Write problem that answers these questions, then

adapts build system• Example: put “-L/path/to/libX –lX” in the link line• Other fixes as well

Two popular configuration management tools

• Autoconf– Unix-based– Game plan:

• You write scripts to test availability on system• Generates Makefiles based on results

• Cmake– Unix and Windows– Game plan:

• You write .cmake files that test for package locations• Generates Makefiles based on results

CMake has been gaining momentum in recent years, because it is one of the best solutions for cross-platform support.

Outline

• Review: permissions• Review: project 1B• Review: basics of build• New: build• New: project 1C• New: memory allocation (maybe)

Unix command: tar

• Anyone know what tar stands for?tar = tape archiver

IBM tape library

Unix command: tar

• Problem: you have many files and you want to…– move them to another machine– give a copy to a friend– etc.

• Tar: take many files and make one file– Originally so one file can be written to tape drive

• Serves same purpose as “.zip” files.

Unix command: tar

• tar cvf 330.tar file1 file2 file3• scp 330.tar @ix:~• ssh ix• tar xvf 330.tar• ls

file1 file2 file

Project 1C

Project 1C

Outline

• Review: permissions• Review: project 1B• Review: basics of build• New: build• New: project 1C• New: memory allocation (yes!)

Memory Segments

• Von Neumann architecture: one memory space, for both instructions and data

• so break memory into “segments”– … creates boundaries to prevent confusion

• 4 segments:– Code segment– Data segment– Stack segment– Heap segment

Code Segment

• Contains assembly code instructions• Also called text segment• This segment is modify-able, but that’s a bad

idea– “Self-modifying code”• Typically ends in a bad state very quickly.

Data Segment

• Contains data not associated with heap or stack– global variables– statics (to be discussed later)– character strings you’ve compiled in

char *str = “hello world\n”

Stack: data structure for collection

• A stack contains things• It has only two methods: push and pop– Push puts something onto the stack– Pop returns the most recently pushed item (and

removes that item from the stack)• LIFO: last in, first out

Imagine a stack of trays.You can place on top (push).

Or take one off the top (pop).

Stack

• Stack: memory set aside as scratch space for program execution

• When a function has local variables, it uses this memory.– When you exit the function, the memory is lost

Stack

• The stack grows as you enter functions, and shrinks as you exit functions.– This can be done on a per variable basis, but the

compiler typically does a grouping.• Some exceptions (discussed later)

• Don’t have to manage memory: allocated and freed automatically

Heap

• Heap (data structure): tree-based data structure

• Heap (memory): area of computer memory that requires explicit management (malloc, free).

• Memory from the heap is accessible any time, by any function.– Contrasts with the stack

Memory Segments

Source: http://www.cs.uwm.edu/classes/cs315/Bacon/

Stack vs Heap: Pros and ConsStack Heap

Allocation/Deallocation

Automatic Explicit

Allocation / Deallocation

How stack memory is allocated into Stack Memory Segment

Code

Data

Heap

Stack

Free

How stack memory is allocated into Stack Memory Segment

Code

Data

Heap

Stack

Free

stack_varCstack_varD

How stack memory is allocated into Stack Memory Segment

Code

Data

Heap

Stack

Free

stack_varCstack_varDstack_varAstack_varB

How stack memory is allocated into Stack Memory Segment

Code

Data

Heap

Stack

Free

stack_varCstack_varD

How stack memory is allocated into Stack Memory Segment

Code

Data

Heap

Stack

Free

stack_varCstack_varD

How stack memory is allocated into Stack Memory Segment

Code

Data

Heap

Stack

Free

stack_varCstack_varD<info for how to get back to main>A (= 3)<Location for RV>

How stack memory is allocated into Stack Memory Segment

Code

Data

Heap

Stack

Free

stack_varCstack_varD<info for how to get back to main>A (= 3)<Location for RV>stack_varA

How stack memory is allocated into Stack Memory Segment

Code

Data

Heap

Stack

Free

stack_varCstack_varD<info for how to get back to main>A (= 3)<Location for RV>stack_varA

Return copies into location specified by calling function

How stack memory is allocated into Stack Memory Segment

Code

Data

Heap

Stack

Free

stack_varC = 6stack_varD = 3

This code is very problematic … why?

foo and bar are returning addresses that are on the stack … they could easily

be overwritten(and bar’s stack_varD

overwrites foo’s stack_varC in this

program)

Nested Scope

Code

Data

Heap

Stack

Free

stack_varA

Nested Scope

Code

Data

Heap

Stack

Free

stack_varAstack_varB

Nested Scope

Code

Data

Heap

Stack

Free

stack_varA

You can create new scope within a function by adding

‘{‘ and ‘}’.

Stack vs Heap: Pros and ConsStack Heap

Allocation/Deallocation

Automatic Explicit

Access Fast Slower

Memory pages associated with stack are almost always immediately

available.

Memory pages associated with heap may be located

anywhere ... may be caching effects

Stack vs Heap: Pros and ConsStack Heap

Allocation/Deallocation

Automatic Explicit

Access Fast Slower

Variable scope Limited Unlimited

Variable scope: stack and heap

bar returned memory from heap

The calling function – i.e., the function that

calls bar – must understand this and take responsibility for calling

free.

If it doesn’t, then this is a “memory leak”.

Memory leaksCode

Data

Heap

Stack

Free

stack_varA

It is OK that we are using the heap … that’s what it is there for

The problem is that we lost the references to the 49 allocations on heap

The heap’s memory manager will not be able to re-claim them … we have effectively limited the

memory available to the program.

Running out of memory (stack)Code

Data

Heap

Stack

Freestack overflow: when the stack runs into the heap.There is no protection for stack overflows.

(Checking for it would require coordination with the heap’s memory manager on every function calls.)

Running out of memory (heap)Code

Data

Heap

Stack

FreeIf the heap memory manager doesn’t have room to make an allocation, then malloc returns NULL …. a more graceful error

scenario.

Allocation too big …

not enough free

memory

Stack vs Heap: Pros and ConsStack Heap

Allocation/Deallocation

Automatic Explicit

Access Fast Slower

Variable scope Limited Unlimited

Fragmentation No Yes

Memory Fragmentation

• Memory fragmentation: the memory allocated on the heap is spread out of the memory space, rather than being concentrated in a certain address space.

Memory FragmentationCode

Data

Heap

Stack

Free

Negative aspects of fragmentation?(1) can’t make big allocations

(2) losing cache coherency

Fragmentation and Big AllocationsCode

Data

Heap

Stack

Free

Even if there is lots of memory available, the memory manager can only accept your request if there is a

big enough contiguous chunk.

Stack vs Heap: Pros and ConsStack Heap

Allocation/Deallocation

Automatic Explicit

Access Fast Slower

Variable scope Limited Unlimited

Fragmentation No Yes

Outline

• Announcements/Review• Project 2B• Project 2C• Memory Overview• Memory Errors• Finish Unix Boot Camp

Memory Errors

• Array bounds read

• Array bounds write

Memory Errors

• Free memory read / free memory write

When does this happen in real-world scenarios?

Memory Errors

• Freeing unallocated memory

When does this happen in real-world scenarios?

Vocabulary: “dangling pointer”: pointer that points to memory that has already been freed.

Memory Errors

• Freeing non-heap memory

When does this happen in real-world scenarios?

Memory Errors

• NULL pointer read / write

• NULL is never a valid location to read from or write to, and accessing them results in a “segmentation fault”– …. remember those memory segments?

When does this happen in real-world scenarios?

Memory Errors

• Unitialized memory read

When does this happen in real-world scenarios?