1

How does the computer generate observations from

various distributions specified after input analysis?

There are two main components to the generation of

observations from probability distributions.

1. Random number generation.

2. Random variate generation.

2

Random number generation – The generation of

U(0,1) random variates (observations from Uniform

(0,1) distribution).

◦ This serves as the foundation for the generation of

observations from other distributions, which is called

random variate generation.

Random Number Generator is the term used to

describe the procedure and parameters used to

generate the U(0,1) observations.

3

Since the “stream” of random numbers generated is

reproducible, random number generation procedures

are also referred to as pseudo random numbergenerators.

The stream or sequence of numbers produced by a

generator should pass statistical tests for randomness.

◦ An outside observer should not be able to tell the difference

(statistically) between a stream of pseudo random numbers

and an actual random number stream.

4

A pseudorandom process appears random, but isn‟t

Pseudorandom sequences exhibit statistical randomness◦ but generated by a deterministic process

Pseudorandom sequences are easier to produce than a

genuine random sequences

Pseudorandom sequences can reproduce exactly the

same numbers

◦ useful for testing and fixing software.

5

Random number generators typically compute the next

number in the sequence from the previous number

The first number in a sequence is called the seed

◦ to get a new sequence, supply a new seed

(current machine time is useful)

◦ to repeat a sequence, repeat the seed

6

7

Desirable Attributes:

◦ Uniformity

◦ Independence

◦ Efficiency

◦ Replicability

◦ Long Cycle Length

8

Each random number Rt is an independent sample

drawn from a continuous uniform distribution

between 0 and 1

1 , 0 x 1pdf: f(x) =

0 , otherwise

9

12/1

4/13/1)2/1(]3/[

)]([)(

2/1]2/[)(

21

0

3

21

0

2

1

0

21

0

x

REdxxRV

xdxxRE

xf(

x)

0

1

PDF:

One early method – the midsquare method (von

Neumann and Metropolis 1940)

◦ Start with a four digit positive integer Z0.

◦ Square Z0 to get an integer with up to eight digits (append

zeros if less than eight).

◦ Take the middle four digits as the next four digit integer Z1.

◦ Place a decimal point to the left of Z1 to form the first

“U(0,1)” observation.

◦ Repeat

10

11

MidSquareExample:

X0 = 7182 (seed)= 51581124

==> R1 = 0.5811= (5811) 2 = 33767721

==> R2 = 0.7677etc.

2

0X

2

0X

12

Note: Cannot choose a seed that guarantees that the sequence will not degenerate and will have a long period. Also, zeros, once they appear, are carried in subsequent numbers.

Ex1: X0 = 5197 (seed) = 27008809

==> R1 = 0.0088 = 00007744

==> R2 = 0.0077

Ex2: X0 = 4500 (seed) = 20250000

==> R1 = 0.2500 = 06250000

==> R2 = 0.2500

2

0X2

1X

2

0X2

1X

The prior method does not work well.

◦ Degenerates to zero.

What are good methods?

◦ Linear Congruential Generators (LCGs).

◦ Composite generators.

◦ Tausworthe generators.

13

Linear Congruential Generators (LCGs).

A LCG generates a sequence of integers Z1, Z2, Z3,… using the following recursive formula,

mod m is short for modulo m or the remainder when

divided by m.

mcaZZ ii mod)( 1

14

Since the mod m operation is used, all Zi‟s will be

between 0 and m-1.

To get the “U(0,1)” random observations each Zi

generated is divided by m.

So are the Ui‟s really U(0,1) random observations ?

,, 22

11

m

ZU

m

ZU

15

Let m=63, a=22, c=4 and Z0 =19.

◦ Generate the first five “U(0,1)” observations.

16

17

i 22*Zi-1+ 4 Zi Ui i 22*Zi-1+ 4 Zi Ui i 22*Zi-1+ 4 Zi Ui i 22*Zi-1+ 4 Zi Ui

0 19 16 356 41 0.6508 32 840 21 0.3333 48 400 22 0.3492

1 422 44 0.6984 17 906 24 0.3810 33 466 25 0.3968 49 488 47 0.7460

2 972 27 0.4286 18 532 28 0.4444 34 554 50 0.7937 50 1038 30 0.4762

3 598 31 0.4921 19 620 53 0.8413 35 1104 33 0.5238 51 664 34 0.5397

4 686 56 0.8889 20 1170 36 0.5714 36 730 37 0.5873 52 752 59 0.9365

5 1236 39 0.6190 21 796 40 0.6349 37 818 62 0.9841 53 1302 42 0.6667

6 862 43 0.6825 22 884 2 0.0317 38 1368 45 0.7143 54 928 46 0.7302

7 950 5 0.0794 23 48 48 0.7619 39 994 49 0.7778 55 1016 8 0.1270

8 114 51 0.8095 24 1060 52 0.8254 40 1082 11 0.1746 56 180 54 0.8571

9 1126 55 0.8730 25 1148 14 0.2222 41 246 57 0.9048 57 1192 58 0.9206

10 1214 17 0.2698 26 312 60 0.9524 42 1258 61 0.9683 58 1280 20 0.3175

11 378 0 0.0000 27 1324 1 0.0159 43 1346 23 0.3651 59 444 3 0.0476

12 4 4 0.0635 28 26 26 0.4127 44 510 6 0.0952 60 70 7 0.1111

13 92 29 0.4603 29 576 9 0.1429 45 136 10 0.1587 61 158 32 0.5079

14 642 12 0.1905 30 202 13 0.2063 46 224 35 0.5556 62 708 15 0.2381

15 268 16 0.2540 31 290 38 0.6032 47 774 18 0.2857 63 334 19 0.3016

What will happen after the 63rd number is generated?

m, a, and c are the parameters of the random number

generator.

◦ There can be an infinite number of different implementations

of a LCG.

The values used for m, a, and c determine whether the

generator is good or bad.

18

The example LCG demonstrates cycling in the prior

table.

◦ Since m=63, it can generate at most 63 numbers before it

repeats the same sequence.

This small random number generator has full period

since it generates all possible (m=63) numbers before

cycling.

◦ A long period (full if possible) is desirable since more

observations can be generated before cycling.

◦ No “gaps”.

19

The example generator has full period but bad

statistical properties (next slide).

A good random number generator will have values for

m, a, and c such that full or close to full period is

obtained, as well as good statistical properties.

◦ Crystal Ball

m = 231 – 1

a = 62089911

c = 0

Period = 231 – 2

20

Theorem (Hull and Dobell 1962)

◦ The LCG Zi = (aZi-1 + c) mod m has full period if and

only if the following three conditions hold.

1. The only positive integer that exactly divides both m and c is 1.

2. If q is a prime number that divides m, then q divides a-1.

3. If 4 divides m, then 4 divides a-1.

The parameters of the LCG dictate the period length

of the LCG as well as other properties of the numbers

generated.

21

A generator that has the maximum possible period is called a full-period generator.

Lower autocorrelations between successive numbers are preferable.

Both generators have the same full period, but the first one has a correlation of 0.25 between xn-1 and xn, whereas the second one has a negligible correlation of less than 2-18.

22

Types of LCGs◦ When c = 0, the LCG is called a multiplicative generator.

◦ When c ≠ 0, the LCG is called a mixed generator.

Most LCGs implemented are multiplicative◦ Can‟t have full period.

How is m selected.◦ A large period is desired → m=231 (based on a 32 bit word

size).

With m=231 it has been proven that the period can be at

most 229 (25% of the values are cycled and gaps may be

present).

23

Multiplicative LCG: c=0

Two types:

24

mxax ii mod1

k

k

m

m

2

2

25

Example:

Using the multiplicative congruential method, findthe period of the generator for a = 13, m = 26, andX0 = 1, 2, 3, and 4. The solution is given in nextslide. When the seed is 1 and 3, the sequence hasperiod 16. However, a period of length eight isachieved when the seed is 2 and a period of lengthfour occurs when the seed is 4.

26

Period Determination Using Various seeds

i Xi Xi Xi Xi

0 1 2 3 41 13 26 39 522 41 18 59 363 21 42 63 204 17 34 51 45 29 58 236 57 50 437 37 10 478 33 2 359 45 7

10 9 2711 53 3112 49 1913 61 5514 25 1115 5 1516 1 3

Maximum possible period 2k-2

Period achieved if multiplier a is of the form 8i± 3,

and the initial seed is an odd integer

One-fourth the maximum possible may not be too

small

Low order bits of random numbers obtained using

multiplicative LCG's with m=2k have a cyclic pattern.

27

28

29

When the modulus m is a prime number and a >1, the maximum period is m-1, no matter whether c=0 or not.

The maximum period m-1 is obtained if and only if „a‟ is a primitive element modulo m.

If m is prime then „a‟ is a primitive element modulo m (or primitive root of m) if and only if an mod m ≠ 1 for n=1, 2, 3, …,m-2.

Recommended. (Prime moduli are best in terms of sequence randomness.)

30

Example:

Starting with a seed of x0=1:

1, 3, 9, 27, 19, 26, 16, 17, 20, 29, 25, 13, 8, 24, 10, 30,

28, 22, 4, 12, 5, 15, 14, 11, 2, 6, 18, 23, 7, 21, 1, …

The period is 30

⇒ 3 is a primitive root of 31

With a multiplier of a = 5: 1, 5, 25, 1,…

The period is only 3 ⇒ 5 is not a primitive root of 31

Primitive roots of 31= 3, …????????.

31

31mod3 1 ii xx

LCGs are a special case of the form Zi = g(Zi-1, Zi-2, ...)

(mod m), Ui = Zi /m, for some function g

Examples:

g(Zi-1) = aZi-1 + c LCG

g(Zi-1, Zi-2, ..., Zi-q) = a1Zi-1 + a2Zi-2 + ... + aqZi-q

multiple recursive generator

g(Zi-1) = a'Z 2i-1 + aZi-1 + c quadratic CG

g(Zi-1, Zi-2) = Zi-1 + Zi-2 Fibonacci (bad)

32

Composite Generators

Combine two (or more) individual generators in some way.

Differencing LCGs

◦ Z1i and Z2i from LCGs with different moduli

◦ Let Zi = (Z1i – Z2i ) (mod m); Ui = Zi / m

◦ Very good statistical properties

◦ Very portable (micros, different languages)

Wichmann/Hill

◦ Use three LCGs to get U1i, U2i, and U3i sequences

◦ Let Ui = fractional part of U1i + U2i + U3i

◦ Long period, good statistics, portability

33

Originated in cryptography

Can achieve very long periods

Theoretical appeal: for properly chosen parameters, can prove that over a cycle,◦ mean 1/2 (as for true U(0,1))

◦ Variance 1/12 (as for true U(0,1))

◦ Autocorrelation 0 (as for true IID sequence)

Define a sequence of binary digits B1,B2, . . ., by

where cj = 0 or 1.

34

2mod1

q

j

jiji bcb

Looks a bit like a generalization of LCG‟s.

Let D = delay operator such that Db(n)=b(n+1)

or

Since in mod 2 arithmetic subtraction is equivalent to

addition, the preceding equation is equivalent to

35

2mod)()()()( 0

2

2

1

1 qibcqibDcqibDcqibD q

q

q

q

q

2mod0

2

2

1

1 cDcDcD q

q

q

q

q

2mod00

2

2

1

1

cDcDcD q

q

q

q

q

2mod00

2

2

1

1

cDcDcD q

q

q

q

q

The polynomial on the left-hand side of this equation iscalled a characteristic polynomial and is traditionally

written using x in place of D

The period of a Tausworthe generator depends upon thecharacteristic polynomial. In particular, the period is thesmallest positive integer n for which xn - 1 is divisible bythe characteristic polynomial. The maximum possibleperiod with a polynomial of order q is 2q - 1. Thepolynomials that give this period are called primitivepolynomials.

36

0

2

2

1

1 cxcxcx q

q

q

q

q

Example: Consider the following polynomial:

x7 + x3 + 1Using the D operator in place of x, we get

or

or using the XOR operator

or

Substituting n-7 for n, we get

37

2mod0)()()( 37 nbnbDnbD

2mod037 nnn bbb

,2,1,0037 nbbb nnn

,2,1,037 nbbb nnn

,9,8,774 nbbb nnn

Starting with b0 = b1 = ... = b6 = 1, we get the following bit

sequence:

38

110

011

011

011

011

4711

3610

259

148

037

bbb

bbb

bbb

bbb

bbb

The complete sequence is:

1111111 0000111 0111100 1011001 0010000 0010001

0011000 1011101 0110110 0000110 0110101 0011100

1111011 0100001 0101011 1110100 1010001 1011100

0111111 1000011 1000000.

Period = 127 or 27-1 bits

⇒ The polynomial x7+x3+1 is a primitive polynomial.

39

A Tausworthe sequence can be easily generated inhardware using Linear-Feedback Shift Registers(LFSRs).

For example, the polynomial x5 + x3 + 1 results in thegenerator bn = bn-2 bn-5. This can be implementedusing the LFSR shown in the Figure presented nextslide.

The circuit consists of six registers, each holding onebit. On every clock cycle, each register‟s content isshifted out, and the new content is determined by theinput to the register.

40

Linear Feedback Shift Register:

x5+x3+1 ⇒ bn= bn-2 ⊕ bn-5

This can be easily implemented using shift registers:

41

Generating U(0,1):

Divide the sequence into successive groups of s bits and

use the first l bits of each group as a binary fraction:

xn = 0.bsnbsn+1bsn+2bsn+3 ...bsn+l-1

Here, s is a constant greater than or equal to l and is

relatively prime to 2q-1.

s ≥ l ⇒ xn and xj for n ≠ j have no bits in common.

Relative prime-ness guarantees a full period 2q-1 for xn.

42

Example: bn = bn-4 ⊕ bn-7

The period 27-1=127

l=8, s=8:

x0 = 0.111111102 = 0.9921910

x1 = 0.000111012 = 0.1132810

x2 = 0.111001012 = 0.8945310

x3 = 0.100100102 = 0.2968810

x4 = 0.000001002 = 0.3632810

x5 = 0.010011002 = 0.4218810

….

43

List of Primitive Trinomials

x2 + x + 1 x3 + x + 1 x4 + x + 1 x5 + x2 + 1

x6 + x + 1 x7 + x + 1 x7 + x3 + 1 x9 + x4 + 1

x10 + x3 + 1 x11 + x2 + 1 x15 + x + 1 x15 + x4 + 1

x15 + x7 + 1 x17 + x3 + 1 x17 + x5 + 1 x17 + x6 + 1

x18 + x7 + 1 x20 + x3 + 1 x21 + x2 + 1 x22 + x + 1

x23 + x5 + 1 x23 + x9 + 1 x25 + x3 + 1 x25 + x7 + 1

x28 + x3 + 1 x28 + x9 + 1 x28 + x13 + 1 x29 + x2 + 1

x31 + x3 + 1 x31 + x6 + 1 x31 + x7 + 1 x31 + x13 +1

If xq + xr + 1 is listed, then xq + xq-r +1 is also primitive.

44

Homework:

Generate random numbers using the primitive polynomial x5+x2+1. (use l=4)

Generate the same sequence using LFSR.

45