1
Real time spectrogram
In the Android tablet
COMP4560 INDIVIDUAL PROJRCT
Yue Wang
Supervisor:
Dr Eric McCreath
A thesis submitted in partial fulfilment of the degree of
Bachelor of Information Technology at The Research School
of Computer Science Australian National University
Oct 2017
2
Abstract
Real time spectrograms give users an understanding of the frequency components of audio
and how they change overtime. This is useful for acoustic engineers. Frequency domain is
represents the signal by giving the amount of different frequency components, and the
algorithm to calculate the frequency result of discrete signal is named Discrete Fourier
Transform (DFT). To reduce the amount of calculation, scientist proposed a more efficient
algorithm which named Fast Fourier Transform (FFT). Even with this fast algorithm
implementing a real time spectrogram is computational demanding.
In this project, I developed an Android applications that produces a real time spectrogram. I
focused on looking at performance issues relating mobile application development.
The report explain the design process of my Android application include code explanation,
test results and issues in development.
Keywords: Time domain, Frequency domain, DFT, FFT, Android development
3
Acknowledgements
I would like to convey my highest respect and deepest gratitude to everyone who helped me
to complete this individual project and report.
Above all, I would like to express my sincere thanks to Dr Eric Mccreath, my supervisor for
his clear instructions and patient guidance throughout my entire project period over the
past several months, and also for my friends for their offering help on my report and giving
advice to solve issues.
Last but not least, sincere thanks to Prof Weifa Liang and Prof Peter Strazdins who were the
course conveners of COMP4560. Thank you for providing us with the tutorial room and
teaching us about presentations and report writing.
4
Content
Abstract ..................................................................................................................................... 2
Acknowledgements ................................................................................................................... 3
Chapter 1 Introduction .............................................................................................................. 6
1.1 Project learning objectives ......................................................................................... 6
1.2 Project value .................................................................................................................... 6
1.3 Contributions ................................................................................................................... 6
Chapter 2 Background ............................................................................................................... 7
2.1 Summary of Android ....................................................................................................... 7
2.2 Android Market Analysis ................................................................................................. 7
2.3 Android’s Features .......................................................................................................... 8
2.4 Time Domain and Frequency Domain ............................................................................. 9
2.5 Discrete Fourier Transform (DFT) .................................................................................... 9
An example of a real discrete Fourier transform ............................................................ 10
2.6 Fast Fourier Transform (FFT) ......................................................................................... 10
Cooley-Tukey algorithm .................................................................................................. 11
2.7 Canvas ............................................................................................................................ 11
2.8 OpenGL ES ..................................................................................................................... 12
Chapter 3 System analysis & environment construction ........................................................ 14
3.1 Develop objectives ........................................................................................................ 14
3.2 Feasibility Analysis ......................................................................................................... 14
3.3 The choice of Android version and other tools ............................................................. 14
3.3.1 Android Studio ........................................................................................................ 15
3.3.2 Genymotion ............................................................................................................ 15
Chapter 4 Application Implementation ................................................................................... 16
4.1 System Design ............................................................................................................... 16
4.2 Application Implementation steps ................................................................................ 16
4.3 Code implement ............................................................................................................ 17
4.3.1 Layout ..................................................................................................................... 17
4.3.2 Record function ...................................................................................................... 17
4.3.3 The Core Algorithm ................................................................................................ 18
5
4.3.4 Draw parts. ............................................................................................................. 19
4.3.5 Tools class ............................................................................................................... 19
4.4 Process analysis ............................................................................................................. 19
Chapter 5 Test result and issues .............................................................................................. 20
5.1 Test Case Design ............................................................................................................ 20
5.2 Test Case detail & result ................................................................................................ 20
5.3 issues and analysis and some solutions ........................................................................ 23
Chapter 6 Future work ............................................................................................................ 25
Conclusion ............................................................................................................................... 26
Appendix 1 Study Contract ...................................................................................................... 27
Appendix 2 Detail Code ........................................................................................................... 29
Appendix 3 System Test Result ............................................................................................... 41
Bibliography ............................................................................................................................. 42
6
Chapter 1 Introduction
In this chapter I introduce the whole view of this project and explain why choose Android OS
as the develop platform
1.1 Project learning objectives
This project involved research on FFT and DFT, gaining a good understanding of the FFT
evaluation. Design a Real Time Spectrogram application on Android Tablet with Java
language based on Android platform. With a focus on looking at performance issues
relating mobile application development. More generally the project would strengthen the
programming and problem solving abilities along with research skill associated with
exploring approaches and ideas and then implementing, testing and evaluating these
approaches and ideas.
1.2 Project value
Real Time Spectrogram Belonging to audio processing field. The purpose is to carry out the
Fourier transform of the audio data acquired in real time by the microphone to obtaining
energy intensity value at different frequencies. However, there is few existing Spectrogram
applications. Many applications give just frequency plot. Based on these points, we have
developed a real-time sound conversation application on Android tablet - "Real Time
Spectrogram".
1.3 Contributions
In the project, to get the spectrogram application going, this involved: sampling audio data
from files, sampling audio data from the mic, implementing FFT algorithm using Cooley-
Tukey algorithm, visualizing the spectrogram. This is able to run in real time both on the
emulator and tablet device when audio was sampled at 44100Hz with a window size of 1024
samples. And each window shifted by 1024 samples. The performance of this application
was evaluated. Also the application was extended to use the GPU for the visualization, this
was done by OpenGL. A comparison was made between the GPU version and the CPU
version and I found on the tablet device the GPU version work marginally fast than the CPU
version.
7
Chapter 2 Background
In the first phase of the project, a lot of time is spent on learning about concepts of Time
domain, Frequency domain, DFT, FFT, Canvas and OpenGL. In this chapter, I will show my
outcome of these research.
2.1 Summary of Android
The original means of Android is refers to "robot". It is also the name of the open source
mobile operating system based on the Linux platform announced by Google on November 5,
2007, which consists of the operating system, middleware, and user interface and
application software. (Kaur & Sharma, 2014)
The openness of Android platform makes it a huge advantage in the competition, open
platform allows any mobile terminal manufacturers to join the Android Alliance. Significant
openness allow more developers to showcase their creativity, with the growing popularity of
users and applications, android platform will be more competitive, which also induce Tablet
PC developers choose the android platform as operating system for their product.
The Android system is developed by Google and the open mobile Alliance which contained
Motorola, Qualcomm, HTC, China Mobile and T-Mobile which are leaders in unlimited
applications and technology domain. To establish an open, standardized mobile platform,
Architecture of the Android OS is designed based on Linux OS so that communication at end
user and application level will be quite easy. Thus, designing an application on Android is
easier than designing an applications on IOS.
2.2 Android Market Analysis
According to KANTAR World Panel report, the android OS market share is still maintained a
leading position in Global Market, which more than 50%, and in some countries like China,
Argentina, Brazil, Mexico, Germany, France, Spain and Italy the market share reach over
80%. As the Figure 1.1 shows that in Australia, Android system market share is still reached
65.5% (Kantar Worldpanel, 2017)
Based on the above data we found that software development based on Android system has
great prospects, develop an application on the Android system can get more potential
customers compared to other platforms.
8
Figure 1.1 (Kantar Worldpanel, 2017)
2.3 Android’s Features
Four main features make Android competitive in the mobile OS market system, the four
features of Android OS is consist of:
• Openness: source code development can be modified without authorization.
• Freedom: get rid of the shackles of operators, can freely contact the network.
• Compatibility: Developers do not need to consider the impact of hardware
• Expandability: allows manufacturers to specify the characteristics of the auxiliary
functions and services.
9
2.4 Time Domain and Frequency Domain
The time domain signal refers to the signal amplitude at the sampling time. Frequency
domain is represent the same signal by giving the amount of different frequency
components, Figure 2.1 shows single frequency component which spread out in the time
domain.
Figure 2.1 (Kantar Worldpanel, 2017; Help, 2012)
2.5 Discrete Fourier Transform (DFT)
Discrete Fourier transform (DFT) is a form in which the continuous Fourier transform is
discrete in the time domain and the frequency domain, and the sampling of the time domain
signal is transformed into the frequency domain in the discrete time Fourier transform
(DTFT). In the form, the sequences at both ends of the transform (both in the time domain
and in the frequency domain) are finite, and in fact these two sets of sequences should be
considered as the main value sequence of the discrete periodic signal. Even for a finite
length of the discrete signal DFT, it should be seen as a cycle of extension into a periodic
signal and then transform. In practice, fast Fourier transform is often used to efficiently
calculate DFT. (Weisstein, 2017)
In order to use the computer for Fourier transform in the fields of scientific computing and
digital signal processing, the function x𝑛 must be defined in discrete rather than continuous
domains, and the finite or periodic conditions must be satisfied. In this case, using the
discrete Fourier transform (DFT), the function x𝑛 is expressed as the following sum form
𝑋𝑛 = ∑ 𝑋𝑘𝑒𝑖2𝜋𝑁
𝑘𝑛
𝑁−1
𝑘=0
n = 0, … , N − 1
Where x𝑘 is the Fourier amplitude. The computational complexity calculated using this
formula is O(𝑛2), and the Fast Fourier Transform (FFT) can improve the complexity to O(n ∗
log 𝑛).
10
An example of a real discrete Fourier transform
Figure 2.2 (Steven W. Smith, 1997)
In Figure 2.2, the length of original signal is 16, So that the signal can be decomposed by nine
cosine waves and nine sine waves (a length of N signal can be decomposed into N
2+ 1 sine
and cosine signals, why? Combined with the following 18 positive cosine map, Combined
with the following 18 positive cosine map, we found it limited by the accuracy of the
computer processing, a length of the signal N, up to only N
2+ 1 different frequencies, more
than the frequency of the computer can handle more than the accuracy range), as shown in
Figure 2.3 :
Figure 2.3 (Steven W. Smith, 1997)
2.6 Fast Fourier Transform (FFT)
In the study of DFT, it is found that DFT requires 𝑛2 times complex multiplication and n ∗
(n − 1) times complex addition. Therefore, when N is a large number, the amount of
calculation will become very huge. To reduce the amount of computation, numerous studies
have been done on DFT and proposed many fast algorithm on how to calculate DFT more
efficient, in all these algorithms, Cooley-Tukey algorithm is called FFT, which is also called as
Butterfly algorithm. (Steven W. Smith, 1997)
11
Cooley-Tukey algorithm
In simple terms, the core idea of a butterfly algorithm is use the dichotomy to shorten the
length original signal until the length of sequence is 2. And do DFT on each subsequence
which can efficient reduce the amount of calculations. For example, In the Cooley-Tukey
algorithm, using the periodic and symmetry of 𝑊𝑛𝑘
𝑊𝑛𝑘 = 𝑒−2𝜋𝑗𝑘𝑛 = cos(−2𝜋
𝑘
𝑛) − 𝑗 ∗ sin(−2𝜋
𝑘
𝑛)
An N-terms sequence (N = 2𝑘, k is a positive integer) is divided into two N 2⁄ -terms sub-
sequences of. Each N 2⁄ point DFT needs N 2⁄ 2 operations, and the DFT transform of two
N 2⁄ points is combined into Nth DFT transform by N operations. After this transformation,
the total number of operations becomes N + 2 ∗ (N
2)2 = N +
N2
2. Continuing with the
example above, when N = 1024, the total number of operations becomes 525312 times,
saving about 50% of the computation. And if we keep this idea of "dividing into two" into the
DFT unit, then the DFT transform of N points only needs Nlog2N operations, N at 1024
points, the operation. The amount of computation is only 10240 times, only 1% of the
previous algorithm, the more points saves more computations, which is the superiority of
FFT. And the time complex of FFT is O(n ∗ log 𝑛)
Figure 2.4 (Steven W. Smith, 1997)
2.7 Canvas
In the custom View, we often need to draw some of the results you want. Here you need to
use the Canvas API. There are two ways to get a canvas object: one by rewriting the
View.onDraw method, the Canvas object in View will be passed as a parameter, and we will
control the Canvas and the result will be directly reflected in the View. But the android SDK
suggested using the View.onDraw parameter provided in the Canvas Class, no need to create
a new Canvas object. All details can founded in the Canvas API Guide (Google, 2017)
12
In the Figure 2.5, we can see that Canvas can draw objects: arcs, argb and color, Bitmap,
circle and oval, point, line, Rectangle, Picture, RoundRect, Text, Vertices, and path. And in
this project I choose point method to draw the final result.
Figure 2.5
2.8 OpenGL ES
After learning the books OpenGL-ES Game Development (Dave Astle, 2004) and OpenGL ES 2.0
Programming Guide (Shreiner, 2008), I have the following understanding of OpenGL:
Open Graphics Library (OpenGL) is a cross-language, cross-platform application
programming interface (API) for rendering 2D and 3D vector graphics. The OpenGL standard
is not platform-dependent, so the same program can run on different graphics cards. The
API only dealS with graphics rendering, does not provide animation, timer, file IO, image file
format processing, GUI and other functions.
13
OpenGL is implemented as a client-server system, and the application as a client, and the
OpenGL on the graphics hardware is the server. As shown in Figure 1, the client program
needs to call the OpenGL interface to achieve 3D rendering, then the OpenGL command and
data will be cached in the RAM, under certain conditions, these commands and data will be
sent to VRAM via the CPU, next under the control of the CPU, use the data and commands in
VRAM to complete the graphics rendering, and save the results into the frame buffer, finally,
the results will be sent to the display and show the results. In modern graphics hardware
systems, it is also possible to send data directly from RAM to VRAM via the CPU or send data
directly from the frame buffer to RAM. The process is like Figure 2.6
Figure 2.6
In simple words, OpenGL creates a 3D model in the buffer and cast 3D space to 2D plane.
Thus all works finished in the buffer, which can reduce the workload of CPU. (Google, 2017)
14
Chapter 3 System analysis & environment construction
3.1 Develop objectives
The purpose of this project is include to learn audio process programming, Android
development and java coding. Familiar with Java language in construction, method, library
about audio conversation. Compare the performance of FFT and DFT, Compare the
performance of GPU and CPU, analysis the reason leads to issues exposed in development
and also find the solution of these issues.
3.2 Feasibility Analysis
The development of any hardware-based program will be limited by time and resources. In
order to reduce the risk of the project, do the feasibility analysis before the program start
1) Economic feasibility
The application of this project is a local application, does not involve any information
exchange with the servers
2) Technical feasibility
The technology involved in development an FFT audio conversion program is quite
complicated, the three main parts is consist of audio acquisition, signal processing and
show the result by a canvas
3.3 The choice of Android version and other tools
The development progress of Real Time Spectrogram on Android Tablet program is used java
language based on JDK7.0 environment, Using Android studio development tools, and take
simulator installed SDK4.1.1 in the windows system PC to test. This chapter provides a brief
description of the architecture, application development tools, and data processing aspects
of the Real Time Spectrogram System.
In this project, we use Android studio compiler which launched by the Google official in
2013, in the test phase using the Genymotion Android simulator. As the upper application of
Android is developed in the Java language, but also need to use the Java SDK provided by the
SUN Company, which contains the JRE (Java Runtime Environment). However, Android
application development is different to Java development, so we also need to use Google
Android SDK which is already included in Android Studio. Thus, the project based on the
above software’s.
15
3.3.1 Android Studio
Android Studio is an Android integrated development tool (wikipedia, 2017), based on IntelliJ
IDEA. Similar to Eclipse ADT, Android Studio provides an integrated Android development
tool for development and debugging. The advantages of Android Studio include:
• Based on Grade build support
• Android-specific refactoring and quick fix
• Prompt tool to capture performance, availability, and
version compatibility issues
• Supports Roofguard and application signatures
• Provide template for Android application designs and
components
• Allow drag the UI controls and make previews
3.3.2 Genymotion
Genymotion is a complete set of tools that provide an Android virtual environment. With
this tool we can easily test our application. The features of Genymotion include:
• Support OpenGL acceleration
• Support full screen and full control
• You can start multiple emulators at the same time
• Support control simulator by shell
• Fully compatible with ADB
• You can set emulator parameters include screen size,
memory size, and CPU count
• Start a virtual device from Android Studio
16
Chapter 4 Application Implementation
In this chapter, I will introduce the application implementation this includes UI design, the
code implementation and parameter settings
4.1 System Design
The main task of this application is to assist the users to easily analyze the energy intensity
of audio signal at different frequencies, it consists of three parts: audio acquisition, FFT
conversion, the graphic output. The main UI surface is designed as Figure 4.1.
Figure 4.1
4.2 Application Implementation steps
The main steps of the application are:
1. Start the application
2. Obtain audio data from the microphone
3. Transform discrete numbers into complex to store
4. Fourier transform of complex numbers
5. Evaluate the complex value
6. Calculate the point color
7. Store color value in two-dimensional array
8. Draw graphic based on two-dimensional array
9. Repeat step (2) to (8)
10. Shut down the application
17
4.3 Code implementation
In this part, I will introduce the code implementation of this application, include layout
implementation, record function, how to implement the main algorithm (FFT algorithm) and
also introduce the output function. All code in this part can be found in Appendix 2
4.3.1 Layout
The layout structure of the application is very easy it consists of three button controls the
program and a screen to show the final graph of the result. Thus, when use XML to set the
layout I choose RelativeLayout label for whole elements and use Linearlayout label as the
screen to show the final result. And use TableRow label to make three button in one line.
4.3.2 Record function
In this part, when we press the start button, the application will do a recording permission
check first then according the return value decide the next work.
If request failed
The application will pop up the notification that “Request recording permission
failed, please go to setup to open the recording”
Else
It will execute the Recording instructions in recording Thread.
There are two ways to implement an asynchronous task mechanism in Android, Handler and
AsyncTask. To Implement the record function, I create a class RecordAudio extend
AsyncTask, which allow RecordAudio can create sub-thread to complete record fuction.
The doInBackground method is executed in the sub-thread, because it is the time-consuming
operation of the loop, so put in the sub-thread, the sub-thread calls AudioRecord and pass
the code rate and other information, after initialization to judge.
When the current started state is true, continue to perform while loop to get the audio
stream (audio_data), then called FFT method to do the calculate and store the result in
result_data. In the next step calculate the RPG value for the corresponding value and record
the point color and call the method publishProgress(current_point) to send to the main
thread in real time. All of these methods have detailed explanation in the follow parts.
The onProgressUpdate method is executed in the main thread and gets the PointColor[] data
stream continuously and store the data into the final result matrix and update continuously,
that’s means in result_color[][] it store the real time point color value needs to be shown on
the screen.
System.currentTimeMillis() methods are added before and after calling FFT method to
record the time the FFT algorithm consumes. And according to these data to evaluate the
performance of the FFT algorithm.
18
4.3.3 The Core Algorithm
As described above, the core idea of the FFT algorithm is divided the original data sequence
into two new sequence and repeat do this step again and again until the sequence only have
two data then do the DFT calculate to achieve the purpose of reducing the calculation.
In order to implement the FFT algorithm, I choose iterative method to group the data. After
n (n is Fourier series) times iteration each subsequence have only two elements, now I do
DFT on this subsequence then combine the result of subsequence to get the DFT result of
the original sequence.
Figure 4.2 (Steven W. Smith, 1997)
Based on a lot of research and study I found the number of the new sequence follows the
binary reverse order as Figure 4.2 which from The Scientist and Engineer's Guide to Digital
Signal Processing, According to the compare of left and right columns some regular pattern
can be obviously found that binary value of two numbers at the same location is reversed.
For example if the sequence have 16 elements, the 2nd number in original sequence is 1,
which the binary value is 0001, exchange the nth and (N − n)𝑡ℎ number, the result is 1000,
which equals 8 and the 2nd number in the sorted sequence is 8 too. Thus, I use this theory
to verify the algorithm accuracy by sampling, after multiple check I found the algorithm is
fully compliance with this rule.
19
4.3.4 Draw parts.
The principle of canvas is create a new bitmap and fill on the pixel points based on cache. In
the case cannot finish a complete figure, it won’t continue draw the next figure. [9]
In the completed project, the final figure output is drawn with a canvas. All these jobs are
finished in the sub thread. In this project, the process in the drawing process is draw points
one by one based on result matrix.
Of course, canvas also has some drawbacks, because of the refresh mechanism of canvas, it
may be limited by the machine performance, if the machine's processing speed is not fast
enough to reach 24 pictures per second users can feel it is not fluent.
And then move draw part into GPU by OpenGL ES, still use draw points way to draw the final
result in the texture. Regrettably, the performance improvement is not obvious this issue
will be explained in the next chapter.
4.3.5 Tools class
The tool class contains the complex class, point color class, calculate rotation factor method:
• The complex class is used to do represents complex numbers.
• The point color class is used to calculate point color according to energy intensity
• The calculate_rotation is used to calculate all rotation factor first to reduce the
calculation in the FFT algorithm.
• The Utils class defines the data structure used for OpenGL
4.4 Process analysis
In this application, the sampling frequency of the microphone is set to 44100HZ, execute an
FFT algorithm for each 1024 points data. That means it needs to do about 40 times 1024
point FFT calculation per second. I found that the result data is intermediate symmetry and
the odd number results equals 0. Thus, the effective result of 1024 data is only 256,
therefore the application only need to calculate 256 point color is enough this founding
reduce about 75% calculations. In the main thread the operation is renew the result matrix,
move the last result in to 0th row and move N𝑡ℎ row data into (N + 1)𝑡ℎ row when n<255.
The job of draw sub thread is show the result of the latest matrix.
20
Chapter 5 Test result and issues
To find the issues and evaluate the performance of the FFT and DFT, also the performance of
the CPU and GPU. A series experiments is designed. In this chapter, I will expound the result
and analysis the issues in the project
5.1 Test Case Design
According to learning from COMP4130-Managing Software Quality and Process, I decide
take three steps to test the application.
1) Take develop test on simulator through the whole develop cycle.
2) Take unit test of the FFT algorithm, draw parts, recording parts to grantee all function
can work as expected. And also evaluate the performance between DFT and FFT.
3) Take function test on this applications to make sure the application can work as
designed.
4) Take system test under different conditions.
Attention: step 2 to 4 will run on the simulator, and the screen size of the simulator is 1536 x
2048 and 320 dpi. Android OS 4.1.1, API16 on laptop with Core(TM) i7-4870HQ CPU @
2.50GHz.
5.2 Test Case detail & result
1) The result of development tests
This test is throughout the development process, the main work of this test is defined bug
which hinder the running of the program. Almost all bugs found in this test are fixed except
the button problem.
2) The result of unit tests
In this part test will divided into three parts, for core algorithm, draw part and recording
part.
In the test of algorithm, I choose n=8, 16, 32, 64, 128, 256, 512, 1024, 2048 as test cases to
test both DFT and FFT algorithm is right or not. After the test, I found the result of FFT I
check it is the same as the algorithm provided in library, reduce the result of my own
algorithm and the result of the algorithm provided in the library, all data is 0.
Then write a loop to test the performance of FFT and DFT, the Figure 5.1 is the time
consuming result of DFT, the Figure 5.2 is the time consuming result of FFT, the x axis is
refers to n point transform, the y axis is refers to time consuming and the standard unit is
second. It obvious that FFT is faster than DFT.
21
Figure5.1 DFT time consuming
Figure5.2 FFT time consuming
When I test draw part, I use a matrix to store the color and draw these
points on the screen, all these can work well without any bugs. All these
codes are the same in the original project except changed input data.
And the result of this test is as Figure 5.3.
The final part need to be test is the recording parts, through record and
play the record file to ensure this part can smooth running without
obvious error.
3) The result of Function test
In the function test, I invite my friends test this application. In the feedback report I found
three buttons cannot worked as design, however, since the priority of this bug is not the
highest, the repair work was postponed. And the canvas version also reported as cannot
support all Screen Size and when rotate the screen the application cannot work unless
Figure5.3
22
restart. And they also complain the UI surface is too monotonous. And these problem can be
reappear on the Samsung Tab P3110. As Figure 5.4
OpenGL version Canvas Version
Figure 5.4
4) The result of System test
In system test, I chose a single-core emulator for testing, which RAM size is set to 1024MB.
In the test result I found that the time consuming is average at about 150ms and I choose
100 times time consuming of FFT algorithm to generate a line graph in Figure 5.5, which far
more than the expected. In addition, the system test results of the CPU version not very
smooth, in the result the adjacent sets of data has the same value, which does not exist in
the GPU version, the output is shown as Figure 5.6.
Figure 5.5
0
50
100
150
200
250
300
350
1 6
11
16
21
26
31
36
41
46
51
56
61
66
71
76
81
86
91
96
Time Consuming
1 cores CPU 1 cores GPU
23
CPU version GPU version
Figure 5.6
5.3 issues and analysis and some solutions
After completing a series of tests, I found three main performance problems are
focused on the FFT algorithm and the draw part.
1) FFT algorithm is not optimized enough
In the first version, my FFT algorithm needs to sort the source data first, and group
two Adjoining data, and then do DFT on each group and integrate the results at last.
However I found it consumed a lot of time in matching. To solve this problem, I used
the iterative method to reduce required operations of sort. Except for this
improvement, Eric advised me that move the method used to calculate all rotation
factor at first instead calculate when needed, and make it static store all rotation
factors in the table. In fact, the effect of this improvement is not obvious, after
discussing with Eric, we infer that the reason for this situation is due to the table size,
many times application consumed to look for rotation factor needed in the table.
In addition, I guess the time consuming beyond the expectations may lead by
hardware constraints. To verify my guess, I added some test cases, as the use of
RAM is only 15MB, and the max RAM of test machine is 1024MB, there is no limit to
the performance of the application. Through variable control of the number of cores
to verify my guess. The average time consuming of 2 cores is about 50ms, and the
average time consuming of 4 cores is about 30ms. When I change the amount of the
cores to 8 the average time consuming is still maintain at 30ms. The logcat as Figure
5.7. With the data support, go back to check the code I determine that the main
reason is that the application have three threads, one for main thread, one for
drawing and one for recording which means if you want to run the application
24
perfectly, the machine need at least 3 cores. And in Appendix 3 I found when the
number of cores over 4 add more cores only can reduce CPU usage.
2 cores 4 cores 8cores
Figure 5.7
2) CPU version of the image output results do not meet expectations
As Figure 5.8, the performance of CPU version have a little problem, the nearby
groups have the same data. This problem cannot be reproduced when more than
three cores. Therefore, I infer that the cause of this problem is the same as the
previous one.
1-core 4-cores 8-cores
Figure 5.8
3) The results of the draw part is not smooth enough
As Figure 5.7 shows, the application cannot finish all calculations per second in time,
it seems that only finish about 20 to 30 times per second cannot meet the
requirement of the applications which need over 40 times per second. To solve this
problem, the frequency of the sample data can be reduced to 22050Hz, thus, the
required calculation reduced from 40 times to 20 times and can release more
resources on drawing. When the machine can draw 24 graphs per second, the result
display will become very smooth.
25
Chapter 6 Future work
The most important part of this project is moving the FFT part to GPU, since GPU has
powerful arithmetic capability. It seems will be more sufficient than execute in CPU. To
verify this conjecture, a serious of experiments is necessary. Then compare the result of GPU
version and CPU version and conclusion the founding. However, given time limitation I did
not complete this.
In the test, the application still have some bugs like button is not work as design, cannot
support all the screen size and after rotate screen cannot show result again in the list of
future work. Except the bugs the performance of draw part is not very smooth, thus,
optimize the code to reduce the operations is also very important.
In addition to the above points, the UI surface of now version, the result graph cannot
intuitively show the meaning of color. To solve this problem, redesign the UI surface is
unavoidable.
The last point is about extend functions, like analysis exist files, and save the data in the file
system.
26
Conclusion
This project was based on the interest on audio process, Android development and Java
coding. With the inspiration from literature about FFT, I learnt lots of knowledge about
digital signal processing, include time domain, frequency domain and also DFT and FFT, all
these research in order to write an efficient algorithm. The processes of application
implement enabled me to be more familiar with Java language in construction, method,
library and so on since I was new to Android.
Moreover, since this project has a huge amount of computing which encouraged me to
optimize the code make the application run smoothly as much as possible.
Also stressed that due to time constraints, this project does not fully cover Renderscript,
OpenCL implementation, only selected OpenGL as a GPU implementation. Thus, these two
parts is moved into the future work.
In conclusion, learnt many new things about audio processing and Android development,
and skill of coding and research. As I mentioned in the introduction, the Android OS is great
potential for development, I can get significant benefit from this project
27
Appendix 1 Study Contract
28
29
Appendix 2 Detail Code
Layout - Activity_main.xml
<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="fill_parent"
android:layout_height="fill_parent"
xmlns:tools="http://schemas.android.com/tools"
tools:layout_editor_absoluteY="16dp"
tools:layout_editor_absoluteX="-91dp">
<LinearLayout
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:orientation="vertical"
android:padding="5dip"
android:weightSum="1">
<TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_marginTop="5dip"
android:textSize="16sp"
android:text="Spectrogram" />
<LinearLayout
android:layout_width="match_parent"
android:layout_height="800dip"
android:layout_marginTop="5dip"
android:orientation="vertical"
android:background="#000000"
android:layout_weight="0.10"
android:id="@+id/Spectrogram">
</LinearLayout>
<TableLayout
android:layout_width="fill_parent"
android:layout_height="wrap_content"
android:layout_marginTop="5dip">
<TableRow>
<Button
android:id="@+id/start"
android:layout_width="fill_parent"
android:layout_height="wrap_content"
android:layout_weight="1"
30
android:text="Start" />
<Button
android:id="@+id/pause"
android:layout_width="fill_parent"
android:layout_height="wrap_content"
android:layout_weight="1"
android:text="Pause"
android:textColor="#ff0000" />
<Button
android:id="@+id/stop"
android:layout_width="fill_parent"
android:layout_height="wrap_content"
android:layout_weight="1"
android:text="Stop" />
</TableRow>
</TableLayout>
</LinearLayout>
</RelativeLayout>
RecordAudio Class
private class RecordAudio extends AsyncTask<Void, PointColor[], Void> {
@Override
protected Void doInBackground(Void... params) {
try {
//Initialize the recording class
bufferSize = AudioRecord.getMinBufferSize((int)frequency,
channelConfiguration, audioEncoding);
AudioRecord audioRecord = new AudioRecord(
MediaRecorder.AudioSource.MIC,(int) frequency,
channelConfiguration, audioEncoding, bufferSize);
short[] audio_data = new short[blockSize_buffer];
audioRecord.startRecording();
while (started) {
//Continue to get audio streams
bufferReadResult = audioRecord.read(audio_data, 0, blockSize_buffer);
for(int k = 0; k < blockSize_buffer; k++ ){
initial_data[k] = new Complex((double)audio_data[k],0);
}
long begintime = System.currentTimeMillis();
result_data = fft(initial_data);
long endtime = System.currentTimeMillis();
31
long costTime = (endtime - begintime);
Log.e("tag","costTime: "+costTime);
for(int k = 0; k < 256; k++ ){
current_point[k] = new PointColor((result_data[(2 * k) + 512].abs()));
}
//Sent to the main thread
publishProgress(current_point);
}
audioRecord.stop();
} catch (Throwable t) {
Log.e("AudioRecord", "Recording Failed");
}
return null;
}
protected void onProgressUpdate(PointColor[]... toTransform) {
for (int i=length - 1; i >= 0; i--) {
if (i == 0) {
for (int j = 0; j < length; j++){
result_color[0][j] = toTransform[0][j];
}
}
else if (i < length) {
for (int j = 0; j < length; j++){
if (result_color[i-1][j] != null){
result_color[i][j] = result_color[i - 1][j];
}
else{
result_color[i][j] = new PointColor(0,0,0);
}
}
}
}
init();
}
}
FFT algorithm
public Complex[] fft(Complex[] x) {
int n = x.length;
// exp(-2i*n*PI)=1，n=1
32
if (n == 1){
return x;
}
// if the number of signal is odd, use DFT
if (n % 2 != 0) {
return dft(x);
}
// choose even number
Complex[] even = new Complex[n / 2];
for (int k = 0; k < n / 2; k++) {
even[k] = x[2 * k];
}
Complex[] evenValue = fft(even);
// choose odd number
Complex[] odd = even;
for (int k = 0; k < n / 2; k++) {
odd[k] = x[2 * k + 1];
}
Complex[] oddValue = fft(odd);
// even + odd
Complex[] result = new Complex[n];
for (int k = 0; k < n / 2; k++) {
int m = (k * 512) / n;
result[k] = evenValue[k].plus(rotation_factor[m].times(oddValue[k]));
result[k + n / 2] = evenValue[k].minus(rotation_factor[m].times(oddValue[k]));
}
return result;
}
DFT algorithm
public Complex[] dft(Complex[] x) {
int n = x.length;
// one signal exp(-2i*n*PI)=1
if (n == 1)
return x;
Complex[] result = new Complex[n];
for (int i = 0; i < n; i++) {
result[i] = new Complex(0, 0);
for (int k = 0; k < n; k++) {
//e^(-i*2pi*k/N) = cos(-2pi*k/N) + i*sin(-2pi*k/N)
33
double p = -2 * k * Math.PI / n;
Complex wk = new Complex(Math.cos(p), Math.sin(p));
result[i].plus(x[k].times(wk));
}
}
return result;
}
Draw thread - canvas
handler = new Handler()
{
public void handleMessage(Message msg)
{
if(msg.what == 0x123);
{
ViewGroup parent = (ViewGroup) view.getParent();
if (parent != null) {
parent.removeAllViews();
}
spec.addView(view);
}
}
};
public void init()
{
new Thread()
{
public void run()
{
view = new DrawView(MainActivity.this, result_color);
view.invalidate();
Message msg = new Message();
msg.what = 0x123;
handler.sendMessage(msg);
}
}.start();
}
34
DrawView - canvas
public class DrawView extends View
{
PointColor[][] a = null;
public DrawView(Context context, PointColor[][] a)
{
super(context);
this.a = a;
}
@Override
protected void onDraw(Canvas canvas)
{
super.onDraw(canvas);
canvas.drawColor(Color.BLACK);
// creat paint
Paint p = new Paint();
p.setAntiAlias(true);
p.setStyle(Paint.Style.STROKE);
p.setStrokeWidth(4);
int viewWidth = this.getWidth();
int viewHeight = this.getHeight();
p.setColor(Color.RED);// set red
p.setStyle(Paint.Style.FILL);
for(int i = 0; i<256; i++)
{
for(int j = 0; j < 256; j++)
{
p.setColor(Color.rgb(a[i][j].R, a[i][j].G, a[i][j].B));// set color
canvas.drawPoint((viewWidth*i)/255, (viewHeight*j)/255, p);// draw a point
}
}
}
}
Complex class
35
public class Complex {
public double re; // the real part
public double im; // the imaginary part
// create a new object with the given real and imaginary parts
public Complex(double real, double imag) {
re = real;
im = imag;
}
// return a string representation of the invoking Complex object
public String toString() {
if (im == 0) return re + "";
if (re == 0) return im + "i";
if (im < 0) return re + " - " + (-im) + "i";
return re + " + " + im + "i";
}
// return abs/modulus/magnitude
public double abs() {
return Math.hypot(re, im);
}
// return angle/phase/argument, normalized to be between -pi and pi
public double phase() {
return Math.atan2(im, re);
}
// return a new Complex object whose value is (this + b)
public Complex plus(Complex b) {
Complex a = this; // invoking object
double real = a.re + b.re;
double imag = a.im + b.im;
return new Complex(real, imag);
}
// return a new Complex object whose value is (this - b)
public Complex minus(Complex b) {
Complex a = this;
double real = a.re - b.re;
double imag = a.im - b.im;
return new Complex(real, imag);
}
// return a new Complex object whose value is (this * b)
public Complex times(Complex b) {
Complex a = this;
double real = a.re * b.re - a.im * b.im;
36
double imag = a.re * b.im + a.im * b.re;
return new Complex(real, imag);
}
// return a new object whose value is (this * alpha)
public Complex scale(double alpha) {
return new Complex(alpha * re, alpha * im);
}
// return a new Complex object whose value is the conjugate of this
public Complex conjugate() {
return new Complex(re, -im);
}
// return a new Complex object whose value is the reciprocal of this
public Complex reciprocal() {
double scale = re*re + im*im;
return new Complex(re / scale, -im / scale);
}
// return the real or imaginary part
public double re() { return re; }
public double im() { return im; }
// return a / b
public Complex divides(Complex b) {
Complex a = this;
return a.times(b.reciprocal());
}
// return a new Complex object whose value is the complex exponential of this
public Complex exp() {
return new Complex(Math.exp(re) * Math.cos(im), Math.exp(re) * Math.sin(im));
}
// return a new Complex object whose value is the complex sine of this
public Complex sin() {
return new Complex(Math.sin(re) * Math.cosh(im), Math.cos(re) * Math.sinh(im));
}
// return a new Complex object whose value is the complex cosine of this
public Complex cos() {
return new Complex(Math.cos(re) * Math.cosh(im), -Math.sin(re) * Math.sinh(im));
}
// return a new Complex object whose value is the complex tangent of this
public Complex tan() {
return sin().divides(cos());
}
// a static version of plus
37
public static Complex plus(Complex a, Complex b) {
double real = a.re + b.re;
double imag = a.im + b.im;
Complex sum = new Complex(real, imag);
return sum;
}
// See Section 3.3.
public boolean equals(Object x) {
if (x == null) return false;
if (this.getClass() != x.getClass()) return false;
Complex that = (Complex) x;
return (this.re == that.re) && (this.im == that.im);
}
}
PointColor class
public class PointColor {
int R,G,B;
public PointColor(int r,int g,int b) {
R = r;
G = g;
B = b;
}
//calculate point color
public PointColor(double b) {
long f;
f = (long)(b*2);
R = (int)(f / 256);
G = R;
B = 0;
}
}
Calculate rotation factor
public static void calculate_rotation (){
for (int n = 0; n < 512; n++) {
// e^(-i*2pi*k/N) = cos(-2pi*k/N) + i*sin(-2pi*k/N)
double p= -2 * n * Math.PI / 512;
rotation_factor[n] = new Complex(Math.cos(p), Math.sin(p));
38
}
}
Myglrender Class
public class Myglrender implements GLSurfaceView.Renderer {
private float[] mArray = { 0f, 0f, 0f };
PointColor[][] a = null;
private float lheight=0;
int index = 0;
// buffer
private FloatBuffer mBuffer;
public Myglrender(PointColor[][] a, int index)
{
// get float buffer
mBuffer = Utils.getFloatBuffer(mArray);
this.a = a;
this.index = index;
}
@Override
public void onSurfaceCreated(GL10 gl, EGLConfig config) {
gl.glClearColor(0.0f, 0.0f, 0.0f, 0.0f);
}
@Override
public void onSurfaceChanged(GL10 gl, int width, int height) {
gl.glViewport(0, 0, width, height);
this.lheight=height;
}
@Override
public void onDrawFrame(GL10 gl) {
gl.glEnableClientState(GL10.GL_VERTEX_ARRAY);
// set point size
gl.glPointSize(4f);
for(int i=0; i<256; i++)
{
for(int j=0; j<256; j++)
{
mArray[0] = (float) ((i-127.5)/127.5);
mArray[1] = (float) ((127.5-j)/127.5);
mBuffer = Utils.getFloatBuffer(mArray);
gl.glVertexPointer(3, GL10.GL_FLOAT, 0, mBuffer);
39
float r = a[i][j].R/255f;
float g = a[i][j].G/255f;
float b = a[i][j].B/255f;
gl.glColor4f(r, g, b, 1f);
gl.glDrawArrays(GL10.GL_POINTS, 0, 1);
}
}
gl.glFlush();
}
}
Utils Class
public class Utils {
/**
* @param vertexs int array
* @return get int buffer
*/
public static IntBuffer getIntBuffer(int[] vertexs) {
IntBuffer buffer;
ByteBuffer qbb = ByteBuffer.allocateDirect(vertexs.length * 4);
qbb.order(ByteOrder.nativeOrder());
buffer = qbb.asIntBuffer();
buffer.put(vertexs);
buffer.position(0);
return buffer;
}
/**
* @param vertexs float array
* @return get float buffer
*/
public static FloatBuffer getFloatBuffer(float[] vertexs) {
FloatBuffer buffer;
ByteBuffer qbb = ByteBuffer.allocateDirect(vertexs.length * 4);
qbb.order(ByteOrder.nativeOrder());
buffer = qbb.asFloatBuffer();
buffer.put(vertexs);
buffer.position(0);
return buffer;
}
/**
* @param vertexs Byte array
* @return get byte buffer
40
*/
public static ByteBuffer getByteBuffer(byte[] vertexs) {
ByteBuffer buffer = null;
buffer = ByteBuffer.allocateDirect(vertexs.length);
buffer.put(vertexs);
buffer.position(0);
return buffer;
}
}
Draw parts - OpenGL
public void init()
{
if(flag) {
glSurfaceView.setRenderMode(GLSurfaceView.RENDERMODE_CONTINUOUSLY);
flag=false;
}
new Thread()
{
public void run()
{
// if support OpenGl ES 2.0
if (IsSupported())
{
myrender.a=result_color;
Message msg = new Message();
msg.what = 0x345;
handler.sendMessage(msg);
}
}
}.start();
}
41
Appendix 3 System Test Result
Monitors Information of 1 cores
Monitors Information of 2 cores
Monitors Information of 4 cores
Monitors Information of 8 cores
42
Bibliography
Dave Astle, D. D. (2004). Opengl-Es Game Development (Game Development Series). ACM.
Google. (2017). Canvas. Retrieved from Developers:
https://developer.android.com/reference/android/graphics/Canvas.html
Google. (2017). Canvas and Drawables. Retrieved from Developers:
https://developer.android.com/guide/topics/graphics/2d-graphics.html
Google. (2017). OpenGL ES. Retrieved from Developers:
https://developer.android.com/guide/topics/graphics/opengl.html
Help, N. L. (2012, August). Differences between Frequency Domain and Time Domain.
Retrieved from National Instruments: http://zone.ni.com/reference/en-
XX/help/370051V-
01/cvi/libref/analysisconcepts/differences_between_frequency_domain_and_time_
domain/
Kantar Worldpanel. (2017, 10 27). Smartphone OS sales market share. Retrieved from Kantar
Worldpanel ComTech: https://www.kantarworldpanel.com/cn-en/smartphone-os-
market-share/
Kaur, P., & Sharma, S. (2014, March 6-8). Google Android a mobile platform: A review.
Engineering and Computational Sciences (RAECS), 2014 Recent Advances in
(14254382), pp. 1-5. doi:10.1109/RAECS.2014.6799598
Shreiner, A. M. (2008). OpenGL ES 2.0 Programming Guide. Pearson Education.
Steven W. Smith. (1997). Chapter 12: The Fast Fourier Transform. In S. W. Smith, The
Scientist and Engineer's Guide to Digital Signal Processing.
Weisstein, E. (2017, Sep 20). Fourier Transform. Retrieved from Wolfram Research:
http://mathworld.wolfram.com/FourierTransform.html
wikipedia. (2017). Android Studio. Retrieved from wikipedia:
https://en.wikipedia.org/wiki/Android_Studio
