Internship Report

David Matos and Tiago Brito

Instituto Superior Tecnico, University of Lisbon, PT,david.r.matos@tecnico.ulisboa.pt,

tiago.de.oliveira.brito@tecnico.ulisboa.pt

Abstract. File systems have been used in modern operating systems tocontrol how data is stored and retrieved from physical storage devices.Since then, file systems have been the focus of several performance andstructural enhancements by the research community as they have a highimpact on system performance. Along side file systems, database man-agement systems have also been an integral part of modern OS. Duringour internship we worked with SQLite, a database management systembuilt on top of the file system abstraction, rather then beside it. The goalof our work was to create a workload for a real-world application whichuses SQLite transactions. In this document we describe the developmentprocesses and results achieved during the intership.

1 Introduction

This report describes the work developed during our five week internship atthe University of Texas at Austin, under the supervision of professor EmmettWitchel [1]. The goal of our work is to help current UTA researchers in their re-search projects. Upon our arrival, professor Emmett Witchel presented us severaldifferent projects on which our help would be useful. Of these different projects,two stood out as the most interesting projects for both of us.

The first project involved adapting a kernel level utility, such as the file sys-tem, and build it as a user library inside an application. Theoretically this hasseveral advantages, such as portability and performance improvements. This isbecause the file system code becomes independent from the kernel code, andthus can be executed at the user level rather than kernel level. This first projectwas the one we wanted to pursue, but since this is a kernel development project,which is generally complex and time consuming, we chose the second project inorder to complete the project in due time (five weeks).

The second project involved building a realistic workload for applicationsusing the SQLite database management system. The goal of this project is tohave a realistic scenario for testing an SQLite client. By building an easily re-producible workload for an application using SQLite we allow researchers to testdifferent versions of SQLite in the same scenario and evaluate the performancedifferences. Since this project seemed more achievable during the timespan of

this internship it was the project chosen by us.

The second project was then developed during the following weeks of the in-ternship. During this period we presented intermediate versions of the prototypesto professor Emmett’s students which were responsible for the project. In thesemeetings we presented research work done on related work, ideas and prototypeimplementations of the work done so far. The development process, includingthe specifics of our project choice and the steps taken during the development ofthis projects, will be described in the following section of this report (section 2).In section 3 we will present the prototypes developed during the internship andfinally, in section 4 we give an general overview of our internship and experiencegained during our stay at the University of Texas at Austin.

2 Development Process

Upon our arrival at the University of Texas at Austin, we met with professorEmmett Witchel. Professor Emmett is our program mentor and, as such, he wasresponsible for assigning us a project to be developed during our five week in-ternship. During our first and only meeting with professor Emmett, he showedinterest in knowing more about previous research work developed by us at Insti-tuto Superior Tecnico and about our research interests. Additionally, professorEmmett presented us some of his own research interests. It became clear thatprofessor Emmett Witchel has a deep interest and knowledge of operating sys-tems and specially in the field of file system and transaction research.

Similarly, our interests and research experience is on operating systems anddistributed systems, although our focus is on the security aspect of these systems.This was good because it allowed professor Emmett to establish common groundbetween his students’ research projects and our research experience. With thecommon ground established, professor Emmett described us the current researchprojects being developed by their students and which projects he believed wouldgain from our cooperation.

From the projects suggested by professor Emmett two stood out as beingmore interesting for the both of us. These were the projects which more closelyresembled our research experience and were the projects that professor Emmettwas convinced could enjoy our help the most. Professor Emmett than proceededto introduce us to his research students responsible for the projects suggested.Since not all of professor Emmett’s students were at the university during thismeeting, we were only introduced to the students responsible for one of theprojects that stood out for us.

The students responsible for this first project introduced and explained usthe concepts and motivation behind their project as well as how our coopera-tion could help them achieve their project’s goals. As explained in section 1 this

first project involved adapting a kernel level utility, such as the file system, andbuild it as a user library inside an application. This first project was the one wewanted to pursue as it would allow us to improve our knowledge of kernel leveldevelopment and specially of common file system utilities such as the EXT4 filesystem. But because kernel development projects are generally very complex,and because we have limited experience in such projects, we believed the projectmight not be viable for our limited timespan of five weeks. In addition, profes-sor Emmett was not convinced with the student’s approach to the project, thuslowering our confidence on this project.

The second project involved building a realistic workload for applications us-ing the SQLite database management system. Although we were not introducedto the students responsible for this research project, the work suggested for us todo for this project during these five weeks seemed more achievable than the firstproject. But because we were interested in improving our kernel developmentskills, we had doubts about choosing this project in regards to the other one.

After being introduced to the suggested projects and the students responsiblefor these projects professor Emmett Witchel finished the meeting by giving usa tour of the laboratory and suggesting a place to work during the week days.After this, professor Emmett suggested us to think about the projects until theend of that week and answer with which project we wanted to cooperate in. Inthe mean time, professor Emmett suggested us to study the first class of hisIntroduction to Operating Systems course, which showed us how to prepare akernel development environment, for the first project. The following subsectionsdescribe our approach to selecting one of the two projects mentioned before.

2.1 Project Selection

After the first and only meeting with professor Emmett Witchel we started re-searching projects similar to what was described during the meeting. In addition,we sent emails to the people responsible for both projects in order to gather moreinformation and clarify some project specific questions. The students responsi-ble for the second project seemed more interesting in our cooperation and sentus two articles [2, 4] of related work so we could familiarize ourselves with theproject.

In addition to reading those papers, we also studied the first class of pro-fessor Emmett’s Introduction to Operating Systems course, although we wouldeventually choose the second project, thus rendering this class unnecessary forthe conclusion of the project. Given the high interest by the students responsi-ble for the second project, together with complexity inherent to the first (kerneldevelopment) project, we decided to choose the second project to be developed

during our internship.

Consequently, we contacted Yige Hu and Vijay Chidambaram (professor Em-mett’s students) to confirm our interest in cooperating in their research projectduring our stay at the University of Texas at Austin. As a response to our email,Yige and Vijay arranged a meeting via Skype, an application that provides videochat and voice call services. The meeting was arranged in this way because bothYige and Vijay were not at the university during the period of our internship.Because these students were not at the university during our internship, we neverfelt the need to work directly on professor Emmett’s laboratory. Our project wasdone comfortably using our own personal computers and in San Jacinto Resi-dence Hall.

The meeting had the goal to clarify project specific questions and to guideus towards the right path of development for their workloads. In addition, thismeeting was used to schedule consequent meetings to update Yige and Vijay onthe progress of our work. The following subsection describes the meetings anddevelopment process we followed during the remaining four weeks.

2.2 Development Steps

First Meeting: the goal of the first meeting was to describe the problem to besolved by us during our internship. The problem was described as the difficultyin testing realistic SQLite workloads consistently. So, in order to overcome thisdifficulty, we our proposed to build a reproducible workload based on realisticapplications, such as email clients, web browsers and social networking applica-tion, which use SQLite. Since we were both new to SQLite and particularly inbenchmarking SQLite performance we were proposed to research the literatureand come up with an approach for the development of the workloads.

First week: as proposed in the first meeting, the following week was used toresearch related work regarding SQLite benchmarking and SQLite workloads.Our attention was focused on finding workloads which would represent realis-tic scenarios, and consequently realistic SQLite use. While researching for theserealistic workloads and scenarios, we came across several articles regarding thetrace and analysis of Input/Output characteristics in file systems where SQLiteis used and analyzed [5–7].

This first article [5] studies the IO characteristics of SQLite transaction inAndroid platform. The authors collected block level IO traces for six months anddeveloped a pattern matching algorithm which allows them to identify the indi-vidual SQLite transactions from the raw IO trace. This article highlights three

key findings, which are as follows. First, SQLite transactions are extremely in-efficiency since over 75% of the entire IO volume in a SQLite transaction comefrom SQLite journaling and EXT4 filesystem journaling. Second, the suspendand the wakeup feature of the smartphone can leave the SQLite transaction toan extreme delay, a few minutes. Third, SQLite transactions are not used effi-ciently by application programmers. Although this article describes preliminarysolutions to these problems, we are not interested in these solutions but ratheron the process of analysis and conclusions of the initial study.

The second article [6] describes an IO trace and analysis framework, An-drotrace, developed for the Android platform. Androtrace’s novelty is how itefficiently traces IO operations and how it supports the analysis of multiple An-droid users by using a server and client model. The authors conclude, throughthe use of the framework, that write IOs dominate the workload of mobile ap-plications. This is interesting because it gives us an insight on what realistic IOworkloads might look like. On one hand, based on this analysis, one can say thatrealistic workloads are write IO intensive. But on the other hand, Androtracedoes not specify the origin of these write IO operations, meaning that thesewrites may be from sources other than SQLite transactions. For this reason, theinformation from this article builds up on previously acquired information, butdoes not describe realistic SQLite workloads and as such is not sufficient for us.

Similarly, the third article [7] offers an in-depth IO characterization of anAndroid smartphone. The authors examined the correlations between seven IOattributes: originating application, file type, IO size, IO type (read/write), ran-dom/sequential, block semantics (Data/Metadata/Journal), and session type(buffered vs. synchronous IO). The authors also developed Mobile Storage Ana-lyzer (MOST) [8], a framework for collecting IO attributes across software layers.In the underlying study done for this article, the authors noticed that SQLiteputs too much burden on the storage. For example, a single SQLite operationresults in at least 11 write operations being sent to the storage. As for storage,more than 50% of writes are for EXT4 Journal updating and 70% of all writesaccount for small data writes, about 4KB. Because in Android the SQLite andEXT4 filesystem require great effort to ensure reliability for transactions andjournaling, respectively. The operations of SQLite and EXT4, when combined,generate unnecessarily excessive write operations to the storage device.

The authors conclude that this not only degrades IO performance but alsosignificantly reduces the lifetime of the underlying NAND flash storage. Thisarticle is revealing for our work because the authors developed IO profiles forsix different application categories, such as Web, Contact, Social Networking,Multimedia, System and Game. Namely this article profiled custom scenariosfor specific well-known applications which fit the mentioned categories. Theseprofiles allow us to understand how realistic workloads are partitioned regardingSQLite reads and writes. With this understanding we can implement a bench-

marking tool which simulates these applications by leveraging SQLite reads andwrites ratios.

Second Meeting: Before the second meeting we sent an email to Yige Hu andVijay Chidambaram describing our research work during the week. During themeeting we talked about our findings and discussed the idea of developing ascript leveraging Mobibench [9] to simulate the profiles described before. Mo-bibench is a tool which allows the user to specify several SQLite parameters tobe tested. Mobibench then executes the benchmarking using the specified pa-rameters. These parameters can be adjusted as to closely resemble the profilesdiscussed above. This would allow researchers to easily benchmark different re-alistic workloads. The idea was well received and as such we proposed to have araw prototype of the mentioned script before the following meeting.

Second Week: During this second week we focused our efforts on developinga prototype of the benchmarking script proposed to Yige and Vijay. We createda BASH script which allows the user to easily specify which of the mentionedprofiles he wants to benchmark. Additionally the script prepares the testingenvironment by installing Mobibench. By the end of this week we achieved inhaving a working version of this script which would benchmark the six differentprofiles using the Mobibench tool.

Third Meeting: This third meeting was used to show Yige and Vijay what wasaccomplished during the week. Unfortunately this meeting was not attended byboth of professor Emmett’s students since only Yige was available at the timeof the meeting. We were still glad to be able to have this meeting in order toclarify questions regarding the next steps to take on this project. We explainedto Yige what the script was capable of achieving and explained what was thegoal of this script.

Yige seemed to understand and support the work done so far and suggestedthe addition of a feature to the script. The suggestion was to allow the user ofthe script to manually adjust the read/write ratio of SQLite transactions testedby Mobibench. We argued that adding this feature would defeat the purposeof having profiles of realistic workloads. But Yige felt that adding this featurewould improve the usability of the script, so we added that to our to-do list.

We then asked Yige if they were interested in having our script to automat-ically average the results of running several tests of the same profile and alsocreate graphs to compare the different results. She agreed to this feature andsuggested us to test how many tests were needed to have a small standard devi-ation during the benchmarks. We scheduled the next meeting to the same dayand time next week and proposed to have the new features implemented until

then.

Third Week: During this third week we tried to implement the new featuresdiscussed during the previous meeting, with Yige. Originally, our BASH scriptallowed the user to run one benchmark for a specific profile. The result wouldbe the specific measures of the benchmark, which are described in full detail insection 3. But, to support the new feature, which would allow the user to runa battery of benchmarking tests on a specific profile, we had to find a way torun our original script a predetermined amount of times, gather all the results,interpret these results by taking the average and standard deviation and finallyproduce the graphs which represent these results.

In order to avoid re-doing the original script we planned on creating a newscript which would leverage the original script in order to produce the intendedoutput. We thought about using Python as the scripting language for the newscript because it is a flexible programming language and supports libraries forcreating graphs. But because leveraging several scripts with different program-ming languages is unpractical we reconsidered writing our original script again,this time using Python. Since the only difference between the scripts is the pro-gramming language’s syntax, re-writing the original script introduced little over-head to our project.

So, during this week we re-implemented our original script with Python andstarted to implement the support for the graph representation and to calculatethe average of several benchmarks of one profile.

Forth Meeting: This meeting was again attended by both Yige and Vijay.We described the state of our work and discussed what was discussed duringthe previous meeting, at the time with Yige only. Unexpectedly, Vijay told usthat what we were doing was not what they wanted. Because this was our forthmeeting we were surprised by this remarked, but listened to what Vijay was sug-gesting to do. It became clear that there was a lack of understanding betweenboth parties regarding what the goals of the project were. We understood theywanted to benchmark realistic SQLite workloads, but they failed to clarify howthese workloads should have been built.

In previous meetings our suggestion of using Mobibench to simulate work-loads was well received, but it seems the goal was not to use simulated workloadsfrom a benchmarking tool such as Mobibench but instead use real-world SQLitequeries. At this point we realized that most of our work done so far was ren-dered useless for the newly perceived goals of the project. Even so, Yige andVijay suggested us to still deliver the script implemented until then, since itmay be useful later on. In order to recover lost time we suggested analyzing areal-world SQLite application and use it as the desired workload. That work was

done during the following week.

Forth Week: In order to build a realistic SQLite workload using real SQLitequeries we had to study real applications. For time constraints we chose to an-alyze only one application corresponding to one of the profiles described above.We searched for open-source applications in order to have access to the SQLitequeries executed by the workload. After extensive search of well-known open-source Android applications we chose K9mail [10], a popular Android email clientapplication.

After inspecting the source code we noticed that all SQLite queries were trig-gered by one Java class. This allowed us to implement a wrapper class whichreplaces the original SQLite queries class and records all queries executed whilethe application is running. After implementing this wrapper class we proposed ascenario to record a realistic workload for our project. The scenario we proposedwas opening the application, sending three emails and closing the application.This scenario is based on a specific scenario presented in the same article wherethe profiles are first described.

The idea is to record all the queries executed during this scenario so theycan later be reproduced for benchmarking purposes. By reproducing the queries,instead of simulating the workload with Mobibench, researchers can test thesame workload and interpret the performance differences between using differ-ent SQLite and file system implementations.

Fifth and Final Meeting: The final meeting was, once again, with Yige onlysince Vijay could not attend again. We described our progress to Yige and triedto clarify how our progresses can help in achieving the goal of the project. Yigeunderstood our approach and how this could be used to build the realistic work-load needed for benchmarking their project. On one hand, our approach works,on the other hand, Yige felt like we could have accomplished the same resultwith a more clever approuch. Considering that using a different, more genericapprouch would be much better for adapting different applications to usableworkloads. This is because writing wrapper classes to all applications we wish touse as a workload is unpratical, but if we took that approach on the last week,we would risk not having results.

The following section describes implementation details regarding the differentsolutions implemented during our internship.

3 Prototypes

During the internship we developed two prototypes to evaluate the throughputof SQLite [3] databases. At first it was asked to develop a workbench that wouldexecute realistic SQLite workloads into a database. However, after we finishedthe first prototype we were informed that it wasn’t useful so they asked us todevelop a new prototype that would gather a real world workload of a database.

3.1 Prototype 1

The first prototype was a workbench to evaluate the throughout of several work-loads in a real SQLite database. To do so we based our work in an existing SQLiteworkbench called MobiBench [9]. Mobibench is a CLI tool that allows to measureIO performance and SQLite of Android filesystem. Mobibench is highly config-urable, allowing to adjust several parameters of a workload. The configurableparameters are:

– p set path name– f set file size in KBytes– r set record size in KBytes– a set access mode– y set sync mode– t set number of thread for test– d enable DB test mode– n set number of DB transaction– j set SQLite journal mode– s set SQLite synchronous mode– g set replay script– q do not display progress

The problem with MobiBench is that it is necessary to give as input theexact parameters of a workload. In order to simulate the behavior of real worldapplications we need to adjust these parameters to match the workloads gener-ated by real applications. The workload of each application varies depending onthe application features. For instance, an email client app will generate severalread operations (gathering the inbox messages every time a user opens the app)and less write operations (every time an email is received it must be stored inthe inbox). A social network application (for example: Facebook or Twitter) willgenerate more write operations, since every post or tweet must be stored in localstorage, and there are more posts and tweets being generated than emails.

So, in order to generate workloads in MobiBench that represent real worldapplications we need to know the number of read and write operations generateby each applications, as well as the size of these operations and frequency. Af-ter researching several articles we discovery two [2, 4] that describe in detail theworkloads of popular applications. Using the numbers in these articles we de-veloped a script that given a profile (a category of an application, for instance,

email) executes the respective workloads in MobiBench. With our script it ispossible to simulate a real application in MobiBench giving only a category ofan application instead of setting manually all the parameters of MobiBench.Besides running realistic workloads, our script also install the latest version ofMobiBench by using the -i option.

The script was developed in bash and takes as input the following parameters:

– h Prints to stdout the usage of this script;– i Git clones and compiles the latest version of MobiBench;– b Starts a benchmarking evaluation given a profile. The available profiles

are:

• web: web browser opening a couple of pages and performing a singleoperation in each page;

• sns: social network application. Simulates 10 scroll down operations inthe wall;

• multimedia: a video application. Simulated the user opening a videoplaying it for a couple of seconds and closing the video;

• system: simulates a system application that runs in background con-stantly;

• game: simulates a game.

After we finished the first version of this script, in the third week, we had ameeting with Yige Hu and Vijay Chidambaram. In this meeting we presentedour work and asked for improvements. However, they told us that although thetool is useful it does not give a reliable evaluation of real world applications,since the numbers used in the script were based in an old article and not inreal operations. It was important to have the exact executed queries by eachapplication, instead of the total number of read and write operations.

3.2 Prototype 2

With only a week and a half left for our internship we propose a second prototype.Our proposition consisted in a database wrapper, which is a layer between theapplication and the database that is responsible for intercepting every queryand register a log. With this wrapper it is possible to collect the exact queriesexecuted by each application. This way a user may simulate a behavior in anapplication and then consult the log with executed queries to later reproducethe interactions. Since we only had a week and half until our return we wrotethe description of this prototype and waited for confirmation before developing.A day later Yige Hu confirmed our project and we started developing it.

We based this prototype on the application K9 Mail [10]. K9 Mail is a wellknown open source e-mail client for Android that uses SQLite to store informa-tion about the user account and the messages. Figure 3.2 represents how K9 Mailinteracts with SQLite. On the left side of the figure K9 Mail executes queries (1)in the SQLite database and gets the results (2). On the right side of the figure weincluded our Database Wrapper and Log components. In practice the database

wrapper consists of a Java class that substitutes the standard SQLiteDatabaseclass of SQLite. So, instead of interacting directly with the database, K9 Mailinteracts with the Database Wrapper (1) which registers every query to a Logfile (2) and then redirects the query to the database (3), finally the result of thequery is returned to the Database wrapper (4) which is forwarded to the K9Mail application (5). There is no difference, from the user perspective, in theapplication. Everything works as it did before introducing the wrapper. Afterinteracting with the application for a short period it is possible to consult allthe executed operations in the database.

K9 Email

SQLite DB

Database Wrapper

Log

K9 Email

SQLite DB

1. queries 2. results

1. queries 5. results

2. queries

3. queries 4. queries

Without Database Wrapper

With Database Wrapper

Fig. 1. Flow diagram of K9 Email without Database Wrapper and with DatabaseWrapper

Figure 3.2 shows the K9 Mail application running in the Android emulator.The red square marks the log with every SQLite query executed so far. Withthis prototype it is possible to gather real workloads of a real application (K9Mail). The prototype allows any execution scenario to be tested. For instance,it is possible to collect the SQLite operations of a simple user interaction (openaccount, read an email and send an email) as well as testing a complex scenario ofexecution (reading several emails, responding to old emails, deleting emails andperforming searches). By using this prototype it is possible to simulate any userinteraction and it is simple collect SQLite operations of different applications.All that needs to be done is import our Database Wrapper in any Androidapplication.

Fig. 2. K9 Mail application running in the emulator and log with database operations.

4 Conclusion

During 5 weeks we study and develop prototypes to simulate real world appli-cations in SQLite databases. We manage to complete two functional prototypesand present a state of the art of SQLite in Android. Both prototypes are ca-pable of producing a realistic experimental evaluation of Android applicationsusing SQLite. The first prototype calculates the throughput of a database withdifferent applications. With the second prototype it is possible to collect theexact queries that were executed by an Android application. We also providedan example of an application (K9 Mail) using the second prototype.

We are satisfied with our work and believe that we achieve our goals forthis internship. However, the fact that 5 weeks if a short period of time todevelop a significant research project and present useful results, it would bemore useful if we had precise directions, in terms of goals of our project and abetter support from our mentors. In the end, we felt that we ended up wastingtime researching and developing tools that were not exactly what was needed. Itwould be beneficial if we had the topic and goals of our project and before goingto the internship. This way we could get ahead and start working in sooner andperhaps present a better work.

In conclusion, the internship was a very fulfilling experience. We got thechance to know and work with different people and learn interesting topics thatare not related with our research fields. Five weeks is a short period to develop arelevant project nevertheless we were able to produce useful tools and contributeto their work.

References

1. Emmett Witchel, PhD. Web site: https://www.cs.utexas.edu/users/witchel/2. Ji, Cheng, et al. ”An Empirical Study of File-System Fragmentation in Mobile Stor-

age Systems.” 8th USENIX Workshop on Hot Topics in Storage and File Systems(HotStorage 16). 2016.

3. Owens, Mike, and Grant Allen. ”SQLite”. Apress LP, 2010.4. Luo, Hao, Hong Jiang, and Myra B. Cohen. ”Why Do We Always Blame The

Storage Stack?.” 8th USENIX Workshop on Hot Topics in Storage and File Systems(HotStorage 16). 2016.

5. Tuan, Dam Quang, Seungyong Cheon, and Youjip Won. ”On the IO characteristicsof the SQLite Transactions.”

6. Lim, Eunryoung, Seongjin Lee, and Youjip Won. ”Androtrace: framework for tracingand analyzing IOs on Android.” Proceedings of the 3rd Workshop on Interactionsof NVM/FLASH with Operating Systems and Workloads. ACM, 2015.

7. Lee, Kisung, and Youjip Won. ”Smart layers and dumb result: IO characteriza-tion of an android-based smartphone.” Proceedings of the tenth ACM internationalconference on Embedded software. ACM, 2012.

8. Lee, Kisung, and Youjip Won. Mobile Storage Analyzer: https://github.com/ESOS-Lab/MOST

9. Mobibench Web site: https://github.com/ESOS-Lab/Mobibench10. Jesse Vincent. Advanced Email for Android: http://k9mail.github.io/