Big Data for Engineers – Exercises Spring 2020 – Week 11 – ETH Zurich MongoDB Introduction This exercise will cover document stores. As a representative of document stores, MongoDB was chosen for the practical exercises. Instructions are provided to install it on the Azure Portal. 1. Document stores A record in document store is a document. Document encoding schemes include XML, YAML, JSON, and BSON, as well as binary forms like PDF and Microsoft Office documents (MS Word, Excel, and so on). MongoDB documents are similar to JSON objects. Documents are composed of field-value pairs and have the following structure: The values of fields may include other documents, arrays, and arrays of documents. Data in MongoDB has a flexible schema in the same collection. All documents do not need to have the same set of fields or structure, and common fields in a collection's documents may hold different types of data. 1.1 General Questions 1. What are advantages of document stores over relational databases? 2. Can the data in document stores be normalized? 3. How does denormalization affect performance? Solution 1) Flexibility. Not every record needs to store the same properties. New properties can be added on the fly (Flexible schema). 2) Yes. References can be used for data normalization.
Transcript
BigDataforEngineers–Exercises
Spring2020–Week11–ETHZurich
MongoDB
IntroductionThis exercise will cover document stores. As a representative of document stores, MongoDB was chosen for the practicalexercises. Instructions are provided to install it on the Azure Portal.
1.DocumentstoresA record in document store is a document. Document encoding schemes include XML, YAML, JSON, and BSON, as well as binaryforms like PDF and Microsoft Office documents (MS Word, Excel, and so on). MongoDB documents are similar to JSON objects.Documents are composed of field-value pairs and have the following structure:
The values of fields may include other documents, arrays, and arrays of documents. Data in MongoDB has a flexible schema in thesame collection. All documents do not need to have the same set of fields or structure, and common fields in a collection'sdocuments may hold different types of data.
1.1GeneralQuestions1. What are advantages of document stores over relational databases?2. Can the data in document stores be normalized?3. How does denormalization affect performance?
Solution1) Flexibility. Not every record needs to store the same properties. New properties can be added on the fly (Flexible schema).
2) Yes. References can be used for data normalization.
3) All data for an object is stored in a single record. In general, it provides better performance for read operations (since expensivejoins can be omitted), as well as the ability to request and retrieve related data in a single database operation. In addition, embeddeddata models make it possible to update related data in a single atomic write operation.
1.2True/FalseQuestionsSay if the following statements are true or false.
1. Document stores expose only a key-value interface.2. Different relationships between data can be represented by references and embedded documents.3. MongoDB does not support schema validation.4. MongoDB encodes documents in the XML format.5. In document stores, you must determine and declare a table's schema before inserting data.6. MongoDB performance degrades when the number of documents increases.7. Document stores are column stores with flexible schema.8. There are no joins in MongoDB.
Solution1. (False) Document stores expose only a key-value interface.2. (True) Different relationships between data can be represented by references and embedded documents.3. (False) MongoDB does not support schema validation.4. (False) MongoDB encodes documents in the XML format.5. (False) In document stores, you must determine and declare a table's schema before inserting data.6. (True) MongoDB performance degrades when the number of documents increases.7. (False) Document stores are column stores with flexible schema.8. (True) There are no joins in MongoDB. Nonetheless,startinginversion3.2,MongoDBsupportsaggregationswith
"lookup"operator,whichcanperforma LEFT OUTER JOIN .
2.MongoDBIn this part of the exercise, you will setup a MongoDB image using AzureContainerInstances(ACI). By using ACI, apps can bedeployed without explicitly managing virtual machines. You can learn more about ACI here.
**Important: please delete your container after finishing the exercise.**
2.1InstallMongoDB1. Open the Azure portal and click "Createaresourece". After searching for container instances , click "Container
2. In the "Basics" tab, select your subscription for this exercise, and create a new resource group.3. Fill in the container name and region. You can select any region you prefer.4. Select "DockerHuborotherregistry" for "Image source", and type in mongo in the "Image" field. By default, Azure will use
Docker Hub as the container registry. Leave other settings as default.
5. In the "Networking" tab, choose a DNS name for your container. Open port27017 which is the default port that MongoDBlistens to. Use TCP for the port.
6. Change nothing on the "Advanced" and "Tags" tabs.7. In the "Review" tab, review your resource settings and click "Create". The deployment should be finished in a couple of
minutes. In fact, fast startup time is one of the benefits of using ACI!
2.2SetupatestdatabaseAfter the container is deployed, we need to connect to the container to create a database user.
1. Select the recently created container resource from Azure portal, click "Settings-Containers", then choose the "Connect"tab. Use /bin/bash as start up command. Click "Connect".
2. Start MongoDB shell by mongo -shell .3. Select the admin database:
mongoimport --db test --collection restaurants --drop --file ./primer-dataset.json
If the dataset is successfully imported, you should see something similar to this:
2.3ConnecttotheMongoDBserverWe have finished setting up the database server. Next, we need to connect to the server using a pymongo client. First, installsome packages:
In order to connect to MongoDB, we need to know the domain name of the host. In the resource console, click "Overview" to seethe basic information of the container. Copy the host URL from the "FQDN" field and paste it in the following cell. Execute it toconnect to the database.
In [3]:
# global variables for MongoDB host (default port is 27017)DOMAIN = 'mymongo.westeurope.azurecontainer.io' # Note: this should be replaced by the URL of yourown container!! PORT = 27017
# use a try-except indentation to catch MongoClient() errorstry: # try to instantiate a client instance client = MongoClient( host = [ str(DOMAIN) + ":" + str(PORT) ], serverSelectionTimeoutMS = 3000, # 3 second timeout username = "root", password = "root", )
db = client.test except errors.ServerSelectionTimeoutError as err: # set the client to 'None' if exception client = None
As a sanity check, we count the number of documents in the restaurants collection that we previously imported. It shouldmatch the number reported by mongoimport .
In [4]:
db.restaurants.count_documents({})
Requirement already satisfied: pymongo==3.10.1 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (3.10.1)WARNING: You are using pip version 19.3.1; however, version 20.1 is available.You should consider upgrading via the 'pip install --upgrade pip' command.Requirement already satisfied: dnspython in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (1.16.0)WARNING: You are using pip version 19.3.1; however, version 20.1 is available.You should consider upgrading via the 'pip install --upgrade pip' command.
2.4MongoDBCRUDoperationsIn this section, we will go through some commonly used CRUD (Create, Read, Update, Delete) operations in MongoDB.
In [5]:
# Create a new collectionscientists = db['scientists']
In [6]:
# Insert some documents.# Note that documents can have nested structures, and the collection can be heterogeneous.scientists.insert_one({ "Name": { "First": "Albert", "Last": "Einstein" }, "Theory": "Particle Physics"})scientists.insert_one({ "Name": { "First": "Kurt", "Last": "Gödel" }, "Theory": "Incompleteness" })scientists.insert_one({ "Name": { "First": "Sheldon", "Last": "Cooper" }})
In [7]:
# Select all documents from the collectionscientists.find()
In [8]:
# As you can see, find() method returns a Cursor object. One must iterate the Cursor object to access individual documentsfor doc in scientists.find(): pprint(doc)
QueryDocumentsFor the db.collection.find() method, you can specify the following optional fields:
a queryfilter to specify which documents to return,
Out[4]:
25359
Out[6]:
<pymongo.results.InsertOneResult at 0x7fb8165fd048>
a queryfilter to specify which documents to return,a queryprojection to specify which fields from the matching documents to return (the projection limits the amount of data thatMongoDB returns to the client over the network),optionally, a cursormodifier to impose limits, skips, and sort orders.
In [9]:
# Using a query filterfor doc in db.scientists.find({"Theory": "Particle Physics"}): pprint(doc)
In [10]:
# Using a projectionfor doc in db.scientists.find({"Theory": "Particle Physics"}, {"Name.Last": 1}): pprint(doc)
In [11]:
# Using a projection, with "_id" output disabledfor doc in db.scientists.find({"Theory": "Particle Physics"}, {"_id": 0, "Name.Last": 1}): pprint(doc)
<pymongo.results.InsertManyResult at 0x7fb8165e2d08>
for doc in scientists.find({"Profession": "Mathematician"}, {"_id": 0, "Name": 1}).sort("Name", 1).skip(1): pprint(doc) print("Using limit:")for doc in scientists.find({"Profession": "Mathematician"}, {"_id": 0, "Name": 1}).sort("Name", 1).skip(1).limit(3): pprint(doc)
In [14]:
# Updating documents
# Adding a new field:scientists.update_many({"Name": "Einstein"}, {"$set": {"Century" : "20"}})pprint(scientists.find_one({"Name": "Einstein"}))
# Changing the type of a field:scientists.update_many({"Name": "Nash"}, {"$set": {"Profession" : ["Mathematician", "Economist"]}})pprint(scientists.find_one({"Name": "Nash"}))
In the lecture, we learnt how to write queries in the syntax of the MongoDB shell. The syntax is a bit different from the syntax of pymongo . Here are a few examples:
MongoDBshell pymongo Note
Insert insert() insert_one() or insert_many() insert() is also valid for pymongo but deprecated.
Update update() update_one() or update_many() update() is also valid for pymongo but deprecated.
Delete delete() delete_one() or delete_many() delete() is also valid for pymongo but deprecated.
Sort criterion JSON document list of (key, direction) pairs
Count db.collection.find(filter).count() db.collection.count_documents(filter) count() is also valid for pymongo but deprecated.
It is not necessary to remember these differences, but you should understand the semantics of a query written in either pymongoor MongoDB shell syntax.
2.5AlargerdatasetNow it's time to play with a dataset of more realistic size! Try to insert a document into the restaurants collection. In addition,you can see the structure of documents in the collection.
3.IndexinginMongoDBIndexes support the efficient resolution of queries. Without indexes, MongoDB must scan every document of a collection to selectthose documents that match the query statement. Scan can be highly inefficient and require MongoDB to process a large volume ofdata.
Indexes are special data structures that store a small portion of the data set in an easy-to-traverse form. The index stores the valueof a specific field or set of fields, ordered by the value of the field as specified in the index.
MongoDB supports indexes that contain either a single field or multiple fields depending on the operations that this index typesupports.
By default, MongoDB creates the _id index, which is an ascending unique index on the _id field, for all collections when thecollection is created. You cannot remove the index on the _id field.
ManagingindexesinMongoDBAn explain() operator provides information on the query plan. It returns a document that describes the process and indexesused to return the query. This may provide useful insight when attempting to optimize a query. Example:
The number of documents examined is indicated in the docsExamined field. The number drops significantly by using an index. Infact, in this example the number of documents examined is exactly the number of documents returned ( nReturned ).
The index specification describes the kind of index for that field. For example, a value of 1 specifies an index that orders items inascending order. A value of -1 specifies an index that orders items in descending order. Notethatindexdirectiononlymattersinacompoundindex.
To remove all indexes, you can use db.collection.drop_indexes() . Example:
In [31]:
print("Before drop_indexes():")for index in db.restaurants.list_indexes(): pprint(index)print("Now we drop all indexes...")db.restaurants.drop_indexes()print("After drop_indexes():")for index in db.restaurants.list_indexes(): pprint(index)
To remove a specific index you can use db.collection.drop_index(index_name) . Example:
In [32]:
print('Create some indexes first...')db.restaurants.create_index([('cuisine', -1), ('borough', 1)]) index_name = db.restaurants.create_index('address.building')print('\nNow we have these indexes:')for index in db.restaurants.list_indexes(): pprint(index) print('\nThen drop_index()...')db.restaurants.drop_index(index_name)print('\nThe remaining indexes are:')for index in db.restaurants.list_indexes(): pprint(index)
