MySQL8Administrator'sGuide
Effectiveguidetoadministeringhigh-performanceMySQL8solutions
ChintanMehtaAnkitBhavsarHetalOzaSubhashShah
MySQL8Administrator'sGuideCopyright©2018PacktPublishingAllrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,or transmitted inanyformorbyanymeans,without thepriorwrittenpermissionof thepublisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the informationpresented. However, the information contained in this book is sold without warranty, either express orimplied.Neithertheauthors,norPacktPublishingoritsdealersanddistributors,willbeheldliableforanydamagescausedorallegedtohavebeencauseddirectlyorindirectlybythisbook.
PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthecompaniesandproductsmentionedinthisbookbytheappropriateuseofcapitals.However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.
CommissioningEditor:AmeyVarangaonkarAcquisitionEditor:AmanSinghContentDevelopmentEditor:AaryamansinghTechnicalEditor:DharmendraYadavCopyEditors:SafisEditingProjectCoordinator:ManthanPatelProofreader:SafisEditingIndexer:TejalDaruwaleSoniGraphics:TaniaDuttaProductionCoordinator:AparnaBhagatFirstpublished:February2018
Productionreference:1140218
PublishedbyPacktPublishingLtd.LiveryPlace35LiveryStreetBirminghamB32PB,UK.
ISBN978-1-78839-519-9
www.packtpub.com
mapt.io
Maptisanonlinedigital librarythatgivesyoufullaccess toover5,000booksand videos, as well as industry leading tools to help you plan your personaldevelopment and advance your career. For more information, please visit ourwebsite.
Whysubscribe?Spendless timelearningandmoretimecodingwithpracticaleBooksandVideosfromover4,000industryprofessionals
ImproveyourlearningwithSkillPlansbuiltespeciallyforyou
GetafreeeBookorvideoeverymonth
Maptisfullysearchable
Copyandpaste,print,andbookmarkcontent
PacktPub.comDidyouknow thatPacktofferseBookversionsofeverybookpublished,withPDFandePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandasaprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwithusatservice@packtpub.comformoredetails.
Atwww.PacktPub.com,youcanalsoreadacollectionoffree technicalarticles,signupforarangeoffreenewsletters,andreceiveexclusivediscountsandoffersonPacktbooksandeBooks.
AbouttheauthorsChintan Mehta is a cofounder of KNOWARTH Technologies(www.knowarth.com)andheadscloud/RIMS/DevOps.Hehas rich,progressiveexperienceinserveradministrationofLinux,AWScloud,DevOps,RIMS,andopensourcetechnologies.HeisanAWSCertifiedSolutionsArchitect.
He has authoredMySQL 8 for Big Data andHadoop Backup and RecoverySolutions, and has reviewed Liferay Portal Performance Best Practices andBuildingServerlessWebApplications.
I would like to thank my coauthors. I would especially like to thank mywonderfulwife,Mittal,andmysweetson,Devam, forputtingupwith the longdays,nights,andweekendswhenIwascampinginfrontofmylaptop.Lastbutnot least, I want to thank my mom, dad, friends, family, and colleagues forsupportingmethroughout.
AnkitBhavsarisaseniorconsultantleadingateamworkingonERPsolutionsat KNOWARTH Technologies. He received an MCA from North Gujaratuniversity. He has had dynamic roles in the development andmaintenance ofERP solutions and astrologyportalsContentManagement that includingOOP,technicalarchitectureanalysis,design,developmentaswellasdatabasedesign,development and enhancement process, data and objectmodeling, in order toprovidetechnicalandbusinesssolutionstoclients.
First, I would like to thank the coauthors, reviewers, the wonderful team atPacktPub, and Aaryaman for this effort. I would especially like to thank mywonderfulwife,Avani,forputtingupwiththelongdays,nights,andweekends.Last,butnotleast,Iwanttothankmymom,dad,friends,family,andcolleaguesforsupportingmethroughoutthewritingofthisbook.
HetalOza anMCA from a reputable institute of India, is working as a leadconsultantatKNOWARTHTechnologies.ShehasrichexperienceinJava-basedsystemswithvariousdatabases.Her10yearsofexperiencecoversallstagesof
softwaredevelopment.Shehasworkedondevelopmentofweb-basedsoftwaresolutions on various platforms. She has good exposure to integration projectswith web-service-based and thread-based architecture. Her knowledge is notboundtoanysinglefieldbecauseshehasworkedonwiderangeoftechnologiesandtools.
Itgavemeimmensepleasuretobeanauthorofthisbook.First,Iwouldliketothankmyhusband,Suhag,andmysweetson,Om,forputtingupwithmeduringthelongdays,nights,andweekendswhenIwascampinginfrontofmylaptop.Second, I would like to thank Chintan Mehta, who showed trust in me andprovidedthisopportunity,andKrupalKhatriforhissupport.IwouldalsoliketothanktheteamatPacktPubfortheirgreathelp.
SubhashShahworks as aprincipal consultant atKNOWARTHTechnologies.HeholdsadegreeininformationtechnologyfromaHNGU.Heisexperiencedin developing web-based solutions using various software platforms. He is astrongadvocateofopensourcesoftwaredevelopmentanditsusebybusinessesto reduce risks and costs.His interests include designing sustainable softwaresolutions.Histechnicalskillsincluderequirementanalysis,architecturedesign,projectdelivery,applicationsetup,andexecutionprocesses.Heisanadmirerofqualitycodeandtest-drivendevelopment.
Iwouldliketothankmyfamilyforsupportingmethroughoutthecourseofthisbook. Itwould have been difficultwithout them being a source of inspiration.ThankstoPacktPublishing,especiallyAaryaman,fortheirsmoothcoordinationand support.Thanks to fellowauthors for being around all the time, for theirdedicationandcommitment.Lastbutnot least, thanks tomycolleagues forallthesupporttheyhaveprovided.
AboutthereviewersSahaj Pathak has been involved with backend technologies such as Java,Spring,Hibernate,anddatabases(MySQL,PostgreSQL,Oracle,andothers).Hisexperience also spans frontend technologies (HTML4/5, jQuery, AngularJS,Node.Js,JavaScript,andCSS2/3).Hehasspeedyversatilitywithanytechnologyandasharpdesireforconsistentchange.
HeworksatKNOWARTHTechnologiesasasoftwareconsultantwherehedealswithbigenterprise-product-basedprojects.
Ravi Shah is a highly versatile IT professional with more than 5 years ofexperience of handling high-end IT projects, with competencies inconceptualizingandsupportingcriticalITframeworksandapplications.Heisateam player, a software engineer with a can-do attitude, and possessesphenomenal timemanagement skills, and stronguser focus.Hehas developedseveralwebapplicationsandmainlyspecializesinhealthcareandinsurance.
He is skilled in all phases of software development, an expert in translatingbusinessrequirementsintotechnicalsolutions,anddevotedtoquality,usability,security and scalability. His expertise mainly includes Liferay, Java, Spring,Struts,Hibernate,MySQL,Lucene,Angular,andAgile.
He is a good trainer delivering training on J2EE and theLiferay portal in hisorganization.
IwouldliketotakethisopportunitytoexpressheartfeltthankstoKNOWARTHTechnologies and Packt Publishing for givingme this opportunity. Also, I amverythankfultomyparentsforalwayssupportingmeinallpossibleways.
Packt is searching for authorslikeyouIfyou'reinterestedinbecominganauthorforPackt,pleasevisitauthors.packtpub.com and apply today. We have worked with thousands of developers and techprofessionals,justlikeyou,tohelpthemsharetheirinsightwiththeglobaltechcommunity.Youcanmakeageneralapplication,apply foraspecifichot topicthatwearerecruitinganauthorfor,orsubmityourownidea.
TableofContents
Preface
Whothisbookisfor
Whatthisbookcovers
Togetthemostoutofthisbook
Downloadtheexamplecodefiles
Conventionsused
Getintouch
Reviews
1. AnIntroductiontoMYSQL8
OverviewofMySQL
MySQLasarelationaldatabasemanagementsystem
LicenserequirementsofMySQL8
Reliabilityandscalability
Platformcompatibility
Releases
CorefeaturesinMySQL
Structureddatabase
Databasestorageenginesandtypes
OverviewofInnoDB
OverviewofMyISAM
Overviewofmemory
Overviewofarchive
OverviewofBLACKHOLEasastorageengine
OverviewofCSV
Overviewofmerge
Overviewoffederated
OverviewoftheNDBcluster
ImprovedfeaturesinMySQL8
Transactionaldatadictionary
Roles
InnoDBautoincrement
Invisibleindexes
Improvingdescendingindexes
TheSETPERSISTvariant
ExpandedGISsupport
Defaultcharacterset
Extendedbit-wiseoperations
InnoDBMemcached
NOWAITandSKIPLOCKED
JSON
Cloud
Resourcemanagement
BenefitsofusingMySQL8
Security
Scalability
Anopensourcerelationaldatabasemanagementsystem
Highperformance
Highavailability
Cross-platformcapabilities
LimitationsofMySQL8
Numberoftablesordatabases
Tablesize
Joins
Windowsplatform
Tablecolumncount
Rowsize
InnoDBstorageengine
LimitationsofInnoDBstorageengine
Restrictions
Datadictionary
LimitationsofgroupreplicationinMySQL8
Limitationsofpartitioning
Constructsprohibition
Operators
Tables
UsecasesofMySQL
Socialmedia
Government
Mediaandentertainment
Frauddetection
Businessmapping
E-commerce
Summary
2. InstallingandUpgradingMySQL8
TheMySQL8installationprocess
Generalinstallationguide
DownloadingMySQL8
Verifyingthepackageintegrity
UsingMD5checksums
Usingcryptographicsignatures
InstallingMySQL8onMicrosoftWindows
Windows-specificconsiderations
MySQL8installationlayout
Choosingtherightinstallationpackage
TheMySQL8installer
Initialsetupinformation
Installationworkflow
InnoDBclustersandboxtestsetup
Serverconfiguration
MySQLinstallerproductcataloganddashboard
MySQLinstallerconsole
MySQL8installationusingaZIPfile
InstallingMySQL8onLinux
InstallationusingtheYumrepository
InstallationusingtheRPMpackage
InstallationusingtheDebianpackage
Post-installationsetupforMySQL8
Datadirectoryinitialization
SecuringtheinitialMySQLaccount
StartingandtroubleshootingMySQL8services
Executingcommandstotesttheserver
UpgradingMySQL8
Upgradingmethods
In-placeupgradeofMySQL
LogicalupgradeforMySQL8
UpgradingprerequisitesforMySQL5.7
MySQL8downgrading
Downgradingmethods
Logicaldowngrade
Manualchangesrequiredbeforedowngrading
Summary
3. MySQL8–UsingProgramsandUtilities
OverviewofMySQL8programs
MySQLprogramsinbrief
Startupprograms
Installation/upgradationprograms
Clientprograms
Administrativeandutilitiesprograms
Environmentvariables
MySQLGUItool
MySQL8command-lineprograms
Executingprogramsfromthecommandline
ExecutingMySQLprograms
ConnectingtotheMySQLserver
Specifyingoptionsforprograms
Optionsonthecommandline
Modifyingprogramoptions
Modifyingoptionswithfiles
group
opt_name
opt_name=value
Includedirectives
Command-lineoptionsaffectingoptionfilehandling
Settingprogramvariableswithoptions
Settingenvironmentvariables
Serverandserver-startupprograms
mysqld-theMySQLserverprogram
Options
mysqld_safe-MySQLserverstartupscript
mysql.server-MySQLServerstartupscript
mysqld_multi-managingmultipleMySQLservers
Installationprograms
comp_err-compilingtheMySQLerrormsgfile
mysql_secure_installation-improvingMySQLinstallationsecurity
mysql_ssl_rsa_setup-creatingSSL/RSAfiles
mysql_tzinfo_to_sql-loadingthetimezonetables
mysql_upgrade-checkingandupgradingMySQLtables
MySQL8clientprograms
mysql-thecommand-linetool
mysqloptions
mysqlcommands
help[arg],\h[arg],\?[arg],?[arg]
charsetcharset_name,\Ccharset_name
clear,\c
connect[db_namehost_name],\r[db_namehost_name]
edit,\e
exit,\q
prompt[str],\R[str]
quit,\q
status,\s
usedb_name,\udb_name
mysqllogging
mysqlserver-sidehelp
Executingsqlfromtextfiles
mysqladmin-clientforadministeringaMySQLserver
mysqlcheck-atablemaintenanceprogram
mysqldump-adatabasebackupprogram
Performanceandscalability
mysqlimport-adataimportprogram
mysqlpump-adatabasebackupprogram
mysqlsh-theMySQLShell
mysqlshow-showingdatabase,table,andcolumninformation
mysqlslap-loademulationclient
MySQL8administrativeprograms
ibdsdi-InnoDBtablespaceSDIextractionutility
innochecksum-offlineInnoDBfilechecksumutility
myisam_ftdump-displayingfull-textindexutility
myisamchk-MyISAMtable-maintenanceutility
myisamlog-displayingMyISAMlogfilecontent
myisampack-generatingcompressed,read-onlyMyISAMtables
mysql_config_editor-MySQLconfigurationutility
mysqlbinlog-utilityforprocessingbinarylogfiles
mysqldumpslow-summarizingslowquerylogfiles.
MySQL8environmentvariables
MySQLGUItools
MySQLWorkbench
MySQLNotifier
MySQLNotifierusage
Summary
4. MySQL8DataTypes
OverviewofMySQL8datatypes
Numericdatatypes
Integertypes
Fixedpointtypes
Floatingpointtypes
Problemswithfloatingpointvalues
Bitvaluetype
Bitvalueliterals
PracticalusesofBIT
Typeattributes
Overflowhandling
Dateandtimedatatypes
DATE,DATETIME,andTIMESTAMPtypes
MySQLDATETIMEfunctions
TIMEtype
Timefunctions
YEARtype
MigratingYEAR(2)toYEAR(4)
Stringdatatypes
CHARandVARCHARdatatypes
BINARYandVARBINARYdatatypes
BLOBandTEXTdatatypes
ENUMdatatype
SETdatatype
JSONdatatype
PartialupdatesofJSONvalues
Storagerequirementsfordatatypes
Choosingtherightdatatypeforcolumn
Summary
5. MySQL8DatabaseManagement
MySQL8serveradministration
Serveroptionsanddifferenttypesofvariables
ServerSQLmodes
SettingtheSQLmode
TheavailableSQLmodes
CombinationSQLmodes
StrictSQLmode
TheIGNOREkeyword
IPv6support
Serversidehelp
Theservershutdownprocess
Datadirectory
Thesystemdatabase
Datadictionarytables
Grantsystemtables
Objectinformationsystemtables
Logsystemtables
Theserver-sidehelpsystemtables
Timezonesystemtables
Replicationsystemtables
Optimizersystemtables
Othermiscellaneoussystemtables
Runningmultipleinstancesonasinglemachine
Settingupmultipledatadirectories
RunningmultipleMySQLinstancesonWindows
Componentsandpluginmanagement
MySQL8serverplugins
Installingtheplugins
Activateplugin
Uninstallplugin
Gettinginformationabouttheinstalledplugins
Rolesandpermissions
Cachingtechniques
Globalization
Charactersets
Charactersetsupport
Addingthecharacterset
Configuringthecharactersets
Languageselection
TimezonesettingsforMySQL8
Localesupport
MySQL8serverlogs
Theerrorlog
Componentconfiguration
Defaulterrorlogdestinationconfiguration
DefaulterrorlogdestinationonWindows
DefaulterrorlogdestinationonUnixandUnix-Likesystems
Thegeneralquerylog
Thebinarylog
Theslowquerylog
TheDDLlog
Serverlogmaintenance
Summary
6. MySQL8StorageEngines
Overviewofstorageengines
MySQLstorageenginearchitecture
Severaltypesofstorageengine
OverviewoftheInnoDBstorageengine
Customstorageengine
Severaltypesofstorageengines
Pluggablestorageenginearchitecture
Thecommondatabaseserverlayer
Settingthestorageengine
TheMyISAMstorageengine
TheMEMORYstorageengine
TheCSVstorageengine
TheARCHIVEstorageengine
TheBLACKHOLEstorageengine
TheMERGEstorageengine
TheFEDERATEDstorageengine
TheEXAMPLEstorageengine
TheInnoDBstorageengine
ACIDmodel
Multiversioning
Architecture
Lockingandtransactionmodel
Configuration
Tablespaces
Tablesandindexes
INFORMATION_SCHEMAtables
Memcachedplugin
Creatingacustomstorageengine
Creatingstorageenginesourcefiles
Addingengine-specificvariablesandparameters
Creatingthehandlerton
Handlinghandlerinstallation
Definingfilenameextensions
Creatingtables
Openingatable
Implementingbasictablescanning
Closingatable
Referenceforadvancedcustomstorageengine
Summary
7. IndexinginMySQL8
Anoverviewonindexing
UsesofindexesinMySQL8
SQLcommandsrelatedtoindexes
CreatinganINDEXcommand
Spatialindexcharacteristics
Non-spatialindexcharacteristics
Dropindexcommand
SPATIALindexcreationandoptimization
InnoDBandMyISAMindexstatisticscollection
Column-levelindexing
Columnindexes
Indexprefixes
FULLTEXTindexes
SpatialIndexes
IndexesintheMEMORYstorageengine
Multiple-columnindexes
B-Treeindex
Hashindex
Indexextension
Usinganoptimizerforindexes
Invisibleanddescendingindexes
Invisibleindex
Descendingindex
Summary
8. ReplicationinMySQL8
Overviewofreplication
WhatisMySQLreplication?
AdvantagesofMySQLreplication
Configuringreplication
Binarylogfilebasedreplication
Replicationmasterconfiguration
REPLICATIONSLAVEconfiguration
Addingslavestoreplication
Globaltransactionidentifiersbasedreplication
MySQLmulti-sourcereplication
Replicationadministrationtasks
Implementingreplication
Replicationformats
Statement-basedversusrow-basedreplication
Replicationimplementationdetails
Replicationchannels
Replicationrelayandstatuslogs
Evaluatingreplicationfilteringrules
Groupreplication
Primary-secondaryreplicationversusgroupreplication
Groupreplicationconfiguration
Groupreplicationusecases
Replicationsolutions
Summary
9. PartitioninginMySQL8
Overviewofpartitioning
Typesofpartitioning
Partitioningmanagement
Partitionselectionandpruning
Restrictionsandlimitationsinpartitioning
Typesofpartitioning
RANGEpartitioning
LISTpartitioning
COLUMNSpartitioning
RANGECOLUMNpartitioning
LISTCOLUMNpartitioning
HASHpartitioning
LINEARHASHpartitioning
KEYpartitioning
Subpartitioning
HandlingNULLinpartitioning
Partitionmanagement
RANGEandLISTpartitionmanagement
HASHandKEYpartitionmanagement
Partitionmaintenance
Obtainpartitioninformation
Partitionselectionandpruning
Partitionpruning
Partitionselection
Restrictionsandlimitationsinpartitioning
Partitioningkeys,primarykeys,anduniquekeys
Partitioninglimitationsrelatingtostorageengines
Partitioninglimitationsrelatingtofunctions
Summary
10. MySQL8–ScalabilityandHighAvailability
OverviewofscalabilityandhighavailabilityinMySQL8
MySQLreplication
MySQLcluster
OracleMySQLcloudservice
MySQLwiththeSolariscluster
ScalingMySQL8
Scalingusingcluster
Clientnode
Applicationnode
Managementnode
Datanode
Datastorageandmanagementofdisk-basedandin-memorydata
Automaticanduser-definedpartitioningoftablesorsharding
oftables
Synchronousdatareplicationbetweendatanodes
Dataretrievalandtransactions
Automaticfailover
Automaticre-synchronizationforself-healingafterfailure
ScalingusingmemcachedinMySQL8
NoSQLAPIs
Scalingusingreplication
Singleserverdependancy
Performance
Backupandrecovery
Loaddistribution
Asynchronousdatareplication
Geographicaldatadistribution
GTIDreplication
ZFSreplication
ChallengesinscalingMySQL8
Businesstypeandflexibility
Understandserverworkload
Read-writeoperationlimit
Maintenance
Masterserverfailure
Synchronization
Databasesecurity
Crossnodetransaction
Growingteamfordevelopment
Managechangerequest
Scale-upandscale-out
Achievinghighavailability
Purposeofhighavailability
Dataavailability
Securityofdata
Synchronizationofdata
Backupofthedata
Competitivemarket
Performance
Updatesinthesystem
Choosingthesolution
Advantagesofhighavailability
Summary
11. MySQL8–Security
OverviewofsecurityforMySQL8
Commonsecurityissues
Generalguidelines
Guidelinesforasecurepassword
Guidelinesforendusers
Guidelinesforadministrators
Passwordandlogging
SecureMYSQL8againstattackers
SecurityoptionsandvariablesprovidedbyMySQL8
Securityguidelinesforclientprogramming
AccesscontrolinMySQL8
PrivilegesprovidedbyMySQL8
Granttables
Verificationofaccesscontrolstages
Stage1-Connectionverification
Stage2-Requestverification
AccountmanagementinMySQL8
Addandremoveuseraccounts
Securityusingroles
SETROLE
CREATEROLE
DROPROLE
GRANT
REVOKE
SETDEFAULTROLE
SHOWGRANTS
Passwordmanagement
EncryptioninMySQL8
ConfiguringMySQL8touseencryptedconnections
Server-sideconfigurationforencryptedconnections
Client-sideconfigurationforencryptedconnections
Commandoptionsforencryptedconnections
ConnectwithMySQL8remotelyfromWindowswithSSH
Securityplugins
Authenticationplugins
SHA-2pluggableauthentication
Client-sidecleartextpluggableauthentication
No-loginpluggableauthentication
Socketpeer-credentialpluggableauthentication
Testpluggableauthentication
Theconnection-controlplugins
CONNECTION_CONTROL
Plugininstallation
VariablesrelatedtoCONNECTION-CONTROL
Thepasswordvalidationplugin
Installpasswordvalidationplugin
Variablesandoptionsrelatedtothepasswordvalidationplugin
MySQL8keyring
Installkeyringplugin
Systemvariablesrelatedtokeyringplugin
Summary
12. OptimizingMySQL8
OverviewofMySQL8optimization
Optimizingthedatabase
Optimizingthehardware
OptimizingMySQL8serversandclients
OptimizingdiskI/O
UsingNFSwithMySQL
Optimizingtheuseofmemory
Optimizinguseofthenetwork
Optimizinglockingoperations
Performancebenchmarking
Examiningthreadinformation
Optimizingdatabasestructure
Optimizingdatasize
Tablecolumns
Rowformat
Indexes
Joins
Normalization
OptimizingMySQLdatatypes
Optimizingformanytables
UseofaninternaltemporarytableinMySQL
Optimizingqueries
OptimizingSQLstatements
Optimizingindexes
Queryexecutionplan
Optimizingtables
OptimizationforInnoDBtables
OptimizationforMyISAMtables
OptimizationforMEMORYtables
Leveragingbufferingandcaching
InnoDBbufferpooloptimization
MyISAMkeycache
Summary
13. ExtendingMySQL8
AnoverviewofextendingMySQL8
MySQL8internals
MySQL8pluginAPI
MySQL8servicesforcomponentsandplugins
AddingnewfunctionstoMySQL8
DebuggingandportingMySQL8
Extendingpluginsandusingservicestocallthem
Writingplugins
Componentandpluginservices
Thelockingservice
Thekeyringservice
Addingnewfunctions
Featuresofauser-definedfunctioninterface
Addinganewuser-definedfunction
Addinganewnativefunction
Debuggingandporting
DebuggingMySQLserver
DebuggingMySQLclient
TheDBUGpackage
Summary
14. MySQL8BestPracticesandBenchmarking
MySQLbenchmarkingandtools
Resourceutilization
Stretchingyourbenchmarkingtimelines
Replicatingproductionsettings
Consistencyofthroughputandlatency
Sysbenchcandomore
Virtualizationworld
Concurrency
Hiddenworkloads
Nervesofyourquery
Benchmarks
Bestpracticesformemcached
Resourceallocation
Operatingsystemarchitecture
Defaultconfigurations
Maxobjectsize
Backlogqueuelimit
Largepagessupport
Sensitivedata
Restrictingexposure
Failover
Namespaces
Cachingmechanism
Memcachedgeneralstatistics
Bestpracticesforreplication
Throughputingroupreplication
Infrastructuresizing
Constantthroughput
Contradictoryworkloads
Writescalability
Bestpracticesfordatapartitioning
Horizontalpartitioning
Verticalpartitioning
PruningpartitionsinMySQL
Bestpracticesforqueriesandindexing
Datatypes
Notnull
Indexing
Searchfieldsindex
Datatypesandjoins
Compoundindex
Shorteningupprimarykeys
Indexingeverything
Fetchingalldata
Lettingtheapplicationdothejob
Existenceofdata
Limitingyourself
Analyzingslowqueries
Querycost
Summary
15. TroubleshootingMySQL8
MySQL8commonproblems
MostcommonMySQLerrors
Accessdenied
Can'tconnectto[local]MySQLserver
LostconnectiontoMySQLserver
Passwordfailswhenenteredincorrectly
Hosthost_nameisblocked
Toomanyconnections
Outofmemory
Packettoolarge
Thetableisfull
Can'tcreate/writetofile
Commandsoutofsync
Ignoringuser
Tabletbl_namedoesn'texist
MySQL8servererrors
Issueswithfilepermissions
Resettingtherootpassword
MySQLcrashesprevention
HandlingMySQLfulldisk
MySQLtemporaryfilesstorage
MySQLUnixsocketfile
Timezoneproblems
MySQL8clienterrors
Casesensitivityinstringsearches
ProblemswithDATEcolumns
ProblemswithNULLvalues
MySQL8troubleshootingapproach
Analyzingqueries
Real-worldscenario
Summary
OtherBooksYouMayEnjoy
Leaveareview-letotherreadersknowwhatyouthink
Preface
For any system, it ismust tomanagedata in anorganizedmanner. In a large-scale system, it is necessary to handle various configurations for securitypurposes.MySQL
is one of the popular solutions used to handle enterprise-level applications. Inthisbook,wewillexplainhowtoconfigureusers,theirroles,multipleinstances,andmuchmore.
ManyorganizationsuseMySQLfortheirwebsitesorcommercialproducts,andit's very challenging for them to manage data storage and analyze data inaccordance with the business requirements. This book will show you how toimplementindexingandqueryoptimizationforbetterperformance.Alongwiththis,we'llcoverhowscalabilityandhighavailabilityoftheMySQLservercanhelptomanagefailurescenarios.Inadditiontothat,replicationandpartitioningconceptsareexplainedindetailwithexamples.
ThebookdescribesvariousfeaturesofMySQL8bytargetingdifferentlevelsofusers, from beginners to database administrators. This book starts from theinstallationwithabasicunderstandingofMySQL8'sconcepts.Theweproceedtoadministrative-levelfeatureswithconfiguration.Attheendofthebook,you
will have learned about very interesting functionalities, such as optimization,extension,andtroubleshooting.
WhothisbookisforThisbook is intended forMySQLadministratorswhoare looking for ahandyguide covering all theMySQL administration-related tasks. If you are aDBAlookingtogetstartedwithMySQLadministration,thisbookwillalsohelpyou.Knowledgeofbasicdatabaseconceptsisrequiredtogetstartedwiththisbook.
WhatthisbookcoversChapter 1, An Introduction to MySQL 8, serves as an introductory guide toMySQL 8. It briefly defines the core features available inMySQL and newlyintroducedorenhancedfeaturesofMySQL8.Inthelaterpartofthechapter,wehighlightthebenefitsofMySQL8alongwithreal-worldapplications.
Chapter 2, Installing and Upgrading MySQL 8, describes detailed steps forinstallingMySQL8ondifferentplatforms.ItalsoexplainshowtoupgradetoordowngradefromMySQL8.
Chapter3,MySQL 8 –Using Programs andUtilities, introduces command-lineprogramsfortheMySQL8serverandclient.ItalsoprovidesinformationontheavailableGUItoolswithitsconfiguration.
Chapter4,MySQL8DataTypes, focusesonadetailedexplanationofMySQL8data types. It also explains data type categorization based on the types ofcontent.We cover data types alongwith their properties in each category.Wealsocoverstoragerequirementsfordatatypes.
Chapter5,MySQL8DatabaseManagement,mainlyexplores theadministrationpart of MySQL 8. This chapter covers components and plugin management,along with user and role management. In addition, it explains globalizationconfiguration, caching techniques, and different types of logs available inMySQL8.
Chapter6,MySQL8StorageEngines, explains several types of storage enginesanddetailsoftheInnoDBstorageengine.Thischapterprovidesinformationoncustom storage engine creation, along with steps to make it pluggable ininstalledMySQL8.
Chapter7,IndexinginMySQL8,explainsindexing,alongwiththepossiblewaysofimplementingit.Itcomparestypesofindexing.
Chapter8,ReplicationinMySQL8,explainsreplicationandthedifferenttypesofreplication available in MySQL 8. It also describes the configuration and
implementationofreplicationalongwithdifferentapproaches.
Chapter 9, Partitioning in MySQL 8, explains the setting of several types ofpartitioning,selection,andpruningofpartitioning.Italsoexplainshowtocopeupwithrestrictionsandlimitationswhilepartitioning.
Chapter 10,MySQL 8 – Scalability and High Availability, explains how to doscaling and how to handle different challenges during implementation. Thereader gets an understanding of diverse ways to achieve high availability inMySQL8.
Chapter 11,MySQL 8 – Security, focuses onMySQL 8 database security. Thischaptercoversgeneralfactorsthataffectsecurity,thesecurityofcoreMySQL8files, accesscontrol, and securing thedatabase system itself.This chapter alsoincludesdetailsofsecurityplugins.
Chapter12,OptimizingMySQL8,explainshowtoconfigureMySQL8forbetterperformance. This chapter also describes use cases with a few performanceresultstovalidate.ThiswillhelpyouknowvarioustouchpointstolookoutforwhendealingwithoptimizingMySQL8.
Chapter13,ExtendingMySQL 8, shows how to extendMySQL 8 and add newfunctions,alongwithdebuggingandportingtoMySQL8.
Chapter 14, MySQL 8 Best Practices and Benchmarking, explains the bestpractices of using MySQL. It also explains various benchmarkings done forMySQL8.
Chapter 15,TroubleshootingMySQL 8, explains many common and real-worldscenariosoftroubleshootingforMySQL8.
TogetthemostoutofthisbookWerecommendthatyougetsomebasicknowledgeofMySQL(anyversion)andSQLcommandsbeforeyoustartreadingthisbook.
Thisbookalsocoverspracticalscenariosandcommandexecution,soifpossible,installatoolforeasyexecutionofMySQLcommands.
Downloadtheexamplecodefiles
Youcandownloadtheexamplecodefilesforthisbookfromyouraccountatwww.packtpub.com.Ifyoupurchasedthisbookelsewhere,youcanvisitwww.packtpub.com/supportandregistertohavethefilesemaileddirectlytoyou.
Youcandownloadthecodefilesbyfollowingthesesteps:
1. Loginorregisteratwww.packtpub.com.
2. SelecttheSUPPORTtab.
3. ClickonCodeDownloads&Errata.
4. Enter the name of the book in the Search box and follow the onscreeninstructions.
Once the file is downloaded, please make sure that you unzip or extract thefolderusingthelatestversionof:
WinRAR/7-ZipforWindows
Zipeg/iZip/UnRarXforMac
7-Zip/PeaZipforLinux
ThecodebundleforthebookisalsohostedonGitHubathttps://github.com/PacktPublishing/MySQL-8-Administrators-Guide.Wealsohaveothercodebundlesfromourrichcatalogofbooksandvideosavailableathttps://github.com/PacktPublishing/.Checkthemout!
ConventionsusedThereareanumberoftextconventionsusedthroughoutthisbook.
CodeInText: Indicates code words in text, database table names, folder names,filenames, file extensions, pathnames, dummy URLs, user input, and Twitterhandles.Hereisanexample:"ItwilldownloadwinMD5Sum.exeontoyourcomputer."
Anycommand-lineinputoroutputiswrittenasfollows:
CREATETABLEworking_days(
yearINT,
weekINT,
daysBIT(7),
PRIMARYKEY(year,week));
Bold:Indicatesanewterm,animportantword,orwordsthatyouseeonscreen.Forexample,wordsinmenusordialogboxesappearinthetextlikethis.Hereisanexample:"ClickontheDownloadWinMD5Sumoptiononthepage."
Warningsorimportantnotesappearlikethis.
Tipsandtricksappearlikethis.
Getintouch
Feedbackfromourreadersisalwayswelcome.
General feedback:[email protected] thebook title in thesubjectofyourmessage.Ifyouhavequestions
aboutanyaspectofthisbook,[email protected].
Errata: Although we have taken every care to ensure the accuracy of ourcontent,mistakes
dohappen.Ifyouhavefoundamistakeinthisbook,wewouldbegratefulifyouwould
report this to us. Please visit www.packtpub.com/submit-errata, selecting your book,clickingontheErrataSubmissionFormlink,andenteringthe
details.
Piracy: IfyoucomeacrossanyillegalcopiesofourworksinanyformontheInternet,
we would be grateful if you would provide us with the location address orwebsite
name.Pleasecontactusatcopyright@packtpub.comwithalinktothematerial.
Ifyouareinterestedinbecominganauthor:Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineither
writingorcontributingtoabook,pleasevisitauthors.packtpub.com.
ReviewsPleaseleaveareview.Onceyouhavereadandusedthisbook,whynotleaveareviewonthesitethatyoupurchaseditfrom?Potentialreaderscanthenseeanduse your unbiased opinion to make purchase decisions, we at Packt canunderstand what you think about our products, and our authors can see yourfeedbackontheirbook.Thankyou!
FormoreinformationaboutPackt,pleasevisitpacktpub.com.
AnIntroductiontoMYSQL8
MySQL is a well-known open source structured database because of itsperformance,easinesstouse,andreliability.Thisisthemostcommonchoiceofweb applications for a relational database. In the currentmarket, thousands ofweb-based applications rely on MySQL including giant industries such asFacebook,Twitter,andWikipedia.
Ithasalsoproven tobe thedatabasechoiceforSoftwareasaService (SaaS)based applications such as Twitter, YouTube, SugarCRM, Supply Dynamics,Workday,RightNow,Omniture,Zimbra,andmanymore.Wewilldiscussthisindetail in the use cases of MySQL section later in the chapter. MySQL wasdeveloped byMySQLAB, a Swedish company, and now it is distributed andsupportedbyOracleCorporation.MySQLcarriesavaluablehistorywithit.
MySQL has continued to improve in order to become an enterprise-leveldatabasemanagement system.MySQL 8 is expected to be a game-changer astodayweareintheageofdigitization.
MySQL8isall tunedtoservemanynewusecases that inpriorversionsweredifficult to achieve. Some of the use cases an enormous amount of data isproduced are social networking, e-commerce, bank/credit card transactions,
emails, data stored on the cloud, and so on. Analysis of all such structured,unstructured, or semi-structured ubiquitous data helps to discover hiddenpatterns,markettrends,correlations,personalpreferences.
"Thereissomuchforeachofus"
- JamesTruslowAdamsLet's takean in-depth lookatMySQL8new features,benefits, use cases alongwitha few limitationsofMySQL8afterwehaveanoverviewofMySQL.Thisisgoingtobeexciting,let'sgetprepared.
OverviewofMySQL
Structured Query Language (SQL) is used to manipulate, retrieve, insert,update,anddeletedatainrelationaldatabasemanagementsystem(RDBMS).Tomakeitsimpler,SQLtellsthedatabasewhattodoandexactlywhatitneeds.SQL is a standard language that all RDBMS systems such as MySQL, MSAccess,MSSQL,Oracle,Postgres,andothersuse.
RDBMSisthebasisforSQLandforallmoderndatabasesystemssuchasMSSQLServer,IBMDB2,Oracle,MySQL,andMicrosoftAccess.
SQL allows users to access data fromMySQL and define andmanipulate thedata. To embed within other languages, you can leverage SQL modules,libraries,andprecompilers,whichcanhelpyoucreate/dropdatabasesandtables,allowuserstocreatetheview,andstoredprocedures,functions,andsoon,inadatabase. It can do various other operations such as allowing users to setpermissionsontables,procedures,andviews.
MySQL as a relational databasemanagementsystem
Datainarelationaldatabaseisstoredinanorganizedformatsothatinformationcanberetrievedeasily.Datawillbestoredindifferent tablesmadeupofrowsand columns. However, the relationship can also be built between differenttables thatefficientlystorehugedataandeffectively retrieve theselecteddata.Thisprovidesdatabaseoperationswithtremendousspeedandflexibility.
Asarelationaldatabase,MySQLhascapabilitiestoestablishrelationshipswithdifferenttablessuchasonetomany,manytoone,andonetoonebyprovidingprimarykeys,foreignkeys,andindexes.Itcanalsoperformjoinsbetweentablestoretrieveexactinformationsuchasinnerjoinsandouterjoins.
SQLisusedasaninterfacetointeractwiththerelationaldatainMySQL.SQLisanAmericanNationalStandardInstitute(ANSI)standardlanguagewhichwecanoperatewithdatasuchascreation,deletion,updating,andretrieval.
LicenserequirementsofMySQL8
Many industries prefer open source technology because of the technology'sflexibility and cost-saving features, whileMySQL has put its footprint in themarketbybecomingthemostpopularrelationaldatabaseforwebapplications.Open sourcemeans that you canview the sourceofMySQLand customize itbasedonyourneedswithoutanycost.Youcandownload thesourceorbinaryfilesfromitssiteandusethemaccordingly.
TheMySQLserveriscoveredundertheGeneralPublicLicense(GNU),whichmeansthatwecanfreelyuseitforwebapplications,studyitssourcecode,andmodify it to suit our needs. It also has the Enterprise Edition as well withadvancedfeaturesincluded.ManyenterprisesstillpurchasethesupportcontractfromMySQL
togetassistanceonvariousissues.
ReliabilityandscalabilityMySQL has great reliability to perform well without requiring extensivetroubleshooting due to bottlenecks or other slowdowns. It also incorporates anumber of performance enhanced mechanisms such as index support, loadutilities,andmemorycaches.MySQL
uses InnoDB as a storage engine, which provides highly efficient ACIDcomplianttransactionalcapabilitiesthatassurehighperformanceandscalability.To handle the rapidly growing database,MySQLReplication and cluster helpscaleoutthedatabase.
PlatformcompatibilityMySQL has great cross-platform availability thatmakes itmore popular. It isflexible to run on major platforms such as RedHat, Fedora, Ubuntu, Debian,Solaris,MicrosoftWindows, and ApplemacOS. It also providesApplicationProgramming Interface (APIs) to interconnect with various programminglanguagessuchasC,C++,C#,PHP,Java,Ruby,Python,andPerl.
Releases
HereisalistofmajorreleasesofMySQLsofar:
Version5.0GAwasreleasedon19thOctober,2005
Version5.1GAwasreleasedon14thNovember,2008
Version5.5GAwasreleasedon3rdDecember,2010
Version5.6GAwasreleasedon5thFebruary,2013
Version5.7GAwasreleasedon21stOctober,2015
Nowit'stimeforthemajorversionrelease--MySQL8--whichwasannouncedon12thSeptember,2016andisstillinthedevelopmentmilestonemode.
CorefeaturesinMySQLLet'slookbackandquicklyglancethroughsomeofthecorefeaturesinMySQL.We will be discussing various features throughout the book in detail as weprogress.
Structureddatabase
Structured databases are traditional databases that have been used by manyenterprisesformorethan40years.However,inthemodernworld,datavolumeis becoming bigger and bigger and a common need has taken its place--dataanalytics. Analytics is becoming difficult with structured databases as thevolumeandvelocityofdigitaldatagrowfasterbytheday;weneedtofindawayto achieve such needs in an effective and efficient way. The most commondatabase that is used as a structured database in the open source world isMySQL.
Manyorganizationsuseastructureddatabasetostoretheirdatainanorganizedwaywiththeformattedrepository.Basically,datainastructureddatabasehasafixedfield,apredefineddatalength,anddefineswhatkindofdataistobestoredsuch as numbers, dates, time, addresses, currencies, and so on. In short, thestructureisalreadydefinedbeforedatagetsinserted,whichgivesaclearerideaofwhatdatacanresidethere.Thekeyadvantageofusingastructureddatabaseisthatdatabeingeasilystored,queried,andanalyzed.
Anunstructureddatabase is theopposite of this; it has no identifiable internalstructure. It can have a massive unorganized agglomerate or various objects.Mainly, the source of structured data is machine-generated, which means
information is generated from the machine and without human intervention,whereasunstructureddataishuman-generateddata.Organizationsusestructureddatabases for data such as ATM transactions, airline reservations, inventorysystems,andsoon.Inthesameway,someorganizationsuseunstructureddatasuch as emails, multimedia content, word processing documents, web pages,businessdocuments,andsoon.
<strong>mysql>SHOWTABLESTATUSLIKE'admin_user'\G;</strong><br/><strong>***************************1.row***************************</strong><br/><strong>Name:admin_user</strong><br/><strong>Engine:InnoDB</strong><br/><strong>Version:10</strong><br/><strong>Row_format:Dynamic</strong><br/><strong>Rows:2</strong><br/><strong>Avg_row_length:8192</strong><br/><strong>Data_length:16384</strong><br/><strong>Max_data_length:0</strong><br/><strong>Index_length:16384</strong><br/><strong>Data_free:0</strong><br/><strong>Auto_increment:3</strong><br/><strong>Create_time:2017-06-1914:46:49</strong><br/><strong>Update_time:2017-06-1915:15:08</strong><br/><strong>Check_time:NULL</strong><br/><strong>Collation:utf8_general_ci</strong><br/><strong>Checksum:NULL</strong><br/><strong>Create_options:</strong><br/><strong>Comment:AdminUserTable</strong><br/><strong>1rowinset(0.00sec)</strong>
<strong>mysql>SHOWENGINES\G;</strong><br/><strong>***************************1.row***************************</strong><br/><strong>Engine:InnoDB</strong><br/><strong>Support:YES</strong><br/><strong>Comment:Supportstransactions,row-levellocking,andforeignkeys</strong><br/><strong>Transactions:YES</strong><br/><strong>XA:YES</strong><br/><strong>Savepoints:YES</strong><br/><strong>***************************2.row***************************</strong><br/><strong>Engine:MRG_MYISAM</strong><br/><strong>Support:YES</strong><br/><strong>Comment:CollectionofidenticalMyISAMtables</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************3.row
***************************</strong><br/><strong>Engine:MEMORY</strong><br/><strong>Support:YES</strong><br/><strong>Comment:Hashbased,storedinmemory,usefulfortemporarytables</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************4.row***************************</strong><br/><strong>Engine:BLACKHOLE</strong><br/><strong>Support:YES</strong><br/><strong>Comment:/dev/nullstorageengine(anythingyouwritetoitdisappears)</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************5.row***************************</strong><br/><strong>Engine:MyISAM</strong><br/><strong>Support:DEFAULT</strong><br/><strong>Comment:MyISAMstorageengine</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************6.row***************************</strong><br/><strong>Engine:CSV</strong><br/><strong>Support:YES</strong><br/><strong>Comment:CSVstorageengine</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************7.row***************************</strong><br/><strong>Engine:ARCHIVE</strong><br/><strong>Support:YES</strong><br/><strong>Comment:Archivestorageengine</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************8.row***************************</strong><br/><strong>Engine:PERFORMANCE_SCHEMA</strong><br/><strong>Support:YES</strong><br/><strong>Comment:Performance
Schema</strong><br/><strong>Transactions:NO</strong><br/><strong>XA:NO</strong><br/><strong>Savepoints:NO</strong><br/><strong>***************************9.row***************************</strong><br/><strong>Engine:FEDERATED</strong><br/><strong>Support:NO</strong><br/><strong>Comment:FederatedMySQLstorageengine</strong><br/><strong>Transactions:NULL</strong><br/><strong>XA:NULL</strong><br/><strong>Savepoints:NULL</strong><br/><strong>9rowsinset(0.00sec)</strong>
OverviewofInnoDB
InnoDBisthedefaultstorageenginebroadlyusedoutofallotheravailablestorageengines.ItwasreleasedwithMySQL5.1asapluginin2008.MySQL5.5andlater has InnoDB as a default storage engine. It has been taken over by OracleCorporationinOctober
2005,fromtheInnobaseOy,whichisaFinland-basedcompany.
InnoDBtablessupportACID-compliant commits, rollback, andcrash recoverycapabilitiestoprotect
user data. It also supports row-level locking, which helps with betterconcurrency
andperformance.ItstoresdatainclusteredindexestoreduceI/Ooperationsfor
all SQL select queries based on the primary key. It also supports FOREIGN KEY
constraintsthatallowbetterdataintegrityforthedatabase.ThemaximumsizeofanInnoDBtablecanscaleupto64TB,whichshouldbegoodenough
toservemanyreal-worldusecases.
OverviewofMyISAMMyISAMwas the default storage engine forMySQLprior to 5.5 1. MyISAM storageenginetablesdonotsupportACID-compliantasopposedtoInnoDB.MyISAM tablessupport table-level locking only, so MyISAM tables are not transaction-safe;however, they are optimized for compression and speed. It is generally usedwhenyouneedtohaveprimarilyreadoperationswithminimaltransactiondata.ThemaximumsizeofaMyISAMtablecangrowupto256TB,whichhelpsinusecasessuchasdataanalytics.MyISAMsupportsfull-textindexing,whichcanhelpincomplexsearchoperations.Usingfull-textindexes,wecanindexdatastoredinBLOBandTEXTdatatypes.
Overviewofmemory
Amemorystorageengineisgenerallyknownasaheapstorageengine.Itisusedtoaccessdataextremelyquickly.ThisstorageenginestoresdataintheRAMsoitwouldn'tneedI/Ooperation.AsitstoresdataintheRAM,alldataislostuponserverrestart.
Thistableisbasicallyusedfortemporarytablesorthelookuptable.Thisenginesupportstable-levellocking,whichlimitshighwriteconcurrency.
Importantnotesaboutmemorytablesareasfollows:
Becausememory table stores data in theRAM,whichhas a very limitedstoragecapacity;ifyoutrytowritetoomuchdataintothememorytable,itwillstartswappingdataintothediskandthenyoulosethebenefitsofthememorystorageengine
These tables don't support TEXT and BLOB data types, and it is not evenrequiredasithaslimitedstoragecapacity
This storage engine can be used to cache the results; lookup tables, for
OverviewofarchiveThisstorageengineisusedtostorelargeamountsofhistoricaldatawithoutany
indexes.Archivetablesdonothaveanystoragelimitations.Thearchivestorageengine
is optimized for high insert operations and also supports row-level locking.Thesetablesstoredata
inacompressedandsmallformat.ThearchiveenginedoesnotsupportDELETEorUPDATEoperations;itonlyallowsINSERT,REPLACE,andSELECToperations.
Overview of BLACKHOLE as astorageengine
Thisstorageengineacceptsdatabutdoesnotstoreit.ItdiscardsdataaftereveryINSERTinsteadofstoringit.
Now,what is theuseof this storageengine;whywouldanybodyuse it?WhywouldwerunanINSERTquerythatdoesn'tinsertanythingintothetable?
This engine is useful for replicationwith large number of servers. A BLACKHOLEstorageengineactsasafilterserverbetweenthemasterandslaveserver,whichdonotstoreanydata,butonlyapplyreplicate-do-*andreplicate-ignore-*rulesandwriteabinlogs.Thesebinlogsareusedtoperformreplicationinslaveservers.WewilldiscussthisindetailinChapter8,ReplicationinMySQL8.
OverviewofCSVThe comma separated values (CSV) engine stores data in the .csv file typeusing the comma-separated values format. This engine extracts data from thedatabaseandcopiesitto.csvoutofthedatabase.IfyoucreateaCSVfilefromthespreadsheetandcopyitinto
the MYSQL data folder server, it can read the data using the select query.Similarly,
ifyouwritedatainthetable,anexternalprogramcanreaditfromtheCSVfile.
This storage engine is used for the exchange of data between software orapplications.
ACSVtabledoesnotsupportindexingandpartitioning.AllcolumnsintheCSVstorage
engineneedtobedefinedwiththeNOTNULLattributetoavoiderrorsduringtablecreation.
Overviewofmerge
This storageengine isalsoknownasanMRG_MyISAM storageengine.This storageenginemergesaMyISAMtableandcreatesittobereferredtoasingleview.Foramergetable,allcolumns
are listed in the same order. These tables are good for data warehousingenvironments.
The table is used to manage log-related tables, generally. You can createdifferent
monthsoflogsinseparateMyISAM tablesandmergethesetablesusingthemergestorageengine.
MyISAMtableshavestoragelimitfortheoperatingsystem,butacollectionofMyISAM(merge) tablesdonothavestorage limits.Sousingamerge tablewouldallowyou
to split data intomultiple MyISAM tables,which can help in overcoming storagelimits.
Merge tables do not support partitioning. Also, you cannot partition a merge
Overviewoffederated
Thisstorageengineallowsyoutocreateasingledatabaseonamultiplephysicalserver.Itopensaclientconnectiontoanotherserverandexecutesqueriesagainstatablethere,retrievingandsendingrowsasneeded.Itwasoriginallymarketedas a competitive feature that supported many enterprise-grade proprietarydatabaseservers,suchasMicrosoftSQLServerandOracle,butthatwasalwaysastretch, tosay the least.Although it seemed toenablea lotof flexibilityandneat tricks, it has proven to be a source ofmanyproblems and is disabled bydefault.Thisstorageengineisdisabledbydefault inMySQL;toenableit,youneedtostarttheMySQLserverbinaryusingthefederatedoption.
OverviewoftheNDBclusterNDBcluster(alsoknownasNDB)isanin-memorystorageengineofferinghighavailabilityanddatapersistencefeatures.
TheNDBclusterstorageenginecanbeconfiguredwitharangeoffailoverandloadbalancing options, but it is easiest to startwith the storage engine at theclusterlevel.NDBclusterusestheNDBstorageenginethatcontainsacompleteset of data, which is dependent only on other datasets available within thecluster.
The cluster portion of the NDB cluster is configured independently of theMySQLservers.InanNDBcluster,eachpartoftheclusterisconsideredtobeanode.
Eachstorageenginehasitsownadvantageandusability,asfollows:
SearchEngine:NDBClusterTransactionsdata:InnoDBSessiondata:MyISAMorNDBClusterLocalizedcalculations:MemoryDictionary:MyISAM
Thefollowingdiagramwillhelpyouunderstandwhichstoreengineyouneedtouseforyourrequirement:
Nowyouhaveabetter ideaaboutvariousstorageenginesalongwithdifferentusecases,whichwillhelpyoutomakeadecisionbasedonyourneeds.
It's time to move on to our next topic where we will look at delightful newfeaturesavailableinMySQL8.
ImprovedfeaturesinMySQL8
The MySQL database development team has recently announced its majorreleaseasMySQL
8DevelopmentMilestoneRelease(DMR). Itcontainssignificantupdatesandfixesforproblemsthatweremuchneeded.
Youmight bewonderingwhy it's 8 after 5.7!Were the intermediate versions,thatis,6and7,missout?Ofcoursenot!Actually,6.0waspreservedaspartofthe changeover to a more frequent and timely release, while 7.0 for theclusteringversionofMySQL.
Let'sseesomeexcitingfeaturesthathavebeenintroducedinthislatestversion,as depicted in the following diagram:
It's time to look atMySQL 8 features in detail, which makes us excited andconvincedaboutthereasonsforamajorversionupgradeofMySQL.
Transactionaldatadictionary
Upuntilthepreviousversion,theMySQLdatadictionarywasstoredindifferentmetadatafilesandnon-transactionaltables,butfromthisversiononwards,itwillhaveatransactionaldatadictionarytostoretheinformationaboutthedatabase.Nomore.frm,.trg,or.par files.All informationwillbestored in thedatabase,which removes the cost of performing heavy file operations. There werenumerous issueswith filesystemmetadata storage such as the vulnerability ofthe filesystem, exorbitant file operations, difficult to handle crash recoveryfailures,orreplication;itwasalsodifficulttoaddnewfeature-relatedmetadata.Now this upgrade has made it simple by storing information in a centralizedmanner,andwillhaveimprovedperformanceasthisdatadictionaryobjectcanbecachedinmemory,similartootherdatabaseobjects.
ThisdatadictionarywillhavedatathatisneededforSQLqueryexecutionsuchas catalog information, character sets, collations, column types, indexes,database information, tables, stored procedures, functions and triggers, and soon.
Roles
InMySQL 8, the privileges module has been improved by introducing roles,which means a collection of permissions. Now we can create roles with anumberofprivilegesandassignthemtomultipleusers.
The problem with the previous version was that we were not able to definegenericpermissionsforagroupofusersandeachuserhasindividualprivileges.Supposeif thereare1,000usersalreadyexistingthathavecommonprivileges,andyouwanttoremovethewritepermissionsforthese1,000users,whatwouldyouhavedone in thepreviousversion?Youwouldhavehad to take the time-consuming approach of updating each user, right? Arrgh! That's a long, longtask.
NowwithMySQL8, it is easy to update any change in privileges.Roleswilldefine all the required privileges and this rolewill be assigned to those 1,000users.Wejustneedtomakeanyprivilegechangesintheroleandalluserswillautomaticallyinherittherespectiveprivileges.
Rolescanbecreated,deleted,grantorrevokepermission,grantorrevokefromtheuseraccount,andcanspecifythedefaultrolewithinthecurrentsession.
InnoDBautoincrement
MySQL 8 has changed the auto-increment counter value store mechanism.Previously, it was stored in thememory,whichwas quite difficult tomanageduring server restarts or server crashes. However, now the auto-incrementcountervalueiswrittenintotheredologwheneverthevaluegetschangedand,on each checkpoint, it will be saved in the system table, which makes itpersistentacrosstheserverrestart.
Withthepreviousversion,updateoftheauto-incrementvaluemayhavecausedduplicateentryerrors.Supposeifyouupdatedthevalueofauto-incrementinthemiddleofthesequencewithalargerthanthecurrentmaximumvalue,butthensubsequent insertoperationscouldnot identify theunusedvalues,whichcouldcause a duplicate entry issue. This has been prevented by persisting the auto-incrementvalue,hencesubsequent insertoperationscanget thenewvalueandallocateitproperly.
If server restarthappened, theauto-incrementvaluewas lostwith thepreviousversionasitwasstoredinmemoryandInnoDBneededtoexecuteaquerytofindoutthemaximumusedvalue.Thishasbeenchanged,asthenewerversionhasthecapabilitytopersistitsvalueacrosstheserverrestart.
Duringtheserverrestart,InnoDBinitializesthecountervalueinmemoryusingthemaximum value stored in the data dictionary table. In case of server crashes,InnoDB initializes the auto-increment counter value that is bigger than the datadictionarytableandtheredolog.
<strong>ALTERTABLEtable1ALTERINDEXix_table1_col1INVISIBLE;</strong><br/><strong>ALTERTABLEtable1ALTERINDEXix_table1_col1VISIBLE;</strong>
ImprovingdescendingindexesDescendingindexesexistedinversion5.7too,buttheywerescannedinreverseorder,whichcausedperformancebarriers.To improveperformance,MySQL8hasoptimizedthisandscanneddescendingindexesinforwardorder,whichhasdrasticallyimprovedperformance.
Italsobringsmultiplecolumnindexesfortheoptimizerwhenthemostefficientscan order has ascending order for some columns, and descending order forothercolumns.
ExpandedGISsupportUntil thepreviousversion, it supportedonlyone coordinate system, a unitless2D
placethatwasnotreferencedtoapositiononearth.NowMySQL8hasaddedsupport for aSpatialReferenceSystem (SRS)with geo-referenced ellipsoidsand2Dprojections.SRShelpsassigncoordinates toa locationandestablishesrelationshipsbetweensetsofsuchcoordinates.ThisspatialdatacanbemanagedindatadictionarystorageastheST_SPATIAL_REFERENCE_SYSTEMStable.
DefaultcharactersetThe default character set has been changed from latin1 to UTF8. UTF8 is thedominatingcharacterset,thoughithadn'tbeenadefaultoneinpreviousversionsofMySQL. Along with the character set default, collation has been changedfromlatin1_swedish_ci to utf8mb4_800_ci_ai.With these changes globally accepted,charactersetsandcollationsarenowbasedonUTF8;oneofthecommonreasonsis because there are around 21 different languages supported by UTF8, whichmakessystemsprovidemultilingualsupport.
Extendedbit-wiseoperations
InMySQL5.7,bit-wiseoperationsandfunctionswereworkingforBIGINT(64-bitinteger)data typesonly.WeneededtopassBIGINTasanargumentanditwouldreturn the result as BIGINT. In short, it had maximum range up to 64 bits toperformoperations.AuserneedstodoconversiontotheBIGINTdatatypeincasetheywanttoperformitonotherdatatypes.Thistypecastingwasnotfeasiblefordatatypeslargerthan64bitsasitwouldtruncatetheactualvalue,whichresultedininaccuracy.
MySQL8hasimprovedbit-wiseoperationsbyenablingsupportforotherbinarydata types such as Binary,VarBinary, and BLOB.Thismakes it possible to performbit-wiseoperationsonlargerthan64-bitdata.
Nomoretypecastingneeded!Thisallowsthetakingofargumentsandreturningresultslargerthan64bits.
InnoDBMemcachedMultiple get operations are now possible with the InnoDB memcached plugin,whichwill really help in improving the read performance.Now,multiple keyvalue pairs can be fetched in a single memcached query. Frequentcommunicationtraffichasalsobeenminimizedaswecangetmultipledatainasingleshot.
RangequeriesarealsosupportedbytheInnoDBMemcachedplugin.Itsimplifiesrangesearchesbyspecifyingaparticularrangeandretrievesvalueswithin thisrange.
<strong>SELECT*FROMtable1WHEREid=5FORUPDATENOWAIT;</strong><br/><strong>SELECT*FROMtable1FORUPDATESKIPLOCKED;</strong>
JSON
JSONsupporthadbeenimplementedinMySQL5.7;itwaswell-acknowledgedfeature.
InMySQL8 ithasaddedvarious functions thatwouldallowus togetdatasetresultsinJSONdataformat,virtualcolumns,andtentatively15SQLfunctionsthatallowyoutosearchanduseJSONdataonserverside.InMySQL8thereareadditionalaggregationfunctionsaddedthatcanbeusedinJSONobjects/arraystorepresent loadeddata ina furtheroptimizedway.Thefollowingare the twoJSONaggregationfunctionsthatwereintroducedinMySQL8:
JSON_OBJECTAGG()
JSON_ARRAYAGG()
Cloud
InMySQL8anewoption is introducedinnodb_dedicated_server,whichwouldbehelpfulforverticalscalingoftheservers.Itactuallyautomatically
detectsthememoryallocatedtothevirtualserverandappropriatelysetMySQL8without
any need to change configuration files. These would be very handy featuresconsidering
theadoptionofvirtualizationandcloudisthere.Infactwiththisconfiguration,
youmightnotevenneed toget shellaccessof server toedit theconfigurationfiles.
You can do this with the new SET PERSIST feature that can set relevantconfigurationfromtheMySQLcommandlineitself,which
can enhance security further as you almost wouldn't need shell access of theserver.
Resourcemanagement
MySQL8hascomeupwithawonderfulresourcemanagementfeaturethatwillallowyou toallocate resource to threads runningona server,whichwouldbeexecutedbasedontheresourcesconfiguredforthegroup.Currently,CPUtimeisaresourcethatcanbeconfiguredforagroup.Withthis,youcantweakyourworkloadswithvirtualresourcemanagementwithinMySQLitself.MySQLwillidentifyonstartupnumbersofvirtualCPUsavailableandafter thatuserswithappropriateprivilegescanmapthevirtualCPUswithresourcegroupandalignthreadmanagementtothesegroups.
Weexpect to seemore featuresby the timeMySQL8 is available forgeneraluse.LetusnowlookatbenefitsofusingMySQL8.
BenefitsofusingMySQL8Whetheryouareadeveloperoranenterprise,youwouldobviouslychooseonethatprovidesgoodbenefitsandresultswhencomparedtootherrelatedproducts.MySQL
providesnumerousadvantagesas thefirstchoice in thiscompetitivemarket. Ithas various powerful features available that make it a more comprehensivedatabase.Let'snowgothroughsomebenefitsofusingMySQL.
Security
Thefirst thing thatcomes tomindissecuringdatabecausenowadaysdatahasbecomepreciousandcanimpactbusinesscontinuityiflegalobligationsarenotmet; in fact, it canbe sobad that it canclosedownyourbusiness inno time.MySQL is themost secure and reliable databasemanagement systemused bymanywell-knownenterprisessuchasFacebook,Twitter,andWikipedia.Itreallyprovidesagoodsecuritylayerthatprotectssensitiveinformationfromintruders.MySQLgivesaccesscontrolmanagementsothatgrantingandrevokingrequiredaccessfromtheuseriseasy.Rolescanalsobedefinedwithalistofpermissionsthatcanbegrantedorrevokedfortheuser.
All user passwords are stored in an encrypted format using plugin-specificalgorithms.
Scalability
Day by day, the mountain of data is growing because of extensive use oftechnologyinnumerousways.Becauseofthis,loadaverageisgoingthroughtheroof.Insomecases,itisunpredictablethatdatacannotexceeduptosomelimitor number of userswill not go out of bounds. Scalable databaseswould be apreferable solution so that, at any point,we canmeet unexpected demands toscale.MySQLisarewardingdatabasesystemforitsscalability,whichcanscalehorizontally and vertically; in terms of data, spreading database and load ofapplicationqueriesacrossmultipleMySQL
servers is quite feasible. It is pretty easy to add horsepower to the MySQLclustertohandletheload.
An open source relationaldatabasemanagementsystemMySQLisanopensourcedatabasemanagementsystemthatmakesdebugging,upgrading, and enhancing the functionality fast and easy. You can view thesourceandmakethechangesaccordinglyanduseitinyourownway.Youcanalso distribute an extended version of MySQL, but you will need to have alicenseforthis.
HighperformanceMySQL gives high-speed transaction processing with optimal speed. It cancache the results, which boosts read performance. Replication and clusteringmake the system scalable for more concurrency and manages the heavyworkload. Database indexes also accelerate the performance of SELECT querystatementsforsubstantialamountofdata.Toenhanceperformance,MySQL8
hasincludedindexesinperformanceschematospeedupdataretrieval.
Highavailability
Today, in theworldofcompetitivemarketing,anorganization'skeypoint is tohave their system up and running. Any failure or downtime directly impactsbusiness and revenue; hence, high availability is a factor that cannot beoverlooked.MySQLisquitereliableandhasconstantavailabilityusingclusterandreplicationconfigurations.
Cluster servers instantly handle failures andmanage the failover part to keepyoursystemavailablealmostallthetime.Ifoneservergetsdown,itwillredirecttheuser'srequesttoanothernodeandperformtherequestedoperation.
Cross-platformcapabilities
MySQL provides cross-platform flexibility that can run on various platformssuchasWindows,Linux,Solaris,OS2,andsoon.IthasgreatAPIsupportforthe allmajor languages,whichmakes it very easy to integratewith languagessuch as PHP,C++, Perl, Python, Java, and so on. It is also part of theLinuxApache MySQL PHP (LAMP) server that is used worldwide for webapplications.
It's now time to get our handsdirty and look atMySQL8; let's startwith theinstallationofMySQL8onaLinuxplatforminourcase.WepreferMySQL8onaLinuxoperatingsystemas thathasbeenacommonusecaseacrossmanyorganizations.WewillbediscussingmoreinstallationinChapter2,InstallingandUpgradingMySQL8.Youcanuse itonotherplatforms thatMySQLsupports,suchasWindows,Solaris,HP-UNIX,andsoon.LinuxprovidesvariouswaystoinstalltheMySQLserver,asfollows:
RPMpackage
YUMrepository
LimitationsofMySQL8Acoin has two sides; similarly, benefits of also usingMySQL8would comealongwithafewlimitations.LetuswalkthroughafewareasofMySQL8now.
NumberoftablesordatabasesThenumberofdatabasesor tablesarenota limitationforMySQL8;however,theoperatingsystemfilelimitcanbealimitationforMySQL8.StorageEngineInnoDBisallowedtoscaleuptofourbilliontablesasitspeaknumber.
TablesizeYoumayhitmaximumtablesizelimit,whichisnotrestrictedfromMySQL8;however,itmaybebecauseofoperatingsystemfilesystemlimits.
JoinsInasinglejoin,onecanuse61tables,whichcanbereferred.Itisalsoapplicableto the tables that are referenced in view definition. Joins that are part ofsubqueriesandviewsarealsoconsideredtobepartofthelimitation.
Windowsplatform
There are few limitations when you have MySQL 8 used on the Windowsplatform:
Memory:32-bitarchitecturehaslimitationtouseonly2GBofRAMforaprocess.
Ports: In case you have a high number of concurrency youmight comeacross Windows platform limitation of having 4000 ports available forclientconnectionsintotal.
Case-insensitivity: The Windows platform doesn't have case sensitivity,which is why tables and databases need to be deliberately managed forcase-insensitivity.
Pipes:|, generally referred as pipe signs, they are not fully supported inWindows. You might come across them in a few scenarios while doingdatabaseadministrationactivities.
Pathname separator: MySQL 8 escape character is \, which is the
pathnameseparatorforWindows.Hencewhileusingpathseparatoryoucandoubleslashas"\\"asanalternativeforapathnameseparator.
TablecolumncountThe table column for each table inMySQL8 has a limit of 4096 columns. Itmightvarybasedonafewotherfactorsforcolumnscountlimit,asstatedinthefollowingsection.
RowsizeMySQLtableshavealimitof65,535bytesforarow,althoughstorageenginessuchasInnoDBarecapableofsupportinglargerchunks.
InnoDBstorageengineLimitations on InnoDB storage engine are what we will talk about a bit morespecificallyasInnoDBnowwithMySQL8willplayaprominentrole.
Limitations of InnoDB storageengine
WewillhaveaquickglanceatafewofthelimitationsofInnoDBstorageengine:
Thenumberofindexessupportedcanbemaximum64foratable
For tables thatusecompressedordynamic row format;3072 is the indexkeyprefixlengthlimit
Fortablesthatusecompactorredundantrowformat;767istheindexkeyprefixlengthlimit
Total columns in a table, which includes virtual generated columns, arelimitedtoamaximumof1,017
16columnsisthemaximumpermittedformulti-columnindexes
ThecombinedInnoDBlogfilesizecannotexceed512GB
MaximumtablesizesupportedbyInnoDBis256TB
AdminAPIisnotsupportedwhileusingunixsocketconnections
Multi-byte characters might give you unreliable aligned columns whileformattingofresultsinInnoDBclusters
Restrictions
WewillnowhaveaquickglanceatafewoftherestrictionsoftheInnoDBstorageengine:
Deletefromtablename: Itdoesn'tactuallydelete thecomplete table, insteaditdeleteseachrowofthetableoneafteranother.
Show table status: It wouldn't provide you accurate data all the time; itprovidesestimates.
When counting rows, the number of rows provided by count(*) is notaccuratebecauseofconcurrency; itwouldcountonly thosecountsvisibletotransactionscurrentlyavailable.
Ifthereismultipleanalyzetablequeriesexecuted,lateronewillbeblockeduntilthefirstonegetscompleted.
InnoDBkeepsanexclusive lockon the indexat theendassociatedwith theauto_incrementcolumn.
Inacasetheauto_incrementintegerrunsoutofthevalue;thefollowinginsertoperationswouldshowusduplicate-keyerrors.
Foreignkeysthatarecascadedcannotactivatetriggers.
There are a few column names reserved by MySQL that InnoDB uses forinternalpurposes.Thefollowingareafewsuchcolumnnames:
DB_ROW_ID
DB_TRX_ID
DB_ROLL_PTR
DB_MIX_ID
Wemightcomeacrossoutputshowninthefollowingexampleincaseofsuchreservedcolumnnamesused:
mysql>CREATETABLEchintan(c1INT,db_row_idINT)
ENGINE=INNODB;ERROR1166(42000):Incorrectcolumnname'db_row_id'
InnoDB locks are released immediately after the transaction is aborted orcommitted,whichisheldbyatransaction.
Theadditionoftablelocksarenotsupported,aslocksareimplicittocommitandunlocktables
Datadictionary
Letushavealookatafewknownlimitationsofdatadictionary:
IndividualMyISAMtablesforbackupandrestorearenotsupportedbymerelycopyingthefiles.
ManuallycreateddirectoriesfordatabasesarenotsupportedbyMySQL8.For instance, using mkdir would have no impact on MySQL server datadictionary.
DDLoperationswouldtakemoretimethanexpectedbecausesuchoperationsarewritten tostorage,undologsandredoinsteadof.frm filesaswhatwewouldhaveseeninpriorversionsofMySQL.
LimitationsofgroupreplicationinMySQL8
It'snowtimetodiscussafewlimitationsofgroupreplicationinMySQL8:
Largetransactions: Transactions that result toGTID contents cannot bereplicatedbetweentherestofthemembersofthegroupifthey'retoolarge.It is suggested to use smaller chunks of data that cannot be replicated inaroundfivesecondstogroupmemberstoavoidfailures.
Clusterfromagroup:Ifyoutrytocreateclustersfromanexistinggroupreplicationsetupitwillresult inanerrorastheinstancewouldalreadybepart of a replication group. This is noticed currently only in MySQL'swizardmodeonly;analternativesolutionfortheissueistodisablewizardmode.
Serializable isolation level: Serializable isolation level is not supportedwhenmulti-primarygroupsareused,whichisthedefaultconfiguration.
DDL and DML operations: If there is concurrent DDL and DML
operationsexecutedagainstthesamedataobjectbutondifferentserversisnotsupportedwhenmulti-primarygroupmodeisused.
Replication checksum: Currently MySQL design limitations createrestrictionsofhavingreplicationeventchecksums.
ConstructsprohibitionThefollowingaretheconstructsthatarenotallowedinexpressionsofpartitions:
DeclaredvariablesUservariablesStoredproceduresStoredfunctionsUDFsPlugins
OperatorsThere are a few operators that are not permitted in partition expressions suchas<<,>>,|,&,~and^.Resultsforarithmeticoperatorssuchas+,-,and*musthaveanintegervalueorNULL.
Tables
Thefollowingareafewspecificareasthatshowuslimitationsofpartitioningontables:
Themaximumnumberofpartitions supportedbyMySQL8 fora table is8192.Thislimitalsoconsiderssub-partitions.
Fulltextindexandsearchisnotsupportedonpartitionedtables.
Tablesthataretemporarycannotbepartitioned.
Logtablescan'tbepartitioned.
ForeignkeysarenotsupportedonpartitionedInnoDBstorageengine.
Thedata typeofpartitionkeys shouldbean integercolumnorcanbeanexpression to an integer. Expression or column values may be NULL;however,expressionsthatincludeENUMarenotsupported.
Upgradingpartitionedtables thathavebeenpartitionedbyKEYwouldhave
tobereloaded,whichstandstrueotherthantheInnoDBstorageengine.
Wehavesofardiscussedoverview, features,benefits,anda fewlimitationsofMySQL.
LetusnowwalkthroughthewonderfulusecasesofMySQL.
UsecasesofMySQL
MySQL has many advantages because it has its foot in many industries andvarioususecases across theglobe.The importance ofMySQLdoesn't dependonlyonhowmuchdatayouhave,it'sratherwhatyouaregoingtodowiththedata.Datacanbesourcedandanalyzedfromunpredictablesourcesandcanbeusedtoaddressmanythings.
Let's now look at use cases with real-life importance made on renownedscenarioswiththehelpofMySQL:
The preceding figure helps us understand where MySQL is serving variousindustries.
Thoughit'snotanextensivelistofindustrieswhereMySQLhasbeenplayingaprominentroleinbusinessdecisions,let'snowdiscussafewoftheindustries.
Socialmedia
Social media content is information, and so are engagements such as views,likes,demographics,shares,follows,uniquevisitors,comments,anddownloads.At the end of the day, what matters is how your social media-related effortscontributetothebusiness.
OnenotableexampleisFacebook,whereMySQLhadbeenusedextensively.Ontop of MySQL where petabytes of data was used to serve likes, shares, andcomments. Facebook has developed the RocksDB storage engine on top of theMySQL InnoDB storage engine, which leverages many advantages of InnoDBstorageengineasFacebookwantedtoprimarilyfocusonstorageoptimization.ThoughcurrentlyMySQLisstillusedlargelyforothercommonapplications.
Government
The era of MySQL has been playing a significant role in government too;government bodies have been using MySQL extensively because of splendidreturnoninvestmentsandpromotingopensource.Infact,thegovernmentsectoriscarryingoutahugenumberofimplementationsofMySQLworldwide.
Thismaycomeasasurprisetoyou;USNavyusesMySQLforitscriticalflightplanningactivities.Therearevariousactivitiessuchasweatherconditions,flightplans, fuel efficiency, maintenance of flights, and many more that are beingtrackedwiththehelpofMySQLasthedatabase.It'sano-brainerthatitneedstorun 24x7 with full redundancy;MySQL was able to achieve this serving USNavyaircraftacrosstheglobe.
MediaandentertainmentYouTube isalsooneof theprominentusersofMySQL.AnytimeyouwatchavideoonYouTube itgetsdata froma relationaldatabaseorablobstoreusingMySQL.YouTubealsousesVitess;aproject thatwas releasedbyYouTube tofrontendMySQL.VitesshelpstodolotsofoptimizationandactsasaproxytoserveeachdatabaserequestusingMySQL.MySQLreplicasareheavilyusedinYouTube's implementation; leveraging MySQL caching was one of the otherprominentfactorsforYouTube.
Frauddetection
Whenitcomestosecurity,frauddetection,orcompliance,andpreciselyifyoursolutionhelpsyouinidentifyingandpreventingissuesbeforetheystrike,thenitbecomesasweetspotforbusiness.Mostofthetime,frauddetectiontakesplacealongtimeafterthefraudhasoccurred,whenyoumighthavealreadysufferedloss. The next stepswould be obviously tominimize the impact of fraud andimproveareasthatcouldhelpyoupreventthisfrombeingrepeated.
Manycompanieswhoareintoanytypeoftransactionprocessingorclaimsusefraud detection techniques extensively. MySQL helps to analyze transactions,claims, and so on in real time, along with trends or anomalous behavior topreventfraudulentactivities.
PayPal is one of such use cases that has built fraud detection system usingMySQL.
PayPal has more than 100 million active users, which is distributed to US,Japanese, andEuropean data centers.High-availability for such use cases is akeycriteriaalongwithperformance,whichMySQLhasbeenabletodeliverasexpected.
BusinessmappingNetflix has millions of subscribers; it uses MySQL for running its billingsystems.
ThecorebillingsystemofNetflixonMySQLisaprominentbackboneforanybusiness.
Netflixhasbillionsofrowsofdataconcurrentlyupdatedandofconsistingdatasince
itsinceptiontwodecadesago.Compliancewasoneofthekeyfactorsalongwithmigration
fromOraclewithminimaldowntime;bothofthesewereachievedwithMySQLandhas
beenexpandingtremendouslyeveryotherday.
E-commerce
Uber is one of the other well-known customers of MySQL. Uber had beengrowingenormouslyworldwide,andscalability,high-availability,andreturnoninvestmentswereafewoftheimportantcriteriatobeworkedupon.UberusesMySQLasitsprimarydatabaseforitsknownprivatecartransportationservice.UberheavilyusesschemalessdatabasearchitectureasitsbackendasalayeronMySQL.
There are many real-world MySQL use cases that have changed humanity,technology, predictions, health, science and research, law and order, sports, e-commerce, power and energy, financial trading, robotics, and many more.MySQLisanintegralpartofourdailyroutine,whichisnotevidentallthetime,butyes,itplaysasignificantroleinwhatwedoinmanyways.
Summary
In this chapter, we started with an overview of MySQL along with majorfeatures of the MySQL database and explored the newly added features inMySQL8.Afterthis,wetookadeepdiveintoexcitingnewfeaturesofMySQL8alongwith
benefitsofusingMySQLforyourbusinessapplications.WeunderstoodMySQL8'scurrent
limitations and restrictions, which is important for us when performing theimplementations.
Finally,weglancedthroughafewimpressiveusecasesfromtherealworldthatplay
prominentrolesinourdailyroutine,andtheyalluseMySQLastheirdatabase.
In the next chapter, we will learn detailed steps for installing MySQL 8 ondifferent
platforms. The chapter also covers methods to upgrade or downgrade fromMySQL8,and
Installing and Upgrading MySQL8
Inthepreviouschapter,weprovidedanoverviewofMySQLalongwithMySQL8'snewfeatures,usecases,andlimitations.MySQLisveryflexibleintermsofplatforms, such as RedHat, Fedora, Ubuntu, Debian, Solaris, MicrosoftWindows, and so on. It has the support of an API to connect with differentlanguages, such as C, C++, C#, PHP, Java, Ruby, and many more. For anyprogrammingplatform,themostimportantandmonotonoustaskistosetuptheenvironment with the necessary software tools. That won't be the case forMySQL
8,asthischapterisfocusedonsettinguptheenvironmentwithMySQL8.
ThischapterexplainsMySQL8's installationsteps indetailwith thenecessaryprerequisites.
SeparateinstallationstepsareprovidedtosetupMySQL8onvariousplatforms.ThechapteralsocoversmethodstoupgradetoordowngradefromMySQL8.
Wewillcoverthefollowingtopicsinthischapter:
TheMySQL8installationprocessThis section will guide readers in MySQL 8 version selection, where to getMySQL 8 from, and how to install MySQL 8. It also explains the post-installationstepsrequiredforsetup.ThischapterprovidesinformationonhowtoupgradeordowngradefromMySQL8.
Generalinstallationguide
MySQL8 isavailableonmanyoperatingsystemswithdifferentversions.TheMySQL
8releaseismanagedintwoways:
Developmentrelease:Thishasthenewestfeaturebutisnotrecommendedforuseinproduction
Generalrelease:Thisisastablereleaseanduserscanuseitforreleaseinproductionalso
NamingconventionsarefollowedineachreleaseofMySQL8,whichindicatesitsstatus.
Eachreleasenameconsistsofthreedigitsandanoptionalsuffix.Forexample,mysql.8.1.2-rc.Thenumbersareinterpretedasfollow:
Thefirstnumber(8)indicatesamajorversionoftherelease.
The second number (1) indicates a minor version of the release. Acombinationofmajorandminornumbersdescribestheseriesoftherelease.
The third number (2) indicates the versionwithin the release series. It isincrementedoneachbugfixrelease.
Themostrecentversionofthereleaseisthemostpreferableforuse.ThesuffixgivenintheexampleindicatesthestabilityoftheMySQL8release.TheMySQL8
releasefollowsthreesuffixes:
DevelopmentMilestoneRelease (dmr):MySQL8 follows themilestonemodel,whereeachmilestoneindicatesthoroughlytestedfeatures.
ReleaseCandidate (rc):Anewfeaturemightget released in thisversionbuttheaimistofixbugswithinthepreviouslyreleasedfeatures.
Absence of a suffix: This indicates General Availability (GA) orproductionrelease.Thisreleaseisstableandpassedthroughearlierstages.
Itisreliableandsuitableforuseinproduction.
As described, preceding each release is the DMR, followed by the RC, andfinallytheGAreleasestatus.Now,afterdecidingtheMySQL8versionfortheinstallation,it'stimetoselectthedistributionformat.
Thebinarydistributionisrecommendedforgeneral-purposeuse.It isavailableinnativeformatsformanyplatforms.Forexample,theRPMpackageforLinuxandDMG
packageforOSX.
DownloadingMySQL8
TogetMySQL8fromtheofficialsite,refertothefollowingURL:http://dev.mysql.com/downloads/.MySQLalsoprovidesamirrorsite:http://dev.mysql.com/downloads/mirrors.html.Whenyoureachthedownloadpage,youcanseetheversionselectiontabatthebottomsideofthepage,wheretwotabsaredisplayed:
GenerallyAvailable(GA)release
Developmentrelease
Basedontheprevioussection,selectthesuitableversionfromthelistandclickontheDownloadbutton.
VerifyingthepackageintegrityThis is a stage where the downloaded package is available and ready for theinstallation.
It'sanoptionalstep,butwerecommendittoavoiderrorsduringtheinstallationprocess.Therearethreedifferentwaysavailabletocheckintegrity:
UsingMD5checksums
Usingcryptographicsignatures
UsingtheRPMintegrityverificationmechanism
<strong>E:\Softwares\md5>md5.exe<br/>E:\Softwares\mysql-installer-community-5.7.19.0.msi</strong><br/><strong>2578BFC3C30273CEE42D77583B8596B5<br/>E:\Softwares\mysql-installer-community-5.7.19.0.msi</strong>
Performthefollowingstepsforthegraphicaltoolexecution:
1. Openthelink:http://www.nullriver.com/index/products/winmd5sum.
2. ClickontheDownloadWinMD5Sumoptiononthepage.ItwilldownloadwinMD5Sum.exeontoyourcomputer.
3. RunthedownloadedInstall-winMD5Sum.exeandinstallitonyourlocalmachine.
4. Aftersuccessfulinstallation,openthewinMD5Sumtool.ThisopensonedialogboxwhereyouhavetoselectthedownloadedMySQL.msifile.
5. Clickonthecalculatebutton.ThiswillcalculatetheMD5checksumofthedownloadedfile.
6. EntertheMD5checksumavailableontheMySQLdownloadpageinthecomparetextboxandpressthecomparebutton.
<strong>cmd>gpg--verifypackage_name.asc</strong>
<strong>cmd>rpm--checksigpackage_name.rpm</strong>
ThistechniqueofverificationismorereliablethantheMD5checksumbutitisverycomplexandrequiresmoreeffortforintegritychecks.
Installing MySQL 8 on MicrosoftWindows
MySQLisavailableforboth32-bitand64-bitversions.Therearedifferentwaysavailable to install MySQL 8 on Microsoft Windows. The most commonapproachistouseaninstaller,whichinstallsandconfiguresMySQL8onyourlocalsystem.
BeforeinstallingMySQLCommunity8.0Server,makesurethattheMicrosoft Visual C++ 2015 redistributable package has beeninstalledonthesystem.
MySQL8 either runs as a standard application or runs as aWindows service.Useit,astheserviceenablesuserstocontrolandmeasureoperationsusingtheWindows service management tool. Three major distribution formats areavailableforeachplatform:
Installerdistribution:ThisincludestheMySQL8serveralongwithotherproductssuchasMySQLWorkbench,MySQL
for Excel, andMySQLNotifier.An installer is also useful for upgradingproductsintootherversions.
Sourcedistribution:Asthenameimplies,thiscontainsallthesourcecodealongwithallthesupportedfiles.TheVisualStudiocompilerisrequiredtomakeitexecutable.
Binary distribution: This distribution is available in ZIP file format. Itcontains all the required files except the installer.Theuserhas tounpackthefileintoaselecteddirectory.
Windows-specificconsiderations
BeforeinstallingMySQL8onMicrosoftWindowsconsiderfollowingpoints:
Antivirussoftware:Asweknow,antivirussoftwareusesthefingerprintingtechnique,whichwillconsiderrapidlychangedfilesasapotentialsecurityrisk.InMySQL8,therearesomedirectoriesthatcontainMySQL8relateddataandtemporarytablesinformationandareupdatedfrequently.So,thereisapossibilitythatantivirussoftwarewillconsiderthosefilesasspam.Thiswillalsoimpactperformance.
Antivirus software provides configurations to exclude some of thedirectories,soitisrecommendedtoexcludetheMySQL8datadirectoryandtempdirectory.MySQL
8, by default, stores temporary data into a Microsoft Windows temporarydirectory.
To change this default configuration inMySQL 8, refer to my.ini file's tempdirparameter.
Large table support:UseMySQL8 onNTFS or any new filesystem tosupportlargetableswhosesizeismorethan4GB.Fortheselargertables,theuserhastodefinetheMAX_ROWSandAVG_ROW_LENGTHpropertiesatthetimeoftablecreation.
MySQL8installationlayout
MicrosoftWindows, by default, considers the C:\Program Files directory for theMySQL8installation.However,wehaveachoiceforthedirectoryselectionatthe time of installation. Whatever the location of the installation, thesubdirectorystructureafterinstallationremainssame.FortheMicrosoftWindowlayout,refertothefollowingtable:
Directory ContentsofDirectory Notes
binmysqldserver,clientandutilityprograms
%PROGRAMDATA%\MySQL\MySQL
Server8.0\ Logfiles,databases
TheWindowssystemvariable%PROGRAMDATA%
defaultstoC:\ProgramData
examples Exampleprogramsandscriptsinclude Include(header)files
lib Libraries
share
Miscellaneoussupportfiles,includingerrormessages,charactersetfiles,sampleconfigurationfiles,SQLfordatabaseinstallation
Youcanlearnmoreaboutthistopicathttps://dev.mysql.com/doc/refman/8.0/en/windows-installation-layout.html.
Choosing the right installationpackage
There are multiple options available for package formats while installingMySQL 8 onWindows. MySQL provides a facility to debug the installationprocessusingprogramdatabase(pdb)files.ThesefilesareavailableinaZIPdistribution:
Theinstallerpackage:This isawizard-basedprocessandiseasy touse.Theinstallerpackageisavailable
for32-bitsonlybutcaninstallMySQL8onthe64-bitconfigurationalso.Itdoes
notcontainthedebuggingcomponentofMYSQL;wehavetodownloaditseparatelyin
the formof aZIP file.The installer package is available in twodifferentformats:
Web Community: As the name implies, this is available for webinstallation.ItmeanstheInternet
is required for the installation using the web community. Its size isapprox19MB.
ItsnameisdefinedasMySQL-installer-communityby theappendingversion.
Community:Thispackage format is used for offline installation. Itssizeisapprox301MB.
Its name is defined as MySQL-installer-web-community by theappendingversion.
An installer is themost commonway for MySQL product installationandupgrade.
TheNoinstallArchives:Thisisamanualinstallationprocessthatcontainsfilesfortheincompleteinstallation
package.As it is amanual process, noGUI is available.The user has tomanually
install andconfigureMySQL8andotherproducts if required.Unlike theinstaller,
itprovides twodifferent files for32-bit and64-bit configuration inaZIPformat.
TheMySQL8installer
The MySQL 8 installer is mainly used to reduce the complexity of theinstallationprocessalongwiththemanagementofMySQLproductsrunningontheWindowsplatform.
Intheproductlist,wecanconsider:
MySQLservers
MySQLapplications
MySQLconnectors
Documentationandsamples
TheMySQL8installerhastwoeditions:
CommunityEdition:Thiscanbedownloadedathttp://dev.mysql.com/download
s/installer/. As described in previous section, bothWeb Community andCommunitypackageformatsareavailablefortheinstaller.
CommercialEdition:Refer tohttps://edelivery.oracle.com/ todownload theCommercialEdition.TheCommercialEditioncontainsalltheproductsthatareavailableintheCommunityEditionalongwiththefollowingproducts:
WorkbenchSE/EE
MySQLEnterprisebackup
MySQLEnterprisefirewall
Initialsetupinformation
Asmentionedpreviously,theinstallerwillguideauserthroughthewizard.OncewestarttheinstallerinourhostmachineitwilldetectalreadyinstalledMySQL
productsandconsider themina listofproducts tobemanaged.Thefollowingarethestepsthatarerequiredintheinitialsetupoftheinstaller:
1. MySQLinstaller licensingandsupportauthentication:This is the stepwhere the user must accept the license agreement before starting theMySQL8installation.Afteracceptingtheterms,theuserisallowedtoadd,update,orremoveMySQLproducts.IntheCommercialEdition,credentialsare required to unbundle products andmustmatchwith the user'sOracleaccountinthesupportsite.
2. Choosingasetuptype:ThisisthestepwheretheusermustselectMySQLproducts for installation. The installer also provides the option of apredefined setup,whichcontains a setofMySQLproducts.So,youhavethe flexibility of selecting one setup type as per your requirements. Thefollowingaresomesetupsavailableintheinstaller.
3. Developerdefault:This installs theversionofMySQL8 server thatwasselectedatthetimeofdownload:
MySQLserver
MySQLshell
MySQLrouter
MySQLWorkbench
MySQLforVisualStudio
MySQLforExcel
MySQLnotifier
MySQLconnectors
MySQLutilities
MySQLdocumentation
MySQLsamplesandexamples
4. Serveronly:ThisinstallsonlytheMySQLserver.
5. Clientonly:Thisisthesameasthedeveloperdefaultsetuptype,exceptitdoesnotcontaintheMySQL8serveroranyclient-specificpackageaddedtoit.
6. Full:ThisinstallsalltheavailableproductsofMySQL,suchasmysql-server,mysql-client,mysqladmin,andafewmore.
7. Custom: This option installs only those products that are selected by theuser from the catalog.Here, the user has the freedom to choose only therequiredproducts,ratherthaninstallingthecompletebundleofproducts.
8. Path conflicts: When the hosting system already contains a MySQLproductandtheuseris tryingtoinstalladifferentversionofthatMySQL
productonthesamepath,thentheinstallerwillshowapathconflicterrorin the wizard. The installer enables the user to take action on the pathconflictinthefollowingways:
ChooseadifferentlocationusingtheBrowsebuttonfromthewizard
Chooseadifferentsetuptypeorversionbycustomselection
Overwritetheexistingfolderbymovingontothenextstep
Cancelthewizardsteps,deleteexistingproducts,andstarttheinstalleragain
9. Check requirements: Each MySQL product has a package-rules.xml fileattachedtoit,whichcontainsalltheprerequisitesoftwarelists.Duringtheinitialsetup,theinstallerwillchecktheavailabilityoftherequiredsoftwareandprompttheusertoupdatethehostincaseofmissingrequirements.
10. MySQLinstallerconfigurationfiles:The installerconfigurationfilesarelocatedatC:\ProgramFiles.Thefollowingaretheconfigurationfiledetails:
FileorFolder
Description
FolderHierarchy
MySQLinstallerforWindows
This folder contains all of the filesneededtorunMySQLinstallerandMySQLinstallerConsole.exe, a command-lineprogramwithsimilarfunctionality.
C:\ProgramFiles(x86)
Templates
TheTemplatesfolderhasonefileforeachversionofMySQLserver.Templatesfilescontainkeysandformulastocalculatesomevalues
C:\ProgramData\MySQL\MySQL
installerfor
Windows\Manifest
dynamically.
package-
rules.xml
This file contains the prerequisites foreveryproducttobeinstalled.
C:\ProgramData\MySQL\MySQL
installerfor
Windows\Manifest
produts.xml
The products file (or product catalog)containsa listofallproductsavailablefordownload.
C:\ProgramData\MySQL\MySQL
installerfor
Windows\Manifest
Product
Cache
The Product Cache folder contains allstandalone MSI files bundled with thefullpackageordownloadedafterward.
C:\ProgramData\MySQL\MySQL
installerforWindows
Reference:https://dev.mysql.com/doc/refman/8.0/en/mysql-installer-setup.html
Installationworkflow
TheMySQLinstallerfollowsaworkflowforeachproduct:
1. Product download: The installer will download all the required productMSIfilesintotheProductCachefolder.
2. Product installation:The installermanages thestatusofeachproductbyReady|Install|Installing|Complete.
3. Product configuration: This phase uses a step-by-step configurationprocess for products. The installer will change the status from Ready |Configure.
4. Installationcomplete:Thisfinalizestheinstallationandtheusercanstartusingtheapplicationaftertheinstallation.
InnoDB cluster sandbox testsetup
Therearetwooptionsavailableforhigh-availabilityimplementationinMySQL8,usingtheinstaller:
StandaloneMySQLserver/ClassicMySQLreplication (default):Thisoption configuresmultiple servers manually or uses the latest version ofMySQLShelltoconfiguretheInnoDBcluster.
InnoDBclustersandboxtestsetup(fortestingonly):Thisisalsoknownasthesandboxcluster.ThisoptionallowsyoutocreateanInnoDBclusteronthe local system for testing only. The MySQL installer toolbar providesconfigurationsforanumberofinstancesinInnoDBclustering.
Clusternodesrunondifferentports.Afterconfiguration,clickontheSummarytabtogettheportdetailsofeachcluster.
Serverconfiguration
The MySQL installer performs some basic configurations for the MySQL 8server,including:
Theinstallerwillcreateamy.iniconfigurationfilefortheMySQL8server.Filecontentswillbedecidedaspertheselectedoptionsoftheinstallationprocess.
By default, the installer will add theWindows service for theMySQL 8server.
TherequireddefaultinstallationanddatapathsoftheMySQL8serverwillbeprovidedbytheinstaller.
TheinstallerwillcreatesomeuseraccountswithrolesandpermissionsforMySQL
8server.ItcancreateaWindowsuserwithlimitedprivilegestotheMySQL8server.
Using Show Advanced Options, MySQL installer allows for definingcustom paths for logging options. For example, you can configure theseparatepathforanerrorlog,showaquerylog,andmuchmore.
ThefollowingserverconfigurationisrequiredforMySQL8:
Serverconfigurationtype:Basedontheserverconfigurationtype,systemresourceswillbeassignedtotheMySQL8server.
Development: By considering the host as a personal workstation, itconfiguresMySQL8tousetheminimumamountofmemory.
Server: As servers, some other applications are also running on themachine,soitwillconfigureamediumamountofmemory.
Dedicated: In case of a dedicatedmachine for theMySQL8 server, thisoptionconfiguresthemaximumuseofavailablememoryfortheMySQL8server.
Connectivity:ThisoptionindicatestheconnectionfortheMySQL8server.Thefollowingoptionsareavailable:
TCP/IP: This option enables TCP/IP connection with MySQL 8.Users are allowed to define the port number alongwith the firewallsettingfortheportonthenetworkaccess.
Namedpipe:Thisoptionallowsyou todefine thepipelinenamefortheconnection.
Sharedmemory:ThisallowsyoutodefinethememorynamefortheMySQL8server.
Advanced configuration: This configuration enables additional loggingfeatureswhichwillmanage logs in individual files.Users are allowed toconfigurepathsforindividualfiles.Forexample,configuringacustompath
forabinarylog.
MySQL Enterprise Firewall: This option is used for the CommercialEditiononly.Check theEnableEnterpriseFirewalloption toenable thefirewall.
Accountsandroles:Accountsandrolesareusedtomanageaccessrightsfor theusers.During the installationprocess, theMySQL installer allowsyoutosetrootaccountpasswordsanduseraccounts.
Rootaccountpassword: It is requiredtoenterrootpasswordduringtheinstallationprocess.Theinstallerwillcheckthepasswordstrengthandgiveawarningifthereisaviolationofapredefinedpolicy.
MySQLuseraccounts:ThisisanoptionalstepwhereanewMySQLuser account defines with existing user roles. Predefined roles havetheirownprivileges.
Windows service: The MySQL 8 service can be configured in thefollowingtwoways:
Configureas aWindow service:This is thedefault option selectedduringtheinstallationprocess.Itfurtherprovidestwooptions:
Start service on system startup: This option is selected bydefault and will start the MySQL 8 service automatically atsystemstartup.
RunWindow service as: This option allows attaching the useraccount with the MySQL 8 service. By default, the systemaccount is selected where the service is considered as networkservice.
Withacustomuser,itfirstsetsprivilegesfortheuserbyusingthe“localsecuritypolicy”inMicrosoftWindows.
Configure as an executable program: This deselects theWindowsServiceoptionduringtheinstallationprocess.
Pluginsandextensions:Thisstepisavailableforanewinstallation.IftheuserwantstoupgradefromanolderMySQLversion,thentheuserneedstochoosetheReconfigureoptionintheMySQLinstaller.
Advance options: To enable this option, select the Show advanceconfiguration check box in theType andNetworking step. This optionenablestheusertodefineaspecificpathforlogfiles,suchasanerrorlog,agenerallog,aslowquerylog,andbinlog.
Applyserverconfiguration:Oncealltheconfigurationhasbeendonebythe user in theMySQL installer, click on the Execute button to make itavailable.WhentheinstallationhasbeencompletedbypressingtheFinishbutton, the MySQL installer and all the MySQL installed products areavailableintheWindowsStartmenu.
MySQL installer product cataloganddashboard
This section contains details on how theMySQL installer handles productcatalogsandmanagesdashboards.
Theproduct catalog is a componentwhere a list of all the releasedMySQLproductsisavailable,whichsupportMicrosoftWindows.TheMySQLinstallerupdates the catalog on a daily basis and the option is also available for themanual update of the catalog. The product catalog performs the followingactionstomanagethelist:
Populatetheavailableproductslistonaregularbasis
Checkfortheproduct'supdateasinstalledinthehost
Theproductcataloglistsalltheproductsthatareavailableinthedevelopment,general,oranyminorrelease.
TheMySQL installer dashboard provides the facility to manage MySQL
productsinstallationinthehostworkstation.
Thefollowingarethewaystomanageproductsusingthedashboard:
The MySQL installer provides a configuration to update the catalog atspecifictimeintervals.Theusercanenableordisableautomaticupdatesbytheconfiguration.
The dashboard shows a special icon at the product level when its newversionisavailable.
Theusercanmanageproductswiththefollowingactions:
Add:Usetodownloadandinstalloneormoreproducts.
Modify:Usetoaddorremovefeaturesininstalledproducts.
Upgrade: Use to upgrade products. Make sure the checkbox isselected at the product level for upgrading in the upgradeableproductspane.
Remove:Usetouninstallproductsfromthepopulatedlist.
The dashboard provides the reconfiguration feature, where the user canchangealreadyconfiguredoptionsandvalues.Afterapplyingchanges,theMySQLinstallerwillstoptheMySQL8serverandrestartitagaintomakethemavailable.
The dashboard provides the facility to download the products catalogwithoutupgradingit.TheDonotupdateatthistimecheckboxisavailabletocheckcurrentchangesrelatedtoproductswithoutdownloading.
Toperformthisfunctionality,selectthecheckboxandclickonthecataloglink.
MySQLinstallerconsoleTheMySQLinstallerincludestheMySQLinstallerConsole.exefile,whichprovidesthefunctionalitytoexecutecommandsusingCommandPrompt.
This functionality is installed by default during the initial installation of theMySQL installer. There are some commands available to manage MySQLproducts.Toseethedetailsofthesecommands,executethehelpcommand.
<strong>[mysqld]</strong><br/><strong>#setbasedirtoyourinstallationpath</strong><br/><strong>basedir=E:\\mysql</strong><br/><strong>#setdatadirtothelocationofyourdatadirectory</strong><br/><strong>datadir=E:\\mydata\\data</strong>
<strong>E:\>“E:\MySQL\MySQLServer8\bin\mysqld”</strong>
<strong>mysqld:readyforconnections</strong><br/><strong>Version:'8.0.4'socket:''port:3306</strong>
<strong>E:\>“E:\MySQL\MySQLServer8\bin\mysqld”--install</strong>
<strong>E:\>“E:\MySQL\MySQLServer8\bin\mysqld”--installMySQL--defaults-file=E:\my-opts.cnf</strong>
<strong>E:\>“E:\MySQL\MySQLServer8\bin\mysqld”--install-manual</strong><br/><strong>E:\>"E:\MySQL\MySQLServer8\bin\mysqld"--remove</strong>
InstallingMySQL8onLinux
ForMySQL8installationonLinux,varioussolutionsareavailable.Theusercanchoose any of the distributions for his requirement. Th following are thedifferentdistributionsandamongthem,threeofwhicharedescribedindetail:
InstallationusingtheYumrepository
InstallationusingtheAPTrepository
InstallationusingtheSLESrepository
InstallationusingtheRPMpackage
InstallationusingtheDebianpackage
InstallationusingDocker
InstallationusingtheNativeSoftwarerepository
InstallationusingJuju
<strong>shell>sudoyumlocalinstallpackage_name.rpm</strong><br/><strong>shell>yumrepolistenabled|grep"mysql.*-community.*"</strong>
<strong>shell>yumrepolistall|grepmysql</strong>
<strong>shell>sudoyum-config-manager--disablemysql57-community</strong><br/><strong>shell>sudoyum-config-manager--enablemysql80-community</strong>
<strong>mysql80-community]</strong><br/><strong>name=MySQL8.0CommunityServer</strong><br/><strong>baseurl=http://repo.mysql.com/yum/mysql-8.0-community/el/6/$basearch/</strong><br/><strong>enabled=1</strong>
<strong>shell>yumrepolistenabled|grepmysql</strong>
<strong>shell>sudoyuminstallmysql-community-server</strong>
<strong>shell>sudoservicemysqldstart</strong><br/><strong>shell>sudoservicemysqldstatus</strong>
Duringtheinitialstartup,thefollowingtasksareperformed:
TheserverisinitializedTheSSLcertificateandkeyfilesaregeneratedinthedatadirectoryThepluginsnamevalidate_password_pluginisinstalledandenabledAsuperaccountiscreated
<strong>shell>sudoservicemysqldstart</strong>
Aswithotherinstallations,theRPMpackageinstallationalsocreatesfilesanddirectoriesinthesystemonfollowingpath:
FilesorResources LocationClientprogramsandscripts /usr/bin
mysqldserver /usr/sbin
Configurationfile /etc/my.cnf
Datadirectory /var/lib/mysql
ErrorlogfileForRHEL,OracleLinux,CentOS,orFedoraplatforms:/var/log/mysqld.logForSLES:/var/log/mysql/mysqld.log
Valueofsecure_file_priv /var/lib/mysql-files
SystemVinitscriptForRHEL,OracleLinux,CentOS,orFedoraplatforms:/etc/init.d/mysqldForSLES:/etc/init.d/mysql
Systemdservice ForRHEL,OracleLinux,CentOS,orFedoraplatforms:mysqldForSLES:mysql
Pidfile /var/run/mysql/mysqld.pid
Socket /var/lib/mysql/mysql.sock
Keyringdirectory /var/lib/mysql-keyring
Unixmanualpages /usr/share/man
Include(header)files /usr/include/mysql
Libraries /usr/lib/mysql
Miscellaneoussupportfiles(forexample,errormessages,andcharactersetfiles)
/usr/share/mysql
<strong>shell>tar-xvfmysql-server_MVER-DVER_CPU.deb-bundle.tar</strong>
<strong>shell>sudoapt-getinstalllibaio1</strong>
<strong>shell>sudodpkg-preconfiguremysql-community-server_*.deb</strong>
<strong>shell>sudoapt-get-finstall</strong>
TheMySQL8configurationfilesareavailableunderthefollowingpathintheDebianpackage:
Configurationfilesarestoredunder/etc/mysqlThedatadirectoryisstoredunder/var/lib/mysqlBinaries,libraries,andheadersarestoredunder/user/binandunderuser/sbin
Post-installationsetupforMySQL8Post-installationisaprocessthatdescribesthebasicstepsorconfigurationthattheuserhastoperformafterMySQL8installation.
<strong>E:\>bin\mysqld–-initialize</strong><br/><strong>E:\>bin\mysqld--initialize-insecure</strong>
<strong>E:\>bin\mysqld--initialize--basedirE:\mysql--datadir:\mydata\data</strong>
Theusercanalsospecifythesedirectoriesinaseparatefile,knownastheOptionfile,underthemysqldparameter.ThisconfigurationisdescribedindetailundertheOptionfilesectioninthischapter.Whentheuserexecuteseitherofthecommands,mysqldperformsthefollowingstepsintheexecution:
1. Itcheckstheexistenceofthedatadirectory
2. TheMySQL8servercreatesasystemdatabaseanditstable,granttables,helptables,andtimezonetables
3. ItinitializesasystemtablespacewithdatastructureforInnoDBtables
4. Theroot@localhostsuperuseraccountandotherreservedaccountswillbecreated
<strong>shell>mysql-uroot-p</strong><br/><strong>Enterpassword:(entertherandomrootpasswordhere)<br/></strong>
<strong>mysql>ALTERUSER'root'@'localhost'IDENTIFIEDBY'newPassword';</strong>
<strong>mysql-uroot</strong>
<strong>mysql>ALTERUSER'root'@'localhost'IDENTIFIEDBY'newPassword';</strong>
Afterassigningthenewpassword,youhavetousethenewpasswordwheneveryouwanttoconnectwithMySQL8.
<strong>shell>sudoservicemysqldstart</strong>
<strong>shell>sudoservicemysqldstatus</strong>
<strong>shell>mysql-uroot-p</strong>
Theprecedingcommandpromptsforthepassword,soenterthepasswordandpresstheEnterkey.TheMySQLpromptwillbedisplayedwhereyoucanentermysqlcommandsforexecution.Duringtheexecutionoftheprecedingcommands,somecommonproblemsmayarrive,soherewewillpresentthefollowingtroubleshootingsuggestions:
1. Checkintothelogfilestofindtheexacterrorthatoccurredduringtheservicestartup.Asmentionedintheprevioussection,theerrorfileandlogfilesarelocatedunderthedatadirectory.Itsnamingconventionsarehost_name.errandhost_name.log.Byreadingthelastfewlinesofthefile,youcanidentifytheproblemthatoccurredduringthelastcommandexecuted.
2. Checkthattherequiredport/socketisavailable.Thefollowingerrorswillindicatethattherequiredportsandsocketsarenotavailableforuse,meaningthattheyareinusewithotherprograms.Toidentifythis,trackalltheproblemsbydisablingtheservice.Anotherreasonisyourfirewallsettingsareblockingtheaccessoftherequiredport,somodifythefirewallsettingandgivepermissiontotherequiredports:
Can'tstartserver:BindontheTCP/IPport:theaddressisalreadyinuseCan'tstartserver:BindontheUnixsocket
3. DefiningthespecificparametersintotheOptionfileis
recommended.IftheparametersarenotdefineintothefilethenMySQL8willconsiderthedefaultparameters,sorefertoalltheavailableparametersprovidedbyMySQL8beforeusingit.
4. Verifythatthedatadirectorypathandpermissionareproperlydefinedornot.ThisdirectoryisusedasthecurrentdirectoryforMySQL8.Tofindthecurrentlysetpathofthedatadirectory,executethemysqldcommandwiththe--verboseand--helpcommand.IfthedatadirectoryislocatedatadifferentplaceotherthantheMySQLinstallationdirectory,thenusethe--datadiroptionwiththemysqldcommand.Forpermission,youwillgetanErrorcode13,whichindicatesthepermissiondeniederror.Toovercomethisissue,changethepermissionoftherequiredfilesandfolder.Anotherwayistologinwiththerootuser,butthisisnotpossibleinallthescenarios,sothefirstapproachisrecommendedtoovercomethepermissionissue.
<strong>shell>bin/mysqladminversion</strong>
<strong>mysql>mysqlshow</strong><br/><strong>mysql>mysqlshowmysql</strong>
Thefirstcommandshowsthelistofdatabasesavailableintheserver.Listsmayvaryasperthesystem,butmysqlandinformation_schemamustbeavailableinthelist.Thesecondcommandlistsallthetablescreatedunderthemysqldatabase.
UpgradingMySQL8In previousversionsofMySQL, thedata dictionary is stored in the file-basedsystemwhileinMySQL8itisstoredinthedatadictionarystructure.So,theup-gradation process will move the file-based structure into the data dictionarystructure. Up-gradation into MySQL 8 is possible from the MYSQL 5.7 GAversion, which means from 5.7.9 or higher. For non-GA versions of 5.7, up-gradationisnotpossible.Beforestartingtheup-gradationprocess,thefollowingpointsneedtobeunderstood.
UpgradingmethodsTwo methods are in use for up-gradation that are differentiated by theirimplementationmethod.Letusdiscussthesemethodsindetail.
<strong>ALTERINSTANCEROTATEINNODBMASTERKEY;</strong>
<strong>SETGLOBALinnodb_fast_shutdown=1;--fastshutdown</strong><br/><strong>SETGLOBALinnodb_fast_shutdown=0;--slowshutdown</strong>
<strong>mysqladmin-uroot-pshutdown</strong>
<strong>mysql_upgrade-uroot-p</strong>
7. Afterup-gradation,shutdownandrestarttheservertocheckwhetherallthechangeshavebeenappliedornot.
<strong>mysqldump-uroot-p--add-drop-table--routines--events--all-databases--<br/>force>data-for-upgrade.sql</strong>
<strong>mysqld--initialize--datadir=/path/to/8.0-datadir</strong>
<strong>mysqld_safe--user=mysql--datadir=/path/to/8.0-datadir</strong>
<strong>mysql-uroot-p--force<data-for-upgrade.sql</strong>
<strong>mysql_upgrade-uroot-p</strong>
<strong>mysqlcheck-uroot-p--all-databases--check-upgrade</strong>
<strong>SELECTTABLE_SCHEMA,TABLE_NAME</strong><br/><strong>FROMINFORMATION_SCHEMA.TABLES</strong><br/><strong>WHEREENGINENOTIN('innodb','ndbcluster')</strong><br/><strong>ANDCREATE_OPTIONSLIKE'%partitioned%';</strong>
<strong>ALTERTABLEtable_nameENGINE=INNODB;</strong><br/><strong>ALTERTABLEtable_nameREMOVEPARTITIONING;</strong>
<strong>SELECTCONSTRAINT_SCHEMA,TABLE_NAME,CONSTRAINT_NAME</strong><br/><strong>FROMINFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS</strong><br/><strong>WHERELENGTH(CONSTRAINT_NAME)>64;</strong>
5. MakesureMySQL5.7doesn'tcontainfeaturesthatarenotavailableinMySQL8,forexample:
IfatableusedastorageenginethatisnotsupportedbyMySQL8,sowasalteredwiththesupportedstorageengineAconfigurationchangewhereyouuseanoptionorvariablethatisnotavailableinMySQL8
MySQL8downgrading
Downgradingisthereverseprocessofup-gradation,wherewewillmovefromahigher version of MySQL to a lower version ofMySQL. In this section, wecoverhowtodowngradefromMySQL8toMySQL5.7.AdowngradethatdoesnotsupportaversionskipmeansthatdowngradingfromMySQL8toMySQL5.6 isnot supported.Within the sameserieswhereaversion skip is supportedmeansyoucandowngradefromMySQL8.ztoMySQL
8.xbyskippingtheMySQL8.yversion.First,wewillexplainsomebasicpointsthatneedtobeunderstoodbeforestartingdowngrading.
<strong>rmib_logfile*</strong>
<strong>mysqld_safe--user=mysql--datadir=/path/to/existing-datadir</strong>
<strong>mysql_upgrade-uroot-p</strong>
6. ShutdownandrestarttheMySQLserveragaintocheckifallthechangeshavebeenappliedornot.
ForMySQLinstallationbasedonAPT,SLES,andtheYumrepositoryinstallations,in-placedowngradesarenotsupported
<strong>mysqldump-uroot-p</strong><span><strong>--add-drop-table--routines--events</strong><br/><strong>--all-databases--force>data-for-downgrade.sql</strong><br/></span>
<strong>mysqladmin-uroot-pshutdown</strong>
<strong>mysqld--initialize--user=mysql</strong>
<strong>mysqld_safe--user=mysql--datadir=/path/to/new-datadir</strong>
<strong>mysql-uroot-p--force<data-for-upgrade.sql</strong>
<strong>mysql_upgrade-uroot-p</strong>
<strong>mysqladmin-uroot-pshutdown</strong><br/><strong>mysqld_safe--user=mysql--datadir=/path/to/new-datadir</strong>
<strong><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>columns_priv<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span><spanclass="tokenkeyword">component</span><spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>db<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>default_roles<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>engine_cost<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>func<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><span
class="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>general_log<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>global_grants<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>gtid_executed<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>help_category<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>help_keyword<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>help_relation<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>help_topic<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=
</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>innodb_index_stats<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>innodb_table_stats<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span><spanclass="tokenkeyword">plugin</span><spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>procs_priv<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>proxies_priv<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>role_edges<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><span
class="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>server_cost<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>servers<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>slave_master_info<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>slave_relay_log_info<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>slave_worker_info<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>slow_log<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>tables_priv<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=
</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>time_zone<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>time_zone_leap_second<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>time_zone_name<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>time_zone_transition<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><br/><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span>time_zone_transition_type<spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span><spanclass="tokenkeyword"><br/>ALTER</span><spanclass="tokenkeyword">TABLE</span>mysql<spanclass="tokenpunctuation">.</span><spanclass="tokenkeyword">user</span><spanclass="tokenkeyword">TABLESPACE</span><spanclass="tokenoperator">=</span>innodb_file_per_table<spanclass="tokenpunctuation">;</span></strong>
<strong>ALTERTABLEmysql.columns_privENGINE='MyISAM'<br/>STATS_PERSISTENT=DEFAULT</strong>
<strong>ALTERTABLEmysql.userdropCreate_role_priv;</strong><br/><strong>ALTERTABLEmysql.userdropDrop_role_priv;</strong>
InnoDBchanges:Beforestartingin-placedowngrading,shutdownMySQLusingtheinnodb_fast_shutdownoption.Shutdowntheserverwithinnodb_fast_shutdown=0.Removingtheredologsisrecommendedforin-placedowngrading.
Summary
Tochoosethepropersoftwarewithitsversionfordevelopmentisanimportantphase,right?Inthischapter,weunderstoodhowtoselecttheproperversionofMySQL8
by understanding its version pattern. We also learned the execution steps ofMySQL
8 installationusing the installerandcommand line inMicrosoftWindows.Forthe Linux platform, we installed MySQL 8 using the Yum repository, RPMpackage,andDebianpackage.Post-installationdescribesthebasicconfigurationtostartwithMySQL8.
Finally, we explained how to upgrade and downgrade from MySQL 8 withexecutionsteps.
Inthenextchapter,wewilllearnaboutvariousprogramsandutilitiesavailableforMySQL8. Itmainly focuses on how to use these programs forMySQL8alongwithcommand-lineexecutions.
MySQL 8 – Using Programs andUtilities
Inthepreviouschapter,weinstalledMySQL8andgottoknowalternativewaystoinstallMySQL8.WealsolearnedhowtomigrateandupgradetoMySQL8.Thefollowingarethesummarytopicsexplainedinthepreviouschapter:
MySQL8Installation
PostInstallationSetup
MySQL8Upgrading
MySQL8Downgrading
In this chapter, the reader will learn about various programs and utilitiesavailable inMySQL8.The readerwill alsoget toknowhow touseprogramsand utilities in MySQL 8. The reader will learn about using command-lineprogramsusedinMySQL8.
The readerwill learn the syntax for theprogramandhow theyarebeingusedwithspecificoptionstoperformspecificoperations.Thefollowingisasummaryofthetopicscoveredinthischapter.
OverviewofMySQL8programs
MySQL8command-lineprograms
MySQL8clientprograms
MySQL8administrativeprograms
MySQL8environmentvariables
MySQLGUItools
OverviewofMySQL8programs
There are various different programs in the MySQL installation. A briefoverviewof theseprograms iscovered in this section.Upcomingsectionswillcover a detailed description for eachof themand the descriptionwill have itsowninvocationsyntaxandoptionstoperformtheoperation.
MostoftheMySQLdistributionwillhavealltheseprograms,apartfromthosethat are platform-specific; for example, server startup script not used inWindows.RPM
(Red-hatpackagemanager)distributionsareveryspecializedandarepartoftheexceptions toallprogramsavailable indistributions.What isspecializedaboutRPMdistributions?
Well, they have different programs for different operations; for example, oneprogramwillbeexecutedfortheserver,asecondprogramwillbeexecutedfortheclient, and soon. If it looks likeoneormoreprograms ismissing inyourinstallation,thendon'tworry.SeeChapter2,InstallingandUpgradingMySQL8,for information on the types of distributions available andwhat is included inthem.Itmightbethecasethatthedistributionwhichyouhavedoesnotincludealltheprogramsandyouneedtoinstallanadditionalpackage.
EachoftheMySQL8programswillhavetheirownoptions,butmostofthemwillhavea--helpoption thatcanbeused to retrievedescriptionsaboutall theoptionsthattheprogramhas.Forexample,trymysql--helponthecommandline(thatisyourshellorCommandPrompt).
The description on the first few lines will have specific version informationabout MySQL that is installed along with operating system and licenseinformation.ThenextlinewillstartwithUsage:mysql[OPTIONS][database],thatis,a syntax of the program command usage, and later lines describe the optionsavailabletobeusedalongwiththemaspertheusagedescription.Thiswasjustaglimpseofwhatwewillbelookingat:programdetailswithoptions,theirusagesanddefaultoptions,overridingdefaultoptionvaluesonvarious command-lineprograms,clientprograms,administrativeprograms,andsoon.
For detailed information about executing programs and specifying programoptionsinthecommandlineseetheMySQL8commandlineprograms section,whichwillbe followedbya listof installations,clientandserverstartup,andotherutilityprograms.
MySQLprogramsinbrief
LetusstartwiththeMySQLserverprogramsfirst!!
mysqld is thefirstprogram,alsoconsidered tobe themainprogramforMySQLinstallation.
Itworkswithseveral scripts tohelpwithstartingandstopping theserver.Thefollowing are the programs divided into categories based on their operationalscope:
Startupprograms
Installation/upgradationprograms
Clientprograms
Administrativeandutilitiesprograms
Startupprograms
ThestartupprogramsaretheprogramsthatareusedduringMySQLstartupandinitiatetherequiredbackgroundservicesbasedontheconfiguration.
mysqld: This is theMySQL server daemon. All the other client programsinteractwiththedatabaseusingthisserverprogram.Itmustbestartedandberunningatalltimesexceptformaintenance.
mysqld_safe:Thisisoneoftheserverstartupprogramscriptsandattemptstostartthemysqldprogram.
mysql.server:Another server startupprogram script,which is used in thosesystems, uses V-style run directories containing scripts. It starts systemservicesatparticularrunlevels.
Itcallsmysqld_safetostarttheMySQLserver.
mysqld_multi:Asthenamesuggests,thisisastartupprogramscripttostartorstopmultipleMySQLserversonthesystem.
Installation/upgradationprograms
Theprogramsregardingtheoperationsofinstallationandupgradationarelistedherewiththeirrespectiveusage:
comp_err: This program is used to compile error message files from errorsourcefilesanditisusedduringtheMySQLbuildorinstalloperation.
mysql_secure_installation: This program is used to update the securityconfigurationinordertoenablesecurityduringinstallationofMySQL.
mysql_ssl_rsa_setup:Asthenamesuggests,thisprogramisforgeneratingSSLcertificatesandkeyfilesandRSAkey-pairfiles,ifthosefilesaremissingandarerequiredtosupportsecureconnections.
mysql_tzinfo_to_sql: This program gets the content of the host system zoneinfodatabase(filesdescribingtimezones)andloadstheinformationinthetimezonetablesofMySQL.
mysql_upgrade: As the name suggests, it is used for upgrade operations. Itchecksforany incompatibilityandmakesrepairs if it isnecessary. Italso
Clientprograms
TheclientprogramsareamongtheprogramsthatarecommonlyusedtoconnecttotheMySQLdatabaseandperformdifferentqueryoperations:
mysql: This is the most commonly used program. It is an interactivecommand-linetoolforexecutingSQLstatementsdirectlyorusingafileinbatchmode.
Detailed information is followed in the next MySQL 8 command lineprogramssection.
mysqladmin: This is the program responsible for performing variousadministrativeoperations,suchascreatingordroppingdatabases,flushingtables, reloading grant tables, reopening log files, and much more. Theprogram is also used to retrieve information from the server, such asversion,process,andstatus.
mysqlcheck: This is the client program used for maintenance of tables,performinganalysis,checks,repairs,andoptimizingtables.
mysqldump:This is theclientprogram thatdumps theMySQLdatabase to afile in text, SQL, orXML formats. It is commonly known as a databasebackupprogram.
mysqlimport:ThisistheclientprogramthatimportstextfilesintorespectivetablesusingLOAD_DATA_INFILE. It isalsocommonlyknownasthedataimportprogram.
mysqlpump:TheclientprogramwhichdumpsMySQLdatabaseintoSQLfile.
mysqlshow:The client that shows informationof databases, tables, columns,andindexes.
mysqlslap: This is the client program that is used to check client loadcapability for the MySQL server. The program mimics multiple clientsaccessingtheserver.
Administrative and utilitiesprograms
The following are the programs that perform various administrative activities.Theyaredepictedalongwithsomeoftheutilitieswhichhelpinadministrativeoperations:
innochecksum:TheprogramfortheInnoDBofflinefilechecksum.
myisam_ftdump: The utility program gives information of full-text indexes inMyISAMtables.
myisamchk: The programused to check, describe, repair andoptimize MyISAMtables.
myisamlog:TheutilityprogramforprocessingaMyISAMlogfilecontents.
myisampack:Theutilityprogramthatproducessmallerread-onlyMyISAMtablesthroughcompression.
mysql_config_editor: The utility program that enables authenticationcredentials storage in an encrypted and secure login path file namedmylogin.cnf.
mysqlbinlog:Theutilityprogramthatcanreadbinary logfilestatements. Intheeventofaservercrash,abinarylogfileexecutedstatementscanbebighelp.
mysqldumpslow:Theutilityprogramthatcanreadandsummarizethecontentsofaslowquerylog.
Environmentvariables
TheMySQL client programs that communicatewith theMySQL server usinglibrariesusethefollowingenvironmentvariables:
MYSQL_UNIX_PORT:Thisvariable is responsible for thedefaultUnixsocket filewhichwillbeusedforconnectingtoalocalhost
MYSQL_TCP_PORT: This variable is responsible for providing the default portnumberandisusedinTCP/IPconnections
MYSQL_PWD:Thisvariableisresponsibleforprovidingthedefaultpassword.
MYSQL_DEBUG: This variable is responsible for providing debug trace optionsduringdebuggingoperations
TMPDIR: This variable is responsible for providing the directory where thetemporaryfilesandtableswillbecreated
Foradetailedlistandusesofenvironmentvariablesinprograms,
MySQLGUItoolTheMySQLWorkbenchGUI tool, providedbyOracle corporation, is used intheadministrationofMySQLserversanddatabases,forcreating,executing,andevaluating queries. It is also used for migrating schema and data from otherrelational database management systems to be used with MySQL. There areotherGUI tools, includingMySQLNotifier,MySQLforExcel,phpMyAdmin,andmanymore.
MySQL8command-lineprogramsIntheprevioussection,wewentthroughvarioustypesofprogramsprovidedbyMySQL
8andoutlinedtheirusageinbrief.
In this section, we will look at command-line programs and learn aboutexecuting programs from command lines.Wewill take a detailed look at theprovisionforoptionsandhowtheycanbeutilizedfortheadministration.
Executing programs from thecommandlineExecutingprogramsfromthecommandline(shellorCommandPrompt)isoneof themostused formsof administration inMySQL.Plentyofprogramshavebeenaddedalongwithoptionsforadministration.
<strong>shell>mysql--verbose--help</strong><br/><strong>shell>mysql--user=root--password=********mysampledb</strong><br/><strong>shell>mysqldump-urootpersonnel</strong><br/><strong>shell>mysqlshow--help</strong>
Therearenonoptionarguments,argumentswithoutanyleadingdash,givingsupplementaryinformationtotheprogram.Asanexample,ifyouseethesecondlineoftheprecedingexample,ithasathirdnonoptionargumentwithadatabasenamemysampledb,sothecommandmysql--user=root--password=********mysampledbtellsthemysqlprogramthatyouwantedtousemysampledbasthedatabasename.
Argumentsbeginningwithsingleordoubledash(-,--)areusedforspecifyingtheprogramoptions.Specifyingtheprogramoptionsindicatesthetypeofconnectiontheprogramwillconnecttotheserverorwillaffectthemodeofoperation.Syntaxfortheoptionsareexplainedwithdetails,seetheSpecifyingoptionsforprogramssection.
<strong>shell>mysql</strong>
<strong>shell>mysql--host=localhost--user=root--password=mypwdmysampledb</strong>
Asyoucanseeintheprecedingexamplewithspecificoptionvalues,thehostistobeconsideredasthelocalhostandtheuservalueisprovidedasmyname.Thepasswordisalsospecifiedand,finally,anon-optionargumentisspecifiedthattellstheprogramtousemysampledbasthedefaultdatabasename.
Specifyingoptionsforprograms
In theearliersection,wehaveseenhowprogramoptionschangetheoperationmodebasedonargumentvaluesspecified for theoption in theclientprogram.Here we will look at several ways to specify options for MySQL programs.Theseinclude:
Providingtheoptionsonthecommandlinefollowedbytheprogramname.This is the common way of providing options but it will be appliedspecificallytotheexecutionoftheprogramatthattimeonly.
Providingtheoptionsintheoptionsfile that isbeingreadbytheprogrambeforeitstartsexecution.Thisisthecommonwayforprovidingtheoptionsthatyouwanttheprogramtouseeachtimeitexecutes.
Providingtheoptions in theenvironmentvariables.Byusing thismethod,youcanalsospecifyoptionsthatyouwanttoapplyeverytimetheprogramisbeingexecuted.
Ingeneralpractice,usingoptionfilesiscommonlyusedforthispurposebutspecifyingoptionvaluesintheenvironmentvariablesisalsoveryusefulin
some cases; for example, when runningmultipleMySQL instances on aUnixsystem.
TheMySQLprogramcheckswhichoptionsare tobegivenfirstbyexaminingthe related environment variables, then it processes option files, and then itconsiders option arguments in the command line. Thus, the command-lineoptions have the highest precedence and the environment variables have thelowest.However,thereisoneexceptionthatappliesandthatisthemysqld-auto.cnfoptionfile inthedatadirectoryprocessedlast,soit takeshigherprecedencetothecommand-lineoptions.
<strong>shell>mysqladmin--count=1k--sleep=10ping</strong>
Forfilenameoptionsvalues,avoidusingthe~metacharacterbecauseitwillnotbeinterpretedasperexpectation.Optionvaluescontainingspacesmustbeenclosedbyquotationmarkswhenavalueisspecifiedonthecommandline.
<strong>--disable-column-names</strong><br/><strong>--skip-column-names</strong><br/><strong>--column-names=0</strong>
Asyoucanseeintheprecedingexample,the=0suffixandthe--skipand--disableprefixeshavethesameeffect.Itisalsoapplicablewhenturningtheoptiononwiththe=1suffixandthe--enableprefix.
Iftheoptionisspecifiedwiththe--looseprefix,andiftheoptionspecifieddoesnotexist,theprogramwillissueawarninginsteadofexiting.
Forsomeoftheprograms,the--maximumprefixisavailabletobeusedwiththeoptionnameforspecifyingthelimit.Itcanalsobeusedwithenvironmentvariables.
Modifyingoptionswithfiles
MostoftheMySQLprogramscanreadstartupoptionsfromtheoptionfiles,alsosometimescalledconfigurationfiles. It isaveryconvenientwayofprovidingthe options that are commonly used and once specified you need not specifythemeach timeyouexecute theprogram.Tocheckwhether theprogramreadsoption files, use the --help option. For example, consider the mysqld program,whichshoulduse--verboseand--help, if it readsoptionfiles.Thehelpmessagewillindicatewhichoptionfileitlooksforandforwhichoptiongroup:
TheMySQLprogramwiththe--no-defualtsoptiondoesnotreadanyoption files apart from .mylogin.cnf. If the server program startedwith theoption persisted_globals_load systemvariable disabled thentheprogramdoesnotreadthemysqld-auto.cnffile.
Themajorityoftheoptionfilesareplaintextfilesthatcanbecreatedandeditedby any text editors. Exceptions among those files are .mylogin.cnf , which hasloginpathoptions,encryptedbythemysql_config_editorutilityprogram.
MySQLchecksoptionfilesforWindowsandUnixsystemsinspecificorderandfollows the precedence that starts from reading global options, such as in theWindowssystem:
%PROGRAMDATA%\MySQL\MySQL Server 8.0\my.ini and %PROGRAMDATA%\MySQL\MySQL Server
8.0\my.cnf
%WINDIR%\my.iniand%WINDIR%\my.cnf
C:\my.iniandC:\my.cnf
BASEDIR\my.iniandBASEDIR\my.cnf
Thefilespecifiedwith--defaults-extra-file,ifany
Loginpathoptionsin%APPDATA%\MySQL\.mylogin.cnf(clientprogramonly)
For system variables persisted with SET_PERSIST or PERSIST_ONLY (if it is theserverprogram)inDATADIR\mysql-auto.cnf
Similarly, in Unix systems it follows the following order of precedence forreadingoptionfiles:
/etc/my.cnf
/etc/mysql/my.cnf
SYSCONFDIR/my.cnf
$MYSQL_HOME/my.cnf(serverprogramonly)
Thefilespecifiedwith--defaults-extra-file,ifany
~/.my.cnfforuser-specificoptions
~/.mylogin.cnfforuser-specificloginpathoptions(clientprogramonly)
For system variables persisted with SET_PERSIST or PERSIST_ONLY (if it is theserverprogram)inDATADIR\mysql-auto.cnf
Intheprecedingoptions,~referstothecurrentuser'shomedirectory.
Emptylinesinoptionfilesareignored,alongwithcomments.Commentscanbespecifiedusing#or;charactersand#canstartinthemiddleofanylineaswell.
groupgroupisthenameoftheprogramorgroupforwhichoptionsaretobeset.Theyare not case sensitive. Once a group line is added to the option file, all thefollowinglinesapplytothenamedgroupuntilanothergrouplineisspecifiedorattheendoftheoptionfile.
opt_name=valueThisissimilartothe--opt_nameinthecommandlinebutinplaceofthevalue,youcanspecifythevaluewithspaces,whichyoucannotinthecommandline.
IncludedirectivesItispossibleusing!includedirectivesintheoptionfilestoincludeanotheroptionfileand!includedirtosearchforspecificdirectoriestocheckforoptionfiles.Forexample,!include/home/dev/myopt.cnfand!includedir/home/dev, fordirectories.Theonly thing that MySQL does not consider is any order during the directorysearch.
Anyoptionfilestobeusedinthe!includedirdirectiveonaWindowssystemmustendwiththe.inior.cnfextensionandinUnixsystemstheymustendwith.cnf.
Command-line options affectingoptionfilehandlingMost of theMySQL programs support option files. As they affect option filehandling,theymustbegiveninthecommandlineandnotaspartofanoptionfile.
Inordertomakethemworkproperly,theymustbegivenbeforeotheroptions.Some of the exceptions are as follows : --print-defaults might be usedimmediatelyafter--login-path,--defaults-file,ordefaults-extra-file.
--no-defaultsand--print-defaultsarealsousedtomodifyoptionfilehandling.
Setting program variables withoptionsManyMySQLprogramshaveinternalvariablesthatwecansetduringruntimeoperations
using the SET statement and also using the same syntax by which we specifyoptionvalues.This
willworkwhentheprogramisstarted.Forexample,ifweusetheoptionvaluesyntax
thenwehavetospecifylikethis:shell>mysql--max_allowed_packet=16M .Tospecifythe runtime option using the SET, we can specify like this: mysql> SET GLOBAL
max_allowed_packet=16*1024*1024.
To findout if theoptionvariable syntax iscorrect, youcango tomysqlandusethefollowing:mysql>showvariableslike'max%'.
<strong>SETUSER=your_user_name</strong>
<strong>MYSQL_TCP_PORT=3306</strong><br/><strong>exportMYSQL_TCP_PORT</strong>
<strong>setenvMYSQL_TCP_PORT3306</strong>
Thecommandstosetenvironmentvariablesthatareexecutedwillimmediatelyaffecttheprograminexecutionbutifyouwantedtogettheenvironmentvariabletopersist,youneedtospecifyitontheinterfaceprovidedbythesystemoryoumaysetitupinthestartupfilethatthecommandprocessorusesatstartup.OnWindows,thiscanbesetupfromthecontrolpanelsoptiontosetenvironmentvariablesandinUnix,thiscanbesetupbasedonthecommand-lineprocessoryouuse.Forbash,youneedtoputthevaluein.bashrcor.bash_profileandfortcshuse.tcshrc.
Server and server-startupprogramsTherearespecificprogramsprovidedbyMySQLwhichyouneedtoexecutefirstinordertomakeMySQLworkcorrectly.Inthefollowingsections,wewilllookattheserverprogramsandrelatedstartupprogramsthatcanbeusedwithseveraloptionsasperyourrequirement.
mysqld - the MySQL serverprogramTheMySQLserver isadaemonprogram.Allotherprogramsconnectwiththedatabase
through this server, so it shouldbe running at all times.Thedaemonprogramusually
getsstartedfromascriptcalledmysqld_safe.Theprogramscriptisrequired,asitsetstheappropriateenvironmentvariables
andexecutesthemysqldprogramwiththerequiredarguments-optionvalues.
Options
Thefollowingaretheoptionsbriefedindetailforvariousoptionsavailablefromcommandline:
-?,-I,--help:Displaystheusageinformationoftheprogram.
-#debuglevel,--debug=debuglevel:Setsthedebugginglevelasspecified.
-b directory, --basedir=directory: Specifies the base directory used todetermineallotherrelateddirectories.
--big-tables: Used to allow large result sets. They are saved as temporaryresultsinafile.
--bind-address=ip-number:SpecifiestheIPaddresstheserverwillbinditto.
-h directory, --datadir=directory: Specifies the directory where the databasedatafilesarestored.
-l [logfile], --log[=logfile]: Add various log information, which includes
connections and error information. If an argument is not provided, thenhostname.logisusedasthelogfile,andherehostnameisthenameoftheservermachine.
--log-isam[=logfile]: Adds changes to the data (ISAM) files in logs. If anargument is notprovided, then isam.log is used as the log file and the loggenerated by this option can only be read and manipulated with thetheisamlogutility.
--log-update[=number]: Logging the database updates info. The log file getsnamedashostname.num,wherethehostname is thenameoftheservermachineandthenumistheargumenttotheoptionorgeneratesauniquenumberiftheargumentisnotspecified.
-L=language, --language=language: To specify the language (English, French,German,andsoon)fortheserver.
-n,--new:Toenablenewroutines(andpossiblyunsaferoutines).
-S,--skip-new:Todisable/enablenewroutines(andpossiblyunsaferoutines).
-Ovariable=value,--set-variablevariable=value:Tospecifyandsetvalueforthevariables
--pid-file=file:TogetthenameofthefilehavingtheprocessID (PID)oftherunningserver.Thedefaultvaluefor thefile ishostname.pid ,wherethehostnameistheservermachine'sname.
-Pport,--port=port:Tospecifythenetworkportnumber.
--secure: To enable network security checks. But this reduces databaseperformance.
--skip-name-resolve: To specify using only IP numbers (not names) for theconnections.Thisincreasesthenetworkperformance.
--skip-networking: To disable network connections, with only local accessbeingallowed.
--skip-thread-priority:Forgivingallthethreadsthesamepriority.
-Sg:Todisableaccesschecking.Thisallowsall theusersfullaccess toallthedatabases.
-Sl:Tospecifynottoperformthreadlocking.
--use-locking:Toenablethreadlocking.
--socket=file:TospecifythefilenamefortheUnixsocket.
-T,--exit-info: Used to display debugging information during the shuttingdownoftheserver.
-v,-V,and--version:Toshowtheversioninformationoftheserver.
mysqld_safe - MySQL serverstartupscriptThisisthemostrecommendedwaytostarttheMySQLserverinaUnix-basedsystemasitaddsafewsafetyfeatures,suchaslogginginformationtoerrorlogifanyerroroccursatruntimeandrestartingtheserverifthereisanerror.
Insomeof theUnixplatforms,MySQLinstallations fromRPMorDebian packages include the systemd support to manage MySQLstartupandshutdownoperationsandsomysqld_safe isnotinstalledonthosesystems.
mysqld_safe attempts to execute mysqld and to override the default behavior tospecifythenameoftheserverthatyouwantedtoexecute.Theoptiontospecifythe directory using --ledir is also available so that mysqld_safewill look for theserver in the directory.Most of the options in mysqld_safe are also available inmysqldandifthespecifiedoptionisunknowntomysqld_safethenitgetspassedontomysqld.mysqld_safe,whichreadsoutalltheoptionsfromthemysqld,server,andmysqld_safe sections in option files. For backward compatibility, mysqld_safe alsoreadssafe_mysqldsectionsbutyoushouldrenamesuchsectiontobeonthecurrentonethatismysqld_safe.
Asstatedpreviously,thereareverycommonoptionsspecifiedinbothmysqldandmysqld_safe,sosomeoftheoptionsareexcludedinthefollowinglistofoptions:
--core-file-size=size
Tospecifythesizeofthecorefilewhichmysqldshouldcreate.
--ledir=dir_name
Ifmysqld is not able to find the server, then use this option to specify the pathnameof the directory inwhich the server is located. This option can be usedonlyonthecommandline,andnotinoptionfiles.Onplatformsthatusesystemd,thevalueshouldbegiveninthevalueofMYSQLD_OPTS.
--mysqld-safe-log-timestamps
Thisoptionistospecifytheformatfortimestampsinthelogoutputproducedbymysqld_safe.
--mysqld=prog_name
To specify the server program name contained in the ledir directory that youwanttostart.Ifmysqld_safecannotfindtheserver,usethe--lediroptiontospecifythepathnametothedirectorywheretheserverwiththespecifiednameislocated.Thisoptionisonlyacceptedonthecommandline,andnotfromtheoptionfiles.
--open-files-limit=count
Thenumberoffileswhichmysqldcanopen.
--plugin-dir=dir_name
Tospecifythepathandnameoftheplugindirectory.
--timezone=timezone
Thisoptionistosetthetimezoneenvironmentvariabletothegivenoptionvalue,dependingonoperatingsystemtimezonespecificationformats.
--user={username|user_id}
Runthemysqldserverasifyouhavethenameoftheuser.Specifytheuser_nameorspecifythenumericuserIDasuser_id.
<strong>basedir=dir_name</strong>
<strong>datadir=data_dir</strong>
<strong>pid-file=file_name</strong>
<strong>service-startup-timeout=seconds</strong>
Tospecifyinsecondshowlongtowaitforconfirmationoftheserverstartup.Iftheserverdoesnotstartwithinthistime,mysql.serverexitswithanerrorindication.Thedefaultvaluefortheoptionis900secondsandavalueof0meansnottowaitatallforstartupandprovidingnegativevaluesmeanstowaitforever(thereshouldnotbeatimeout).
<strong>shell>mysqld_multi[options]{start|stop|reload|report}[GNR[,GNR]...]</strong>
Intheprecedingsyntax,start,stop,reload(stopandrestart)andreportreferstotheoperationtobeperformed.BasedontheGNRlistvaluesspecified,youcanperformtargetedoperationsonsingleormultipleservers.
PleasemakesurethatthedatadirectoryforallserversisfullyaccessibletotheUnixaccountbywhichthespecificmysqldprocessisstarted.Donotusearootaccountunlessyouknowexactlywhatyouaregoingtodowithit.
InstallationprogramsTheprogramsdiscussedinthissectionareusedduringtheinstallationprocessorwhen upgrading theMySQL, somake sure you understand it correctly beforedoinganymodificationsontheprogram.
comp_err - compiling theMySQLerrormsgfileThis creates the errmsg.sys file which is used by mysqld to identify the errormessages and display individual error codes. comp_err is normally runautomaticallywhenMySQLisbuilt.Theerrmsg.sysfileiscompiledfromthetextfile located inMySQLdistributionsat sql/share/errmsg-utf8.txt. It alsogeneratessql_state.h,mysqld_ername.h,andmysqld_error.hheaderfiles.
comp_err has several options and can be retrieved using the --help option in thepreviouscommand.
<strong>shell>mysql_secure_installation[options]</strong>
Youcanusethe--helpoptionhereandretrievealistofotheroptionswheneverrequired.
<strong>shell>mysql_ssl_rsa_setup[options]</strong>
Usethe--helpoptionhereandretrievealistofotheroptionsifrequired.
Usingsslmysql_ssl_rsa_setuplowersthebarriertosslandmakesiteasiertogeneratetherequiredfilesbutthefilesthataregeneratedareself-signed,whichisnotverysecure.YoucanconsiderobtainingaCAcertificatefromtherespectiveauthority.
<strong>shell>mysql_tzinfo_to_sqltz_dir</strong><br/><strong>shell>mysql_tzinfo_to_sqltz_filetz_name</strong><br/><strong>shell>mysql_tzinfo_to_sql--leaptz_file</strong>
Afterrunningthemysql_tzinfo_to_sqlprogram,itishighlyrecommendedtorestarttheserversothatitwillnotuseanypreviouslycachedtimezonedata.
mysql_upgrade - checking andupgradingMySQLtablesAs the name suggests, this is used for upgrade operations. It checks for anyincompatibilityandmakes repairs if it is necessary and also updates the granttableswithanychangesinnewversionsofMySQL.Italsoupdatesthesystemtables so you can take advantage of any new privilege or compatibility thatmighthavebeenaddedinthenewerversion.
BeforeperforminganupgradealwaysbackupyourcurrentMySQLinstallation.
Ifmysql_upgradefindsthatatablehaspossibleincompatibility,itperformsatablecheckandattemptsrepairingthetable,andifitcannotrepairit,itwillaskforamanualtablerepair.
mysql_upgradeshouldbeexecutedeachtimeMySQLisupgraded.Itcommunicatesdirectly with the MySQL server and sends the required SQL statements toperformanupgrade.
Oncewerunmysql_upgrade,weshouldrestart theserver. Ifanychangesmade tosystemtablesaretakenintoeffect,beforerunningit,youshouldmakesuretheserverisrunning.
Executemysql_upgradewiththefollowingsyntax:
shell>mysql_upgrade[options]
Youcanusethe--helpoptionhereandretrievealistofotheroptionswheneverrequired.
MySQL8clientprogramsThe MySQL 8 client programs are the programs that are commonly used toconnecttotheMySQLdatabaseandperformdifferentqueryoperations.
Theprogramsinformationdetailedinthefollowingsubsectionincludesmysql—commandlinetoolswithmanycommandsandrelatedoptionsandconfigurationforlogging,mysqlcheck,mysqldump,mysqlimport,mysqlsh,mysqladminandsoon.
mysql-thecommand-linetoolThis is themost commonly used program.The command-line tool is used forexecutingSQLstatementsdirectlyorusingafileinbatchmode.Ithassupportforboth interactiveandnon-interactivemodes. In this section,wewill lookatthemysqlcommand lineand thevariousoptions,commands, logging,andotherrelatedprograms.
mysqloptions
mysqlisacommandlinetoolthathasbeenprovidedforalongtimeandsoithasplentyofoptions togetyourworkdone.The following is the tableofoptionswithformatsanddescriptions:
Format Description--auto-rehash Enablesautomaticrehashing--auto-vertical-
output Enablesautomaticverticalresultsetdisplay--batch Donotusethehistoryfile--binary-as-hex Displaysbinaryvaluesinhexadecimalnotation
--binary-modeDisables\r\n-to-\ntranslationandtreatmentof\0asend-of-query
--bind-addressUsesspecifiednetworkinterfacetoconnecttoMySQLserver
--character-sets-dir Directorywherecharactersetsareinstalled--column-names Writescolumnnamesinresults--column-type-info Displaysresultsetmetadata
--commentsAscertainswhethertoretainorstripcommentsinstatementssenttotheserver
--compress Compressesallinformationsentbetweenclientandserver
--connect-expired-
password
Indicatestoserverthatclientcanhandleexpiredpasswordsandboxmode
--connect_timeout Numberofsecondsbeforeconnectiontimeout--database Thedatabasetouse
--debugWritesdebugginglog;supportedonlyifMySQLwasbuiltwithdebuggingsupport
--debug-check Printsdebugginginformationwhenprogramexits
--debug-infoPrintsdebugginginformation,memory,andCPUstatisticswhenprogramexits
--default-auth Authenticationplugintouse--default-character-
set Specifiesdefaultcharacterset--defaults-extra-
file Readsnamedoptionfileinadditiontousualoptionfiles--defaults-file Readsonlynamedoptionfile--defaults-group-
suffix Optiongroupsuffixvalue--delimiter Setsthestatementdelimiter--enable-cleartext-
plugin Enablescleartextauthenticationplugin--execute Executesthestatementandquit--force ContinuesevenifanSQLerroroccurs--get-server-public-
key PathnametofilecontainingRSApublickey--help Displayshelpmessageandexit--histignore Patternsspecifyingwhichstatementstoignoreforlogging--host ConnectstoMySQLserverongivenhost--html ProducesHTMLoutput
--ignore-spaces Ignoresspacesafterfunctionnames
--init-command SQLstatementtoexecuteafterconnecting--line-numbers Writeslinenumbersforerrors--local-infile EnablesordisableforLOCALcapabilityforLOADDATAINFILE--login-path Readsloginpathoptionsfrom.mylogin.cnf--max_allowed_packet Maximumpacketlengthtosendtoorreceivefromserver
--max_join_size Theautomaticlimitforrowsinajoinwhenusing--safe-updates
--named-commandsEnablesnamedmysqlcommands
--net_buffer_length BuffersizeforTCP/IPandsocketcommunication--no-auto-rehash Disablesautomaticrehashing--no-beep Donotbeepwhenerrorsoccur--no-defaults Readsnooptionfiles
--one-databaseIgnoresstatementsexceptthoseforthedefaultdatabasenamedonthecommandline
--pager Usesthegivencommandforpagingqueryoutput--password Passwordtousewhenconnectingtoserver--pipe OnWindows,connecttoserverusingnamedpipe--plugin-dir Directorywherepluginsareinstalled--port TCP/IPportnumbertouseforconnection--print-defaults Printdefaultoptions--prompt Settheprompttothespecifiedformat--protocol Connectionprotocoltouse--quick Donotcacheeachqueryresult--raw Writescolumnvalueswithoutescapeconversion
--reconnectIftheconnectiontotheserverislost,automaticallytriestoreconnect
--i-am-a-dummy,--
safe-updates
AllowsonlyUPDATEandDELETEstatementsthatspecifykey
values--secure-auth Donotsendpasswordstoserverinold(pre-4.1)format
--select_limit TheautomaticlimitforSELECTstatementswhenusing--safe-updates
--server-public-key-
path PathnametofilecontainingRSApublickey
--shared-memory-
base-name
Thenameofsharedmemorytouseforshared-memoryconnections
--show-warnings Showswarningsaftereachstatementifthereareany
--sigint-ignoreIgnoresSIGINTsignals(typicallytheresultoftypingControl+C)
--silent Silentmode--skip-auto-rehash Disablesautomaticrehashing--skip-column-names Donotwritecolumnnamesinresults--skip-line-numbers Skipslinenumbersforerrors--skip-named-
commands Disablesnamedmysqlcommands--skip-pager Disablespaging--skip-reconnect Disablesreconnecting
--socketForconnectionstolocalhost,theUnixsocketfileorWindowsnamedpipetouse
--ssl-ca PathoffilethatcontainslistoftrustedSSLCAs
--ssl-capathPathofdirectorythatcontainstrustedSSLCAcertificatesinPEMformat
--ssl-cert PathoffilethatcontainsX509certificateinPEMformat--ssl-cipher Listofpermittedcipherstouseforconnectionencryption--ssl-crl Pathoffilethatcontainscertificaterevocationlists
--ssl-crlpathPathofdirectorythatcontainscertificaterevocationlistfiles
--ssl-key PathoffilethatcontainsX509keyinPEMformat
--ssl-mode Securitystateofconnectiontoserver
--syslog Logsinteractivestatementstosyslog--table Displaysoutputintabularformat--tee Appendsacopyofoutputtonamedfile--tls-version Protocolspermittedforencryptedconnections--unbuffered Flushesthebufferaftereachquery--user MySQLusernametousewhenconnectingtoserver--verbose Verbosemode--version Displaysversioninformationandexit
--verticalPrintsqueryoutputrowsvertically(onelinepercolumnvalue)
--waitIftheconnectioncannotbeestablished,waitandretryinsteadofaborting
--xml ProducesXMLoutput
Reference:https://dev.mysql.com/doc/refman/8.0/en/mysql-command-options.html
Togetmoreinformationonindividualoptions,usethatoptionalongwiththe--helpoption.
mysqlcommandsEachof theSQLstatements thatyouissuearesent to theserverforexecution.Thereisalsoalistofcommandsthatmysqlitselfinterprets.Togetthelistofallthosecommands,type\hor\helpatthemysql>prompt.
Eachofthecommandshavebothlongandshortformthatcanbeused;exceptshort form cannot be used inmulti-line comments. The long form is not casesensitivebuttheshortformcommandiscasesensitive.
help[arg],\h[arg],\?[arg],?[arg]Thehelparg[]commandisusedtodisplayhelpmessages,alongwithlistingalltheavailablecommandsinmysql.
connect[db_namehost_name], \r[db_namehost_name]Toreconnecttheserverbyprovidingdatabaseandhost_namearguments.
mysqlloggingThemysqlprogramcandologgingasperthefollowingtypes.
On Unix systems, it writes the logs to the history file with the default name.mysql_historyinthehomedirectory.
Onallplatforms,ifthe--syslogoptionisprovided,itwritesthestatementstothesystemloggingimplementation.
OnUnix,itissyslog,onWindows,itiseventlogs,onLinuxdistributions,itoftengoestothe/var/log/messagefile.
<strong>mysql>helpsearch_string</strong>
<strong>mysql>helpme</strong><br/><strong>Nothingfound</strong><br/><strong>Pleasetrytorun'helpcontents'froalistofallaccessibletopics</strong>
Ifsearch_stringmatchesmultiplecontentsofatopic,thenitshowsalistofmatchingtopicitems.Atopiccanalsobeusedassearch_stringandlooksfortheentryforthetopic.Italsocontainsthewildcardcharacter%and_,whichhavethesamemeaningformatchingoperationsperformedbytheLIKEoperator.
<strong>shell>mysqldb_name<text_file</strong>
<strong>shell>mysql<text_file</strong>
<strong>mysql>sourcefile_name</strong>
Byusingthe--verboseoption,eachstatementgetsdisplayedjustbeforetheresultproducedbyit.
<strong>shell>mysqladmin[options]command[command-arg][command[command-org]]...</strong>
mysqladminsupportsplentyofprogramcommands,startingfromcreatedb_nametocreatenewdatabasewithnamedb_name,debugtogetdebuginformation,dropdb_nametodropadatabase,flush-xxxx,wherexxxxcanbereplacedwithlogs,hosts,privileges,status,tables,threads,andsoon.killidtokilltheserverthreadormultiplethreads,passwordnew_passwordtosetanewpassword,pingtochecktheserver'savailability,shutdowntostoptheserver,start-slavetostartreplicationonaslaveserver,stop-slavetostopreplicationontheslaveserver,variablestodisplayserversystemvariablesandtheirrespectivevalues.
mysqladminstatuscommandsgiveresultswithvaluesofuptime,threads,questions,slowqueries,opens,andflushtableswithrelevantinformation.
Alongwiththecommandlist,thereareoptionsthatcomeinhandywhenretrievingspecificinformationfromtheserver.Suchinformationcanberetrievedusingthemysqladmin--helpcommand.
<strong>shell>mysqlcheck[options]db_name[tbl_name...]</strong><br/><strong>shell>mysqlcheck[options]--databasesdb_name...</strong><br/><strong>shell>mysqlcheck[options]--all-databases</strong>
mysqlcheckhasaspecialfeaturethatisthedefaultbehaviorofcheckingtables.Itcanbechangedbyrenamingthebinary,suchasrenamingmysqlchecktothemysqlrepairprogrambycreatingacopyofmysqlcheckandaddingasymboliclinktomysqlcheck,afterwhichmysqlrepaircanrepairtablesinstead.Thisalsoworkswiththemysqlanalyzeandmysqloptimizeoptionstomakethemthedefaultoperationforthemysqlcheckcommand.
Similartootheradministeringprograms,thisprogramalsohavemanyoptionsthatcanbeusedtogetspecificinformation,andbyusingthemysqlcheck--helpcommand,alistofoptionscanberetrieved.
mysqldump - a database backupprogramThis program is a utility program used for making a logical backup bygenerating a set of SQL statements which can be executed to reproduce theoriginaldatabasetabledataandobjectdefinition.Itdumpsoneormoredatabasefor backup or may transfer to another SQL server. It can also generate dataoutputindifferentformats,suchasCSV,XML,orotherdelimitedtextfiles.
<strong>shell>mysqldump[options]db_name[tbl_name...]</strong><br/><strong>shell>mysqldump[options]--databasesdb_name...</strong><br/><strong>shell>mysqldump[options]--all-databases</strong>
Therearemorethan25optionsavailableformodifyingtheoperationofthemysqldumpcommandandtheycanberetrievedbyusingthemysqldump--helpcommand.Specificmodificationscanbeusedinthiscommandfordebuggingoptions,helpoptions,connectionoptions,DDLoptions,andsoon,basedonyourrequirement.
<strong>shell>mysqlimport[options]db_nametextfile1[textfile2...]</strong>
OptionsforthecommandscanbespecifiedontheCLIorinthe[mysqlimport]or[client]groupoftheoptionsfileasperyourrequirement.Itprovidedoptionstoretrieveandmodifyimportoperationsforthedata,suchasusingadifferentdelimiterformat,debugging,forcingthepathofafile,providingdefaultvalues,ignoringandlockingtables,andsoon,whichcanberetrievedbyusingthemysqlimport--helpcommand.
<strong>shell>mysqlpump--all-databases</strong>
<strong><spanclass="tokenprompt">shell></span><spanclass="tokencommand">mysqlpump</span><spanclass="tokenconstant">--include-databases</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>db1,db2</span><spanclass="tokenconstant">--exclude-tables</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>db1.t1,db2.t2</span></strong>
Theprecedingcommanddumpsdatabasesdb1anddb2butitwillexcludetablet1fromthedatabasedb1andtablet2fromthedatabasedb2.
mysqlpumpusesparallelismtoachievetheconcurrentprocessinganditcanbebetweendatabasesorwithinadatabase.--default-parallelism=NspecifiesthedefaultnumberofthreadsusedinthequeuecreatedbytheprogramandthevalueofNis2bydefault.--parallel-schemas=[N:]db_listsetsuptheprocessingqueueasperthedatabasenamelistprovided.Thus,additionalqueuesandthenumberofthreadscanbecontrolled.
mysqlpumpdoesnotdumpperformance_schema,nbdinfo,orsysbydefaultbutcandosobyspecifyingthe--include-databasesoptionandsimilarlyitalsodoesnotdumptheINFORMATION_SCHEMA.
mysqlsh-theMySQLShellThe advanced command line client and editor for MySQL is the very wellknownMySQLShell.IthascapabilitiesforscriptinginPythonandJavaScript.WhenconnectedtotheMySQLserverusingtheXProtocol,theXDevAPIcanwork with documents and relational data. It includes the AdminAPI, whichenablesyoutoworkwithanInnoDBcluster.
MySQLShell hasmanyoptions associatedwith it, but the important ones arelistedasfollows:
--port=port_num,-Pport_num
TheTCP/IPportnumbertousewithport_num.Thedefaultportis33060.
--node
CreatesanodesessiontoasingleserverusingtheXProtocolandisdeprecatedin8.0.3.
--js
StartsJavaScriptmode.
--py
StartsPythonmode.
--sql
StartsSQLmode.
--sqlc
StartsSQLmodeinClassicSession.
--sqln
StartsSQLmodeinNodeSession.
--sqlx
StartsSQLmodebycreatinganXprotocolconnection.
--ssl*
Optionsbeginningwith--ssl specifyconnecting theserverusingSSLandalsofinding certificates and SSL keys. It works the same way as for theMySQLServerand itacceptsSSLoptions :--ssl-crl,--ssl-crlpath,--ssl-mode,--ssl-ca, --ssl-capath,--ssl-cert,--ssl-cipher,--ssl-key,--tls-version.
Otheroptionsnotlistedcanberetrievedusingthemysqlsh--helpcommand.
mysqlshow - showing database,table,andcolumninformationThis is the client mainly used to quickly check which databases, their tables,columns,andindexesexistornot. Itprovidesaninterfacetosomeof theSQLSHOWstatements.
Theexecutionsyntaxforthecommandisasfollows:
shell>mysqlshow[options][db_name[tbl_name[col_name]]]
By executing the preceding command, you will get information about thedatabases,tables,orcolumnsforwhichyouhaveprivileges:
AlistofdatabasesisshownifnodatabaseisgivenAlistofallmatchingtablesinthedatabaseisshownifnotableisgivenA list of all the matching columns and column types in the table isdisplayedifnocolumnisgiven
Inthepreviouscommandexecution,ifyouusedtheSQLwildcardcharacter(*,?,%,_), thennamesthatarematchedby thewildcardaredisplayed.*and?wildcardcharacters,ifgiven,areconvertedinto SQL % and _ wildcard characters. It might create confusionwhen trying to display a columnwith a table or a column namewith_inthename,itdisplaysonlynamesmatchingthepattern,butit can be fixed easily by adding an extra % at the end of thecommandlineasaseparateargument.
Theprogramhasmanyoptionstogetthedesiredinformationbyusingspecificoptionarguments.Afewoftheimportantonesaregivenasfollows:
--character-sets-dir=dir_name
Specifiesthedirectorynamewherecharactersetsareinstalled.
--compress,-C
Ifboththeclientandtheserversupportcompression,alltheinformationsentiscompressed.
--enable-cleartext-plugin
Enablesthecleartextauthenticationplugin.
--get--server-public-key
Requests the RSA public key from the server required for key-pair-basedpassword exchange. Also, if a client connects to the server using a secureconnection,RSA-basedpasswordexchangeisnotneededandisignored.
--keys,-k
Displaystableindexes.
--ssl*
Specifies options starting with --ssl to connect to the server using an SSLconnectionbyusingcertificatesandSSLkeys.
Manyotheroptionsnotlistedherecanberetrievedbyusingthemysqlshow--helpcommand.
<strong>shell>mysqlslap[options]</strong>
<strong>mysqlslap<spanclass="tokenconstant">--delimiter</span>=<spanclass="tokenatrule">";"</span><spanclass="tokenconstant">--create</span>=<spanclass="tokenatrule">"CREATETABLEt(iint);INSERTINTOtVALUES(21)"</span><spanclass="tokenconstant">--query</span>=<spanclass="tokenatrule">"SELECT*FROMt"</span><spanclass="tokenconstant">--concurrency</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=2</span>0</span><spanclass="tokenconstant">--iterations</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=1</span>00</span></strong>
<strong>mysqlslap<spanclass="tokenconstant">--concurrency</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=7</span></span><spanclass="tokenconstant">--iterations</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>20</span><spanclass="tokenconstant">--number-int-cols</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>2</span><spanclass="tokenconstant">--number-char-cols</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=2</span></span><spanclass="tokenproperty">--auto-generate-sql</span></strong>
Here,mysqlslapwillbuildastatementwithatableoftwoINTcolumnsandtwoVARCHARcolumnsandsevenclientsquerying20timestoeachofthem.Italsosupportsspecifyingstatementfilesforcreatingandqueryingseparatelyandrunningtheloadtest.Itprovidesmanysuchalterationsinloadtestingexecutionwithoptionsthatyoucancheckbyexecutingthemysqlslap--helpcommandonthecommandline.
MySQL8administrativeprogramsThis section describes different administrative programs along with someutilities that will help in doing administration operations such as performingchecksum,compressionandextraction,andsoon.
ibdsdi - InnoDB tablespace SDIextractionutilityThis is a utility program that extracts serialized dictionary information fromInnoDB tablespace files. Serivalized dictionary information that is SDI datawillalwaysbepresent inallpersistent InnoDB tablespace files. It canbe runon file-per-tableontablespacefilesandgeneraltablespacefiles,systemtablespacefiles,anddatadictionarytablespacefiles,butusingatemporarytablespaceorundoingatablespaceisnotsupported.
ibd2sdicanbeusedwhiletheserverisofflineoratruntime.ItreadsuncommitteddataofSDIfromaspecifiedtablespaceandundoeslogsandredoeslogsthatarenotaccessible.
Executionfortheidb2sdiwilllooklikethefollowingcommandline:
shell>ibd2sdi[options]file_name1[file_name2file_name3...]
ibd2sdi also supports multiple tablespaces but does not run on more that onetablespace at a time as the InnoDB system tablespace. Specifying each file willworkasfollows:
shell>ibd2sdiibdata1ibdata2
ibd2sdioutputsSDIdatainJSONformat.
Therearemanyoptionsavailablefortheprogramwhichcanberetrievedusingtheibd2sdi--helpcommand.
<strong>shell>innochecksum[options]file_name</strong>
innochecksumcommandalsohasfewoptionstodisplayinformationofthepagesbeingverifiedandcanberetrievedwithinnochecksum--helpcommand.
myisam_ftdump - displaying full-textindexutilityThis is a utility for displaying information about MyISAM tables and FULLTEXTindexes.Itwillscananddumptheentireindex,whichcanbeaslowandlengthyprocess.Iftheserverisalreadyrunning,thenyouneedtomakesuretoinsertaFLUSHTABLESstatementfirst.
Executionforthemyisam_ftdumpcommandwilllooklikethefollowingcodeblock:shell>myisam_ftdump[options]<table_name><index_num>
Inthepreviousexample,table_nameshouldbethenameoftheMyISAMtablewiththe.MYIindexextension.
Suppose the test database has a table named mytexttable with the followingdefinition:CREATETABLEmytexttable ( id INTNOTNULL, txt TEXTNOTNULL,PRIMARYKEY(id),FULLTEXT(txt))ENGINE=MyISAM;
Theindexcreatedonidis0onFULLTEXTindexandonthetxtitis1.Iftheworkingdirectory is the test database directory, then execute myisam_ftdump as follows:shell>myisam_ftdumpmytexttable1
myisam_ftdump can also be used to generate a list of index entries in order offrequencyofoccurrenceasfollows(thefirstlineinWindowsandthesecondlineforUnixsystems):shell>myisam_ftdump-cmytexttable1|sort/Rshell>myisam_ftdump-cmytexttable1|sort-r
myisam_ftdump also has several options that can be retrieved by using themyisam_ftdump--helpcommand.
myisamchk - MyISAM table-maintenanceutility
myisamchk is a command line tool for getting information about database tables,checking,repairing,
andoptimizingnon-partitionedMyISAMtables.ItworkswithMyISAMtables.
CHECK_TABLEandREPAIR_TABLEstatementscanalsobeusedtocheckandrepairMyISAMtables.
Executionforthemyisamchkcommandisshowninthefollowingcodeblock:
shell>myisamchk[OPTIONS]tables[.MYI|.MYD]
Beforerunningthemyisamchkcommand,youmustmakesurethatanyotherprogramisnotusingthetables.Otherwise,
itwilldisplaywarningmessagesaying:warning:clientsareusingor haven't closed the table properly. To do this effectively, shutdowntheMySQLserverorlockalltablesbeingused
bymyisamchk.
<strong>shell>myisamlog[options][file_name[tbl_name]...]</strong><spanclass="tokenpunctuation"><br/></span>
Thedefaultoperationismarkedasanupdateandifrecoveryisdoneallwrites,updates,anddeletesaredoneanderrorsarecountedonly.myisam.logisthedefaultlogfilename.
Theprogramhassomeoptionsusedtospecifyoffsets,recovery,openanumberoffiles,andafewmore,whichcanberetrievedbyusingthemyisamlog--helpcommand.
myisampack - generatingcompressed, read-only MyISAMtablesThisisautilityprogramthatcompressesMyISAMtables.Itcompresseseachofthecolumnsintablesseparatelyandcompressesthedatafilebyabout40%to70%.
MySQL preferably uses the mmap() function to do memory mapping oncompressedtables;otherwise,itusesnormalread/writefileoperations.
myisampackdoesnotsupportpartitionedtables.
Oncetablesarepacked,theybecomereadonly.
Stopping theserverand thengoingforcompress tables issafeway toperformthecompressoperation.
Executionsyntaxformyisampack looks like the followingblockon thecommandline:
shell>myisampack[options]file_name...
Specifytheindexfilenamewithorwithoutthe.MYIfileandalsoaddthepathnameifyouarenotinthedatabasedirectory.
Aftercompressingatablewithmyisampack,youshouldusemyisampack-rqtorebuildthe indexesof thecompressed table. It supports someof thecommonoptions,such as versioning, debugging, and so on along with specific compressionchecks,suchas--test,--backup,--join=big_tbl_name,--silent,andsoon.Ifyouwanttocheckindetail,youcanexecutethemyisampack--verbose--helpcommandonthecommandline.
mysql_config_editor - MySQLconfigurationutilityThisisautilitytostoreandupdatetheauthenticationcredentialsinanencryptedloginpathfilewiththename.mylogin.cnf.
Forfurtherdetails,usethemysql_config_editor--verbose--helpcommandtoexecuteonthecommandline.
<strong>shell>mysqlbinlog[options]log_file...</strong>
Thereareseveraloptionsthatmodifytheformatoftheoutputandusageofmysqlbinlog.Theyarelistedasfollows:
Itcanbeconvertedintoahexdumpformatthatcontainsbyteposition,eventtimestamp,andtypeoftheeventthatoccurredItalsoprovidesaroweventdisplayformatthatdisplaysdatamodificationsinformationintheformofpseudo-SQLstatementsItisalsousedformakingabackupofthebinarylogfilesbyprovidingtherequiredoptionvalues,suchasapathforthebackupfileandtypeorformatoftheoutputinthebackupWhenyouconnecttotheMySQLserverusingmysqlbinlog,itprovidesaspecificserverIDtoidentifyitselfandrequestsbinarylogfilesfromtheserver
Theprogramhassomecommonoptionswhicharenotmentionedbutcanberetrievedusingthemysqlbinlog--helpcommand.
<strong>shell>mysqldumpslow[options][log_file...]</strong>
Youcanmodifytheoutputbyusingsomeoptions,suchaslimitingthenumberofresult-tN,whereNisthenumberofqueryresultstobedisplayed.-sstandsforsorttypebyquerytime,locktime,orbyrowscount,and-rforreversingthesortorder.
MySQL8environmentvariables
Inthissection,wewilllookatthenumberofenvironmentvariablesthatareused
directly or indirectly for different MySQL programs, changing their behaviorwith
theuseofenvironmentvariables.
Optionsprovidedonthecommandlinetakeprecedenceovervaluesspecifiedinthe
option files and on the environment variables, and similarly values in optionstake
precedenceover the environmentvariables; so inmost cases, it ispreferred touse
anoptionfileinsteadofenvironmentvariablestomodifythebehavior.
The following is the list of environment variables and descriptions for thevariable:
CXX:ThenameofyourC++compilerforrunningCMake
CC:ThenameofyourCcompilerforrunningCMake
DBI_USER:ThedefaultusernameforPerlDBI
DBI_TRACE:TraceoptionsforPerlDBI
HOME:Thedefaultpathforthemysqlhistoryfileis$HOME/.mysql_history
LD_RUN_PATH:Usedtospecifythelocationoflibmysqlclient.so.
LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN:Enables the mysql_clear_passwordauthenticationplugin
LIBMYSQL_PLUGIN_DIR:Directoryinwhichtolookforclientplugins
LIBMYSQL_PLUGINS:Clientpluginstopreload
MYSQL_DEBUG:Debugstraceoptionswhendebugging
MYSQL_GROUP_SUFFIX:optionalgroupsuffixvalue(suchasspecifying--defaults-group-suffix)
MYSQL_HISTFILE: The path to the mysql history file; if this variable is set, itsvalueoverridesthedefaultfor$HOME/.mysql_history
MYSQL_HISTIGNORE:Patternsspecifyingstatementsthatmysqlshouldnotlogto$HOME/.mysql_history,orsyslogif--syslogisgiven
MYSQL_HOME:Thepath to thedirectory inwhich theserver-specificmy.cnf fileresides
MYSQL_HOST:Thepath to thedirectory inwhich theserver-specificmy.cnf fileresides
MYSQL_PWD: The default password when connecting to mysqld; using this isinsecure
MYSQL_TCP_PORT:DefaultTCP/IPportnumber
MYSQL_TEST_LOGIN_FILE:Nameofthe.mylogin.cnfloginpathfile
PATH:UsedbytheshelltofindMySQLprograms
PKG_CONFIG_PATH:Locationofthemysqlclient.pcpkg-configfile
TMPDIR:Directoryinwhichtemporaryfilesarecreated
TZ:Thisshouldbesettoyourlocaltimezone
UMASK:User-filecreationmodewhencreatingfiles
UMASK_DIR:User-directorycreationmodewhencreatingdirectories
USER:DefaultusernameonWindowswhenconnectingtomysqld
MYSQL_TEST_LOGIN_FILE is the pathname for the login path file that is created bymysql_config_editor.
TheUMASKandUMASK_DIRvariablesareusedasmodesinsteadofmasks.
It is necessary to set PKG_CONFIG_PATH if using pkg-config for building MySQLprograms.
MySQLGUItoolsTherearemanyMySQLGUItoolsavailableforperformingvariousoperations,startingwithcreatingdatabasestoperformingdailyadministrationtasks.
MySQLWorkbench
MySQLWorkbench is a graphical tool to work with the MySQL server anddatabases.It
fully supportsMySQLversions 5.1 and above. In this section,wewill brieflydiscuss
thecapabilitiesofMySQLWorkbench.
FivemainfunctionprovidedbyMySQLWorkbenchareasfollows:
SQL development: This creates and manages database connections,andconfigurationforconnection
parameters. It executes SQL statements using the built-in SQL editoranditreplaces
thestandaloneapplicationquerybrowserprovidedearlier.
Datamodeling: This createsmodels of database schema graphically,
withreverseandforwardengineering
betweentwodifferentschemaaswellasonalivedatabase.Itprovidesacomprehensive
table editor, with easy to use facilities for editing tables, columns,triggers,indexes,
options,inserts,partitioning,routines,views,andprivileges.
Serveradministration:Thiscreates,maintains,andadministersserverinstances.
Data migration: This allows migration from PostgreSQL, SQLite,SybaseASE,MicrosoftSQLServer,
and other relational databasemanagement system objects, tables, anddatatoMySQL.
It also facilitates migration from earlier versions to the latest releaseversion.
MySQLEnterprisesupport:Thisprovidesenterpriselevelsupportfortheproduct;forexample,MySQLEnterprise
backup,MySQLAudit.
MySQL workbench is available in two different editions, theCommercialEditionand
the Community Edition. The Community Edition is providedwithoutanycost.TheCommercial
Edition provides additional features, such as databasedocumentationgeneration.
MySQLNotifierMySQL Notifier is a simple tool used to monitor and adjust the status oflocal/remoteMySQL server instances. It is an indicator which is placed in asystemtray.ItisinstalledwiththeMySQLinstalleritself.
MySQLNotifier acts as quick launcherwith list actions clubbed together thatcanbeactandmonitoredveryeasilyfromthesystemtrayitselfandalongwiththat it keeps monitoring based on specified interval and notifies on statuschange.
MySQLNotifierusage
MySQLNotifierstaysinsystemtrayandprovidesoneclickoptionforMySQLstatusandmaintenance.FollowingsareimportantusageofMySQLNotifier.
MySQLNotifierprovidesstart,stop,andrestartofMySQLSeverinstances
MySQL Notifier configures MySQL server services and automaticallydetectsandaddsnewMySQLserverservices
MySQLNotifiermonitorsbothlocalandremoteMySQLinstances
Summary
In this chapter, we dived deep into the ocean of commands that are used foralmostalltheactivitiesfortheMySQL
server, starting with installation, server start up, client programs toadministrative
programs, and several utility programs to cater for different purposes on aroutine
basis for database administration. The chapter also provided a workingknowledgeof
makingdatabasebackupsand importing thedatabaseswithorwithoutspecifictables.
ThenextchapterwillfocusonMySQL8datatypesindetail.Itwillcategorizedata
types based on their content types and itwill describe properties in detail foreach
MySQL8DataTypes
In the previous chapter, we learned how to use MySQL 8 command-lineprogramsandutilitiestoperformvariousoperationsontheMySQL8database.Itisalwaysgoodtohaveholdoncommand-linetools.Itprovidesflexibilitytowork innon-GUIenvironments.Thefocusof thischapter isdata types. Isn't itfascinatingtoknowthetypeofdatatheprogramminglanguagesupportsorthestorage engine can store? It is a fundamental feature of any programminglanguageordatabase.Atthesametime,itisthemostignoredtopic,aswell.Themajority of programmers don't spend enough time assessing the storagerequirements for variables used in a piece of code. Actually, it is extremelyimportant to understand the basic and custom data types that the databasesupports,whichiswhythischapterexists.
Thefollowingisalistoftopicstobecoveredinthischapter:
OverviewofMySQL8datatypes
Numericdatatypes
Dateandtimedatatypes
OverviewofMySQL8datatypes
All standard SQL data types are supported in MySQL. These data types areclassifiedinafewcategories,suchasnumerictypes,stringtypes,dateandtimetypes,andtheJSONdatatype.Whenweassignadatatypetoacolumn,certainconventionsmustbefollowed.TheseconventionsarenecessaryforMySQLtoallowvaluestobestoredinacolumn:
Mdenotesthemaximumdisplaywidthforintegertypes.Forfloatingpointandfixedpointtypes,itisthetotalnumberofdigitsthatcanbestored.Forstring types, it is the maximum length. The maximum value alloweddependsonthedatatype.
D is applicable to floating points and fixed point types. It denotes thenumberofdigitsafterthedecimalpoint.Themaximumallowedvalueis30,butmustbelessthanorequaltoM-2.
fsp is applicable to date and time types. It denotes the fractional secondsprecision,whichmeansthenumberofdigitsfollowingthedecimalpointforthefractionalpartofseconds.
This overview is brief so that the detailed descriptions of the features of eachdatatypecanbecoveredintopicstofollow.
NumericdatatypesTheMySQL8numericdatatypesincludeintegerorexactdatatypes,decimalorapproximatedatatypes,andbitdatatypes.
Bydefault,REALdatatypevaluesarestoredasDOUBLE.IfwehavesettheREAL_AS_FLOATflagonMySQL,REALdatatypevaluesarestoredasFLOAT.FLOAToccupieslessspacecomparedtoDOUBLE.
<strong>CREATETABLEemployees</strong><br/><strong>(salaryINTEGER(5)UNSIGNED);</strong>
<strong>CREATETABLEemployees</strong><br/><strong>(idINT(255));</strong>
<strong>CREATETABLEdocuments</strong><br/><strong>(document_noINT(5)ZEROFILL);</strong>
Wespecifiedthevaluetobestoredas111;itwillbestoredas00111ifweprovidedtheZEROFILLoption.
FixedpointtypesFixed point types represent numbers with a fixed number of digits after thedecimalor radixpoint.MySQLhas DECIMALandNUMERIC as fixedpoint, or exact,value data types.These values are stored in a binary format. Fixed point datatypes are useful, especially in storing monetary values in multiplication anddivisionoperations.Thevalue of a fixedpoint data type is an integer numberscaledbyaspecificfactor,accordingtothetype.Forexample,thevalueof1.11canberepresentedinfixedpointas111,withascalingfactorof1/100.Similarly,1,110,000canberepresentedas1110,withascalingfactorof1000.
ThefollowingcodeblockdemonstratesthedeclarationofaDECIMALdatatype:
CREATETABLEtaxes
(tax_rateDECIMAL(3,2));
Intheprecedingexample,3istheprecisionand2isthescale.Anexamplevaluecouldbe4.65,where4istheprecisionand65isthescale:
Precision:DenotesthenumberofsignificantdigitsstoredforthevaluesScale:Representsthenumberofdigitsafterthedecimalpoint
Precisionandscaledefinetherangeofvaluesthatcanbestoredinthecolumn.So,intheprecedingcolumndeclaration,tax_ratecanstorevaluesfallingbetween-9.99and9.99.
ThesyntaxfordefiningtheDECIMALtypeinstandardSQLisasfollows:
DECIMAL(M)
InMySQL,thisisequivalentto:
DECIMAL(M,0)
DeclaringacolumnwithDECIMALisequivalenttoDECIMAL(M,0)inMySQL.
InMySQL,10isthedefaultvalueforM,ifit'snotprovided.
Themaximumnumberof digits supported for the DECIMAL type is 65, includingprecisionandscale.Wecan limit thenumberofdigitsonvalueswhichcanbeenteredforacolumnbyusingprecisionandscale.Ifauserentersavaluewithalarger number of digits than permitted in scale, the valuewill be truncated tomatchthepermittedscale.
DECIMAL is often considered tobe an alternative to DOUBLE or FLOAT. Asmentionedearlier, DECIMAL numbers are an exact representation of REAL numbers inmathematics. The only problem with the DECIMAL data type is that it occupiesmuch more space, even for small numbers. For example, to store a value of0.000003,thecolumndeclarationshouldhavethedatatypedefinedasDECIMAL(7,6).
Ifthescaleis0,thecolumnvaluesdon'thavedecimalpointsorfractionalvalues.
FloatingpointtypesFloatingpointnumbersrepresentrealnumbersincomputing.Realnumbersareusefulformeasuringcontinuousvalues,suchasweight,height,orspeed.
MySQLhas twofloatingpointdata types forstoringapproximatevalues:FLOATandDOUBLE.
For floating point numbers, precision is an important factor. Precision definesthemeasureofaccuracy.MySQLsupportssingleprecisionanddoubleprecisionfloating point numbers. It consumes four bytes to store a single precisionfloatingpointnumberwiththeFLOATdatatype,whereasitconsumeseightbytestostoreadoubleprecisionfloatingpointnumberwiththeDOUBLEdatatype.
In MySQL, REAL is a synonym for DOUBLE PRECISION. As mentioned earlier, ifREAL_AS_FLOAT isenabled,acolumndefinedwiththeREALdatatypewillbetreatedsimilarlytoFLOAT.
TheprecedingdescriptiondepictsFLOATorDOUBLEassimilartoDECIMAL.No,itisnot.There isahugedifference.Asdescribedearlier, fixedpointdata typessuchasDECIMAL or NUMERIC can store exact values, up to the maximum digit after thedecimal point, whereas floating point data types, such as FLOAT or DOUBLE, storeapproximatevalues.Thevaluesstoredaredetailedenough,butnotcompletelyaccurate.Thereremainsaminorinaccuracy.
Let'sunderstandthisthroughthefollowingcodeexample:
mysql>CREATETABLEtyped_numbers(idTINYINT,float_valuesFLOAT,decimal_valuesDECIMAL(3,2));
mysql>INSERTINTOtyped_numbersVALUES(1,1.1,1.1),(2,1.1,1.1),(3,1.1,1.1);
mysql>SELECT*FROMtyped_numbers;
+------+--------------+------------------+
|id|float_values|decimal_values|
+------+--------------+------------------+
|1|1.1|1.10|
|2|1.1|1.10|
|3|1.1|1.10|
+------+--------------+------------------+
mysql>SELECTSUM(float_values),SUM(decimal_values)FROMtyped_numbers;
+--------------------+---------------------+
|SUM(float_values)|SUM(decimal_values)|
+--------------------+---------------------+
|3.3000000715255737|3.30|
+--------------------+---------------------+
Intheprecedingexample:
1. WecreatedatablecontainingFLOATandDECIMALtypecolumns.2. We inserted the same values in two columns, named float_values and
decimal_values.3. Weexecutedaselectquerytofetchthesumofstoredvalues.
Thoughthesamevalues,theoutputisdifferent.Thesumofdecimal_valueslooksmoreprecisecompared to thatof thefloat_values.Thesumoffloat_values looksless precise. This is because of internal rounding performed by the MySQLenginefor floating point data types,which results in the approximation storedvalue.
Standard SQL has a provision for specifying precision while defining a FLOATcolumn. The precision is in bits specified following the keyword FLOAT withinparenthesis.MySQLalsosupportsspecifyingprecisionvaluesforFLOATorDOUBLE,buttheprecisionisusedtodeterminethesize:
Precisionfrom0to23resultsina4bytesingleprecisionFLOATcolumnPrecisionfrom24to53resultsinan8bytedoubleprecisionDOUBLEcolumn
ThefollowingisanexampleofFLOATcolumndeclarationattributes:
FLOAT(M,D)
where,
M-numberofdigitsintotal
D-numberofdigitsmaybeafterthedecimalpoint
So,thecolumndefinedasthefollowingwillstoreavaluesuchas99.99:
FLOAT(4,2)
While storing floating point values,MySQLperforms rounding. So, the valueinserted as 99.09 into a FLOAT(4, 2) column may be stored as 99.01 as anapproximateresult.
Though the floating point column definition supports specifyingprecision, it is advisable to use FLOAT or DOUBLE PRECISION with no
<strong>mysql>CREATETABLEtemp(idINT,col1DOUBLE,col2DOUBLE);</strong><br/><br/><strong>mysql>INSERTINTOtempVALUES(1,5.30,2.30),(1,-3.00,0.00),</strong><br/><strong>(2,0.10,-10.00),(2,-15.20,4.00),(2,0.00,-7.10),</strong><br/><strong>(3,0.00,2.30),(3,0.00,0.00);</strong><br/><br/><strong>mysql>SELECTid,SUM(col1)asv1,SUM(col2)asv2FROMtemp</strong><br/><strong>GROUPBYidHAVINGv1<>v2;</strong><br/><strong>+------+--------+--------+</strong><br/><strong>|id|v1|v2|</strong><br/><strong>+------+--------+--------+</strong><br/><strong>|1|2.3|2.3|</strong><br/><strong>|2|-15.1|-13.1|</strong><br/><strong>|3|0.0|2.3|</strong><br/><strong>+------+--------+--------+</strong>
Intheprecedingexample,itseemsthatthefirsttworowsintheoutputhavesimilarnumbers.Itispossiblethattheymightnotbe,inthecaseoffloatingpointtypes.Ifwewanttoensure,intheprecedingcase,thatsimilar-lookingvaluesareconsidered,wehavetocomparethedifferenceagainstapredefinednumberwithprecision.Forexample,intheprecedingcase,ifwemodifytheHAVINGclausetochecktheconditionABS(v1-v2)>0.1,itwillreturntheexpectedoutput.
Asinterpretationoffloatingpointnumbersisplatformdependent,ifwetrytoinsertavaluewhichisoutsideoftherangeoffloatingpointdatatypesupportedvalues,itmayinsert+-infor+-0.
BitvaluetypeHave you ever come across a requirement to store binary representations ofnumbers?Canyouthinkofsuchusecases?Onesuchusecaseistostoreweeklyworkingdaysinformationforayear.Wewilltouchbaseonthisexamplelaterinthesection.
TheBITdatatypeisusedtostorebinarybitsorgroupsofbitvalues.ItisalsooneoftheoptionstostoreBoolean,yes/noor0/1values.
TheBITtypecolumncanbedefinedas:
column_nameBIT
or
column_nameBIT(m)
wherem=numberofbitstobestored
ForaBITdatatype,mcanvaryfrom1to64.Supplyingmisoptional.Thedefaultvalueformis1.
ThefollowingisanexampleofhowaBITcolumncanbedefined:
CREATETABLEworking_days(
yearINT,
weekINT,
daysBIT(7),
PRIMARYKEY(year,week));
AftertheBITdatatypecolumndeclaration,nextisstoringbitvaluesinacolumn.The bit values are a combination of zeros (0s) and ones (1s). The b'value'notationisusedtospecifybitvalues.
Thefollowingaretheexamplesofhowtostore11and55inaBITcolumn:
CREATETABLEbit_values(valBIT(7));
INSERTINTObit_valuesVALUES(b'1011');
INSERTINTObit_valuesVALUES(b'110111');
WhathappensifthevaluestoredintheBITcolumnislessthanthenumberofbits(m)specifiedinthecolumndefinition?MySQLwillpadthevaluewith0sontheleft of the number. So, for the preceding example, the values stored will be
0001011and0110111,respectively.
HowdowedefineaBITcolumntostoreboolean_values?Thefollowingcodeblockshowsthat:
CREATETABLEboolean_values(valueBIT(1));
or
CREATETABLEboolean_values(valueBIT);
INSERTINTOboolean_valuesVALUES(b'0');
INSERTINTOboolean_valuesVALUES(b'1');
<strong>mysql>SELECTb'1010110',CHARSET(b'1010110');</strong><br/><strong>+--------------+----------------------+</strong><br/><strong>|b'1010110'|CHARSET(b'1010110')|</strong><br/><strong>+--------------+----------------------+</strong><br/><strong>|V|binary|</strong><br/><strong>+--------------+----------------------+</strong><br/><br/><strong>mysql>SELECT0b1100100,CHARSET(0b1100100);</strong><br/><strong>+--------------+----------------------+</strong><br/><strong>|0b1100100|CHARSET(0b1100100)|</strong><br/><strong>+--------------+----------------------+</strong><br/><strong>|d|binary|</strong><br/><strong>+--------------+----------------------+</strong>
PracticalusesofBITLet'scontinuewiththeworkingdaysperweekinayearexample.Pleaserefertotheworking_daystableschemaprovidedearlier.
HowcanwespecifythattheMondayandFridayintheweek4 intheyear2017are non-working days? The following is the INSERT query for this: INSERTINTOworking_daysVALUES(2017,4,0111011);
Ifwe fetch the working_days records using the SELECT query, the following is theoutput:
mysql>SELECTyear,week,daysFROMworking_days;
+--------+---------+--------+
|year|week|days|
+--------+---------+--------+
|2017|4|59|
+--------+---------+--------+
In theprecedingoutput, thedays, thoughbeingofbitdata types,showintegervalues.Howcanweshowbitvaluesintheoutput?
TheansweristheBIN()MySQLfunction.Thefunctionconvertsanintegervaluetoitsbinaryrepresentation:
mysql>SELECTyear,week,BIN(days)FROMworking_days;
+--------+---------+------------+
|year|week|days|
+--------+---------+------------+
|2017|4|111011|
+--------+---------+------------+
Asyoucan see, the leadingzeros are removed from thedays' bit value in theoutput.Toaccomplishtherepresentationintheoutput,ontopoftheBINfunction,wecanusetheLPADMySQLfunction:
mysql>SELECTyear,week,LPAD(BIN(days),7,'0')FROMworking_days;
+--------+---------+------------+
|year|week|days|
+--------+---------+------------+
|2017|4|0111011|
+--------+---------+------------+
TypeattributesAs shown earlier, while defining integer columns, we can also specify anoptionaldisplaywidthattribute.Forexample,INT(5)indicatesanintegernumberwithadisplaywidthof5digits.WhenthiscolumnisusedintheSELECTquery,theoutputwilldisplaythenumberleftpaddedwithspaces.So,ifthevaluestoredintheINT(5)columnis123,thenitwillbedisplayedas__123.The_willbeaspaceintheactualoutput.
However,thedisplaywidthdoesn'tlimittherangeofvalueswhichcanbestoredintheINT(5)column.Thequestionthenarises:Whatifwestoreavalueforwhichthedisplaywidth is larger than thedisplaywidthspecified?Thedisplaywidthdoesn't prevent values wider than the display width of a column from beingdisplayed correctly. So, values wider than the column display width aredisplayedinfullwidth,usingmorethanthenumberofdigitsspecifiedwiththedisplaywidth.
Asmentionedearlier,MySQLcolumndefinitionprovidesanoptionalattributecalled ZEROFILL. This optional attribute, when specified, replaces left paddedspaceswithzeros.Forexample,foracolumndefinedasthefollowing,avalueof82isretrievedas00082:
INT(5)ZEROFILL
This optional attribute is useful where the proper formatting of numbers isimportant.
TheZEROFILLattribute is ignoredwhen thecolumnvalue isused inexpressionsorinaUNIONquery.
MySQLcreatestemporarytableswhencomplicatedjoinsareusedinaquerytostoreintermediateresults.Insuchacase,wemayfaceissuesifwespecifiedacolumnwithdisplaywidth.Inthesecases,MySQLconsidersthatthedatavaluesfitwithinthedisplaywidth.
Another important attribute is UNSIGNED.The UNSIGNED attribute permits only non-
negative values to be stored in the column. It is also useful when we needsupportforalargerrangeofvaluestobestoredwiththesamedatatype.
UNSIGNEDisalsosupportedbyfloatingpointtypesandfixedpointtypes.
If we specify a ZEROFILL attribute for a column, UNSIGNED isautomaticallyaddedtothecolumn.
Another important attribute for integer and floating point columns isAUTO_INCREMENT. When we insert a NULL value in the column defined with theAUTO_INCREMENTattribute,MySQLstoresvalue+1insteadofNULL.Avalueof0willbetreated the sameas that of a NULL value, unless the NO_AUTO_VALUE_ON_ZEROmode isenabled.Here,thevalueisthelargestvaluestoredinthecolumn.ItisextremelyimportantthatthecolumnisdefinedasNOTNULL.Otherwise,theNULLvaluewillbestoredasNULL,eventhoughtheAUTO_INCREMENTattributeisprovided.
Overflowhandling
Whenanout-of-rangevalueisstoredinthenumerictypecolumninMySQL,thevaluestoreddependsontheMySQLmode:
Ifstrictmode is enabled,MySQLwillnot accept thevalueand throwanerror.Theinsertoperationfails.
If restrictive modes are enabled, the value is clipped by MySQL to anappropriatevalue,andthatiswhatisstoredinthecolumn.
DateandtimedatatypesDATE,TIME,DATETIME,TIMESTAMP,andYEAR formthegroupofdateandtimedatatypesforstoringtemporalvalues.Eachofthesetypeshasarangeofpermittedvalues.Apartfromthepermittedvalues,aspecialzerovaluecanalsobeusedtospecifyaninvalidvaluewhichMySQLcannotrepresent.Thezerovaluecanbe00-00-0000.MySQLallowsthisvaluetobestoredinadatecolumn.ThisissometimesmoreconvenientthanstoringNULLvalues.
Thefollowingarethegeneralconsiderationswemusttakecareofwhileworkingwithdateandtimetypes.
ThewayMySQLtreatsstorageandretrievaloperationsfordateortimetypesisdifferent in the context of the format.Basically, for a date or time type valuestoredinthetable,MySQLretrievesvalues inastandardoutput format. In thecaseofinputtingadateor timetypevalue,MySQLattemptstoapplydifferentformatsonthesuppliedinputvalue.So,itisexpectedthatthesuppliedvalueisvalid,orunexpectedresultsmayoccurifusedvaluesinunsupportedformats.
ThoughMySQLcaninterpret inputvalueswithseveraldifferentformats,partsof the date valuemust be supplied in a year-month-day format. For example,2017-10-22or16-02-14.
Supplying a two-digit year creates ambiguity forMySQL to interpret the yearbecause of the unknown century. The following are the rules that must befollowed,usingwhichMySQLinterpretstwo-digityearvalues:
Yearvaluesbetween70-99areconvertedto1970-1999Yearvaluesbetween00-69areconvertedto2000-2069
Itispossibletoconvertavaluefromonetemporaltypetoanothertemporaltypefollowingcertainrules.Wewilldiscusstheseruleslaterinthechapter.
Ifthedateortimevalueisusedinanumericcontext,MySQLwillautomaticallyconvertthevaluetoanumber.
Wehaveoneinterestingusecase.Wewanttodevelopanauditlogfeaturewherewestoreeveryuser-enteredvalue.Supposethatinoneofthedatefields,theuserenteredan invaliddate,2017-02-31.Will thisbe stored in theaudit log table?Certainly not. How do we complete the feature, then? MySQL has theALLOW_INVALID_DATESmode.Ifenabled,itwillallowinvaliddatestobestored.Withthismodeenabled,MySQLverifies that themonth is in the rangeof1-12anddayisintherangeof1-31.
AsODBCcannothandlezerovaluesfordateortime,suchvaluesusedthroughConnector/ODBCareconvertedtoNULL.
Followingtableshowszerovaluesfordifferentdatatypes:
DataType ZeroValueDATE 0000-00-00TIME 00:00:00
DATETIME 0000-00-0000:00:00TIMESTAMP 0000-00-0000:00:00
YEAR 0000
Reference:https://dev.mysql.com/doc/refman/8.0/en/date-and-time-types.html
Thepreceding table shows zero values fordifferent temporaldata types.Thesearespecialvalues,astheseareallowedbyMySQLandareveryusefulincertaincases.Wecanalsospecifyzerovaluesusing'0'or0.MySQLhasaninterestingmodeconfiguration:NO_ZERO_DATE.Ifthisconfigurationisenabled,MySQLshowsawarningifthetemporaltypehasavaluewiththedateaszero.
DATE, DATETIME, andTIMESTAMPtypesThis section describes the most commonly used MySQL date and time datatypes: DATE, DATETIME, and TIMESTAMP. This section explains the similarities anddifferencesbetweenthesedatatypes.
TheDATEdatatypeissuitablewhenthevalueswewishtostorehaveadatepart,but the timepart ismissing.ThestandardMySQLdate format isYYYY-MM-DD.Thedatevaluesare retrievedanddisplayed in the standard formatunlessDATEfunctionsareapplied.TheMySQLsupportedrangeofvaluesis1000-01-01to 9999-12-31. Supported, here, means the values may work, but there is noguarantee.ThesameisthecasefortheDATETIMEdatatype.
TheDATETIMEdatatypeissuitableforvaluescontainingdateandtimeparts.ThestandardMySQLDATETIMEformatisYYYY-MM-DDHH:MM:SS.Thesupportedrangeofvaluesis1000-01-0100:00:00to9999-12-3123:59:59.
Similar toDATETIME, the TIMESTAMP data type is also suitable for values containingdateandtimeparts.However,therangeofvaluessupportedbytheTIMESTAMPdatatypeis1970-01-0100:00:01UTCto2038-01-1903:14:07UTC.
Thoughtheylooksimilar,theDATETIMEandTIMESTAMPdatatypesdiffersignificantly:
TheTIMESTAMPdata typerequires4bytes tostoredateand timevalues.TheDATETIMEdatatyperequires5bytestostoredateandtimevalues.TIMESTAMPcanstorevaluestill2038-01-1903:14:07UTC.Ifwewishtostorevaluesbeyond2038,theDATETIMEdatatypeshouldbeused.TIMESTAMPconsidersUTCasthetimezonewhilestoringvalues.DATETIMEstoresvalueswithouttimezoneconsideration.
Let'suseanexampletounderstandthedifferencebetweenDATETIMEandTIMESTAMPwithinthecontextoftime_zone.
Supposetheinitialtime_zonevalueissetto+00:00:
SETtime_zone='+00:00';
Let'screateatablecalleddatetime_temp.Thetablehastwocolumns;oneisDATETIMEandanotherisofthetypeTIMESTAMP.Wewillstorethesamedateandtimevaluesinbothcolumns.WiththehelpoftheSELECTquery,wewilltrytounderstandhowtherepresentationsdifferinoutput:
mysql>CREATETABLEdatetime_temp(
tsTIMESTAMP,
dtDATETIME);
mysql>INSERTINTOdatetime_temp
VALUES(NOW(),NOW());
mysql>SELECTts,dtFROMdatetime_temp;
+------------------------+-------------------------+
>|ts|dt|
+------------------------+-------------------------+
|2017-10-1418:10:25|2017-10-1418:10:25|
+------------------------+-------------------------+
Intheprecedingexample,NOW()istheMySQLfunctionwhichreturnsthecurrentdateandtimevalues.Lookingattheoutput,itseemsthatboththeTIMESTAMPandDATETIMErepresentationsaresame.Itisbecausethetime_zonevalueissettoUTC.Bydefault,TIMESTAMPshowsthedatetimevalueconsideringtheUTCtime_zone.Ontheotherpart,DATETIMEshowsdatetimewithoutatime_zone.
Let'schangethetime_zoneandobservetheoutput:
mysql>SETtime_zone='+03:00';
mysql>SELECTts,dtFROMdatetime_temp;
+------------------------+-------------------------+
|ts|dt|
+------------------------+-------------------------+
|2017-10-1421:10:25|2017-10-1418:10:25|
+------------------------+-------------------------+
Lookingattheoutput,itisclearthattheTIMESTAMPconsidersthetime_zonevaluesetinMySQL.So,theTIMESTAMPvaluegotadjustedwhenwechangedthetimezone.DATETIMEisn'timpacted,sotheoutputisnotchanged,evenafterchangingthetimezone.
IfTIMESTAMPisusedtostoredateandtimevalues,wemustconsideritseriously when migrating data to a different server located in adifferenttimezone.
If higher precision for the time value is required, DATETIME and TIMESTAMP can
include trailing fractional seconds as small asmicroseconds (sixdigits).So, ifwe insert adate timevaluewithamicrosecondsvalue, itwillbe stored in thedatabase. The format, including the fractional part, is YYYY-MM-DDHH:MM:SS[.fraction], and the range is from 1000-01-01 00:00:00.000000 to9999-12-3123:59:59.999999.The rangeforTIMESTAMP, including the fraction, is1970-01-0100:00:01.000000to2038-01-1903:14:07.999999.
ThefractionalpartisseparatedfromthetimevaluebyadecimalpointbecauseMySQLdoesn'trecognizeanyotherdelimiterforfractionalseconds.
DateandtimevaluesstoredwiththeTIMESTAMPdata typeareconvertedfromtheserver'stimezonetoUTCforstorageandfromUTCtotheserver'stimezoneforretrieval.IfwestoredaTIMESTAMPvalueandthenchangedtheserver's timezoneandretrievedthevalue,theretrievedvaluewouldbedifferentfromtheonewestored.
ThefollowingisthelistofpropertiesofdatevalueinterpretationinMySQL:
MySQL supports a relaxed format for values specified as string. In arelaxed format, any punctuation character can be used as the delimiterbetweendatepartsortimeparts.Thisisalittlebitconfusing.Forexample,avalue10:11:12mightlooklikeatimevaluebecauseoftheuseof:,butisinterpretedasa2010-11-12date.The only recognized delimiter between the rest of the time part and thefractionalsecondspartisthedecimalpoint.It is expected that month and day values are valid. With strict modedisabled,invaliddatesareconvertedtorespectivezerovaluesandawarningmessageisshown.TIMESTAMPvaluesthatincludezerointhedayormonthcolumnarenotavaliddate.Theexceptiontothisruleisthezerovalue.
IfMySQL is runwith MAXDBmode enabled, TIMESTAMP is identical toDATETIME. If this mode is enabled at the time of table creation,TIMESTAMPvaluesareconvertedtoDATETIME.
<strong>mysql>SET@dt=NOW();</strong>
<strong>mysql>SELECT@dt;</strong><br/><strong>+---------------------+</strong><br/><strong>|@dt|</strong><br/><strong>+---------------------+</strong><br/><strong>|2017-10-1513:43:17|</strong><br/><strong>+---------------------+</strong>
<strong>mysql>SELECTDATE(@dt);</strong><br/><strong>+------------------+</strong><br/><strong>|DATE(@dt)|</strong><br/><strong>+------------------+</strong><br/><strong>|2017-10-15|</strong><br/><strong>+------------------+</strong>
<strong>mysql>SELECTTIME(@dt);</strong><br/><strong>+------------------+</strong><br/><strong>|TIME(@dt)|</strong><br/><strong>+------------------+</strong><br/><strong>|13:43:17|</strong><br/><strong>+------------------+</strong>
<strong>mysql>SELECT</strong><br/><strong>HOUR(@dt),</strong><br/><strong>MINUTE(@dt),</strong><br/><strong>SECOND(@dt),</strong><br/><strong>DAY(@dt),</strong><br/><strong>WEEK(@dt),</strong><br/><strong>MONTH(@dt),</strong><br/><strong>QUARTER(@dt),</strong><br/><strong>YEAR(@dt);</strong><br/><strong>+-----------+-------------+-------------+---------+----------+</strong><br/><strong>|HOUR(@dt)|MINUTE(@dt)|SECOND(@dt)|DAY(@dt)|WEEK(@dt)|</strong><br/><strong>+-----------+-------------+-------------+---------+----------+</strong><br/><strong>+------------+--------------+-----------+</strong><br/><strong>|MONTH(@dt)|QUARTER(@dt)|YEAR(@dt)|</strong><br/><strong>+------------+--------------+-----------+</strong><br/><strong>+-----------+-------------+-------------+---------+----------+</strong><br/><strong>|13|43|17|15|41|</strong><br/><strong>+-----------+-------------+-------------+---------+----------+</strong><br/><strong>+------------+--------------+----------
-+</strong><br/><strong>|10|4|2017|</strong><br/><strong>+------------+--------------+-----------+</strong>
TIMEtypeMySQL DATETIME or TIMESTAMP data types are used to represent specific times atparticular dates. How about storing only the time of the day or the timedifferencebetweentwoevents?MySQL'sTIMEdatatypeservesthepurpose.
The standardMySQL format for storing or displaying TIME data type values isHH:MM:SS. The time value represents the time of the day, which is less than 24hours, but the TIME data type, asmentioned earlier, can also be used to storedelapsed time or time difference between two events. So, the TIME column canstorevaluesgreaterthan24hours.
TheMySQLTIMEcolumnisdefinedasfollows:
column_nameTIME;
TherangeofvaluesthatcanbestoredintheTIMEdatatypecolumnis-838:59:59to838:59:59.
The MySQL TIME column can also store the fractional seconds part up tomicroseconds (six digits), similar to the DATETIME column. Considering thefractionalsecondprecision, therangeofvaluesvaries from-838:59:59.000000to838:59:59.00000.
TheMySQLTIMEcolumncanalsohaveanoptionalvalue:
column_nameTIME(N);
whereNrepresentsnumberoffractionalpart,whichisupto6digits.
The TIME value usually takes 3 bytes for storage. In the case of the TIME valueincluding fractional secondprecision, itwill requireadditionalbytes,basedonthenumberoffractionalsecondprecision.
The following table shows the number of additional bytes required to storefractional-secondprecision:
FractionalSecondPrecision Storage(bytes)
0 0
1,2 1
3,4 2
5,6 3
MySQLsupportsabbreviatedvaluesfortheTIMEcolumn.TherearetwodistinctwaysforMySQLtointerpretabbreviatedvalues:
Iftheabbreviatedvaluehasacolon(:),MySQLinterpretsitastimeoftheday.Forexample,11:12isinterpretedas11:12:00andnotas00:11:12.If theabbreviatedvaluedoesn'thaveacolon(:),MySQLassumes that thetworightmostdigitsrepresentseconds.Thismeansthevalueis interpretedaselapsedtime,ratherthantimeoftheday.Forexample, '1214'and1214areinterpretedbyMySQLas00:12:14.
The decimal point is the only delimiter accepted by MySQL to separatefractionalsecondprecisionfromtherestofthetimevalueparts.
MySQL, by default, clips the values that lie outside of the permitted range ofvalues to the closest endpoint of the range. For example, -880:00:00 and880:00:00 are stored as -838:59:59 and 838:59:59. Invalid TIME values areconverted to00:00:00.As00:00:00 itself is avalid TIME value, it is difficult toknowifthevalue00:00:00wasstoredintentionally,orconvertedfromaninvalidTIMEvalue.
MySQLacceptsstringandnumericvaluesastheTIMEvalues.
<strong>mysql>SELECT</strong><br/><strong>CURRENT_TIME()AS'CUR_TIME',</strong><br/><strong>ADDTIME(CURRENT_TIME(),020000)AS'ADDTIME',</strong><br/><strong>SUBTIME(CURRENT_TIME(),020000)AS'SUBTIME';</strong><br/><br/><strong>+----------+-----------+-----------+</strong><br/><strong>|CUR_TIME|ADDTIME|SUBTIME|</strong><br/><strong>+----------+-----------+-----------+</strong><br/><strong>|10:12:34|12:12:34|08:12:34|</strong><br/><strong>+----------+-----------+-----------+</strong>
TheUTC_TIME()functioncanbeusedtofetchtheUTCtime.
<strong>manufacturing_yearYEAR</strong><br/><strong>or</strong><br/><strong>manufacturing_yearYEAR(4)</strong>
OnenotablethingisthatearlierMySQLversionssupportedtheYEAR(2)typecolumndeclaration.ThesupportforYEAR(2)hasbeendiscontinuedfromMySQL8.ItispossiblethatwemightwanttoupgradetheolderMySQLdatabasetotheMySQL8database.Inalatersection,wewillexplainthemigrationdetailsfromYEAR(2)toYEAR(4).
MySQLrepresentsYEARvaluesinaYYYYformat.Therangeofvaluesisfrom1901to2155and0000.
ThefollowingisthelistofformatssupportedforinputtingYEARvalues:
Fourdigitnumberfrom1901to2155.Fourdigitstringfrom1901to2155.Oneortwodigitnumberwiththerangeof0to99.YEARvaluesfrom1to69areconvertedto2001to2069andfrom70to99areconvertedto1970to1999.Oneortwodigitstringwiththerangeof0to99.YEARvaluesfrom1to69areconvertedto2001to2069andfrom70to99areconvertedto1970to1999.Insertinganumeric0hasadisplayvalueof0000andaninternalvalueof0000.Ifwewanttoinsert0andwantittobeinterpretedas2000,weshouldspecifyitasastring0or00.TheresultofafunctionthatreturnsanacceptablevalueYEARcontext,forexample,NOW().
MySQLconvertsinvalidYEARvaluesto0000.
<strong>mysql>CREATETABLEtemp(yearYEAR(2));</strong><br/><strong>ERROR1818(HY000):SupportsonlyYEARorYEAR(4)column.</strong>
TheALTERTABLEquery,whichrebuildsthetable,willautomaticallyconvertYEAR(2)toYEAR(4).TheYEAR(2)column,afterupgradingthedatabasetotheMySQL8database,remainsasYEAR(2),butthequeriesgiveerrors.
TherearemultiplewaystomigratefromYEAR(2)toYEAR(4):
UsingtheALTERTABLEquerywithFORCEattributeconvertstheYEAR(2)columntoYEAR(4).Itdoesn'tconvertthevalues,though.IftheALTERTABLEqueryisappliedtoareplicationmaster,thereplicationslaveswillreplicatetheALTERTABLEstatement.So,thechangewillbeavailableonallthereplicationnodes.Usingbinaryupgrade,withoutdumpingorreloadingdata,isanotherwayofupgradingYEAR(2)toYEAR(4).Runningmysql_upgradesubsequentlyexecutesREPAIR_TABLEandconvertsYEAR(2)toYEAR(4)withoutchangingvalues.Similartothepreviousalternative,thiswillbereplicatedinreplicationslavesifitisappliedtoareplicationmaster.
Animportantthingtonoteisthatwhileupgrading,wemustnotdumptheYEAR(2)datawithmysqldumpandreloadthedumpfileafterupgrading.ThismethodhasthepotentialtochangetheYEAR(2)valuessignificantly.
BeforeYEAR(2)toYEAR(4)migration,applicationcodemustbereviewedfor:
CodethatselectstheYEARvalueintwodigits.Codethatdoesn'thandlenumeric0insertions.Inserting0into
StringdatatypesWhichis themostwidelyrequiredanduseddata typeforrepresentingvalues?Stringorcharacterdatatypes;it'seasy,right?MySQLsupportsawiderangeofstringdata types to fulfill different storage requirements. Stringdata types arecategorized into two categories: fixed length and variable length. CHAR, VARCHAR,BINARY, VARBINARY, BLOB, TEXT, ENUM, and SET are the MySQL-supported string datatypes. The storage requirement for each data type is different and will beexplainedlaterinaseparatesection.
CHARandVARCHARdatatypesTheCHARdatatypeisafixed-lengthstringdatatypeinMySQL.TheCHARdatatypeis often declaredwith amaximumnumber of characters that can be stored asfollows:
dataCHAR(20);
In the preceding example, the data column can store string values that arecapableofstoringmaximumcharacters.
CHARandVARCHARaresimilarinmanyways,withcertaindifferences.TheCHARdatatype ispreferred if the stringvalues tobe storedareof fixed size. ItwillgivebetterperformancecomparedtoifVARCHARisusedforfixedsizestrings.
Thelengthsvaryfrom0to255.ThevalueintheCHARcolumncannotexceedthemaximumlengthdeclaredatthetimeoftablecreation.Ifthelengthofthestringislessthanthemaximumallowedlength,MySQLaddspaddingontherighttothe length specified.At the timeof retrieval, trailing spaces are removed.Thefollowingisanexample:mysql>CREATETABLEchar_temp(dataCHAR(3));
mysql>INSERTINTOchar_temp(data)VALUES('abc'),('a');
mysql>SELECTdata,LENGTH(data)FROMchar_temp;+-------+--------------+|data|LENGTH(data)|+-------+--------------+|abc|3|+-------+--------------+|a|2|+-------+--------------+
Aswecanobserveintheprecedingexample,thesecondrecordwasinsertedas'
a',butintheoutput,thetrailingspaceisremoved.So,thelengthisdisplayedtobe2insteadof3.
MostMySQLcollationshaveapadattribute.Itdetermineshowtrailingspacesare treated for comparison of non-binary strings. There are two types ofcollations:PADSPACEandNOPAD. IncaseofPADSPACEcollation, trailingspacesarenot considered in comparison. Strings are comparedwithout regard to trailingspaces.
In the case of NO PAD collation, the trailing spaces are treated as any othercharacter.Thefollowingisanexample:mysql>CREATETABLEemployees(emp_nameCHAR(10));
mysql>INSERTINTOemployeesVALUES('Jack');
mysql> SELECT emp_name = 'Jack', emp_name = 'Jack ' FROMemployees;+-------------------+--------------------+|emp_name='Jack'|emp_name='Jack'|+-------------------+--------------------+|1|1|+-------------------+--------------------+mysql>SELECTemp_nameLIKE'Jack',emp_nameLIKE'Jack'FROMemployees;+----------------------+------------------------+|emp_nameLIKE'Jack'|emp_nameLIKE'Jack'|+----------------------+------------------------+|1|0|+----------------------+------------------------+
LIKE is a MySQL operator used for comparison in the WHERE clause. It isspecificallyusedforpatternsearchinginastring.TrailingspacesaresignificantwhencomparingstringvalueswiththeLIKEoperator.
IfPAD_CHAR_TO_FULL_LENGTHmodeisenabled,atthetimeofretrieval,thetrailingspaceswillnotberemoved.
The MySQL VARCHAR data type is a variable length string data type with amaximumlengthofupto65,535characters.VARCHARvaluesarestoredbyMySQLasaoneortwobytelengthprefix,alongwithactualdata.Theactualmaximumlengthof a VARCHAR is subject to themaximum row size,which is 65,536 bytessharedamongallcolumns.
IftheVARCHARvaluerequireslessthan255bytes,onebyteisusedfordetermininglengthprefix.Ifthevaluerequiresmorethan255bytes,twobytesareusedfordetermininglengthprefix.
IfMySQLstrictmodeisenabledandavaluetobeinsertedintheCHARorVARCHARcolumnvalueexceedsthemaximumlength,anerrorwillbegenerated.Ifstrictmode is disabled, the valuewill be truncated to themaximum allowed lengthwithawarning.
UnlikeintheCHARdatatype,valuestobestoredinVARCHARarenotpadded.Also,trailingspacesarenotremovedwhenthevaluesareretrieved.
BINARY and VARBINARY datatypesAnothersetofMySQLstringdatatypesisBINARYandVARBINARY.ThesearesimilartoCHAR and VARCHAR data types.An important difference between CHAR/VARCHAR andBINARY/VARBINARY is that BINARY/VARBINARY data types contain binary strings thancharacter strings. BINARY/VARBINARY uses binary character sets and collation.BINARY/VARBINARY are different from CHAR BINARY and VARCHAR BINARY data types. Thebasicdifferenceliesinthecharactersetandcollationreferredto.
Themaximumlength forpermittedvalues is similar to thatofCHARandVARCHAR.The only difference is that the length of BINARY and VARBINARY is in bytes, ratherthancharacters.
HowwouldMySQLcomparebinaryvalues?Theansweristhatthecomparisonhappensbasedonthenumericvaluesforthebytesinthevalues.
Similar to CHAR/VARCHAR data types, the values are truncated if the length of thevalue exceeds the column length, and awarning is generated. This is if strictmodeisnotenabled.Ifstrictmodeisenabled,anerrorisgenerated.
BINARY values are right-padded with the pad value 0x00 (zero bytes) to thespecifiedcolumnlength.Thepadvalueisaddedoninsert,butnotrailingbytesareremovedonretrieval.WhilecomparingBINARYvalues,allbytesareconsideredsignificant.ThisappliestoORDERBYandDISTINCToperators,aswell.Zerobytesandspaces are different when compared with 0x00 < space. The following is anexampleofinsertingabinaryvalue:
mysql>CREATETABLEtemp(
dataBINARY(3));
mysql>INSERTINTOtemp(data)VALUES('a');
In this case, 'a' becomes 'a \0' on insertion. 'a\0' is converted to 'a\0\0'. Onretrieval,valuesremainunchanged.
VARBINARYisavariablelengthstringdatatype.UnlikeBINARY,forVARBINARY,padding
isnotaddedoninsertionandbytesarenotstrippedonretrieval.SimilartoBINARY,allbytesaresignificantincomparisonforVARBINARY.
If the table has a unique index on columns, insertion of values in the columndifferingonlyinnumberoftrailingpadbyteswillgiveaduplicate-keyerror.Forexample,ifsuchacolumncontains'a'andwetrytoinsert'a\0',itwillcauseaduplicate-keyerror.
ThefollowingexampleexplainsthepaddingofBINARYvaluesincomparison:
mysql>CREATETABLEbin_temp(dataBINARY(3));
mysql>INSERTINTObin_temp(data)VALUES('c');
mysql>SELECTdata='c',data='c\0\0'frombin_temp;
+------------+-------------------+
|data='c'|data='c\0\0'|
+------------+-------------------+
|0|1|
+------------+-------------------+
In the case that it is required to retrieve the same value as specified withoutpadding,VARBINARYispreferable.
Ifthevalueretrievedmustbethesameasthevaluespecifiedforstoragewithnopadding, it might be preferable to use VARBINARY or one of the BLOB data typesinstead.
BLOBandTEXTdatatypesInwhatsituationcouldweberequiredtostoredatainaBinaryLargeObject(BLOB)column?Anyidea?Storingafileorimage,yousaid?Itispartiallytrue.Beforewe make a decision to store the images or files in a database or filesystem,weneedtoassessthesituation.Ifthefilesarestoredinafilesystemandmigratedovertoanotheroperatingsystem,itispossiblethatfilepointerscouldgetcorrupted.Itwillrequireadditionaleffortstofixthefilepointers.Insuchacase, storing files in a database is preferable. However, it might impactperformanceifwestorealargecloggedfileorimagedatainthedatabase.
BLOBisMySQL'ssolutiontostoringlargebinaryinformationofvariablelengths.MySQL has four BLOB types: TINYBLOB, BLOB, MEDIUMBLOB, and LONGBLOB. The onlydifferenceamongthesedatatypesisthemaximumlengthofvalueswecanstore.Thestoragerequirementsfor thesedata typesareexplainedin latersectionsofthechapter.
Similar to BLOB, TEXT data types are TINYTEXT, TEXT, MEDIUMTEXT, and LONGTEXT. Thesehave maximum lengths and storage requirements similar to that of BLOB datatypes.
Like BINARY data types, BLOB values are stored as byte strings and have binarycharacter setsandcollation.Comparisonsandsortingaredoneon thenumericvaluesofthecolumnvalues.TEXTvaluesarestoredasnon-binarystrings.
InthecaseofBLOBorTEXTdatatypes,ifthevaluecontainsexcesstrailingspaces,MySQL truncates with a warning, regardless of the MySQL mode. MySQLdoesn't pad BLOB or TEXT column values on insertion and doesn't strip bytes onretrieval.
ForaTEXTcolumnwhichisindexed,theindexcomparisonsaddtrailingspacesaspadding at the end of the values. So, a duplicate-key error may occur oninsertioniftheonlydifferencebetweenanexistingTEXTvalueandtheTEXTvaluetobeinsertedisinthetrailingspaces.BLOBcanberegardedasVARBINARYandTEXTcanberegardedasVARCHAR,withnorestrictiononthelengthofthevalues.
ThefollowingarethedifferencesbetweenVARBINARY,VARCHARandBLOB,TEXT:
When creating indexes on BLOB or TEXT columns, we must specify indexprefixlengthBLOBandTEXTcannothavedefaultvalues
BLOBorTEXTvaluesarerepresentedinternallyasobjectswithseparateallocations,unlikeotherdatatypes,forwhichthestorageisallocatedoncepercolumn.
ENUMdatatype
MySQL provides a data type for which lists of permitted values can bepredefinedwhenthetableiscreated.ThedatatypeisENUM.Ifwewanttorestricttheuser from insertingvaluesoutside a rangeof values,we shoulddefine thecolumn of data type ENUM. MySQL encodes the user input string values intonumbersforENUMdatatypes.
ENUMprovidesthefollowingmentionedbenefits:
Compactdatastorage
Readablequeriesandoutput
The following is an example that showcases when ENUM is useful: mysql>CREATETABLEsubjects(nameVARCHAR(40),streamENUM('arts','commerce','science'));
mysql> INSERT INTO subjects (name, stream) VALUES
('biology','science'),('statistics','commerce'),('history','arts');
mysql>SELECTname,streamFROMsubjectsWHEREstream='commerce';
+------------+----------+
|name|stream|
+------------+----------+
|statistics|commerce|
+------------+----------+
mysql>UPDATEsubjectsSETstream='science'WHEREstream='commerce';
ENUMvaluesrequireonebyteofstorage.Storingonemillionsuchrecordsinthistablewouldrequireonemillionbytesofstorage,opposedtothesixmillionbytesrequiredbytheVARCHARcolumn.
Thefollowingareimportantlimitationstoconsider:
ENUMvaluesarestoredinternallyasnumbers.So,iftheENUMvalueslooklikenumbers,literalvaluesmaymixupwiththeirinternalindexnumbers.
UsingENUM columns in ORDERBY clauses requires extra care. ENUM values areassignedindexnumbersbasedontheorderoflisting.ENUMvaluesaresortedbasedontheirindexnumbers.So,itisimportanttomakesurethattheENUMvalues list is in alphabetical order. Also, the column should be sortedlexicallythanbyindexnumbers.
TheENUMvaluemustbeaquotedstringliteral.
Each ENUM value has an index beginning with 1. The index of the emptystringorerrorvalue is0.Wecan find invalidENUMvaluesbyquerying thetablewithenum_column_value=0intheWHEREclause.TheindexofNULLvalueisNULL.IndexreferstothepositionofavaluewithintheENUMlistofvalues.
MySQL automatically removes trailing spaces from ENUM member valueswhen a table is created. Upon retrieval, values from an ENUM column aredisplayed in the case used in the columndefinition. If a number is to bestoredintheENUMcolumn,thenumberistreatedasanindexintothepossiblevalues.Thevaluestoredis theENUMvaluewith that index. In thecaseofaquoted numeric value, it is still interpreted as an index if there is no
matchingstringinthelistofenumeratedvalues.
If an ENUM column is declared to contain NULL values, the NULL value isconsideredavalidvalueforthecolumnandNULLbecomesthedefaultvalue.IfNULLisnotallowed,thefirstENUMvaluebecomesthedefaultvalue.
IfENUMvaluesareusedinanumericcontext,theindexisused.Thefollowingisan example query to use ENUM values in a numeric context:mysql> SELECTstream+1FROMsubjects;+--------------+|stream+1|+--------------+|4||3||2|+--------------+
SETdatatypeMySQLSETisadatatypewhichcanhavezeroormorevalues.Apermittedlistofvaluesisspecifiedatthetimeoftablecreation.Eachvaluemustbefromwithinthelistofpermittedvalues.Multiplesetmembersarespecifiedbyacomma(,)separated listofvalues.ASET canhaveamaximumof64distinctmembers. Ifstrictmodeisenabled,anerrorisgeneratedifduplicatevaluesarefoundinthecolumndefinition.
ItmustbetakencarethatSETmembervaluesdonotcontaincommas;otherwise,theyareinterpretedasSETmemberseparators.
A column specified as SET('yes', 'no') NOT NULL can have any of the followingvalues:
''
'yes'
'no'
'yes,no'
Trailing spaces are removed automatically from SET member values. Uponretrieval,SETcolumnvaluesaredisplayedusingthelettercasewhichwasusedinthecolumndefinition.
The following is an example of inserting values in the SET data type:mysql>CREATETABLEtemp(hobbiesSET('Travel','Sports','FineDining','Dancing'));
mysql>INSERTINTOtemp(hobbies)VALUES(9);
TheSETvaluesarestoredintheMySQLtableasabitmapinwhicheachelementis represented by one bit. In the preceding case, each element in the SET isassigned a bit. If the row has a given element, the associated bitwill be one.Becauseof thisapproach,eachelementhasanassociateddecimalvalue.Also,becauseof the bitmap, though there are only four values, SET will occupy onebyte.Thefollowingisthetableexplainingthis:
Element SETvalue Decimalvalue
Travel 00000001 1
Sports 00000010 2
FineDining 00000100 4
Dancing 00001000 8
MultipleSETelementscanberepresentedbyaddingtheirdecimalvalues.Intheprecedingcase,thedecimalvalue9isinterpretedasTravel,Dancing.
TheSETdatatypeisnotsocommonlyused.Thisisbecausealthoughitisastringdatatype,itisabitcomplexinimplementation.Thevaluesthatcanbestoredarelimitedto64elements.WecannotaddcommasaspartofSETvalues,becauseacommaisastandardSETvalueseparator.Fromadatabasedesignpointofview,usingSETmeansthedatabaseisnotnormalized.
JSONdatatypeJSONstandsforJavaScriptObjectNotation.Supposethatwewanttostoreuserpreferencesfor aweb application in the database.Usually,wemay choose tocreateaseparatetablewithid,user_id,key,valuefields.Thismayworkwellforasmall number of users, but in the case of thousands of users, the cost ofmaintenance is unaffordable compared to the value it adds to the webapplication.
InMySQL, we can utilize the JSON data type for this requirement.MySQLsupports thenative JSONdata type,which enables efficient storage for JSONdocuments.MySQLsupportsautomaticvalidationofJSONdocumentsstoredinthe JSONcolumn.Trying to store invalid JSONdocumentsproduces an error.JSONdocuments stored inJSONcolumnsareconverted toan internal format.Theformatisbinary,andstructuredtoenabletheservertolookupsubojbectsornestedvaluesdirectly,bykeyorarrayindex,withoutreadingothervalues.
A JSON column cannot have a default value. The JSON data type requiressimilar storage to that of LONGTEXT or LONGBLOB. JSON columns are not indexeddirectly,unlikeotherstringdatatypes.
ThefollowingisanexampleofinsertingJSONvaluesinatable:
mysql>CREATETABLEusers(
user_idINTUNSIGNEDNOTNULL,
preferencesJSONNOTNULL);
mysql>INSERTINTOusers(user_id,preferences)
VALUES(1,'{"page_size":10,"hobbies":{"sports":1}}');
mysql>SELECTpreferencesFROMusers;
+---------------------------------------------------------+
|preferences|
+---------------------------------------------------------+
|{"hobbies":{"sports":1},"page_size":10}|
+---------------------------------------------------------+
Intheprecedingexample,wehaveformattedtheJSONvalue.Asanalternative,wecan also use the built-in JSON_OBJECT function.The function accepts a list ofkey/valuepairsandreturnsaJSONobject.Anexamplefollows:
mysql>INSERTINTOusers(user_id,preferences)
VALUES(2,JSON_OBJECT("page_size",1,"network",JSON_ARRAY("GSM","CDMA","WIFI")));
TheprecedingINSERTquerywill insert theJSONvalue{"page_size":1,"network":["GSM","CDMA","WIFI"]}.WecanalsousenestedJSON_OBJECTfunctions.TheJSON_ARRAYfunctionreturnsaJSONarraywhenpassedasetofvalues.
Ifthesamekeyisspecifiedmultipletimes,onlythefirstkey/valuepairwillberetained. In thecaseof theJSONdata type, theobjectkeysaresortedand thetrailing space between the key/value pairs is removed. The keys in the JSONobjectmustbestrings.
InsertingaJSONvalueinaJSONcolumnsucceedsonlyiftheJSONdocumentis valid. In the case that the JSON document is invalid,MySQL produces anerror.
MySQLhas onemore important anduseful functionwhichoperates on JSONvalues. The JSON_MERGE function takes multiple JSON objects and produces asingle,aggregateobject.
TheJSON_TYPE function takesa JSONasanargumentand tries toparse it intoaJSONvalue.Itreturnsthevalue'sJSONtypeifitisvalidandproducesanerrorifotherwise.
PartialupdatesofJSONvalues
WhatshouldwedoifwewanttoupdateavalueinaJSONdocumentstoredinaJSON
datatype?Oneoftheapproachesistoremovetheolddocumentandinsertanewdocument, with updates. The approach doesn't seem good, right?MySQL 8.0supportspartial,inplaceupdateofaJSONdocumentstoredinaJSONdatatypecolumn. The optimizer requires that an update must meet the followingconditions:
ThecolumnmustbeofJSONtype.
Oneofthreefunctions,JSON_SET(),JSON_REPLACE()orJSON_REMOVE(),canbeusedto update the column. MySQL doesn't permit direct assignment of thecolumnvalueasapartialupdate.
The input column and target column must be the same. For example, astatement such as UPDATE temp SET col1 = JSON_SET(col2, 'one', 10) cannot beperformedasapartialupdate.
Thechangesonlyupdate existing arraysor objects, andnonewelementsareaddedtotheparentobjectorarray.
Thereplacementvaluemustnotbelargerthanthevaluebeingreplaced.
Storage requirements for datatypes
This section explains storage requirements for different data types inMySQL.The storage requirements depend on different factors. The storage enginesrepresentdatatypesandstorerawdatadifferently.
Atablehasamaximumrowsizeof65,535bytes,evenif thestorageengineiscapableofsupportinglargerrows.BLOBandTEXTdatatypesareexcluded.
Thefollowingtableexplainsthestoragedetailsfornumericdatatypes:
DataType
Storagerequired
TINYINT 1byteSMALLINT 2bytes
MEDIUMINT 3bytes
INT,INTEGER 4bytesBIGINT 8bytes
FLOAT(p)
4bytesif0<=p<=24,
8bytesif25<=p<=53
FLOAT 4bytes
DOUBLE[precision],REAL 8bytes
DECIMAL(M,D),NUMERIC(M,D) VariesBIT(M) Approximately(M+7)/8bytes
Reference:https://dev.mysql.com/doc/refman/8.0/en/storage-requirements.html
The following table explains the storage requirements for DATE and TIME datatypes:
DataType
StorageRequired
YEAR 1byteDATE 3bytesTIME 3bytes+fractionalsecondsstorageDATETIME 5bytes+fractionalsecondsstorageTIMESTAMP 4bytes+fractionalsecondsstorage
The following table explains the storage required for fractional secondsprecision:
FractionalSecondsPrecision
StorageRequired
0 0bytes
1,2 1byte
3,4 2bytes
5,6 3bytes
Thefollowingtableexplainsstoragerequirementsforstringdatatypes:
DataType
StorageRequired
CHAR(M)
M×wbytes,0<=M<=255,wherewisthenumberofbytesrequiredforthemaximum-lengthcharacterinthecharacterset
BINARY(M) Mbytes,0<=M<=255
VARCHAR(M),VARBINARY(M)
L+1bytesifthecolumnvaluesrequire0−255bytes,L+2bytesifthevaluesmayrequire
morethan255bytes
TINYBLOB,TINYTEXT L+1bytes,whereL<28
BLOB,TEXT L+2bytes,whereL<216
MEDIUMBLOB,MEDIUMTEXT L+3bytes,whereL<224
LONGBLOB,LONGTEXT L+4bytes,whereL<232
ENUM('value1','value2',...) 1or2bytes,dependingonthenumberofenumerationvalues(65,535valuesmaximum)
SET('value1','value2',...)1,2,3,4,or8bytes,dependingonthenumberofsetmembers(64membersmaximum)
Reference:https://dev.mysql.com/doc/refman/8.0/en/storage-requirements.html
Inthecaseofstringdatatypes,variablelengthstringsarestoredusingthelengthof the value and the length prefix. The length prefix varies from one to fourbytes,dependingonthedatatype.
Storage requirements for theJSONdata typearesimilar to thatofLONGBLOBandLONGTEXT.However,astheJSONdocumentsarestoredinbinaryrepresentations,itimposesanoverheadinstoringJSONdocuments.
Choosing the right data type forcolumn
Asageneralpractice,weshouldusethemostprecisetypeforstoringdata.Forexample, a CHAR data type should be used to store a string value that varies inlengthfrom1to255characters.AnotherexampleisthatMEDIUMINTUNSIGNEDshouldbeusedtostorenumbersrangingfrom1to99999.
Basic operations such as addition, subtraction, multiplication, and division withDECIMALdataareperformedwiththeprecisionof65decimaldigits.
Basedon the importanceofaccuracyorspeed,useofFLOATorDOUBLE shouldbechosen.FixedpointvaluesstoredinBIGINTcanbeusedforhigherprecision.
Thesearegeneralguidelines,butthedecisiontousetherightdatatypeshouldbemade based on the detailed characteristics explained separately for each datatypeintheearliersections.
Summary
It was an interesting chapter with important content to learn, right? In thischapter, we understood the significance of data types in MySQL. We sawdifferentcategories inwhichMySQLdata typesareclassified.Welearnedandunderstoodthecharacteristicsandspecificationsofeachdatatypeindepth.Wealso learnedMySQL datamanipulation functions and understood some of theMySQL settings and modes. In the later section of the chapter, we learnedstorage requirements of data types. Finally, we learned general guidelines forchoosingtherightdatatypes.
Moving on to next chapter,wewill learnMySQL databasemanagement. Thechapter will focus on server administration, understanding the basic buildingblocksoftheMySQL
server,suchasthedatadictionary,systemdatabase,andsoon.Thechapterwillexplain how we can run multiple server instances on a single machine andMySQLrolesandpermissions.
MySQL8DatabaseManagement
In the previous chapter,we learned aboutMySQL 8 data types, explaining indetailwhich data types are available and how they are categorized. There arevarious properties associated with each of these data types, and the storagecapacityvarieswitheachtype.Thepreviouschapteralsoprovidedyouwithanin-depthunderstandingofMySQL
8 data types. Now its time to get some practical knowledge on MySQL 8administrativefeatures.Isn'titinterestingtoknowmoreabouttheadministrativefeaturesofMySQL
8, how configuration will be done for it, and much more? It's extremelyimportant for an administrator to have detailed knowledge on howMySQL 8worksforglobalization,howlogsaremaintained,andhowtoenhancecapabilityoftheserver.Now,let'sstartwithsomefundamentalconcepts.
Wewillcoverthefollowtopicsinthischapter:
MySQL8serveradministration
Datadirectory
Thesystemdatabase
Runningmultipleinstancesonasinglemachine
Componentsandpluginmanagement
Rolesandpermissions
Cachingtechniques
Globalization
MySQL8serverlogs
MySQL8serveradministrationTherearemanyoperatingparametersthatavailablewithMySQL8,andamongthem all the required parameters are set by default during the installationprocess. After installation, you are allowed to change the option file byremoving or adding a comment sign (#) at the start of the line of a specificparametersetting.Theuserisalsoallowedtosetparametersatruntimebyusingcommandlineargumentsortheoptionfile.
<strong>mysqld--help</strong>
<strong>mysqld–verbose--help</strong>
ServerSystemvariable:TheMySQLservermanagesmanysystemvariables.MySQLprovidesthedefaultvalueforeachsystemvariable.Systemvariablescanbesetusingthecommandlineorcanbedefinedintheoptionfile.MySQL8hastheflexibilitytochangethesevariablesatruntimewithoutserverstartorstop.Formoredetailsreferto:https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html.Serverstatusvariable:TheMySQLserverusesmanystatusvariablestoprovideinformationaboutitsoperation.Formoredetailsreferto:https://dev.mysql.com/doc/refman/8.0/en/server-status-variables.html.
ServerSQLmodesMySQL 8 provides different modes that will affectMySQL support and datavalidation checks. This option makes it easier for the user to useMySQL indifferentenvironments.
TosetdifferentmodesMySQLprovidesthesql_modesystemvariablewhichcanbesetateitheraglobalorsessionlevel.Refertothefollowingpointsindetailtounderstandmodes:
<strong>SETGLOBALsql_mode='modes';</strong><br/><strong>SETSESSIONsql_mode='modes';</strong>
<strong>SELECT@@GLOBAL.sql_mode;</strong><br/><strong>SELECT@@SESSION.sql_mode;</strong>
TheavailableSQLmodes
ThissectiondescribesalltheavailableSQLmodes.Outofthem,thefirstthreeare
themostimportantSQLmodes:
ANSI:Thismodeisusedtochangesyntaxandbehavior,bymakingitclosertostandardSQL.
STRICT_TRANS_TABLES:As thenameimplies, thismodeisrelated to transactionandit ismainlyusedfortransactionalstorageengines.Whenthismodeisenablefornontransactionaltables,MySQL8willconvertinvalidvaluestotheclosestvalidvalueandinserttheadjustedvalueintothecolumn.Ifthevalueismissing,thenMySQL8willinsertanimplicitdefaultvaluerelatedto thecolumn'sdata type. In thiscase,MySQL8willgenerateawarningmessage instead of an error message, and continue with the statementexecutionwithoutbreaking it.In thecaseof transactional tables,however,MySQL8givesanerrorandwillbreaksexecution.
TRADITIONAL: This mode generally behaves like traditional SQL database
system.Itindicatesgiveerrorinsteadofawarningwhenanincorrectvalueinsertedintothecolumn.
ALLOW_INVALID_DATES: This mode checks only the month range and the daterangeofthedatevalue.Inotherwords,themonthrangemustbebetween1to12anddaterangemustbebetween1to31.ThismodeisapplicableforDATEandDATETIMEdatatypesandnotfortimestampdatatype.
ANSI_QUOTES:Used toconsider" asan identifierquotecharacter insteadofastringquotecharacter.When thismode isenabled,youcannotusedoublequotationtoquotestringliteral.
ERROR_FOR_DIVISION_BY_ZERO:Used tohandle thecaseofdivisionbyzero.ThismodeoutputalsodependsonstrictSQLmodestatus:
Ifthismodeisnotenabled,divisionbyzeroinsertsNULLandproducesnowarning.
If thismode is enabled, division by zero inserts NULL and produces awarning.
Ifthismodeandstrictmodeareenabled,divisionbyzeroproducesanerror,unlessIGNORE isgivenaswell.ForINSERTIGNOREandUPDATEIGNORE,divisionbyzeroinsertsNULLandproducesawarning.
HIGH_NOT_PRECEDENCE: Thismode is used to set a high precedence for the NOToperator. For example, when the mode is enabled the expression NOT a
BETWEENbANDcisparsedasNOT(aBETWEENbANDc)insteadof(NOTa)BETWEENbANDc.
IGNORE_SPACE:Thismodeappliestobuilt-infunctionsratherthanuserdefinedfunctionsorstoredprocedures.
NO_AUTO_CREATE_USER: This mode is used to prevent GRANT statements byautomaticallycreatingnewuseraccountsunlessauthenticationinformationisspecified.
NO_AUTO_VALUE_ON_ZERO:Thismodeisusedforautoincrementalcolumns.When0 is foundMySQLcreatesanewsequencenumber for the field, and thatwillcreateproblemswhenyouareloadingdump.Enablethismodebeforereloadingdumptoresolvethisproblem.
NO_BACKSLASH_ESCAPES:Ifthismodeisenabled,backslashbecomesanordinarycharacter.
NO_DIR_IN_CREATE:ThisoptionisusefulforslavereplicationserverswheretheINDEXDIRECTORYandDATADIRECTORYdirectivesareignoredontablecreation.
NO_ENGINE_SUBSTITUTION: Used to provide substitution of the default storageengine.When thismode is enabledand thedesiredengine isunavailable,MySQLgivesanerrorandatableisnotcreated.
NO_FIELD_OPTIONS:This indicates,don'tprintMySQLspecificcolumnoptionsintheoutputofSHOW_CREATE_TABLE.
NO_KEY_OPTIONS: This indicates, don't printMySQL specific index options intheoutputofSHOW_CREATE_TABLE.
NO_TABLE_OPTIONS:Thisindicates,don'tprintMySQLspecifictableoptionsintheoutputofSHOW_CREATE_TABLE.
NO_UNSIGNED_SUBTRACTION: When this mode is enabled, it makes sure thatsubtractionresultmustbeasignedvalueeventhoughanyoftheoperandisunsigned.
NO_ZERO_DATE:Theeffectofthismodedependsonthestrictmodeasdefinedbelow:
If it is not enabled, 0000-00-00 is allowed andMySQLproduces nowarningoninsertion
Ifthismodeisenabled,0000-00-00isallowedandMySQLrecordsawarning
If both this mode and strict mode are enabled, 0000-00-00 is notallowedandMySQL
producesanerroroninsertion
NO_ZERO_IN_DATE: This mode effect is also depending on the strict mode asdefinedbelow:
If it is not enabled, dates with zero parts are allowed and MySQLproducesnowarningoninsertion
Ifthismodeisenabled,dateswithzeropartsareallowedandproduceawarning
Ifthismodeandstrictmodeareenabled,dateswithzeropartsarenotallowedandMySQLproducesanerror
ONLY_FULL_GROUP_BY: If this mode is enabled, MySQL will reject queries inwhich select list, order by list, and the HAVING condition refer to nonaggregatedcolumns.
PAD_CHAR_TO_FULL_LENGTH:ThismodeisappliedonthecolumnwhosedatatypeissetasCHAR.Whenthismodeisenabled,MySQLretrievescolumnvaluesbypaddingtotheirfulllength.
PIPES_AS_CONCAT:Whenthismodeissetasenabled||willbeconsideredasastringconcatenationoperatorinsteadofOR.
REAL_AS_FLOAT:Bydefault,MySQL8willconsiderREALasasynonymofDOUBLE,butwhen this flag is enabledMySQLwill consider REAL as a synonymofFLOAT.
STRICT_ALL_TABLES:Inthismodeinvaliddatavaluesarerejected.
TIME_TRUNCATE_FRACTIONAL:ThismodeindicatesiftruncationisallowedonTIME,DATE, and TIMESTAMP columns or not. The default behavior is to performroundingonthevaluesinsteadoftruncation.
CombinationSQLmodes
MySQL8alsoprovidessomespecialmodesascombinationsofmodevalues:
ANSI: It includes the effects of the REAL_AS_FLOAT, PIPES_AS_CONCAT, ANSI_QUOTES,IGNORE_SPACE,andONLY_FULL_GROUP_BYmodes.
DB2: It includes the effects of the PIPES_AS_CONCAT, ANSI_QUOTES, IGNORE_SPACE,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,andNO_FIELD_OPTIONSmodes.
MAXDB: It includes the effects of PIPES_AS_CONCAT, ANSI_QUOTES, IGNORE_SPACE,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,NO_FIELD_OPTIONS,andNO_AUTO_CREATE_USER.
MSSQL: It includes the effects of PIPES_AS_CONCAT, ANSI_QUOTES, IGNORE_SPACE,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,andNO_FIELD_OPTIONS.
MYSQL323:ItincludestheeffectsoftheMYSQL323andHIGH_NOT_PRECEDENCEmodes.
MYSQL40:ItincludestheeffectsoftheMYSQL40andHIGH_NOT_PRECEDENCEmodes.
ORACLE: It includes the effects of the PIPES_AS_CONCAT, ANSI_QUOTES, IGNORE_SPACE,
NO_KEY_OPTIONS,NO_TABLE_OPTIONS,NO_FIELD_OPTIONS,andNO_AUTO_CREATE_USERmodes.
POSTGRESQL:ItincludestheeffectofthePIPES_AS_CONCAT,ANSI_QUOTES,IGNORE_SPACE,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,andNO_FIELD_OPTIONSmodes.
TRADITIONAL: It includes theeffectsof the STRICT_TRANS_TABLES, STRICT_ALL_TABLES,NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER andNO_ENGINE_SUBSTITUTIONmodes.
<strong>ALTERTABLE</strong><br/><strong>CREATETABLE</strong><br/><strong>CREATETABLE...SELECT</strong><br/><strong>DELETE(bothsingletableandmultipletable)</strong><br/><strong>INSERT</strong><br/><strong>LOADDATA</strong><br/><strong>LOADXML</strong><br/><strong>SELECTSLEEP()</strong><br/><strong>UPDATE(bothsingletableandmultipletable)</strong>
Youcangoto:https://dev.mysql.com/doc/refman/8.0/en/sql-mode.htmlforadetailedlistoftheerrorsassociatedwithstrictSQLmodeinMySQL.
TheIGNOREkeyword
MySQLprovidesan IGNORE keywordwhich is optional for statement execution.TheIGNORE keyword is used todowngrade errors towarnings and applicable toseveralstatements.
Formultiplerowstatements,theIGNOREkeywordallowsyoutoskiptheparticularrow,insteadofaborting.ThefollowingstatementssupporttheIGNOREkeyword:
CREATE TABLE ... SELECT: Individual CREATE and SELECT statements do not havesupport on this keyword, but when we insert into the table using SELECTstatement, rows that duplicate an existing row on a unique key value arediscarded.
DELETE:IfthisstatementexecuteswiththeIGNOREoptionMySQLavoiderrorsoccurredduringexecution.
INSERT:Duplicate values in unique key and data conversion issueswill behandled by this keyword during row insertion. MySQL will insert theclosestpossiblevaluesintothecolumnandignoretheerror.
LOADDATAandLOADXML:AtthetimeofloadingdataifduplicationisfoundthestatementwilldiscarditandcontinueinsertionfortheremainingdataiftheIGNOREkeywordisdefined.
UPDATE: In cases of duplicate key conflict on unique key during statementexecution,MySQL
willupdatethecolumnwiththeclosestidentifiedvalues.
TheIGNOREkeywordalsoappliesonsomespecificerrors,listedhere:https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html.
IPv6support
MySQL8providessupportforIPv6,withthefollowingcapabilities:
MySQL server will accept TCP/IP connections from clients with IPv6connectivity
MySQL 8 account names permit IPv6 addresses, which enables DBA tospecifyprivileges
fortheclientsthatareconnectedwithserver,usingIPv6
The IPv6 functions enable conversions between string and internal IPv6addressformats,
andcheckingwhetherthevaluesrepresentavalidIPv6addressornot
<strong>mysql-urootmysql<fill_help_tables.sql</strong>
Atthetimeofinstallationprocesscontentinitializationoccurs.Incaseofupgradingitwillbeperformso;executetheabovecommandmanually.
Theservershutdownprocess
Theservershutdownprocessperformsthefollowingsteps:
1. Theshutdownprocessisinitiated:Thereareseveralwaystoinitializetheshutdown process. Execute the mysqladmin shutdown commandwhich can beexecutedonanyplatform.Therearesomesystemspecificwaystoinitializethe shutdownprocess; for example,Unix based systemswill start to shutdown when it receives a SIGTERM signal. In the same way, Windowbasedsystemswillstarttoshutdownwhentheservicemanagertellsthemto.
2. The server creates a shutdown thread if necessary: Based on theshutdowninitializationprocess,theserverwilldecidetocreatenewthreador not. If it is requested by the client, a new threadwill be created. If asignalisreceived,thentheservermightcreateathreadoritmighthandleitby itself. If the server tries to create a separate thread for the shutdownprocessandanerroroccurs,thenitproducesthefollowingmessageintheerrorlog:
Error:Can'tcreatethreadtokillserver
3. Theserverstopsacceptingnewconnections:Whentheshutdownactivityisinitiated,theserverwillstopacceptingnewconnectionrequests,usingahandlerofnetwork interfaces.TheserverwillbeusingWindows featuressuchasnamedpipe,TCP/IPport,theUnixsocketfile,andsharedmemoryonWindowsinordertolistentonewconnectionrequests.
4. The server terminates current activity: Once the shut down process isinitialized, the serverwill start tobreak theconnectionwith theclient. Inthenormal scenario, the connection threadswill diequickly,but theoneswhichareworkingorareinanongoingactivitystagewilltakealongtimetodie.Soifathreadisexecutingopentransactionsandifitgetsrollbackinthemiddleofexecutionthentheusermightgetonlypartiallyupdateddata.Ontheotherhand,ifthethreadisworkingonatransaction,thentheserverwillwaituntilthetransactioniscompleted.Also,theusercanterminatetheongoingtransactionbyexecutingtheKILLQUERYorKILLCONNECTIONstatements.
5. Theservershutsdownorclosesstorageengines:Inthisphase,theserverflushes the cache and closes all the open tables.Here, the storage engineperformsall theactionsnecessaryfor tables.InnoDB flushes itsbufferpool,writesthecurrentLSNintotablespaceandterminatesitsthread.TheMyISAMflushesthependingindex.
6. Theserverexits:Inthisphase,theserverwillprovideoneofthefollowingvaluestothemanagementprocesses:
0=successfultermination(norestartdone)
1=unsuccessfultermination(norestartdone)
2=unsuccessfultermination(restartdone)
Datadirectory
ThedatadirectoryistheplacewhereMySQL8storesalltheinformationthatismanagedbyitself.Eachsub-directoryofthedatadirectoryrepresentsadatabasedirectory and its related data.All theMySQL installations have the followingstandarddatabases:
The sys directory: This represents the sys schema, which contains theobjectsusefulforthePerformanceSchemainformationinterpretation.
The performance schema directory: This directory is used to observe theinternalexecutionoftheMySQLserveratrun-time.
The mysql directory: This directory is related to the MySQL systemdatabase,whichcontainsthedatadictionarytableandthesystemtables.Itcontains the information that is required by theMySQL server once it isrunning.
ThesystemdatabaseThe system databasemainly contains the data dictionary tables that stores theobject'smetadataandsystemtables forotheroperationalpurposes.Thesystemdatabasecontainsanumberofsystemtables.Wewilllearnmoreabouttheminthecomingsections.
DatadictionarytablesThe data dictionary tables contains themetadata about data objects. Tables ofthis directory are invisible and are not read by general SQL queries suchasSELECT,SHOWTABLES,INFORMATION_SCHEMA.TABLES,andsoon.MySQLmainlyexposesthemetadatausingtheINFORMATION_SCHEMAoption.
GrantsystemtablesThese tables are used to manage and provide grant information of users,database and relevant privileges. MySQL 8 uses grant tables as transactionaltables,notnontransactional(MyISAM,forexample)tables,soalltheoperationsonthe transaction are either completed or failed; no partial case will be madepossible.
Objectinformationsystemtables
These tables contains information related to the stored programs, componentsandserver-sideplugins.Thefollowingmaintablesareusedtostoreinformation:
Component:Worksasaregistryfortheserver.TheMySQL8serverloadsallthecomponentslistedbythistableonserverstartup.
Func: This table contains information related to all the user-definedfunctions (UDF). MySQL 8 will load all the UDFs listed in this tableduringserverstartup.
Plugin: Contains the information related to the server-side plugins. TheMySQL8serverloadsalltheavailablepluginsduringstartup.
LogsystemtablesThesetablesareusefulforloggingandusingcsvstorageengines.Forexample,thefunctionsgeneral_logandslow_log.
The server-side help systemtablesThese tables are useful to store help information. The following tables areavailableinthiscategory:
help_category:Providesinformationaboutthehelpcategorieshelp_keyword:Provideskeywordsassociatedwithhelptopicshelp_relation:Helpsinmappingsbetweenhelpkeywordsandtopicshelp_topic:Helptopiccontents
Timezonesystemtables
Thesetablesareusefultostoretimezoneinformation.Thefollowingtablesareavailableinthiscategory:
time_zone:ProvidesthetimezoneIDsandwhethertheyuseleapseconds
time_zone_leap_second:Willcomeinhandywhenleapsecondsoccur
time_zone_name:HelpsinmappingsbetweentimezoneIDsandnames
time_zone_transitionandtime_zone_transition_type:Timezonedescriptions
Replicationsystemtables
These tables are useful to support the replication feature. It helps to storereplication related information when it is configured to as mentioned infollowingtables.Thefollowingtablesareavailableinthiscategory:
gtid_executed:UsedforcreatingthetableforstoringGTIDvalues
ndb_binlog_index: Provides the binary log information for MySQL Clusterreplication
slave_master_info, slave_relay_log_info and slave_worker_info: Used to storereplicationinformationonslaveservers
OptimizersystemtablesThis tables are useful for optimizer. The following tables are available in thiscategory:
innodb_index_statsandinnodb_table_stats:UsedforgettingtheInnoDBpersistentoptimizerstatisticsserver_cost: Contains the optimizer cost estimates for general serveroperations.engine_cost: Contains the estimates for operations specific to particularstorageengines
Other miscellaneous systemtables
Tables that don't fall into the above-mentioned categories fall under thiscategory.
Thefollowingtablesareavailableinthiscategory:
servers:UsedbytheFEDERATEDstorageengine
innodb_dynamic_metadata: Used by the InnoDB storage engine to store fastchanging tablemetadata such as auto-increment counter values and indextreecorruptionflags
Youcanlearnmoreaboutthedifferentsystemtablesat:https://dev.mysql.com/doc/refman/8.0/en/system-database.html.
Running multiple instances on asinglemachine
There might be some situations where you are required to install multipleinstances on a single machine. It may be to check the performance of twodifferentversions,orperhapsthereisaneedtomanagetwoseparatedatabasesondifferentMySQLinstances.
The reason can be anything, but MySQL allows user to execute multipleinstances on the same machine by providing different configuration values.MySQL8allowsuserstoconfigureparametersbymakinguseofthecommandline,optionfile,orbysettingenvironmentvariables.TheprimaryresourceusedbyMySQL 8 for this is the data directory and itmust be unique for the twoinstances. We can define the value for the same using the --datadir=dir_namefunction.Apartfromthedatadirectory,wewillalsoconfigureuniquevaluesforthefollowingoptionsaswell:
--port=port_num
--socket={file_name|pipe_name}
--shared-memory-base-name=name
--pid-file=file_name
--general_log_file=file_name
--log-bin[=file_name]
--slow_query_log_file=file_name
--log-error[=file_name]
--tmpdir=dir_name
Setting up multiple datadirectories
As described above, each of theMySQL instancesmust have a separate datadirectory.
Theuserisallowedtodefineseparatedirectoriesusingthefollowingmethods:
Create a new data directory: In this method we must follow the sameprocedure which was defined in Chapter 2, Installing and UpgradingMySQL. For Microsoft Windows, when we install MySQL 8 from Ziparchives,copyitsdatadirectorytothelocationwhereyouwanttosetupthenewinstance.InthecaseofanMSIpackagealongwiththedatadirectory,create a pristine template data directory named data under the installationdirectory.Oncetheinstallationiscomplete,copythedatadirectorytosetupadditionalinstances.
Copyanexistingdatadirectory:Inthismethod,wewillcopyanexistinginstance's data directory to the new instance's data directory. To copy anexistingdirectory,performthefollowingsteps:
1. StoptheexistingMySQLinstance.Makesureit'scleanlyshutdownsothatnopendingchangesareavailableinthedisk.
2. Copythedatadirectorytothenewlocation.
3. Copythemy.cnformy.inioptionfileusedbytheexistinginstancetothenewlocation.
4. Modify the new option as per the new instance. Make sure all theuniqueconfigurationsaredoneproperly.
5. Startthenewinstancewiththenewoptionfile.
<strong>[mysqld]</strong><br/><strong>basedir=C:/mysql-5.5.5</strong><br/><strong>port=3307</strong><br/><strong>enable-named-pipe</strong><br/><strong>socket=mypipe1</strong>
<strong>[mysqld]</strong><br/><strong>basedir=C:/mysql-8.0.1</strong><br/><strong>port=3308</strong><br/><strong>enable-named-pipe</strong><br/><strong>socket=mypipe2</strong>
<strong>C:\>C:\mysql-5.5.5\bin\mysqld--installmysqld1--<br/>defaults-file=C:\my-opts1.cnf</strong><br/><strong>C:\>C:\mysql-8.0.1\bin\mysqld--installmysqld2--<br/>defaults-file=C:\my-opts2.cnf</strong>
<strong>C:\>C:\mysql-5.5.9\bin\mysqld--installmysqld1</strong><br/><strong>C:\>C:\mysql-8.0.4\bin\mysqld--installmysqld2</strong>
<strong>C:\>NETSTARTmysqld1</strong><br/><strong>C:\>NETSTARTmysqld2</strong>
MySQL8serverpluginsMySQL 8 server has a plugin API that enables the creation of servercomponents.WithMySQL8, you have the flexibility of installing a plugin atruntimeoratstartup.
InthefollowingtopicswewilllearnaboutthelifecycleoftheMySQL8serverplugins.
Installingtheplugins
The loadingof thepluginsvarieswith their typesandcharacteristics.Togetaclearerpictureofthis,let'sgothoughthefollowing:
Built-in plugins: The server knows about the built-in plugins and loadsthemautomatically,onstartup.
Theuserisallowedtochangethestateofpluginsbyanyoftheiractivationstatuses,whichwillbediscussedinthefollowingsection.
Pluginsregistered in the mysql.plugin systemtable:On startupMySQL8serverwillloadallthepluginswhichareregisteredinthemysql.plugintable.Iftheserverisstartedwiththe--skip-grant-tablesoption,theserverwillnotloadthepluginslistedthere.
Plugins named with command-line options: MySQL 8 provides the --plugin-load, --plugin-load-add, and --early-plugin-load options for loadingplugins with the command line. The --plugin-load and --plugin-load-add
options load the plugins on server startup after the built-in plugins areinstalled.
But,we can use the --early-plugin-load option to load the plugins, prior toinitializationofbuilt-inpluginsandstorageengines.
Plugins installed with the INSTALL PLUGIN statement: This is a permanentpluginregistrationoption,whichwillregistertheplugininformationinthemysql.plugin table. It will also load all the plugins available in the pluginlibrary.
Activateplugin
To control the state (like the activation or deactivation) of plugins,MySQL 8providesthefollowingoptions:
--plugin_name=OFF:Disables the named plugin. Some of the built-in plugins,suchastheasmysql_native_passwordplugin,arenotaffectedbythiscommand.
--plugin_name[=ON]: This command enables the specified plugin. If plugininitialization failed during startup MySQL 8 will start with the plugindisabled.
--plugin_name=FORCE:Thisisthesameastheabovecommand,excepttheserverdoesnotstart.Thismeansthatitforcestheservertostartwiththepluginifitismentionedonstartup.
--plugin_name=FORCE_PLUS_PERMANENT: The same as the FORCE option, butadditionallypreventsthepluginfrombeingunloadedatruntime.
UninstallpluginMySQL 8 uses the UNINSTALL PLUGIN statement to uninstall the plugin, withoutconsideringwhether itwas installed during the runtime or at startup. But thisstatementwillnotallowustouninstallthebuilt-inpluginsandthepluginsthatwere installedby the--plugin_name=FORCE_PLUS_PERMANENToption.This statement justunloads the plugin and removes it from the mysql.plugin table, so it requiresadditionaldeleteprivilegesonthemysql.plugintable.
<strong>SELECT*FROMinformation_schema.PLUGINS;</strong>
<strong>SHOWPLUGINS;</strong>
Themysql.plugintablecontainsdetailsregardingallthepluginswhichwereregisteredbytheINSTALLPLUGINfunction.
<strong>CREATEROLEhrdepartment;</strong><br/><strong>grantallonhr_employeetohrdepartment;</strong>
Theabovecodewillhelpustocreatethehrdepartmentroleandgrantallthenecessaryaccesstoit.ThistopicwillbecoveredindetailedinChapter11,Security.
CachingtechniquesCache is a mechanism used to improve performance. MySQL uses severalstrategiestocacheinformationinthebuffer.MySQL8makeuseofthecacheatthe storage engine level to handle its operations. It also applies the cache inpreparedstatementsandstoredprogramstoimproveperformance.MySQL8hasintroduced various system level variables to manage cache, such asbinlog_stmt_cache_size, daemon_memcached_enable_binlog, daemon_memcached_w_batch_size,host_cache_size, and many more. We will cover caching in detail in Chapter 12,OptimizingMySQL8.
GlobalizationGlobalization is a feature which provides multi-language support for anapplication, such as enabling the use of native languages. It ismuch easier tounderstandmessagesinourownnativelanguagethanotherlanguages,right?Toachievethis,globalizationcomesintothepicture.Usingglobalizationausercanstore,retrieveandupdatedataintomanylanguages.Therearecertainparametersthataretobeconsideredinglobalization.Wewilldiscussthemindetail inthefollowingsections.
Charactersets
Beforegoingintodetailaboutcharactersetsitisrequiredtounderstandwhata
charactersetactuallyis,aswellasitsrelatedterms,right?Let'sstartwiththe
termitself;thecharactersetisasetofsymbolsandencoding.Anotherimportant
term related to character set is collation, the set of rules used for comparingcharacters.Let'stakeasimpleexampletounderstand
thecharactersetsandcollation.Considertwoalphabets,PandQ,andassignanumbertoeach,sothatP=1andQ=2.Now,assumePisasymboland1isitsencoding.Here,thecombinationofboththelettersand
theirencodingisknownasthecharacterset.Nowsuppose,wewanttocomparethese
values; the simplestway isby referring theencodingvalues.With this as1 isless
than 2 we can sayP is less thanQ which is known as collation. This is thesimplestexampletounderstandcharacter
sets and collation, but in real lifewe havemany characters, including specialcharacters,
andinthesamewaycollationshavemanyrules.
<strong>mysql>showcharacterset;</strong><br/><strong>+----------+---------------------------------+---------------------+--------+</strong><br/><strong>|Charset|Description|Defaultcollation|Maxlen|</strong><br/><strong>+----------+---------------------------------+---------------------+--------+</strong><br/><strong>|armscii8|ARMSCII-8Armenian|armscii8_general_ci|1|</strong><br/><strong>|ascii|USASCII|ascii_general_ci|1|</strong><br/><strong>|big5|Big5TraditionalChinese|big5_chinese_ci|2|.........</strong><br/><strong>.........</strong><br/><strong>+----------+---------------------------------+---------------------+--------+</strong><br/><strong>41rowsinset(0.01sec)</strong>
<strong>mysql>SHOWCOLLATIONWHERECharset='ascii';</strong><br/><strong>+------------------+---------+----+---------+----------+---------+---------------+</strong><br/><strong>|Collation|Charset|Id|Default|Compiled|Sortlen|Pad_attribute|</strong><br/><strong>+------------------+---------+----+---------+----------+---------+---------------+</strong><br/><strong>|ascii_bin|ascii|65||Yes|1|PADSPACE|</strong><br/><strong>|ascii_general_ci|ascii|11|Yes|Yes|1|PADSPACE|</strong><br/><strong>+------------------+---------+----+---------+----------+---------+---------------+</strong><br/><strong>2rowsinset(0.00sec)</strong>
<strong>CREATETABLEemployee(</strong><br/><strong>firstnameCHAR(10)CHARACTERSETlatin1,</strong><br/><strong>lastnameCHAR(10)CHARACTERSETascii</strong><br/><strong>);</strong><br/><br/><strong>INSERTINTOemployeeVALUES('Mona','Singh');</strong><br/><br/><strong>selectconcat(firstname,lastname)fromemployee;</strong><br/><strong>+----------------------------+</strong><br/><strong>|concat(firstname,lastname)|</strong><br/><strong>+----------------------------+</strong><br/><strong>|MonaSingh|</strong><br/>
<strong>+----------------------------+</strong><br/><strong>1rowinset(0.00sec)</strong>
UTF-8formetadata:Metadatameansthedataaboutthedata.Intermsofdatabasewecansaythatanythingthatdescribesdatabaseobjectsisknownasmetadata.Forexample:columnnames,usernames,andmanymore.MySQLfollowsthebelowtworulesformetadata:
Includeallcharactersinallthelanguagesformetadata;thisenablesausertousehisownlanguageforcolumnnameandtablename.Manageonecommoncharactersetforallmetadata.Otherwise,theSHOWandSELECTstatementsfortablesinINFORMATION_SCHEMAwillnotworkproperly.
TofollowtheaboverulesMySQL8storesmetadataintotheUnicodeformat.ConsiderthatMySQLfunctionssuchasUSER(),CURRENT_USER(),SESSION_USER(),SYSTEM_USER(),DATABASE(),andVERSION()havetheUTF-8charactersetbydefault.MySQL8serverhasdefinedcharacter_set_systemtospecifycharactersetsformetadata.MakesurethatthestorageofmetadatainUnicodedoesnotmeanthatcolumnheadersandtheDESCRIBEfunctionwillreturnvaluesintheformofthemetadatacharacterset.Itworksasperthecharacter_set_resultssystemvariable.
Addingthecharacterset
ThissectiondescribeshowtoaddcharactersetsinMySQL8.Thismethodmayvarybasedonthecharactersettype-itmightbesimpleorcomplexdependingonthecharactertype.ThefollowingfourstepsarerequiredforaddingcharactersetsintoMySQL8:
1. Add a <charset> element for MYSET to the sql/share/charsets/Index.xml file. Forthesyntax,referthealreadydefinedfilefortheothercharacterset.
2. Inthisstep,theprocessisdifferentforsimpleandcomplexcharactersets.For simple character sets, create a configuration file, MYSET.xml, in thesql/share/charsets directory to describe the character set properties. In thecaseofcomplexcharactersets,theCsourcefileisrequired.Forexample,create the ctype-MYSET.c type in the strings directory. For each <collation>element,providectype-MYSET.cfile.
3. Modifytheconfigurationinformation:
1. Editmysys/charset-def.c,andregisterthecollationsforthenewcharacterset.Addtheselinestothedeclarationsection:
#ifdefHAVE_CHARSET_MYSET
externCHARSET_INFOmy_charset_MYSET_general_ci;
externCHARSET_INFOmy_charset_MYSET_bin;
#endif
Add these lines to the registration section: #ifdefHAVE_CHARSET_MYSETadd_compiled_collation(&my_charset_MYSET_general_ci);add_compiled_collation(&my_charset_MYSET_bin);#endif
2. Ifthecharactersetusesctype-MYSET.c,editstrings/CMakeLists.txtandaddctype-MYSET.ctothedefinitionoftheSTRINGS_SOURCESvariable.
3. Editcmake/character_sets.cmakewiththefollowingchanges:
AddMYSETtothevalueofwithCHARSETS_AVAILABLEinalphabeticorder.
AddMYSET tothevalueofCHARSETS_COMPLEXinalphabeticorder.Thisis needed even for simple character sets, or CMake will notrecognizeDDEFAULT_CHARSET=MYSET.
4. Reconfigure,recompile,andtest.
ConfiguringthecharactersetsMySQL 8 provides the --character-set-server and --collation-server options toconfigurethecharactersets.Thedefaultcharactersethasbeenchanged
fromlatin1 toUTF8.UTF8 is thedominatingcharacter set, though ithadn'tbeenadefaultoneinpriorversions
ofMySQL.Withthesechangesgloballyaccepted,charactersetsandcollationsarenowbasedonUTF8;oneofthecommonreasonsisbecausetherearearound21differentlanguagessupported
byUTF8,whichmakessystemsprovidemultilingualsupport.Beforeconfiguringcollation,
refertothecollationlistavailableathttps://dev.mysql.com/doc/refman/8.0/en/show-collation.html.
<strong>SETlc_messages='fr_FR';</strong>
<strong>mysqld--lc_messages_dir=/usr/share/mysql--lc_messages=nl_NL</strong>
IfthemessagefiledoesnotexistunderthedirectorythenMySQL8willignorethevalueofthelc_messagesvariableandconsiderthevalueofthelc_messages_dirvariableasalocationinwhichtolook.IftheMySQL8serverdoesnotfindthemessagefile,thenitshowsamessageintheerrorlogfileandusesEnglishforthemessages.
<strong>mysql>SETGLOBALtime_zone=timezone;</strong>
<strong>mysql>SETtime_zone=timezone;</strong>
Thissessionvariableaffectsthedisplayandstorageofzonespecificvalues.Forexample,valuesreturnedbytheNOW()andCURTIME()functions.Ontheotherhand,thisvariabledoesnotaffectsvalueswhicharedisplayedandstoredinUTCformat,suchaswiththeUTC_TIMESTAMP()function.
<strong>mysql>SETNAMES'utf8';</strong><br/><strong>QueryOK,0rowsaffected(0.09sec)</strong><br/><br/><strong>mysql>SELECT@@lc_time_names;</strong><br/><strong>+-----------------+</strong><br/><strong>|@@lc_time_names|</strong><br/><strong>+-----------------+</strong><br/><strong>|en_US|</strong><br/><strong>+-----------------+</strong><br/><strong>1rowinset(0.00sec)</strong><br/><br/><strong>mysql>SELECTDAYNAME('2010-01-01'),MONTHNAME('2010-01-01');</strong><br/><strong>+-----------------------+-------------------------+</strong><br/><strong>|DAYNAME('2010-01-01')|MONTHNAME('2010-01-01')|</strong><br/><strong>+-----------------------+-------------------------+</strong><br/><strong>|Friday|January|</strong><br/><strong>+-----------------------+-------------------------+</strong><br/><strong>1rowinset(0.00sec)</strong><br/><br/><strong>mysql>SELECTDATE_FORMAT('2010-01-01','%W%a%M%b');</strong><br/><strong>+-----------------------------------------+</strong><br/><strong>|DATE_FORMAT('2010-01-01','%W%a%M%b')|</strong><br/><strong>+-----------------------------------------+</strong><br/><strong>|FridayFriJanuaryJan|</strong><br/><strong>+-----------------------------------------+</strong><br/><strong>1rowinset(0.00sec)</strong><br/><br/><strong>mysql>SETlc_time_names='nl_NL';</strong><br/><strong>QueryOK,0rowsaffected(0.00sec)</strong><br/><br/><strong>mysql>SELECT@@lc_time_names;</strong><br/><strong>+-----------------+</strong><br/><strong>|@@lc_time_names|</strong><br/><strong>+-----------------+</strong><br/><strong>|nl_NL|</strong><br/><strong>+-----------------+</strong><br/><strong>1rowinset(0.00sec)</strong><br/><br/><strong>mysql>SELECTDAYNAME('2010-01-01'),MONTHNAME('2010-01-01');</strong><br/><strong>+-----------------------+-------------------------+</strong><br/><strong>|DAYNAME('2010-01-01')|MONTHNAME('2010-01-01')|</strong><br/><strong>+-----------------------+------------------
-------+</strong><br/><strong>|vrijdag|januari|</strong><br/><strong>+-----------------------+-------------------------+</strong><br/><strong>1rowinset(0.00sec)</strong><br/><br/><strong>mysql>SELECTDATE_FORMAT('2010-01-01','%W%a%M%b');</strong><br/><strong>+-----------------------------------------+</strong><br/><strong>|DATE_FORMAT('2010-01-01','%W%a%M%b')|</strong><br/><strong>+-----------------------------------------+</strong><br/><strong>|vrijdagvrjanuarijan|</strong><br/><strong>+-----------------------------------------+</strong><br/><strong>1rowinset(0.00sec)</strong></strong>
Theerrorlog
Thislogisusedtorecorddiagnosticmessageslikeerror,warningsandnotesthatoccur from the startup of MySQL 8 through till its end. MySQL 8 providesvariousconfigurationsandcomponentsforuserstogeneratelogfilesaspertheirrequirements.Whenwe startwriting into files some basics questions come tomind;whatdowewrite?Howdowewrite it?Wheredowewrite it to?Let'sstartwithfirstquestion.MySQL8
usesthelog_error_verbositysystemvariableandassignsthebelowfilteringoptionstodecidewhattypeofmessagesshouldbewrittenintotheerrorlogfile:
ErrorOnly
ErrorsandWarnings
Errors,WarningsandNotes
TowriteatthedestinationplaceMySQLusesthebelowformatwherethetimestamp depends on the log_timestamps system variable: timestamp thread_id[severity]message
Afterwritinglogfiles,thefirstquestionthatcomestomindis,howdoweflushtheselogs?Forthat,MySQL8providesthreeways;FLUSHERRORLOGS,FLUSHLOGS,ormysqladminflush-logs.Thesecommandswillcloseandreopenthelogfiletowhichitiswriting.Whenwetalkabouthowtowriteandwheretowrite,therearesomanythingstounderstand.
Componentconfiguration
MySQL 8 uses the log_error_services system variable to control error logcomponents. It allows users to define multiple components by semicolons,separatedfor theexecution.Here,componentswillbeexecuted in theorder inwhichtheyaredefined.Theuserisallowedtochangethevaluesofthisvariablewiththefollowingconstraints:
INSTALLCOMPONENT:Toenableanylogcomponentwemustfirstinstallitusingthiscommand,andthenusethecomponentbylistingitinlog_error_servicessystem variable. Follow the following commands to add thelog_sink_syseventlogcomponent:
INSTALLCOMPONENT'file://component_log_sink_syseventlog';
SETGLOBALlog_error_services='log_filter_internal;
log_sink_syseventlog';
Afterexecutionof the installationcommandMySQL8will register thecomponent into the mysql.component system table tomake it available forloadingoneachstartup.
UNINSTALL COMPONENT: To disable any of the log components, first remove itfromthelog_error_servicessystemvariablelistandthenuninstallitwiththiscommand.Executethebelowcommandtouninstallacomponent:
UNINSTALLCOMPONENT'file://component_log_sink_syseventlog';
Toenableerrorlogcomponentsoneachstartup,defineitinthemy.cnffileoruseSET_PERSIST.Whenwedefineitinmy.cnfittakeseffectfromthenextrestart, whereas SET_PERSIST will give an immediate effect. Use thefollowingcommandforSET_PERSIST:SETPERSISTlog_error_services='log_filter_internal;log_sink_internal;log_sink_json';
MySQL8alsoallowsuserstowriteerrorlogsintosystemlogs:forMicrosoft,considerEventlog,andforUnixbasedsystems,considersyslog.Toenableerrorloggingintosystemlogfibf,configurelog_filter_internalandthesystemlogwriterlog_sink_syseventlog componentsand follow the same instructionsexplainabove.AnotherwayistowriteaJSONstringintothelogfileconfigurationlog_sink_jsoncomponent.AninterestingpointaboutaJSONwriteristhatitwillmanagefilenamingconventionsbyaddingNN(two-digitnumbers).Forexample,considerfilenamesasfile_name.00.json,file_name.01.json,andsoforth.
Default error log destinationconfigurationErrorlogscanbewrittenintologfilesoronconsole.Thissectiondescribeshowtoconfigurethedestinationoferrorlogondifferentenvironments.
Default error log destination onWindows
--console: If this is given then the console will be considered the defaultdestination.--console takesprecedenceover--log-error incaseswherebotharedefined.Ifthedefaultlocationisconsole,thenMySQL8
serversetsthelog_errorvariable'svalueasstderror.
--log-error: If this is not given, or given without naming a file, then thedefault file name is host_name.err and the file will be created in the datadirectoryunlessthe--pid-fileoptionisspecified.Ifthefilenameisspecifiedin–pid-file option, then the naming conventionwould be aPID file basenamewithasuffixof.errinthedatadirectory.
Default error log destination onUnixandUnix-LikesystemsAll the abovementioned scenarios inMicrosoftWindowswill bemanagedbythe–log_erroroptioninUnixsystems.
--log-error:Ifthisisnotgiventhenthedefaultdestinationistheconsole.Ifnofilenameisgiven,thenaswithWindowsitwillcreateafileinthedatadirectorywith the host_name.err name.The user is allowed to specify –log-errorinanoptionfileunderthemysqldormysqld_safesections.
<strong>SETGLOBALgeneral_log='OFF';</strong>
Oncethelogisdisabled,renamethelogfileandenablethelogagainwiththeONoption.Similarly,toenableordisablethelogatruntimeforparticularconnectionsusethesessionsql_log_offvariablewiththeONorOFFoption.Onemoreoptionisaligningwiththegenerallogfile,thatis,--log-output.Byusingthisoption,wecanspecifythedestinationoflogoutput;itdoesnotmeanlogsareenabled.
Thethreefollowingdifferentoptionsareavailablewiththiscommand:
TABLE:LogtotablesFILE:LogtofilesNONE:Donotlogintotablesorfiles.NONE,ifpresent,takesprecedenceoveranyotherspecifiers.
Ifthe--log-outputoptionisomitted,thenthedefaultvalueisfile.
<strong>mysql>SETGLOBALbinlog_format='STATEMENT';</strong><br/><strong>mysql>SETSESSIONbinlog_format='STATEMENT';</strong>
Therearetwoexceptionalcaseswherewecannotchangeformat:
WithinastoredprocedureorfunctionIncaseswheretherowbasedformatissetandtemporarytableisopen
MySQL8hasthe--binlog-row-event-max-sizevariabletocontrolthesizeofthebinarylogfileintermsofbytes.Assignasavaluetothisvariableamultipleof256;thedefaultvalueofthisoptionis8192.IndividualstorageenginesofMySQL8havetheirowncapabilitiesforlogging.Ifastorageenginesupportsrowbasedlogging,thenitisknownasrow-loggingcapable,andifastorageenginesupportsstatementbasedloggingthenitisknownasstatement-loggingcapable.Refertothebelowtableformoreinformationonstorageengineloggingcapabilities.
Storageengine
Rowloggingsupported Statementloggingsupported
ARCHIVE Yes YesBLACKHOLE Yes YesCSV Yes YesEXAMPLE Yes NoFEDERATED Yes YesHEAP Yes Yes
InnoDB Yes YeswhenthetransactionisolationlevelisREPEATABLE,READ,orSERIALIZABLE;Nootherwise.
MyISAM Yes YesMERGE Yes YesNDB Yes No
Asdescribeinthissectionthebinarylogwillworkbasedontypesofstatementlikesafe,unsafe,orbinaryinjected,ontheloggingformatsuchasROW,STATEMENT,orMIXED,andwiththeloggingcapabilitiesofstorageengineslikerowcapable,statementcapable,both,orneither.Tounderstandallthepossiblecasesofbinaryloggingrefertothetablegiveninthislink:https://dev.mysql.com/doc/refman/8.0/en/binary-log-mixed.html.
Theslowquerylog
Slow query logs are used to record SQL statements that takes long time toexecute.
MySQL8hasdefinedthefollowingtwosystemvariablesfortimeconfigurationofslowquery:
long_query_time:Thisisusedtodefinetheidealtimeforqueryexecution.IfaSQL statement takes longer than this time, then it is considered a slowqueryandastatementisrecordedintothelogfile.Thedefaultvalueis10seconds.
min_examined_row_limit:Thisistheminimumtimerequiredfortheexecutionofeachquery.Thedefaultvalueis0seconds.
MySQL8will not consider the initial time of acquiring a lock into executiontimeandwill returnslowquery logs intoafileonceall locksarereleasedandqueryexecutioniscompleted.WhenMySQL8isstarted,slowqueryloggingisdisabledbydefault;tostartthislogusetheslow_query_log[={0|1}]command,where0 indicates slow query log is disabled and 1 or without argument is used to
enabled it. To log administrative statements and querieswithout indexing, usethe log_slow_admin_statements and log_queries_not_using_indexes variables. Here,administrativestatementsincludeALTERTABLE,ANALYZETABLE,CHECKTABLE,CREATEINDEX,DROPINDEX,OPTIMIZETABLE, and REPAIRTABLE.MySQL8 allowsusers to specify thenameofthelogfileusing--slow_query_log_file=file_namecommand.Ifnofilenameis specified, thenMySQL8willcreatea filewith the host_name-slow.lognamingconventioninthedatadirectory.Towriteminimalinformationintothislogfileusethe--log-short-formatoption.
AlltheabovedescribedparametersarecontrolledbyMySQL8inthefollowingsequence:
1. The query must either not be an administrative statement, orlog_slow_admin_statementsmustbeenabled
2. The query must have taken at least long_query_timeseconds, orlog_queries_not_using_indexesmust be enabled and the querymust have usednoindexesforrowlookups
3. Thequerymusthaveexaminedatleastmin_examined_row_limitrows
4. The query must not be suppressed according to thelog_throttle_queries_not_using_indexessetting
The--log-outputoptionisalsoavailableforthislogfile,andhasthesameimplementationandeffectasthegeneralpurposelog.
TheDDLlog
Asnameimplies,thislogfileisusedtorecordalltheDDLstatementexecutionrelated details.MySQL8 uses this log file to recover from crashes that occurduring themetadataoperationexecution.Let's takeoneexample tounderstandthescenarios:
Drop table t1, t2: We must be sure that both the t1 and t2 tables aredropped
WhenweexecuteanyDDLstatement,arecordoftheseoperationsiswrittenintotheddl_log.logfileundertheMySQL8datadirectory.Thisfileisabinaryfileandnotinhumanreadableformat.Theuserisnotallowedtoupdatethecontentsofthislogfile.
Metadata statements recording is not required in normal execution ofMySQLserver;enableitonlyifitisrequired.
Summary
ThiswasaninterestingchapterforanyMySQL8user,wasn'tit?InthischapterweunderstoodhowMySQL8managesdifferentlogfilesandwhichlogfiletouseatwhattime.Atthesametimewealsocoveredmanyoftheadministrativefeatures,suchasglobalization,systemdatadatabase,andcomponentandpluginconfiguration,andexplainedhowtorunmultipleinstancesonasinglemachine.Thelaterpartofthechaptercoveredlogmaintenance.
Moving on to the next chapter, we will provide you with information aboutstorage engines, such aswhat the different types of storage engine are,whichone to use for your application, and how to create our own custom storageengineforMySQL8.
MySQL8StorageEngines
In the previous chapter, we learned about setting up a new system, datadictionary,andsystemdatabase.Detailedinformationwasprovidedoncachingtechniques, globalization, different types of components, and pluginconfiguration,alongwithseveraltypesoflogfileswhichareveryimportantforadministration.
ThischaptergivesdetailedinformationonMySQL8storageengines.ItexplainstheInnoDB storageengineand its features indetailandalsoprovidesapracticalguideline on custom storage engine creation and how tomake it pluggable sothatitcanbeinstalledinMySQL8.Thetopicsthatwewillbecoveringinthischapterareasfollows:
Overviewofstorageengines
Severaltypesofstorageengines
TheInnoDBstorageengine
Creatingacustomstorageengine
Overviewofstorageengines
StorageenginesareMySQLcomponentsforhandlingtheSQLoperationsusedin different types of tables. MySQL storage engines are designed to managedifferenttypesoftasksindifferenttypesofenvironments.Itisveryimportanttoknow and choose which storage engine is best suited for the system orapplication requirements. In following sections, wewill get to know in detailaboutthetypesofstorageengines,thedefaultstorageengine,andthecreationofcustomstorageengines.
Letusgothroughandseewhythestorageengineisaveryimportantcomponentindatabases, includingMySQL8.Storageenginesworkwithdatabaseenginestoperformvarioustypesoftasksindifferentenvironments.Theyexecutecreate,read,update, anddeleteoperations in the formof statementsondata from thedatabase. It looksquitesimplewhenyouprovide theENGINEparameterwith thecreate table statement but there is configuration for plenty of operations to bedoneonthedataforeachoftherequestssentviaSQLstatements.
Itismuchmorethanjustpersistingdata-theenginetakescareoffeaturessuchas storage limits, transactions, locking granularity/level, multi-versionconcurrencycontrol,geospatialdata types,geospatial indexing,B-tree indexes,T-tree indexes, Hash indexes, full-text search indexes, clustered indexes, data
caches, index caches, compressed data, encrypted data, cluster databases,replication, foreignkeys,backup,querycaches,andupdatingstatistics for thedatadictionary.
MySQL storage enginearchitecture
The MySQL storage engine's pluggable architecture allows a databaseprofessionalto
select any storage engine for the specialization required in any particularapplication.
TheMySQLStorageenginearchitectureprovidesaneasyapplicationmodelandAPIwith
the consistency that isolates the database administrator and the applicationprogrammer
from all the low-level implementation details underlying at the storage level.Thus,
the application always works above different storage engines' differentcapabilities.
Itprovidesstandardmanagementandsupport services thatarecommonforall
underlying
storageengines.
Storage engines perform activities on the data that is persisted at the physicalserver
level. Such modular and efficient architecture provides solutions to specificneeds
of any particular application, such as transaction processing, high availabilitysituations,
or data warehousing, and at the same time has the advantage of independentinterfaces
andservicesfromtheunderlyingstorageengines.
The database administrator and the application programmer interact with theMySQL
database by Connector APIs and services on top of the storage engines. Theapplication
isshieldedbytheMySQLserverarchitecturefromthedetailedlevelcomplexityof
the storage engines by providing easy to use APIs that are consistent andapplicable
onallthestorageengines.Iftheapplicationrequireschangesintheunderlying
storageengine,orifoneormorestorageenginesareaddedtosupporttheneedsof
the application, nomajor codingorprocess changes are required toget thingsworking.
Severaltypesofstorageengine
Now we know the importance of storage engines and critical decisions toidentifywhich storageengines touse fromplentyof storageengines availablefor MySQL 8. Let us take a look at what is available and with whichspecifications. InnoDB is the name that first entered your thoughtswhen youstartedthinkingofstorageengines,right?
InnoDBisthedefaultandmostgeneral-purposestorageengineinMySQL8andit is recommendedbyOracle touse for tablesaswellas forspecialusecases.TheMySQL
server has a pluggable storage engine architecture that enables storage engineloadingaswellasunloadingfromthealreadyrunningMySQLserver.
To identify which storage engines your server supports is made very easy inMySQL
8.Weonly have to go to theMySQL shell or prompt and use the SHOWENGINESstatement.Hitthestatementwhenpromptedandresultwillbethelistofengineswith a few columns, such as Engine, Support, Transactions, Savepoints, andComment.
Values in Support column, DEFAULT,YES, and NO, indicate that a storageengineisavailableandcurrentlysetasthedefaultstorage.
Overview of the InnoDB storageengineInnoDB is the default and most general-purpose storage engine in MySQL 8,providinghighreliabilityandhighperformance.
If you have not configured a different default storage engine, then issuing theSQL
statementCREATETABLEwithouttheENGINE=clausecreatesa tablewith thestorageengineInnoDBasthedefaultengineinMySQL8.
The featuresandadvantagesofferedby theInnoDB storageengineareexplainedlaterintheTheInnoDBstorageenginesection.
Customstorageengine
storageenginearchitectureinMySQL5.1andallthelaterversionsandMySQL8havetakenadvantageoftheflexiblestorageenginearchitecture.
Thestorageenginepluggablearchitectureprovides thecapability tocreateandaddnewstorageengineswithoutrecompilationoftheserver,addingdirectlytoarunning MySQL server. The architecture makes it very easy to develop anddeploynewstorageenginestoMySQL8.
We will develop a new storage engine by using the pluggable feature of theMySQLstorageenginearchitectureintheupcomingCreatingacustomstorageenginesection.
Severaltypesofstorageengines
Inthissection,wewilltakeacloserlookatthewidelyusedstorageenginesthatare supported byMySQL8.But before checking on them, let us see how thestorage engine architecture has made it pluggable and provided flexibility toenableusingmultiplestorageenginesinthesameschemaorserver.
ThefollowingisthelistofstorageenginessupportedinMySQL8:
InnoDB: The default storage engine for MySQL 8. It is an ACID compliant(transaction-safe) storage engine that has commit, roll back, and crash-recoveryforprotectingtheuserdataandreferential-integrityconstraints tomaintaindataintegrity,andmuchmore.
MyISAM:Thestorageenginewithtableshavingasmallfootprint.Ithastable-level locking and so is mostly used in read-only or read-mostly dataworkloads,suchasindatawarehousingandwebconfigurations.
Memory:ThestorageenginepreviouslyknownastheHEAPengine.ItkeepsdatainRAM,whichprovidesfasterdataaccess,mostlyusedinquicklookupsofnon-criticaldataenvironments.
CSV:Thestorageenginewithtablesascomma-separatedvaluesintextfilesandtables.
TheyarenotindexedandaremostlyusedforimportinganddumpingdatainCSVformat.
Archive:Thestorageenginecomprisescompact,unindexedtables, intendedtostoreandretrieveahugeamountofhistorical,archived,orsecurityauditdata.
Blackhole: The storage engine with tables that can be used for replicationconfiguration.Aqueryalwaysreturnsanemptyset.DMLSQLstatementsaresenttoslaveservers.Itacceptsdatabutdataisnotstored,suchasinaUnix/dev/nulldeviceuse.
Merge:ThestorageengineprovidesthecapabilitytologicallygroupaseriesofsimilarMyISAM tablesand refer to themasoneobject insteadofseparatetable.
Federated:ThestorageenginethatcanlinkmanyseparatephysicalMySQLservers into one logical database. It is ideal for datamarts or distributedenvironments.
Example: The storage engine that does nothing but works as a stub. It isprimarilyusedby thedeveloperswho illustratehow tobeginwritingnewstorageenginesintheMySQLsourcecode.
MySQL does not restrict using the same storage engine for anentire server or schema; instead, specifying the engine at tablelevelmakesitflexiblebasedonthetypeofdataandtheusecaseoftheapplication.
<strong>mysql>INSTALLPLUGINMyExampleSONAME'MyExample.so';</strong>
<strong>mysql>UNINSTALLPLUGINMyExample;</strong>
The common database serverlayer
The MySQL pluggable storage engine is responsible for executing I/Ooperationsontheactualdataandalso tocater to thespecificapplicationneedsthat includes enabling and enforcing the required features whenever required.Using a specific or single storage engine will more likely result in moreefficiency and higher database performance because the engine enables thefeatures only needed for a particular application, and resulting in less systemoverheadonthedatabase.
A storage engine supports the following unique infrastructure components orkeys:
Concurrency:Someapplicationshavegranular lock levels (such as row-level locks) requirements more than others. Overall performance andoverhead due to locking can be affected by choosing the right/wronglockingstrategyandthisalsoincludesmulti-versionconcurrencycontrolorsnapshotreadcapabilities.
Transaction support: Very well-defined requirements exist, such as ACIDcomplianceandmoreiftheapplicationneedstransactions.
Referentialintegrity:TheservercanenforcerelationaldatabasereferentialintegrityusingDDL-definedforeignkeysifrequired..
Physicalstorage:Thisincludeseverythingfromthepagesizeoftablesandindexesandalsoincludestheformatusedforstoringdataonaphysicaldiskaswell.
Indexsupport:This includes indexingstrategiesbasedon theapplicationneeds,aseachofthestorageengineshavetheirownindexingmethods.
Memorycaches:Thecachingstrategiesbasedontheapplicationneeds,aseach of the storage engines have their own caching methods along withcommonmemorycachesacrossallthestorageengines.
Performance aids: This involves bulk insert handing, database checkpointing,multiple I/O threads for parallel operations, thread concurrency,andmore.
Miscellaneous target features: This may includes support for securityrestrictionsoncertaindatamanipulationoperations,geospatialoperations,andothersimilarfeatures.
Each of the preceding infrastructure components are designed to support aspecific set of features for a particular application's needs and so it is veryimportanttounderstandtheapplicationrequirementverycarefullyandselecttheright storage engine, as it may impact on the overall system efficiency andperformance.
<strong>CREATETABLEtable1(i1INT)ENGINE=INNODB;</strong><br/><strong>CREATETABLEtable3(i3INT)ENGINE=MEMORY;</strong><br/><strong>ALTERTABLEtable3ENGINE=InnoDB;</strong>
<strong>SETdefault_storage_engine=MEMORY;</strong>
ThedefaultstorageengineforTEMPORARYtablesusingCREATETEMPORARYTABLEcanbesetseparatelybysettingthedefault_tmp_storage_enginevariableateitherstartuporruntime.
<strong>--myisam-recover-options=mode</strong>
ThisoptionwillsetthemodeintheautomaticrecoveryofcrashedtablesinMyISAM.
SpacesneededforkeysinMyISAM,B-TreeindexesareusedbyMyISAMtablesandspacecompressionisusedinStringindexes.Ifastringisthefirstpartoftheindexthenprefixcompressionisalsodone,whichoverallmakestheindexfilesizesmaller.Theprefixcompressionhelpsifmanystringshaveasimilarprefix.ByusingthetableoptionPACK_KEYS=1inMyISAMtables,prefixcompressioncanalsobeappliedonthenumbersiftherearemanynumberswithasimilarprefix.
PartitioningisnotsupportedforMyISAMtablesinMySQL8.0.
SomeoftheimportanttablescharacteristicsforMyISAMtablesareasfollows:
Alldatavaluesstoredhavethelowbytefirstorder,whichmakesthedataindependentofmachineandoperatingsystemsAllnumerickeyvaluesarestoredwithhighbytefirstorder,whichpermitsbetterindexcompressionMyISAMtableislimitedwith(232)2(1.844E+19)rowsMyISAMtableislimitedtoamaximumnumberof64indexespertableMyISAMtablecolumnsislimitedtoamaximumnumberof16columnsperindexConcurrentinsertsaresupportedinMyISAM,ifatabledoesnothaveanyfreeblocksinthemiddleofthedatafilesTEXTandBLOBtypecolumnscanalsobeindexedinMyISAMInindexedcolumns,NULLvaluesarepermittedEachofthecolumnscanhaveadifferentcharactersetItalsosupportforatrueVARCHARtypecolumnwithastarting
lengthstoredof1or2byte,tableswithVARCHARcolumnswithafixedordynamicrowlength,andUNIQUEconstraintswithanarbitrarylengthMyISAMtablestorageformats:ThefollowingthreedifferenttypesofstorageformatslistedaresupportedinMyISAM:
Statictable:ThedefaultformatforthetablesintheMyISAMstorageengine,withfixed-sizedcolumnsDynamictable:Asthenamesuggests,theformatthatcontainsvariablesizedcolumns,includingVARCHAR,BLOB,orTEXTCompressedtable:Thetableformatforkeepingread-onlydataandcompressedformatsinMyISAMstorageenginetables
Thefirsttwoformats,fixedanddynamic,arechosenautomaticallybasedonthecolumntypeused.Thecompressedformatcanbecreatedbyusingthemyisampackutility.
MyISAMtableproblems:Thefileformathasbeenextensivelytestedbutsomecircumstancesarisethatresultincorrupteddatabasetables.Letuslookatsuchcircumstancesandthewaytorecoverthosetables.
Wecouldgetcorruptedtablesintheeventofanyofthefollowingevents:
IfthemysqldprocessiskilledinthemiddleofawriteIfthereisanunexpectedcomputershutdownIfthereisanyhardwarefailureIfatableisbeingmodifiedatthesametimebytheMySQLserverandanexternalprogram,suchasmyisamchkTheMySQLorMyISAMcodehasasoftwarebug
CheckthehealthofthetablewiththeCHECKTABLEstatementandattempttorepairanycorruptedMyISAMtablebyusingtheREPAIRTABLEstatement.
ThereisalsopossibleissueyougetwithMyISAMtablesandthatistablesarenotbeingclosedproperly.Inordertoidentifyifthetableisclosedproperlyornot,eachMyISAMindexfilekeepsacounterintheheader.Thecountercanbeincorrectunderthefollowingcircumstances:
IfatableiscopiedwithoutissuingLOCKTABLESandFLUSHTABLES
MySQLcrashedbeforethefinalcloseduringanupdatemysqldisusingthetableandatthesametimeitismodifiedbyanotherprogram:myisamcheck--recoverormyisamchk--update-state
TheMEMORYstorageengine
TheMEMORYstorageengine,alsopreviouslyknownastheHEAPengine,keepsdatainRAM,whichprovidesfasterdataaccess.Itismostlyusedinquicklookupsofnon-criticaldataenvironments.Itcreatesspecial-purposetableswithcontentsstoredinmemorybut thedata isvulnerable to crashes,poweroutages, andhardwareissues. Therefore, these tables are used in temporary work areas or possiblyusingreadonlydatathatiscachedafterthedataispulledfromothertables.
You should choose whether use MEMORY or NDB Cluster. You should check if theapplicationisrequiredforimportant,highlyavailable,orfrequentlyupdateddataandconsiderwhetherNDBCluster is thebetterchoiceornot.NDBClusterprovidesthe same features as the MEMORY engine, butwith higher performance levels andadditionalfeaturesnotprovidedbyMEMORYengine.Theseinclude:
Low contention between clients by multiple thread operations and row-levellocking
Scalabilitywithstatementsmixes,includingwrites
Datadurability;itsupportsoptionaldisk-backedoperations
Shared-nothing architecture, providingmultiple-host operationswithout asinglepointoffailure,enabling99.999%availabilityfortheapplication
Automaticdatadistributionsacrossnodes
Supportforvariablelengthdatatypes,includingBLOBandTEXT
PartitioningisnotsupportedinMEMORYtables.
Performancedependsonhowbusytheserverisandtheeffectofsinglethreadexecutionwithtablelockoverheadduringupdatesprocessing.ThetablelockingduringupdatesprocessingcausesaslowdownofconcurrentusagefrommultiplesessionsonMEMORYtables.
MEMORYtablescharacteristics:Tabledefinitionsarestoredon theMySQLdata dictionary and do not create any files on the disk. The following are thetablefeaturehighlights:
100%dynamichashingforinsertsandspaceisallocatedinsmallblocks.
Noextrakeyspaceoroverflowareaorextraspaceforfreelistsisrequired.Reuseofdeletedrowswhennewrecordsinsertedbyputtingrowsinlinkedlists.
Fixed length row-storage format, VARCHAR , is stored with fixed length.CannotstoreBLOBorTEXTcolumns.
AUTO_INCREMENTcolumnsaresupported.
IndexinginHASHandBTREEtypesaresupportedbytheMEMORYstorageengine.MEMORYtableshaveamaximumof64indexespertable,amaximumof16columnsperindexandamaximumkeylengthof3,072bytes.MEMORYtablesalsocanhavenon-uniquekeys.
User createdand temporary tables: Internal temporary tables are createdbythe server on the fly while processing queries. Two types of tables differ instorageconversion,wheretheMEMORYtablesarenotsubjecttoconversion:
Whenaninternaltemporarytablebecomestoolarge,itisconvertedtoon-diskstoragebytheserverautomatically
UsercreatedMEMORYtablesareneverconvertedbytheserver
Dataloadingcanbeperformedusingthe--init-fileoption,usingINSERTINTO...SELECTorLOADDATAINFILEstatementsfromanypersistencedatasourceifrequired.
TheCSVstorageengine
This storage engine stores data in the formof comma-separatedvalues in textfiles.
The engine is always compiled into theMySQL server and the source can beexaminedfromthestorage/csvdirectoryofyourMySQLdistribution.
Thedatafilecreatedbytheserverbeginswiththegiventableandtheextensionof.CSV.Thedata file isaplain text filecontainingdata in thecomma-separatevaluesformat.
TheMySQLservercreatesacorrespondingmetafilealongwithaCSVtablethatstores information about the state of the table and the count for the rows thatexists in the table. The metafile is also stored with the table name at thebeginningwiththe.CSMextension.
RepairingandcheckingCSVtables:ThestorageenginesupportsCHECKandREPAIRstatementstoverifyandpossiblyrepairadamagedCSVtable.YoucanusetheCHECKTABLEstatementtoverifyorvalidatethetableandusetheREPAIRTABLEstatementtorepairatablethatcopiesvalidrowsfromanexistingCSV
datafileandreplacesanexistingfilewithnewlycopied/recoveredrows.
Duringrepair,onlyrowsfromtheCSVdatafiletothefirstdamagedrowgetscopiedtothenewtableorcopieddatafile.
Therestoftherowsafterthedamagedrowgetsremovedfromthetable,includingvalidrows,soIsuggestthatyoutakeenoughbackupofthedatafilepriortoproceedingwiththerepair.
Indexing or partitioning is not supported in the CSV storage engine and all thetablescreatedwiththeCSVstorageenginemusthavetheNOTNULLattributeonallthecolumns.
TheARCHIVEstorageengine
TheARCHIVEstorageenginecreatesspecial-purposetablesthatareusedforstoringhugeamountsofunindexeddatawithaverysmallfootprint.
WhentheARCHIVEtableiscreated,itbeginswiththetablenameandendswiththe.ARZextension.Duringoptimizationoperations,afilewithan.ARNextensionmayappear.
The AUTO_INCREMENT column attribute is supported by the engine. It also supportsINSERT,REPLACE,SELECT,andBLOBcolumns(allbutspatialdatatypes)butitdoesnotsupportDELETE,UPDATE,ORDER,orBYoperations.
PartitioningisnotsupportedbytheARCHIVEstorageengine:
Storage:Theengineuseslosslessdatacompressionwithzlibandtherowsget compressed as inserted. It supports the CHECK TABLE operation. Severaltypesofinsertionareusedintheengine:
INSERT statementsends rows intoacompressionbuffer,and thebuffergetsflushedasnecessary.
InsertioninthecompressionbufferisprotectedbythelockandflushwillonlyoccurifSELECTisrequested.
Oncecompletedabulkbuffercanbeseen.Itcanonlybeseenifanyother inserts occur at the same time.Here flushwill not occur uponSELECT,unlesswhileloadinganynormalinsert.
Retrieval:Afterretrieval,rowsgetsuncompressedasrequestedanditdoesnot use any rowcache.A complete table scan is performed for the SELECToperation:
SELECT checks howmany rows are available currently and reads onlythatnumberofrows.
Itisperformedasaconsistentreadoperation.
ThenumberofrowsreportedbySHOWTABLESTATUSisalwaysaccuratefortheARCHIVEtables.
Use OPTIMIZE TABLE or REPAIR TABLE operations to achieve bettercompression.
TheBLACKHOLEstorageengine
TheBLACKHOLE storage engine acts as a black hole. It accepts data but does notstoreitandaqueryalwaysreturnsanemptyresult.
TheserveronlyaddsthetabledefinitionintheglobaldatadictionarywhenyoucreateaBLACKHOLEtableandnofilesareassociatedwiththetable.
AllkindsofindexingissupportedintheBLACKHOLEstorageengineandsothesamecanbeincludedinthetabledefinition.
PartitioningisnotsupportedintheBLACKHOLEstorageengine.
Insertiontothetabledoesnotstoreanydatabutifbinaryloggingisenabledforstatements, then the statements are logged and replicated to the slave servers.Suchamechanismisusefulasafilterorrepeater.
TheBLACKHOLEstorageenginehasthefollowingpossibleuses:
Dumpfilesyntaxverification
Overhead measurement using binary logging enabled or disabled with aBLACKHOLEperformancecomparison
Itcanalsobeusedforfindinganyperformancebottlenecks,exceptforthestorageengineitself
Autoincrementcolumns:Astheengineisano-opengine,itwillnotincrementany field values but it has implications in the replication, which can be veryimportant.Considerascenariothathasthefollowingconditions:
1. ThemasterserverhasaBLOCKHOLE tablewithanautoincrementfieldwithaprimarykey
2. ThesametableexistsontheslaveserverbutusestheMyISAMengine
3. Insertionisperformedintothemasterserver'stablewithoutsettinganyautoincrementvalueintheINSERTstatementorusingtheSETINSERT_IDstatement
In the preceding scenario, the replicationwill fail on the primary key columnwithaduplicateentry.
TheMERGEstorageengine
The MERGE storage engine, known also as the MRG_MyISAM engine, is collection ofsimilartablesthatcanbeusedasonetableinstead.Here,"similar"meansthatallthetableshavesimilarcolumndatatypesandindexinginformation.
Itisnotpossibletomergetableswiththecolumnslistedinadifferentorderortohavethesamedata types inrespectivecolumnsorhaveindexinginadifferentorder.
Thefollowingisthelistofdifferencesintablesthatwillnotrestrictamerge:
Namesofrespectivecolumnsandindexescanbedifferent.
Commentsinbetweentables,columns,andindexescanbedifferent.
AVG_ROW_LENGTH,MAX_ROWS,orPACK_KEYStableoptionscanbedifferent.
WhenaMERGEtableiscreated,MySQLalsocreatesa.MRGfileonthediskwiththenamesofunderlyingMyISAM tablesbeingusedasone.Theformatof the table isstoredintheMySQLdatadictionaryandtheunderlyingtablesdonotrequireto
beinthesamedatabaseastheMERGEtable.
HavingprivilegesareamustforSELECT,UPDATE,andDELETEontheMyISAMtablesthatare being mapped with the MERGE table and so SELECT, INSERT, UPDATE, and DELETEstatementsontheMERGEtablecanbeused.
Executing the DROP TABLE statement on the MERGE table will drop only thespecificationfortheMERGEandnothingisimpactedontheunderlyingtables.
UsingMERGEtableshasthefollowingsecurityissues.IftheuserhasaccesstotheMyISAMtablet1,thentheusercancreatetheMERGEtablem1thatcanaccesst1.Now,iftheuser'sprivilegesonthetablet1arerevoked, the user can still continue accessing table t1 by usingtablem1.
TheFEDERATEDstorageengine
TheFEDERATEDstorageenginecanlinkmanyseparatephysicalMySQLserversintoonelogicaldatabase
and so it can let you access data from a remoteMySQL serverwithout usingeither
replicationorclustertechnology.
WhenwequerytothelocalFEDERATEDtable,thatautomaticallypullsthedatafromtheremotefederatedtablesandthe
dataisnotrequiredtobestoredonlocaltables.
TheFEDERATEDstorageengineisnotsupportedbydefaultintheMySQLserverbutstartingtheserver
withthe--federatedoptionwillenabletheFEDERATEDengineoption.
WhentheFEDERATEDtableiscreatedthetabledefinitionisthesameasothertables,butthephysical
storage of the associated data is handled on the remote server instead. TheFEDERATEDtableconsistsofthefollowingtwoelements:
Aremoteserverwithadatabasetableconsistingofatabledefinitionandtheassociatedtabledata.
ThistypeoftablecanbeanysupportedbytheremoteserverthatincludesMyISAMorInnoDBaswell.
A local server with a database table consisting of a table definition thesameastherespective
table on the remote server. The table definition is stored in the datadictionary
andnoassociateddatafileonthelocalserverisstored.Instead,inadditionto
the table definition, it keeps a connection string that is pointing to theremote
tableitself.
The following is the flowof information between the local and remote serverwhen
aSQLstatementisexecutedontheFEDERATEDtable:
1. The engine checks each of the columns the table has and builds anappropriateSQL
statementthatreferstotheremotetable.
2. TheMySQLclientAPIisusedforsendingtheSQLstatementtotheremoteserver.
TheEXAMPLEstorageengineTheEXAMPLEstorageengineisonlyastubengineandthepurposeoftheengineistoprovideexamplesintheMySQLsourcecode,whichhelpsdeveloperstowritenewstorageengines.
ToworkwiththeEXAMPLEenginesourcecode,lookatthestorage/exampledirectoryoftheMySQLsourcecodedistributiondownload.
NofilesarecreatedifthetableiscreatedwiththeEXAMPLEengine.DatacannotbestoredintheEXAMPLEengineanditreturnsemptyresults.
IndexingandpartitioningisnotsupportedintheEXAMPLEstorageengine.
TheInnoDBstorageengine
InnoDB is themost general-purpose storage engine and is the default engine inMySQL8,providinghighreliabilityandhighperformance.
ThefollowingarethekeyadvantagesofferedbytheInnoDBstorageengine:
Its DML operations follows the ACID model and transactions have commit,rollback,andcrash-recoveryfeaturestoprotectuserdata
Oracle-style gives consistent reads and row level locking increases theperformanceofmulti-userconcurrency
EachInnoDB table has a primary key index, known as the clustered index,thatarrangesdataonthediskinordertooptimizequeriesbasedonprimarykeyandminimizesI/Oduringprimarykeylookups
By supporting foreign keys, inserts, deletes, and updates are checked,ensuring consistency across different tables in order to maintain dataintegrity
ThefollowingarethekeybenefitsofusingInnoDBtables:
If the server crashesdue to anyhardwareor software issue, regardlessofwhat changeswere being processed in the server at that time, you're notrequired to do anything special after restarting the server. It has a crashrecoverysystemthattakescareofchangesthatwerecommittedduringthecrash of the server. It will go to those changes and start where theprocessingwasleftoff.
The engine has it's own buffer pool used for caching table and indexingdata to memory based on data accessed. Frequently used data is fetcheddirectly from the cache memory and so it speeds up processing. Indedicated servers, it takes up to 80%of physicalmemory assigned to beusedinthebufferpool.
Splittingrelateddata to tablesusingforeignkeysetupenforcesreferentialintegritywhichprevents inserting anyunrelateddata to a secondary tablewithouttherespectivedataintheprimarytable.
In case of corrupt data in thememory or disk, the checksummechanismgivesanalertaboutthecorruptdatabeforewegettouseit.
Change buffering automatically optimizes Insert, Update, and Delete. InnoDBalsoallowsconcurrentreadandwriteaccesstothesametableandcachingdatachangestostreamlinethediskI/O.
When the same data rows are accessed from the table repeatedly, theAdaptive Hash Index feature makes the lookups faster and givesperformancebenefits.
Compressionisallowedontablesandassociatedindexes.
Monitoring on internal workings and performance details of the storageengineiseasybyqueryingINFORMATION_SCHEMAorPerformanceSchematables.
Now let us look at each of the areas of the storage engine where InnoDB isenhancedoroptimizedtoprovideveryefficientandenhancedperformance.
ACIDmodelThe ACID model is a group of database design principles with an emphasis onreliability,whichismostimportantformissioncriticalapplicationsandbusinessdata.
MySQLhascomponentssuchastheInnoDBstorageenginethatcloselyadheretothe ACIDmodel. Therefore, data is safe and not corrupted, even in exceptionalcasesofhardwaremalfunctionsorsoftwarecrashes.
WithMySQL8,InnoDB supportsatomicDDL, ensuring that the DDL operations arefullycommittedorrolledback,eveniftheserverishaltedwhileperformingtheoperation.NowDDLlogscanbewrittentothemysql.innodb_ddl_logconfigurationforthedatadictionary tables,enabling theinnodb_print_ddl_logsconfigurationoptiontoprintDDLrecoverylogstostderr.
MultiversioningInnoDBisamultiversionedstorageengine.Thatmeans ithas thecapability tokeep old versions of changed row data information and support transnationalfeatures, such as concurrency and roll back. The information is stored in thetablespace,datastructure,andnamedrollbacksegment.
Internally,foreachoftherowsgettingstoredinthedatabase,InnoDBcreatesthreefields: 6-byte DB_TRX_ID, 7-byte DB_ROLL_PTR (called a roll pointer) and 6-byteDB_ROW_ID. With these fields, InnoDB creates clustered indexes to keep theinformationofchangedrowdatainthedatabase.
Architecture
Inthissection,wewillgiveabriefintroductiontothemajorcomponentsoftheInnoDBarchitecture:
Buffer pool: Area of main memory where tables and indexing data arecachedtospeedupprocessing
Changebuffer: Special data structurewhere changes to secondary indexpagesarecached
Adaptive Hash Index: Enables in-memory database, such as lookups,operations on systemswith balanced and appropriate combinations of thebufferpool'smemoryandworkload
Redologbuffer:Memoryareawheredataisheldtobewrittenontheredolog
System tablespace: Storage area where the doublewrite buffer, undo logs,and thechangebuffer,prior to theMySQL8datadictionary information,arestored
Doublewritebuffer:Storageareainthesystemtablespacewherepagesarewrittenthatareflushedfromthebufferpool
Undologs:Collectionofundo log recordswhichareassociatedwithanysingletransaction
File-per-table tablespaces: Single-table tablespace added to its own datafile
General tablespaces: Shared tablespace created by the CREATE TABLESPACE
syntax
Undotablespace:Oneormorefileswithundologs
Temporarytablespace:Utilizedfornon-compressedtemporarytablesandtheirrelatedobjects
Redo log: Disk-based data structure used for correcting incompletetransactiondataduringcrashrecovery
With MySQL 8, the InnoDB storage engine utilizes the global MySQL datadictionaryandnotitsownstorageengine-specificdatadictionary.
Lockingandtransactionmodel
ThissectiongivesbriefinformationonlockingusedbyInnoDBandthetransactionmodelimplementedbyInnoDB.InnoDBusesthefollowingdifferentlocktypes:
Sharedandexclusivelocks:Twotypesofstandardrow-level lockingareimplemented. A shared lock allows you to read a row to differenttransactions;anexclusivelockholdstoupdateordeletearowanddoesnotallowyoutoevenreadtherowtoanydifferenttransaction.
Intention locks:Table level locks to supportmultiple granularity lockingbywhichInnoDBpracticallymaintainsthecoexistenceofrow-levellocksandentiretable-levellocks.
Record locks: Index record lock that prevents any other transaction toinsert,update,ordeletetherecord.
Gaplocks:Lockappliesonagap(range)betweenindexrecords.
Next-keylocks:Combinationofindexrecordlockplusgaplockonthegapfortheprecedingindexrecord.
Insert intention locks:Typeof gap lockwhich is set by INSERToperationjustbeforetherowinsertion.
AUTO-INClocks: Special table-level lock for inserting recordswith theAUTO_INCREMENTcolumn.
Predicate locks for spatial indexes: Lock on spatial index, enablingsupportforisolationlevelsintableswithspatialindexes.
The goal of following the transaction model is to unite traditional two-phaselocking with the best of the multiversioning database properties. Row-levellockingisperformedandqueriesarerunwithnonlockingconsistentreads.InnoDBtakescareoftransactionisolationlevels,autocommit,rollbackandcommit,andlocking reads. It allows nonlocking consistent reads as applicable. InnoDB alsouses a mechanism to avoid phantom rows and a configuration to supportautomaticdeadlockdetection.
Configuration
Thissectionprovidesbrief informationabout theconfigurationandproceduresusedintheInnoDBinitializationstartupfordifferentInnoDBcomponents:
InnoDBstartupconfiguration:This involvesspecifyingstartupoptions, logfileconfiguration,storageconsiderations,systemtablespacedatafiles,undotablespaces,temporarytablespaces,pagesizes,andmemoryconfigurations
InnoDB forread-onlyoperation:This enables aMySQL instance for read-only operation, using the --innodb-read-only=1 option,which is very helpfulwhenusingread-onlymediasuchasCDorDVD
InnoDBbufferpoolconfiguration:Configuresthebufferpoolsize,multipleinstances,flushing,andmonitoring
InnoDB change buffering: Configures the change buffer options forsecondaryindexcaching
Thread concurrency for InnoDB: Concurrent thread count limitconfiguration
ThenumberofbackgroundInnoDBI/Othreads:ConfiguresthenumberofbackgroundthreadsservicingI/Oread/writeoperationsondatapages
Using asynchronous I/O on Linux: A configuration to use nativeasynchronousI/OsubsytemsonLinux
TheInnoDBmaster thread I/Orate:Configuresoverall I/Ocapacity for amasterthreadworkinginthebackground,responsibleformultipletasks
Spin lock polling: Configures a spin wait delay period to control themaximumdelayforfrequentpollingbetweenmultiplethreadsrequestingtoacquiremutexesorrw-locks
InnoDBpurgescheduling:Configurespurgethreadsforapplicablescalability
Optimizer statistics for InnoDB: Configures persistent and non-persistentoptimizerstatisticsparameters
Themergethresholdforindexpages:ConfiguresMERGE_THRESHOLDtoreducemerge-splitbehavior
Enabling automatic configuration for a dedicated MySQL Server:Configures the dedicated server option --innodb_dedicated_server , whichmakesautomaticconfigurationforthebufferpoolsizeandlogfilesize
Tablespaces
ThissectionprovidesbriefinformationontablespacesandoperationsrelatedtotablespacesperformedinInnoDB:
ResizingtheInnoDB systemtablespace: Increasinganddecreasing thesizeof the system tablespace with configuration while starting/restarting theMySQLserver.
Changing the number or size of InnoDB redo log files: Configuresinnodb_log_files_in_group and innodb_log_file_size values respectively in my.cnfpriortostarting/restartingtheMySQLserver.
Usingrawdiskpartitionsforthesystemtablespace:Configurestherawdiskpartitionstobeusedasdatafilesinthesystemtablespace.
InnoDBFile-Per-Tabletablespaces:Thefeatureinnodb_file_per_table enabledbydefaultwhichensuresthateachofthetablesandassociatedindexesarestoredinaseparate.idbdatafile.
Configuringundotablespaces:Aconfigurationtosetthenumberofundo
tablespaceswhereanundologresides.
Truncatingundotablespaces:Configuresinnodb_undo_log_truncate toenabletruncating undo tablespace files exceeding themaximum limit defined ininnodb_max_undo_log_size.
InnoDB general tablespaces: A shared tablespace created using the CREATETABLESPACEstatement.Itissimilartoasystemtablespace.
InnoDB tablespaceencryption:Support fordataencryption in tablesstoredas file-per-table tablespaces which use the AES block-based encryptionalgorithm.
Tablesandindexes
This section provides brief information on InnoDB tables and indexes and theirrelatedoperations:
CreatingInnoDBtables:CreatestablesusingtheCREATETABLEstatement.
ThephysicalrowstructureofanInnoDB table:Dependson the specifiedrow format during the table creation. If not specified, uses the default,DYNAMIC.
Moving or copying InnoDB tables: Different techniques for moving orcopyingsomeorallInnoDBtablestoadifferentinstanceorserver.
Converting tables from MyISAM to InnoDB: Considers guidelines and tipswhile converting MyISAM tables to InnoDB tables, except a partitioned table,whichisnotsupportedwithMySQL8.
AUTO_INCREMENThandlinginInnoDB:ConfiguresthemodeforAUTO_INCREMENTwiththe innodb_autoinc_lock_mode parameter as 0,1, and 2 for traditional,consecutive, or interleaved, respectively, where interleaved is the default
modefromMySQL8.
LimitsonInnoDBtables:Atablecancontainamaximumof1,017columns,amaximumof64secondaryindexes,andseveralotherlimitsdefinedbasedonthepagesize,tablesize,anddata-rowformats.
Clustered and secondary indexes: InnoDB uses a special index called aclusteredindex.Therestoftheindexesarecalledsecondaryindexes.
ThephysicalstructureofInnoDBindex:Forspatialindexes,InnoDBusestheR-treedatastructure,aspecializeddatastructure.Forrestoftheindexes,theB-treedatastructureisused.
Sorted index builds: Bulk load when creating or rebuilding indexes forinserts. They are known as sorted index builds, and are not supported inspatialindexes.
InnoDBFULLTEXTindexes:Createdfortext-basedcolumns-char,varchar,ortexttype.Theyhelptospeedupqueriesandsearchingoperations.
<strong>mysql>SHOWTABLESFROMINFORMATION_SCHEMALIKE'INNODB%';</strong>
Tablesaboutcompression:Thenumberofcompressionoperationsandtheamountoftimespentforcompression-relatedinformationprovidedintheINNODB_CMPandINNODB_CMP_RESETtables.MemoryallocationduringcompressionisprovidedintheINNODB_CMPMEMandINNODB_CMPMEM_RESETtables.Transactionandlockinginformation:INNODB_TRXhasinformationontransactionscurrentlyexecutingandthedata_locksanddata_lock_waitstablesfromthePerformanceSchematablegiveinformationaboutthelocks.Schemaobjecttables:ThisprovidesmetadatainformationabouttheInnoDBschemaobjects.FULLTEXTindextables:ThisprovidesmetadatainformationaboutFULLTEXTindexes.Bufferpooltables:Thisprovidesstatusinformationandmetadataaboutthepagesinthebufferpool.Metricstable:Thisprovidesperformanceandresourcerelatedinformation.Temporarytableinformationtable:Thisprovidesmetadatainformationaboutallusersandsystem-createdtemporarytablescurrentlyactiveinanInnoDBinstance.RetrievingInnoDBtablespacemetadata:ThisprovidesmetadatainformationaboutallthetypesoftablespacesinanInnoDBinstance.
Anewview,INNODB_TABLESPACES_BRIEF,hasbeenaddedtoprovidethename,path,flag,space,andspacetypedata.
Anewtable,INNODB_CACHED_INDEXES,hasbeenaddedtoprovidethenumberofindexpagescachedinthebufferpoolforeachindex.
Memcachedplugin
MySQL8providesyouwiththeInnoDBmemcachedpluginnameddaemon_memcached,which can help us in managing data easily. It will automatically store andretrieve data from InnoDB tables and provide get, set, and incr operations thatremoveperformanceoverheadbyskippingSQLparsing,whichspeedsupdataoperations. The memcached plugin uses the integrated memcached daemon thatautomaticallyretrievesandstoresdatafromandtotheInnoDBtable,enablingtheMySQLservertosenddataquicklytothekey-valuestore.
ThefollowingarethemajorbenefitsofusingtheInnoDBmemcachedplugin:
Accesses theInnoDB storageenginedirectly, reducingparsingandplanningSQLoverhead
memcached uses the same process space as the MySQL server, reducingnetworkoverhead
Datawrittenorrequestedinthememcachedprotocolistransparentlywrittenorqueried from InnoDB tables, reducing having to go through SQL layeroverhead
Simplifiesapplicationlogicbyautomaticallytransferingbetweendiskandmemory
TheMySQLdatabasestoresdatasothatitisprotectedagainstcorruption,crashes,oroutages
Ensures high availability using the daemon_memcached plugin on the masterserverandMySQLreplicationincombination
Repeated data requests are cached using the InnoDB buffer pool, providinghighspeedprocessing
As the data is stored in the InnoDB tables, thedata consistency is enforcedautomatically
The InnoDB memcached plugin supports multiple get operations (fetching multiplekey/valuepairsinasinglememcachedquery)andrangequeries.
Creatingacustomstorageengine
MySQLABintroducedpluggablestorageenginearchitectureinMySQL5.1andalllater
versions, including MySQL 8, have taken advantage of the flexible storageenginearchitecture.
Thestorageenginepluggablearchitectureprovides thecapability tocreateandadd
newstorageengineswithoutrecompilingtheserver,addingdirectlytoarunningMySQL
server.Thearchitecturemakes itveryeasy todevelopanddeploynewstorageengines
toMySQL8.
When developing new storage engine, it is required to take care of all thecomponents
that work for and with storage engines. These include installation handlers,
operations
ontablesuchascreating,opening,andclosing,DML,indexing,andsoon.
Inthissection,wewillcoverhowyoucanstartdevelopinganewstorageengineon
ahigh-levelbasiswithreferenceto theMySQLdocumentationprovidedinthedevelopment
community. The creation of a custom storage engine requires a workingknowledgeofdevelopmentwithCandCPP,andcompilationwithcmakeandVisualStudio.
Creating storage engine sourcefilesTheeasiestwaytoimplementanewstorageengineistobeginbycopyingandmodifying
theEXAMPLEstorageengine.Thefilesha_example.ccandha_example.hcanbefoundinthestorage/exampledirectoryoftheMySQLsourcedistribution.
Whencopying the files,change thenames fromha_example.ccandha_example.h tosomethingappropriatetoyourstorageengine,suchasha_foo.ccandha_foo.h.
Afteryouhavecopiedandrenamedthefiles,youmustreplaceall instancesofEXAMPLEandexamplewiththenameofyourstorageengine.
Adding engine-specific variablesandparametersAplugincanimplementstatusandsystemvariablesandinthissectionwehavecovered the changes to variables and parameters with appropriate values anddatatypes.
Theserverplugininterfaceenablespluginstoexposestatusandsystemvariablesusingthestatus_varsandsystem_varsmembersofthegeneralplugindescriptor.
status_varsisamemberofthegeneralplugindescriptor.Ifthevalueisnot0,thenitpointstoanarrayofthest_mysql_show_varstructurewhereeachofthemdescribeone statusvariable followedby a structurewith all themembers set to0.Thedefinition for the st_mysql_show_var structure is as follows: structst_mysql_show_var{constchar*name;char*value;enumenum_mysql_show_typetype;};
Whenthepluginisinstalled,thepluginnameandthenamevaluearejoinedwithanunderscoretoformthenamedisplayedbytheSHOWSTATUSstatement.
Thefollowinglistshowsthepermissiblestatusvariabletypevaluesandwhatthecorrespondingvariableshouldbe:
SHOW_BOOL:ThisisapointertothebooleanvariableSHOW_INT:ThisisapointertotheintegervariableSHOW_LONG:ThisisapointertothelongintegervariableSHOW_LONGLONG:ThisisapointertothelonglongintegervariableSHOW_CHAR:ThisisaStringindexSHOW_CHAR_PTR:ThisisapointertoStringindexesSHOW_ARRAY:Thisisapointertoanotherst_mysql_show_vararraySHOW_FUNC:ThisisapointertoafunctionSHOW_DOUBLE:Thisisapointertoadouble
Allsessionandglobalsystemvariableshavetobepublishedtomysqldbeforetheyare used.This is precisely done by constructing a NULL terminated arrayof thevariablesandlinkingtoitinthepluginpublicinterface.
All mutable and plugin system variables are stored internally in the HASH
structure.
The display of the server command-line help text is generated by compilingDYNAMIC_ARRAY of all the relevant variables, sorting and iterating through them todisplayeachoption.
During the plugin installation process, the server processes command-lineoptions,immediatelyafterthepluginhasbeensuccessfullyloadedbuttheplugininitializationfunctionisyettobecalled.
Pluginsloadedatruntimedonotbenefitfromanyconfigurationoptionsandmusthave usable defaults. Once they are installed, they are loaded at mysqld
initialization timeandconfigurationoptionscanbesetat thecommand lineorwithinmy.cnf.
Thethdparametershouldbeconsideredasread-onlyinplugins.
<strong>handlertonexample_hton={</strong><br/><strong>"EXAMPLE",/*Nameofthestorageengine*/</strong><br/><strong>SHOW_OPTION_YES,/*Itshouldbedisplayedinoptionsornot*/</strong><br/><strong>"Examplestorageengine",/*Descriptionofthestorageengine*/</strong><br/><strong>DB_TYPE_EXAMPLE_DB,/*Typeofstorageengineitshouldreferto*/</strong><br/><strong>NULL,/*Initializehandlerton*/</strong><br/><strong>0,/*slotavailable*/</strong><br/><strong>0,/*definesavepointsize.*/</strong><br/><strong>NULL,/*handleclose_connection*/</strong><br/><strong>NULL,/*handlesavepoint*/</strong><br/><strong>NULL,/*handlerollbacktosavepoint*/</strong><br/><strong>NULL,/*handlereleasesavepoint*/</strong><br/><strong>NULL,/*handlecommit*/</strong><br/><strong>NULL,/*handlerollback*/</strong><br/><strong>NULL,/*handleprepare*/</strong><br/><strong>NULL,/*handlerecover*/</strong><br/><strong>NULL,/*handlecommit_by_xid*/</strong><br/><strong>NULL,/*handlerollback_by_xid*/</strong><br/><strong>NULL,/*handlecreate_cursor_read_view*/</strong><br/><strong>NULL,/*handleset_cursor_read_view*/</strong><br/><strong>NULL,/*handleclose_cursor_read_view*/</strong><br/><strong>example_create_handler,/*Createanewhandlerinstance*/</strong><br/><strong>NULL,/*handledropdatabase*/</strong><br/><strong>NULL,/*handlepaniccall*/</strong><br/><strong>NULL,/*handlereleasetemporarylatches*/</strong><br/><strong>NULL,/*UpdaterelevantStatistics*/</strong><br/><strong>NULL,/*StartConsistentSnapshotforreference*/</strong><br/><strong>NULL,/*handleflushlogs*/</strong><br/><strong>NULL,/*handleshowstatus*/</strong><br/><strong>NULL,/*handlereplicationReportSenttoBinlog*/</strong><br/><strong>HTON_CAN_RECREATE</strong><br/><strong>};</strong>
<strong>statichandler*tina_create_handler(TABLE*table);</strong>
<strong>statichandler*myisam_create_handler(TABLE*table)</strong><br/><strong>{</strong><br/><strong>returnnewha_myisam(table);</strong><br/><strong>}</strong>
<strong>constchar**ha_tina::bas_ext()const</strong><br/><strong>{</strong><br/><strong>returnha_tina_exts;</strong><br/><strong>}</strong>
Byprovidingextensioninformation,youcanalsoskipimplementingDROPTABLEfunctionality,astheMySQLserverwillimplementthesamebyclosingthetableanddeletingallfileswiththeextensionsspecified.
<strong>virtualintcreate(constchar*name,TABLE*form,HA_CREATE_INFO*info)=0;</strong>
<strong>typedefstructst_ha_create_information</strong><br/><strong>{</strong><br/><strong>CHARSET_INFO*table_charset,*default_table_charset;/*charsetintable*/</strong><br/><strong>LEX_STRINGconnect_string;/*connectionstring*/</strong><br/><strong>constchar*comment,*password;/*storingcommentsandpasswordvalues*/</strong><br/><strong>constchar*data_file_name,*index_file_name;/*dataandindexfilenames*/</strong><br/><strong>constchar*alias;/*valuepointerforalias*/</strong><br/><strong>ulonglongmax_rows,min_rows;</strong><br/><strong>ulonglongauto_increment_value;</strong><br/><strong>ulongtable_options;</strong><br/><strong>ulongavg_row_length;</strong><br/><strong>ulongraid_chunksize;</strong><br/><strong>ulongused_fields;</strong><br/><strong>SQL_LISTmerge_list;</strong><br/><strong>enumdb_typedb_type;/*valuefordb_type*/</strong><br/><strong>enumrow_typerow_type;/*valueforrow_type*/</strong><br/><strong>uintnull_bits;/*NULLbitsspecifiedatstartofrecord*/</strong><br/><strong>uintoptions;/*ORofHA_CREATE_optionsspecification*/</strong><br/><strong>uintraid_type,raid_chunks;/*raidtypeandchunksinfo*/</strong><br/><strong>uintmerge_insert_method;</strong><br/><strong>uintextra_size;/*lengthofextradatasegments*/</strong><br/><strong>booltable_existed;/*1increateiftableexisted*/</strong><br/><strong>boolfrm_only;/*1ifnoha_create_table()*/</strong><br/><strong>boolvarchar;/*1iftablehasaVARCHAR*/</strong><br/><strong>}HA_CREATE_INFO;</strong>
Storageenginescanignorethecontentsof*infoand*formbecausethecreationandtheinitializationofthedatafilesisonlyreally
<strong>intopen(constchar*name,intmode,inttest_if_locked);</strong>
<strong>#defineHA_OPEN_ABORT_IF_LOCKED0/*default*/</strong><br/><strong>#defineHA_OPEN_WAIT_IF_LOCKED1/*waitiftableislocked*/</strong><br/><strong>#defineHA_OPEN_IGNORE_IF_LOCKED2/*ignoreiflocked*/</strong><br/><strong>#defineHA_OPEN_TMP_TABLE4/*Tableisatemptable*/</strong><br/><strong>#defineHA_OPEN_DELAY_KEY_WRITE8/*Don'tupdateindex*/</strong><br/><strong>#defineHA_OPEN_ABORT_IF_CRASHED16</strong><br/><strong>#defineHA_OPEN_FOR_REPAIR32/*openevenifcrashedwithrepair*/</strong>
Thetypicalstorageenginewillimplementsomeformofsharedaccesscontrolinordertopreventfilecorruptioninamulti-threadedenvironment.Forexample,seetheget_share()andfree_share()methodsofsql/example/ha_tina.ccforimplementingfilelocking.
<strong>ha_tina::store_lock</strong><br/><strong>ha_tina::external_lock</strong><br/><strong>ha_tina::info</strong><br/><strong>ha_tina::rnd_init</strong><br/><strong>ha_tina::extra-ENUMHA_EXTRA_CACHECacherecordinHA_rrnd()</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::rnd_next</strong><br/><strong>ha_tina::extra-ENUMHA_EXTRA_NO_CACHEEndcachingofrecords(def)</strong><br/><strong>ha_tina::external_lock</strong><br/><strong>ha_tina::extra-ENUMHA_EXTRA_RESETResetdatabasetoafteropen</strong>
Thefollowingmethodscanbeimplementedtotakecareofspecificoperations:
Implementingthestore_lock():Thismethodcanmodifythelocklevel,ignoringoraddinglocksformanytablesImplementingtheexternal_lock():ThismethodiscalledwhentheLOCKTABLESstatementisissuedImplementingthernd_init():ThismethodisusedintablescanningforresettingcountersandpointersatthestartofatableImplementingtheinfo(uinfflag):ThismethodisusedtoprovideextratableinformationtotheoptimizerImplementingtheextra():Thismethodisusedtoprovideextrahintsinformationtothestorageengine
Implementingthernd_next():ThismethodiscalledoneachrowofscanninguntilEOFisreachedorthesearchconditionissatisfied
<strong>intha_tina::close(void)</strong><br/><strong>{</strong><br/><strong>DBUG_ENTER("ha_tina::close");</strong><br/><strong>DBUG_RETURN(free_share(share));</strong><br/><strong>}</strong>
Storageenginesusetheirownsharemanagementsystems.Theyshouldusetherequiredmethodsinordertoremovethehandlerinstancefromthesharefortherespectivetableopenedintheirhandler.
Ifyourstorageengineiscompiledasasharedobject,duringloadingifyougetanerrorsuchasundefinedsymbol:_ZTI7handler,thenmakesureyoucompileandlinkyourextensionusingthesameflagsastheserveruses.TheusualreasonforthiserroristhatLDFLAGSaremissingthe-fno-rttioption.
Reference for advanced customstorageengineWe have gone through the preceding sections in detail, giving high-levelinformation
for custom storage engine components and the required changes. Forimplementing INSERT, UPDATE, DELETE, indexing, and so on, in a custom storageengine, requires a working knowledge of development with C/CPP andcompilation with cmake and Visual Studio. For advanced development for thecustomstorageengines,pleaserefertothedetailed
informationgivenathttps://dev.mysql.com/doc/internals/en/custom-engine.html
Summary
Bynow,youhavelearnedthedifferentdatabaseenginesavailableinMySQL8andwelearnedwhyweshouldcareaboutstorageenginesandavailablestorageengineoptions inMySQL8.Wecovered indetail theInnoDB storageengineandrelatedimportantfeaturesalreadyprovidedwithintheInnoDBstorageengine.Nowyou are practically able to create a custom storage engine as per the systemrequirement andmake it pluggable inMySQL 8.An important aspectwas tochooseasuitablestorageengineforyoursystem,whichiscovereddetail.
Inthenextchapter,youwilllearnabouthowindexingworksinMySQL8,thenewfeaturesintroducedrelatedtoindexing,thedifferenttypesofindexing,andhowtouseindexingonyourtables.Alongwiththat,acomparisonwillalsobeprovided along with in-depth knowledge of various ways of indeximplementation.
IndexinginMySQL8
In the previous chapter,we learned about storage engines.Nowwe are awarewhat types of storage engines are available and which ones to use for ourrequirements. The previous chapter also covered the InnoDB storage engine indetail, along with other storage engine information. It also described how todefineacustomstorageengineforuse,withapracticalexample.Nowit'stimeto understand one more important functionality of MySQL 8 and that is,indexing.
Wewill cover different types of indexeswith their functionalities,whichwillencourage you to use indexes and provided youwith guidance on how to usethem.So,yourjourneyintoindexeshasstarted!Let'sgo.
Wewillcoverthefollowingtopicsinthischapter:
Anoverviewonindexing
Column-levelindexing
B-Treeindexes
Anoverviewonindexing
TodefineanindexonatableisthebestwaytoimprovetheperformanceoftheSELECT operation. An index acts like a pointer for the table rows and permitsqueriesto
quicklypoint tomatching rowsbasedon theWHERE condition.MySQL8allowsyoutocreateindexesonallthedatatypes.Althoughindexing
providesgoodperformanceonqueries,itisrecommendtodefineitintheproperway,
becauseunnecessaryindexeswastespaceandtime(forMySQL8tofindwhichindex
is best to use). In addition to that, indexes also add costs to INSERT, UPDATE, andDELETE operations, because during these operations,MySQL8will update eachindex.
Aswedescribedpreviously,anindexisadatastructurethatimprovesthespeedof
operations.Basedonthestructure,anindexisbifurcatedintotwomajorforms—
aclustered
indexandanon-clusteredindex:
Clustered index: A clustered index defines the order in which data isphysicallystoredinatable.
Therefore,onlyoneclusteredindexisallowedpertable.Itgreatlyincreasesthe
speedofretrievalwhendataisretrievedinasequentialmanner,eitherinthesame
order or in reverse order. A clustered index also provides betterperformancewhen
arangeofitemsareselected.Aprimarykeyisdefinedasaclusteredindex.
Non-clustered index: A non-clustered index doesn't define the order inwhichdataisphysicallystored.
Thismeansanon-clusteredindexisstoredinoneplace,anddataisstoredinanother
place.Therefore,morethanonenon-clusteredindexisallowedpertable.Itrefers
tonon-primarykeys.
Asweknow,theprimarykeyrepresentsthecolumn,orsetofcolumns,whichismost
widelyusedforfetchingrecordsfromthe table.Theprimarykeyhasan indexassociated
withitandisusedforfastqueryperformance.Itprovidescomparativelyfasterperformance
becauseaprimarykeydoesnotallowaNULLvalue,sonocheckisrequiredonNULLvalues. It is recommended that if your table does not have a columnor set ofcolumns
to define as a primary key, then you define one auto increment field as theprimary
key for better performance. On the other hand, if your table contains manycolumns
andthereisaneedtoexecuteaquerywithacombinationofmultiplecolumns,then
it is advisable to the less frequently used data and transfer in onto a separatetable.
Relatealltheseparatetableswithprimaryandforeignkeyreferences,whichwill
helpyouinmanagingdata,andqueryretrievalprovidesyougoodperformance.
SQL commands related toindexesMySQL 8 provides two main commands related to indexes. We will discussthesecommandsinthefollowingsections.
<strong>CREATE[UNIQUE|FULLTEXT|SPATIAL]INDEXindex_name</strong><br/><strong>[index_type]</strong><br/><strong>ONtbl_name(index_col_name,...)</strong><br/><strong>[index_option]</strong><br/><strong>[algorithm_option|lock_option]...</strong><br/><strong>index_col_name:</strong><br/><strong>col_name[(length)][ASC|DESC]</strong><br/><strong>index_option:</strong><br/><strong>KEY_BLOCK_SIZE[=]value</strong><br/><strong>|index_type</strong><br/><strong>|WITHPARSERparser_name</strong><br/><strong>|COMMENT'string'</strong><br/><strong>|{VISIBLE|INVISIBLE}</strong><br/><strong>index_type:</strong><br/><strong>USING{BTREE|HASH}</strong><br/><strong>algorithm_option:</strong><br/><strong>ALGORITHM[=]{DEFAULT|INPLACE|COPY}</strong><br/><strong>lock_option:</strong><br/><strong>LOCK[=]{DEFAULT|NONE|SHARED|EXCLUSIVE}</strong>
Usingcol_name(length)syntax,theuserisabletospecifyanindexprefixlength,whichwillconsideronlyaspecifiednumberofcharactersfromthestringvalue.Atthetimeofdefining,theprefixconsidersthefollowingpoints:
TheprefixisoptionalforCHAR,VARCHAR,BINARY,andVARBINARYcolumnindexesTheprefixmustbespecifiedinthecaseofBLOBandTEXTcolumnindexesMySQL8willconsiderprefixesasanumberofcharactersfornon-binarystringtypes(CHAR,VARCHAR,TEXT)andanumberofbytesforbinarytypes(BINARY,VARBINARY,BLOB)Theprefixisnotallowedforthespatialcolumns
Adetailedexampleoftheprefixoptionisdescribedlaterinthischapter,undertheColumnindexessection.AUNIQUEindexisa
constraintwhichindicatesthatallthevaluesintheindexwillbeunique.Ifyoutrytoaddvalueswhichalreadyexist,thenMySQL8displaysanerror.AlltypesofstorageengineswithaUNIQUEindexpermitmultiplenullvalues.InthecaseofprefixeswhenyouuseNULLvalues,makesurecolumnvaluesareuniquewithintheprefixes.Ifanindexprefixexceedsitssize,thenMySQL8willhandletheindexasfollows:
Foranon-uniqueindex:IfthestrictSQLmodeisenabled,thenMySQL8willthrowanerror,andifthestrictmodeisdisabled,thentheindexlengthisreducedtothemaximumcolumndatatypesizeandwillproduceawarning.Forauniqueindex:Inthiscase,MySQL8producesanerrorregardlessoftheSQLmode,becauseitmightbreaktheuniquenessofthecolumn.Thismeansyouhavedefinedacolumnwith25lengthandtriedtodefineanindexonthesamecolumnwithaprefixlength27;thenMySQL8throwsanerror.
Spatialindexcharacteristics
MySQL8followsthefollowingrulesforspatialindexcharacteristics:
ItisonlyavailableforInnoDBandMyISAMstorageengines;ifyoutrytouseitforotherstorageengines,thenMySQL8givesanerror.
ANULLvalueisnotallowedforindexedcolumns.
A prefix attribute is not allowed for this column. Full width will beconsideredfortheindex.
<strong>createtableemployee(idint(11)notnull,namevarchar(50));</strong><br/><strong>CREATEINDEXemp_name_indexONemployee(name)USINGBTREE;</strong>
<strong>CREATEINDEXname_indexONemployee(name)COMMENT<br/>'MERGE_THRESHOLD=40';</strong>
Ifthepage-fullpercentageforanindexislessthantheMERGE_THRESHOLDvalue,thentheInnoDBstorageenginewillmergetheindexpagewithaneighboringindexpage.
VISIBLE,INVISIBLE:Thisparameterdefinestheindexvisibility.Bydefault,allindexesarevisible.Theoptimizerwillnotuseaninvisibleindexduringtheoptimizationprocess.
TheALGORITHMandLOCKattributeswillhaveanimpactwhenyoutrytousethetableforreadingorwriting,andwhilesimultaneouslymodifyingitsindexing.
<strong>DROPINDEXindex_nameONtbl_name</strong><br/><strong>[algorithm_option|lock_option]...</strong><br/><strong>algorithm_option:</strong><br/><strong>ALGORITHM[=]{DEFAULT|INPLACE|COPY}</strong><br/><strong>lock_option:</strong><br/><strong>LOCK[=]{DEFAULT|NONE|SHARED|EXCLUSIVE}</strong>
<strong><spanclass="tokenkeyword">DROP</span><spanclass="tokenkeyword">INDEX</span>name_index<spanclass="tokenkeyword">ONemployee;</span></strong>
<strong>CREATETABLEgeom_data(dataGEOMETRYNOTNULL,SPATIALINDEX(data));</strong>
<strong>SELECTDISTINCTST_SRID(column_name)FROMtable_name;</strong>
Ifthequeryreturnsmultiplerows,thenitindicatesthatthecolumncontainsmixedSRIDs.Ifso,changethecontentsofthecolumnforthesameSRIDvalue.DefineanexplicitSRIDforthecolumn.RecreatetheSPATIALINDEX.
InnoDB and MyISAM indexstatisticscollection
MySQL 8 will consider table statistics based on the value group, which isnothing but a set of rows with the same prefix values. The storage enginecollectsstatisticsrelatedtothetable,whichareusedbytheoptimizer.Fromtheoptimizationperspective,averagevaluegroupsizeisanimportantstatistic.Iftheaveragevalueofthegroupsizeincreases,thentheindexisnotmeaningful.
So, it's better to target a small number of rows for each index. This can beachievedbytablecardinality,whichisnothingbutanumberofvaluegroup.ForInnoDB and MyISAM tables, MySQL 8 provides control on statistics by themyisam_stats_methodand theinnodb_stats_method systemvariables.The followingarethepossiblevaluesforthesevariables:
nulls_ignored:ItindicatesthatNULLvaluesareignored
nulls_equal:ItindicatesallNULLvaluesareidentical
nulls_unequal:ItindicatesallNULLvaluesarenotidentical
The innodb_stats_method system variable has a global value, while themyisam_stats_methodsystemvariablehasbothglobalandsessionvalues.Whenwesetaglobalvalueofavariable,itwillaffectstatisticscollectionfortablesfromthe corresponding storage engine. In the case of session value statistics,collectionisavailableonlyforthecurrentclientconnection.Thismeansthatyouhave to regenerate a table's statistics for the other client on the same tablewithoutaffectingotherclients,andneedtosetitinasessionvalue.ToregenerateMyISAMstatistics,useeitherofthesemethods:
Executethemyisamchk--stats_method=method_name--analyzecommand
Make changes to the table tomake its statistics out of date, and then setmyisam_stats_methodandissueanANALYZETABLEstatement
Therearesomepointsthatmustbeconsideredbeforeusingthesetwovariables:
These variables are available only for InnoDB and MyISAM tables. For otherstorageengines,onlyonemethodisavailabletocollecttablestatistics,anditisveryneartothenulls_equalmethod.
MySQL8providesawaytogeneratestatisticsforatableexplicitly,butthisisnot always thecase.Sometimes,MySQL8mayalsogenerate statisticsautomatically,ifitisrequired.Forexample,inthecaseofanyoperations,ifsomeoftheSQL
statementsmodified table data, thenMySQL 8will automatically collectstatistics.
Considerbulkinsertordeleteoperations.
Wecannotsaywhichmethodwasusedtogeneratestatisticsforatable.
Column-levelindexingMySQL 8 allows you to create an index on a single column, as well as onmultiplecolumns.
Themaximumnumberofindexespertableandmaximumindexlengthdependonthestorageengine.Mostly,all thestorageenginesallowat least16indexesper table and total index lengthsof at least256bytes,butmostof the storageenginespermithigherlimits.
ColumnindexesThisisthemostcommonwaytodefineanindexwhereonlyasinglecolumnisinvolved.
MySQL8storesacopyofcolumnvaluesinadatastructuresothatrowscanbeaccessed
quickly.MySQL8usesaB-Treedatastructuretoenablevaluestobeaccessedquickly.TheB-Treeexecutionwill
workbasedonoperators,suchas=,<,>,BETWEEN,IN,andmanymore,whichweredefined in thewhere condition.Youcangetdetailson theB-Treedata structureanditsexecutionin
thenexttopic.Wewilldiscussthecharactersofcolumnindexesinthecomingsections.
<strong>CREATETABLEperson(personal_dataTEXT,INDEX(personal_data(8)));</strong>
Thiscommandcreatesanindexonthepersonal_datacolumnbyconsideringthefirsteightcharacters.Theprefixlengthvaries,basedonthestorageengine.TheInnoDBstorageengineallowsuptoa767byteprefixlengthforREDUNDANTorCOMPACTrowformats,whileinthecaseofDYNAMICorCOMPRESSEDrowformats,itallowsupto3072bytes.InthecaseoftheMyISAMstorageengine,theprefixcanbedefinedforupto1,000bytes.
Prefixlengthwillbemeasuredinbytesforbinarystringtypes,suchasBINARY,VARBINARY,andBLOB,whileinthecaseofnon-binarystringtypes,itwillbeconsideredasanumberofcharacters.
FULLTEXTindexes
Asthenameimplies,FULLTEXT indexesallowCHAR,VARCHAR,andTEXTcolumnsonly.This index is supported by InnoDB and MyISAM storage engines. In this type,indexingwilltakeplaceonanentirecolumn,ratherthanaprefixlength.MySQL8evaluates the full text expressionduring theoptimizationphaseof thequeryexecution. Before making estimations, optimization evaluates the full textexpressionintheprocessofdevelopinganexecutionplan.Asaresult,theEXPLAINqueryoffulltextisslowerthannon-fulltextqueries.Fulltextqueriesareusefulinthefollowingscenarios:
WhenaFULLTEXTqueryreturnseitheradocumentIDoradocumentIDandsearchrank
When a FULLTEXT query sorts the matching rows by descending order andusesaLIMITclausetofetchNnumberofrows,applyonlyasingleORDERBYclauseindescendingorder,anddon'tuseaWHEREclauseinitforoptimization
When a FULLTEXT query fetches a COUNT(*) value from the rowswithout anyadditional WHERE clauses, apply the WHERE clause as WHERE MATCH(text) AGAINST('other_text'),withoutany>0comparisonoperator
SpatialIndexesMySQL 8 allows you to create indexes on spatial data types. The InnoDB andMyISAM storage engines support R-Tree for spatial data, while other storageengines useB-Tree. SinceMySQL 5.7, spatial indexes are supported in MyISAMandInnoDBdatabaseengines.
Indexes in the MEMORY storageengineMemorystorageenginessupportbothHASHindexesandB-Treeindexes,butHASHindexesare,bydefault,setfortheMEMORYstorageengine.
<strong>CREATETABLEEmployee(</strong><br/><strong>idINTNOTNULL,</strong><br/><strong>lastnamevarchar(50)notnull,</strong><br/><strong>firstnamevarchar(50)notnull,</strong><br/><strong>PRIMARYKEY(id),</strong><br/><strong>INDEXname(lastname,firstname)</strong><br/><strong>);</strong>
<strong>SELECT*FROMEmployeeWHERElastname='Shah';</strong><br/><strong>SELECT*FROMEmployeeWHERElastname='Shah'ANDfirstname='Mona';</strong><br/><strong>SELECT*FROMEmployeeWHERElastname='Shah'AND(firstname='Michael'ORfirstname='Mona');</strong><br/><strong>SELECT*FROMEmployeeWHERElastname='Shah'ANDfirstname>='M'ANDfirstname<'N';</strong>
<strong>SELECT*FROMEmployeeWHEREfirstname='Mona';</strong><br/><strong>SELECT*FROMEmployeeWHERElastname='Shah'ORfirstname='Mona';</strong>
Remember,inthecaseofamultiple-columnindex,anyleft-mostprefixoftheindexcanbeusedbytheoptimizerforsearchingrows.Forexample,iftheindexwasdefinedonthreecolumnsinsequence,column1,column2,andcolumn3,thenyoucanuseindexedcapabilitieson(column1,column2,column3),(column1),(column1,column2),bydefiningitintheWHEREclause.
<strong>CREATETABLEEmployee(</strong><br/><strong>lastnamevarchar(50)notnull,</strong><br/><strong>firstnamevarchar(50)notnull,</strong><br/><strong>dobdatenotnull,</strong><br/><strong>genderchar(1)notnull,</strong><br/><strong>key(lastname,firstname,dob)</strong><br/><strong>);</strong>
Asperthetabledefinitionindexwillcontainsvaluesincombinationofthreecolumnsfirstname,lastname,anddob.Itwillsortvaluesaspertheordergivenpreviously;thismeansthatifsomeemployeeshavesimilarnames,thentheywillbesortedbytheirbirthdates.ConsiderthefollowingtypesofquerieswhichwillbenefitfromtheB-Treeindex:
Matchtherangeofvalues:FindstheemployeeswhoselastnamesarePatelandGupta.Matchthefullvalues:FindsemployeewhosenameisMohanPatelandwasbornon28/11/1981.Matchleftmostprefix:Findsallemployeewithalastname.Theseuseonlythefirstcolumnintheindex.Matchwithcolumnprefix:FindstheemployeeswhoselastnamesbeginwithM.Itusesonlythefirstcolumnintheindex.Matchonepartexactlyandwithrangeofanothercolumn:FindstheemployeewhoselastnameisPatelandwhosefirstnamestartswithA.
ThefollowingarequerieswhereB-Treeisnotuseful:
Don'tuseanyconditionafterarangecondition:Forexample,youhaveputtheWHEREconditionlastname='Patel'andfirstnamelike‘A%'anddob='28/11/1981'.Here,onlythefirsttwocolumnsareconsideredfortheindex,becauseLIKEisarangecondition.
Don'tskipanyofthecolumnsdefinedintheindex:ThismeansyouarenotallowedtouselastnameanddobtofindanemployeewherefirstnameismissingintheWHEREcondition.Lookupdoesn'tstartwithleft-mostsideofindexedcolumns:Forexample,theindexwillnotworkifyoufindtheemployeewhosefirstnameisMohanandwhosedobisonacertaindate.Inthisquery,definedcolumnsarenotleft-mostintheindex.Inthesameway,theindexdoesnotworkinthecasewhereyoufindtheemployeewhoselastnameendswithsomething.
Hashindex
It is very difficult to find a single value from a large database by followingcompletetreetraversalswithmultiplelevels.Toovercomethisproblem,MySQLhasprovidedanother index type,which isknownasahash index.This indexcreatesahashtableratherthanatree,whichisveryflatinstructurecomparedtoaB-Treeindex.Hashingmainlyuseshashfunctionstogeneratetheaddressesofdata.Twoimportanttermsrelatedtohashingare:
Hashfunction:Themappingfunctionwhichwillbeusefultomapsearch-keyswiththeaddresswhereactualrecordsarestored.
Bucket:Abucketisaunitofstoragewhereahashindexstoresthedata.Abucket indicates a complete disk block, which will store one or morerecords.
Along with the hashing mechanism, a hash index has some specialcharacteristics,describedasfollows:
Thewholekeyisusedtosearchtherow.WhileinthecaseofaB-Tree,onlytheleft-mostprefixofthekeyisusedtofindrows.
TheoptimizerwillnotuseahashindextospeedupORDERBYoperations.Inotherwords,thisindexisneverusedtofindthenextentry.
Hashindexesareusedforequalitycomparisonbyusing=or<=>operators.Itwillneverusecomparisonoperatorswhichwillreturnarangeofvalues.
Forexample,the<(lessthan)operator.
The range optimizer cannot actually gauge howmany rows are availablebetweenthetwovalues.And,ifweuseahash-indexedMEMORYtableinsteadofInnoDBorMyISAM,thenitmayaffectthequeriesaswell.
<strong>SEToptimizer_switch='use_index_extensions=off';</strong>
<strong>CREATETABLEtable1(</strong><br/><strong>c1INTNOTNULLDEFAULT0,</strong><br/><strong>c2INTNOTNULLDEFAULT0,</strong><br/><strong>d1DATEDEFAULTNULL,</strong><br/><strong>PRIMARYKEY(c1,c2),</strong><br/><strong>INDEXkey1(d1)</strong><br/><strong>)ENGINE=InnoDB;</strong><br/><br/><strong>--Insertvaluesintotable</strong><br/><strong>INSERTINTOtable1VALUES</strong><br/><strong>(1,1,'1990-01-01'),(1,2,'1991-01-01'),</strong><br/><strong>(1,3,'1992-01-01'),(1,4,'1993-01-01'),</strong><br/><strong>(1,5,'1994-01-01'),(2,1,'1990-01-01'),</strong><br/><strong>(2,2,'1991-01-01'),(2,3,'1992-01-01'),</strong><br/><strong>(2,4,'1993-01-01'),(2,5,'1994-01-01'),</strong><br/><strong>(3,1,'1990-01-01'),(3,2,'1991-01-01'),</strong><br/><strong>(3,3,'1992-01-01'),(3,4,'1993-01-01'),</strong><br/><strong>(3,5,'1994-01-01'),(4,1,'1990-01-01'),</strong><br/><strong>(4,2,'1991-01-01'),(4,3,'1992-01-01'),</strong><br/><strong>(4,4,'1993-01-01'),(4,5,'1994-01-01'),</strong><br/><strong>(5,1,'1990-01-01'),(5,2,'1991-01-01'),</strong><br/><strong>(5,3,'1992-01-01'),(5,4,'1993-01-01'),</strong><br/><strong>(5,5,'1994-01-01');</strong>
<strong>--Indexextensionissetasoff</strong><br/><strong>SEToptimizer_switch='use_index_extensions=off';</strong><br/><br/><strong>--Executeselectquerywithexplain</strong><br/><strong>EXPLAINSELECTCOUNT(*)FROMtable1WHEREc1=3ANDd1='1992-01-01';</strong><br/><br/><strong>--Outputofexplainquery</strong><br/><strong>***************************1.row***************************</strong><br/><strong>id:
1</strong><br/><strong>select_type:SIMPLE</strong><br/><strong>table:table1</strong><br/><strong>type:ref</strong><br/><strong>possible_keys:PRIMARY,key1</strong><br/><strong>key:PRIMARY</strong><br/><strong>key_len:4</strong><br/><strong>ref:const</strong><br/><strong>rows:5</strong><br/><strong>Extra:Usingwhere</strong>
<strong>--Indexextensionissetason</strong><br/><strong>SEToptimizer_switch='use_index_extensions=on';</strong><br/><br/><strong>--Executeselectquerywithexplain</strong><br/><strong>EXPLAINSELECTCOUNT(*)FROMtable1WHEREc1=3ANDd1='1992-01-01';</strong><br/><br/><strong>--Outputofexplainquery</strong><br/><strong>***************************1.row***************************</strong><br/><strong>id:1</strong><br/><strong>select_type:SIMPLE</strong><br/><strong>table:table1</strong><br/><strong>type:ref</strong><br/><strong>possible_keys:PRIMARY,key1</strong><br/><strong>key:key1</strong><br/><strong>key_len:8</strong><br/><strong>ref:const,const</strong><br/><strong>rows:1</strong><br/><strong>Extra:Usingindex</strong>
Now,wewillcheckthedifferencebetweenthesetwoapproaches:
Thekey_lenvaluechangesfrom4bytesto8bytes,whichindicatesthatkeylookupsuseboththecolumnsd1andc1,notonlyd1.Therefvaluechangesfrom(const)to(const,const)whichindicateskeylookupusestwokeypartsinsteadofone.Therowscountchangesfrom5to1,whichindicatesthatInnoDBrequiresfewerrowsthanthefirstapproachtoproducetheresult.TheExtravaluechangesfromUsingwheretoUsingindex.Itindicatesthattherowscanbereadbyusingonlytheindex,
<strong>CREATETABLEt1(c1INT,c2INTAS(c1+1)STORED,INDEX(c2));</strong>
<strong>SELECT*FROMt1WHEREc1+1>100;</strong>
<strong>mysql>explainSELECT*FROMt1WHEREc1+1>100;</strong><br/><strong>***************************1.row***************************</strong><br/><strong>id:1</strong><br/><strong>select_type:SIMPLE</strong><br/><strong>table:t1</strong><br/><strong>partitions:NULL</strong><br/><strong>type:range</strong><br/><strong>possible_keys:c2</strong><br/><strong>key:c2</strong><br/><strong>key_len:5</strong><br/><strong>ref:NULL</strong><br/><strong>rows:1</strong><br/><strong>filtered:100.00</strong><br/><strong>Extra:Usingindexcondition</strong>
<strong>nameTEXTAS(JSON_EXTRACT(emp,'$.name'))STORED</strong>
<strong>nameTEXTAS(JSON_UNQUOTE(JSON_EXTRACT(emp,'$.name')))STORED</strong>
<strong>c2INTAS(c1)STOREDincolumndefinition.</strong>
Useaindexhintiftheoptimizertriestousethewrongindex,whichwilldisableditandforcetheoptimizertouseadifferentchoice
Invisibleanddescendingindexes
Theinvisibleindexisaspecialfeaturewhichwillmarkanindexasunavailablefortheoptimizer.MySQL
8 will maintain invisible indexes and keep them up-to-date when data ismodified.
This will apply on indexes other than primary key. As we know, indexes arevisible by default; we have to make them invisible explicitly at the time ofcreation, or by using the alter command. MySQL 8 provides the VISIBLE andINVISIBLEkeywordstomaintainindexvisibility.Adescendingindexisthemethodofstoringkeyvaluesindescendingorder.Adescendingindexismoreefficient,asitcanbescannedintheforwardorder.Let'sseetheseindexesindetail,withexamples.
InvisibleindexAsmention previously, an invisible index is not used by the optimizer. Thenwhatistheuseofthisindex?Thisquestioncomesintoourmind,right?Wewillexplaintoyousomeoftheusecasesforinvisibleindexes:
Whenmanyindexesaredefined,butyouarenotsurewhichindexisnotinuse. In this case, you can make an index invisible and check theperformance impact. If it has an impact, then you can make that indexvisibleonanimmediatebasis.Special cases where only one query is using an index. In this case, aninvisibleindexisagoodsolution.
In the following example,wewill create an invisible index using CREATE TABLE,CREATEINDEX,orALTERTABLEcommands:
CREATETABLE`employee`(
`id`int(11)NOTNULLAUTO_INCREMENT,
`department_id`int(11),
`salary`int(11),
PRIMARYKEY(`id`)
)ENGINE=InnoDB;
CREATEINDEXidx1ONemployee(department_id)INVISIBLE;
ALTERTABLEemployeeADDINDEXidx2(salary)INVISIBLE;
Tochangethevisibilityoftheindexes,usethefollowingcommands:
ALTERTABLEemployeeALTERINDEXidx1VISIBLE;
ALTERTABLEemployeeALTERINDEXidx1INVISIBLE;
Togetinformationaboutanindex,executeINFORMATION_SCHEMA.STATISTICStableorSHOWINDEXcommandsinthefollowingways:
mysql>SELECT*FROMinformation_schema.statisticsWHEREis_visible='NO';
***************************1.row***************************
TABLE_CATALOG:def
TABLE_SCHEMA:db1
TABLE_NAME:employee
NON_UNIQUE:1
INDEX_SCHEMA:db1
INDEX_NAME:idx1
SEQ_IN_INDEX:1
COLUMN_NAME:department_id
COLLATION:A
CARDINALITY:0
SUB_PART:NULL
PACKED:NULL
NULLABLE:YES
INDEX_TYPE:BTREE
COMMENT:
INDEX_COMMENT:
IS_VISIBLE:NO
mysql>SELECTINDEX_NAME,IS_VISIBLEFROMINFORMATION_SCHEMA.STATISTICS
->WHERETABLE_SCHEMA='db1'ANDTABLE_NAME='employee';
+------------+------------+
|INDEX_NAME|IS_VISIBLE|
+------------+------------+
|idx1|NO|
|idx2|NO|
|PRIMARY|YES|
+------------+------------+
mysql>SHOWINDEXESFROMemployee;
***************************1.row***************************
Table:employee
Non_unique:1
Key_name:idx1
Seq_in_index:1
Column_name:department_id
Collation:A
Cardinality:0
Sub_part:NULL
Packed:NULL
Null:YES
Index_type:BTREE
Comment:
Index_comment:
Visible:NO
MySQL 8 provides a use_invisible_indexes flag in the optimizer_switch systemvariable tocontrol invisible indexesusedbythequeryoptimizer.If thisflag ison, then the optimizer uses invisible indexes in execution plan construction,while if the flag is off, the optimizer ignores invisible indexes. MySQL 8providesa facility to use an implicit primary key if you have defined a UNIQUEindexonaNOTNULLcolumn.Onceyoudefine the indexonthisfield,MySQL8doesnotallowyoutomakeitinvisible.Inordertounderstandthisscenario,let'stake one example with the following table. Let's try to execute the followingcommandtomaketheidx1indexinvisible:
CREATETABLEtable2(
field1INTNOTNULL,
field2INTNOTNULL,
UNIQUEidx1(field1)
)ENGINE=InnoDB;
Theserverwillnowgiveanerror,asshowninthefollowingcommands:
mysql>ALTERTABLEtable2ALTERINDEXidx1INVISIBLE;
ERROR3522(HY000):Aprimarykeyindexcannotbeinvisible
Nowlet'saddtheprimarykeyintothetable,usingthefollowingcommand:
ALTERTABLEtable2ADDPRIMARYKEY(field2);
Now,wewill try tomaketheidex1 invisible.This time, theserverallowsit,asshowninthefollowingcommands:
mysql>ALTERTABLEtable2ALTERINDEXidx1INVISIBLE;
QueryOK,0rowsaffected(0.06sec)
Records:0Duplicates:0Warnings:0
<strong>CREATETABLEt1(</strong><br/><strong>aINT,bINT,</strong><br/><strong>INDEXidx1(aASC,bASC),</strong><br/><strong>INDEXidx2(aASC,bDESC),</strong><br/><strong>INDEXidx3(aDESC,bASC),</strong><br/><strong>INDEXidx4(aDESC,bDESC)</strong><br/><strong>);</strong>
<strong>ORDERBYaASC,bASC--optimizercanuseidx1</strong><br/><strong>ORDERBYaDESC,bDESC--optimizercanuseidx4</strong><br/><strong>ORDERBYaASC,bDESC--optimizercanuseidx2</strong><br/><strong>ORDERBYaDESC,bASC--optimizercanuseidx3</strong>
<strong>Query1:SELECT*FROMt1ORDERBYaDESC;</strong><br/><strong>Query2:SELECT*FROMt1ORDERBYaASC;</strong><br/><strong>Query3:SELECT*FROMt1ORDERBYaDESC,bASC;</strong><br/><strong>Query4:SELECT*FROMt1ORDERBYaASC,bDESC;</strong><br/><strong>Query5:SELECT*FROMt1ORDERBYaDESC,bDESC;</strong><br/><strong>Query6:SELECT*FROMt1ORDERBYaASC,bASC;</strong>
ThefollowingstatisticalgraphisprovidedbyMySQL8on10millionrows,withrespecttothepreviouslymentionedqueries:
Reference:https://mysqlserverteam.com/mysql-8-0-labs-descending-indexes-in-mysql/
Therearesomeimportantpointsthatyoushouldrememberatthetimeofusingdescendingindexes,whichareasfollows:
Allthedatatypessupportedbyascendingindexesarealsosupportedbydescendingindexes.DescendingindexesaresupportedforBTREEbutnotforHASH,FULLTEXT,andSPATIALindexes.IfyoutrytouseASCandDESCkeywordsexplicitlyforHASH,FULLTEXT,andSPATIALindexes,thenMySQL8willgenerateanerror.DescendingindexesonlysupportInnoDBstorageengines,buttheInnoDBSQLparserdoesnotusedescendingindexes.Changebufferingisnotsupportedforsecondaryindexesiftheprimarykeyincludesadescendingindex.DISTINCTcanuseanyindex,includingadescendingkey,butforMIN()/MAX(),nodescendingkeypartsareused.Bothnon-generatedandgeneratedcolumnsallowuseofdescendingindexes.
Summary
Everythingbecomesveryinterestingwhenyouareawareofhowitworks,right?Wehopeyouhavefoundthesamethingaboutindexinginthischapter.Wehavecovered very useful informationwhichwill help you to define indexes on theright columns to get better performance. In addition to that, we have alsodescribedvarioustypesofindexingwiththeirstoragestructures.
Inthenextchapter,wewillprovideyouwithinformationaboutreplication.Wewillexplaintheconfigurationandimplementationofreplicationindetail.
ReplicationinMySQL8
Inthepreviouschapter,wediveddeepintoMySQL8indexing.Indexesareanimportantentityforanydatabasemanagementsystem.TheyhelpimproveSQLquery performance by limiting the number of records to be visited. Databaseadministrators working on performance improvement must be aware of thisimportant technique. This chapter explains in detail the types of indexes andtheiradvantages.Thischapteralsoexplainshowindexingworks inMySQL8.It'sgoingtobeaprettyinformativechapter!
Moving further along the same line, in this chapter, wewill discuss databasereplication.
Howmucharewealreadyawareaboutdatabasereplication?Itdoesn'tactuallymatter.
This chapter covers insightful details about database replication. If you havepriorknowledgeofdatabasereplication,thischapterwilladdtoit.Ifyouhaveonly just heard about it for the first time, you will find every detail that isrequired to make it work in this chapter. So, are we ready to jump in? Thefollowingisalistoftopicsthatwewillbecoveringinthischapter:
Overviewofreplication
Configuringreplication
Implementingreplication
Groupreplicationversusclustering
Replicationsolutions
OverviewofreplicationWewillwalkthroughthebasicsofdatabasereplicationinthissection.Wewillunderstandwhat replication is, theadvantages itprovides,and thescenarios inwhichreplicationcanbebeneficial.
<strong><spanclass="tokenconstant">GTID</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>source_id:transaction_id</span></strong>
<strong>1A22AF74<spanclass="tokenoperator">-</span>17AC<spanclass="tokenoperator">-</span>E111<spanclass="tokenoperator">-</span>393E<spanclass="tokenoperator">-</span>80C49AB653A2:1</strong>
Thesequencenumberfortransactionscommittedstartswith1.Itcanneverbe0.
AGTID-basedMySQLreplicationmethodistransactionalandsothisiswhyitismorereliablethanabinarylogfile-basedreplicationmethod.GTIDguaranteesthereplicationaccuracyandconsistencybetweenmasterandslavedatabasesaslongasallthetransactionscommittedonmasterdatabaseservershavealsobeenappliedonalloftheslavedatabaseservers.
Asmentionedearlier,MySQLdatabasereplicationisusuallyasynchronous.However,MySQL8supportsdifferenttypesofsynchronizationforreplication.Theusualmethodofsynchronizationisasynchronous.Itmeansoneserveractsasamasterdatabaseserver.Itwritesalleventstoabinarylogfile.Otherdatabaseserversactasslaves.Slavedatabaseserversreadandcopyposition-basedeventrecordswithinbinarylogfilesfromthemasterdatabaseserver.So,itisalwaysfromamasterdatabaseservertoaslavedatabaseserver.MySQL8alsosupportssemisynchronoussynchronizationmethods.Insemisynchronousmethodsofreplication,anytransactioncommittedonamasterdatabaseserverisblockeduntilthemasterdatabaseserverreceivesacknowledgementfromatleastoneoftheslavedatabaseserversthatithasreceivedandloggedthetransactionevent.Delayedreplicationisanotherreplicationmethodsupportedby
MySQL8.Indelayedreplication,slavedatabaseserversintentionallylogthetransactioneventbehindmasterdatabaseserversbysomeamountoftime.
AdvantagesofMySQLreplication
Aswe arenow familiarwithwhatMySQLdatabase replication is, it's time toassessiftheaddedcomplexityofmaintainingmultipledatabaseserversisworthitornot.
TheadvantagesofMySQL8databasereplicationareasfollows:
1. Scale out solutions: As described earlier, usually web applications areread-heavy applications. The read operations aremuch higher in numberthan the write operations. The applications provide features that requireheavy,complexSQLqueries tobeexecutedonthedatabaseserver.Thesearenotthequeriesthattakemillisecondstoexecute.Suchcomplexqueriesmaytakeafewsecondstominutesforexecution.Executionofsuchqueriesputupheavy loadondatabaseserver. Insuchcases, it isalwaysbetter tohave such read operations performed on a separate database server thanmasterdatabaseservers.
Write database operations will always be performed on master databaseservers.Doyouknowwhy?It'sbecauseittriggersdatabasemodifications.Events of these modifications must be written to binary log files for
replicationsynchronizationbyslaveservers.
Also, the synchronization is from master to slaves. So, if we performedwritedatabaseoperationsonslaves,thosewillneverbeavailableonmasterdatabaseservers.Thisapproachimprovesperformanceofwriteoperationswith increased speed for read operations as the read operations areperformedacrossthenumberofslaveservers.
2. DataSecurity:Security,ingeneral,isanimportantfeaturethateverywebapplicationneeds.
The security can on at an application layer or on a database layer. Datasecurityprotectsagainstlossofdata.Datasecurityisachievedbybackingup a database on a regular basis. If replication is not set up, backing upproduction databases requires the application to be put on maintenancemode. This is required because simultaneous access to a database by anapplicationandthebackupprocessmaycorruptthedata.
Withreplicationinplace,wecanuseoneoftheslavesforbackup.Astheslavedatabaseserverisalwaysinsynchronizationwiththemasterdatabaseserver,wecanbackuptheslavedatabaseserver.Forthat,wecanmaketheslavedatabaseserverstopreplicatingfromamasterdatabaseserverwhilethebackupprocessisrunning.Thisdoesn'trequirethewebapplicationtostop using amaster database server. In fact, it doesn't impact themasterdatabaseserverinanyway.Anotherdatasecurityaspectistoproviderole-basedaccesstoproductionormasterdatabaseservers.Wecanhaveonlyafewroleswhocanaccessthemasterdatabaseserverfromthebackend.Therest of the users or roles have access to the slave database server. Thisreducestheriskofaccidentaldatalossbecauseofhumanerror.
3. Analytics: Analytics and reporting are always important features for adatabasebackedapplication.
These features require fetching of information from a database on afrequent basis so that analysis on the data can be performed. If databasereplication issetup,wecanfetch thedata requiredforanalytics fromtheslave database server without affecting the master database serverperformance.
4. Long distance data distribution: It is a common requirement forapplication developers to replicate production data on local developmentenvironments. In a database replication enabled infrastructure, a slavedatabase server can be used to prepare database copy on a developmentdatabaseserverwithoutconstantaccesstothemasterdatabaseserver.
ConfiguringreplicationIn this section, we will learn configuration for different types of MySQL 8replication methods. It includes step by step instructions for setting up andconfiguringreplication.
Binarylogfilebasedreplication
OneofthemostcommontraditionalmethodsofMySQLdatabasereplicationisthebinarylogfilepositionmethod.Thissectionfocusesonconfigurationofthebinary log file position method of replication. Before we jump into theconfiguration section, itwouldbegood to revise andunderstand thebasics ofbinarylogpositionbasedreplication.
Asdescribedearlier,oneoftheMySQLdatabaseserversactsasmasterandtherestoftheMySQLdatabaseserversbecomeslaves.Themasterdatabaseserveristheoriginforthedatabasechanges.Themasterdatabaseserverwriteseventsbasedonupdatesorchangestothedatabaseinthebinarylogfile.Theformatofthe information recordbeingwritten in the binary log file varies basedon thedatabasechangebeingrecorded.
MySQL REPLICATION SLAVE database servers are configured so that they read thebinarylogeventsfromthemasterdatabaseserver.Slavesexecutetheeventsonlocaldatabasebinarylogfiles.
Thiswayslavessynchronizethedatabasewiththemasterdatabase.Whenslavedatabaseserversreadthebinarylogfilefromthemasterdatabaseserver,slavesgetanentirecopyofthebinarylogfile.Oncethebinarylogfileisreceived,itis
up to the slaves todecidewhichstatements toexecuteon the slavebinary logfile.Itispossibletospecifythatallstatementsfromthemasterdatabaseserverbinarylogfileshouldbeexecutedontheslavedatabaseserversbinarylogfile.Itisalsopossibletoprocesseventsfilteredbyparticulardatabasesortables.
Onlyslavedatabaseserverscanbeconfiguredtofiltereventsfrommasterdatabaseserverlogfiles.Itisnotpossibletoconfigurethemasterdatabaseservertologonlyspecificevents.
MySQL 8 provides a system variable that helps identify the database serveruniquely.
AllthedatabaseserversparticipatinginMySQLreplicationmustbeconfiguredtohaveauniqueID.Eachoftheslavedatabaseserversmustbeconfiguredwiththemasterdatabaseserverhostname,logfilename,andpositionwithinthelogfile.Once set up, it is possible tomodify these details fromwithin aMySQLsessionusingtheCHANGEMASTERTOstatementexecutedontheslavedatabaseserver.
Whentheslavedatabaseserverreadstheinformationfromthemasterdatabasebinary log file, it keeps track of a record of the binary log coordinates. Thebinary log coordinates consists of the filename and position within the file,whichisreadandprocessedfromthemasterdatabaseserver.Theefficiencyofslavedatabaseserversreadingthebinarylogfilefromthemasterdatabaseserveris very high because multiple slave database servers can be connected to themasterdatabaseserverandprocessdifferentpartsofthebinarylogfilefromthemasterdatabaseserver.Themasterdatabaseserveroperationsremainunaffectedbecause the connecting and disconnecting of slave database servers from themasterdatabaseserveriscontrolledbyslavesthemselves.Asmentionedearlier,eachslavedatabaseserverkeepstrackofthecurrentpositionwithinthebinarylog file. So, it is possible for the slave database server to disconnect andreconnect with the master database server and resume the binary log fileprocessing.
A number of methods for setting up database replication are available inMySQL.Theexactmethod for replicationdependson ifdataalreadyexists inthedatabaseandhowreplicationisbeingsetup.EachofthefollowingsectionsareastepforconfiguringMySQLreplication.
ReplicationmasterconfigurationBeforewesetupthereplicationmasterdatabaseserver,itmustbeensuredthatthedatabaseserverhasauniqueIDestablishedandbinaryloggingisenabled.Itmay be required to restart the database server after these configurations aremade.ThemasterdatabaseserverbinarylogisthebasisforMySQL8databasereplication.
Toenablebinarylogging,thelog_binsystemvariableshouldbesettoON.BinaryloggingisenabledforaMySQLdatabaseserverbydefault. Ifmysqld isused toinitialize the data directory manually with a --initialize or --initialize-insecureoption, the binary logging is disabled by default. It has to be enabled byspecifyingthe--log-binoption.The--log-binoptionspecifiesthebasenametobeusedforthebinarylogfiles.
Ifthefilenameisnotspecifiedwiththestartupoption,thebinarylogfilenameswillbe setbasedon thedatabase serverhostname. It is recommended that thebinarylogfilename isspecifiedwith the--log-binoption. If the logfilename isspecifiedwith --log_bin=old_host_name-bin, the log filenamewill be retained evenafterthedatabaseserverhostischanged.
Tosetupthemasterdatabaseserver,opentheMySQLconfigurationfileonthemasterdatabaseserver:
sudovim/etc/mysql/my.cnf
Intheconfigurationfile,makethefollowingchanges.
Firstofall,findthesectionthatbindstheservertolocalhost:
bind-address=127.0.0.1
ReplacethelocalIPaddresswiththeactualdatabaseserverIPaddress.ThisstepisimportantbecausetheslavescanaccessthemasterdatabaseserverusingthepublicIPaddressofthemasterdatabaseserver:
bind-address=175.100.170.1
The following changes are required to configure a unique ID for the masterdatabase server. It also includes the configuration required for setting up themasterbinarylogfile:
[mysqld]
log-bin=/var/log/mysql/mysql-bin.log
server-id=1
Now,let'sconfigurethedatabasetobereplicatedontheslavedatabaseservers.Ifmorethanonedatabaseisrequiredtobereplicatedonslavedatabaseservers,repeatthefollowinglinemultipletimes:
binlog_do_db=database_master_one
binlog_do_db=database_master_two
Once these changes are done, restart the database server using the followingcommand:
sudoservicemysqlrestart
Now, we have the master database server set up. The next step is to grantprivilegestotheslaveuserasfollows:
mysql>mysql-uroot-p
mysql>CREATEUSER'slaveone'@'%'IDENTIFIEDBY'password';
mysql>GRANTREPLICATIONSLAVEON*.*TO'slaveone'@'%'IDENTIFIEDBY'password';
Theprecedingcommandscreatestheslaveuser,grantsprivilegesonthemasterdatabaseserver,andflushesdatabasecachedprivileges.
Now,wehavetobackupthedatabasethatwewanttoreplicate.Wewillbackupthedatabaseusingthemysqldumpcommand.Thisdatabasewillbeusedforcreatingtheslavedatabase.Themasterstatusoutputdisplaysthenameofthebinarylogfilename,currentposition,andthenameofthedatabasetobereplicated:
mysql>USEdatabase_master_one;
mysql>FLUSHTABLESWITHREADLOCK;
mysql>SHOWMASTERSTATUS;
+------------------+----------+---------------------+------------------+
|File|Position|Binlog_Do_DB|Binlog_Ignore_DB|
+------------------+----------+---------------------+------------------+
|mysql-bin.000001|102|database_master_one||
+------------------+----------+---------------------+------------------+
1rowinset(0.00sec)
mysqldump-uroot-pdatabase_master_one>database_master_one_dump.sql
Beforewe take the database backup using the mysqldump command,we have tolock the database to check the current position.This informationwill be usedlatertosetuptheslavedatabaseserver.
After the database dump is taken, the database should be unlocked using thefollowingcommands:
mysql>UNLOCKTABLES;
mysql>QUIT;
Weare donewith all the configuration required to set up a replicationmasterdatabaseserverandmakeitaccessiblebytheREPLICATIONSLAVEdatabaseservers.
Thefollowingoptionshaveanimpactonthemasterdatabaseserversetup:
1. innodb_flush_log_at_trx_commit=1 and sync_binlog=1 options should be set toachieve higher durability and consistency. The options can be set in themy.cnfconfigurationfile.
2. The skip-networking optionmust not be enabled. If it is enabled, the slavecannotcommunicatewiththemasteranddatabasereplicationfails.
REPLICATION SLAVEconfigurationSimilar to themaster database server, each slave database servermust have auniqueID.Oncesetup,thiswillrequiredatabaseserverrestart:
[mysqld]
server-id=2
Forsettingupmultipleslavedatabaseservers,auniquenonzeroserver-idmustbe configured that is different from that ofmaster or anyother slavedatabaseservers.Binaryloggingonaslavedatabaseserverisnotrequiredforreplicationtobesetup.Ifenabled,abinarylogfileonaslavedatabaseservercanbeusedfordatabasebackupsandcrashrecovery.
Now,createanewdatabasethatwillbecomethereplicaofthemasterdatabaseand import the database from the database dump prepared from the masterdatabaseasfollows:
mysql>CREATEDATABASEdatabase_slave_one;
mysql>QUIT;
#mysql-uroot-pdatabase_slave_one</path/to/database_master_one_dump.sql
Now,wehavetoconfigureafewotheroptionsinthemy.cnffile.Likethebinarylog, a relay log consists of numbered files with database change events ascontentsof the file. It alsocontainsan index file thathas thenamesofall theused relay log files.The followingconfigurations set the relay log file, binarylogfile,andnameoftheslavedatabase,whichisareplicaofthemasterdatabaseasfollows:
relay-log=/var/log/mysql/mysql-relay-bin.log
log_bin=/var/log/mysql/mysql-bin.log
binlog_do_db=database_slave_one
A database server restart is required after this configuration change. The nextstepistoenableslavereplicationfromwithinMySQLshellprompt.Executethefollowingcommand toset themasterdatabase informationrequiredby theslavedatabaseserver:
mysql>CHANGEMASTERTOMASTER_HOST='12.34.56.789',MASTER_USER='slaveone',MASTER_PASSWORD='password',MASTER_LOG_FILE='mysql-bin.000001',MASTER_LOG_POS=103;
Asafinalstep,activatetheslaveserver:
mysql>STARTSLAVE;
If the binary logging is enabled on the slave database server, a slave canparticipateinacomplexreplicationstrategy.Insuchareplicationsetup,databaseserverA acts as amaster for database server B.B acts as a slave to the Amasterdatabaseserver.Now,BinturncanactasamasterdatabaseserverfortheCslavedatabaseserver.Somethinglikethiscanbeseenasfollows:
A->B->C
AddingslavestoreplicationIt is possible to add a new slave database server to an existing replicationconfiguration.Thisdoesn'trequirethemasterdatabaseservertobestopped.Theapproach shouldbe tomake a copy of an existing slave database server.Oncecopied,wehavetomodifythevalueforaserver-idconfigurationoption.
Thefollowinginstructionssetupanewslavedatabasetoanexistingreplicationconfiguration. First, an existing slave database server should be shut down asfollows:
mysql>mysqladminshutdown
Now,adatadirectoryfromtheexistingslaveshouldbecopiedtothenewslavedatabase server.Alongwith the data directory, binary logs and relay log filesmustbecopiedaswell.Itisrecommendedtousethesamevaluefor--relay-logforthenewslavedatabaseserverasthatoftheexistingslavedatabaseserver.
Ifthemasterinfoandrelayloginforepositoriesusefilesthenthosefilesmustbecopied from an existing slave database server to a new slave database server.These files hold a master's current binary log coordinates and a slave's relaylogs.
Now,starttheexistingslaveserverthatwasstoppedearlier.
Now,weshouldbeabletostartthenewslavedatabaseserver.Wemusthavetheuniqueserver-idconfiguredbeforestartingthenewslaveserver,ifitisnotsetupalready.
<strong>GTID=source_id:transaction_id</strong>
<strong>gtid_set:</strong><br/><strong>uuid_set[,uuid_set]...</strong><br/><strong>|''</strong><br/><br/><strong>uuid_set:</strong><br/><strong>uuid:interval[:interval]...</strong><br/><br/><strong>uuid:</strong><br/><strong>hhhhhhhh-hhhh-hhhh-hhhh-hhhhhhhhhhhh</strong><br/><br/><strong>h:</strong><br/><strong>[0-9|A-F]</strong><br/><br/><strong>interval:</strong><br/><strong>n[-n]</strong><br/><strong>(n>=1)</strong>
<strong>[mysqld]</strong><br/><strong>server-id=1</strong><br/><strong>log-bin=mysql-bin</strong><br/><strong>binlog_format=ROW</strong><br/><strong>gtid_mode=on</strong><br/><strong>enforce_gtid_consistency</strong><br/><strong>log_slave_updates</strong>
<strong>>CREATEUSER'slaveuser'@'%'IDENTIFIEDBY'password';</strong><br/><strong>>GRANTREPLICATIONSLAVEON*.*TO'slaveuser'@'%'IDENTIFIED<br/>BY'password';</strong><br/><strong>>mysqldump-uroot-pdatabaseName>databaseName.sql</strong>
<strong>>mysql-uroot-pdatabaseName</path/to/databaseName.sql</strong>
<strong>[mysqld]</strong><br/><strong>server_id=2</strong><br/><strong>log_bin=mysql-bin</strong><br/><strong>binlog_format=ROW</strong><br/><strong>skip_slave_start</strong><br/><strong>gtid_mode=on</strong><br/><strong>enforce_gtid_consistency</strong><br/><strong>log_slave_updates</strong>
<strong>sudoservicemysqlrestart</strong>
<strong>>CHANGEMASTERTOMASTER_HOST='170.110.117.12',MASTER_PORT=3306,</strong><br/><strong>MASTER_USER='slaveuser',MASTER_PASSWORD='password',MASTER_AUTO_POSITION=1;</strong>
<strong>STARTSLAVE;</strong>
<strong>----GTIDstateatthebeginningofthebackup--</strong><br/><strong>mysql>SET@@GLOBAL.GTID_PURGED='b9b4712a-df64-11e3-b391-60672090eb04:1-7';</strong>
MySQLmulti-sourcereplicationThissectionfocusesonreplicatingfrommultipleimmediatemastersinparallel.The method is known as multi-source replication. With multi-sourcereplication,aREPLICATIONSLAVE receives transactionsfrommultiplesourcesat thesametime.AchanneliscreatedbyaREPLICATIONSLAVEforeachmasterfromwhichitshouldreceivetransactions.
Themulti-source replication configuration requires at least twomasters and aslavetobeconfigured.ThemasterscanbeconfiguredusingbinarylogpositionbasedreplicationorGTID-basedreplication.ReplicationrepositoriesarestoredinFILEorTABLEbasedrepositories.ATABLEbasedrepositoryiscrashsafe.MySQLmulti-sourcereplicationrequiresaTABLEbasedrepository.TherearetwowaystosetupaTABLErepository.
One is to start mysqld with options as follows: mysqld —master-info-repostiory=TABLE&&–relay-log-info-repository=TABLE
Another preferred way of doing this is to modify the my.cnf file as follows:[mysqld]master-info-repository=TABLErelay-log-info-repository=TABLE
ItispossibletomodifyanexistingREPLICATIONSLAVEthatisusingaFILErepositoryto use a TABLE repository. The following commands convert the existingrepositoriesdynamically:STOPSLAVE;SETGLOBALmaster_info_repository='TABLE';SETGLOBALrelay_log_info_repository='TABLE';
The following commands can be used to add a new GTID-based replicationmaster to an existing multi-source REPLICATION SLAVE. It adds a master to theexisting slave channel: CHANGE MASTER TOMASTER_HOST='newmaster', MASTER_USER='masteruser',MASTER_PORT=3451, MASTER_PASSWORD='password',MASTER_AUTO_POSITION=1FORCHANNEL'master-1';
Thefollowingcommandscanbeusedtoaddanewbinarylogfilepositionbasedreplicationmastertoanexistingmulti-sourceREPLICATIONSLAVE.Itaddsamastertothe existing slave channel: CHANGE MASTER TOMASTER_HOST='newmaster', MASTER_USER='masteruser',MASTER_PORT=3451, MASTER_PASSWORD='password'MASTER_LOG_FILE='master1-bin.000006', MASTER_LOG_POS=628FORCHANNEL'master-1';
ThefollowingcommandsSTART/STOP/RESETalltheconfiguredreplicationchannels:STARTSLAVEthread_types;--TostartallchannelsSTOPSLAVEthread_types;--TostopallchannelsRESETSLAVEthread_types;--Toresetallchannels
The following commands START/STOP/RESET a named channel using a FOR CHANNEL
clause:STARTSLAVEthread_typesFORCHANNELchannel;STOPSLAVEthread_typesFORCHANNELchannel;RESETSLAVEthread_typesFORCHANNELchannel;
<strong>mysql>SHOWSLAVESTATUS\G</strong><br/><strong>***************************1.row***************************</strong><br/><strong>Slave_IO_State:Waitingformastertosendevent</strong><br/><strong>Master_Host:master1</strong><br/><strong>Master_User:root</strong><br/><strong>Master_Port:3306</strong><br/><strong>Connect_Retry:60</strong><br/><strong>Master_Log_File:mysql-bin.000004</strong><br/><strong>Read_Master_Log_Pos:931</strong><br/><strong>Relay_Log_File:slave1-relay-bin.000056</strong><br/><strong>Relay_Log_Pos:950</strong><br/><strong>Relay_Master_Log_File:mysql-bin.000004</strong><br/><strong>Slave_IO_Running:Yes</strong><br/><strong>Slave_SQL_Running:Yes</strong><br/><strong>Replicate_Do_DB:</strong><br/><strong>Replicate_Ignore_DB:</strong><br/><strong>Replicate_Do_Table:</strong><br/><strong>Replicate_Ignore_Table:</strong><br/><strong>Replicate_Wild_Do_Table:</strong><br/><strong>Replicate_Wild_Ignore_Table:</strong><br/><strong>Last_Errno:0</strong><br/><strong>Last_Error:</strong><br/><strong>Skip_Counter:0</strong><br/><strong>Exec_Master_Log_Pos:931</strong><br/><strong>Relay_Log_Space:1365</strong><br/><strong>Until_Condition:None</strong><br/><strong>Until_Log_File:</strong><br/><strong>Until_Log_Pos:0</strong><br/><strong>Master_SSL_Allowed:No</strong><br/><strong>Master_SSL_CA_File:</strong><br/><strong>Master_SSL_CA_Path:</strong><br/><strong>Master_SSL_Cert:</strong><br/><strong>Master_SSL_Cipher:</strong><br/><strong>Master_SSL_Key:</strong><br/><strong>Seconds_Behind_Master:0</strong><br/><strong>Master_SSL_Verify_Server_Cert:No</strong><br/><strong>Last_IO_Errno:0</strong><br/><strong>Last_IO_Error:
</strong><br/><strong>Last_SQL_Errno:0</strong><br/><strong>Last_SQL_Error:</strong><br/><strong>Replicate_Ignore_Server_Ids:0</strong>
<strong>mysql>SHOWPROCESSLIST\G;</strong><br/><strong>***************************4.row***************************</strong><br/><strong>Id:10</strong><br/><strong>User:root</strong><br/><strong>Host:slave1:58371</strong><br/><strong>db:NULL</strong><br/><strong>Command:BinlogDump</strong><br/><strong>Time:777</strong><br/><strong>State:Hassentallbinlogtoslave;waitingforbinlogtobeupdated</strong><br/><strong>Info:NULL</strong>
<strong>mysql>SHOWSLAVEHOSTS;</strong><br/><strong>+-----------+--------+------+-------------------+-----------+</strong><br/><strong>|Server_id|Host|Port|Rpl_recovery_rank|Master_id|</strong><br/><strong>+-----------+--------+------+-------------------+-----------+</strong><br/><strong>|10|slave1|3306|0|1|</strong><br/><strong>+-----------+--------+------+-------------------+-----------+</strong><br/><strong>1rowinset(0.00sec)</strong>
<strong>mysql>STOPSLAVE;</strong><br/><strong>mysql>STARTSLAVE;</strong>
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">STOP</span><spanclass="tokenkeyword">SLAVE</span><spanclass="tokenkeyword">IO_THREAD</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">STOP</span><spanclass="tokenkeyword">SLAVE</span><spanclass="tokenkeyword">SQL_THREAD</span></strong><spanclass="tokenpunctuation"><strong>;</strong><br/><br/><strong><span
class="tokenprompt">mysql></span><spanclass="tokenkeyword">START</span><spanclass="tokenkeyword">SLAVE</span><spanclass="tokenkeyword">IO_THREAD</span>;<br/><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">START</span><spanclass="tokenkeyword">SLAVE</span><spanclass="tokenkeyword">SQL_THREAD</span>;</strong><br/></span>
Implementingreplication
Thebasisforreplicationisthatthemasterdatabaseserverkeepstrackofallthechanges taking place on the master database. The changes are tracked in thebinary log files in the form of events since the server was started. SELECToperationsarenotrecordedastheymodifyneitherthedatabasenorthecontents.EachoftheREPLICATIONSLAVEpullacopyofthebinarylogfilefrommasterinsteadofamasterdatabasepushingthelogfiletotheslave.Theslaveinturnexecutesthe events as it is read from the master's binary log file. This maintains theconsistencybetween master and slave servers. InMySQL replication, each slavefunctions independently from master and other slave servers. So, it is up to theslave to request the master's binary log file at a convenient time withoutimpactingthemasterorslavefunctioning.
The focus for this section of the chapter is onMySQL replication details.Wehavealreadyunderstoodthebasics,whichwillhelpusunderstandtheindepthdetails.
Replicationformats
As we already know by now,MySQL replication works based on replicatingeventsfromthemasterservergeneratedbinarylogs.Later,theseeventsarereadandprocessedbytheslave.Whatwedonotyetknowistheformatinwhichtheevents are recorded in binary log files.Replication formats is the emphasis ofthissection.
When the events are recorded in the master's binary log files, the replicationformat used depends on the binary log format used. Basically, two binaryloggingformatsexist:statementbasedandrowbased.
Withstatement-basedbinarylogging,SQLstatementsarewritteninthemaster'sbinarylogfile.ReplicationontheslaveworksbyexecutingtheSQLstatementson the slave database. This approach is called statement-based replication. ItcorrespondswiththeMySQLstatement-basedbinaryloggingformat.
ThiswastheonlytraditionalformatthatexisteduntilMySQLversions5.1.4andearlier.
With row-based binary logging, the events written in the master binary logindicatehow individual table rows changed.Replication in this caseworksbythe slave copying the events representing changes to the table rows. This is
calledrow-basedreplication.
Row-basedloggingisthedefaultMySQLreplicationmethod.
MySQLsupportsconfiguration tomixstatement-basedandrow-based logging.Thedecisiontousetheloggingformatdependsonthechangebeinglogged.Thisis known as mixed-format logging. Statement-based logging is the defaultformatwhenmixed-formatloggingisused.Basedonthetypeofstatementsandstorageenginebeingused, thelogautomaticallyswitchestorow-basedformat.Replication based on the mixed logging format is known as mixed-formatreplication.
Thebinlog_format systemvariablecontrols the logging formatused ina runningMySQL server. SYSTEM_VARIABLES_ADMIN or SUPER privileges are required to set thebinlog_formatsystemvariableatasessionorglobalscope.
Statement-based versus row-basedreplication
In the earlier section,we learned three different logging formats. Each one ofthese has its own advantages and disadvantages. In usual cases,mixed formatshouldprovidethebestcombinationofintegrityandperformance.However,toachieve the best performance from either statement-based or row-basedreplication, the advantages and disadvantages described in this section arehelpful.
Statement-basedreplicationisatraditionalandproventechniqueincomparisonwithrow-basedreplication.Thenumberofrecordsoreventsrecordedinthelogfiles is smaller. If a statement impactsmany rows, only one statementwill bewrittentothebinarylogfile.Incaseofrow-basedreplication,arecordwillbeenteredforevery tablerowmodified thoughaspartof thesinglestatement. Inessence,thismeansstatement-basedreplicationrequiresmuchlessstoragespaceforlogfiles.Italsomeansbackingupandrestoringorreplicatingtheeventsismuchquicker.
Alongwiththeadvantagesdescribedpreviously,statement-basedreplicationhasdisadvantagesaswell.AsthereplicationworksbasedontheSQLstatements,it
is possible that not all the statements thatmodify data can be replicatedwithstatement-basedreplication.
Afewexamplesaredescribedasfollows:
SQLstatementsdependonauser-definedfunctionthatisnon-deterministicwhenthevaluereturnedbysuchuser-definedfunctionsdependonfactorsotherthantheparameterssuppliedtoit.
UPDATEandDELETEstatementswithaLIMITclausewithoutanORDERBYclause isnon-deterministic as it is possible that theordermayhave changedwhilereplicating.
FORUPDATEorFORSHARE locking read statements that useNOWAIT or SKIP LOCKEDoptions.
User-definedfunctionsmustbeappliedontheslavedatabases.
SQLstatementsusingfunctionssuchasLOAD_FILE(),UUID(),USER(),UUID_SHORT(),FOUND_ROWS(), SYSDATE(), GET_LOCK(), and so on cannot be replicated properlyusingstatement-basedreplication.
INSERTorSELECTstatementsrequirehighernumberofrowlevellocks.
UPDATEwithtablescanrequireslockingahighernumberofrows.
Complex SQL statements must be evaluated and executed on the slavedatabaseserverbeforerowsareinsertedorupdated.
Let's seeadvantagesprovidedby row-based replication.Row-based replicationis the safest form of replication because instead of depending on SQLstatements, itdependsonthevaluesstoredin the tablerows.So,everychangecanbereplicated. It requires fewer rowlocks incaseINSERT...SELECT statements.UPDATEandDELETEstatementswithWHEREclausesthatdonotusekeysrequirefewerrowlevellocks.
Themajordisadvantagewithrow-basedreplicationisthatitgeneratesmoredata
thatmustbelogged.Withstatement-basedreplication,oneDMLSQLstatementis sufficient for logging though it modifies many rows. In case of row-basedreplication, it requires logging for every row that changed.Thebinary log filegrowsveryquicklywithrow-basedreplication.IttakeslongertimetoreplicatedeterministicuserdefinedfunctionsthatgeneratelargeBLOBvalues.
<strong>mysql>SHOWPROCESSLIST\G</strong><br/><strong>***************************1.row***************************</strong><br/><strong>Id:2</strong><br/><strong>User:root</strong><br/><strong>Host:localhost:32931</strong><br/><strong>db:NULL</strong><br/><strong>Command:BinlogDump</strong><br/><strong>Time:94</strong><br/><strong>State:Hassentallbinlogtoslave;waitingforbinlogtobeupdated</strong><br/><strong>Info:NULL</strong>
<strong>mysql>SHOWPROCESSLIST\G</strong><br/><strong>***************************1.row***************************</strong><br/><strong>Id:10</strong><br/><strong>User:systemuser</strong><br/><strong>Host:</strong><br/><strong>db:NULL</strong><br/><strong>Command:Connect</strong><br/><strong>Time:11</strong><br/><strong>State:Waitingformastertosendevent</strong><br/><strong>Info:NULL</strong><br/><strong>***************************2.row***************************</strong><br/><strong>Id:11</strong><br/><strong>User:systemuser</strong><br/><strong>Host:</strong><br/><strong>db:NULL</strong><br/><strong>Command:Connect</strong><br/><strong>Time:11</strong><br/><strong>State:Hasreadallrelaylog;waitingfortheslaveI/Othreadtoupdateit</strong><br/><strong>Info:NULL</strong>
Intheoutput,thread10istheslave'sI/Othreadandthread11istheslave'sSQLthread.TheI/Othreadiswaitingforthemaster'sbinlogdumpthreadtosendbinarylogcontents.TheSQLthreadhasreadallthestatementsloggedintheslaverelaylogs.FromtheTimecolumn,itcanbedeterminedhowslowtheslaveisrunningbehindthe
Replicationchannels
Areplicationchannelisapathoftransactionflowfromamastertoaslave.Thissection explains how channels can be used in replication. TheMySQL serverautomatically creates a default channelwith the name as "" (empty string) onstartup.Thedefaultchannelisalwayspresentandcan'tbecreatedordestroyedby the user. Replication statements work on the default channel if no otherchannel is created. This section describes statements that are applied toreplicationchannelswhenthereexistsatleastonenamedchannel.
Inmulti-source replication, the slave database server opens multiple channels,oneforeachmaster.Eachchannelhas itsownrelaylogandSQLthreads.Thereplicationchannelhasahostnameandportassociation.
Multiplechannelscanbeassignedtothesamehostnameandportcombination.A maximum of 256 channels can be added to one slave in a multi-sourcereplication topology inMySQL8.The channelmust have a nonempty uniquename.
TheFORCHANNELclauseisusedwithvariousMySQLstatementsforthereplicationoperationstobeperformedonindividualchannels.Theclausecanbeappliedtothefollowingstatements:
CHANGEMASTERTO
STARTSLAVE
STOPSLAVE
RESETSLAVE
SHOWRELAYLOGEVENTS
FLUSHRELAYLOGS
SHOWSLAVESTATUS
Apartfromthese,thefollowingfunctionshaveanadditionalchannelparameter:
MASTER_POS_WAIT()
WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS()
Formulti-sourcereplicationtoworkcorrectly,thefollowingstartupoptionsmustbeconfigured:
--relay-log-info-repository:Asdescribedearlier, thismustbe set to TABLE formulti-source replication. In MySQL 8, the FILE option is deprecated andTABLEisthedefaultoption.
--master-info-repository:ThismustbesettoTABLE.
--log-slave-updates:Transactionsreceivedfromthemasterarewrittentothebinarylogs.
--relay-log-purge:Eachchannelpurgesitsownrelaylogsautomatically.
--slave-transaction-retries:SQLthreadsofallchannelsretrytransactions.
--skip-slave-start:Noreplicationthreadsstartonanychannels.
--slave-skip-errors: Execution continues and errors are skipped for allchannels.
--max-relay-log-size=size: The relay log file is rotated after reaching themaximumsize.
--relay-log-space-limit=size:Upperlimitfortotalsizeofallrelaylogsforeachindividualchannel.
--slave-parallel-workers=value:Numberofslaveparallelworkersperchannel.
--slave-checkpoint-group:WaitingtimebyI/Othread.
--relay-log-index=filename:Eachchannel'srelaylogindexfilename.
--relay-log=filename:Eachchannel'srelaylogfilename.
--slave-net-timeout=N:EachchannelwaitsforNsecondstocheckforbrokenconnection.
--slave-skip-counter=N:EachchannelskipsNeventsfromthemaster.
Replicationrelayandstatuslogs
TheREPLICATIONSLAVEservercreateslogsthatholdthebinarylogeventssentfromthemasterdatabase
servertotheslavedatabaseserver.Theinformationisrecordedaboutthecurrent
statusandlocationintherelaylog.Threetypesoflogsareusedinthisprocess:
1. Relaylog:Therelayloghaseventssentfromthemaster'sbinarylog.Theeventsarewritten
byaslave'sI/Othread.Eventsfromtheslave'srelaylogareexecutedontheslave
bytheslave'sSQLthread.
2. Master info log: The master info log has information about status andcurrentconfigurationforthe
slave's connection to themaster database server.The informationheldby
themaster
infologincludeshostname,logincredentials,andcoordinatesindicatingaslave's
position on reading themaster's binary log.These logs arewritten to themysql.slave_master_infotable.
3. Relayloginfolog:Therelayloginfologstoresinformationregardingtheexecutionpointwithinthe
slave's relay log. The relay log info log is written in amysql.slave_relay_log_infotable.
No attempt should be made to insert or update rows in theslave_master_info or slave_relay_log_info tables manually. This maycauseunexpectedbehavior.ItisnotsupportedinMySQL
replication.
Theslave relay logconsistsof an index file alongwitha setofnumbered logfiles.
Theindexfilecontainsthenamesofallrelaylogfiles.MySQLdatadirectoryis
the default location for the relay log files. The relay log file indicates anindividually
numbered file containing events. Whereas the relay log denotes the set ofnumbered
relaylogfilesandanindexfilecollectively.Theformatforrelaylogfilesis
the same as that of the binary log files. The index filename for relay log ishost_name-relay-bin.indexbydefault for thedefaultchannelandhost_name-relay-bin-channel.index for non-default replication channels. The default locations for therelaylogfile
andrelaylogindexfilecanbeoverriddenwiththe--relay-logand--relay-log-index
serverstartupoptions.Iftheslave'shostnameischangedafterreplicationhas
been set up and the slave uses default host-based relay log filenames, it canthrow
errors such as Failed to open the relay log andCould not find target logduringrelayloginitialization.Thismayfailthereplication.Sucherrorsmaybe
avoided by using --relay-log and --relay-log-index options to specify relay logfilenamesexplicitly.Usingtheseoptionsonslavesetup
willmakethenamesindependentoftheserver'shostname.
Evaluating replication filteringrules
Thissectionfocusesonthefilteringrulesandhowserversevaluatetheserules.
Basically,if themasterdoesn't logthestatement,theslavedoesn'treplicatethestatement. If the master logs the statement in its binary log file, the slavereceives the statement. However, it is up to the slave database server if itprocessesthestatementorignoresit.Optionsareavailableforthemasterserverto controlwhich databases and tables should be replicated on the slaves. Therecommendedwayistousefiltersontheslavetocontroltheeventsthataretobeexecutedontheslavedatabaseserver.Thedecisionaboutwhethertoexecuteor ignore the statements received from the master are made based on the --replicate-* options used when the slave was started. Once the slave server isstarted, the CHANGE REPLICATION FILTER statement can be used to set the optionsdynamically.
All replication filtering options follow the same rules for casesensitivity as names of databases and tables, including thelower_case_table_namessystemvariable.
Groupreplication
Thissectionof thechapterexplainswhatgroupreplicationis,settingupgroupreplication, configure andmonitor group replication. Basically,MySQL groupreplication is a plugin that enables us to create elastic, highly-available, fault-tolerantreplicationtopologies.
Thepurposeofthegroupreplicationistocreateafaulttolerantsystem.Tocreatea fault tolerant system, the components should be made redundant. Thecomponent should be removed without impacting the way system operates.There are challenges in setting up such a system. The complexity of such asystem is of a different level. Replicated databases require maintenance andadministration of several servers instead of just one. The servers cooperatetogether to create a group, which raises the problems related to networkpartitioning and split-brain scenarios. So, the ultimate challenge is to haveagreementfrommultipleserversonthestateofthesystemanddataaftereverychangeappliedonthesystem.Thismeansthattheserversneedtooperateasadistributedstatemachine.
MySQL group replication can provide such a distributed state machinereplicationwithstrongcoordinationbetweenservers.Theserversthatbelongtothesamegroupcoordinatethemselvesautomatically.Inagroup,onlyoneserver
accepts updates at a time.The election of primary is done automatically.Thismodeisknownassingle-primarymode.
MySQLprovidesagroupmembershipservice,whichisresponsibleforkeepingtheviewofthegroupconsistentandavailableforallservers.Theviewiskeptupdatedwhen the servers join or leave the group. In case, any of the serversleaves the group unexpectedly, the failure detection mechanism notifies thegroupaboutviewchange.
Thisbehaviorisautomatic.
Themajorityofthegroupmembershavetoagreeontheorderofthetransactiontocommitintheglobalsequenceoftransactions.Itisuptotheindividualservertodecidewhetherthetransactionshouldbecommittedoraborted,butallserversmake the same decision. The system does not proceed until themembers areunable to reach toagreementasa resultofsplitdue tonetworkpartition.Thismeansthesystemhasbuilt-in,automatic,split-brainprotectionmechanism.AllthisisdonebyGroupCommunicationSystem(GCS)protocols.Itprovidesafailure detection mechanism, group membership service, safe and completelyorderedmessagedelivery.The implementationof thePaxosalgorithmisat thecoreofthistechnology,whichactsasthegroupcommunicationengine.
Primary-secondary replicationversusgroupreplicationThissectionfocusesonsomebackgrounddetailsofhowreplicationworks.ThiswillbeusefulinunderstandingtherequirementsforgroupreplicationandhowitisdifferentfromtheclassicasynchronousMySQLreplication.
The following figure showcases how traditional asynchronous primary-secondaryreplicationworks.Theprimaryisthemasterandthesecondaryisoneormoreslavesconnectedtothemaster,asshowninthefollowingfigure:
Figure1.MySQLAsynchronousReplication
MySQLalsosupportssemisynchronousreplication inwhich themaster waitsfor at least one of the slaves to acknowledge the transaction receipt:
Figure2.MySQLSemisynchronousReplication
Thebluearrowsinthefiguresindicatethemessagespassedbetweenserversandtheclientapplication.
With group replication, a communication layer is provided that guaranteesatomicmessages and total ordermessage delivery. All read-write transactionsare committed only after they are approved by the group. The read onlytransactionsarecommittedimmediatelyasitdoesnotneedcoordination.So,ingroupreplication,thedecisiontocommitatransactionornotisnotunilateralbytheoriginatingserver.Whenthetransactionisreadyforcommit,theoriginatingserver broadcasts the write values and corresponding write set. All serversreceivethesamesetoftransactionsinthesameorder.So,allserversapplythesame transactions in the same order. This way all servers remain consistentwithinthegroup:
Figure3.MySQLGroupReplicationProtocol
GroupreplicationconfigurationThissectionfocusesonconfiguringgroupreplication.
Firstofall,openthemy.cnfconfigurationfileandaddthefollowingentriesinthemysqldsectionasfollows:
[mysqld]
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=binlog
binlog_format=ROW
transaction_write_set_extraction=XXHASH64
loose-group_replication_bootstrap_group=OFF
loose-group_replication_start_on_boot=OFF
loose-group_replication_ssl_mode=REQUIRED
loose-group_replication_recovery_use_ssl=1
These are general configurations related to global transaction IDs and binaryloggingrequiredforgroupreplication.
Thenextstepistosetupgroupreplicationconfigurations.Theseconfigurationsinclude group UUID, group members white listing, and indicating seedmembers:
#Sharedreplicationgroupconfiguration
loose-group_replication_group_name="929ce641-538d-415d-8164-ca00181be227"
loose-group_replication_ip_whitelist="177.110.117.1,177.110.117.2,177.110.117.3"
loose-group_replication_group_seeds="177.110.117.1:33061,177.110.117.2:33061,177.110.117.3:33061"
...Choosing
The following configuration is required for deciding whether a single-mastergroupormulti-mastergroupistobesetup.Forenablingthemulti-mastergroup,uncomment
loose-group_replication_single_primary_modeand
loose-group_replication_enforce_update_everywhere_checks directives. It will set up amulti-masterormulti-primarygroup:
...
#SingleorMulti-primarymode?Uncommentthesetwolines
#formulti-primarymode,whereanyhostcanacceptwrites
loose-group_replication_single_primary_mode=OFF
loose-group_replication_enforce_update_everywhere_checks=ON
Itmustbeensuredthattheseconfigurationsarethesameonalltheservers.AnychangestotheseconfigurationsrequireMySQLgroupstoberestarted.
Thefollowingconfigurationsaredifferentoneachoftheserversinthegroup:
...
#Hostspecificreplicationconfiguration
server_id=1
bind-address="177.110.117.1"
report_host="177.110.117.1"
loose-group_replication_local_address="177.110.117.1:33061"
Theserver-idmustbeuniqueacrossall servers in thegroup.Theport33061 isthe one used bymembers to coordinate for group replication.MySQL serverrestartisrequiredafterthesechangesaremade.
Ifnotalreadydone,wehavetoallowaccesstotheseportsusingthefollowingcommands:
sudoufwallow33061
sudoufwallow3306
The next step is to create a replication user and enable the replication plugin.Thereplicationuserisrequiredforeachservertoestablishgroupreplication.Weneedtoturnbinaryloggingoffduringthereplicationusercreationprocessastheuserwillbedifferentforeachserver,asshowninthefollowingblock:
SETSQL_LOG_BIN=0;
CREATEUSER'mysql_user'@'%'IDENTIFIEDBY'password'REQUIRESSL;
GRANTREPLICATIONSLAVEON*.*TO'mysql_user'@'%';
FLUSHPRIVILEGES;
SETSQL_LOG_BIN=1;
Now,use CHANGE MASTER TO to configure the server to use the credentials for thegroup_replication_recoverychannel:
CHANGEMASTERTOMASTER_USER='mysql_user',MASTER_PASSWORD='password'FORCHANNEL'group_replication_recovery';
Now,weareallset toinstall theplugin.Connect totheserverandexecutethefollowingcommand:
INSTALLPLUGINgroup_replicationSONAME'group_replication.so';
Usethefollowingstatementtoverifyifthepluginisactivatedornot:
SHOWPLUGINS;
Thenextstepistostartupthegroup.Executethefollowingstatementsononememberofthegroup:
SETGLOBALgroup_replication_bootstrap_group=ON;
STARTGROUP_REPLICATION;
SETGLOBALgroup_replication_bootstrap_group=OFF;
Now,wecanstartgroupreplicationonanotherserver:
STARTGROUP_REPLICATION;
WecancheckthegroupmemberslistusingthefollowingSQLquery:
mysql>SELECT*FROMperformance_schema.replication_group_members;
+---------------------------+--------------------------------------+
|CHANNEL_NAME|MEMBER_ID|
+---------------------------+--------------------------------------+
|group_replication_applier|13324ab7-1b01-11e7-9dd1-22b78adaa992|
|group_replication_applier|1ae4b211-1b01-11e7-9d89-ceb93e1d5494|
|group_replication_applier|157b597a-1b01-11e7-9d83-566a6de6dfef|
+---------------------------+--------------------------------------+
+---------------+-------------+--------------+
|MEMBER_HOST|MEMBER_PORT|MEMBER_STATE|
+---------------+-------------+--------------+
|177.110.117.1|3306|ONLINE|
|177.110.117.2|3306|ONLINE|
|177.110.117.3|3306|ONLINE|
+---------------+-------------+--------------+
3rowsinset(0.01sec)
Groupreplicationusecases
The MySQL group replication feature provides a way to build fault tolerantsystemsby replicating the stateof the system throughout a setof servers.Thegroup replication system stays available as long as themajority of servers arefunctioning even if some of the servers fail. Server failures are tracked by agroupmembershipservice.
Thegroupmembership service relieson thedistributed failuredetector,whichsignalsifanyserverleavesthegroup,voluntarilyorduetounexpectedhalt.Thedistributedrecoveryprocedureensuresthatwhentheserversjointhegroup,theyarebroughtup todateautomatically.Therefore,continuousdatabaseservice isguaranteed with MySQL group replication. There is one problem though.Although the database service is available, the clients connected to itmust beredirected to a different serverwhen the server crashes.The group replicationdoes not attempt to resolve it. It should be dealt with by a connector, loadbalancer,router,orsomeothermiddleware.
ThefollowingarethetypicalusecasesofMySQLgroupreplication:
1. Elastic replication: Group replication is suitable for fluid environments
where the number of servers grow or shrink dynamically withminimumsideeffects.Theexampleiscloud-baseddatabaseservices.
2. Highly available shards: MySQL group replication can be used toimplement highly available write scale-out shards where each replicationgroupmapstooneshard.
3. Alternative to master-slave replication: Group replication can be ananswer to contention problems arising in certain situations with singlemasterserverreplication.
4. Autonomic systems: MySQL group replication can be deployed for theautomationbuiltintothereplicationprotocol.
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">UPDATE</span>performance_schema<spanclass="tokenpunctuation">.</span>setup_instruments<spanclass="tokenkeyword">SET</span>ENABLED<spanclass="tokenoperator">=</span><spanclass="tokenstring">'YES'</span><spanclass="tokenkeyword">WHERE</span><spanclass="tokenkeyword">NAME</span><spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'stage/sql/Applyingbatchofrowchanges%'</span><spanclass="tokenpunctuation">;</span></strong>
TheMySQL8replicationprocesscanworkeventhoughthesourcetableonthemasterandthedestinationtableontheslaveusesdifferentenginetypes.Thedefault_storage_enginesystemvariableisnotreplicated.Thisisahugeadvantageinreplicationwhereindifferentenginetypescanbeusedfordifferentreplicationscenarios.Anexampleisascale-outscenariowherewewantallreadoperationstobeperformedontheslavedatabaseserver,whereasallwriteoperationsshouldbeperformedonthemasterdatabaseserver.Insuchacase,wecanuseatransactionalInnoDBengineonthemasterandanon-transactionalMyISAMenginetypeontheslavedatabaseserver.
Consideranexampleofanorganizationthatwantstodistributesalesdatatodifferentdepartmentstospreadtheloadfordataanalysis.MySQLreplicationcanbeusedtohaveasinglemasterreplicatedifferentdatabasestodifferentslaves.Thiscanbeachievedbylimitingthebinarylogstatementsbyusingthe--replicate-wild-do-tableconfigurationoptiononeachslave.
OnceMySQLreplicationissetup,asthenumberofslavesconnectedtothemasterincrease,theloadalsoincreases.Thenetworkloadonthemasteralsoincreasesaseachslaveissupposedtoreceiveafullcopyofthebinarylogs.Themasterdatabaseserverisalsobusy
processingrequests.Inthisscenario,itbecomesnecessarytoimprovetheperformance.Oneofthesolutionstoimproveperformanceistocreateadeeperreplicationstructurethatenablesreplicationofamastertoonlyoneslave.Therestoftheslavesconnecttotheprimaryslavefortheiroperations.
Summary
In this chapter,we learned insightfuldetails aboutMySQL8 replication,whatreplicationis,andhowithelpssolvespecificproblems.Wealsolearnedhowtosetupstatement-basedandrow-basedreplicationtypes.Alongtheway,wealsolearnedaboutthesystemvariablesandserverstartupoptionsforreplication.Inthelaterpartofthechapter,wediveddeepintogroupreplicationandhowitisdifferent from the traditional method of MySQL replication.We also learnedlogging and replication formats. Last but not least, we learned differentreplicationsolutionsinbrief.Wecoveredalotofstuff,huh?
It'snowtimetomoveontoournextchapter,wherewewillbesettingupseveraltypesofpartitioning,andexploringtheselectionofpartitioning,andpruningofpartitioning.Italsoexplainshowtocopewithrestrictionsandlimitationswhilepartitioning. The reader will be able to understandwhich type of partitioningsuitsasituationaspertherequirement.
PartitioninginMySQL8
In the previous chapter, replication inMySQL8was explained.This includeddetailed explanations of replication, configuration, and implementation. Thechapter also explained group replication versus clustering, and covered thereplicationapproachasasolution.
Inthischapter,wewilldopartitioninginMySQL8.Partitioningistheconceptof managing and maintaining data with specific operations with severaloperators,anddefiningrulestocontroloverpartitioning.Basically,itprovidesaconfigurationhookformanagingtheunderlyingdatafilesinaspecifiedway.
Wewillcoverthefollowingtopicsonpartitioning:
Overviewofpartitioning
Typesofpartitioning
Partitionmanagement
Partitionselection
<strong>CREATETABLEtp(tp_idINT,amtDECIMAL(5,2),trx_dateDATE)</strong><br/><strong>ENGINE=INNODB</strong><br/><strong>PARTITIONBYHASH(MONTH(trx_date))</strong><br/><strong>PARTITIONS4;</strong>
Partitioningisapplicableonallindexesandalldataofthetable.Itisnotapplicableoneitherindexesordata,andviceversaisalsonotapplicable.Itcanbeapplicableonbothindexesanddatatogetheranditcannotbeappliedonpartofthetable.Theprecedingtabletphasnouniqueorprimarykeysdefinedbutingeneralpracticeweusuallyhaveprimarykeys,uniquekeys,orbothaspartofthetable,andpartitioningcolumnchoicedependsuponthesekeysifanyofthemispresent.Thepartitioningcolumnchoiceisgivenindetailinthepartitioningkeys,primarykeys,anduniquekeyssection.Tosimplifytheconceptofpartitioningtheexamplesgivenmaynotincludethesekeys.
Typesofpartitioning
SeveraltypesofpartitioningaresupportedinMySQL8,listedasfollows:
RANGE Partitioning: Assigns rows to partitions from the column values thatcomebetweenthegivenrangeofvalues
LIST Partitioning: Assigns rows to partitions from the column values thatmatcheswithoneofthegivensetofvalues
COLUMNSPartitioning:AssignsrowstopartitionswithmultiplecolumnvalueswitheitherRANGEorLISTpartitioning
HASH Partitioning: Assigns partition based on user specified expressionsevaluatedoncolumnvalues
KEYPartitioning: In addition to HASH partitioning, allows the use ofmultiplecolumnvalues
Subpartitioning: In addition to partitioning, allows further division inpartitionedtables,alsoknownascompositepartitioning
Differentrowsofthetablecanbeassignedtodifferentphysicalpartitions;thisisknownashorizontalpartitioning.Differentcolumnsofthetablecanbeassignedtodifferentphysicalpartitions;thisisknownasverticalpartitioning.MySQL8
currentlysupportshorizontalpartitioning.
FortheLIST,RANGE,andLINEARHASH typesofpartitioning,thevalueofpartitioningcolumns isgiven to thepartitioning function.Thepartitioning function returnsan integer value that is the partition number in which the record should bestored.Thepartitionfunctionmustbenonrandomandnonconstant.Thepartitionfunctioncannotcontainqueriesandcanuse theSQLexpressionwhich returnseitheranintegerorNULL,wheretheintegerasintvalmustfollowtheexpression-MAXVALUE<=intval<=MAXVALUE.Here,-MAXVALUErepresentsthelowerlimitandMAXVALUEistheupperlimitfortheintegertypevalue.
Thestorageenginemustbethesameforallpartitionsofthesametable,howeverthereisnorestrictiononusingdifferentstorageenginesfordifferentpartitionedtablesinthesamedatabaseorMySQLserver.
PartitioningmanagementTherearedifferentways touseSQL statements tomodifypartitioned tablesandperform operations such as add, redefine, merge, drop, or split existingpartitionedtables.InformationaboutpartitionedtablesandpartitionscanalsobeobtainedwithSQLstatements.
MIN_ROWS and MAX_ROWS can be used to configure the maximum and minimumnumberofrowsandcanbestoredinthepartitiontable.
PartitionselectionandpruningExplicit selectionof partition and subpartition is alsoprovided. It enables rowmatching to the conditionsgiven in thewhereclause. Inpartition, thepruningconceptsdescribeddonotscanpartitionswherenopossiblematchingvaluescanbe present, and are applied using queries, whereas partition selection isapplicableforbothqueriesandmanyoftheDMLstatements.
Restrictions and limitations inpartitioning
Stored procedures or functions, user defined functions or plugins, and uservariables or declaredvariables are restricted in partitioning expressions.Therearealsoseveralrestrictionsandlimitationsapplicabletopartitioninggiveninthedetailedsection.
Seethefollowinglistforsomeoftheadvantagesofpartitioning:
Partitioningfacilitatesstoringmoredatainonetablethancanbeheldonafilesystempartitionorsingledisk.
Datathathasbecomeuselesscanberemovedeasilybydroppingapartitionorpartitionsthatonlycontaintheuselessdata.Insomecaseswherespecificdata is required to be added separately, this can be done easily withpartitioninginsingleormultiplepartitionsbasedonthespecifiedrule.
Queryoptimizationthatoccursautomaticallybasedonpartitioneddatabynotsearchingfordatainpartitionsthatarenotapplicableasperthewhere
condition.
Inadditiontopartitionpruning,partitionselectionissupportedexplicitlybywhichwhereclauseisappliedonaspecifiedpartitionormultiplepartitions.
Greater query throughput is achieved by separating data search intomultipledisks.
Typesofpartitioning
In this section,youwillunderstanddifferent typesofpartitioningandalso thepurposeofusingspecificpartitioning.ThefollowingisalistofthepartitioningtypesthatareavailableinMySQL8:
RANGEpartitioning
LISTpartitioning
COLUMNSpartitioning
HASHpartitioning
KEYpartitioning
Subpartitioning
In addition to the above list, we will also see NULLhandling in MySQL 8Partitioningindetailedsection.
Averycommonusecase fordatabasepartitioning is segregatingdatabydate.MySQL
8 does not support date partitioning, which some database systems provideexplicitly, but it is easy to create partitioning schemes with date, time, ordatetime columns, or that are based on date/time related expressions thatevaluatevaluesfromthesecolumntypes.
You can use the date, time, or datetime types as column values for partitioncolumnswithoutanymodificationsifusingKEYorLINEARKEYpartitioning,whereasinotherpartitioningtypesanexpressiongivingbackanintegerorNULLvalue isrequired.
Irrespective of which type of partitioning you use, partitions always getnumbered automatically with an integer number in sequence of the partitionscreated. If, for example, the table uses four partitions, they are numbered as0,1,2,and3foreachofthepartitionsaspercreationsequence.
Whenyouspecifynumbersofpartitions,itmustbeevaluatedtoapositive,nonzero integer without any leading zeros. Decimal fractions are not allowed aspartitionnumbers.
Names of partitions are not case-sensitive and should follow conventions orrules just like other MySQL identifiers such as tables. The options used inpartitiondefinitionarealreadyprovidedbytheCREATETABLEsyntax.
Now,let'slookatpartitionindetailandexamineeachofthetypestolearnhowtheyaredifferenttoeachother.
RANGEpartitioningInthistypeofpartitioning,asthenamestates,RANGEisgiveninanexpressionthatevaluateswhetheravalueliesinthegivenrangeornot.RangesaredefinedwiththeVALUESLESSTHANoperatorandtheyshouldnotbeoverlappingandcontiguous.
For thenext fewexamples, supposeweare creating a tableholding employeepersonalrecordsfor25foodstores.Thestoresarenumberedfrom1to25andisachainof25foodstores,asshowninthefollowingblock:
CREATETABLEemployee(
employee_idINTNOTNULL,
first_nameVARCHAR(30),
last_nameVARCHAR(30),
hired_dateDATENOTNULLDEFAULT'1990-01-01',
termination_dateDATENOTNULLDEFAULT'9999-12-31',
job_codeINTNOTNULL,
store_idINTNOTNULL
);
Nowlet'sdopartitioningofthetable,soyoucanpartitionthetablebyrangeasperyourneed.Supposeyouconsiderusingdivision tosplit thedata fivewayswiththestore_idrangeforpartitioning.Forthis,thetablecreationdefinitionwilllookasfollows:
CREATETABLEemployee(
employee_idINTNOTNULL,
first_nameVARCHAR(30),
last_nameVARCHAR(30),
hired_dateDATENOTNULLDEFAULT'1990-01-01',
termination_dateDATENOTNULLDEFAULT'9999-12-31',
job_codeINTNOTNULL,
store_idINTNOTNULL
)
PARTITIONBYRANGE(store_id)(
PARTITIONp0VALUESLESSTHAN(6),
PARTITIONp1VALUESLESSTHAN(11),
PARTITIONp2VALUESLESSTHAN(16),
PARTITIONp3VALUESLESSTHAN(21),
PARTITIONp4VALUESLESSTHAN(26)
);
So, as per the above partitioning scheme, all inserted rows that contain theemployeesworking at stores 1 to 5 are stored in the p0 partition, employeesworkingatstores6to10arestoredinthep1partition,andsoon.Ifyoutakealookatthepartitiondefinition,partitionsareorderedfromthelowesttohighest
store_id column values, and the PARTITION BY RANGE syntax looks similar toprogrammingstatementsif…elseif…statements,doesn'tit?
Well,youarethinkingaboutwhatwillhappenifarecordcomeswithstore_id26;this would result in an error as the server does not knowwhere to place therecord.Therearetwowaystokeepthiserrorfromoccurring:
1. ByusingtheIGNOREkeywordwiththeINSERTstatement.2. ByusingMAXVALUEinsteadofaspecifiedrange(26).
Andyes,ofcourse,youcanextendthelimitsbyusingtheALTERTABLEstatementtoaddnewpartitionsforstores26-30,30-35,andsoon.
Similar to store_id, you could also partition the table based on the job codes -based on the range of column values. Suppose if 5 digit codes are used formanagementpositions,4digitcodesareusedforofficeandsupportpersonnel,and3digitcodesareusedforregularworkers, thenthepartitiontablecreationdefinitionwouldbeasfollows:
CREATETABLEemployee(
employee_idINTNOTNULL,
first_nameVARCHAR(30),
last_nameVARCHAR(30),
hired_dateDATENOTNULLDEFAULT'1990-01-01',
termination_dateDATENOTNULLDEFAULT'9999-12-31',
job_codeINTNOTNULL,
store_idINTNOTNULL
)
PARTITIONBYRANGE(job_code)(
PARTITIONp0VALUESLESSTHAN(1000),
PARTITIONp1VALUESLESSTHAN(10000),
PARTITIONp2VALUESLESSTHAN(100000)
);
You can also specify partitioning with one of the two columns of date typevalues.Supposeyouwish topartitionbasedon theyear eachof the employeejoined - soby thevalueof YEAR(hired_date).Now the table definitionwill be asfollows:
CREATETABLEemployee(
employee_idINTNOTNULL,
first_nameVARCHAR(30),
last_nameVARCHAR(30),
hired_dateDATENOTNULLDEFAULT'1990-01-01',
termination_dateDATENOTNULLDEFAULT'9999-12-31',
job_codeINTNOTNULL,
store_idINTNOTNULL
)
PARTITIONBYRANGE(YEAR(hired_date))(
PARTITIONp0VALUESLESSTHAN(1996),
PARTITIONp1VALUESLESSTHAN(2001),
PARTITIONp2VALUESLESSTHAN(2006),
PARTITIONp3VALUESLESSTHANMAXVALUE
);
Accordingtothisscheme,allemployeesrecordedhiredbefore1996willbestoredinthepartitionp0,thenrecordswithahiredatebefore2001willbestoredinthepartitionp1,recordsbetween2001and2006inp2,andtherestoftherecordswillbestoredinpartitionp3.
Partition schemes based on time intervals can be implemented using thefollowingtwooptions:
1. PartitionthetablebyRANGEanduseafunctionoperatingonthedate,timeordatetime column values to return an integer value for the partitioningexpression
2. Partition the table by RANGE COLUMN and use the date, time, or datetimecolumnsasthepartitioncolumn
RANGE COLUMN is supported in MySQL 8 and is described in detail in the COLUMNPARTITIONINGsection.
LISTpartitioningAsthenamestates,LISTpartitioninguses lists for tablepartitioning.The list iscomma separated integer values defined while partitioning with VALUES IN
(value_list);here,value_listreferstocommaseparatedintegerliterals.
LIST partitioning is similar to RANGE partitioning in many ways, but there aredifferences.Theoperatorusedineachpartitioningisdifferent.Theoperatorusesa list of comma separated values to bematchedwith the column value or thepartitionexpressionevaluatingtointegervalue.
Consideringtheemployeetableasanexample,thebasicdefinitionforthetableusingthecreatetablesyntaxwillbeasfollows:
CREATETABLEemployee(
employee_idINTNOTNULL,
first_nameVARCHAR(30),
last_nameVARCHAR(30),
hired_dateDATENOTNULLDEFAULT'1990-01-01',
termination_dateDATENOTNULLDEFAULT'9999-12-31',
job_codeINTNOTNULL,
store_idINTNOTNULL
);
Supposeyouwish todistribute these25foodstoresamongfivezones—North,South, East, West, and Central, with the store ID numbers (1,2,11,12,21,22),(3,4,13,14,23,24), (5,6,15,16,25), (7,8,17,18), and (9,10,19,20) respectively forthezones.
Partitioningthetablewiththezoneslistwillprovidethefollowingdefinitionfortablepartition:
CREATETABLEemployee(
employee_idINTNOTNULL,
first_nameVARCHAR(30),
last_nameVARCHAR(30),
hired_dateDATENOTNULLDEFAULT'1990-01-01',
termination_dateDATENOTNULLDEFAULT'9999-12-31',
job_codeINTNOTNULL,
store_idINTNOTNULL
)
PARTITIONBYLIST(store_id)(
PARTITIONpNorthVALUESIN(1,2,11,12,21,22),
PARTITIONpSouthVALUESIN(3,4,13,14,23,24),
PARTITIONpEastVALUESIN(5,6,15,16,25),
PARTITIONpWestVALUESIN(7,8,17,18),
PARTITIONpCentralVALUESIN(9,10,19,20)
);
Asyoucanseeintheprecedingstatement,partitioningperzonesmeansitwillbeeasytoupdaterecordsforstoresbasedonzoneswithinparticularpartitions.Supposetheorganizationsoldthewestzonetoanothercompany;thenyoumightneedtoremoveallemployeerecordsfromthewestzoneusingthepWestpartitioninquery.ExecutingALTERTABLEemployeeTRUNCATEPARTITIONpWestwouldbemucheasierandefficient thantheDELETEstatementDELETEfromemployeewherestore_idIN(7,8,17,18);also,youcanusetheDROPstatementforemployeerecordsremoval-ALTER TABLE employee DROP PARTITION pWest. Along with the previous statementexecution you will also remove the pWest PARTITION from the table partitiondefinition,andwillneedtousetheALTERstatementagaintoaddthepWestPARTITIONandrestorethepartitiontableschemeyouhadearlier.
SimilartoRANGEpartitioning,youcanalsouseLISTpartitioningusinghashorkeyto produce composite partitioning,which is also known assubpartitioning. Youwill get to know more details on subpartitioning as a dedicated section forsubpartitioningfollows.
In LIST partitioning there is no catch-all mechanism such as MAXVALUE that cancontainallpossiblevalues.Instead,youhavetomanagetheexpectedvalueslistinthevalues_listitself,otherwisetheINSERTstatementwillresultinanerrorwherethetablehasnopartitionforvalue9,asinthefollowingexample:
CREATETABLEtpl(
cl1INT,
cl2INT
)
PARTITIONBYLIST(cl1)(
PARTITIONp0VALUESIN(1,3,4,5),
PARTITIONp1VALUESIN(2,6,7,8)
);
INSERTINTOtplVALUES(9,5);
Asyoucansee in theprecedingINSERT statement,value9 isnotpartof the listgivenduringpartitionschemaandsothereisanerror.Ifyouusemultiplevalueinsert statements, the same error can result in failure for all inserts and norecordswillbeinserted;insteadusetheIGNOREkeywordtoavoidsucherrors,asinthefollowingINSERTstatementexample:
INSERTIGNOREINTOtplVALUES(1,2),(3,4),(5,6),(7,8),(9,11);
COLUMNSpartitioning
Asthenamesuggests,thistypeofpartitioningusescolumnsthemselves.Wecanusetwoversionsofcolumnpartitioning.OneisRANGECOLUMNandtheotherisLISTCOLUMN. In addition to both RANGE COLUMN and LIST COLUMN partitioning, MySQL 8supports using non-integer types of column that can be used to define valuerangesorlistvalues.Thelistofpermitteddatatypesareasfollows:
AllcolumntypesofINT,BIGINT,MEDIUMINT,SMALLINT,andTINYINTaresupportedfortheRANGEandLISTpartitioningcolumns,butothernumericcolumntypessuchasFLOATorDECIMALarenotsupported
DATEandDATETIMEaresupportedbutothercolumntypesrelating todateandtimearenotsupportedaspartitioningcolumns
ThestringcolumntypesBINARY,VARBINARY,CHARandVARCHARaresupportedbuttheTEXTandBLOBcolumntypesarenotsupportedaspartitioningcolumns
Now,let'sseeRANGECOLUMNpartitioningandLISTCOLUMNpartitioningindetailonebyone.
<strong>CREATETABLEtable_name</strong><br/><strong>PARTITIONBYRANGECOLUMNS(column_list)(</strong><br/><strong>PARTITIONpartition_nameVALUESLESSTHAN(value_list)[,</strong><br/><strong>PARTITIONpartition_nameVALUESLESSTHAN(value_list)][,</strong><br/><strong>...]</strong><br/><strong>)</strong><br/><strong>column_list:</strong><br/><strong>column_name[,column_name][,...]</strong><br/><strong>value_list:</strong><br/><strong>value[,value][,...]</strong>
<strong>CREATETABLEtrc(</strong><br/><strong>pINT,<br/>qINT,<br/>rCHAR(3),<br/>sINT<br/>)</strong><br/><strong>PARTITIONBYRANGECOLUMNS(p,s,r)(</strong><br/><strong>PARTITIONp0VALUESLESSTHAN(5,10,'ppp'),</strong><br/><strong>PARTITIONp1VALUESLESSTHAN(10,20,'sss'),</strong><br/><strong>PARTITIONp2VALUESLESSTHAN(15,30,'rrr'),</strong><br/><strong>PARTITIONp3VALUESLESSTHAN(MAXVALUE,MAXVALUE,MAXVALUE)</strong><br/><strong>);</strong>
<strong>INSERTINTOtrcVALUES(5,9,'aaa',2),(5,10,'bbb',4),(5,12,'ccc',6);</strong>
<strong>CREATETABLEcustomer_z(</strong><br/><strong>first_nameVARCHAR(30),<br/>last_nameVARCHAR(30),<br/>street_1VARCHAR(35),<br/>street_2VARCHAR(35),<br/>cityVARCHAR(15),<br/>renewalDATE<br/>)</strong><br/><strong>PARTITIONBYLISTCOLUMNS(city)(</strong><br/><strong>PARTITIONpZone_1VALUESIN('Ahmedabad','Surat','Mumbai'),</strong><br/><strong>PARTITIONpZone_2VALUESIN('Delhi','Gurgaon','Punjab'),</strong><br/><strong>PARTITIONpZone_3VALUESIN('Kolkata','Mizoram','Hyderabad'),</strong><br/><strong>PARTITIONpZone_4VALUESIN('Bangalore','Chennai','Kochi')</strong><br/><br/><strong>);</strong>
SimilartoRANGECOLUMNpartitioning,itisnotrequiredtoprovideanyexpressionintheCOLUMNS()clausethatconvertsthecolumnvaluetoanintegerliteral,andnothingotherthanthelistofcolumnnamesitselfispermitted.
<strong>CREATETABLEemployee(</strong><br/><strong>employee_idINTNOTNULL,</strong><br/><strong>first_nameVARCHAR(30),</strong><br/><strong>last_nameVARCHAR(30),</strong><br/><strong>hired_dateDATENOTNULLDEFAULT'1990-01-01',</strong><br/><strong>termination_dateDATENOTNULLDEFAULT'9999-12-31',</strong><br/><strong>job_codeINTNOTNULL,</strong><br/><strong>store_idINTNOTNULL</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(store_id)</strong><br/><strong>PARTITIONS4;</strong>
Intheabovestatement,ifyouexcludethePARTITIONSclause,thenumberofpartitionsautomaticallydefaultstoone.
<strong>CREATETABLEemployee(</strong><br/><strong>employee_idINTNOTNULL,</strong><br/><strong>first_nameVARCHAR(30),</strong><br/><strong>last_nameVARCHAR(30),</strong><br/><strong>hired_dateDATENOTNULLDEFAULT'1990-01-01',</strong><br/><strong>termination_dateDATENOTNULLDEFAULT'9999-12-31',</strong><br/><strong>job_codeINTNOTNULL,</strong><br/><strong>store_idINTNOTNULL</strong><br/><strong>)</strong><br/><strong>PARTITIONBYLINEARHASH(YEAR(hired_date))</strong><br/><strong>PARTITIONS4;</strong>
Anadvantageofusinglinearhashingisfasterpartitioningoperations,andadisadvantageislessevendatadistributioncomparedtoregularhashingpartitioning.
<strong>CREATETABLEtk1(</strong><br/><strong>tk1_idINTNOTNULLPRIMARYKEY,<br/>noteVARCHAR(50)<br/>)<br/>PARTITIONBYKEY()<br/>PARTITIONS2;</strong>
<strong>CREATETABLEtk2(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2CHAR(10),</strong><br/><strong>cl3DATE</strong><br/><strong>)</strong><br/><strong>PARTITIONBYLINEARKEY(cl1)</strong><br/><strong>PARTITIONS3;</strong>
Asyoucanseeintheprecedingexamplestatement,similartoHASHpartitioning,KEYpartitioningalsosupportsLINEARKEYpartitioningandhasthesameeffectasLINEARHASHpartitioning.
<strong>CREATETABLEtrs(trs_idINT,soldDATE)</strong><br/><strong>PARTITIONBYRANGE(YEAR(sold))<br/>SUBPARTITIONBYHASH(TO_DAYS(sold))<br/>SUBPARTITIONS2(<br/>PARTITIONp0VALUESLESSTHAN(1991),<br/>PARTITIONp1VALUESLESSTHAN(2001),<br/>PARTITIONp2VALUESLESSTHANMAXVALUE<br/>);</strong>
Asyoucanseeintheprecedingexamplestatement,tabletrshasthreeRANGEpartitionsandeachofthepartitionsp0,p1,p2isfurtherdividedintotwomoresubpartitions.Effectively,theentiretableisdividedintosixpartitions.
SubpartitioningispossibleontablespartitionedusingRANGEorLISTpartitioning,andsubpartitioningcanusetheKEYorHASHpartitioningtypes.Thesyntaxrulesforsubpartitioningarethesameasinregularpartitioning,withtheexceptiontospecifythedefaultcolumninKEYpartitioningasitdoesnottakethecolumnautomaticallyforsubpartitioning.
Thefollowingisalistofpointstoconsiderwhenusingsubpartitioning:
NumberofpartitionsmustbesameforeachofthepartitionsdefinedNamemustbespecifiedwiththeSUBPARTITIONINGclauseorspecifyadefaultoptioninsteadNamesspecifiedforsubpartitioningmustbeuniqueacrossthetable
HandlingNULLinpartitioning
There is nothing specific toMySQL 8 that disallows NULL in partitioning as acolumn value, partitioning expression, or the value from the user-definedexpression. Even if NULL is permitted as a value ,the value returned from theexpression must be an integer and so MySQL 8 has implementation forpartitioningsuchthatittreatsNULLaslessthananynon-NULLvalueasdoneintheORDERBYclause.
BehaviorforNULLhandlingvariesamongdifferenttypesofpartitioning:
HandlingNULLinRANGEpartitioning:IfaNULLvaluecontainedinthecolumnisinserted,therowwillbeinsertedinthelowestpartitionspecifiedinrange
Handling NULL with LIST partitioning: If the table has a partitioningdefinitionwithLISTpartitioningand itspartitionsaredefinedwithavaluelistthatexplicitlyspecifiesNULLasavalueinvalue_list,theninsertionwillbesuccessful; otherwise, it will give an error table that has no partitionspecifiedforNULL
Handling NULL with HASH and KEY partitioning: NULL is handled differently
when table partitioning is defined with HASH or KEY partitioning, and if apartitionexpressionreturnsNULLitiswrappedwithzerovalue.Sothatbasedonpartitioningtheinsertionoperationwillsuccessfullyinserttherecordtopartitionbeingzero.
Partitionmanagement
There are plenty ofways to use SQL statements in order tomodify partitionedtables—youcandrop,add,merge,split,orredefinepartitionswiththeALTERTABLEstatement. There are also ways to retrieve partitioned tables and partitioninformation.
Wewillseeeachoftheseinthefollowingsections:
RANGEandLISTpartitionmanagement
HASHandKEYpartitionmanagement
Partitionmaintenance
Obtainpartitioninformation
RANGE and LIST partitionmanagementPartitionaddinganddroppingishandledinasimilarwayfortheRANGEandLISTpartition types.A tablepartitionedby RANGE or LIST partitioning canbedroppedusingtheALTERTABLEstatementwiththeDROPPARTITIONoptionavailable.
Make sureyouhave the DROP privilegebefore executing the ALTER TABLE ... DROPPARTITION statement. DROP PARTITION will delete all the data and also remove thepartitionfromthetablepartitiondefinition.
The following example illustrates the DROP PARTITION option with the ALTER TABLEstatement:
SET@@SQL_MODE='';
CREATETABLEemployee(
idINTNOTNULLAUTO_INCREMENTPRIMARYKEY,
first_nameVARCHAR(25)NOTNULL,
last_nameVARCHAR(25)NOTNULL,
store_idINTNOTNULL,
department_idINTNOTNULL
)
PARTITIONBYRANGE(id)(
PARTITIONp0VALUESLESSTHAN(5),
PARTITIONp1VALUESLESSTHAN(10),
PARTITIONp2VALUESLESSTHAN(15),
PARTITIONp3VALUESLESSTHANMAXVALUE
);
INSERTINTOemployeeVALUES
('','Chintan','Mehta',3,2),('','Bhumil','Raval',1,2),
('','Subhash','Shah',3,4),('','Siva','Stark',2,4),
('','Chintan','Gajjar',1,1),('','Mansi','Panchal',2,3),
('','Hetal','Oza',2,1),('','Parag','Patel',3,1),
('','Pooja','Shah',1,3),('','Samir','Bhatt',2,4),
('','Pritesh','Shah',1,4),('','Jaymin','Patel',3,2),
('','Ruchek','Shah',1,2),('','Chandni','Patel',3,3),
('','Mittal','Patel',2,3),('','Shailesh','Patel',2,2),
('','Krutika','Dave',3,3),('','Dinesh','Patel',3,2);
ALTERTABLEemployeeDROPPARTITIONp2;
In theprecedingstatement,afterexecuting theALTER TABLE employee DROP PARTITIONp2;statement,youcanseethatalldataisremovedfrompartitionp2.Incaseyouwant to remove all the data but also need to keep the table definition and thepartitioningscheme,youcanusetheTRUNCATEPARTITIONoptiontoachieveasimilar
result.
InordertoaddnewLISTorRANGEpartitionstoexistingpartitionedtablesyoucanusetheALTERTABLE...ADDPARTITIONstatement.
ByusingtheSHOWCREATETABLEstatementyoucanverifyandseeiftheALTERTABLEstatement has the desired effect on the table definition and the partitioningschema.
<strong>CREATETABLEclient(</strong><br/><strong>client_idINT,</strong><br/><strong>first_nameVARCHAR(25),</strong><br/><strong>last_nameVARCHAR(25),</strong><br/><strong>signedDATE</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(MONTH(signed))</strong><br/><strong>PARTITIONS12;</strong>
<strong>ALTERTABLEclientCOALESCEPARTITION8;</strong>
Intheprecedingstatementthenumber8representsthenumberofpartitionstoberemovedfromthetable.Youcannotremovemorepartitionsthanalreadyexistinthetablepartitioningschema.Similarly,youcanaddmorepartitionsusingtheALTERTABLE...ADDPARTITIONstatement.
<strong>ALTERTABLEtrpREBUILDPARTITIONp0,p1,p2;</strong>
<strong>ALTERTABLEtopOPTIMIZEPARTITIONp0,p1,p2;</strong>
<strong>ALTERTABLEtapANALYZEPARTITIONp1,p2;</strong>
<strong>ALTERTABLEtrpREPAIRPARTITIONp3;</strong>
<strong>ALTERTABLEtcpCHECKPARTITIONp0;</strong>
ThereisanoptiontouseALLinsteadofaspecificpartition,specifiedinallaboveoptions,inordertoperformtheoperationonallthepartitions.
<strong>SHOWCREATETABLEemployee;</strong>
Theoutputfromtheprecedingstatementhasseparateinformationforpartitioningschema,includingcommoninformationforthetableschema.
Similarly,youcanretrieveinformationaboutpartitioningfromtheINFORMATION_SCHEMA.PARTITIONStable.
TheEXPLAINoptiongivesalotofinformationonpartitioningwithcolumn.Forexampleitgivesnumberofrowsobtainedfromthequeryspecifictopartitions.Thepartitionwouldbesearchedasperthequerystatement.Italsogivesinformationaboutkeys.
EXPLAINisalsousedtogetinformationfromnonpartitionedtables.Itdoesnotgiveanyerroriftherearenopartitions,butgivesaNULLvalueinthepartitionscolumn.
PartitionselectionandpruningInthissection,youwillseehowpartitioningcanoptimizeSQLstatementsclauseexecution with the optimizer known as partition pruning, and the use of SQLstatements to effectively use partition data for selection and performmodificationoperationsonthepartitioning.
PartitionpruningPartitionpruningisrelatedtotheoptimizationconcept inpartition.Inpartitionpruning the concept described as Do not scan partitions where no possiblematchingvaluescanbepresentisappliedbasedonthequerystatements.
Supposethereisapartitionedtable,tp1,createdwiththefollowingstatement:
CREATETABLEtp1(
first_nameVARCHAR(30)NOTNULL,
last_nameVARCHAR(30)NOTNULL,
zone_codeTINYINTUNSIGNEDNOTNULL,
dojDATENOTNULL
)
PARTITIONBYRANGE(zone_code)(
PARTITIONp0VALUESLESSTHAN(65),
PARTITIONp1VALUESLESSTHAN(129),
PARTITIONp2VALUESLESSTHAN(193),
PARTITIONp3VALUESLESSTHANMAXVALUE
);
Intheprecedingexampletabletp1,supposeyouwanttoretrievearesultfromthefollowingSELECTstatement:
SELECTfirst_name,last_name,dojfromtp1wherezone_code>126ANDzone_code<131;
Now,youcanseefromtheprecedingstatementthattherearenorowsthathavedatainpartitionsp0orp3asperthestatement,soweonlyneedtosearchthedatainp1orp2formatchingcriteria.So,bylimitingthesearch,itispossibletospendlesstimeandeffortmatchingandsearchingforthedatathroughallthepartitionsinthetable.Thiscuttingawayoftheunmatchedpartitionsisknownaspruning.
The optimizer can make use of partition pruning for performing the queryexecution much faster compared to nonpartitioned tables that have the sameschema,data,andquerystatements.
TheoptimizercandopruninginthefollowingcasesbasedontheWHEREconditionreduction:
partition_columnIN(constant1,constant2,...,contantN)
partition_column=constant
Inthefirstcase, theoptimizerevaluates thepartitioningexpressionforeachofthe values in the list and creates a list of partitions that are matched duringevaluation,andthenscanningorsearchingisperformedonlyonthepartitionsinthispartitionlist.
In the second case, the optimizer only evaluates the partitioning expressionbased on the given constant or specific value and determines which partitioncontainsthevalue,andsearchingorscanningisperformedonlyonthispartition.Therecanbeuseofanotherarithmeticcomparisoninsteadofequalsforthistypeofcase.
Currently,pruningisnotsupportedonINSERTstatementsbutissupportedinSELECT,UPDATE,andDELETEstatements.
Pruning is also applicable to short rangeswhere theoptimizer can convert therangesintoanequivalentlistofvalues.Theoptimizercanbeappliedwhenthepartitioning expression consists of equality or range that can be reduced toequalitiessetorifanincreasingordecreasingrelationshipisrepresentedbythepartitioningexpression.
Pruning can also applicable to the column types of DATE or DATETIME if thepartitioningusestheTO_DAYS()orYEAR()function,andalsoapplicableifsuchtablesusetheTO_SECONDS()functionintheirpartitioningexpression.
Supposeyouhaveatable,tp2,asperthefollowingstatement:
CREATETABLEtp2(
first_nameVARCHAR(30)NOTNULL,
last_nameVARCHAR(30)NOTNULL,
zone_codeTINYINTUNSIGNEDNOTNULL,
dojDATENOTNULL
)
PARTITIONBYRANGE(YEAR(doj))(
PARTITIONp0VALUESLESSTHAN(1971),
PARTITIONp1VALUESLESSTHAN(1976),
PARTITIONp2VALUESLESSTHAN(1981),
PARTITIONp3VALUESLESSTHAN(1986),
PARTITIONp4VALUESLESSTHAN(1991),
PARTITIONp5VALUESLESSTHAN(1996),
PARTITIONp6VALUESLESSTHAN(2001),
PARTITIONp7VALUESLESSTHAN(2006),
PARTITIONp8VALUESLESSTHANMAXVALUE
);
Now, in the preceding statement the following statements can benefit from
partitionpruning:
SELECT*FROMtp2WHEREdoj='1982-06-24';
UPDATEtp2SETregion_code=8WHEREdojBETWEEN'1991-02-16'AND'1997-04-26';
DELETEFROMtp2WHEREdoj>='1984-06-22'ANDdoj<='1999-06-22';
Forthelaststatement,theoptimizercanactasfollows:
1. Finding thepartition thathas the lowendof therangeasYEAR('1984-06-22')givesthevalue1984,foundinthep3partition.
2. FindingthepartitionthathasthehighendoftherangeasYEAR('1999-06-22')givesthevalue1999,foundinthep5partition.
3. Scan only the above two identified partitions and any partitions that liebetweenthem.
So, in the abovementioned case thepartitions tobe scannedare p3,p4, and p5only,andtherestofthepartitionscanbeignoredwhilematching.
The preceding examples use RANGE partitioning but partition pruning is alsoapplicableonothertypesofpartitioningaswell.Supposeyouhavethetabletp3schemaasperthefollowingstatement:
CREATETABLEtp3(
first_nameVARCHAR(30)NOTNULL,
last_nameVARCHAR(30)NOTNULL,
zone_codeTINYINTUNSIGNEDNOTNULL,
descriptionVARCHAR(250),
dojDATENOTNULL
)
PARTITIONBYLIST(zone_code)(
PARTITIONp0VALUESIN(1,3),
PARTITIONp1VALUESIN(2,5,8),
PARTITIONp2VALUESIN(4,9),
PARTITIONp3VALUESIN(6,7,10)
);
Fortheprecedingtableschema,considerifthisstatementSELECT*FROMtp3WHEREzone_codeBETWEEN1AND3istobeexecuted.Theoptimizerdetermineswhichofthepartitionscanhavethevalues1,2,and3andfindsp1andp0,soitskipstherestofthepartitionsp3andp2.
Column values with a constant can be pruned, as in the following examplestatement:
UPDATEtp3setdescription='ThisisdescriptionforZone5'WHEREzone_code=5;
PartitionselectionExplicitselectionofpartitionandsubpartitionisalsosupportedandthisenablesrowmatchingtoconditionsgiveninthewhereclause-thisisknownaspartitionselection. It is very similar to partition pruning as only specific partitions arescannedformatching,butdiffersinthefollowingtwokeyaspects:
ThepartitionstobescannedarespecifiedbytheissuerofthestatementandarenotautomaticsuchaswithpartitionpruningThe partition pruning is limited to queries, whereas partition selectionsupportsbothqueriesandanumberofDMLstatements
SQLstatementssupportedforexplicitpartitionselectionarelistedasfollows:
INSERT
SELECT
UPDATE
REPLACE
LOADDATA
LOADXML
DELETE
The following syntax with the PARTITION option is used for explicit partitionselection:
PARTITION(partition_names)
partition_names:
partition_name,...
Theprecedingoptionisalwaysfollowedbythetablestructureortableschemaitbelongs to. partition_names stands for the list of comma separated names ofpartitions or subpartitions that will be used in partitioning. Partition andsubpartitionnamesinpartition_namescanbeinanyorderorevenoverlapbuteachname from the list must be the existing partition or subpartition name of thespecific table, otherwise the statement will fail with the error messagepartition_namedoesn'texist.
IfthePARTITIONoptionisused,onlylistedpartitionsandsubpartitionsarecheckedformatching rows.PARTITION option can also be used in the SELECT statement toretrieverowsbelongingtoanygivenpartition.
Supposeyouhavethetableemployeecreatedwiththefollowingstatements:
SET@@SQL_MODE='';
CREATETABLEemployee(
idINTNOTNULLAUTO_INCREMENTPRIMARYKEY,
first_nameVARCHAR(25)NOTNULL,
last_nameVARCHAR(25)NOTNULL,
store_idINTNOTNULL,
department_idINTNOTNULL
)
PARTITIONBYRANGE(id)(
PARTITIONp0VALUESLESSTHAN(5),
PARTITIONp1VALUESLESSTHAN(10),
PARTITIONp2VALUESLESSTHAN(15),
PARTITIONp3VALUESLESSTHANMAXVALUE
);
INSERTINTOemployeeVALUES
('','Chintan','Mehta',3,2),('','Bhumil','Raval',1,2),
('','Subhash','Shah',3,4),('','Siva','Stark',2,4),
('','Chintan','Gajjar',1,1),('','Mansi','Panchal',2,3),
('','Hetal','Oza',2,1),('','Parag','Patel',3,1),
('','Pooja','Shah',1,3),('','Samir','Bhatt',2,4),
('','Pritesh','Shah',1,4),('','Jaymin','Patel',3,2),
('','Ruchek','Shah',1,2),('','Chandni','Patel',3,3),
('','Mittal','Patel',2,3),('','Shailesh','Patel',2,2),
('','Krutika','Dave',3,3),('','Dinesh','Patel',3,2);
Now,ifyoucheckwithpartitionp1,youseethefollowingoutputasrowsaddedinpartitionp1:
mysql>SELECT*FROMemployeePARTITION(p1);
+----+-----------+------------+----------+---------------+
|id|last_name|last_name|store_id|department_id|
+----+-----------+------------+----------+---------------+
|5|Chintan|Gajjar|1|1|
|6|Mansi|Panchal|2|3|
|7|Hetal|Oza|2|1|
|8|Parag|Patel|3|1|
|9|Pooja|Shah|1|3|
+----+-----------+------------+----------+---------------+
5rowsinset(0.00sec)
IfyouusethisstatementSELECT*FROMemployeeWHEREidBETWEEN5AND9;,itwillgivethesameoutput.
Inordertoretrieverowsfrommultiplepartitionsyoucanuseacommaseparatedlistofpartitionnames.Forexample,SELECT * FROM employee PARTITION (p1,p2), willresult in all the rows from partitions p1 and p2 and exclude the remainingpartitions.
Any supported partitioning types can be used using partitioning selectionstatements.MySQL8automaticallyaddspartitionnameswhenatableiscreatedwiththeLINEARHASHorLINEARKEYpartitioningtypesspecifiedwithoutanynames,and this is also applicable to subpartitions aswell.While executing the SELECTstatementonthistableyoucanspecifypartitionnamesgeneratedbyMySQL8forpartitionspecificdataretrieval.
ThePARTITION option is also applicableon the SELECT statement for the INSERT...SELECTstatement,bywhichwecaninsertdataretrievedfromspecificpartitionsorsubpartitionsaswell.
ThePARTITIONoptionisalsoapplicableontheSELECTstatementwithjoinqueriesontableswithspecificpartitionorsubpartitiondata.
Restrictions and limitations inpartitioningIn this section, you will see the restrictions and limitations in MySQL 8partitioning, covering prohibited constructs, performance considerations, andlimitation aspects related to storage engines and functions in detail, to gainoptimumbenefitsfromthetablepartitioning.
<strong>CREATETABLEtk1(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>UNIQUEKEY(cl1,cl2)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl3)</strong><br/><strong>PARTITIONS4;</strong><br/><br/><strong>CREATETABLEtk2(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>UNIQUEKEY(cl1),</strong><br/><strong>UNIQUEKEY(cl3)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl1+cl3)</strong><br/><strong>PARTITIONS4;</strong>
<strong>CREATETABLEtk1(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>UNIQUEKEY(cl1,cl2,cl3)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl3)</strong><br/><strong>PARTITIONS4;</strong><br/><br/><strong>CREATETABLEtk2(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>UNIQUEKEY(cl1,cl3)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl1+cl3)</strong><br/><strong>PARTITIONS4;</strong>
<strong>CREATETABLEtk4(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2INTNOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4
INTNOTNULL,</strong><br/><strong>UNIQUEKEY(cl1,cl3),</strong><br/><strong>UNIQUEKEY(cl2,cl4)</strong><br/><strong>);</strong>
<strong>CREATETABLEtk5(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>PRIMARYKEY(cl1,cl2)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl3)</strong><br/><strong>PARTITIONS4;</strong><br/><br/><strong>CREATETABLEtk6(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>PRIMARYKEY(cl1,cl3),</strong><br/><strong>UNIQUEKEY(cl2)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(YEAR(cl2))</strong><br/><strong>PARTITIONS4;</strong>
<strong>CREATETABLEtk7(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>PRIMARYKEY(cl1,cl2)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl1+YEAR(cl2))</strong><br/><strong>PARTITIONS4;</strong><br/><br/><strong>CREATETABLEtk8(</strong><br/><strong>cl1INTNOTNULL,</strong><br/><strong>cl2DATENOTNULL,</strong><br/><strong>cl3INTNOTNULL,</strong><br/><strong>cl4INTNOTNULL,</strong><br/><strong>PRIMARYKEY(cl1,cl2,cl4),</strong><br/><strong>UNIQUEKEY(cl2,cl1)</strong><br/><strong>)</strong><br/><strong>PARTITIONBYHASH(cl1+YEAR(cl2))</strong><br/><strong>PARTITIONS4;</strong>
Ifthetabledoesnothaveauniquekeyorprimarykeythentherestrictionisnotapplicable,andanycolumnorcolumnscanbeusedinthepartitioningexpressionaspercompatiblecolumntypesforthepartitioningtype.AllaboverestrictionsarealsoapplicabletotheALTERTABLEstatementsaswell.
PartitioninglimitationsrelatingtostorageenginesPartitioning support is not provided by MySQL server but from the storageengine's
ownornativepartitioninghandlerinMySQL8.InMySQL8,theInnoDBstorageengineonlyprovidesanativepartitioninghandlerandsothepartitioned
tablecreationisnotapplicablewithanyotherstorageengine.
ALTER TABLE ... OPTIMIZE PARTITION doesnotwork correctlywith the InnoDB storageengine, so instead use the ALTER TABLE ... REBUILD PARTITION and ALTER TABLE ...
ANALYZEPARTITIONoperationsforsuchtables.
Partitioninglimitationsrelatingtofunctions
In partitioning expressions only the following listed MySQL functions areallowedinMySQL8:
ABS():Itprovidesanabsolutevalueforthegivenargument
CEILING(): It provides the smallest integer number possible for the givenargument
DAY():Itprovidesthedayofthemonthforthegivendate
DAYOFMONTH():ItprovidesthedayofthemonthforthegivendatesameasDAY()
DAYOFWEEK():Itprovidestheweekdaynumberforthegivendate
DAYOFYEAR():Itprovidesthedayoftheyearforthegivendate
DATEDIFF():Itprovidesthenumberofdaysbetweentwogivendates
EXTRACT():Itprovidespartofthegivenargument
FLOOR():Itprovidesthelargestintegervaluepossibleforthegivenargument
HOUR():Itprovidesthehourfromthegivenargument
MICROSECOND():Itprovidesthemicrosecondsfromthegivenargument
MINUTE():Itprovidestheminutefromthegivenargument
MOD(): It performs theModulo operation and provides the remainder of NdividedbyMwhereMOD(N,M)
MONTH():Itprovidesthemonthfromthegivenargument
QUARTER():Itprovidesthequarterfromthegivenargument
SECOND():Itprovidesthesecondfromthegivenargument
TIME_TO_SEC():Itprovidesthesecondfromthegiventimevalueargument
TO_DAYS():Itprovidesthenumberofdaysfromyear0forthegivenargument
TO_SECONDS():Itprovidesthenumberofsecondsfromtheyear0forthegivenargument
UNIX_TIMESTAMP()(withTIMESTAMPcolumns): Itprovides thesecondssince '1970-01-0100:00:00'UTCforthegivenargument
WEEKDAY():Itprovidestheweekdayindexforthegivenargument
YEAR():Itprovidestheyearforthegivenargument
YEARWEEK():Itprovidestheyearandweekforthegivenargument
PartitionpruningsupportstheTO_DAYS(),TO_SECONDS(),TO_YEAR(),andUNIX_TIMESTAMP()functionsinMySQL8.
Summary
Inthischapter,welearnedaboutdifferenttypesofpartitioningandtheneedforpartitions. We also covered detailed information on managing all types ofpartitions.
Welearnedaboutpartitionpruningandselectionofpartitionswhichisusedbythe optimizer. We also discussed applicable limitations and restrictions toconsiderwhileusingpartitioning.
Inthenextchapter,youwilllearnhowtodoscalinginMySQL8,anddiscovercommonchallengesfacedwhenprovidingscalabilityinMySQL8.Youwillalsolearn how to make the MySQL server highly available and achieve highavailability.
MySQL 8 – Scalability and HighAvailability
In this chapter, we will cover the following important topics for MySQL 8scalabilityandhighavailability:
OverviewofscalabilityandhighavailabilityinMySQL8
ScalingMySQL8
ChallengesinscalingMySQL8
Achievinghighavailability
Beforewemoveontothedetails,let'shaveanoverviewofscalabilityandhighavailabilityinMySQL8
Overview of scalability and highavailabilityinMySQL8
In any type of application, be it mobile, web portals, websites, social, e-commerce, enterprise or cloud applications, data is the core portion of thebusiness.Dataavailability isconsideredanutmostconcernforanybusinessororganization.Datalossoranydowntimeofanapplicationcanresultinaheavylossintermsofmoneyandalsoimpactthecreditofthecompanyinthemarket.
Ifweconsideranexampleofanonlineshoppingsitewhichhasanicelycoveredmarket in a specific area, with customers and good business credit. If thisbusiness faced an issue with data loss or any application server or databaseserver downtime, itwould impact thewhole business.Many customerswouldlosefaithinthebusinessandalsothebusinesswouldsufferalossbothintermsoffinanceandcredit.
Thereisnosingleformulathatcanprovideasolution.Differentbusinesseshavetheirownapplicationrequirements,businessneeds,distinctprocesses,differentinfrastructure in different locations, and operational competencies. In thesecircumstances,technologyplaysamajorroleinachievinghighavailability.
Aspertherequirementsofscalabilityandhighavailability,MySQLcanbeusedfor various applications, and as per need it is capable of overcoming failures,includingfailuresofMySQL,failuresfromtheoperatingsystem,oranyplannedmaintenance activity thatmay impact availability. Scalability in simple terms,that has the capability to distribute database load and application queriesbetweenMySQLservers.
Theattributesthatmatterwhenchoosingtherightsolutionforhighavailabilitydepend on to what extent the system can be called highly available, as suchrequirements vary from system to system. For smaller applications,where theuser load is not expected to be very high, setting up the replication or clusterenvironment can result in very high cost. In such cases, providing the correctconfigurationoftheMySQLcanalsobeenoughtoreduceapplicationload.
The following sections briefly describe the primary solutions supported byMySQL8
forhighavailability.
MySQLreplicationMySQL replication allows data from one server to be replicated onto themultipleMySQL
servers. MySQL replication provides master-slave design, so that one of theservers from the group acts as amasterwherewrite operations are performedfromtheapplicationand then themasterservercopies thedata to themultipleslaveservers.ReplicationisawellestablishedsolutionforhighavailabilityandisusedbythesocialgiantssuchasFacebook,Twitter,andsoon.
MySQLclusterThisisanotherpopularhighavailabilitysolutionforMySQL.Clusteringenablesdata to be replicated tomultipleMySQL serverswith automated sharing. It isdesignedforbetteravailabilityandthroughput.
OracleMySQLcloudserviceOracleMySQLcloudserviceprovidesanefficientmeanstohelpbuildasecure,cost-effective MySQL database as a service for applications used in modernworld.Itprovestobescalableandcost-efficientwithlessresourceutilizationformanagingtheservicewhencomparedtoon-premises.
MySQLwiththeSolariscluster
The sun Solaris cluster provided by the MySQL data service provides amechanism for orderly startup and shutdown, faultmonitoring, and automaticfailoveroftheMySQL
service.ThefollowingMySQLcomponentsareprotectedbythesunclusterHAfortheMySQLdataservice.
There are some further options available using third-party solutions. Eacharchitecture that is used to achieve highly available database services isdifferentiatedbythelevelsofuptimethateachoffers.Thesearchitecturescanbegroupedintothreemaincategories:
Datareplication
Clusteredandvirtualizedsystems
Geographically-replicatedclusters
Basedonthebestanswertothequestion,youcanselecttherightoptionforyour
application with optimal cost and a highly available solution. This discussiongivesusafairoverviewofMySQL8'shighavailability.
ScalingMySQL8
Scalability is theability todistribute the loadofanyapplicationqueriesacrossvariousMySQL instances.For somecases, it is unpredictable thatdata cannotexceed up to some limit or the number of users will not go out of bounds.Scalabledatabaseswouldbeapreferablesolutionsothat,atanypoint,wecanmeetunexpecteddemandsofscale.MySQLisarewardingdatabasesystemforits scalability, which can scale horizontally and vertically; in terms of data,distributionofclientqueriesacrossvariousMySQLinstancesisquitefeasible.ItisprettyeasytoaddhorsepowertotheMySQLclustertohandletheload.
The requirements for achievingHigh Availability (HA) and scalability mayvary from system to system.Each system requires a different configuration inorder to achieve these abilities. There are many questions that come to mindwhen we think about scaling in MySQL, and while we perform scalingoperationsinMySQL:
Whyisscalingrequired?
WhataretheadvantagesofscalinginMySQL?
Whatpointsneed toputacross inourmindswhenweperformscaling inMySQL?
Howwillscalingwork?
Isitsecurefordata-doesitprovidesuretyofdatasecurity?
Plusmanymore...
Let's takea real timeexample tounderstandwhyweneed scaling inMySQL.Wehave anonline e-commercewebsite that has covered a smallmarket,withlimitedusersandlimitedhitsonthewebsite,withasingledatabaseserver.Thebusiness isgrowingupnicely; theperformanceof thebusiness iscontinuouslyincreasingandtheusercountisincreasing,andwithoursingledatabaseserverall requests and performance cannot be scaled at all time. This may possiblyresult ina servercrashand thebusinessmight face loss in termsofprofit andcreditinthemarket.Toavoidsuchasituation,scalabilitywillperformamajorpart.Ifanyrequestfromacustomerfailsduetoanyreason,orifthenodegoesdown, the other node will take care of it quickly and give the appropriateresponsetothecustomer.
Scaling is required for the continuous increase in performance of databaseresponsetimeandtoimprovetheproductivityoftheproduct.Itwillhelptheendproductintermsofdatascalability,performance,andbetterresults.ClusterandreplicationarebothkeyfeaturesinMySQLthatcanbeleveragedforscaling.
ScalingusingclusterBasicclusterarchitectureisdividedintofourdifferentlayers:
ClientnodeApplicationnodeManagementnodeDatanode
Theseareshowninthefollowingimage:
ClientnodeTheclientnodeisanenduserorapplicationthatsendsarequestforanyqueryinterms of read data or write data from a different device, such as a computer,mobile,tablet,andsoon.
Applicationnode
The application node ismeant to provide the bridge between the logic of theapplication and the nodes containing the data in MySQL. Applications canaccess the data, which is stored in theMySQL cluster, by SQL, with one ormanyMySQL servers using the function of SQL. In the application we havemultipletechnologiesfromwhereweconnecttotheMySQLserver.WeconnectMySQLserverwithstandardtheMySQLconnectors,whichgivesustheabilitytoconnectwithawiderangeofaccesstechnologies.
Asanotheroption,wehaveNDBAPI;ahighperformanceinterfacethatcanbeused to control real-time user experiences and provide better throughput. InMySQLwehaveNDBAPI,whichaddsalayeradditionallytoNoSQLinterfacesthatconsistcapabilitytoaccesstheclusterdirectly.Applicationnodescanfetchdatafromallthedatanodes,sotheonlycauseoffailurecanbetheunavailabilityofapplicationservices,astheapplicationcanusealldatanodestoperformdatamanipulation.
ManagementnodeThemanagementnodeperformstheimportantroleofpublishingrelevantclusterinformationacrossthenodesinitscluster,alongwithnodemanagement.NodesformanagementworkatstartupwhenallnodeswanttojointheMySQLclusterandalsowhenreconfigurationofthesystemisrequired.Themanagementnodecan be stopped and restart all services without damaging or impacting anongoingoperation,execution,orprocessingofdataandtheapplicationnode.
DatanodeThedatanodesstores thedata.Tablesget sharedacross thedatanodes,whichalsohelpstohandleloadbalancing,replication,andhighavailabilityfailover.
Data nodes are the main nodes of aMySQL cluster solution. It provides thefollowingfunctionalityandbenefits:
Datastorageandmanagementofdisk-basedandin-memorydata
Inashared-nothingscenario,dataisstoredinatleastonereplicawithouttheuseof shared-disk space.MySQLcreate one replica of the databasewhich does asynchronous replication process. If any data node fails due to any specificreason,thereplicateddatawilltakecareofitandprovidetherespectiveoutput.It does a synchronous copy of the node so it consists of the same data as themainnodedata.
We can store the data either in memory or partially on disk based on therequirement.
Data that frequently change are suggested tobe stored in-memory. In-memorydataisroutinelycheckedwiththelocaldiskandcoordinatestoupdatethedatatotherestofthedatanodes.
Automatic and user-definedpartitioningof tablesor shardingoftablesMySQL cluster provides low latency, high throughput, scalability, and highavailability.
Thisadoptshorizontalscalingandautoshardingtoserveheavyloadread/writeoperations
throughthedifferentNoSQLqueries.AnNDBclusterisasetofdifferentnodeswhere
eachtaskisrunningonitsownprocessor.
Synchronous data replicationbetweendatanodesWhen we have data replication for the data node it follows synchronousreplication,soatanytimeallnodedatawillbein-sync.Ifanynodefailsforanyreason, the other nodes have the samedata and sowill be able to provide thedataforaquery.
So, without any downtime for data response, MySQL provides a perfectsolution.
DataretrievalandtransactionsMySQLsupportseachofthetransactionsthatcanbemapped,asitiscommittedonthemasterserverandappliedontheslaveserver.Thismethodisnotreferringtobinlog filesor therelevantposition in thebinlog file.GTID replication issolelyworking based on transactions; it becomes very easy to identify whether themasterandtheslaveserversareinsyncornot.
AutomaticfailoverIf any data node fails for any reason, the other nodes take responsibility andgivestheresponsetotherequest.Replicationofthedatabaseisveryhelpful incriticalconditionsofdowntimeorafailureinanyofthenodes.
Automatic re-synchronization forself-healingafterfailureIf any node is failed it will start automatically and again perform thesynchronizationofdatatotherestofthenodes,whichareactivenodes,andcopyallrecentdatainthenode.Inthatcaseitdoesself-healingofthefailures.
Scaling using memcached inMySQL8
In MySQL 8, using memcached is one of the ways to achieve scalability.Memcachedisasimpleandhighlyscalablesolutionforstoringdatainkeyandvalue form in cachewhenevermemory is available.Memcached is commonlyusedforquickaccessofdata.
Data stored inmemorydoesn't have I/Ooperationsperformed for fetching thedata.
As all the information is stored inmemory, the access speed for data ismuchfaster than compared to loading every time from disk and results in a betterqueryexecutiontimeonthedatabaseserver.Thispluginalsohasthefeatureofserialization,which converts binary files, code blocks, or any other objects tostringsthatcanbestored,andprovidesasimplemeanstoretrievesuchobjects.Whilespecifyingamemoryallocationitshouldnotbelargerthantheavailablephysicalmemoryoftheserver.
If you specify too large of a value then some of the memory allocated formemcached will use swap space and not physical memory. This may lead to
delays when storing and retrieving values because data is swapped to diskinsteadofstoringthedatadirectlyinmemory:
Thepreceding imagedepictsmemcachedarchitecture,whichdisplays the flowofthedatafrommemcachedtoaclientoranenduser,orarequestofthedatafromanapplication.
Thedatainmemcachednevergetsstoredinthedatabase.It'salwaysavailableinmemoryitself.Ifeitherofthememcachedserversfail,datawillbefetchedfromthedatabase, so itwillnot impact end-users fordata retrievalorhaveamajorperformance impact on the application. The only thing need to keep in mindwhile we use a memcached server is that data related to any importantinformation, for instance a financial transaction, should not be placed inmemcached.Inthatcaseifthereisafailureinmemcached,thedatamightnotberetrieved. In a memcached server data integrity is not healthy as it stores inmemory, soduring failure itwouldbegood tohavedata that is important notsaved inmemcached.When configuring amemcached server,memory size isthe key factor. If there is improper configuration, then you can expect a badsituation.
Thiswaywecanusememcached to scale theMySQLserver for an increaseddataresponsetime,andtoprovidefasterperformance.ItwillreducetheloadonMySQLserverandmultipleserversasapartofcachegroupandalsoprovidesaninterfaceformultiplelanguages.Itissuggestedtobeusedideallywhenthereareheavyreadoperations.
NoSQLAPIsMySQLcluster provides numerousways to help access thedata store.Oneofmostgeneric
wayisleveragingSQL;however,inreal-worldusecaseswecanalsodependonnative
APIs, which allow the fetching of the data from within the database withoutaffecting
performance or adding further complexity by developing an application toconvertSQL.
Scalingusingreplication
Replication is thecopyingofaMySQLdatabase.MySQLprovides replicationwith a different approach. MySQL has a feature of replication that providesscale-outsolutions,datasecurity,longdistancedatadistributionandmanymorebenefits.WehavediscussedthisatlengthinChapter8,Replication inMySQL8.The following imageexplains thebasicarchitectureof replication inMySQL:
ReplicationisoneofthebestfeaturesofMySQL.Itsimplycreatesacopyofthedatatothenewserveroranotherphysicalmachine,whichwill importthedatafrom the master server. Whenever the data is required it will populate theaccurateresults.
Itfollowsthemasterandslaveapproachforthereplication.Themasterdatabaseis the actual database of the application and the slave database is created byMySQL
inthedatabaseserverofanotherphysicalserver,whichcontainsreplicateddatafrom the master server. We can configure the slave database for the specificoperation,suchaswhenthequeryrelatestoreadingthedatafromthedatabase;wecanexecutethisontheslaveserver.Inthiscasethemasterdatawillhavelessloadthanearlier.
Supposewehavearatioofthe40%writedataqueryand60%readdataquery;inthiscaseifwehaveasingleserveritwillhandlealloperationsrelatedtotheread and write operation. But, as defined in the preceding image we havereplicated the database in two different servers and read operations are
performedontheslaveservers,sowecanmakeuseofoneoftheslaveservertoperform complex read queries. This makes it possible to generate reports fordoingdataanalysisonMySQL8,asperformingcomplexreadquerieswillnotimpacttheoverallapplicationperformance.
In standardMySQL replication the master server creates binary log files andmaintains the indexof the log files tomaintainandkeep trackof log rotation.Thebinary log files serve the recordsupdatesandare sent to the slave server.Whentheslaveserverconnectstothemasterdatabaseserveritconsidersthelastpositionithasreadinthelogfiles,afterwhichtheslaveserverthenreceivesanyupdatesthathavetakenplacesincethattime.Theslavesubsequentlyblocksandwaitsforthemastertonotifyitforfurtherupdates.
Thequestioninmindiswhydoweneedreplication?Or,whatisthepurposeofthereplication?If replicationrequiresanotherdatabaseserver,complexity,andadditionalconfiguration,itincreasesthemaintenanceandmonitoringtime.Still,wehavemanyadditionalbenefitsforbusinessanddatabaseadmin.
SingleserverdependancyInanycase,ifthemasterdatabaseserverfailswecaneasilyswitchourdatabaseconnectiontothereplicatedslaveservertoprovidestabilityincriticalsituations.This includes if there is network failure, server failure, hardware issues, andmanymorereasonsforfailure.
PerformancePerformance is the main part in the database. When we have a distributeddatabaseovermultipleserverswecanconnectdifferentapplicationstodifferentdatabaseserverstoincreaseperformance.Thisfeaturereducestheresponsetimeofthequery.
BackupandrecoveryReplicationhelpsbackup themasterdatabase. It ismoreefficient thanstoringthe database on disk. Users can store the database in the master using thereplicated database as a backup instead of digging up the backup files.Whenrequiredtorestorethedataofthemasterserverausercaneasilygetitfromtheslaveserver,withnoneedtoworkonthebackupfilesandgoaboutfindingthelastupdatesandotheroperations.
Loaddistribution
Byusing the replication loadof thedatabase,queryexecutioncanbe reduced;wecan split read andwriteoperationsover thedatabases. Ifweexecutewriteoperationsinthemasterdatabaseandreadoperationsintheslavedatabasethatwillimprovetheresponsetimeoftheapplication.Wecancreateloadbalancedenvironments inMySQL,which share the load of all requests to the databaseserver. The load balancer then further sends requests to the database that canhandleeachtransactionwithmuchbetterthroughput.
AsynchronousdatareplicationAsynchronousdata replicationmeans that data is copied fromonemachine toanother,with a resultant delay. This delay is based on networking bandwidth,resourceavailability,oratimeintervalsetbytheadministratorinconfiguration.The correct configuration and time setting provides an accurate result inresponse. It's basedon thenetwork administrator's configuration.Synchronousdata replication implies thatdata iscommitted tooneormoremachinesat thesametime.
GeographicaldatadistributionGroupreplicationmakesitpossibletocopythemaster'sdatatotheslaveserverthatresidesataremotelocationandperformthereadoperationsforaseparategroupofclientwithoutimpactingthemaster'soperations.
GTIDreplication
Global transaction identifiers (GTID) uses transaction based replication ofdatainsteadofbinarylogfilebasedreplication.
Until and unless transactions that have been operated and committed on themaster servers are present on all the slave servers, GTID will not considerreplicationinaconsistentstate.
InMySQL8 replicationcanbedoneeither in asynchronousmodeor in semi-synchronousmode. In asynchronousmode,write operations are performed onthemasterserverimmediately,whilereplicationinslavesisdoneperiodicallyaspertheconfiguration.
Inthesemi-synchronousmodeofreplication,ifsemi-synchronousconfigurationisenabledonthemasterandatleastoneslaveserver,atransactiononthemasternode waits before getting a transaction time out until the semi-synchronousenablednodeconfirms that requireddataorupdatehasbeen received.Andontime-out, the master node again looks for the semi-synchronous slave andperformsthereplication.
MySQL8providesanewreplicationmethod,GTID,whereauniqueidentifieriscreatedandconnectedwitheach transactionsavedorcommittedon themaster
server.Theuniquenessoftheseidentifiersisinallserversthatareintheserverwhere it's created, and also in the replicated servers. GTID have one to onemappingbetweenalltransactions.
The concept of the log file referencing a position within files is not requiredwhenstartinganewslavecreationorfailovertoanewMySQLmaster.Youcanuseeitherrow-basedorstatement-basedreplicationwithGTIDs.
UsingaglobaltransactionIDprimarilyprovidestwomajorbenefits:
It's easy to change a master server to connect with a slave serverduringfailover:GTIDisuniqueoutofallserversinthereplicationgroup.TheslaveserverrememberstheglobaltransactionIDofthelasteventfromtheoldmaster.Thismeansitbecomeseasytoidentifywheretoreinitializereplication on the newMySQLmaster, as the global transaction IDs areknownthroughouttheentirereplicationhierarchy.
The status of the slave provides a crash-safemethod: The slave holdscurrent position information in the mysql.gtid_slave_pos system table. If thistable is using a transactional storage engine (such as InnoDB, which is thedefault),thenfurtherupdatesaredoneinthesametransaction.
AGTIDsisauniquekeycreatedandassociatedwitheachtransaction(insertandupdateoperations)committedonthemasterserver.Thekeyisnotonlyuniquetothemasterserver,butit'suniqueacrossallserversinreplication.
ZFSreplication
TheZFSfile systemhas theability toprovisiona snapshotof theserver files,such as file system contents, transfer the snapshot to another machine, andextract thesnapshot to recreate the filesystemonadifferentserver.Userscancreateasnapshotatanytimeandcancreateasmanysnapshotsasrequired.Bycontinually creating, transferring, and restoring snapshots, it can providesynchronizationbetweenoneormoremachinesinafashionsimilartoDRBD.
We have seen all the possible ways to scale a database in MySQL, usingdifferenttechniques.
As per business need and flexibility we can perform scaling with databasebackup.
Scaling is not an easy task but its possible in MySQL 8, with the properknowledge and understanding of the requirements of the business and aconfigurationprovidedbyMySQL8.Fordatabasescaling,wemusthaveproperunderstanding of the entire workflow of the database and communicationapproach.
ChallengesinscalingMySQL8
Wehaveseenhowscalingworksandtheadvantagesandpurposeofscalingintheprevioustopic.WhenwestartworkingwithscalinginMySQL8,whattypeof challengeswillwe face, andwhat steps need to be kept inmindwhilewework towardsscaling?Wehave toaccount for ifwearedoingscalingand themasterserverfails,limitsarereached,readandwriteoperationsarenotabletohandle the requests of the application, or while re-platforming the database.Scaling is not an easy task; it needs to ensure it is able to handle increasingtransactions without any difficulty. At the time of performing the scaling weneedtokeepinmindmanypoints,suchasthewriteandreadoperationlimitsinthemasterandslaveservers.Databaseloadbalancingisoneof theapproachesthathelpreducethetrafficofthetransaction,butagainitneedsperfection,andneedstounderstandtheloadbalancingconfigurationproperly.
Thefollowingarethemajorchallengesfacedwhenweperformscaling.
Businesstypeandflexibility
This is thefirstpoint thatneeds tokept inmindwhileweperformthescaling.Business type or business behavior is the core part; if the business is an e-commerce, we already know e-commerce businesses have a number offunctionalities and very critical data about clients, such as product details,monopolyofthebusinessforoffersanddiscounts.
Themainthingiscustomerdetailsandpaymentinformationsuchascreditcarddetails,debitcarddetails,andcustomerfeedback.
Inthiscase,whilewearedoingscalinginMySQL8,allparametersneedtokeptinmind,suchasdatabasebackup,security,roles/privilegesofthedatabaseandbackwardcompatibilityofthescaling.Whiledoingscalingbyclusteringalldatanodesneedtobeonthesamepage.Iftheapplicationisdevelopedusingmultipletechnologies and we perform the scaling for each of the stack, we can havedifferentdatanodesavailable;inthiscasethedatabasesyncisoneofthemostimportantthingsthatneedtobesureinconfigurationwhilescaling.Whichtypeofdatashouldresideincachedmemoryinmemcachedandindiskshouldalsobeclearbeforewedesignscaling.
Thebehavioroftheapplicationaccessesdatafromtheshareddatanodes.Ifwe
have an e-commerce site and we perform the sharding for the same and at acertain level the data are not available for the clientwho uses the data of theothershardserverforanyreason,atthattimethecross-nodetransactionwouldberequired.It'scompletelybasedonthebusinessbehavior,anddependsonhowflexible thebusiness iswhenitcomestoacceptingchangesregardingdatabasescaling.
Understandserverworkload
For thesetupof flexibility, scale,andperformance improvementmanyoptionsand
actions are available inMySQL 8.Many people face issueswhile performingsuchactivities
becausetheydonothaveenoughunderstandingorknowledgetohandlevarioustechnology
stacks and configuration option selections that can improve scalability,performance,
security, and the flexibility of the application and deployment activity. Theseconfiguration
options includingclustering, replication, sharding,memcached, storageengine,and
manymore,which can bewell designed to handle thewholeworkload of theapplication.
Read-writeoperationlimit
Whathappenifthereadandwritelimitisreachedandthetransactionincreaseson the master database server. MySQL has limitations to the capacity; forinstance if a number of customers are visiting the site at the same time that aread-write operation is running and the server or node are not synced, then atthattimeitwillcreateconfusionormisunderstandingfortheenduser.Or,inane-commerce site, ifonecustomer ispurchasing theproduct,which is last itemleft instock,andatsametimeanothercustomersearchesfor thesameproductandit'sstillavailable,inthiscasebothoperationsarenotinsyncintermsofthereadandwriteoperationofthedatabase.
Intheend,theothercustomermightpurchasethesameproduct,whichwedon'thaveinthewarehouse.Thisimpactsinventorycalculation,andcustomershavedoubtsabouttheprocessofthepurchasecycle.Inthatcasewewouldloosethefaithofthecustomerinthebusinessandthecreditofthebusinesswouldalsobeimpacted.
Anotherapproachistohavedatabasesharding.Shardingcanbesimplystatedaspartitioning thedatabase inmultipleservers.Shardinghelps to reduce the loadon a single database or the master database. If we have databases shardinggeographically,andfordifferentcountryorregionwehavedifferentserversfor
thedatabase,wecansolvetheissueofthelimitofreadandwriteoperationsontheMySQLserver.Butagain,thetechniquewhichweusefortheshardingalsodeterminestheperformanceofthedatabase.WehavealreadylearnedaboutthisindetailinChapter9,PartitioninginMySQL8.
Maintenance
WhilewehaveperformedscalinginMySQL8,wemustknowhowtomanagemasterandslaveservers,andwhichconfigurationisrequiredwhileperformingscaling.What are the steps thatneed tobe takencareof at the timewhen theserver is in a critical stage? What steps needs to be performed at time ofsharding,clustering,orreplicationofthedatabaseserver?
Scalingispossiblebutitsnotaneasyoperation.Ifwewanttoperformscalingwe should know that the database can handle more transactions without anyissue.Weshouldknowtheappropriateconfigurationtobedonetoovercomethedefault limits on themaster server for thewrite and read operation. Once it'scompleted,weneed to perform similarly steps to configure the slave databaseserver,which should only have read operations available for the end user andshouldalwaysbeinsyncwiththemasterdatabase.
Ifwehavemultipleservers,thenthemaintenanceoftheserveralsobecomesacostlyoverhead.Alltheserverneedstobeonsamepage,configurationshouldbeinpropermanner,andthecostoftheserverwillalsoaffectthebusiness.Ifthenumberofdataconstantly increasesat that time, server spacealsoneeds tobemanagedinanappropriatemanner.
MasterserverfailureIf themasterserverfailsanddata isnotavailable to thecustomerat that time,the enduserwill get frustrated and thebusinesswill behampered in termsofcreditinmarketandinlosingthecustomeralso.Thebusinesswillhavetosufferfromtheloss.
SynchronizationWhetherweperformthescalingwithclusteringorreplication,inbothcasesweneedtosecuresynchronization.Allslaveserversshouldhavethesamedatabaseasthemasterserver.Ifwriteoperationsareperformedonthemasterserverandreadoperationsareperformedontheslaveserver,atthattimealldataneedstobesyncedup.Allresultsshouldbesame,andifanyservergoesdownatatimewhendatawasnotsynced,itwillcreateissuesregardingthelossofdata.
Databasesecurity
How do we secure the database if we have different servers and sharding isperformed?
If we have different database servers at different locations and access to thedatabaseisnotrestrictedtotheuserspecificatthattimethentheissueofadataleak isastrongpossibility.Wehave tocompletelyunderstandaccesspointsofthe data in terms of IP configuration of the database server, with appropriaterolesandprivilegesforthedatabaseuserswhoperformvariousactivities.WhichIPs have access and which IPs need to restrict the data transaction from theserver?While we are performing the cross node transaction on the database,accurate data should be available; it should not give the permission to accessrestricteddatafromtheserver.
CrossnodetransactionCross node transaction is required when we have multiple nodes after doingscalingandonenoderequirestheothernodedataasapartofinput.Forinstance,ifwehavedifferentnodesatdifferentlocationsandwehaveasingleinventoryfor all the locations at that time, one user request for any product that is notavailableinthatdatanodeatthattimewillhavetocommunicatewithotherdatanodesfortheinformationoftheproduct,basedontheuser'srequest.
Growingteamfordevelopment
While theapplicationmayhaveapositive responseand itscontinuoussuccessincreasethebusinessteam,theexpansionofthedatabaseadministratorswillalsoberequired.
WhenweperformingsharingandscalingorreplicationinMySQL8,werequireappropriateteammemberswiththeproperknowledgeandexperiencetohandlecontinuous expansion and the management of database servers. It's not onlylimited to the setup of database servers; we also need to keep an eye onmaintenanceoftheserverandkeepwatchingtheserveractivityalso.
Managechangerequest
When we have a change in any database structure and we have alreadyperformedthescalingorreplicationthanafewthingsneedtobetakencareofaspartofachangerequest,or ifweaddanewfeatureoranenhancementof thefunctionality.Thisincludesthingssuchasupdatingsharingkeys,modifyingthedata distribution with replication of the nodes, updating the queries to takereplication latency into account to avoid stale data with on-going managingshards,databalancing,andensuringthatdataisavailablewithnewupdates.
Scale-upandscale-out
Scale-updescribestheprocessofmaximizingthecapacitythatasingleMySQLnode canhandle.Theprocessof scaling-up can involveoptimally tuningyourdatabasesoftwareandchoosingtherightstorageengine,aspreviouslydiscussedin Chapter 6,MySQL 8 Storage Engines, and selecting appropriate hardware.Therearelimitsonhowfaryoucanscale-upanodeandthesearedeterminedbysome combination of data size, schema complexity, CPU cycles, systemmemory, and disk IO operations. While scale-out has been garnering muchattentionbecauseoftheneedtohandleincreasinglymassivedatasets,itisveryimportanttorememberthatthebetterwescale-up,thefewerscale-outnodesthatwewillrequireandsothelessweneedtospendonhardware.
Scale-outcanbeusedtodeliversolutionsthatcoverseveraldifferentusecases.
Someofmostcommononesaretoincreasereadcapacitybyusingreplicationortousedatabase sharding to increase totaldatabase size andoverall transactionthroughput.
All of these are the key challenges faced while scaling MySQL 8. Thesechallenges need to be considered while we are performing scaling of thedatabaseinMySQL8.
Asinglemistakecanputabusinessintoansituationwhichnoneofuswouldliketo be in. Scaling is the the better way to improve the performance of thedatabase.
Achievinghighavailability
Highavailability refers tosystems thataredurableandcanperformoperationswithoutanyhindranceonthedatathatisrequiredfortheresponseoranyrequestfrom any mobile, web portals, websites, social, e-commerce, enterprise, andcloud applications. Data availability is considered an utmost concern for anybusinessororganization.Anyissueswithdowntimemayhaveanimpactonthebusinesscredit,andinsomecasesbusinesseshavetosufferfinancialloss.
For instance, if we have an e-commerce application with a single databaseserver, if that server goes down for reasons such as hardware failure, networkissue,virus,oroperatingsystemissues,itimpactsthedataalso.Ane-commerceapplicationmay have a large number of customer hits at same time, and anyserver failures toserve theresponse touser requestswill impact theuser; theywillsearchforotheroptionsforthepurchasecommodity.
MySQL8hascapabilitiestoprovidebackendfortheapplicationtohelpachievehighavailabilityandprepareafullyscalablesystem.Anabilityofthesystemtokeeptheconnectionpersistent, incaseapartoftheinfrastructurefails,andtheabilityofthesystemtorecoverfromsuchfailuresisconsideredhighavailability.A failureof the systemcanbecausedbyeither amaintenanceactivityononepartofthesystem,suchasahardwareorsoftwareupgrade,orbythefailureof
Purposeofhighavailability
The requirements for achieving HA and scalability may vary from system tosystem.
Eachsystemrequiresadifferentconfigurationinordertoachievetheseabilities.
MySQL 8 also supports different approaches, such as the replication of dataacross multiple MySQL servers, or preparing multiple data centers based ongeographical locations and serving the client requests from the data centersclosest to the locationof the client.Such solutions canbeused to achieve thehighestuptimeofMySQL.
Today, in theworldofcompetitivemarketing,anorganization'skeypoint is tohave their system up and running. Any failure or downtime directly impactsbusiness and revenue. Hence, high availability is a factor that cannot beoverlooked.MySQLisquitereliableandhasconstantavailabilityusingclusterandreplicationconfigurations.
Cluster servers instantly handle failures andmanage the failover part to keepyoursystemavailablealmostallthetime.Ifoneservergetsdown,itwillredirecttheuser'srequesttoanothernodeandperformtherequestedoperation.
DataavailabilityDataisalwaysavailableinanysituation.Inanyapplication,dataisthecorepart,which isactually thewealthof theapplicationowner. Ifwehaveahealthcaresystem and at the time of medical check up of any patient their data is notavailable, due to server downtime or any other reason, it might block furtherprocessofthedoctorandinthiscaseimpactsthelifeofthepatient.
Securityofdata
Thefirstthingthatcomestomindissecuringdata,becausenowadaysdatahasbecomeprecious and it can impact business continuity if legal obligations arenotmet;infact,itcanbesobadthatitcanclosedownyourbusinessinnotime.MySQL is themost secureand reliabledatabasemanagement system,usedbymanywell-knownenterprisessuchasFacebook,Twitter,andWikipedia.Itreallyprovidesagoodsecuritylayerthatprotectssensitiveinformationfromintruders.MySQLgivesaccesscontrolmanagementsothatgrantingandrevokingrequiredaccessontheuser iseasy.Rolescanalsobedefinedwitha listofpermissionsthatcanbegrantedorrevokedfortheuser.
All user passwords are stored in an encrypted format using plugin-specificalgorithms.
SynchronizationofdataWhilewehaveasingledatabaseserver,ifitgoesdownforanyreasonwewouldlose the whole database, and if we have database backup available up to thecurrentday,wecanrestorethedatabasetillthatday,butallcurrenttransactionswouldalsobelostinthiscase.Thelasttransactiondatawouldnotbeavailableatthattime.
BackupofthedataDatabasebackuptill the last transactionshouldbe in theplanwhenabusinesshasanyserverbaseapplicationwhereasingledatabaseserverisperformingallthetasks.Whendoinghighavailability,includeallscenariosofthebackupandrestoreoperationinthearchitecture.
CompetitivemarketInthemarketmanycompetitorsareavailablewiththesamenatureofbusiness.In this case, if a business is having issueswith data availability to end users,customersmight not continuewith that business and insteadmove to anotherprovider.Itsanintegralpartofbusinesscontinuity.
Performance
High availability is also important in terms of the performance of the dataoperation.
Ifwehaveasingleserverandalloperationsareperformedonthatserveronly,itwillreachitslimitatsomestage,wheretheservercapacityisexhausted.So,inthatcase,ifwehavehighavailabilityarchitectureimplementeditwouldprovideameans to load the balance of a transaction and the performance of the datamanipulation operation. Replication and clustering enables for betterconcurrencyandmanagestheworkload.
UpdatesinthesystemWhile any online site or application requires updates or any new productionrelease
is planned it directly impacts the end users. If an application has only limitedusers
at that time, we canmanage the update regarding all end-users via emails ormessages
within the application before the release.But in caseswhere there are a largenumber
ofuserinasingleapplication,atthattimeitwillimpactthebusiness.Itwill
stop all users at the same time, anddue to this running transactionswould beimpacted.
Choosingthesolution
Again,we have to think about selecting the right solution for the availability.ManythingsneedtobekeptinmindwhileweplanhighavailabilityinMySQL.The requirements for achieving HA and scalability may vary from system tosystem.Eachsystemrequiresadifferentconfigurationinordertoachievetheseabilities.
SuchsolutionscanbeusedtoachievethehighestuptimeofMySQLwithregardtothefollowing:
Thelevelofavailabilityrequired
Thetypeofapplicationbeingdeployed
Acceptedbestpracticeswithinyourownenvironment
InMySQL, replication and clustering are the best options for achieving highavailability.
Allapplicationshavetheirownarchitecture,andnatureoftheirbusinessneeds
Advantagesofhighavailability
The following are the advantages that we have when we perform highavailabilityinMySQL:
MySQL is quite reliable and has constant availability using cluster andreplicationconfigurations.
Cluster servers instantly handle failures and manage the failover part tokeepyoursystemavailablealmostallthetime.Ifoneservergoesdown,itwill redirect theuser's request to anothernode andperform the requestedoperation.
Anabilityofthesystemtokeeptheconnectionpersistent,incaseapartofinfrastructure fails, and the ability of the system to recover from suchfailureisconsideredashighavailability.
MySQL 8 also supports different approaches such as replication of dataacrossmultipleMySQLserversorpreparingmultipledatacentersbasedongeographicallocationsandservingtheclientrequestsfromthedatacentersclosesttothelocationoftheclient.
MySQLgiveshighspeedtransactionprocessingwithoptimalspeed.Itcancachetheresults,whichboostsreadperformance.
Replication and clustering enables better concurrency and manages theworkload.Groupreplicationbasicallytakescareofcommittingtransactionsonce most of the members in group replication have acknowledged thetransactionhasbeenreceivedconcurrently.
Thishelpscreatebetterthroughputiftheoverallnumberofwritesdoesnotexceedthecapacityofthemembersinthegroupreplication.
Clustering enables data to be replicated tomultipleMySQL serverswithautomatedsharing.Itisdesignedforbetteravailabilityandthroughput.
Memcached removes the SQL layer and directly accesses the InnoDBdatabasetables.
Hence,overheadoperations likeSQLparsingwill no longerbe executed,whichreallyimpactstheperformance.
MemcachedwithMySQLalsoprovidesyouwithawaytomakein-memorydatapersistentsothatwecanuseitforvariousdatatypeswithoutlosingit.
MemcachedAPIsareavailableindifferentprogramminglanguagessuchasPerl,Python,Java,PHP,C,andRuby.WiththehelpofaMemcachedAPI,an application can interact with the Memcached interface to store andretrieveinformation.
Summary
Inthischapter,westartedwithanoverviewofscalabilityandhighavailabilityinMySQL 8, which covered the various scalability needs, advantages, methods,andkeypointstobenotedwhenwemakescalabledesignsofMySQL8.Wealsodiscussed the shortcomings that we generally come across when we performscalabilityandhowtoovercomechallengeswithappropriatesolutions.Wehavelearned about scaling in MySQL 8 and troubleshooting challenges in scalingMySQL8.WealsolearnedaboutmanydiversewaystoachievehighavailabilityinMySQL8.
In thefollowingchapter,wewill learnhowto takecareofMySQL8security.We will learn about general factors that affect security, the security of coreMySQL8files,accesscontrol,andsecuringthedatabasesystemitself.Wewillalso learn thedetailsofsecuritypluginsandgainan in-depthunderstandingofdatabasesecurityingeneralforrelationaldatabases.
MySQL8–Security
Inpreviouschapters,welearnedabout thescalabilityofMySQL8andhowtotroubleshoot challenges when scaling MySQL 8. Apart from that, we alsolearnedhowtomakeMySQL
8highlyavailableforuse.Nowadays,securityisimportantforanyapplication,right?
Whenwetalkaboutsecurity,itincludesaccountmanagement,roles,privileges,andmore. Considering these aspects,wewill cover all of these topics in thischapter.
This chapter mainly focuses on MySQL 8 database security and its relatedfeatures.
Thefollowingtopicsarecoveredinthischapter:
OverviewofsecurityforMySQL8
Commonsecurityissues
OverviewofsecurityforMySQL8
The term security is not bound to a specific topic; it covers a wide range oftopics related to MySQL 8. Before starting a detailed discussion on it, let'smentionsomeimportantpointsrelatedtosecurity:
Considersecuritywithinadatabasewhereusersandtheirprivilegesrelatedtovariousdatabaseobjectsneedtomanage.
Passwordsecurityforusers.
Security configuration during the installation process, which includesvarious typesof files, suchas log files,data files, andmanymore.Thesefilesmustbeprotectedfortheirread/writeoperations.
To handle system level failure scenarios, you must have a backup andrecoveryplan.
This includes all the required files, such as database files, configurationfiles,andmanymore.
Manage network security of the system where MySQL 8 was installed,whichpermitsalimitednumberofhostsfortheconnection.
Nowyourridebeginswithanotherimportantandveryinterestingtopic.Herewego.
CommonsecurityissuesBeforegoingintodetailoncomplexissues,youmustfirstunderstandsomebasicpointsthatwillhelpyoupreventmisuseorattacks.
Guidelines for a securepasswordIn this section,we describe guidelines for securing passwordswith respect todifferentusersandcoverhowtomanageitduringtheloggingprocess.MySQL8provides the validate_password plugin to define the policy for acceptablepasswords.
<strong>cmd>mysql-uroot--password=your_pwd</strong><br/><strong>--OR</strong><br/><strong>cmd></strong>
<strong>cmd>mysql-uroot-p</strong><br/><strong>Enterpassword:*********</strong>
<strong>[client]</strong><br/><strong>password=your_pass</strong>
<strong>shell>chmod600.my.cnf</strong>
Guidelinesforadministrators
For a database administrator, the following guidelines should be followed tosecurepasswords:
Usevalidate_passwordtoapplythepolicyonacceptedpasswords
MySQL8usesthemysql.usertabletostoreuserpasswords,soconfigurethesysteminawaythatonlyadministrativeuserscanaccessthistable
Usersshouldbeallowedtoresetaccountpasswordsinthecaseofexpiredpasswords
Applyprotectiononthelogfileifitcontainspasswords
Manage access to the plugin directory and the my.cnf file, because it canmodifycapabilitiesprovidedbytheplugins
PasswordandloggingMySQL8allowsyoutowritepasswordsasplaintextinSQLstatements,suchasCREATEUSER,SETPASSWORD,andGRANT.Ifweexecutethesestatements,MySQL8willwritepasswordsastextinlogfiles,andtheyarevisibletoalltheusersthathaveaccesstothelogfiles.Toovercomethisproblem,avoiddirectupdatesongranttablesusingthementionedSQLstatements.
Secure MYSQL 8 againstattackers
TosecureMySQL8againstattackers,stronglyconsiderthefollowingpoints:
SetapasswordforallMySQLaccounts.Neverdefineanaccountwithnopassword,becausethispermitsaccesstoyouraccountbyanyuser.
TomakeaconnectionwithMySQL8,usesecureprotocols/channels,suchascompressedprotocols,MySQL8 internalSSLconnections,orSSHforencryptedTCP/IPconnections.
ForaUnix-basedsystem,setread/writeprivilegesonthedatadirectoryfortheUnixaccountwhichisusedforrunningmysqld.Don'tusetherootusertostarttheMySQL8server.
Use thesecure_file_privvariable tospecify thedirectoryfor readandwritepermission. Using this variable, you can restrict non-administrative usersfrom accessing important directories.Use this variable to set permissionson plugin_dir; it is very important. In the same way, do not provide FILE
privilegestoalltheusers,becauseitpermitsuserstowritefilesanywhereinthesystem.
Usethemax_user_connectionsvariabletorestrictthenumberofconnectionsperaccount.
At the time of creating grant table entries, use wildcards properly. It ispreferabletouseIPsinsteadofDNS.
Followsecurityguidelinesduringstoredprogramandviewcreation.
Security options and variablesprovidedbyMySQL8
The following mysqld options and variables are provided by MySQL 8 forsecurity:
Name
Cmd-Line
OptionFile
SystemVar
StatusVar
VarScope
allow-suspicious-udfs Yes Yesautomatic_sp_privileges Yes Globalchroot Yes Yesdes-key-file Yes Yeslocal_infile Yes Global
old_passwords Yes Bothsafe-user-create Yes Yessecure-auth Yes Yes Global-Variable:secure_auth Yes Globalsecure-file-priv Yes Yes Global-Variable:
secure_file_priv Yes Global
skip-grant-tables Yes Yesskip-name-resolve Yes Yes Global-Variable:
skip_name_resolve Yes Global
skip-networking Yes Yes Global-Variable:
skip_networking Yes Global
skip-show-database Yes Yes Global-Variable:
skip_show_database Yes Global
Reference:https://dev.mysql.com/doc/refman/8.0/en/security-options.html
Security guidelines for clientprogramming
Don't trust any data entered by the application user, because there is thepossibility that the user has entered a drop or delete statement for theMySQLdatabase. So, there is always the risk of security leaks and data loss. As anadministratorofaMySQLdatabase,thefollowingchecklistshouldbefollowed:
ThesizeofthedatamustbecheckedbeforepassingittoMySQL8.
TomakeMySQL8morerestrictive,enablethestrictMySQLmode.
Fornumericfields,entercharacters,specialcharacters,andspaces insteadof numeric itself. Change field values to their original forms by yourapplicationbeforesendingthemtotheMySQL8server.
Use twodifferentusers forapplicationconnection to thedatabaseand fordatabaseadministration.
Modifydatatypesfromnumerictocharactertypesbyaddingquotesinthe
case of dynamicURLs andweb forms.Also add%22 ("),%23 (#), and%27(')indynamicURLs.
Previouslydefinedfunctionalitiesareavailablebuiltintoalloftheprogramminginterfaces. For example, Java JDBC provides prepared statements withplaceholders,andRubyDBIprovidesthequote()method.
AccesscontrolinMySQL8
Privilegesaremainlyusedtoauthenticateusersandwillverifyusercredentialsand check if a user is allowed for the requested operation or not. When weconnectwiththeMySQL8server,itwillfirstchecktheidentityoftheuserbytheprovidedhostandusername.Afterconnection,whenarequestcomesin,thesystem will grant privileges according to the user's identity. Based on thisunderstanding,wecansaythataccesscontrolcontainstwostageswhenwetrytoconnectwiththeMySQL8serverusingtheclientprogram:
Stage 1: The MySQL server will either accept or reject the connection,basedontheprovidedidentity
Stage2:AftergettingaconnectionfromtheMySQLserver,whentheusersendsarequestforperforminganyoperation,theserverwillcheckwhethersufficientprivilegesareavailablefortheuserornot
TherearesomelimitationsoftheMySQL8privilegesystem:
Userisnotallowedtosetapasswordonspecificobjects,suchasatableoraroutine.
MySQL8allowsitgloballyattheaccountlevel.
As an adminuser,we cannot specifyprivileges in away that create/droptableisallowedbutcreate/dropdatabaseofthattableisnotallowed.
Youarenotallowedtorestrictuseraccessexplicitly,whichmeansthatexplicitlymatchingtheuserandrefusingitsconnectionisnotpossible.MySQL8managesthecontentofgranttablesinmemory,sointhecaseofINSERT,UPDATE,andDELETEstatements,executionongranttablesrequirestheservertorestartforeffect.
To avoid server restarts, MySQL has provided a command for flushingprivileges.Wecanexecutethiscommandinthreedifferentways:
1. ByissuingFLUSHPRIVILEGES.
2. Usingmysqladminreload.
3. Usingmysqladminflush-privileges.
Whenwereloadgranttables,itwillworkasperthefollowingmentionedpoints:
Tableandcolumnprivileges:Changesoftheseprivilegeswillbeavailablefromthenextclient'srequest
Databaseprivileges:ChangesoftheseprivilegeswillbeavailablethenexttimetheclientexecutesaUSEdbnamestatement
Global privileges and passwords: Changes of these privileges areunaffectedforaconnectedclient;itwillbeapplicablefromthesubsequentconnections
PrivilegesprovidedbyMySQL8
Privilegesdefinewhichoperations arepermissible to theuser accounts.Basedonthelevelofoperationandthecontextinwhichitisapplied,itwillwork.Itismainlyclassifiedasfollows:
Database privileges: Applied on the database, and all objects of thedatabasewithinit.Itcanbegrantedtoasingledatabaseordefinedgloballytoapplyonalldatabases.
Administrativeprivileges:Itisdefinedatthegloballevel,sonotrestrictedtoasingledatabase.ItenablesuserstomanageoperationoftheMySQL8server.
Databaseobject'sprivileges:Itisusedtodefineprivilegesonthedatabaseobjects,suchastables,views,indexes,andstoredroutines.Itcanbeappliedonaspecificobjectofthedatabase,canbeappliedonallobjectsofagiventypeinadatabase,orcanbeappliedgloballyforalltheobjectsofagiventypeinalldatabases.
MySQL8willstoreaccountprivilegerelatedinformationintogrant tablesand
store the contents of these tables intomemory upon server start-up for betterperformance.
Privilegesarefurtherclassifiedintermsofstaticanddynamicprivileges:
Staticprivileges:Theseareavailablebuiltinwiththeserverandcannotbeunregistered. These privileges are always available for the user to begranted.
Dynamicprivileges:Theseprivilegescanberegisteredorunregisteredatruntime. If privileges are not registered, then they are not available to begrantedforuseraccounts.
Granttables
Granttablescontaininformationrelatedtouseraccountsandgrantedprivilegesfor that user. MySQL 8 automatically inserts data into these tables when weexecuteanyaccountmanagementstatementsinthedatabase,suchasCREATEUSER,GRANT,andREVOKE.MySQLallowsinsert,update,ordeleteoptionsongranttablestotheadminuser,butit'snotapreferableapproach.ThefollowingtablesoftheMySQL8databasecontaingrantinformation:
user:Itcontainsdetailsrelatedtouseraccounts,globalprivileges,andothernon-privilegecolumns
password_history:Itcontainshistoryofpasswordchanges
columns_priv:Itcontainscolumnlevelprivileges
procs_priv:Itcontainsprivilegesrelatedtostoredproceduresandfunctions
proxies_priv:Itcontainsprivilegesforproxyusers
tables_priv:Itcontainstablelevelprivileges
global_grants: It contains details related to dynamic global privilegesassignments
role_edges:Itcontainsedgesforrolesubgraphs
db:Itcontainsprivilegesatthedatabaselevel
default_roles:Itcontainsdetailsrelatedtodefaultuserroles
Granttablescontainscopeandprivilegecolumns:
Scopecolumn:Thiscolumndefinesthescopeofrowsinthetables,whichmeansthecontextunderwhichtherowapplies.
Privilegecolumn: This column indicateswhich operation is permitted totheuser.TheMySQLservercombines informationfromthevariousgranttablestobuildacompletedetailofauser'sprivileges.
FromMySQL8.0onward,granttablesusetheInnoDBstorageenginebymanagingtransactionalstates,butbeforethat,MySQLusedtheMyISAMenginebymanagingnontransactional states. This change enables users to manage all accountmanagement statements in the transactional mode, so in the case of multiplestatements, either all of them are successfully executed or none of them areexecuted.
Stage1-Connectionverification
This is the connection verification stage, so after verification,MySQL 8 willeither accept or reject your connection request.Verificationwill be performedwiththefollowingconditions:
1. Basedonauser'sidentity,withitspassword.
2. Whetherauser'saccountislockedornot.
The server will deny access if either of these cases fails. Here, the identitycontainstheusernameandhostnamefromwhichtherequestiscoming.MySQLperforms a locking check on the account_locked column of the user table and acredential check on the three columns of the user table scope: Host, User, andauthentication_string.
Stage2-RequestverificationOnceaconnectionisestablishedwiththeMySQLserver,stage2comesintothepicture,wheretheMySQLservercheckswhichoperationyouwant toperformandwhetherthatispermissibletoyouornot.Forthisverification,MySQLusesthe privilege columns of the grant tables; it might be coming from user, db,tables_priv,columns_priv,orprocs_privtables.
AccountmanagementinMySQL8As the name implies, this topic describes how to manage user accounts inMySQL8.Wewilldescribehowtoaddnewaccounts,howtoremoveaccounts,howtodefineusernamesandpasswordsfortheaccounts,andmore.
<strong>#1mysql>CREATEUSER'user1'@'localhost'IDENTIFIEDBY'user1_password';</strong><br/><strong>#2mysql>GRANTALLPRIVILEGESON*.*TO'user1'@'localhost'WITHGRANTOPTION;</strong><br/><br/><strong>#3mysql>CREATEUSER'user2'@'%'IDENTIFIEDBY'user2_password';</strong><br/><strong>#4mysql>GRANTALLPRIVILEGESON*.*TO'user2'@'%'WITHGRANTOPTION;</strong><br/><br/><strong>#5mysql>CREATEUSER'adminuser'@'localhost'IDENTIFIEDBY'password';</strong><br/><strong>#6mysql>GRANTRELOAD,PROCESSON*.*TO'adminuser'@'localhost';</strong><br/><br/><strong>#7mysql>CREATEUSER'tempuser'@'localhost';</strong><br/><br/><strong>#8mysql>CREATEUSER'user4'@'host4.mycompany.com'IDENTIFIEDBY'password';</strong><br/><strong>#9mysql>GRANTSELECT,INSERT,UPDATE,DELETE,CREATE,DROPONdb1.*TO'user4'@'host4.mycompany.com';</strong>
<strong>mysql>DROPUSER'user1'@'localhost';</strong>
Thiscommandwilldropthe'user1'accountfromthesystem.
SecurityusingrolesThesameasauseraccountrolehavingprivileges,wecanalsosaythataroleisacollectionofprivileges.Asanadminuser,wecangrantand revokeprivilegesfrom the roles. MySQL 8 provides the following commands, functions, andvariablesrelatedtoroleconfiguration.
SETROLESETROLEchangestheactiveroleswithinthecurrentsession.Refertothefollowingcommands related to SET ROLE: mysql> SET ROLE NONE; SELECTCURRENT_ROLE();+----------------+|CURRENT_ROLE()|+----------------+|NONE|+----------------+mysql>SETROLE'developer_read';SELECTCURRENT_ROLE();+----------------+|CURRENT_ROLE()|+----------------+|`developer_read`@`%`|+----------------+
The first commandwilldeactivate all roles for theuser in the current session.YoucanseetheeffectwiththeCURRENT_ROLE();function.Inthesecondcommand,wearesetting the'developer_read' roleasdefault,and thenchecking thecurrentroleusingthepredefinedfunctionagain.
CREATEROLECREATEROLE isused tocreatea role; refer to thefollowingcommand,whichwillcreatearolewiththename'developer_role':
CREATEROLE'developer_role';
DROPROLEDROPROLE isusedtoremovearole.Refertothefollowingcommand,whichwillremovethe'developer_role'role:
DROPROLE'developer_role';
<strong>GRANTALLONmy_db.*TO'developer_role';</strong>
<strong>GRANT'developer_role'TO'developer1'@'localhost';</strong>
<strong>CREATEUSER'user1';</strong><br/><strong>CREATEROLE'role1';</strong><br/><strong>GRANTSELECTONmydb.*TO'user1';</strong><br/><strong>GRANTSELECTONmydb.*TO'role1';</strong><br/><strong>CREATEUSER'user2';</strong><br/><strong>CREATEROLE'role2';</strong><br/><strong>GRANT'user1','role1'TO'user2';</strong><br/><strong>GRANT'user1','role1'TO'role2';</strong>
Inthisexample,user1androle1iscreatedandGRANTisappliedontheminasimplewaybyusingtheGRANTcommand.Now,foruser2androle2,wehaveappliedGRANTfromtheuser1androle1,respectively.
<strong>REVOKEdeveloper_roleFROMuser1;</strong><br/><strong>REVOKEINSERT,UPDATEONapp_db.*FROM'role1';</strong>
Thefirstcommandisusedtoremove'developer_role'foruser1,andthesecondcommandisusedtoremoveinsertandupdateprivilegesfromthe'role1'on'app_db'.
<strong>mysql>SETDEFAULTROLEapp_developerTOroot@localhost;</strong><br/><br/><strong>mysql>SELECTCURRENT_ROLE();</strong><br/><strong>+---------------------+</strong><br/><strong>|CURRENT_ROLE()|</strong><br/><strong>+---------------------+</strong><br/><strong>|`app_developer`@`%`|</strong><br/><strong>+---------------------+</strong><br/><strong>1rowinset(0.04sec)</strong>
Aftersettingthedefaultrole,restarttheserverandexecutethecurrent_role()functiontocheckwhetheraroleisassignedornot.
<strong>mysql>showgrantsforapp_developer;</strong><br/><strong>+-------------------------------------------+</strong><br/><strong>|Grantsforapp_developer@%|</strong><br/><strong>+-------------------------------------------+</strong><br/><strong>|GRANTUSAGEON*.*TO`app_developer`@`%`|</strong><br/><strong>+-------------------------------------------+</strong><br/><strong>1rowinset(0.05sec)</strong>
<strong>mysql>showgrantsforroot@localhost;</strong>
<strong>[mysqld]</strong><br/><strong>mandatory_roles='app_developer'</strong>
<strong>SETPERSISTmandatory_roles='app_developer';</strong>
ThisstatementapplieschangesontherunningMySQL8instanceandalsosavesitforsubsequentrestarts.Ifyouwanttoapplychangesfortherunninginstanceandnotforotherrestarts,thenusethekeywordGLOBALinsteadofPERSIST.
<strong>ALTERUSER'testuser'@'localhost'PASSWORDEXPIRE;</strong>
Thiswillmarkapasswordasexpiredforthementioneduser.Forpasswordpolicies,youhavetodefinethedurationintermsofthenumberofdays.MySQLusesthesystemvariabledefault_password_lifetime,whichcontainsapositiveintegernumbertodefinethenumberofdays.Wecandefineitinthemy.cnffileorcandefineitatruntimeusingthePERSISToption:
Passwordreuserestrictions:Usedtopreventtheuseofoldpasswordsagain.MySQL8definesthisrestrictionbasedontwoparameters-thenumberofchangesandtimeelapsed;theycanbeusedseparatelyorincombination.MySQL8definespassword_historyandpassword_reuse_intervalsystemvariables,respectively,toapplyrestrictions.Wecandefinethesevariablesinthemy.cnffile,orcanpersistthem.password_history:Thisvariableindicatesthatnewpasswordscannotbeset/duplicatedfromtheoldpasswords.Here,considerthemostrecentoldpasswordsasperthespecifiednumber.password_reuse_interval:Thisvariableindicatesthatthepasswordcannotbeset/duplicatedfromtheoldpassword.Here,intervaldefinesthespecificperiodandMySQL8willchecknewpasswordwithallthepasswordswhichwerefallsunderthatperiodforauser.Forexample,iftheintervalissetas20days,thenthenewpasswordshouldnothaveexistedinthelast20daysofchangeddata.Passwordstrengthassessment:Usedtodefinestrongpasswords.Itisimplementedusingthevalidate_passwordplugin.
EncryptioninMySQL8
When there is a need to transfer data over the network, it is a must to useencryptionfortheconnection.Ifyouareusingunencrypteddata,thensomeonewhohasaccess to thenetworkcaneasilywatchallofyour trafficandcanseewhatdataistransferredbetweentheclientandserver.Toprotectyourdataoverthenetwork,useencryption.
Make sure the encryption algorithmusedcontains security elements toprotectyour connection fromknown attacks, like changing the order of amessage orreplay twiceondata.Basedonyour application requirements, youcanchooseeither an encrypted or unencrypted type connection. MySQL 8 performsencryptionperconnectionusingTransportLayerSecurity(TLS)protocol.
Configuring MySQL 8 to useencryptedconnectionsThissectiondescribeshowtoconfigure theserverandclientfor theencryptedconnection.
<strong>[mysqld]</strong><br/><strong>ssl-ca=ca.pem</strong><br/><strong>ssl-cert=server-cert.pem</strong><br/><strong>ssl-key=server-key.pem</strong>
The--ssloptionisenabledbydefault,soonserverstartup,MySQL8willtrytofindthecertificateandkeyfileunderthedatadirectory,eventhoughyouhavenotdefineditinthemy.cnffile.Ifthosefilesarefound,thenMySQL8willprovideanencryptedconnection,orelsecontinuewithoutanencryptedconnection.
Client-side configuration forencryptedconnections
Attheclientside,MySQLusesthesame–ssloptionsusedattheserversidetospecify thecertificateandkey file,butapart fromthat, ithas–ssl-modeoptions.By default, the client is allowed to set up an encrypted connection with theserver if the server permits it. For further control, the client programuses thefollowing–ssl-modeoptions:
--ssl-mode=REQUIRED:Thisoptionindicates thatanencryptedconnectionmustbeestablished,andfailsifitisnotestablished
--ssl-mode=PREFFERED:Thisoptionindicatestheclientprogramcanestablishanencrypted connection if the server permits it, or else establish anunencryptedconnectionwithoutafail
--ssl-mode=DISABLED:Thisoptionindicatestheclientprogramisunabletouseanencryptedconnection,andonlyanunencryptedconnectionisallowed
--ssl-mode=VERIFY_CA: This option is the same as REQUIRED, but in addition to
that,itverifiestheCAcertificateagainsttheconfiguredCAcertificateandreturnsafailifnomatchesarefound
--ssl-mode=VERIFY_IDENTITY:ItisthesameastheVERIFY_CAoption,butinadditiontothat,itwillperformthehostnameidentity
Command options for encryptedconnections
The followingoptions are available inMySQL8 for anencryptedconnection.Youcanuse theseoptionson thecommand line,oryoucandefine theminanoptionfile:
Format Description--skip-ssl Donotuseencryptedconnection--ssl Enableencryptedconnection--ssl-ca FilethatcontainsalistoftrustedSSLCertificateAuthorities
--ssl-
capath
DirectorythatcontainstrustedSSLCertificateAuthoritycertificatefiles
--ssl-cert FilethatcontainsX509certificate--ssl-
cipher Listofpermittedciphersforconnectionencryption
--ssl-crl Filethatcontainscertificaterevocationlists
--ssl-
crlpath Directorythatcontainscertificaterevocationlistfiles
--ssl-key FilethatcontainsX509key--ssl-mode Securitystateofconnectiontoserver--tls-
version Protocolspermittedforencryptedconnections
Reference:https://dev.mysql.com/doc/refman/8.0/en/encrypted-connection-options.html
Connect with MySQL 8 remotelyfromWindowswithSSH
ToconnectremotelywithMYSQL8byusingSSHfromtheMicrosoftWindowssystem,performthefollowingsteps:
1. InstalltheSSHclientonyourlocalsystem.
2. Afterstarting theSSHclient, set thehostnameanduser IDbywhichyouwanttoconnectwiththeserver.
3. Configureportforwardingasfollowsandsavetheinformation:
For remote forwarding configure: local_port:3306,remote_host:mysqlservername_or_ip,remote_port:3306
For local forwarding configure: local_port:3306, remote_host:localhost,remote_port:3306
4. LogintotheserverwiththecreatedSSHsession.
5. In your localMicrosoftWindows machine, start any ODBC application,suchasMicrosoftAccess.
6. In your local system, create new file and try to linkwithMySQL serverusing the ODBC driver. Make sure you have defined localhost in theconnectioninsteadofmysqlservername.
SecuritypluginsMySQL8providesseveralpluginstoimplementsecurity.Thesepluginsprovidevariousfeaturesrelatedtoauthenticationprotocols,passwordvalidation,securestorage,andmuchmore.Let'sdiscussthevarioustypesofpluginsindetail.
<strong>CREATEUSER'testsha256user'@'localhost'</strong><br/><strong>IDENTIFIEDWITHsha256_passwordBY'userpassword';</strong>
SHA-2pluggableauthentication
SHA-2pluggableauthentication is thesameas theSHA-256pluggableplugin,except its plugin name is caching_sha2_password.When compare to sha256_password,thispluginhasthefollowingadvantages:
1. If you are using Unix socket-file and shared-memory protocols, thensupportisprovidedforclientconnections.
2. In-memorycachingisavailableinSHA-2plugins,whichprovidesfasterre-authenticationforuserswhohaveconnectedpreviously.
3. This plugin provides RSA-based password exchange, which worksregardlessoftheSSL
libraryprovidedbyMySQL8.
Client-side cleartext pluggableauthenticationThis plugin is used to send passwords to the server without hashing orencryption. It is available at the client side with the name mysql_clear_password.MySQL8providesitbuilt-inwithintheclientlibrary.
<strong>[mysqld]</strong><br/><strong>plugin-load-add=mysql_no_login.so</strong>
<strong>INSTALLPLUGINmysql_no_loginSONAME'mysql_no_login.so';</strong>
<strong>UNINSTALLPLUGINmysql_no_login;</strong>
<strong>[mysqld]</strong><br/><strong>plugin-load-add=auth_socket.so</strong>
<strong>INSTALLPLUGINauth_socketSONAME'auth_socket.so';</strong>
<strong>UNINSTALLPLUGINauth_socket;</strong>
TestpluggableauthenticationA test plugin is provided byMySQL 8 to check account credentials and logsuccessesorfailuresontheserverlogs.Itisnotabuilt-inpluginandneedstobeinstalledbeforeuse.Thisisavailableforboththeserverandclientside,andisnamed test_plugin_server and auth_test_plugin, respectively. MySQL 8 uses theauth_test_plugin.so library for this plugin. To install and uninstall this plugin,performthesamestepsmentionedintheprecedingplugin.
Theconnection-controlpluginsMySQL 8 uses these plugins to introduce an increasing delay in the serverresponsetotheclientaftersomespecificnumberoffailedconnectionattempts.MySQLhasprovidedtwopluginsforconnectioncontrol.
<strong>CONNECTION_CONTROL_FAILED_LOGIN_ATTEMPTS</strong>
ThispluginimplementsusesoftheINFORMATION_SCHEMAtabletoprovidedetailsonmonitoringofthefailedconnections.
<strong>[mysqld]</strong><br/><strong>plugin-load-add=connection_control.so</strong>
<strong>INSTALLPLUGINCONNECTION_CONTROLSONAME<br/>'connection_control.so';</strong><br/><strong>INSTALLPLUGINCONNECTION_CONTROL_FAILED_LOGIN_ATTEMPTSSONAME<br/>'connection_control.so';</strong>
Variables related toCONNECTION-CONTROL
ThefollowingvariablesareprovidedbyCONNECTION-CONTROLplugins:
Connection_control_delay_generated: This is a status variable, mainly used tomanagethecounter.Itindicateshowmanytimestheserveraddedadelayinits response on failed connection attempts. It also depends on theconnection_control_failed_connections_threshold system variable, because thisstatusvariabledoesnotincrementthecountunlessthenumberofattemptsreachesthelimitdefinedbythethresholdvariable.
connection_control_failed_connections_threshold:Thisisasystemvariablewhichindicates how many consecutive failed attempts are allowed to clientsbeforetheserveraddsadelayoneachattempt.
connection_control_max_connection_delay:Thisisasystemvariablewhichdefinesthemaximumdelay time inmillisecondsfor theserver responseonfailedconnections attempts. MySQL 8 will consider this variable once thethresholdvariablecontainsavaluehigherthanzero.
connection_control_min_connection_delay: This system variable defines theminimum delay time in milliseconds for the server to failed connectionattempt.MySQL8will consider this variable once the threshold variablecontainsavaluehigherthanzero.
Thepasswordvalidationplugin
Forpasswordvalidation,MySQLprovidesapluginnamedvalidate_password.Itismainlyusedtotestpasswordsandimprovesecurity.Thefollowingarethetwomajorcapabilitiesofthisplugin:
VALIDATE_PASSWORD_STRENGTH():AnSQL function used to find the strength of apassword.Ittakesapasswordasanargumentandreturnsanintegervaluebetween0and100.Here,0indicatesaweakpasswordand100indicatesastrongpassword.
Check password as per policy in SQL statements: For all the SQLstatementswhichuseapasswordasacleartextvalue,thepluginwillcheckthe provided password against the policy of the password and, based onthat, return a response. In the case of a weak password, the plugin willreturn an ER_NOT_VALID_PASSWORD error. ALTERUSER,CREATE USER, GRANT, SET PASSWORDstatements,andthePASSWORD()functionarealwayscheckedbythispluginifthepasswordisdefinedascleartextinanargument.
<strong>[mysqld]</strong><br/><strong>plugin-load-add=validate_password.so</strong>
<strong>INSTALLPLUGINvalidate_passwordSONAME'validate_password.so';</strong>
Variables and options related tothepasswordvalidationplugin
MySQL 8 provides following system variables, status variables and optionsrelatedtopasswordvalidationplugin.
validate_password_check_user_name:Thisisasystemvariable,andisenabledbydefaultinMySQL8.Asthenameimplies,itisusedtocompareapasswordwiththeusernameofthecurrentlyeffectiveuser.
If thepasswordmatcheswith theusernameor its reverse,MySQL8willreject the password, irrespective of the VALIDATE_PASSWORD_STRENGTH() functionvalue.
validate_password_dictionary_file:This systemvariablecontains thepathnameofthedirectorywhichisusedbythevalidate_passwordplugin.Youcansetitat runtimewithout a server restart, and it is available once the plugin isinstalled. Set the password policy value as 2(strong), if you define thedirectory for the password check. Possible values for password policy isdescribeundervalidate_password_policysystemvariable.
validate_password_length: This system variable is available once a plugin isinstalled and is used to define the minimum number of characters for apasswordtocheckwiththevalidate_passwordplugin.
validate_password_mixed_case_count: This system variable is available once apluginisinstalledandisusedtodefinetheminimumnumberoflowercaseanduppercasecharactersforapasswordcheck.
validate_password_number_count: This system variable is available once thepluginisinstalledandisusedtodefinetheminimumnumberofdigitsforthepasswordcheck.
validate_password_special_char_count: This system variable is available once aplugin is installed and is used to define the minimum number of non-alphanumericcharactersinapasswordcheck.
validate_password_policy: This system variable is available once a plugin isinstalled,andit indicateshowapluginshouldbehavein thecaseofothersystem variables. The following values of this variable describe thebehaviorofthevalidate_passwordplugin:
Policy TestsPerformed
0orLOW Length
1orMEDIUM Length;numeric,lowercase/uppercase,andspecialcharacters
2orSTRONG
Length;numeric,lowercase/uppercase,andspecialcharacters;dictionaryfile
Reference:https://dev.mysql.com/doc/refman/8.0/en/validate-password-options-variables.html
validate_password_dictionary_file_last_parsed: This is a status variable used toindicatethetimewhenadirectoryfilewaslastparsed.
validate_password_dictionary_file_words_count: This is a status variable used toindicatethenumberofwordsreadfromthedirectoryfile.
--validate-password[=value]:Thisoption isused todefinehowa server loadsthe validate_password plugin on startup. This option is available only if thepluginwasregisteredwithINSTALLPLUGINor if it is loadedwith the--plugin-load-addfeature.
MySQL8keyringMySQL8providesakeyringservice,whichallowstheMySQLserver'sinternalcomponentsandpluginstostoretheirsensitiveinformationforlateruse.Forthisfeature,MySQL8usesthekeyring_fileplugin,whichwillstoredataintothefilelocatedontheserverhost.ThispluginisavailableinalldistributionsofMySQL,suchastheCommunityEditionandEnterpriseEdition.
<strong>mysqld]</strong><br/><strong>plugin-load-add=keyring_file.so</strong>
<strong>INSTALLPLUGINkeyring_fileSONAME'keyring_file.so';</strong>
System variables related tokeyringplugin
MySQL8providesbelowsystemvariablerelatedtokeyringplugin:
keyring_file_data:Thissystemvariableisavailableonceapluginisinstalledand is used to define a pathname of the data file which is used by thekeyring_fileplugintostoresecuredata.Keyringoperationsaretransactional,sothispluginusesabackupfileduringwriteoperationtohandlearollbackscenario. In this case, the backfile is also named with the same namingconventionasdefinedinthekeyring_file_datasystemvariable,withthesuffixas.backup.
Summary
Inthischapter,westartedwithanoverviewofsecurity,andthentheridebeganwith theMySQL8 security related features.Firstwediscussed somecommonsecurity issues, thenwe showed how to assign privileges and how tomanageaccess control in MySQL 8. Encryption was also covered in this chapter, tosecureyoursensitivedata.
And finally,we covered some important security plugins,which are useful toimplementsecurityinMySQL8.
It's now time to move on to our next chapter, where we will be configuringMySQL8
foroptimization.Foroptimization,wewillcoverdifferentareasofthedatabase,suchasoptimizingqueries,optimizingtables,optimizingbufferingandcaching,andmuchmore.Apartfromserverconfiguration,italsocovershowtoconfigureaclientforoptimization.
OptimizingMySQL8
Inthepreviouschapter,welearnedaboutsecurity,whichisanimportantaspectofanyproduction-gradeapplication.Thechapterstartedwithanintroductiontosecurity and identifying common security issues. Moving on, the chaptercovered access control mechanisms, account management, and encryption inMySQL8.WelearnedvariousMySQL
8 security plugins in the later part of the chapter. Security is an importantbenchmark for every production-grade application. That's why the previouschapterisanimportantone.
Movingalongasimilarline,withtheobjectiveofdevelopinghighlyoptimizeddatabases, this chapter focuses on optimization methods. It starts with anoverviewofwhatoptimizationmeansinMySQL8. It takes thereader throughMySQL 8 server and client optimization, optimizing database structure, andoptimizingcommonqueriesanddatabasetables.Laterinthechapter,emphasisisgiventobufferingandcachingtechniques.
Thefollowingisalistoftopicstobecovered:
OverviewofMySQL8optimization
OptimizingMySQL8serversandclients
Optimizingdatabasestructure
Optimizingqueries
Optimizingtables
Leveragingbufferingandcaching
Overview of MySQL 8optimization
Let's start with understanding MySQL 8 optimization. Optimization is theprocessof
identifying performance bottlenecks and implementing optimized solutions toovercome
these issues. Optimization in MySQL 8 involves performance measurement,configuration,
andtuningatseveraldifferentlevels.Itisanimportanttaskforanadministrator
to optimize the performance at different levels, like individual SQL queries,entire
database applications, database servers, or distributed database servers.Performance
optimizationattheCPUandmemorylevelsimprovesscalability.Italsoallowsthe
database to handlemore complex queries without slowing down the databaseserver.
The performance of a database depends on multiple factors. At the databaselevel,
thesefactorscanbetables,queries,andconfigurations.Databaseserverstartups
anddatabasequeryexecutionsareacoupleoftheeventswhentheseconstructsimpact
theCPUorperformI/O(Input/Output)operationsatthehardwarelevel.Thisisa
responsibility of the MySQL 8 database administrator: to make sure that thehardware
performancestandsatanoptimumlevel.Itisrequiredthatthehardwareisusedwith
the maximum efficiency possible. At the software level, performanceoptimizationstarts
bylearninggenericrulesandguidelinesandmeasuringperformancewithclocktime.
Gradually,weunderstand the internals of variousdatabaseoperations.Wecanmeasure
theperformance in termsofCPUcyclesand I/Ooperations.Toattain thebestdatabase
performance, we can optimize the software and hardware configurations at abasiclevel.
At an advanced level, we can improve MySQL itself by developing customstorageengines
andhardwareappliances,whichexpandtheMySQLecosystem.
Optimizingthedatabase
What is the most important factor in making the database perform at theoptimumspeed?
Theansweris,basicdatabasedesign.Thefollowingisachecklisttokeepaneye
onfordatabasedesign:
Thedatabasecolumnshavetobeoftherightdatatypes.Tablesmusthaveappropriate
columns for the purposes to be served. Applications that have frequentoperations
to be performed on the database have many tables with fewer columns,whereasapplications
thatanalyzelargeamountsofdatahavelimitedtableswithmanycolumns.
Aswelearnedinoneofthepreviouschapters,databaseindexingplaysan
important
role in enhancing query performance. So, it is important to have correctindexesin
placeforqueryexecutionefficiency.
We discussed database storage engines, such as MyISAM or InnoDB, in earlierchapters.Useofanappropriatestorageengineforeachindividualtable
is important.InnoDB ispreferable for transactionaldatabase tables,whereasMyISAMispreferablefordefiningnon-trasactionaldatabasetables.Thechoiceofstorage
engineplaysavitalroleindefiningtheperformanceandscalabilityofthedatabase.
In the chapter onMySQL8 data types,we learned about row formats indetail.Itis
againimportantforeachtohaveanappropriaterowformat.Thechoiceofrowformat
dependsonthestorageenginechosen.Compressedtablesoccupylessdiskspaceand
requirefewerdiskI/Ooperations.ForInnoDBtables,compressionisavailableforallreadandwriteoperations.Onthecontrary,
compressionisavailableforread-onlyMyISAMtables.
TheMySQLdatabasesupportsmultiplelockingstrategies.Thelockingcanbeatatable-level
orarow-level.Theapplicationmustuseanappropriatelockingstrategy.Bygranting
shared access wherever appropriate, it becomes possible to run databaseoperations
concurrently.Also,itshouldbepossibletorequestexclusiveaccess,sothatcritical
databaseoperationscanbeexecutedwithdataintegrityissuesandprioritycanbe
maintained. In this case, thechoiceof storageengine is again significant.The InnoDB storage engine handles most locking issues without userinvolvement.Itallowsfor
betterconcurrencyand reduces theamountofexperimentationand tuningforthecode.
The memory areas must use the correct caching size. It should be largeenoughtohold
frequently accessed data, and at the same time, not so large that theyoverloadphysical
memoryandcausepaging.TheInnoDBbufferpoolandMyISAMkeycachearethemainmemoryareastobeconfigured.
For newly created tables, MyISAM is the default storage engine. Inpractical use, InnoDB advanced performance features mean thattableswith InnoDB storage enginesoutperform the MyISAM tables foranoperations-heavydatabase.
Optimizingthehardware
Growthisthenatureofeverysoftwareapplication.Astheapplicationgrows,sodoes the database. The database becomesmore andmore busy in performingoperations. At a certain point, the database application eventually hits thehardware limits. An administrator must evaluate the possibility of tuning theapplicationor re-configuring theserver toavoid these issues. It shouldalsobeevaluated whether deploying more hardware resources would help. Systembottlenecksusuallyarisefromthefollowingsources:
Diskseeks:Aspartofthediskreadoperation,findingapieceofdatatakestimeforthedisk.
The mean time for finding a piece of data is usually lower than 10millisecondswithmoderndisks.So, in theory, it shouldbe100seekspersecond.Withtechnologicalevolution,thenewdiskshaveimprovementsonthedisktime,butitisveryhardtooptimizeforsingletables.Tooptimizetheseektime,itisnecessarytodistributedataacrossmorethanonedisk.
Diskreadingandwriting:Toreadorwritedatafromadisk,itisrequiredforthedisktobeatthecorrectposition.Onediskdeliversatleast10to20
MB of throughput per second (throughput is the amount of data read orwritten per second). So, the read and write throughput is more easilyoptimizedthantheseektime,aswecanreadinparallelfrommultipledisks.
CPUcycles:Wemustprocessthedatawhenitisinthemainmemorytogetthedesiredresult.
With large tables, the amount of memory is the most common limitingfactor.Withsmalltables,however,speedisusuallynotanissue.
Memory bandwidth: In an uncommon scenario, the main memorybandwidthbecomesabottleneckwhentheCPUneedsmoredatathancanbefitintheCPUcachememory.
OptimizingMySQL 8 servers andclientsThissectionfocusesonoptimizationforMySQL8databaseserversandclients,starting
with optimizing the server and followed by optimizing MySQL 8 client-sideentities.This
section ismore relevant todatabaseadministrators, to ensureperformanceandscalability
acrossmultiple servers. Itwouldalsohelpdeveloperspreparingscripts (whichincludes
settingupthedatabase)andusersrunningMySQLfordevelopmentandtestingtomaximize
theproductivity.
OptimizingdiskI/O
Inthissection,wewill learnwaystoconfigurestoragedevicestodevotemoreand
fasterstoragehardwaretothedatabaseserver.Amajorperformancebottleneckis
disk seeking (finding the correct place on the disk to read or write content).When
theamountofdatagrowslargeenoughtomakecachingimpossible,theproblemwith
diskseedsbecomesapparent.Weneedatleastonediskseekoperationtoread,and
several disk seek operations towrite things in large databaseswhere the dataaccess
is donemore or less randomly.We should regulate orminimize the disk seektimes
usingappropriatedisks.
In order to resolve the disk seek performance issue, increasing the number ofavailable
diskspindles,symlinkingthefilestodifferentdisks,orstrippingdiskscanbe
done.Thefollowingarethedetails:
Using symbolic links:Whenusing symbolic links,we can create aUnixsymboliclinksforindexanddata
files. The symlink points from default locations in the data directory toanother
disk in the case of MyISAM tables. These links may also be striped. Thisimprovestheseekandreadtimes.The
assumption is that the disk is not used concurrently for other purposes.Symbolic
linksarenotsupportedforInnoDBtables.However,wecanplaceInnoDBdataandlogfilesondifferentphysicaldisks.
Striping:Instriping,wehavemanydisks.Weputthefirstblockonthefirstdisk,thesecond
blockontheseconddisk,andsoon.TheNblockonthe(N%number-of-disks)disk.Ifthestripesizeisperfectlyaligned,
the normal data size will be less than the stripe size. This will help toimprove
theperformance.Stripingisdependentonthestripesizeandtheoperatingsystem.
Inanidealcase,wewouldbenchmarktheapplicationwithdifferentstripe
sizes.
The speed difference while striping depends on the parameters we haveused,likestripe
size.Thedifference inperformancealsodependson thenumberofdisks.Wehaveto
choose ifwewant tooptimizefor randomaccessorsequentialaccess.Togainreliability,
wemaydecide to set upwith striping andmirroring (RAID0+1).RAIDstandsforRedundantArrayofIndependentDrives.Thisapproachneeds2xN drives to holdN drives of data.With a good volumemanagementsoftware,wecanmanagethissetup
efficiently.
Thereisanotherapproachtoit,aswell.Dependingonhowcriticalthetypeofdata
is,wemayvarytheRAIDlevel.Forexample,wecanstorereallyimportantdata,such
ashostinformationandlogs,onaRAID0+1orRAIDNdisk,whereaswecanstoresemi-important
dataonaRAID0disk.InthecaseofRAID,paritybitsareusedtoensuretheintegrity
ofthedatastoredoneachdrive.So,RAIDNbecomesaproblemifwehavetoomany
writeoperationstobeperformed.Thetimerequiredtoupdatetheparitybitsinthis
caseishigh.
If it is not important tomaintainwhen the filewas last accessed,we can
mountthe
filesystemwiththe-onoatimeoption.Thisoptionskips theupdatesonthefilesystem,whichreducesthedisk
seek time. We can also make the file system update asynchronously.Dependingupon
whetherthefilesystemsupportsit,wecansetthe-oasyncoption.
UsingNFSwithMySQL
While using a Network File System (NFS), varying issues may occur,dependingontheoperatingsystemandtheNFSversion.
Thefollowingarethedetails:
DatainconsistencyisoneissuewithanNFSsystem.Itmayoccurbecauseofmessagesreceivedoutoforderorlostnetworktraffic.WecanuseTCPwithhardandintrmountoptionstoavoidtheseissues.
MySQLdataandlogfilesmaygetlockedandbecomeunavailableforuseifplacedonNFSdrives.IfmultipleinstancesofMySQLaccessthesamedatadirectory,itmayresultinlockingissues.ImpropershutdownofMySQLorpower outage are other reasons for filesystem locking issues. The latestversionofNFSsupportsadvisoryandlease-basedlocking,whichhelpsinaddressing the locking issues.Still, it isnot recommended toshareadatadirectoryamongmultipleMySQLinstances.
Maximumfilesizelimitationsmustbeunderstoodtoavoidanyissues.WithNFS2,onlythelower2GBofafileisaccessiblebyclients.NFS3clients
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>performance_schema<spanclass="tokenpunctuation">.</span>setup_instruments<spanclass="tokenkeyword">WHERE</span><spanclass="tokenkeyword">NAME</span><spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'%memory%'</span><spanclass="tokenpunctuation">;</span></strong>
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>performance_schema<spanclass="tokenpunctuation">.</span>setup_instruments<spanclass="tokenkeyword">WHERE</span><spanclass="tokenkeyword">NAME</span><spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'%memory/innodb%'</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span>
<spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>NAME<spanclass="tokenpunctuation">|</span>ENABLED<spanclass="tokenpunctuation">|</span>
TIMED<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span>
<spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/adaptivehashindex<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/buf_buf_pool<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/dict_stats_bg_recalc_pool_t<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/dict_stats_index_map_t<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/dict_stats_n_diff_on_level<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/other<spanclass="token
punctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/row_log_buf<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/row_merge_sort<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/std<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb/trx_sys_t::rw_trx_ids<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|</span>NO<spanclass="tokenpunctuation">|<br/></span></span><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="token
punctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span></strong>
<strong><spanclass="tokenconstant">performance-schema-instrument</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>'memory/%=COUNTED'</span></strong>
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>performance_schema<spanclass="tokenpunctuation">.</span>memory_summary_global_by_event_name<spanclass="tokenkeyword">WHERE</span>EVENT_NAME<spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'memory/innodb/buf_buf_pool'</span>\G;<br/><br/>EVENT_NAME:<spanclass="tokenkeyword">memory</span><spanclass="tokenoperator">/</span>innodb<spanclass="tokenoperator">/</span>buf_buf_pool<br/>COUNT_ALLOC:<spanclass="tokennumber">1<br/></span>COUNT_FREE:<spanclass="tokennumber">0<br/></span>SUM_NUMBER_OF_BYTES_ALLOC:<spanclass="tokennumber">137428992<br/></span>SUM_NUMBER_OF_BYTES_FREE:<spanclass="tokennumber">0<br/></span>LOW_COUNT_USED:<spanclass="tokennumber">0<br/></span>CURRENT_COUNT_USED:<spanclass="tokennumber">1<br/></span>HIGH_COUNT_USED:<spanclass="tokennumber">1<br/></span>LOW_NUMBER_OF_BYTES_USED:<spanclass="tokennumber">0<br/></span>CURRENT_NUMBER_OF_BYTES_USED:<spanclass="tokennumber">137428992<br/></span>HIGH_NUMBER_OF_BYTES_USED:<spanclass="tokennumber">137428992</span></strong>
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenfunction">SUBSTRING_INDEX</span><spanclass="tokenpunctuation">(</span>event_name<spanclass="tokenpunctuation">,</span><spanclass="tokenstring">'/'</span><spanclass="tokenpunctuation">,</span><spanclass="tokennumber">2</span><span
class="tokenpunctuation">)</span><spanclass="tokenkeyword">AS</span><br/>code_area<spanclass="tokenpunctuation">,</span>sys<spanclass="tokenpunctuation">.</span>format_bytes<spanclass="tokenpunctuation">(</span><spanclass="tokenfunction">SUM</span><spanclass="tokenpunctuation">(</span>current_alloc<spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">)</span><br/><spanclass="tokenkeyword">AS</span>current_alloc<br/><spanclass="tokenkeyword">FROM</span>sys<spanclass="tokenpunctuation">.</span>x$memory_global_by_current_bytes<br/><spanclass="tokenkeyword">GROUP</span><spanclass="tokenkeyword">BY</span><spanclass="tokenfunction">SUBSTRING_INDEX</span><spanclass="tokenpunctuation">(</span>event_name<spanclass="tokenpunctuation">,</span><spanclass="tokenstring">'/'</span><spanclass="tokenpunctuation">,</span><spanclass="tokennumber">2</span><spanclass="tokenpunctuation">)</span><br/><spanclass="tokenkeyword">ORDER</span><spanclass="tokenkeyword">BY</span><spanclass="tokenfunction">SUM</span><spanclass="tokenpunctuation">(</span>current_alloc<spanclass="tokenpunctuation">)</span><spanclass="tokenkeyword">DESC</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="token
punctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>code_area<spanclass="tokenpunctuation">|</span>current_alloc<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="token
punctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/innodb<spanclass="tokenpunctuation">|</span>843.24MiB<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/performance_schema<spanclass="tokenpunctuation">|</span>81.29MiB<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/mysys<spanclass="tokenpunctuation">|</span>8.20MiB<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/sql<spanclass="tokenpunctuation">|</span>2.47MiB<spanclass="tokenpunctuation">|
</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/memory<spanclass="tokenpunctuation">|</span>174.01KiB<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/myisam<spanclass="tokenpunctuation">|</span>46.53KiB<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/blackhole<spanclass="tokenpunctuation">|</span>512bytes<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/federated<spanclass="tokenpunctuation">|</span>512bytes<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/csv<spanclass="tokenpunctuation">|</span>512bytes<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>memory/vio<spanclass="tokenpunctuation">|</span>496bytes<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span>
<spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span></strong>
<strong>mysql>showglobalstatus;</strong><br/><strong>+-----------------------------+--------+</strong><br/><strong>|Variable_name|Value|</strong><br/><strong>+-----------------------------+--------+</strong><br/><strong>|Aborted_clients|0|</strong><br/><strong>|Aborted_connects|1|</strong><br/><strong>|Acl_cache_items_count|0|</strong><br/><strong>|Binlog_cache_disk_use|0|</strong><br/><strong>|Binlog_cache_use|0|</strong><br/><strong>|Binlog_stmt_cache_disk_use|0|</strong><br/><strong>|Binlog_stmt_cache_use|0|</strong><br/><strong>|Bytes_received|443|</strong><br/><strong>|Bytes_sent|346|</strong><br/><strong>|Threads_cached|0|</strong><br/><strong>|Threads_connected|1|</strong><br/><strong>|Threads_created|1|</strong><br/><strong>|Threads_running|2|</strong><br/><strong>+-----------------------------+--------+</strong>
<strong>mysql>showstatuslike'%Thread%';</strong><br/><strong>+------------------------------------------+-------+</strong><br/><strong>|Variable_name|Value|</strong><br/><strong>+------------------------------------------+-------+</strong><br/><strong>|Delayed_insert_threads|0|</strong><br/><strong>|Performance_schema_thread_classes_lost|0|</strong><br/><strong>|Performance_schema_thread_instances_lost|0|</strong><br/><strong>|Slow_launch_threads|0|</strong><br/><strong>|Threads_cached|0|</strong><br/><strong>|Threads_connected|1|</strong><br/><strong>|Threads_created|1|</strong><br/><strong>|Threads_running|2|</strong><br/><strong>+------------------------------------------+-------+</strong>
OptimizinglockingoperationsAsdiscussedinoneoftheearlierchapters,MySQL8useslockingmechanismstomanagecontention.Contentionoccurswhenconcurrentlyexecutingqueriesinmultiplethreadstry togetaholdofonetableat thesametime.If thesequeriesareperformedonthetableconcurrently,thetabledataisleft inaninconsistentstate. MySQL 8 supports two types of locking: internal locking and externallocking.
Internal locking is performedbymultiple threadswithin theMySQLserver tomanagecontentionfortablecontents.ThistypeoflockingisperformedentirelybytheMySQLserver,withoutinvolvinganyotherprograms.So,whyitiscalledinternal locking? In the caseof external locking, theMySQL server andotherprograms lock table files to decide which programs can access the table at atime.
Thefollowingarethetwomethodsforinternallocking:
Row-levellocking.Table-levellocking.
Row-level locking in MySQL supports simultaneous write access to multiplesessions. This enables multi-user and highly concurrent applications. Whileperformingmultiple concurrentwrite operations on a single table, it is highlypossiblethatadeadlockmayoccur.
Inordertoavoidsuchadeadlocksituation,alockingmechanismacquireslocksat the beginning of the transaction using the SELECT ... FOR UPDATE statement foreachsetofrowstobemodified.MySQLappliesthestatementsinthesameorderwithin each transaction if transactions lock more than one table. The InnoDBdatabase engine automatically detects deadlock conditions and rolls back theaffectedtransactions.Consideringthis,deadlocksaffectperformance.
Thedeadlockdetectionmaycauseslowdownifmanythreadswaitforthesamelock in highly concurrent systems. In such cases, it becomesmore efficient todisabledeadlockdetection.Wecanrelyontheinnodb_lock_wait_timeoutsettingfor
transaction rollback when deadlock occurs. Using the innodb_deadlock_detect
configurationoption,wecandisablethedeadlockdetection.
Thefollowingaretheadvantagesofrow-levellocking:
Whendifferentsessionsaccessdifferentrowsinatable,thenumberoflockconflictsisfewerThenumberofchangestoberolledbackisfewerItbecomespossibletolockasingletablerowforalongtime
Table-levellockingisusedbyMySQLforMyISAM,MEMORY,andMERGE tables. In thecase of table-level locking,MySQL permits only one session to update thesetablesatatime.Withtable-levellocking,thesestorageenginesbecomesuitablefor read-onlyor single-user applications.These storage engines request all therequiredlocksatonce,whenthequerybegins,toavoidanydeadlocks.Italwayslocksthetablesinthesameorder.Themajordrawbackwithtable-levellockingis that it affects concurrency. If other sessions need tomodify the table, theymustwaituntiltheconcurrentdatachangestatementfinishes.
Thefollowingaretheadvantagesoftable-levellocking:
Itrequireslessmemorycomparedtorow-levellockingWhenusedonalargepartofthetable,it isfast,becauseonlyonelockisrequiredIfGROUPBYoperationsareperformedfrequently,itisfast
ThefollowingisthestrategyforMySQLtograntwritelocksontables:
1. Putawritelockonthetableiftherearenowritelocksonthetable2. Puta lock request in thewrite lockqueue if the tablealreadyhasawrite
lock
ThefollowingisthestrategyforMySQLtograntreadlocksontables:
1. Putareadlockonthetableiftherearenoreadlocksonthetable2. Putalockrequestinthereadlockqueueifthetablealreadyhasareadlock
Morepriorityisgiventotableupdatesthantableretrievals.Thelockisavailabletothewritelockrequestsfirst,andthentothereadlockrequestswhenalockis
released.
Thefollowingisanexampletoanalyzetablelockcontention:
mysql>SHOWSTATUSLIKE'Table_locks%';
+-----------------------+-------+
|Variable_name|Value|
+-----------------------+-------+
|Table_locks_immediate|5|
|Table_locks_waited|0|
+-----------------------+-------+
TheMyISAMstorageengineinherentlysupportsmultipleconcurrentinsertsinordertoreducecontentionbetweenreadersandwritersforatable.ItallowstheMyISAMtable to insert rows in themiddleofadata file. If the tabledoesnothaveanyfreeblocksinthemiddleofthedatafile,therowsareinsertedattheendofthefile.ThisenablesMySQLtoexecuteINSERTandSELECTqueriesonthesametable,concurrently. concurrent_insert is the global system variable which controls thebehavioroftheMyISAMstorageenginetoallowexecutionofconcurrentINSERTandSELECTstatements.IfthissystemvariableissettoAUTO,concurrentINSERTandSELECTareallowed.
IfconcurrentinsertsarenotpossibleandwewanttoperformmultipleINSERTandSELECToperationsonatabletab1,wecanusethetemporarytabletemp_tab1toholdthetab1tabledataandupdatethetab1tablewiththerowsfromthetemp_tab1table.Thefollowingisanexamplewhichdemonstratesthisscenario:
mysql>LOCKTABLEStab1WRITE,temp_tab1WRITE;
mysql>INSERTINTOtab1SELECT*FROMtemp_tab1;
mysql>DELETEFROMtemp_tab1;
mysql>UNLOCKTABLES;
<strong>BENCHMARK(loop_count,expression)</strong>
<strong>mysql>selectbenchmark(1000000,1+1);</strong><br/><strong>+-------------------------+</strong><br/><strong>|benchmark(1000000,1+1)|</strong><br/><strong>+-------------------------+</strong><br/><strong>|0|</strong><br/><strong>+-------------------------+</strong><br/><strong>1rowinset(0.15sec)</strong>
Fromtheprecedingexample,wecanfindthatthetimetakentocalculate1+1for1000000timesis0.15seconds.
<strong>mysql>showprocesslist;</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>|Id|User|Host|db|Command|Time|</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>+------------------------+-----------------------+</strong><br/><strong>|State|Info|</strong><br/><strong>+------------------------+-----------------------+</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>|4|event_scheduler|localhost|NULL|Daemon|214901|</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>|8|root|localhost:58629|NULL|Query|0|</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>+------------------------+-----------------------+</strong><br/><strong>|Waitingonemptyqueue|NULL|</strong><br/><strong>+------------------------+-----------------------+</strong><br/><strong>|starting|showfullprocesslist|</strong><br/><strong>+------------------------+-----------------------+</strong>
<strong>mysql>select*frominformation_schema.processlist;</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>|ID|USER|HOST|DB|COMMAND|TIME|</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>+------------------------+----------------------------------------------+</strong><br/><strong>|STATE|INFO|</strong><br/><strong>+------------------------+----------------------------------------------+</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>|8|root|localhost:58629|NULL|Query|0|</strong><br/><strong>+----+-----------------+-----------------+------+---------+--------+</strong><br/><strong>|4|event_scheduler|localhost|NULL|Daemon|215640|</strong><br/><strong>+----+-
----------------+-----------------+------+---------+--------+</strong><br/><strong>+------------------------+----------------------------------------------+</strong><br/><strong>|executing|select*frominformation_schema.processlist|</strong><br/><strong>+------------------------+----------------------------------------------+</strong><br/><strong>|Waitingonemptyqueue|NULL|</strong><br/><strong>+------------------------+----------------------------------------------+</strong>
Themysqladminprocesslistcommand.Theperformanceschemathreadstable,stagetables,andlocktables.
Wemustbeabletoviewtheinformationofuserthreads.ThePROCESSprivilegeisrequiredtoviewtheinformationaboutthreadsbeingexecuted.Toaccessthreads,aMutexaccessisnotrequired.IthaslessimpactontheMySQLserverperformance.AccessingINFORMATION_SCHEMA.PROCESSLISTandSHOWPROCESSLISTrequiresaMutexandhasanimpactonperformance.Threadsalsoprovidedetailsofbackgroundthreads.INFORMATION_SCHEMA.PROCESSLISTandSHOWPROCESSLISTdonotprovideinformationaboutbackgroundthreads.
Thefollowingtableshowstheinformationcontainedineachprocesslistentry:
Information Details
Id Clientconnectionidentifierfortheclientthatthethreadisassociatedwith.
User,Host Accountassociatedwiththethread.db DefaultdatabaseforthethreadorNULL.Command,State Itindicateswhatthethreadiscurrentlydoing.
Time Itindicateshowlongthethreadhasbeeninthecurrentstate.
Info Itcontainstheinformationofthestatementbeingexecutedbythethread.
Thefollowingisthethreadstatevaluesassociatedwithgeneralqueryprocessing:
Aftercreate:Itoccurswhenthethreadcreatesatable,includinginternaltemporarytablesAnalyzing:ItoccurswhenthethreadiscalculatingMyISAMkeydistributionCheckingpermissions:ItoccurswhencheckingiftheserverhastherequiredprivilegestoexecutetheSQLstatementCheckingtable:ItoccurswhenthethreadisperformingatablecheckoperationCleaningup:ItoccurswhenthethreadhasprocessedonecommandandfreesthememoryClosingtables:ItoccurswhenthethreadisflushingthechangedtabledatatodiskandclosingtheusedtablesAlteringtable:ItoccurswhentheserverisprocessingtheALTERTABLEstatementCreatingindex:ItoccurswhenthethreadisprocessingALTERTABLE...ENABLEKEYSfortheMyISAMtableCreatingtable:Itoccurswhenthethreadiscreatingatableend:Itoccursattheend,butbeforethecleanupofALTERTABLE,CREATEVIEW,DELETE,INSERT,SELECT,orUPDATEstatementsexecuting:Itoccurswhenthethreadhasbegunexecutingastatementinit:ItoccursbeforetheinitializationofALTERTABLE,DELETE,INSERT,SELECT,andUPDATEstatements
Thefollowingisthelistofcommonstatesinthemaster'sbinlogdumpthreadforreplicationmasterthreads:
Finishedreadingonebinlog;switchingtonextbinlogMasterhassentallbinlogtoslave;waitingformoreupdatesSendingbinlogeventtoslaveWaitingtofinalizetermination
ThefollowingisalistofcommonstatesforaslaveserverI/Othread:
CheckingmasterversionConnectingtomasterQueueingmastereventtotherelaylogReconnectingafterafailedbinlogdumprequestReconnectingafterafailedmastereventreadRegisteringslaveonmasterRequestingbinlogdumpWaitingforitsturntocommitWaitingformastertosendeventWaitingformasterupdateWaitingforslaveMutexonexitWaitingfortheslaveSQLthreadtofreeenoughrelaylogspaceWaitingtoreconnectafterafailedbinlogdumprequestWaitingtoreconnectafterafailedmastereventread
ThefollowingisalistofcommonstatesforaslaveserverSQLthread:
KillingslaveMakingtemporaryfile(append)beforereplayingLOADDATAINFILE
Makingtemporaryfile(create)beforereplayingLOADDATAINFILE
ReadingeventfromtherelaylogSlavehasreadallrelaylog;waitingformoreupdatesWaitingforaneventfromcoordinatorWaitingforslaveMutexonexit
WaitingforslaveworkerstofreependingeventsWaitingforthenexteventinrelaylogWaitinguntilMASTER_DELAYsecondsaftermasterexecutedevent
OptimizingdatabasestructureAsadatabaseadministrator,wemust look for efficientways toorganize tableschema, tables, and columns.Weminimize I/O, plan ahead, and keep relateditems together to tune the application code in order to keep performance highwithanincreaseindatavolume.Itusuallystartswithefficientdatabasedesign,whichmakesiteasierfor teammemberstowritehigh-performanceapplicationcode.Italsomakesthedatabaselikelytosustainitselfasapplicationsevolveorarerewritten.
Optimizingdatasize
In order to minimize the space on the disk, we should start designing thedatabase tables. This results in huge performance improvements, as it reducestheamountofdatatobewrittentoandreadfromthedisk.Smallertablesusuallyneedlessmainmemory,whilethecontentsareactivelyprocessedduringqueryexecution.Anyreductionintabledataspaceresultsinaneedforsmallerindexesthatcanbeprocessedfaster.
As discussed in the chapter onMySQL 8 data types,MySQL supports manydifferent storage engines and row formats. We can decide the storage andindexingmethodtobeusedforeachtable.Itisabigperformancegaintochoosethepropertableformat.
Tablecolumns
Weshouldusethesmallestfeasibledatatypeforatablecolumn.Thisresultsinthe most efficient approach. MySQL supports specialized data types to savememory and disk space. For example, we should use integer types whereverpossible to get smaller tables.Comparing MEDIUMINT and INT,MEDIUMINT is a betterchoice,asituses25%lessspacecomparedtoINT.
Wemustdeclarecolumns tobe NOTNULLwhereverpossible.ThisenablesbetteruseofindexesandeliminatestheoverheadoftestingwhethereachvalueisNULLornot.ItresultsinfasterSQLoperations.Wecanalsosaveonebitpercolumnofstoragespace.WeshoulduseNULLifwereallyrequireit.NULLvaluesshouldnotbeallowedasaresultofdefaultsettingsforeverycolumn.
We can attain huge performance gain for a table andminimize storage spacerequirementbyusingfollowingtechniques:
RowformatAsadefault,theDYNAMICrowformatisusedwhencreatingInnoDB tables.Wecanconfigureinnodb_default_row_format touse row formatsother than DYNAMIC.We canalso specify the ROW_FORMAT option explicitly in a CREATE TABLE or ALTER TABLE
statement.
The row formats include COMPACT, DYNAMIC, and COMPRESSED. They decrease rowstoragespaceatthecostofincreasedCPUuseforsomeoperations.
Fortheaverageworkload,whichislimitedbythecachehitratesanddiskspeed,itwillbefaster.IfitislimitedbytheCPUspeed,itwillbeslower.
TherowformatsalsooptimizetheCHARdatatypecolumnstoragewhenitusesavariable length character set.With the REDUNDANT row format, the CHAR(N)columnvalueoccupiesNtimesthemaximumbytelengthinthecharacterset.TheInnoDBstorageengineallocatesvariableamountsofstoragewithintherangeofNtoNtimesthemaximumbytelengthinthecharacterset.
Afixed-sizerowformatisusedifwedonothavevariable-lengthcolumns,suchasVARCHAR,TEXT,orBLOB,inthecaseofMyISAMtables.
Indexes
A table's primary index must be as short as feasible. This enables easyidentification of each row. It is efficient, too. In the case of InnoDB tables, theprimarykeycolumn isduplicated ineach secondary indexentry. Ifwehaveashortprimarykey,itsavesspaceinthecaseofmanysecondaryindexes.
We should create only those indexes which improve query performance. Theindexesimproveinformationretrieval,buttheyslowdowntheinsertandupdateoperations. Indexes must be created with proper attention to the performanceimpact. If it is required to access a table by searching on a combination ofcolumns, it is preferred to have a composite index on the combination ofcolumns, rather than a separate index on each of the columns.Themost usedcolumnshouldbethefirstpartoftheindex.Ifitisacommonrequirementtousemany columns in selected operations on the table, it is advisable to have thecolumn with the most duplicates as the first column in the index. This givesbettercompressionoftheindex.
If a long string column is supposed to have a unique prefix as the first fewcharacters, it is advisable to indexonly theprefix,usingMySQL's support forindexingon the leftmostpartof thecolumn.Shorter indexesarepreferred,notonlyforthelessspacetheyrequire,butalsobecausetheyprovidemorehitsin
Joins
Ifatableisscannedveryoften,itisbeneficialtosplitthetableintotwotables,iffeasible. This holds true especially if it is a dynamic-format table. It is alsopossible to use smaller static format tables, which can be used to search forrelevantrowswhilescanningthetables.
The columnswith identical information should be declared in different tableswithidenticaldatatypes.Thisspeedsupjoinsbasedonmatchingcolumns.
Columnnamesmustbekeptsimple,soastousethesamenameacrosstables.Itsimplifies join queries. For example, in a customer table, we should use thecolumnnameofname,ratherthanusingcustomer_name.InordertomakethenamesportabletootherSQLservers,weshouldkeepthecolumnnamesshorterthan18characters.
Normalization
Thedatainthetablecolumnsmustbekeptnon-redundant,consideringthethirdnormalforminthenormalizationtheory.Ifthecolumnholdsrepeatinglengthyvalues, such as names or addresses, it is preferable to assign unique IDs andrepeat these IDs acrossmultiple smaller tables. In the event of searching, joinqueriesshouldbeusedbyreferencingIDsinthejoinclauses.
In an application, if the preference is speed and not disk space or themaintenancecostsofusingmultiplecopiesofdata, it is advisable toduplicatethe information or create summary tables to gain more speed. An examplescenario could be a business intelligence system,where data is analyzed fromlargetables.Inthiscase,normalizationrulesarenotstrictlyfollowed.
OptimizingMySQLdatatypes
Thefollowingaretheguidelinesforoptimizingnumericdatatypes:
Numeric columnsmust be preferred over string columns to store uniqueIDsorother
valuesthatcanberepresentedaseitherstringsornumbers.Itisfasterandoccupies
less memory to transfer and compare, because large numeric values arestoredinfewer
bytescomparedtostrings.
Itisfastertoaccessinformationfromadatabasethanfromatextfile.Thisis
especially truewhen numeric data is used. Information in the database isstoredin
a more compact format than in the text file. So, it requires fewer diskaccesses.
Thefollowingaretheguidelinesforoptimizingcharacterandstringdatatypes:
The binary collation order (logical sequence) should be used for fastercomparisons
andsortoperations.Thebinaryoperatorcanalsobeusedwithinaquerytousebinary
collationorder.
WithanInnoDBtable,whenweusearandomlygeneratedvalueasaprimarykey,itshouldbeprefixed
withanascendingvalue,suchasthedateandtime,iffeasible.Inthiscase,primary
keyvalues are stored nearer to eachother, physically. InnoDB can insert orretrievesuchvaluesfaster.
The binary VARCHAR data type should be used instead of BLOB for columnvaluesthatareexpectedtoholdlessthan8KBofdata.Iftheoriginal
table does not have any BLOB columns, the GROUP BY and ORDER BY clausesgeneratetemporarytables.ThesetemporarytablescanusetheMEMORYstorageengine.
In order to avoid string conversions while running a query, the columnsshouldbe
declared with the same character set and order wherever possible whencomparingthe
valuesfromdifferentcolumns.
Ifthetableholdsstringcolumnswhicharenotfrequentlyusedinretrievaloperations,
splitting the string columns into a separate table shouldbe considered. Intheretrieval
operations, join queries should be used with a foreign key wherevernecessary.MySQL
readsadatablockcontainingallthecolumnsofarowwhenitretrievesanyvalue
fromarow.Itallowsmorerowstofitwithineachdatablockwhenwekeeptherows
small, with only frequently used columns. These compact tables reducememoryusage
anddiskI/O.
ThefollowingaretheguidelinesforoptimizingBLOBdatatypes:
The performance requirements for a BLOB column may be different whenretrievinganddisplayinginformation.So,storingthe
BLOB specific table in a different storage device or a separate databaseinstanceshould
be considered. For example, it is required to retrieve a BLOB in a largesequentialdiskread.So,atraditionalharddriveoranSSDdevicemight
bettersuitneeds.
Inordertoreducethememoryrequirementsforaquerywhichdoesnotusea BLOB column, for a table with several columns, splitting the BLOB intoseparate tablesand referencingwith joinqueries shouldbeconsidered, asneeded.
Ifa tablecolumnisa largeblobwith textualdata,compressingshouldbeconsidered
first.Iftheentiretableiscompressedbythestorageengine,suchasInnoDBorMyISAM,thistechniqueshouldnotbeused.
<strong>>mysqladminstatus</strong><br/><strong>Uptime:262200Threads:2Questions:16Slowqueries:0Opens:111Flushtables:2Opentables:87Queriespersecondavg:0.000</strong>
<strong>mysql>SHOWGLOBALSTATUSLIKE'%Opened_Tables%';</strong><br/><strong>+---------------+-------+</strong><br/><strong>|Variable_name|Value|</strong><br/><strong>+---------------+-------+</strong><br/><strong>|Opened_tables|112|</strong><br/><strong>+---------------+-------+</strong>
UseofaninternaltemporarytableinMySQL
TheMySQL 8 server creates temporary internal tables while processing SQLstatements, in some cases. The following are the conditions under which theservercreatestemporarytables:
UNIONstatements
ViewswhichusestheTEMPTABLEalgorithm,UNION,oraggregation
Derivedtables
Commontableexpressions
Tablescreatedforsubqueryorsemijoinmaterialization
StatementsthatcontainORDERBYandGROUPBYclauses
StatementswithDISTINCTcombinedwithORDERBY
QueriesthatusetheSQL_SMALL_RESULTmodifier
INSERT...SELECTstatementsthatselectfromandinsertintothesametable
MultipletableUPDATEstatements
GROUP_CONCAT()orCOUNT(DISTINCT)expressions
TheEXPLAINstatementcanbeusedtodeterminewhetherthestatementrequiresatemporarytable.
TheEXPLAINstatementhaslimitations.Itwillnotindicateifthestatementrequiresatemporarytableforderivedormaterializedtemporarytables.
The Created_tmp_tables status variable keeps track of the number of temporarytablescreatedininternalmemory.WhentheMySQLservercreatesatemporarytable, it increments the value in the Created_tmp_tables status variable.Created_tmp_disk_tablesisanotherstatusvariablethatkeepstrackofthenumberoftablescreatedonthedisk.
Basedonthequeryconditions,theserverpreventstheuseoftemporarytablesinmemory.Insuchcases,theservercreatesatableonthedisk.Thefollowingaresomeinstances:
IfthetablehasaBLOBorTEXTcolumn
Ifthestatementhasastringcolumnwithamaximumlengthlargerthan512bytesintheSELECTlist,ifUNIONorUNIONALLisused
IftheSHOWCOLUMNSandDESCRIBEstatementsuseBLOBasthetypeofthecolumn
Thefollowingare theconditions inwhichaUNION isevaluatedwithoutcreatingtemporarytables:
TheunionisUNIONALLandnotUNIONorUNIONDISTINCT
OptimizingqueriesSimilartotables,databasequeriesarethemostcrucialelementofanydatabase.Applications interactwith the databases using queries.Queries are also calledexecutableSQLstatements.This section focuseson techniques to improve theperformanceofqueryexecution.
<strong><spanclass="tokenpunctuation">(</span><spanclass="tokenpunctuation">(</span>a<spanclass="tokenoperator">AND</span>b<spanclass="tokenpunctuation">)</span><spanclass="tokenoperator">AND</span>c<spanclass="tokenoperator">OR</span><spanclass="tokenpunctuation">(</span><spanclass="tokenpunctuation">(</span><spanclass="tokenpunctuation">(</span>a<spanclass="tokenoperator">AND</span>b<spanclass="tokenpunctuation">)</span><spanclass="tokenoperator">AND</span><spanclass="tokenpunctuation">(</span>c<spanclass="tokenoperator">AND</span>d<spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">)</span><br/><spanclass="tokenprompt">-></span><spanclass="tokenpunctuation">(</span>a<spanclass="tokenoperator">AND</span>b<spanclass="tokenoperator">AND</span>c<spanclass="tokenpunctuation">)</span><spanclass="tokenoperator">OR</span><spanclass="tokenpunctuation">(</span>a<spanclass="tokenoperator">AND</span>b<spanclass="tokenoperator">AND</span>c<spanclass="tokenoperator">AND</span>d<spanclass="tokenpunctuation">)</span></strong>
<strong><spanclass="tokenpunctuation">(</span>a<spanclass="tokenoperator"><</span>b<spanclass="tokenoperator">AND</span>b<spanclass="tokenoperator">=</span>c<spanclass="tokenpunctuation">)</span><spanclass="tokenoperator">AND</span>a<spanclass="tokenoperator">=</span><spanclass="tokennumber">5</span><br/><spanclass="tokenprompt">-></span>b<spanclass="tokenoperator">></span><spanclass="tokennumber">5</span><spanclass="tokenoperator">AND</span>b<spanclass="tokenoperator">=</span>c<spanclass="tokenoperator">AND</span>
a<spanclass="tokenoperator">=</span><spanclass="tokennumber">5</span></strong>
<strong><spanclass="tokenpunctuation">(</span>B<spanclass="tokenoperator">>=</span><spanclass="tokennumber">5</span><spanclass="tokenoperator">AND</span>B<spanclass="tokenoperator">=</span><spanclass="tokennumber">5</span><spanclass="tokenpunctuation">)</span><spanclass="tokenoperator">OR</span><spanclass="tokenpunctuation">(</span>B<spanclass="tokenoperator">=</span><spanclass="tokennumber">6</span><spanclass="tokenoperator">AND</span><spanclass="tokennumber">5</span><spanclass="tokenoperator">=</span><spanclass="tokennumber">5</span><spanclass="tokenpunctuation">)</span><spanclass="tokenoperator">OR</span><spanclass="tokenpunctuation">(</span>B<spanclass="tokenoperator">=</span><spanclass="tokennumber">7</span><spanclass="tokenoperator">AND</span><spanclass="tokennumber">5</span><spanclass="tokenoperator">=</span><spanclass="tokennumber">6</span><spanclass="tokenpunctuation">)</span><br/><spanclass="tokenprompt">-></span>B<spanclass="tokenoperator">=</span><spanclass="tokennumber">5</span><spanclass="tokenoperator">OR</span>B<spanclass="tokenoperator">=</span><spanclass="tokennumber">6</span></strong>
Optimizingindexes
Thebasicuseofindexesistoquicklyfindtherowswithspecificcolumnvalues.
Iftheindexisnotpresent,MySQLbeginswiththefirstrowandreadsthroughtheentiretabletofindallthematchingrows.Ittakesmoretime,dependingonhowlargeatableis.Iftheindexispresentfortheappropriatecolumns,MySQLisabletoquicklydeterminethepositiontoseektointhemiddleofthedatafile,withoutlookingatthewholetabledata.
ThefollowingisalistofoperationsforwhichMySQLusesindexes:
Tofindmatchingrows,basedonaWHEREclause,quickly.
MySQLuses the indexwith the smallest number of rows (most selectiveindex)inthecaseofchoosingfrommultipleindexestoeliminaterowsfromconsideration.
Theoptimizeruses the leftmostprefixof the index to lookuprows if thetable has a composite index. For example, in a tablewith three columnsindexed(oncol1,col2,col3),theoptimizercanlookforrowswithindexed
searchcapabilitieson(col1),(col1,col2),and(col1,col2,col3).
MySQLusesindexeswhileitfetchesrowsfromothertablesusingjoins.Ifthe indexesaredeclaredas thesame typeandsize,MySQLcanuse themefficientlyonthecolumn.
TheVARCHARandCHARareconsideredthesamewhendeclaredasthesamesize.
MySQLalsousesindexestofindtheminimum(MIN())ormaximum(MAX())valueforanindexedcolumnkey_col.ThepreprocessorcheckswhetheritisusingWHEREkey_part_N=constantonallkeypartstooptimizeit.
It is also possible to optimize the query to retrieve values withoutconsultingthedatarows.(Acoveringindexisanindexthatprovidesalltheresultsforaquery.)Ifthequeryusesonlythosecolumnsfromatablewhichare included in some index, the selected valueswill be fetched from theindextree.Thiswillhaveahigherspeedinretrievingvalues.
Queryexecutionplan
TheMySQLoptimizerconsidersoptimizationtechniquestoefficientlyperformthe lookups involved in the query, depending on the details of the tables,columns,andindexes,andtheconditionsintheWHEREclause.Aquerycanalsobeperformedwithoutreadingalltherowsonahugetable.
An SQL join can also be performedwithout comparing every combination ofrows.Aqueryexecutionplan is a setofoperations that theMySQLoptimizerchoosestoperformthemostefficientquery.ItisalsoknownastheEXPLAINplan.Asanadministrator,thegoalistorecognizetheaspectsofthequeryexecutionplanwhichindicateifaqueryisoptimized.
The EXPLAIN statement is used to determine the query execution plan. ThefollowingisthesetofinformationprovidedbytheEXPLAINstatement:
The EXPLAIN statement works with SELECT, DELETE, INSERT, UPDATE, and REPLACEstatements.
MySQLdisplays information from theMySQLoptimizer about thequeryexecution planwhen EXPLAIN is usedwith the SQL statement. Thismeans
MySQL explains the process with which the statement is executed. Itincludesinformationabouthowtablesarejoined,andinwhichorder.
If EXPLAIN displays the execution plan for the statement execution in thenamedconnection if it isusedwithFORCONNECTION connection_id insteadofexplainableSQLstatement.
EXPLAINdisplaysadditionalexecutionplaninformationforSELECTstatements.
EXPLAINisalsousefulinexaminingquerieswhichinvolvepartitionedtables.
EXPLAIN supports a FORMAT option, which can be used to select the outputformat.TheTRADITIONALformatdisplaystheoutputinatabularformat.Thisisthedefaultformatoption.
The JavaScript Object Notation (JSON) format option producesinformationintheJSONformat.
Based on the output from the EXPLAIN statement, it can be figured out whereindexescanbeaddedto the tables,so that thestatementexecutesfaster. Itcanalso be found whether the optimizer joins the tables in the optimized order.BeginthestatementwithSELECTSTRAIGHT_JOIN,insteadofjustSELECT,togiveahintto theoptimizer touse the joinordercorresponding to theorder the tablesarenamedintheSELECTstatement.AsSTRAIGHT_JOINdisablessemi-jointransformations,itmaypreventtheuseofindexes.
The optimizer trace is another tool to find the information on the queryexecution.
ItispossiblethattheoptimizertracemayprovideinformationdifferingfromthatofEXPLAIN.Theformatandcontentoftheoptimizertracearesubjecttovariation,basedontheversions.
ThefollowingtableshowstheoutputformatoftheEXPLAINstatement:
Column JSONName Details
id select_id TheSELECTidentifier
select_type NoneTheSELECTtype
table table_name Thetablefortheoutputrowpartitions partitions Thematchingpartitionstype access_type Thejointypepossible_keys possible_keys Thepossibleindexestochoosekey key Theindexactuallychosenkey_len key_length Thelengthofthechosenkeyref ref Thecolumnscomparedtotheindexrows rows Estimateofrowstobeexaminedfiltered filtered PercentageofrowsfilteredbytableconditionExtra None AdditionalInformation
Reference:https://dev.mysql.com/doc/refman/8.0/en/explain-output.html#explain-output-column-table
OptimizingtablesDatabase tables are the most basic building blocks for any database. In thissectionofthechapter,wewillfocusonoptimizingtables.Thesectionprovidesdetailed guidelines for improving performance through table optimizationtechniques.
OptimizationforInnoDBtables
TheInnoDB storage engine is preferred in production environments in situationswherereliabilityandconcurrencyareimportant.ItisthedefaultstorageengineforMySQLtables.
ThissectionfocusesonoptimizingdatabaseoperationsforInnoDBtables.
ThefollowingaretheguidelinestooptimizeInnoDBtables:
Useof the OPTIMIZETABLE statement should be considered to reorganize thetableandcompact thewastedspaceonce thedata reachesastablesizeorthetablehasincreasedbytensofmegabytes.
ItrequireslessdiskI/Otoperformfulltablescansforreorganizedtables.
The OPTIMIZE TABLE statement copies the data in the table and rebuilds theindexes. It is beneficial because of improved packing of the data withinindexes, and fragmentation reductionwithin the table spaces on the disk.The benefits may vary, depending on the data in each table. It may benoticeable that thegainsaresignificant insomecases,andnot forothers.
Thegainsmayalsodecreaseovertimeuntil thenexttableoptimizationisdone.Theoperationcanbeslow if the table is largeor the indexesbeingrebuiltdonotfitinthebufferpool.
AlongprimarykeyinaInnoDBtablewastesalotofdiskspace.Itshouldbeavoided.
InInnoDBtables,preferenceshouldbegiventotheVARCHARdatatypeinsteadoftheCHARdatatypetostorevariablelengthstrings,orforcolumnswhichareexpected to contain NULL values. A CHAR(N) column always occupies Ncharacters to storedata, even if thevalue is NULL. Smaller tables aremoresuitabletofitinthebufferpoolandreduceddiskI/O.
Considerusinga COMPRESSED row format forbig tables, or tables containinglotsofrepetitivetextornumericdata.
OptimizationforMyISAMtables
Forread-onlyorread-mostlydata,orforlowconcurrencyoperations,theMyISAMstorage engine fits the best. This is because table locks limit the ability toperform simultaneousupdates. In this section, the focuswill beonoptimizingqueriestobeexecutedonMyISAMtables.
ThefollowingaretheguidelinesforspeedingupqueriesonMyISAMtables:
AvoidexecutingcomplexSELECTqueriesonfrequentlyupdatedMyISAMtables.It prevents problemswith table locking that occur because of contentionbetweenwritersandreaders.
TheMyISAM storageenginesupportsconcurrent inserts. If the tabledata filedoesnothavefreeblocksinthemiddle,wecanINSERTnewrowsinitatthesametimethatotherthreadsarereadingfromthetable.Considerusingthetable to avoiddeleting rows if it is important tobe able todo concurrentread-write operations. Another option is to execute OPTIMIZE TABLE todefragment the table after deletion of the rows. This behavior can becontrolledormodifiedbysettingtheconcurrent_insertsystemvariable.
Avoidallvariable-lengthcolumnsforfrequentlychangingMyISAMtables.Thedynamicrowformatisusedbythetableifitincludesevenasinglevariablelengthcolumn.
The myisamchk --sort-index --sort-records=1 command can be used to sort anindex.Italsosortsdataaccordingtotheindex.
This makes the queries run faster if we have unique indexes, based onwhichwewanttoreadallrowsintheorderaccordingtotheindex.Ittakesalongtimewhenwesortalargetablethiswayforthefirsttime.
Ifweusuallyretrieverowsintheorderofexpression1,expression2,andsoon,use ALTER TABLE ... ORDER BY expression1, expression2,.., and so on. Thiswillgive higher performance, if this option is used after extensive changes tothetable.
OptimizationforMEMORYtables
MySQLMEMORYtablesshouldbeconsideredforuseonlyfornoncriticaldatathatisaccessed often and is read-only and rarely updated.The application should bebenchmarkedagainstequivalentInnoDBorMyISAM tablesunderrealisticworkloadstoconfirmthatadditionalperformanceisworththeriskoflosingdata.
Weshouldexaminethekindsofqueriesagainsteachtableforbestperformancewith MEMORY tables.We should also specify the type of use for each associatedindex.ItcanbeaB-treeindexorahashindex.UsetheUSINGBTREEorUSINGHASHclauseontheCREATEINDEXstatement.
LeveragingbufferingandcachingThissection focusesonusingbufferingandcaching techniques to increase theperformance.
InnoDBbufferpooloptimization
TheInnoDBstorageenginemaintainsastorageareaknownasthebufferpool.Itisusedfor
cachingdataandindexesinthememory.ItisimportanttoknowhowtheInnoDBbufferpoolworks,soastotakeadvantageofittokeepfrequentlyaccesseddata
inmemory.ItisanimportantaspectofMySQLtuning.
The following are the general guidelines for improving performance with theInnoDBbufferpool:
In an ideal case, the size of the buffer pool should be set large enough,whileleaving
enoughmemoryforotherprocessesontheservertorunwithoutexcessivepaging.With
larger buffer pools,more InnoDB functions, like an in-memorydatabase. Inthiscase,itreadsdatafromthedisk
once,andthenaccessesthedatafrommemoryinsubsequentreads.
Wecanconsidersplittingthebufferpoolintomanypartsfor64-bitsystemswith
large memory sizes. This minimizes contention for memory duringconcurrentoperations.
Thefrequentlyaccesseddatashouldbekeptinmemory.
ItispossibletocontrolwhenandhowInnoDBperformsread-aheadrequeststoprefetchpagesintothebufferpoolasynchronously.
InnoDBuses tworead-aheadalgorithmsto improveI/Operformance.Linearreadaheadpredicts
whatpagesmightbeneededsoon,basedonthepagesbeingaccessedinthebufferpool
sequentially. Random read ahead predicts when pages might be neededbasedonthepages
in the buffer pool, regardless of the order in which pages are read. Theinnodb_read_ahead_thresholdconfigurationparametercontrolsthesensitivityoflinearreadahead.Wecanenable
randomreadaheadsbysettinginnodb_random_read_aheadtoON.
innodb_buffer_pool_read_ahead determines the number of pages read into theInnoDBbufferpool.innodb_buffer_pool_read_ahead_evicteddeterminesthenumberofpagesreadintothebufferpoolbytheread-aheadbackground
thread that was subsequently evicted without having been accessed byqueries.The
innodb_buffer_pool_read_ahead_rnd determines the number of random readaheadsinitiatedbyInnoDB.
MyISAMkeycache
The MyISAM storage engine incorporates a strategy that is supported by manydatabasemanagementsystemstominimizethediskI/O.Thecachemechanismis employed by MyISAM to keep the most frequently accessed table blocks inmemoryasfollows:
A special structureknownas akey cache ismaintained for indexblocks.Themostusedindexblocksareplacedinthestructurecontaininganumberofblockbuffers.
MySQL relies on the native operating system filesystem cache for datablocks.
Thekey_buffer_sizesystemvariabledetermines thesizeof thekeycache. If it isset to zero, no key cache is used. The key cache is also not used if thekey_buffer_sizevalueistoosmalltoallocatetheminimumorderofblockbuffers.All theblockbuffers in thekeycachestructureareof thesamesize.Thissizecanbeequal to,greater than,or less than the sizeof the table indexblock. Inusualcases,oneofthesetwovaluesisamultipleoftheother.
When it is required to access data fromany table indexblock, the server firstchecks if it is available in some block buffer of the key cache. If the data isavailable,theserveraccessesdatafromthekeycacheratherthanonthedisk.Ifthedata isnotavailable, theserverselectsacacheblockbuffer thatcontainsadifferent table index block and replaces the data in it by copying the requiredtableindexblock.
Theindexdatacanbeaccessedassoonasthenewindexblockisavailableinthecache.
TheMySQLserverfollowstheLeastRecentlyUsed(LRU)strategy.Accordingto it, it selects the least recently used index blockwhile choosing a block forreplacement.ThekeycachemodulecontainsallusedblocksintheLRUchain(aspecial list).Thelist isorderedbythe timeofuse.It is themostrecentlyusedwhentheblockisaccessed.Theblockisplacedattheendofthelist.Blocksatthebeginningofthelistaretheleastrecentlyusedwhentheblocksneedtobereplaced.So,theblockatthetopbecomesthefirstcandidateforeviction.
The block is considered dirty if the block selected for replacement has beenmodified.
Theblockcontentsareflushedtothetableindexfromwhichtheycamepriortoreplacement.
Based on the following conditions, the threads can access key cache bufferssimultaneously:
Thebufferwhichisnotbeingupdatedcanbeaccessedbymultiplesessions
Thebufferwhichisbeingupdatedcausessessionsthatrequirewaitinguntiltheupdateiscompletetouseit
As long as the sessions are independent and do not interfere with eachother, multiple sessions can initiate requests resulting in cache blockreplacements
Inthisway,sharedaccesstothekeycacheimprovesperformancesignificantly.
Summary
In this chapter, we learned, in detail, the techniques to optimize MySQL 8components.
The chapter started with the basics of optimization, including hardware andsoftwareoptimizationguidelines.WealsodiscussedoptimizationguidelinesfortheMySQL8
server and client, database structure, queries, and tables. We also coveredoptimization for tables belonging to different storage engines, such as MyISAM,InnoDB,andMEMORY.Welearnedthetools,suchasEXPLAINandEXPLAINANALYZE,neededto understand the query execution plan. In the later part of the chapter, welearnedbufferingandcachingtechniquestoimproveperformance.
It's time to move on to the next chapter now. The next chapter focuses ontechniques to extend MySQL 8. The chapter will cover in-depth details ofMySQL8plugins,whichhelp toextend thedefaultMySQL8 features. Itwillalsoexplain the services tocall theseplugins.Thechapterwilldiscussaddingnewfunctions,debugging,andportingmethods. It isgoing tobean importantchapterfordatabaseadministrators.
ExtendingMySQL8
Inthepreviouschapter,welearnedhowtooptimizeMySQL8.Wealsolearnedwhat configurationsneed to bedone to achieveoptimization, and alsohow toleveragecachingandbufferingforoptimization.Wewent throughtheusecasestudystepbystepforachievingoptimizationinthefollowingcomponents:
OptimizingMySQL8serverandclient
Optimizingdatastructures
Optimizingqueries
Optimizingtables
In this chapter,wewill learn about extendingMySQL8.Wewill checkwhatMySQL8
components are allowed to extend, and we will look at how to customizeMySQL 8 for specific business needs. You will learn about the fundamentalcomponentspriortoextendingMySQL8andthefeaturesoftheMySQLplugin
API thatwill be used to extendMySQL8. The following is the list of topicscoveredinthischapter:
AnoverviewofextendingMySQL8
Extendingpluginsandusingservicestocallthem
Addingnewfunctions
Debuggingandporting
AnoverviewofextendingMySQL8
In thissection,youwill learnaboutoneof themostexciting topicsonhowtoextendMySQL8asperyourneeds.ThereareseveralcomponentsofMySQL8thatyoushouldunderstandwellpriorattemptingtoextendMySQL8.HereisalistofthecomponentsthatareimportantforextendingMySQL8:
MySQL8internals
MySQL8pluginAPI
MySQL8servicesforcomponentsandplugins
AddingnewfunctionstoMySQL8
DebuggingandportingMySQL8
MySQL8internals
TherearefewthingsyoushouldknowbeforeyoustartworkingontheMySQLcode. To contribute or track MySQL development you should follow theinstructionsfortheinstallationofsourcecodedistributionasperyoursystemoroperating system platform. The source code includes internal documentation,whichisveryimportanttounderstandhowMySQL
internally works from developer's perspective. You can also subscribe to theinternalsmailinglistfromhttps://lists.mysql.com/internals,whichincludespeoplewhoworkonMySQLcode,andyoucanalsodiscusstopicsrelatedtoMySQLdevelopmentorpostingpatches:
MySQL 8 threads: MySQL server creates threads such as connectionmanager threads, signal threads, read and write threads if using InnoDBstorageengine,schedulerthreadstohandleconnection,andreplicationandeventprocessing.
MySQL 8 test suite: MySQL 8 provides the test systems included withUnixsourcedistributiontohelpusersanddevelopersperformingregressiontestingwithMySQLcode.Youcanalsowriteyourowntestcasesusingthe
MySQL8pluginAPI
MySQL 8 provides support for plugin API by which server componentsthemselvescanbecreated.Thepluginscanbeloadedduringserverstartupandcanalsobeloadedandunloadedduringruntime;thereisnoneedtorestart theserver.TheAPIisverygenericinthatitdoesnotspecifywhatpluginscandointerms of limitation but instead they are allowed to do more than build-incomponents. The API supports interfaces for components such as storageenginesplugins,full-textparserplugins,serverextensionsandsoon.
Theplugin interfacemakesuseof theplugin table in theMySQL8database tostore the information about installed plugins permanently by using the INSTALLPLUGIN statement. During the MySQL 8 installation process the plugin table iscreated.Forsingleserverinvocationthepluginscanalsobeinstalledusingthe--plugin--loadoption,butusing thisoptiondoesnot record the installedplugin totheplugintable.
MySQL8 also provides supportAPI for client plugins to be used for specificpurposes such as enabling the server connection by client through differentauthenticationmethods.
MySQL 8 services forcomponentsandplugins
The MySQL 8 server plugins can access and initiate server plugin services;similarly, the server components can also access and request componentservices.TheMySQL8
plugin Services interface complements the API plugin by exposing serverfunctionality, which can be called by plugins. The following are the pluginservicecharacteristics:
The services enable plugins to access the server code using ordinaryfunctioncallsandcanalsocalluser-definedfunctions
Theservicesareportableandcanworkonmultipleplatforms
The services provide versioning support that protects againstincompatibilitiesbetweenpluginsandservices
Theservicesalsoprovidesupportfortestingpluginservices
MySQL provides two services types for plugins and components, listed asfollows:
1. Thelockingservice:Thelockingserviceinterfaceisprovidedattwolevels—that is, at C level and at SQL level. The interface works on locknamespace,lockname,andlockmodeattributes.
2. Thekeyringservice:Thekeyringserviceprovidesaninterfaceforsecurelystoringsensitiveinformationforinternalservercomponentsandpluginstoretrievelater.
AddingnewfunctionstoMySQL8
YoucanaddyourownfunctionstoMySQL8,andthiscanbedonewithanyoneof the three supported types of function. The new function can be called thesame way we invoke the built-in functions such as ABS(), and that is trueirrespectiveofwhichfunctiontypeyouhavenewlyadded.ThefollowinglistisofthesupportedthreetypesofnewfunctioninMySQL8:
1. Addingafunctionthroughtheuser-definedfunction(UDF)interface.
2. Addingafunctionasnative(built-in)MySQLfunction.
3. Addingafunctionbycreatingastoredfunction.
DebuggingandportingMySQL8
PortingMySQL 8 to other operating systems is currently supported bymanyoperatingsystems;thelistofsupportedoperatingsystemsisprovidedathttp://www.mysql.com/support/supportedplatforms/database.html. In case you have added a newportandarerunningintoproblemswiththenewport,youmightusedebuggingofMySQL8.
There are different possible ways to start debugging based on where you arerunningintotheproblems—theycouldbeinMySQLserverorinMySQLclient.Dependingontheproblem'slocation,youcanstartdebugginginMySQLserveror client respectively and also get help from the DBUG package to trace theprogram'sactivities.
Extending plugins and usingservicestocallthem
In this section, you will gain an understanding of how the plugin API, itsinterface, and the MySQL services interact with one another and provideextensionsinMySQL8.
ThepluginsarealsoconsideredascomponentsintheMySQL8architecture,andthereforeyoucanuse them toprovidepluggable features.ThepluginAPIandthepluginservicesinterfaceshavethefollowingdifferences:
ThepluginAPIenablespluginsthatwillbeusedbytheserver.Thecallingandinvokingofpluginsisinitiatedbytheserver,sothepluginscanextendthe server's functionality or can register themselves in order to receiveserverprocessingnotifications.
The plugin services interface allows plugins to call the server code. Thecallingandinvokingofservicefunctionsisinitiatedbythepluginssothatthe same server functionality can be leveraged bymany plugins withoutrequiringindividualimplementationforthefunctionality.
Writingplugins
To create a plugin library, providing the required descriptor information is amust,asitspecifieswhichpluginsthelibraryfilecontains.Writingtheinterfacefunctionforeachofthepluginsspecifiedisalsonecessary.
Everyserverpluginmusthaveageneraldescriptorprovidinginformationtotheplugin APIs, and a type specific descriptor providing information about theinterface for specified plugin types. The structure for specifying a generaldescriptor is the same for all theplugin types, and the type specificdescriptorcan vary based on the requirements of the plugin's behavior or function. Theserverplugininterfaceallowspluginstoexposesystemvariablesandstatus.
Client-sidepluginshaveaslightlydifferentarchitecturethanthatofserversideplugins.Forexample,eachpluginmusthavedescriptorinformation,butthereisnoseparatedivisionbetweengeneralandtypespecificdescriptors.
PluginscanbewritteninCorC++oranyotherlanguagethatcanuseCcallingconventions.
Pluginsareloadedandunloadeddynamically,hencetheoperatingsystemmustdynamically support where you have dynamically compiled the callingapplication.Specifically,forserverpluginsthismeansthatmysqldmustbelinked
dynamically.
Aswecannotbesureofwhatapplicationwilluse theplugin, thedependencieson the symbolsof the callingapplication shouldbeavoidedbytheclientpluginwriters.
The following are the types of supported plugin creations that can implementseveralcapabilities:
Authentication
Passwordvalidationandstrengthchecking
Protocoltracing
Queryrewriting
Securekeyringstorageandretrieval
Storageengines
Full-textparsers
Daemons
INFORMATION_SCHEMAtables
Semisynchronousreplication
Auditing
Componentandpluginservices
YoucanidentifythecomponentservicesandfunctionsprovidedbyMySQLbylooking into the include/mysql/components and respective services directories of theMySQL8sourcedistribution.
Similarly, you can identify the plugin services and functions provided byMySQL by looking into the include/mysql directory of the MySQL 8 sourcedistributionandtherelevantfilesasfollows:
Theplugin.hfileincludestheservices.hfile,whichservices.hfilecontainsalltheavailableservice-specificheaderfileswithinit
Service-specificheaderfileswillhavenamesintheformofservice_xxx.h
ThefollowingisalistofavailablecomponentservicesinMySQL8:
component_sys_variable_register, component_sys_variable_unregister: For registeringandunregisteringsystemvariables
log_builtins,log_builtins_string:Forlogcomponentsservices
mysql_service_udf_registration, mysql_service_udf_registration_aggregate: Forenablingregistrationandunregistrationofscalarandaggregateuser-definedfunctionsincomponentsandplugins
mysql_string:ForstringserviceAPIs
pfs_plugin_table:FordynamicPerformanceSchematablemanipulation
ThefollowingislistofavailablepluginsservicesinMySQL8:
get_sysvar_source:Forretrievingsystemvariablesettings
locking_service: For lock implementation with C language and SQL levelinterfaces,havingtheattributesnamespace,name,andmode
my_plugin_log_service:Forwritingerrorsmessagestologs
my_snprintf: For string formatting to keep the output consistent acrossplatforms
status_variable_registration:Forregisteringthestatusvariable
my_thd_scheduler:Forthreadschedulerselection
mysql_keyring:Forkeyringstorageservice
mysql_password_policy:Forpasswordstrengthandvalidationchecking
plugin_registry_service: For accessing the component registry and relatedservices
security_context:Formanagingthreadsecuritycontexts
thd_alloc:Formemoryallocation
thd_wait:Forreportingtosleeporstall
Now,youhaveaclearunderstandingofpluginservicesandcomponentservices.MySQL
8 provides the following types of services to support plugins and componentsservices:
1. Thelockingservice
2. Thekeyringservice
Thefollowingsectionsgivedetailedinformationonbothtypesofservices.
Thelockingservice
Thelockingserviceinterfaceisprovidedattwolevels:ClevelandatSQLlevel.
The interface works on the lock namespace, lock name, and lock modeattributes. The C language interface is callable as a plugin service from user-definedfunctionsorserverplugins,andtheSQLlevelinterfaceisusedassetofuser-definedfunctions,beingmappedtocalltheserviceroutines.
Thefollowingarethecharacteristicsofthelockinginterface:
Lock namespace, lock name, and lockmode are three three attributes oflocks.
Locks are identified by forming a lock namespace and lock namecombination.
Lockmodecanbeeitherreadorwrite.Readlocksaresharedwhereaswritelocksareexclusive.
Lock names and namespaces can have amaximum of 64 characters and
mustbenon-NULL
andnon-emptystrings.
Locknamesandnamespacearetreatedasbinarystringssocomparisonwillbecase-sensitive.
Functionsareprovidedtoacquireandreleaselocksanddonotrequireanyspecialprivilegestocallthefunctions.
Detectsdeadlockduringlockacquisitioncallsindifferentsessions;acallerischosenandterminatedforitslockacquisitionrequestandcallersessionsholdingreadlocksarepreferredoverthesessionsholdingwritelocks.
A typical session can request formultiple locks acquisitionwith a singlelock acquisition call. It provides atomic behavior for the request andsucceedsifalllocksareacquiredorfailsifanyofthelockacquisitionsfail.
Multiple locksfor thesame lock identifiercanbeacquiredby thesessionwhere the lock instances canbewrite locks, read locks,or amixofbothreadandwritelocks.
Acquired locks are released from the session by explicitly calling therelease-lockfunction,orimplicitlyifthesessiongetsterminated.
All locks in the given namespace when released are released togetherwithinthesession.
<strong>boolmy_key_generate(constchar*key_id,constchar*key_type,<br/>constchar*user_id,size_tkey_len)</strong>
<strong>boolmy_key_fetch(constchar*key_id,constchar**key_type,<br/>constchar*user_id,void**key,size_t*key_len)</strong>
<strong>boolmy_key_remove(constchar*key_id,constchar*user_id)</strong>
<strong>boolmy_key_store(constchar*key_id,constchar*key_type,<br/>constchar*user_id,void*key,size_tkey_len)</strong>
Addingnewfunctions
NewfunctionscanbeaddedwithanyofthethreesupportedtypesinMySQL8.Each of the types have their own advantages and disadvantages. Where andwhich type of function should be added or implemented depends on therequirementsofthefunction.
ThefollowingisthelistofthesupportedthreetypesofnewfunctioninMySQL8,whichwewilllookatinthefollowingsection:
1. Addingafunctionthroughtheuser-definedfunctioninterface.
2. Addingafunctionasanative(built-in)MySQLfunction.
3. Addingafunctionbycreatingastoredfunction.
Features of a user-definedfunctioninterface
A user-defined function interface provides independent capabilities to a userpurposefunction.
ThefollowingfeaturesandcapabilitiesareprovidedbytheMySQLinterfaceforuser-definedfunctions:
Functions can accept arguments of integer, string, or real values and canreturnvaluesforthesametypes
Simplefunctionscanbedefinedtooperateonasinglerowata timeor itcanbeaggregatefunctionstooperateongroupsofrows
Functionsaregiveninformationtoenablethemsothattheycancheckthetypes,names,andnumbersofargumentspassed
Beforepassingargumentstothegivenfunction,youcanalsoaskMySQLtocoercearguments
Adding a new user-definedfunction
TheUDFfunctionsmustbewritten inCorC++and theunderlyingoperatingsystemmustsupportdynamicloadingbehavior.Thereisafile,sql/udf_example.cc,that defines five UDF functions and it's included in the MySQL sourcedistributions.
Analyzing the filewill letyouknowhowcallingconventionswork forUDFs.User-defined function related symbols and data structures are defined in theinclude/mysql_com.hfileandthefileisincludedinthemysql.hheaderfile.
Typicalcodecontained in theUDFsgetsexecuted in therunningserver,soallconstraints are applicable when writing UDF code—server code. Currentlyapplicableconstraintsmayget revisedwhenaserver isupgraded,and thiscanpossiblyresultintotheneedtorewriteUDFcode,soitisessentialtobecarefulwhenwritingcodefortheUDF.
InordertouseUDF,linkingmysqlddynamicallyisamust.Foranyfunctiontobeused in SQL statements there must be underlying C or C++ functions. Theconvention for separating SQL and C/C++ code is followed where xxx() in
uppercaseindicatesanSQLfunctioncallwhereasxxx()withlowercaseindicatesaC/C++functioncall.
EncapsulateyourCfunctionasshowninfollowingsentencewhenyouareusingC++: extern"C"{ ... }This way it is ensured thatyour C++ function names are readable in the completed user-definedfunction.
TowriteandimplementtheinterfacefunctionnameXXX(),themainfunctionxxx()isamustandadditionallyrequiresoneormorefunctiontobeimplementedfromthefollowing:
xxx():Themainfunctionwherethefunctionresultisbeingproduced
xxx_init():The initialization function for themain function xxx(), it canbeusedforanyofthefollowingpurposes:
CheckingnumberofargumentstobepassedontoXXX()
Verifying argument types with a declaration when calling the mainfunction
Allocatingmemorytothemainfunctionwheneverrequired
Result'smaximumlengthverification
Settingadecimalnumberlimitformaximumintheresult
SpecifyingwhethertheresultcanbeNULLornot
xxx_deinit(): Represents deinitialization for the main function anddeallocatesmemoryifanyisallocatedbytheinitializationfunctionforthemainfunction
AggregateUDFsarehandledasinthefollowingsequenceinMySQL8:
1. Call xxx_init() so that it allocates the required memory to store resultinformation.
2. Sortthetable/resultasspecifiedbytheGROUPBYfunction.
3. Callxxx_clear()sothatitresetthecurrentaggregatevalueforthefirstrowineachnewgroup.
4. Callxxx_add()thataddstheargumenttothecurrentaggregatevalue.
5. Callxxx() toget the resultofaggregatedataongroupbychangesorafterprocessingthelastrow.
6. Repeatsteps3-5untilallspecified/resultedrowsareprocessed.
7. Callxxx_deinit()tofreeanyallocatedmemoryfortheUDF.
All the functionsmust be thread-safe, including themain function as well asother additional functions as required, along with the initialization anddeinitializationfunctions.
Similartotheabovesequence,thefollowingareimportantaspectsthatneedtobetakencareofwhileaddingnewuser-definedfunctions:
UDFargumentprocessing
UDFreturnvaluesanderrorhandling
UDFcompilingandinstalling
UDFsecurityprecautions
<strong>doubleItem_func_newname::val()</strong><br/><strong><span>longlongItem_func_newname::val_int()<br/></span>String*Item_func_newname::Str(String*str)</strong>
Ifyourobjectisinheritedfromanyofthestandarditemsthenyouprobablyneedtodefineonlyoneoftheprecedingfunctions,astheparentobjectwilltakecareotherofthefunction.YoucanrefertotheItem_str_funcclassthathasdefinedtheval()functionthatexecutestheatof()functiononthereturnedvalueofthe::str()function.
5. Ifthefunctionisnondeterministic—thatis,ifthereturnedresultvariesatdifferentinvocationsforfixedgivenarguments-thenthefollowingstatementneedstobeincludedintheitemconstructor,indicatingthatthefunctionresultsshouldnotbecached:current_thd->lex->safe_to_cache_query=0;.
6. Youprobablyalsoneedtodefinethefollowingobjectfunctionforyournativefunction:
voidItem_func_newname::fix_length_and_dec()
Thefunctionshouldatleastincludethemax_lengthcalculationonthegivenargumentsYoushouldalsosetmaybenull=0ifyourmainfunctioncannotreturnanyNULLvaluesYoucanrefertoItem_func_mod::fix_length_and_decforthesame
Threadsafetyisamustforallfunctions.Youshouldnotbeusinganystaticorglobalvariablesinthefunctionswithoutbeingprotectedbymutexes.
Debuggingandporting
PortingMySQL 8 to other operating systems is currently supported bymanyoperatingsystems.Thelistofthelatestsupportedoperatingsystemsisprovidedathttp://www.mysql.com/support/supportedplatforms/database.html. If you have added orattemptedtoaddnewports(supportedplatforms)andarerunningintoproblems,youmightusedebuggingofMySQL8tofindandfixtheproblems.
First,youshouldgetthetestprogrammysys/thr_locktoworkbeforedebuggingmysqld.Thismakessurethatyourthreadinstallationcanhavearemotechancetowork!
Therearedifferentpossibilitiesforstartingdebugging,basedonwhereyouarerunning into theproblems - it couldbe inMySQLserveror inMySQLclient.Dependingontheproblem'slocationyoucanstartdebugginginMySQLserverorMySQLclient respectively,and for tracing theprogram'sactivitiesyouwillgethelpfromtheDEBUGpackage.
TheMySQL source code includes internal documentationwrittenusingDoxygen,whichisveryhelpfulinunderstandingthedeveloperperspectiveonhowMySQLworks.
Inthissection,youwillseedetailedinformationonthefollowingtopics:
DebuggingMySQLserver
If you are using some of very new functionality inMySQL and facing someissues—let's say the server is crashing—youcan try running mysqldwith the --skip-new option. This option tells the MySQL server to disable all new andpotentiallyunsafefunctionality.
In caseswhere mysqld is not getting started, verify the my.cnf files, as they caninterferewiththesetup!Youcanchecktheargumentsinmy.cnfwiththemysqld--print-defaults option and then start mysqld with the --no-defaults option to avoidusingthem.
Incaseswheremysqldstarts toeatupmemoryorCPUorhangs,youcancheckmysqladminprocessliststatusandfindoutifaqueryexecutedbysomeoneistakingalongtime.Incaseswhereyouarefacingperformanceissuesorproblemsandnewclientsarenotabletoconnect,youcanusemysqladmin-i10processliststatus.
Youcanalsousethedebugcommandmysqladmin,whichdumpsinformationaboutquery usage,memory usage, and locks in use to theMySQL log file and cansolve someproblems foryou.Thiscommandalsoworks incaseyouhavenotcompiledMySQLfordebugging,providingsomeusefulinformation.
Incaseswhereyouarefacinganyissuewiththetablegettingslower,youshould
try to optimize the table using myisamchk or OPTIMIZE_TABLE. You should probablychecktheslowqueries,ifthereareany,usingEXPLAINtofindandfixtheproblemwithqueries.
ThefollowingaretheimportantareastoconsiderwhendebugginginMySQL8:
CompilingMySQLfordebugging:Incaseofveryspecificproblemsyoucanalways try todebugMySQL.Todo thatyoumustconfigureMySQLwith the -DWITH_DEBUG=1 option. The debugging configuration automaticallyenableslotsofextrasafetycheckfunctionsthatmonitorthehealthofmysqld.
Creating trace files: You can attempt to find the problem by creating atrace file. To do that you must have mysqld compiled with debuggingsupport.Youcan thenuse the --debug option,whichwill add trace logs in/tmp/mysqld.traceonUnixand\mysqld.traceonWindows.
Using WER with PDB to create a Windows crashdump: ProgramdatabasefilesareincludedintheZIParchivedebugbinariesandtestsuiteas a separate distribution ofMySQL. These files provide information ondebugging for a MySQL installation problem. They can be used withWinDbgorVisualStudiotodebugmysqld.
Debugging mysqld under gdb: You can use this option when you arefacingissueswiththreadsorwhenthemysqldserverhangspriortoreadyforconnections.
Using a stack trace: You can also use this option when mysqld diesunexpectedlyandfindouttheproblem.
Using server logs to find causes of errors in mysqld: You can use thisoptionbyenablingthegeneralquerylog-prior to that,youshouldcheckallyourtablesusingthemyisamchkutilityandverifyifthereareanyproblemsfromthelogs.
Making a test case if you experience table corruption: This option isusedwhenyouare facingan issuewith tablecorruptionand isapplicableonlytoMyISAMtables.
DebuggingMySQLclientIn cases where you are facing an issue inMySQL client you can also debugwithinMySQLclientaswell,butinordertodosoyoumusthavetheintegrateddebug package. You need to configure MySQL with -DWITH_DEBUG=1 to enabledebugginginMySQLclient.
Prior to running MySQL client, you should set the environment variableMYSQL_DEBUGasfollows:shell>MYSQL_DEBUG=d:t:O,/tmp/client.traceshell>exportMYSQL_DEBUG
ThismakesMySQL client generate a trace file in /tmp/client.trace forUnix or\client.traceforWindows.
Incaseswhereyouhaveproblemswithyourownclientcode,youcanattempttoconnect to the server by running your query using the client that is known towork.Fordoingthisyoushouldrunmysqldindebuggingmode:shell>mysql --debug=d:t:O,/tmp/client.trace
This tracewillprovideuseful information ifyouwant tomailabugreport fortheproblem.
Incaseswhereyourclientcrashesatsomelegallookingcode,youcancheckthatyourmysql.hheaderfileincludesfilematcheswithyourMySQLlibraryfile.Thisis one of the very common mistakes, using an older mysql.h file from an oldMySQLinstallationwithanewMySQLlibrary,resultinginthisissue.
<strong><spanclass="tokenoperator">--</span>debug<spanclass="tokenoperator">=</span>d<spanclass="tokenoperator">:</span>t<br/><spanclass="tokenoperator">--</span>debug<spanclass="tokenoperator">=</span>d<spanclass="tokenoperator">:</span>f<spanclass="tokenpunctuation">,</span>main<spanclass="tokenpunctuation">,</span>subr1<spanclass="tokenoperator">:</span>F<spanclass="tokenoperator">:</span>L<spanclass="tokenoperator">:</span>t<spanclass="tokenpunctuation">,</span>20<br/><spanclass="tokenoperator">--</span>debug<spanclass="tokenoperator">=</span>d<spanclass="tokenpunctuation">,</span>input<spanclass="tokenpunctuation">,</span>output<spanclass="tokenpunctuation">,</span>files<spanclass="tokenoperator">:</span>n<br/><spanclass="tokenoperator">--</span>debug<spanclass="tokenoperator">=</span>d<spanclass="tokenoperator">:</span>t<spanclass="tokenoperator">:</span>i<spanclass="tokenoperator">:</span>O<spanclass="tokenpunctuation">,</span>\\mysqld<spanclass="tokenpunctuation">.</span>trace</strong>
Summary
Inthischapter,youlearnedhowtoextendMySQL8throughcustomfunctionsandAPIs.
Youalsogottoknowaboutwritingfunctionsandtheassociatedcharacteristicsof the plugin services and APIs. You can now create your own function orplugin,catertospecificbusinessrequirements,andalsodebugifafunctiondoesnotworkasperexpectations,andtestwhetheritdoes.
Innextchapter,youwilllearnaboutMySQL8bestpracticesandbenchmarkinginMySQL
8.Youwilllearnaboutbenchmarkingandtoolsusedforbenchmarking.YouwillalsolearnbestpracticesforsomeofveryimportantfeaturesofMySQL8,suchasmemcached,replication,datapartitioning,andindexing.
MySQL 8 Best Practices andBenchmarking
Inthepreviouschapter,youlearnedhowtoextendMySQL8.Itcoveredalotofinterestingaspects,suchasextendingpluginsandcallingthembyusingservicesinMySQL8,addinganddebuggingnewfunctionstoMySQL8,andsoon.Inthischapter,wewillgothroughthebestpracticesofMySQL8,whichisamuch-awaited version that promises to address many of the shortfalls of the priorversionsandhasexcitingnewfeatures.
MySQL8promisesnot tobejustastandalonedatabase,but itwillalsoplayasignificantroleinvariousareas,includingbigdatasolutions.Wewilllearnhowbest practices can be implemented for optimal use of features in MySQL 8.Benchmarkingwillenhanceourunderstandingfurther.
Wewillcoverthefollowingtopicsinthischapter:
MySQLbenchmarkingandtools
Bestpracticesforthememcached
Bestpracticesforreplication
Bestpracticesfordatapartitioning
Bestpracticesforqueriesandindexing
Due to prominent optimizations and changes,MySQL 8 advanced its versiondirectlyfromthereleaseofMySQL5.7.MySQL8willnothavethelimitationoffiles,whichwaspreviously restricting thenumberof databases that you couldhave.Therearemanymoreexcitingfeatures,whichwehavecoveredinChapter1, Introduction toMySQL 8. MySQL 8 can now store millions of tables in adatabase.Itwillalsomakemodificationstotablesswiftly.
I am excited to go through this chapter, asMySQL 8 best practices not onlyimpactyourdatabaseperformance,scalability,security,andavailability,butwillalso,on thewhole,exposehowyoursystemperformsfor theenduser.This isourendgoal, isn't it?Let's lookatsomebenchmarksthathavebeenderivedinour test lab, which will raise your eyebrows for sure:
MySQLbenchmarkingandtools
WehavegonethroughvariousnewfeaturesandimprovementsinMySQL8.Itmakesusmoreexcited,asperformanceisalwayswhatwecrave.WithMySQL8notbeinggenerallyavailableyet,Oraclehasn'tpublisheditsbenchmarkresults.Wedidn'twaitforittodosoandcarriedoutourownanalysisinafewareas.
Configuration best practices ofMySQL is the cherry on the cake;without thecherry,thecakeseemsincomplete.Inadditiontoconfigurations,benchmarkinghelps us validate and find bottlenecks and address them. Let's look at a fewspecific areas thatwill help us understand the best practices for configurationandperformancebenchmarking.
ResourceutilizationIOactivity,CPU,andmemoryusageissomethingthatyoushouldnotmissout.These metrics help us know how the system is performing while doingbenchmarking and at the time of scaling. It also helps us derive impacts pertransaction.
Stretching your benchmarkingtimelines
We may often like to have a quick glance at performance metrics; however,ensuringthatMySQLbehavesinthesamewayforalongerdurationoftestingisalsoakeyelement.Thereissomebasicstuffthatmightimpactonperformancewhen you stretch your benchmark timelines, such as memory fragmentation,degradation of IO, impact after data accumulation, cachemanagement, and soon.
Wedon'twantourdatabasetogetrestartedjusttocleanupjunkitems,correct?
Therefore, it issuggested to runbenchmarkingfora longdurationforstabilityandperformancevalidation.
Replicatingproductionsettings
Let's benchmark in a production-replicated environment. Wait! Let's disabledatabase replication in a replica environment until we are done withbenchmarking.Gotcha!
Wehavegotsomegoodnumbers!
Itoftenhappensthatwedon'tsimulateeverythingcompletelythatwearegoingto configure in theproduction environment. It couldprove tobe costly, aswemightunintentionallybebenchmarkingsomethinginanenvironmentthatmighthave an adverse impactwhen it's in production.Replicate production settings,data, workload, and so on in your replicated environment while you dobenchmarking.
Consistency of throughput andlatencyThroughput and latency go hand in hand. It is important to keep your eyesprimarily focused on throughput; however, latency over time might besomethingtolookoutfor.
Performancedips,slowness,orstallswerenoticedinInnoDBinitsearlierdays.Ithas improveda lot since then,but as theremightbeothercasesdependingonyour workload, it is always good to keep an eye on throughput along withlatency.
SysbenchcandomoreSysbench is a wonderful tool to simulate your workloads, whether it bethousands of tables, transaction intensive, data in-memory, and so on. It is asplendidtooltosimulateandgivesyounicerepresentation.
VirtualizationworldIwould like tokeep this simple;baremetalascompared tovirtualization isn'tthesame.Hence,whiledoingbenchmarking,measureyourresourcesaccordingtoyourenvironment.Youmightbe surprised to see thedifference in results ifyoucompareboth.
ConcurrencyBig data is seated on heavy data workload; high concurrency is important.MySQL 8 is extending itsmaximumCPU core support in every new release,optimizing concurrency based on your requirements and hardware resourcesshouldbetakencareof.
HiddenworkloadsDonotmissoutfactorsthatruninthebackground,suchasreportingforbigdataanalytics,backups,andon-the-flyoperationswhileyouarebenchmarking.Theimpactofsuchhiddenworkloadsorobsoletebenchmarkingworkloadscanmakeyourdays(andnights)miserable.
NervesofyourqueryOops!Didwemisstheoptimizer?Notyet.Anoptimizerisapowerfultoolthatwillreadthenervesofyourqueryandproviderecommendations.It'satoolthatIusebeforemakingchangestoaqueryinproduction.It'sasaviorwhenyouhavecomplexqueriestobeoptimized.
These are a few areas that we should look out for. Let's now look at a fewbenchmarks that we did on MySQL 8 and compare them with the ones onMySQL5.7.
BenchmarksTo start with, let's fetch all the column names from all the InnoDB tables. Thefollowingisthequerythatweexecuted:
SELECTt.table_schema,t.table_name,c.column_name
FROMinformation_schema.tablest,
information_schema.columnsc
WHEREt.table_schema=c.table_schema
ANDt.table_name=c.table_name
ANDt.engine='InnoDB';
The following figure showshowMySQL8performeda thousand times fasterwhenhavingfourinstances:
Followingthis,wealsoperformedabenchmarktofindstatictablemetadata.Thefollowingisthequerythatweexecuted:
SELECTTABLE_SCHEMA,TABLE_NAME,TABLE_TYPE,ENGINE,ROW_FORMAT
FROMINFORMATION_SCHEMA.TABLES
WHERETABLE_SCHEMALIKE'chintan%';
The following figure shows howMySQL 8 performed around 30 times fasterthanMySQL5.7:
Itmadeuseager togointoabitmoredetail.So,wethoughtofdoingonelasttesttofinddynamictablemetadata.
Thefollowingisthequerythatweexecuted:
SELECTTABLE_ROWS
FROMINFORMATION_SCHEMA.TABLES
WHERETABLE_SCHEMALIKE'chintan%';
The following figure shows howMySQL 8 performed around 30 times fasterthanMySQL5.7:
MySQL8.0bringsenormousperformanceimprovementstothetable.Scalingtoone million tables, which is a need for many big data requirements, is nowachievable.WelookforwardtomanymorebenchmarksbeingofficiallyreleasedonceMySQL8isavailableforgeneralpurposes.
Let'snowlookatournexttopic,whichwillmakeyourlifeeasier.It'sallabouttakingthingsintoconsiderationforbestpracticesofmemcached.
BestpracticesformemcachedMultiple get operations are now possible with the InnoDB memcached plugin,whichwill really help in improving the read performance.Now,multiple keyvalue pairs can be fetched in a single memcached query. Frequentcommunicationtraffichasalsobeenminimized,aswecangetmultipledatainasingleshot.
Thekeytakeawaysthatyoushouldconsiderformemcachedconfigurationbestpracticesarewhatwewillbegoingthroughnow.
ResourceallocationMemory allocation for memcached shouldn't be allocated over the availablephysicalmemoryorwithoutconsideringotherresourcesthatwouldbeutilizingmemory. If we over-allocatememory, there is a high chance thatmemcachedwould havememory allocated from the swap space. Thismay lead to delayswhileinsertingorfetchingvaluesbecausetheswapspaceisstoredonthedisk,whichisslowerthanin-memory.
OperatingsystemarchitectureAs theoperating systemarchitecturehas32-bits,oneneeds tobecautious.Aswe know, there are limitations to provision resources in a 32-bit operatingsystemarchitecture.
Similarly, memcached with 4 GB RAM with a 32-bit operating systemarchitectureshouldn'tbesetmorethan3.5GBRAM,asitcanbehavestrangelyinperformanceandcanalsoresultincrashes.
Defaultconfigurations
Somekey default configuration parameters should always be fine-tuned basedonyourneeds:
Memory allocation: By default, this is 64 MB; instead it should bereconfiguredbasedonyourrequirementsandtesting
Connections: By default, this is 1,024 concurrent connections; instead itshouldbereconfiguredbasedonyourrequirementsandtesting
Port:Bydefault,thislistensonport11211;insteaditshouldlistentoanotherportforsecuritypurposes
Network interface:Bydefault, thisacceptsconnectionsfromallnetworkinterfaces;insteaditshouldbelimitedforsecuritypurposes
MaxobjectsizeYoushouldlookatconfiguringthemaximumobjectsize,whichbydefaultis1MB.
However, it canbebumpedup to128MB. It ispurelybasedonwhat typeofdatayouaregoingtostoreand,accordingly,itsmaximumobjectsizeshouldbeallowed. Allowing overhead data to be stored in memcached can have anadverse impact, as theremaybemuchmoredata to retrieve,which can causefailures.
Backlogqueuelimit
Thebacklogqueue limit isallabout thenumberofconnections tomemcachedthat should be kept in queue if it reaches the limit of allowed connections.Ideally,yournumberofconnectionsallowedshouldbeconfiguredinawaythatshouldsuffice formostofyourneeds.Thebacklogqueue limitcanbehelpfulwhenthereisanunexpectedpeakloadonmemcached.Ideally,itshouldnotgobeyond20%ofthetotalconnectionsoritcouldimpacttheexperienceofsystemfetchinginformationfrommemcachedbecauseofheavydelays.
LargepagessupportOnsystemsthatsupportlargememorypages,youshouldenablememcachedtoleverage them. Large pages support helps allocate a large data chunk to storedataandalsoreducesthenumberofcachesmissedcallsusingthis.
SensitivedataStoring sensitive data in memcached could be a security threat, as somebodywith access to memcached could view the sensitive information. You shouldobviously take precautions to limit the exposure ofmemcached.You can alsohavesensitiveinformationencryptedbeforestoringitonmemcached.
RestrictingexposureMemcacheddoesn'thavemanysecurityfeaturesbuiltin.Onemeasureinvolvesexposingmemcachedaccesswithintherequiredboundaries.Ifyourapplicationserver needs to talk tomemcached, it only allowsmemcached to be accessedfrom that server with the help of system firewall rules, such as IP Tables orsimilartechniques.
FailoverMemcacheddoesn'thavegoodfailovertechniques.Itissuggestedthatyouhaveyour application configured in a way to failover to an unavailable node andregeneratedataintoanotherinstance.Itisgoodtohaveatleasttwomemcachedconfiguredtoavoidfailureowingtotheunavailabilityoftheinstance.
NamespacesYou can leverage namespaces provided by memcached, which basically addsprefixestothedatabeforestoringit inmemcached.Itcanhelpwhenyouhavemultiple applications talking to memcached. This is helpful and, using somebasicprinciplesofnamingconventions,youcanderiveasolution.Ifthereisdatathatisstoringfirstnamesandlastnames,youcanuseprefixes,suchasFNandLN,respectively.Thiswouldhelpyoueasilyidentifyandretrievedatafromtheapplication.
Cachingmechanism
One of the easiestways to start leveraging caching inmemcached is to use atwo-columntable;youcanleveragenamespacesprovidedbymemcached,whichbasicallyaddsprefixes.
Thefirstcolumnswouldbeaprimarykey,anddatabaseschemashouldbe theaddressrequirementofauniqueidentifierwiththehelpofprimarykeymappingalongwithuniqueconstraints.Incaseyouwanttohaveasingleitemvaluebycombining multiple column values, you should make sure you chooseappropriatedatatypes.
QuerieswithasingleWHEREclausecanbemappedeasilyintomemcachedlookupswhileusing=orINoperatorsinthequeriesthemselves.IncaseswheremultipleWHERE clauses areusedor complexoperations areparsed, such as <,>,LIKE, andBETWEEN,memcachedwouldgetyou throughchallenges. It is suggested thatyouhave such complex operations using traditional SQL queries added to yourdatabase.
ItwouldbebeneficialtocacheentireobjectsinmemcachedinsteadofoptingtocacheindividualrowsfromMySQL8.Forinstance,forabloggingwebsite,youshouldcachetheentireobjectoftheblogportinmemcached.
MemcachedgeneralstatisticsTohelpyouunderstand the statisticsofmemcachedbetter,wewillprovideanoverviewofhealthandperformance.Statisticsreturnedbymemcachedandtheirmeaning are shown in the following table:Terms used to define the value foreachofthestatisticsare:
32u:32-bitunsignedinteger64u:64-bitunsignedinteger32u:32u:Two32-bitunsignedintegersseparatedbyacolonString:Characterstring
Statistic Datatype Description
pid 32u ProcessIDofthememcachedinstance.
uptime 32u Uptime(inseconds)forthismemcachedinstance.
time 32u Currenttime(asepoch).
version string Versionstringofthisinstance.
pointer_size string Sizeofpointers for thishost specified inbits (32or64).
rusage_user 32u:32u Total user time for this instance(seconds:microseconds).
Total system time for this instance
rusage_system 32u:32u (seconds:microseconds).
curr_items 32u Currentnumberofitemsstoredbythisinstance.
total_items 32u Totalnumberofitemsstoredduringthelifeofthisinstance.
bytes 64u Current number of bytes used by this server tostoreitems.
curr_connections 32u Currentnumberofopenconnections.
total_connections 32u Total number of connections opened since theserverstartedrunning.
connection_structures 32u Number of connection structures allocated by theserver.
cmd_get 64u Totalnumberofretrievalrequests(getoperations).
cmd_set 64u Totalnumberofstoragerequests(setoperations).
get_hits 64u Number of keys that have been requested andfoundpresent.
get_misses 64u Numberofitemsthathavebeenrequestedandnotfound.
delete_hits 64u Numberofkeysthathavebeendeletedandfoundpresent.
delete_misses 64u Number of items that have been delete and notfound.
incr_hits 64u Number of keys that have been incremented andfoundpresent.
incr_misses 64u Numberof items thathavebeen incrementedandnotfound.
decr_hits 64u Number of keys that have been decremented andfoundpresent.
decr_misses 64u Numberofitemsthathavebeendecrementedandnotfound.
cas_hits 64u Number of keys that have been compared andswappedandfoundpresent.
cas_misses 64u Number of items that have been compared andswappedandnotfound.
cas_badvalue 64uNumber of keys that have been compared andswapped, but the comparison (original) valuedidnotmatchthesuppliedvalue.
evictions 64u Numberofvaliditemsremovedfromcachetofreememoryfornewitems.
bytes_read 64uTotal number of bytes read by this server fromnetwork.
bytes_written 64u Total number of bytes sent by this server tonetwork.
limit_maxbytes 32u Numberofbytesthisserverispermittedtouseforstorage.
threads 32u Numberofworkerthreadsrequested.
conn_yields 64u Numberofyieldsforconnections(relatedtothe-Roption).
Reference:https://dev.mysql.com/doc/refman/8.0/en/ha-memcached-stats-general.html
These are a few useful items that should be kept handy for best practices ofmemcached. It'snow time forus tomoveaheadand lookatbestpractices forreplication.
BestpracticesforreplicationMySQL8hasmadesomegreatimprovementsonthereplicationside.MySQL8is all about scalability, performance, and securitywith the utmost integrity ofdata,whichisexpectedtobeagame-changerinbigdatatoo.
ThroughputingroupreplicationGroupreplicationbasically takescareofcommitting transactionsoncemostofthemembers in group replication have acknowledged the transaction receivedconcurrently.
This results in a better throughput if the overall number of writes doesn'texceeding the capacity of themembers in group replication. If there is a casewherecapacity isnotplannedappropriately,youwouldnotice lagsonaffectedmembersascomparedtoothermembersinthegroup.
InfrastructuresizingInfrastructure sizing is a common success factor for performance and the bestpractices checklist. If infrastructure sizing is not proper or uneven across thenodes in group replication, it could adversely impact the replicationfundamentalstopology.Eachcomponentshouldbeconsideredwhileconsideringthethroughputrequiredfromthecomponents.
Constantthroughput
To achieve constant throughput is a good success factor. What if you startexperiencing aworkload that starts affecting the rest of themembers ingroupreplication?Itmightbeacasewhereyourmasterkeepsonacceptingadditionalworkload and is lagging behind, after which it might return to an acceptablelevel before burning out all the resources. Additionally, you can implement aqueuingmethodology that can prevent you from burning down resources andonlyallowsyoutopassonworkloadstothemembersthatarepredefinedbasedoncapacity.
While considering a queuing methodology, you mustn't allow queues growexponentially.
This would impact the end user, as there would be a lag in the data beingupdated.
However,youneedstodecidebasedonyourneedsandthebusinessrequirementtoachieveconstantthroughputacrossthesystem.
Contradictoryworkloads
Fundamentally,group replication isdesigned toallowupdates fromanyof themembers in the group. Rollback of transactions based on overlap of rows ischecked for each of the transactions; the rest are committed and sent to beupdated to other members in the group. If several updates on the same rowhappen frequently, it can result inmultiple rollbacks. Youmight come acrosscyclic situations where one server updates, requests others to update, and, inparallel, another has already updated for the same row. This would result inrollback.
Topreventsuchascenario,youcanhavethelastmemberofthegroupapplytheupdate, afterwhichyouproceed to anotherone.Youcanhave similar updatesrouted only from the same node where the earlier one had been executed topreventthechancesofcyclicrollbackconditions.
Writescalability
Distribute your write workload by sharing out write operations, which mightresultinbetterthroughputandbetterscalabilityonwriteperformance.Itwouldbe dependent on contradictory workloads that you would be expecting in thesystem.This ishelpfulwhenyourpeakworkloadisbeingexecutedisonethatcansharetheload.Incommoncases, ifyouhavegoodcapacityplanningdonewithwritescalability,youwouldseetrivialimprovement.
Refer to the following diagram that depicts this:
Youwillnotice thatwith thehelpofmulti-masters todistribute,your loadhasbetterthroughput.Italsoconsidersthegroupsizeinmulti-masterconfiguration.
Best practices for datapartitioning
In general terms, partitioning is logically dividing anything into multiplesubgroups so that each subgroup can be identified independently and can becombinedintoasinglepartition.
Let's now learn different partitioning methods and how partitioning can helpwheretherearelargedatatables.
For any organization, it is very important to store data in such away that thedatabase provides scalability, performance, availability, and security. Forinstance,inahighlyaccessede-commercestore,therearethousands,ormore,ofordersplacedfrequently.
Sotomaintainday-to-dayorderdeliveryshowingadashboardofcurrentorders,whatisrequiredistoqueryatableshowingordersfromthepastfiveyears;theprocesswill take a long time to executewith the currentdata.Here, historicalorderdataisneededfortheanalyticalpurposeoffindinguserbehaviorortrends,butthiswillberequiredtobeperformedonlimiteddatasets.
Therearevariouswaystoachievethebestsuitablesolutionforhighavailability,
scalability,andhighlyperformingarchitecture;thekeyingredientispartitioning.
Inadatabase,dataineachtableisstoredinphysicalfilegroups.Sobydividingthisdatatablefromasinglefilegrouptoamultiplefilegroupcanreducethefilesizeandhelpuscreateascalableandhigh-performingdatabase.
Thefollowingarethekeybenefitsofusingpartitioninginadatabase:
Scalability:Asdatawillbesharedamongmorethanonepartition,serverscanbe configured to usemultiple nodes andpartitions canbe configuredamong multiple nodes. Doing so will eliminate any hardware limits andallow the database to scale up to a large extent to accommodate highvolumedata.
Highperformance:Asdataisstoredamongmultiplepartitions,eachquerywill be executed on a small portion of the data. For example, in an e-commercestorewithanorderhistoryofmorethantwoyears,togetalistoforders placed in the current month will require checking only a singlepartitionandnottheentireorderhistory,thusreducingthequeryexecutiontime.Tofetchthequeryonmorethanonepartition,wecanalsorunthisinparallel,thusreducingtheoveralltimetofetchdatafromthedatabase.
High availability: In partitioning, data is divided across multiple filegroups. Each file group is logically connected but can be accessed andworked on independently. So if one of the file groups or partitions getscorrupted or one of the nodes in the server fails, then we will not loseaccess to the entire table, but only a section of the database will not beavailable, thuseliminating thechancesofsystemfailureandmakingyoursystemhighlyavailable.
Security: Itmaybe that someof thedata in tables requireshigh securitymeasurements to avoid data theft or data leaks. By partitioning, you canprovideadditionalsecurity tooneormorepartitionstoavoidanysecurityissues,thusimprovingdataaccessibilitywithdatasecurity.
In general terms, partitioning is logically dividing anything into multiple
subgroups so that each subgroup can be identified independently and can becombined into a single partition. Let's understand what partitioning means intermsofRDBMS.
Partitioningisgenerallyusedtodividedataintomultiplelogicalfilegroupsforthepurposeofperformance,availability,andmanageability.Whendealingwithbigdata,thenormaltendencyofdataistobeintermsofbillionsofrecords.Soto improve performance of the database, it is better to divide data amongmultiple file groups. These file groups can be on a singlemachine or sharedacrossmultiplemachinesandidentifiedbyakey.Thesefilegroupsareknownaspartitioneddata.
Datainthetablecanbepartitionedintwoways:
Horizontalpartitioning
Verticalpartitioning
HorizontalpartitioningWhenthenumberofrowsinthetableisverylarge,thetablecanbedividedintomultiplepartitions; this isknownashorizontalpartitioning.Whenhorizontalpartitioning is used, each partition of the table contains the same number ofcolumns. It is possible to access all partitions at once, oryou can access eachpartitionindividually.
Verticalpartitioning
In vertical partitioning, the columns of the tables are partitioned to achieveperformanceandbettermanagementofthedatabase.Verticalpartitioningcanbeachievedintwoways.Thefirstoneisbynormalizingtables.Insteadofhavingtoomanycolumnsin the table,columnscanbedividedintomultiple tablesbydividingthedata.Thesecondoneisbycreatingseparatephysicalfilegroupsfordefinedcolumnsinthetable.
VerticalpartitioningiscurrentlynotsupportedinMySQL8.
Let'slookatafewofthebenefitsassociatedwithpartitioning:
Ifatablecontainshistoricaldata,suchaslogsofanapplication,dataolderthansixmonthsdoesnotprovideanysignificancetotheapplicationtobeactive. If partitioning is created based onmonths, you can easily removeoneofthepartitions.
In thesameprecedingcaseof logs, ifwewant to filterdatabetween twodates, theMySQL optimizer can identify the specific partitions, where itcanfindthefilteredrecords,whichcanresultinmuchfasterqueryresults,
asthenumberofrowstocheckisreduceddrastically.
MySQL8alsosupportsqueryingdataonparticularpartitions.Itcanreducethenumberofrecordstocheckwhenyouknowthepartitionthatneedstobequeriedforthedatarequired.
PruningpartitionsinMySQL
Pruningistheselectiveextractionofdata.Aswehavemultiplepartitions,itwillgothrougheachpartitionduringretrieval,whichistimeconsumingandimpactsonperformance.Someofthepartitionswillalsobeincludedinsearchingwhilethe requested data is not available inside that partition, which is an overheadprocess. Pruning helps here to search for only those partitions that have therelevant data, which will avoid the unnecessary inclusion of those partitionsduringretrieval.
Thisoptimization that avoids the scanningofpartitionswhere there canbenomatchingvaluesisknownasthepruningofpartitions.Inpartitionpruning,theoptimizer analyzes FROM and WHERE clauses in SQL statements to eliminateunneededpartitions,andscansthosedatabasepartitionsthatarerelevant to theSQLstatement.
Best practices for queries andindexingIt would be difficult to write the best queries for reference and reuse. It willalwaysvarybasedon thenatureofyourapplication,architecture,design, tablestructure,andsoon.However,precautionscanbe takenwhilewritingMySQLqueriesforbetterperformance,scalability,andintegrity.
Let'sgothroughafewofthebestpracticesthatweshouldkeepinmindwhiledesigningorwritingMySQLqueries.
Datatypes
A database table could consist of multiple columns with data types, such asnumerics or strings. MySQL 8 provides various data types rather than justlimitingtonumericsorstrings:
Small isgood.AsMySQL loadsdata inmemory,a largedata sizewouldhaveanadverseimpactonitsperformance.Smallersetscanaccommodatemoredatainmemoryandreduceoverheadsofresourceutilization.
Fixyour length. Ifyoudon't fix thedata type length, itwouldhave togoandfetch therequired informationeach timeitneeds to.So,wherever it'spossible,youcanlimitthedatalengthbyusingthechardatatype.
NotnullNotnulldataissomethingthatMySQLdoesn'tlikemuch.Notnullcolumnsusemorestorage,impacttheperformance,andrequireadditionalprocessingwithinMySQL.
Optimizingsuchqueries referring tonulldata isdifficultaswell.Whenanulldatacolumnisindexed,itusesadditionalbytesforeachentry.
IndexingIndexingisimportant,asitcanimprovetheperformanceofyourbadlydesignedqueryand table structure or it can even turn awell-designedquery into a badone,whichcanimpactperformancetoo.
SearchfieldsindexGenerally,wedoindexingonfieldsthatareusedasfiltersinMySQLqueries.Itobviously helps reading faster but can adversely impact writes/updates soindexingonlywhatyouneedwouldbeasmartdecision.
DatatypesandjoinsMySQLcandojoinsfordatatypesthataredifferentbuttheperformancecanbeimpacted if MySQL is asked to use different data types for join fields, as itwouldhavetoconvertfromonetoanotherforeachrow.
CompoundindexIfaqueryissupposedtorefertomultiplecolumnsofatable,acompositeindexforsuchcolumnsmightbehelpful.Acompoundindexrefersthecolumnsfromtheresultssetbythefirstcolumn,secondcolumn,andsoon.
Theorderofcolumnsplaysasignificantroleintheperformanceofthequery,sowhiledesigningthetablestructureandindex,youneedtouseiteffectively.
Shorteningupprimarykeys
Small isgoodforprimarykeys too.Shorteningupprimarykeyswouldbenefitanalogously to howwediscusseddatatypes.Becauseof smaller primarykeys,yourindexsizewouldbesmallerandhencetheusageofcachewouldbeless,soitcanaccommodatemoredatainmemory.
It is preferred to use numeric types, as these would be much smaller thancharacters toachieve thegoalofshorteningupprimarykeys. Itcanbehelpfulwhiledoingjoins,asgenerally,primarykeysarereferredforthejoining.
IndexingeverythingIndexing everything is a good idea; however,MySQL won't do this. Do youknow thatMySQLwill do a full table scan if it is supposed to scan an indexhigherthan30%?
Donotindexvaluesthatdon'tneedtobeindexed.
We need to keep inmind that indexing helps—if done correctly—in fetchingdata;however,whilewriting/updatingdata,itisanoverhead.
Fetchingalldataselect *... - Arrghh! Do not use this unless it is really needed. So far, myexperience hasn't needed this. Fetching all datawill slow down the executiontimeandimpactheavilyonresourceutilizationoftheMySQLserver.Youneedtoprovideaspecificcolumnnameorappropriateconditions.
LettingtheapplicationdothejobLet the application alsodo the job forMySQL.Youcan avoidhaving clausessuch as order by by letting applications do the ordering. Doing ordering inMySQL is much slower than in applications. You can identify queries thatshouldbeplannedtobetakencareofbytheapplication.
ExistenceofdataCheckingtheexistenceofdatawiththehelpoftheEXISTSclauseismuchfaster.TheEXISTSclausewillreturntheoutputassoonasitfetchesthefirstrowfromthefetcheddata.
LimitingyourselfLimit yourself to the data that you need to fetch.Always ensure that you useappropriate limits while fetching the data, as unwanted data being fetchedwouldn'tbeusefulandwouldimpactperformance.UsetheLIMITclauseinyourSQLqueries.
AnalyzingslowqueriesThisisagoodpracticetofollow.Wemightmissoutqueriestoeitheroptimizeorrealize having adverse impact as data grows. Youmight have changes in therequirementofthedatatobefetchedwherewemightmissseeingtheimpactofthe queries. It is good to always keep a watch on slow queries that can beconfiguredinMySQLandoptimizethem.
Querycost
What is the cost of your query? Explain is the right answer to this. Use theexplainqueryparameter toknowwhat is impactingyourquery—whether it isafull table scan, index scans, range access, and so on. Use the informationprovided by explain wisely, to optimize the query further. It is a wonderful,quickhandytoolofMySQL.Ifyouknowthatyouhavedoneyourbest,indexingcomesasasaviortooptimizeitfurtherbasedonyourneeds.
Best practices while writing a query start with requirements, designs,implementations, and ongoingmaintenance. It's a complete life cycle that wecan'tdiversify.Understandingschemas,indexes,andanalysesplaysasignificantrole.Whatmatterstousistheresponsetimeandoptimumresourceutilization.
Ipersonallylovetodeepdiveintothismuchmorethanwecanmentionhere—it'saworldofrelations!Yourquerywillmeetaroworcolumnofatableorgetjoinedwith another table.On topof this, if youhaven't done it right, you aretrying to find a relation from a subset that is not required.Howdowe forgetindexesthataresaviorsifusedappropriately?Allthesetogetherwouldshowourrelationsandwouldpromptlyrespondtoarequestedquery.
Summary
Iamsurethatwhilereadingthechapter,youhavekeptinmindthethingstobetakencareoforrecollectingthem,ifthere'sanythingmissinginyourMySQL8implementation.
In these chapter, we discussed best practices for MySQL 8 which would behelpful at various stages, such as implementation, usage, management, andtroubleshootingandwouldactaspointersforbestpracticesofMySQL8;thesemight varybasedondifferent use cases.Proper testing andverificationwouldhelpaffirmthebenefitsofhavingbestpracticesimplemented.
WehavebroadlycoveredsomeexcitingtopicsaboutMySQL8benchmarksanda few configuration parameters along with best practices of memcached. WediscussedMySQL replication best practices, inwhichwewent through a fewcritical pointers. Lastly, MySQL queries and indexing pointers were alsodiscussedwithbestpracticesfordatapartitioning.
Anything written in this chapter would be less, but the pointers provided arenecessary.
By now,we should have a good understanding ofMySQL8; it's now time tosolveproblems.
Let'snowmoveon to thenextchapterand lookathowwecouldcomeacrossmany common issues, identifying error codes alongwith real-world scenariosfortroubleshootingMySQL8.
TroubleshootingMySQL8
In the previous chapter, we learned an important aspect of the MySQL 8database,benchmarking, andbestpractices.Benchmarkinghelps incomparingthecurrentdatabaseperformanceagainsttheexpectedperformancematrices.Welearned what benchmarking is and the tools that can be used to find thebenchmarkperformanceofaMySQL8server.Inalaterpartofthechapter,welearned about the best practices to be followed for memcached, replication,partitioning,andindexing.BestpracticeshelpensuretheoptimumconfigurationoftheMySQL8database.
In this chapter, the focuswill beonunderstanding the commonerrors thatwemayencounterwhileworkingwith theMySQL8database.Theerrorsmaybeservererrorsorclienterrors.Wewilllookatawaytodeterminethattheproblemhasoccurred.
Wewillalsolearntroubleshootingandresolutiontechniquesforerrors.Inalaterpartofthechapter,wewilllookintoreal-worldscenarioswherethesetechniquesareapplicable.Thefollowingisthelistoftopicstobecovered:
MySQL8commonproblems
MySQL8commonproblems
When troubleshooting is an issue, the first thing to be done is to find out theprogramorpieceofequipmentthatiscausingitwhenwerunintoaproblem.
Thefollowingaresymptomsthatindicateahardwareorkernelproblem:
The keyboard is not functioning. It can be checked by pressing theCapsLockkey.IfthelightontheCapsLockkeydoesnotlightup,itisanissuewiththekeyboard.Similarly,themousenotmovingindicatesanissuewiththemouse.
ping is an operating system command to check the accessibility of onemachine from another machine. The machine from which the pingcommand is executed is called the localmachine, whereas the machinepinged is called the remote machine. If the remote machine does notrespond to the local machine's pings, it indicates a hardware or networkrelatedissue.
Itmay indicate an issuewith the operating system kernel program if theprogramsotherthanMySQLarenotworkingcorrectly.
Itmayindicateanissuewiththeoperatingsystemorhardwareifthesystemrestartsunexpectedly. Ina typicalcase,auser-levelprogramshouldneverbeabletotakethesystemdown.
Totroubleshoottheissue,oneormoreofthefollowingcanbedone:
Runadiagnostictooltocheckhardware
Ensuretherelevantlibraryfilesareuptodate
Check for the availability of updates, patches, or service packs for theoperatingsystem
Checkallconnections
ECC memory is an error correcting code memory. It can detect and correct mostcommon internaldatacorruption issues. It isadvisable touseECCmemory inordertodetectthememoryissuesatanearlystage.
Thefollowinginstructionsmayhelpfurtheridentifytheissue:
Examining the system log files may help to discover the reason for theproblem.MySQL
log filesmust also be checked in case there appears to be an issuewithMySQL.
Operating system specific commands can be used to check issues withmemory,filedescriptors,diskspace,orothercriticalresources.
A bug can be identified in the operating system kernel if a problematicrunawayprocessdoesnotdieeventhoughwehaveexecutedacommandtokillit.
Ifthereappearsnottobeaproblemwiththehardware,attemptsshouldbe
made to identify the program that may be causing the problem. Usingoperatingsystemspecificcommands,suchasTaskmanageronWindows,psandtoponLinux,orsimilarprograms,wecanidentifyprogramsthateatupCPUorblocksystemprocesses.
Itispossibletorecovertheaccesstothemachineeventhoughthekeyboardis lockedup.Thiscanbedoneby loggingon to the systemfromanothermachine.Executethekbd_mode-acommanduponsuccessfullogin.
MySQLuserscanreportissuesbyusingoneofthemultiplechannelsprovidedbyMySQL.
After having examined all the possible alternatives, if it can be decided thateither theMySQL server or theMySQLclient causes the problem, a user caneither create a bug report for the mailing list or contact the MySQL supportteam.Thebugreportermustprovidedetailedinformationaboutthebug,systeminformation,andbehaviorandtheexpectedbehavior.ThereportermustdescribethereasonbasedonwhyitseemstobeaMySQLbug.Itisusefultoknowthefollowinginformationiftheprogramfails:
With the help of the top command, check if the program in question hastakenupalltheCPUtime.Insuchcases,weshouldallowaprogramtorunfor a while because it is possible that the program may be executingintensivecomputationalinstructions.
ObservetheresponsefromtheMySQLserverwhenaclientprogramtriesto connect to it. Has it stopped responding? Did the server provide anyoutput?
If it is found that the MySQL server is causing problems in the mysqldprogram,trytoconnectusingthemysqladminprogramtocheckwhethermysqldresponds. The mysqladmin -u root ping or mysqladmin -u root processlist
commandscanbeused.
Didthefailedprogrammakeasegmentationfault?
MostcommonMySQLerrorsThis section provides a list of the most common MySQL errors that usersencounterveryfrequently.
Accessdenied
MySQL provides a privilege system that authenticates the user who connectsfroma host, and associates the userwith access privileges on a database.Theprivileges include SELECT, INSERT, UPDATE, and DELETE and are able to identifyanonymoususersandgrantprivilegesforMySQLspecificfunctions,suchasLOADDATAINFILEandadministrativeoperations.
Theaccessdeniederrormayoccurbecauseofmanycauses.Inmanycases,theproblemiscausedbecauseofMySQLaccounts that theclientprogramsuse toconnectwiththeMySQLserverwithpermissionfromtheserver.
<strong><spanclass="tokencommand">>mysqladmin</span>version<br/><spanclass="tokencommand">>mysqladmin</span>variables<br/><spanclass="tokencommand">>mysqladmin</span><spanclass="tokenproperty">-h</span>`hostname`versionvariables<br/><spanclass="tokencommand">>mysqladmin</span><spanclass="tokenproperty">-h</span>`hostname`<spanclass="tokenconstant">--port</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>3306</span>version<br/><spanclass="tokencommand">>mysqladmin</span><spanclass="tokenproperty">-h</span>host_ipversion<spanclass="tokencommand"><br/>>mysqladmin</span><spanclass="tokenconstant">--protocol</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>SOCKET</span><spanclass="tokenconstant">--socket</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>/tmp/mysql.sock</span>version</strong>
Intheprecedingcommands,hostnameisthehostnameofthemachineonwhichtheMySQLserverisrunning.host_ipistheIPaddressoftheservermachine.
LostconnectiontoMySQLserver
ThelostconnectiontoMySQLservererrorcanoccurbecauseofoneofthethreelikely
causesexplainedinthissection.
One of the potential reasons for the error is that the network connectivity istroublesome.
Networkconditionsshouldbecheckedif this isafrequenterror. If theduringquery message is part of the error message, it is certain that the error hasoccurredbecause
ofnetworkconnectionissues.
The connection_timeout system variable defines the number of seconds that themysqldserverwaitsforaconnectionpacketbeforeconnectiontimeoutresponse.Infrequently,
this error may occur when a client is trying for the initial connection to theserver
andtheconnection_timeoutvalueissettoafewseconds.Inthiscase,theproblemcanberesolvedbyincreasing
theconnection_timeoutvaluebasedon the thedistanceandconnectionspeed.SHOWGLOBAL STATUS LIKE and Aborted_connects can be used to determine if we areexperiencingthismorefrequently.Itcanbecertainly
saidthatincreasingtheconnection_timeoutvalueisthesolutioniftheerrormessagecontainsreadingauthorizationpacket.
It is possible that the problemmay be faced because of largerBinaryLargeOBject(BLOB)valuesthanmax_allowed_packet.Thiscancausea lostconnectionto the MySQL server error with clients. If the ER_NET_PACKET_TOO_LARGE error isobserved,itconfirmsthatthemax_allowed_packetvalueshouldbeincreased.
<strong>>mysql-uuser_name-p</strong><br/><strong>Enterpassword:</strong>
Onafewsystems,itmayhappenthatthepasswordworksfinewhenspecifiedinanoptionfileoronthecommandline.ButitdoesnotworkwhenenteredinteractivelyontheCommandPromptattheEnterpassword:prompt.Itoccursbecausethesystem-providedlibrarytoreadthepasswordslimitsthepasswordvaluestoasmallnumberofcharacters(usuallyeight).ItisanissuewiththesystemlibraryandnotwithMySQL.Asaworkaroundtothis,changetheMySQLpasswordtoavaluethatiseightorfewercharactersorstorethepasswordintheoptionfile.
<strong>Host'host_name'isblockedbecauseofmanyconnectionerrors.</strong><br/><strong>Unblockwith'mysqladminflush-hosts'</strong>
<strong>>mysqld_safe--max_connect_errors=10000</strong>
<strong>mysql>SETGLOBALmax_connect_errors=10000;</strong>
ItshouldbecheckedfirstthatthereisnothingwrongwithTCP/IPconnectionsfromthehostifthehost_nameisblockederrorisreceivedforaparticularhost.Increasingthevalueofthemax_connect_errorsvariabledoesnothelpifthenetworkhasproblems.
Toomanyconnections
This error indicates that all available connection are in use for other clientconnections.
Themax_connectionsisthesystemvariablethatcontrolsthenumberofconnectionstotheserver.The
default value for the maximum number of connections is 151. We can set alargervalue
than151forthemax_connectionssystemvariabletosupportmoreconnectionsthan151.
The mysqld server process actually allows one more than max_connections
(max_connections + 1) value clients to connect. The additional one connection iskeptreservedforaccounts
with CONNECTION_ADMIN or the SUPER privilege. The privilege can be granted to theadministrators with access to the PROCESS privilege. With this access, theadministratorcanconnecttotheserverusingthe
reservedconnection.TheycanexecutetheSHOWPROCESSLISTcommandtodiagnose
OutofmemoryIfthemysqldoesnothaveenoughmemorytostoretheentirerequestofthequeryissued by the MySQL client program, the server throws the following error:mysql:Outofmemoryatline42,'malloc.c'mysql:needed8136byte(8k),memoryinuse:12481367bytes(12189k)ERROR2008:MySQLclientranoutofmemory
Inorder to fix theproblem,wemust firstcheck if thequery iscorrect.Doweexpectthequerytoreturnsomanyrows?Ifnot,weshouldcorrectthequeryandexecuteitagain.Ifthequeryiscorrectandneedsnocorrection,wecanconnectmysql with the --quick option. Using the --quick option results in themysql_use_result() CAPI function for fetching the result set. The function addsmoreloadontheserverandlessloadontheclient.
<strong><spanclass="tokencommand">>mysql</span><spanclass="tokenconstant">--max_allowed_packet</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>32M</span></strong>
ThedefaultvaluefortheMySQLserveris64MB.Itshouldbenotedthatthereisnoharminsettingalargervalueforthissystemvariable,astheadditionalmemoryisallocatedasneeded.
ThetableisfullThetable-fullerroroccursinoneofthefollowingconditions:
ThediskisfullThetablehasreachedthemaximumsize
TheactualmaximumtablesizeintheMySQLdatabasecanbedeterminedbytheconstraintsimposedbytheoperatingsystemonthefilesizes.
<strong>Can'tcreate<spanclass="tokenoperator">/</span>writetofile'\\sqla3fe_0<spanclass="tokenpunctuation">.</span>ism'<spanclass="tokenpunctuation">.</span></strong>
<strong>>mysqld--tmpdirC:/temp</strong>
<strong>[mysqld]</strong><br/><strong>tmpdir=C:/temp</strong>
CommandsoutofsyncIftheclientfunctionsarecalledinthewrongorder,thecommandsoutofsyncerror is received. Itmeans that the command cannot be executed in the clientcode.Asanexample,ifweexecutemysql_use_result()andtrytoexecuteanotherquery before executing mysql_free_result(), this error may occur. It may alsohappen if we execute two queries that return a result set without calling themysql_use_result()ormysql_store_result()functionsinbetween.
<strong>Foundwrongpasswordforuser'some_user'@'some_host';ignoringuser</strong>
TheaccountisignoredbytheMySQLpermissionsystemasaresult.Tofixtheproblem,weshouldassignanewvalidpasswordfortheaccount.
<strong>Table'tbl_name'doesn'texist</strong><br/><strong>Can'tfindfile:'tbl_name'(errno:2)</strong>
Insomecases,theusermaybereferringtothetableincorrectly.ItispossiblebecausetheMySQLserverusesdirectoriesandfilesforstoringdatabasetables.Dependingupontheoperatingsystemfilemanagement,thedatabaseandtablenamescanbecasesensitive.
Fornoncase-sensitivefilesystems,suchasWindows,thereferencestoaspecifiedtableusedwithinaquerymustusethesamelettercase.
MySQL8servererrorsThissectionfocusesonMySQL8servererrors.Thesectiondescribestheerrorsrelated toMySQLserveradministration, tabledefinitions,andknownissues intheMySQL8server.
<strong>ERROR:Can'tfindfile:'path/with/file_name'(Errcode:13)</strong>
<strong><spanclass="tokenprompt">></span><spanclass="tokencommand">UMASK</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>384</span><spanclass="tokencomment">#=600inoctal</span><br/><spanclass="tokenprompt">></span><spanclass="tokencommand">export</span>UMASK<br/><spanclass="tokenprompt">></span><spanclass="tokencommand">mysqld_safe</span></strong>
<strong><spanclass="tokenprompt">></span><spanclass="tokencommand">UMASK_DIR</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>504</span><spanclass="tokencomment">#=770inoctal</span><br/><spanclass="tokenprompt">></span><spanclass="tokencommand">export</span>UMASK_DIR<br/><spanclass="tokenprompt">></span><spanclass="tokencommand">mysqld_safe</span>&</strong>
<strong><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">USER</span><spanclass="tokenstring">'root'</span>@<spanclass="tokenstring">'localhost'</span><spanclass="tokenkeyword">IDENTIFIED</span><spanclass="tokenkeyword">BY</span><spanclass="tokenstring">'NewPassword'</span><spanclass="tokenpunctuation">;</span></strong>
<strong><spanclass="tokenprompt">C:\></span><spanclass="tokencommand">cd</span><spanclass="tokenatrule">"C:\ProgramFiles\MySQL\MySQLServer8.0\bin"</span><br/><spanclass="tokenprompt">C:\></span><spanclass="tokencommand">mysqld</span><spanclass="tokenconstant">--init-file</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>C:\\mysql-root-reset.txt</span></strong>
<strong>>kill'cat/mysql-data-directory/host_name.pid'</strong>
<strong><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">USER</span><spanclass="tokenstring">'root'</span>@<spanclass="tokenstring">'localhost'</span><spanclass="tokenkeyword">IDENTIFIED</span><spanclass="tokenkeyword">BY</span><spanclass="tokenstring">'NewPassword'</span><spanclass="tokenpunctuation">;</span></strong>
<strong><spanclass="tokenprompt">></span><spanclass="tokencommand">mysqld</span><spanclass="tokenconstant">--init-file</span><spanclass="tokenattr-value"><spanclass="tokenpunctuation">=</span>/home/me/mysql-reset-root</span>&</strong>
<strong>>mysql</strong>
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">FLUSH</span><spanclass="tokenkeyword">PRIVILEGES</span><spanclass="tokenpunctuation">;</span></strong>
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">ALTER</span><spanclass="tokenkeyword">USER</span><spanclass="tokenstring">'root'</span>@<spanclass="tokenstring">'localhost'</span><spanclass="tokenkeyword">IDENTIFIED</span><spanclass="tokenkeyword">BY</span><br/><spanclass="tokenstring">'NewPassword'</span><spanclass="tokenpunctuation">;</span></strong>
5. Restarttheserverandloginwiththerootuserandnewlysetpassword.
<strong>C:\ProgramFiles\MySQL\MySQLServer8.0\bin>mysqladminversion-uroot-p</strong><br/><strong>Enterpassword:*****</strong><br/><strong>mysqladminVer8.0.3-rcforWin64onx86_64(MySQLCommunityServer(GPL))</strong><br/><strong>Copyright(c)2000,2017,Oracleand/oritsaffiliates.Allrightsreserved.</strong><br/><br/><strong>OracleisaregisteredtrademarkofOracleCorporationand/orits</strong><br/><strong>affiliates.Othernamesmaybetrademarksoftheirrespective</strong><br/><strong>owners.</strong><br/><br/><strong>Serverversion8.0.3-rc-log</strong><br/><strong>Protocolversion10</strong><br/><strong>ConnectionlocalhostviaTCP/IP</strong><br/><strong>TCPport3306</strong><br/><strong>Uptime:9days4hours4min52sec</strong><br/><br/><strong>Threads:2Questions:4Slowqueries:0Opens:93Flushtables:2Opentables:69Queriespersecondavg:0.000</strong>
Theresolve_stack_dumpisautilityprogramthatresolvesanumericstackdumptosymbols.Toanalyzetherootcauseofwherethemysqldserverprocessdied,wefindinthestacktraceerrorlogs.Itcanberesolvedwiththeresolve_stack_dumpprogram.Itmustbenotedthatitispossiblethatvariablevaluesfoundintheerrorlogsmaynotbeaccurate.
CorrupteddataorindexfilescancausetheMySQLservertocrash.Thesefilesareupdatedonthediskusingthewrite()systemcalledupontheexecutionofeachSQLstatementandbeforetheclientisnotifiedabouttheresult.Itmeansthatthecontentsinthedatafilesaresafeevenintheeventofamysqldcrash.Thewritingoftheunflusheddataonthediskistakencareofbytheoperatingsystem.The--flushoptioncanbeusedwithmysqldtoforceMySQLtoflusheverythingtodiskaftereverySQLstatementexecution.
OneofthefollowingcanbethereasonforMySQLcorruptedtables:
Ifthedatafileorindexfilecrashesthenitcontainscorrupteddata.AbugintheMySQLserverprocesscausedtheservertodieinthemiddleofanupdate.Anexternalprogrammanipulatedthedataandindexfilesatthesametimeasmysqldwithouttablelocking.Inthemiddleofanupdate,theMySQLserverprocesswaskilled.Manymysqldserversarerunningonthesystem.Theserversusethesamedatadirectory.Thesystemdoesnothavegoodfilesystemlocksortheexternallockingisdisabled.Itispossiblethatabugisfoundinthedatastoragecode.WecantrytochangethestorageenginebyusingALTERTABLEontherepairedcopyofthetable.
HandlingMySQLfulldisk
ThissectionfocusesontheresponsefromMySQLtodisk-fullerrorsandquota-exceedederrors.ItismorerelevanttowritesinMyISAMtables.Itcanbeappliedtowritesinbinarylogfilesandanindexfile.Itexcludesthereferencestorowsandrecordsthatshouldbeconsideredanevent.
MySQLperformsthefollowingwhenadisk-fullconditionoccurs:
MySQL ensures that there is enough space available towrite the currentrow.
TheMySQLserverwritesanentryinthelogfileevery10minutes.Itwarnsaboutthedisk-fullcondition.
Thefollowingactionsshouldbetakentoremedytheproblem:
Thediskspaceshouldbefreed tomakesureenoughspace isavailable toinsertallrecords.
Wecanexecute the mysqladminkill command toabort the thread.Thenexttimeitchecksthedisk,thethreadisaborted.
Itmayhappen thata few threadsarewaiting for the table thatcaused thedisk-full situation.Outof several locked threads,killing the thread that iswaitingonthedisk-fullconditionwillenableotherthreadstocontinue.
TheREPAIRTABLEorOPTIMIZETABLEstatementsareexceptionsto theprecedingcondition.OtherexceptionsincludeindexescreatedinabatchaftertheLOADDATA INFILE or ALTER TABLE statements. These SQL statements can createtemporary fileswith largevolumes.Thismaycreatebigproblems for therestofthesystem.
MySQLtemporaryfilesstorageThevalueoftheTMPDIRenvironmentvariableisusedbyMySQLontheUnixasthepathnameof thedirectory to store temporary files.MySQLuses a systemdefault,suchas/tmp,/var/tmp,or/usr/tmpiftheTMPDIRisnotset.
The values of the TMPDIR, TEMP, and TMP environment variables are checked byMySQLonWindows. IfMySQLfindsoneset, ituses thatvalueanddoesnotcheckforremainingvalues.Ifnoneofthesethreevariablesareset,MySQLusesthesystemdefault,whichisC:\windows\temp\.
Ifthetemporaryfiledirectoryinthefilesystemistoosmall,wecanusethemysqld--tmpdir option to specifyadirectoryon the filesystemwithenough space.Forthereplication,onslavemachines,wecanuse--slave-load-tmpdirandspecifythedirectory for holding temporary files during the replication of LOAD DATA INFILEstatements. It is possible to set a list of several paths used in a round-robinfashionwiththe--tmpdiroption.OntheUnixsystem,thepathscanbeseparatedby thecoloncharacter(:),whereasonWindows, thesemicoloncharacter(;)canbeusedtoseparatethepaths.
To effectively distribute the load, multiple temporary directorypathsshouldbelongtodifferentphysicaldisksandnotthedifferentpartitionsofthesamedisk.
For the MySQL server working as a replication slave, we must take care ofsetting the --slave-load-tmpdir option so as not to point to a directory in thememory-basedfilesystemortoadirectorythatiscleareduponserverorserverhostrestarts.ToreplicatethetemporarytablesorLOADDATAINFILEoperations,thereplication slave requires its temporary files on the machine restart. Thereplicationfailsifthefilesinthetemporaryfiledirectoryarelost.
MySQL takes care of removing the temporary files when the mysqld serverprocessisterminated.OnUnix-likeplatforms,itcanbedonebyunlinkingafileafteropeningit.Oneofthemajordisadvantageofthisisthatthenamedoesnotappear in directory listings. It also happens thatwe cannot see a big file thatoccupiesthefilesystem.
ibtmp1isthenameofthetablespacefilethattheInnoDBstorageengineusestostoretemporarytables.ThefileislocatedinthedatadirectoryofMySQL.Ifwewantto specify a different filename and location, the innodb_temp_data_file_pathoptioncanbeusedontheserverstartup.
If the ALGORITHM=COPY technique is used by the ALTER TABLE operation on the InnoDBtable, the storage engine creates a temporary copy of the original table in thesamedirectory.The temporary table filenamesstartwith the#sql- prefix.TheyonlyappearbrieflywhiletheALTERTABLEoperationisbeingperformed.
If the InnoDB table is rebuilt by the ALTER TABLE SQL statement using theALGORITHM=INPLACEmethod,theInnoDBstorageenginecreatesanintermediatecopyofthe original table in the same directory as that of the original table. Theintermediate table filenames start with the #sql-ib prefix. They only appearbrieflywhiletheALTERTABLEoperationisbeingperformed.
The innodb_tmpdir option cannot be applied to intermediate table files. Theseintermediatefilesarealwayscreatedandstoredinthesamedirectoryasthatoftheoriginaltable.
The ALTER TABLE SQL statements that rebuild the InnoDB table with theALGORITHM=INPLACE method create temporary sort files in the default MySQLtemporary directory. The default temporary directory is denoted by $TMPDIR onUnix,%TEMP%onWindows,orthedirectorymentionedbythe--tmpdiroption.tmpdirmayneed tobe reconfigured if the temporarydirectory is not large enough tostoresuchfiles.Asanalternative,wecandefineanothertemporarydirectoryforonlineInnoDBALTERTABLEstatementsusingtheinnodb_tmpdiroption.Theinnodb_tmpdiroptioncanbeconfiguredatruntime,usingtheSETGLOBALorSETSESSIONstatements.
Replicating an innodb_tmpdir configuration should be considered in replicationenvironmentsifalltheservershavethesameoperatingsystemenvironment.Inother cases, an innodb_tmpdir setting replication can result in a failed replicationwhile executing online ALTER TABLE operations. It is recommended to configureinnodb_tmpdir for each server separately if the operating environments aredifferent.
<strong>chmod+t/tmp</strong>
<strong>[mysqld]</strong><br/><strong>socket=/path/to/socket</strong><br/><br/><strong>[client]</strong><br/><strong>socket=/path/to/socket</strong>
<strong>mysqladmin--socket=/path/to/socketversion</strong>
TimezoneproblemsTheMySQLservermustbetoldtheuser'scurrenttimezoneifwehaveproblemwith
SELECTNOW()returningavalueinUTCinsteadoftheuser'scurrenttimezone.Itisalsoapplicable
ifUNIX_TIMESTAMP() returnsawrongvalue. It shouldbedone for theenvironmentrunningtheserver;
forexample,mysqld_safeormysql.server.
Wecanalsoset theserver timezonebyusing the--timezone=timezone_nameoptionwithmysqld_safe. Itcanalsobesetbyassigning thevalue to theTZenvironmentvariablebeforethemysqldisstarted.
Theallowedlistofvaluesfor--timezoneorTZdependsonthesystem.
MySQL8clienterrorsThissection focusesonerrors thatoccuron theMySQL8client.The jobofaMySQL
clientis toconnecttotheMySQLserversoastoexecutetheSQLqueriesandget the results from theMySQL8database.This section lists errors related toexecutionofthequeries.
<strong>col_name<spanclass="tokenkeyword">COLLATE</span>utf8mb4_0900_as_cs<spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'a%'</span></strong><br/><strong>col_name<spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'a%'</span><spanclass="tokenkeyword">COLLATE</span>utf8mb4_0900_as_cs</strong><br/><strong>col_name<spanclass="tokenkeyword">COLLATE</span>utf8mb4_bin<spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'a%'</span></strong><br/><strong>col_name<spanclass="tokenoperator">LIKE</span><spanclass="tokenstring">'a%'</span><spanclass="tokenkeyword">COLLATE</span>utf8mb4_bin</strong>
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SET</span><spanclass="tokenkeyword">NAMES</span><spanclass="tokenstring">'utf8mb4'</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SET</span><spanclass="tokenvariable">@s1</span><spanclass="tokenoperator">=</span><spanclass="tokenstring">'MySQL'</span><spanclass="tokenkeyword">COLLATE</span>utf8mb4_bin<spanclass="tokenpunctuation">,</span><spanclass="tokenvariable">@s2</span><spanclass="tokenoperator">=</span><spanclass="tokenstring">'mysql'</span><spanclass="tokenkeyword">COLLATE</span>utf8mb4_bin<spanclass="tokenpunctuation">;</span><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenvariable">@s1</span><spanclass="tokenoperator">=</span><spanclass="tokenvariable">@s2</span><spanclass="tokenpunctuation">;<br/></span><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="token
punctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>@s1=@s2<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>0<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="token
variable">@s1</span><spanclass="tokenkeyword">COLLATE</span>utf8mb4_0900_ai_ci<spanclass="tokenoperator">=</span><spanclass="tokenvariable">@s2</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>@s1COLLATE
utf8mb4_0900_ai_ci=@s2<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">|</span>1<spanclass="tokenpunctuation">|</span></span><br/><spanclass="tokenoutput"><spanclass="tokenpunctuation">+</span><spanclass="tokenpunctuation">-</span><spanclass="token
punctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">-</span><spanclass="tokenpunctuation">+</span></span></strong>
<strong><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>table_name<spanclass="tokenkeyword">WHERE</span><spanclass="tokendatatype">date_col</span><spanclass="tokenoperator">>=</span><spanclass="tokenstring">'2011-06-02'</span><spanclass="tokenpunctuation">;</span></strong>
WhenaconstantstringiscomparedtoDATE,TIME,DATETIME,orTIMESTAMPusing<,<=,=,>=,>,orBETWEENoperators,MySQLconvertsthestringintoaninternallongintegervalue.MySQLdoesthissoastoachieveafastercomparison.However,thefollowingexceptionsareapplicabletothisconversion:
ComparingtwocolumnsComparingaDATE,TIME,DATETIME,orTIMESTAMPcolumntoanexpressionUseofacomparisonmethodotherthanthosejustlisted,suchasINorSTRCMP()
Thecomparisonisdonebyconvertingtheobjectsintostringvaluesandperformingastringcomparisonincaseoftheseexceptions.
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">INSERT</span><spanclass="tokenkeyword">INTO</span>my_table<spanclass="tokenpunctuation">(</span>phone<spanclass="tokenpunctuation">)</span><spanclass="tokenkeyword">VALUES</span><spanclass="tokenpunctuation">(</span><spanclass="tokenboolean">NULL</span><spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">INSERT</span><spanclass="tokenkeyword">INTO</span>my_table<spanclass="tokenpunctuation">(</span>phone<spanclass="tokenpunctuation">)</span><spanclass="tokenkeyword">VALUES</span><spanclass="tokenpunctuation">(</span><spanclass="tokenstring">''</span><spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">;</span></strong>
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenboolean">NULL</span><spanclass="tokenpunctuation">,</span><spanclass="tokennumber">1</span><spanclass="tokenoperator">+</span><spanclass="tokenboolean">NULL</span><spanclass="tokenpunctuation">,</span><spanclass="tokenfunction">CONCAT</span><spanclass="tokenpunctuation">(</span><spanclass="tokenstring">'Invisible'</span><spanclass="tokenpunctuation">,</span><spanclass="tokenboolean">NULL</span><spanclass="tokenpunctuation">)</span><spanclass="tokenpunctuation">;</span></strong>
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>my_table<spanclass="tokenkeyword">WHERE</span>phone
<spanclass="tokenoperator">=</span><spanclass="tokenboolean">NULL</span><spanclass="tokenpunctuation">;</span></strong>
<strong><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>my_table<spanclass="tokenkeyword">WHERE</span>phone<spanclass="tokenoperator">IS</span><spanclass="tokenboolean">NULL</span><spanclass="tokenpunctuation">;</span><br/><spanclass="tokenprompt">mysql></span><spanclass="tokenkeyword">SELECT</span><spanclass="tokenoperator">*</span><spanclass="tokenkeyword">FROM</span>my_table<spanclass="tokenkeyword">WHERE</span>phone<spanclass="tokenoperator">=</span><spanclass="tokenstring">''</span><spanclass="tokenpunctuation">;</span></strong>
MySQL 8 troubleshootingapproach
In this section of the chapter,wewill focus on theMySQL8 troubleshootingapproach.
WhydoweneedtotroubleshootMySQL8?Thereasonsfortroubleshootingareasfollows:
FasterexecutionofSQLqueries
Performanceenhancement
Efficientuseofresources
TheprimarysetofresourcesincludeCPU,diskIO,memory,andnetwork.TherearetwoapproachestomeasureMySQLperformance:
Inaqueryfocusedapproach,itisimportanttomeasurehowfastthequeriesgetexecuted
In resource focused approach, it is important that the queries use fewerresources.
LetustakeadeeperlookatwaystotroubleshootMySQLproblems.
<strong>mysql>SELECT*FROMtbl_nameWHEREprimary_key=1;</strong><br/><strong>mysql>SELECT*FROMtbl_nameWHEREprimary_key_part1=1ANDprimary_key_part2=2;</strong>
ref:Foreachcombinationofrowsfromtheearliertables,allrowswithmatchingindexvaluesarereadfromthereftable.Ifthejoinusesonlytheleftmostprefixofthekey,therefisused.
Real-worldscenario
MySQL query optimization is referred to as improving the time of queryexecution.
Forexample,whenaqueryisnotperformingwellmeansthatthequeryistakinga longer time than expected for execution.The timeof the query execution isimportant but there are other matrices as well that are used to measureperformance.Thissectionexplainswhatshouldbemeasuredandhowitshouldbedoneaspreciselyaspossible.
Thefollowingquestionarises:whyshouldweoptimizethequery?Doesitreallyrequireoptimizationifitonlytakesahundredthofasecond?Yes,itdoesrequireoptimizationunlessthequeryisexecutedrarely.Weshouldoptimizethequeriesthataremostexpensive.
Let'sdiscussareal-timeexample.Inoneoftheapplications,wehadareportthatwas generated based on a complex query andwas taking toomuch time.Theexecution time was in minutes. To optimize such a complex query, weconsideredthefollowingapproach:
1. Analyze the query plan using EXPLAIN: MySQL provides two ways toanalyze theperformanceof a query.One is the EXPLAINmethod,whichwehavealreadylearnedaboutintheprecedingsectionofthischapter.
Another tool is SHOW STATUS. Usually, we should prefer to use EXPLAIN tounderstandthequeryplanofaSELECTquery.Inthecaseofthereportquery,weconvertafewofthenon-SELECTqueriestoSELECTqueries.Thishelpsusinunderstandingthequeryexecutionplanfornon-SELECTqueriesaswell.Forexample,rewritinganUPDATEquerytoSELECTcanbedonebyusingtheWHEREclauseintheUPDATEquery,whichispassedontotheSELECTquery.Wecouldalsofindfewmissingindexesonthetables.
2. SHOW STATUS: The SHOW STATUS statement outputs the internal counters forMySQL. The counters are incremented by MySQL upon every queryexecution.Withthehelpofthesecounters,wecouldunderstandthetypesofoperationsperformedbytheserverinaggregate.Italsohelpsinindicatingtheworkdonebyeachindividualquery.
ThefollowingarethemeasurementsperformedforMySQLservervariables:
Select_: This counter is incrementedwhenever a SELECT query is executed.Thiscountercanalsobeusedtoidentifyifatablescanisperformed.
Key_read:Thisvariableprovidesadditional informationon theusageof thekeyindex.
Last_query_cost:Thisvalue indicateshowexpensive the lastexecutedquerywas.
Thefollowingarethestepstoperformqueryoptimization:
1. Querytheexecutionafewtimestoensureitreturnsthesameresult.
2. ExecuteSHOWSTATUS.Theoutputshouldbesaved.
3. Executethequery.
4. ExecuteSHOWSTATUStoobservethedifferencesfromthepreviousexecution.
5. ExecuteEXPLAINifrequired.
The following parameters should be analyzed for query performanceoptimization:
Tableindex
Sorting
Overallperformance
Rowleveloperations
DiskI/Ooperations
Summary
Inthislastchapterofthebook,welearnedanimportantaspectofanydatabase:troubleshootingerrorsthatwemayencounterusingtheMySQLserverorclient.We started the discussion by understanding what troubleshooting is. Wediscussed different ways for initial diagnostics of the error. We understoodcommonMySQL errors and what the error messages mean.We also learnedaboutthewaystofixtheseerrors.WealsolearnedabouttheMySQLserverandclienterrorsandfixesfortheseerrors.Inthelaterpartofthechapter,welearnedabouttheMySQLtroubleshootingapproachandlookedatareal-worldscenario.Prettyimportantstuffforthelastchapter,huh?
That'sitforthebook.
OtherBooksYouMayEnjoy
Ifyouenjoyedthisbook,youmaybeinterestedintheseotherbooksbyPackt:
MySQL8forBigDataShabbirChallawala, JaydipLakhatariya,ChintanMehta,KandarpPatel ISBN:978-1-78839-718-6
ExplorethefeaturesofMySQL8andhowtheycanbeleveragedtohandleBigData
Unlock the new features of MySQL 8 for managing structured andunstructuredBigData
IntegrateMySQL8andHadoopforefficientdataprocessing
PerformaggregationusingMySQL8foroptimumdatautilization
ExploredifferentkindsofjoinandunioninMySQL8toprocessBigDataefficiently
AccelerateBigDataprocessingwithMemcached
IntegrateMySQLwiththeNoSQLAPI
ImplementreplicationtobuildhighlyavailablesolutionsforBigData
MySQL8CookbookKarthikAppigatla
ISBN:978-1-78839-580-9
InstallandconfigureyourMySQL8instancewithoutanyhassle
Get togripswithnewfeaturesofMySQL8 likeCTE,Windowfunctionsandmanymore
Perform backup tasks, recover data and set up various replicationtopologiesforyourdatabase
MaximizeperformancebyusingnewfeaturesofMySQL8likedescendingindexes,controllingqueryoptimizerandresourcegroups
Learn how to use general table space to suit the SaaS or multi-tenant
applications
Analyze slow queries using performance schema, sys schema and thirdpartytools
Manage and monitor your MySQL instance and implement efficientperformance-tuningtasks
Leaveareview-letotherreadersknowwhatyouthink
Pleaseshareyourthoughtsonthisbookwithothersbyleavingareviewonthesite that you bought it from. If you purchased the book fromAmazon, pleaseleaveusanhonestreviewonthisbook'sAmazonpage.Thisisvitalsothatotherpotential readers can see and use your unbiased opinion to make purchasingdecisions,wecanunderstandwhatourcustomersthinkaboutourproducts,andourauthorscanseeyourfeedbackonthetitlethattheyhaveworkedwithPackttocreate. Itwillonly takeafewminutesofyour time,but isvaluable tootherpotentialcustomers,ourauthors,andPackt.Thankyou!