Date post: | 23-Jan-2017 |
Category: |
Internet |
Upload: | jorge-cardoso |
View: | 77 times |
Download: | 0 times |
OpenStack: Eine Cloud ohne Fehler? Höhere Zuverlässigkeit durch gezieltes Provozieren von Fehlern!
Using Fault Injection to Increase Cloud Reliability!
Götz Brasche / Jorge Cardoso CTO IT PL RnD and Director CSI / Lead Architect Cloud Operations and Analytics
21.06. - 22.06.2016
Cologne, Germany
1
• 47% aller Probleme in Datenzentren resultieren aus Ausfällen von Cloudiensten
• Dauer der Ausfälle reicht von 40 Minuten bis zu fünf Tagen
• Kosten pro Ausfall durchschnittlich 690.204 USD
• Kosten pro Minute knapp 7.000 USD
2
Unplanned downtimeis caused by*
software bugs … 27% hardware … 23%human error … 18%network failures … 17%natural disasters … 8%
* Marcus, E., and Stern, H. Blueprints for High Availability: Designing Resilient Distributed Systems. John Wiley & Sons, Inc., 2003.
3
Google's 2007 found annualized failure rates (AFRs) for drives
1 year old 1.7%3 year old >8.6%
Eduardo Pinheiro, Wolf-Dietrich Weber, and Luiz André Barroso. 2007. Failure trends in a large disk drive population. In Proc. of the 5th USENIX conference on File and Storage Technologies (FAST '07). USENIX Association, Berkeley, CA, USA, 2-2.
4
OpenStack is Growing
Apr 6, 2016
March 2016
5
OpenStack User Survey: A snapshot of OpenStack users’ attitudes and deployments. April 2016. (https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf). Fig. 2.1, Page 9.
Key User Interests
6
What Companies Think About…
Cloud Reliability
7
Gartner• Member of the Magic
Quadrant for x86 Server
Virtualization
Infrastructure
• Member of the Magic
Quadrant for Integrated
Systems
Cloud
Infrastructure
Product
Innovation AwardFrost & Sullivan 2013
10,000Huawei employees working
in cloud computing and
dedicated to meeting every
IT requirement
No. 1Industry-leading performance
according to the SPECvirt
server virtualization
performance benchmark
Best of Show
Award
NominationInterop 2013
DCD Blueprint
AwardChina's first in the data
center industry
100,000Desktops in the world's
largest-scale deployment
Huawei Cloud Computing Investment & Rewards
8
OpenStack and FusionSphere
9
Fusion Sphere Architecture
VMware vSphere
Third-party Virtualization Architecture
FusionCompute FusionStorage FusionNetwork
Huawei Virtualization Architecture
Server Storage Network & Security
Physical Infrastructure
Cloud Storage APIHuawei Open API
eBackup
UltraVR
Backup & DR
SNMP/REST NBI
FusionSphere SOI
FusionManager
Portal
RBAC
Alarm
Log
Open APIResource
ManagementConfiguration API Adapter
Cloud Storage
ManagementVDC/VPC
10
Market Recognitions Grows…
2 times higher scalability
Industry-leading performance in the
SPECvirt test
For the first time in 3 years, Gartner has
introduced a new company, Huawei
(FusionSphere), into the Magic Quadrant
for x86 Server Virtualization
Infrastructure.
HypervisorSPECvirt
ScoreRanking
FusionSphere 5.0 632 1
Linux 6.4 (KVM) 625 2
ESXi 5.1 472 3
http://www.spec.org/virt_sc2013/results/specvirt_sc2013_perf.html
vSphere
5.1
vSphere
6.0
FusionSphere
5.0
3264
128
Nodes per cluster
• High performance
− < 5% CPU performance overheads
− Support for database, email, and
ERP and CRM services
• High reliability
− Proactive event detection
− Active/Standby management nodes
− Upgrade without service interruption
− Multi-level disaster recovery plans
Support for Critical Applications
VMware
MicrosoftOracle
Parallel
CitrixHUAWEI
LEADERSCHALLENGERS
NICHE PLAYERS VISIONARIES
As of July 2014COMPLETENESS OF VISION
Source: Gartner (July 2014)
AB
ILIT
Y T
O E
XE
CU
TE
Red Hat
Member of the Gartner Magic Quadrant
for x86 Server Virtualization
Infrastructure
11
FusionSphere 5.1 Key Performance Indicators
Physical Server/VM Performance Indicator Value
Max. number of vCPUs (virtual SMP) per VM 128
Max. memory size per VM 4 TB
Max. virtual disk capacity per VM 64 TB
Max. number of virtual disks per VM 60
Max. number of virtual NICs per VM 12
Max. number of logical CPU cores per physical server 480
Max. memory size per physical server 12 TB
Max. number of powered-on VMs per physical server 1024
12
FusionSphere 5.1 Key Performance Indicators
Management Indicator Value
Max. number of physical servers per logical cluster 128
Max. number of VMs per logical cluster 3000
Max. number of logical clusters supported by a VRM node 32
Max. number of hosts supported by a VRM node 1024
Max. number of VMs supported by a VRM node 10,000
Max. number of VRM nodes that can be cascaded 16
Max. number of physical servers supported by cascaded VRM nodes 4096
Max. number of VMs supported by cascaded VRM nodes 80,000 (best practice in the industry)
15
FAILURES ARE INEVITABLE!
THE BEST WE CAN DO IS BE PREPARED FOR THEM AND LEARN FROM THEM
TEST, REPAIR, LEARN & PREDICT !
Kripa Krishnan, Technical Program Director from Google
16
One reason [Netflix]: It’s the lack of control over the underlying hardware, the inability to configure it to ensure 100% uptime
Why does using a cloud infrastructure requires advanced approaches for resiliency?
17
A program designed to increase resilience by purposely injecting
major failuresDiscover flaws and subtle dependencies
Amazon AWS: GameDay
“That seems totally bizarre on the face of it, but as you dig down, you end up finding
some dependency no one knew about previously […] We’ve had situations where we
brought down a network in, say, São Paulo, only to find that in doing so we broke our
links in Mexico.”
18
Google DIRT (Disaster Recovery Test) Annual disaster recovery & testing exercise
8 years since inception
Multi-day exercise triggering (controlled) failures in systems and process
Premise 30-day incapacitation of headquarters following a disaster
Other offices and facilities may be affected
When “Big disaster”: Annually for 3-5 days
Continuous testing: Year-round
Who 100s of engineers (Site Reliability, Network, Hardware, Software, Security, Facilities)
Business units (Human Resources, Finance, Safety, Crisis response etc.)
Google: DiRT
Source http://flowcon.org/dl/flowcon-sanfran-2014/slides/KripaKrishnan_LearningContinuouslyFromFailures.pdf
19
Netflix: Chaos MonkeyFewer alerts for
ops team
Amazon EC2 and Amazon RDS Service Disruption in the US East RegionApril 29, 2011
September 20th, 2015: Amazon’s DynamoDB service experienced an availability issue in their US-EAST-1
Transfer traffic to east region
20
Huawei: Butterfly Effect
-- Butterfly Effect System --Enables to Automatically Test and Repair OpenStack and Cloud
Applications
CLOUD APPLICATION
HUAWEI FusionSphere
The system works by intentionally injecting different failures, test the ability to
survive them, and learn how to predict and repair failures preemptively
Failure
Repair
Test
21
Approach to Cloud Reliability
1
2 3
4
7
5
6
Design & Deploy
Test Infrastructure
Monitoring Facilities
Design & Execute Fault-Injection Plan
Identify Damages
Predict Future Errors
Automatic Repair
Repair & Learn
OpenStack
Inject Fault F51
Fault F51(mysql unavailable)
OpenStack
22
Design & Execute Fault-Injection PlanBest way to avoid failure: Fail constantly
Kill cinder database(Simulate update failure)
Introduce delay in messages(Full-scale traffic shows where the real bottlenecks are)
Operation ErrorOPENSTACK_KEYSTONE_URL = "http://%s:5000/v2.0" % OPENSTACK_HOST
Operation Error/etc/nova/nova.confDelete: auth_strategy=keystone
Remove driver to HDRemove access to NFS(Simulate hardware failure)
The main testing framework of OpenStack is called Tempest, an opensource project with more than 2000 tests: only black-box testing (test only access the public interfaces)
2
OpenStack
OpenStackimage list
server create
security group create
Learning
OpenStackimage list
server create
security group create
Learning
OpenStack
Inject Fault F51
Learning
OpenStackimage list
server create
security group create
Learning
OpenStackimage list
server create
security group create
Learning
OpenStack
Monday
OpenStackimage list
Monday
OpenStackimage list
Monday
OpenStack
Tuesday
OpenStackimage list
Tuesday
OpenStackimage list
Tuesday
OpenStackimage list
Tuesday
?
OpenStackimage list
Tuesday
Fault F51(mysql unavailable)
OpenStack
Inject Fault F51
Faults Behavior
Fault F51(mysql unavailable)
Root Cause
$ openstack image list
2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUGdogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.553 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004acd50> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2808:05:30.555 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.565 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2808:05:30.566 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.567 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.573 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0049b090> acquired _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.576 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.581 23842 DEBUG oslo.middleware.request_id [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] <BE_ID> (backward<<<) new x-openstack-request-id: <BE_ID><iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181>__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:95 <<keystone-main.service>>2016-04-28 08:05:30.585 11693 DEBUG eventlet.wsgi.server [-] (11693) accepted ('192.168.4.231', 51774) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 <<glance-api.service>>2016-04-28 08:05:30.588 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1>__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<glance-api.service>>2016-04-28 08:05:30.589 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<glance-api.service>>2016-04-28 08:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Determining version of request: GET /v1/images/detail Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:47 <<glance-api.service>>2016-04-28 08:05:30.590 11693DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Matched version: v1 process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:72 <<glance-api.service>>2016-04-2808:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Using url versioning process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:60 <<glance-api.service>>2016-04-28 08:05:30.591 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] new path /v1/images/detail process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:73 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.auth.identity.v2 [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Making authentication request tohttp://192.168.4.231:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:86 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.session [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - -REQ_DOMA] <BE_ID> (bridge>>>) Call from keystoneclient/session.py to Keystone <BE_ID> from request /usr/lib/python2.7/site-packages/keystoneclient/session.py:373 <<glance-api.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<keystone-admin.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__/usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<keystone-admin.service>>2016-04-28 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 <<keystone-admin.service>>2016-04-28 08:05:30.598 23841 INFOkeystone.common.wsgi [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] POST http://192.168.4.231:35357/v2.0/tokens <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException_enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0740f610> acquired _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.600 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.603 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.618 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148<<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f002c5ed0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.628 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-
Root Cause Localization
$ egrep ‘ERROR | TRACE’ /var/log/nova/nova-api.log | wc
$ grep ‘req-’ /var/log/glance/glance-api.log
Typically, the first thing to look in the logs when you hit a
problem is for the error message or the stack trace that has
the details on which part of the python module the problem
is generated, for that particular component.
This should explain the problem in detail, as well as its origin.
You should also look for the exception that gets logged for
any failure, which will also help identify the root cause of the
failure. If there are no exception/stack trace messages getting
dumped to the logs, it means that the problem might not be of
severe impact and the user can continue with their operation.
However, in such cases, at least a warning message will get
logged to make sure that we don't miss out on anything that's
happening in the FusionSphere environment.
The root cause of most of the issues seen in FusionSphere can
be triaged with the default logging levels. If the support team
requires more detailed logging, you can enable the debug
logs and perform the action again to capture all the details.
To have a stable environment, you want to detect failure
promptly and determine causes efficiently. With a distributed
system, it's even more important to track the right items to
meet a service-level target. By knowing where the logs are and
how to manage them, you can analyze most issues you
encounter, allowing you to keep your environment running
smoothly.
Limitations of Troubleshooting Approaches
Although today’s programs are orders of magnitude more complex than those of 30 years ago, many people still use printf to log to console or local disk, and use some combination of manual inspection and regular expressions to locate specific messages or patterns.FEBRUARY 2012, VOL. 55, NO. 2, COMMUNICATIONS OF THE ACM
Manual, complex, error-prone, and expensive
http://www.slideshare.net/tomoya/openstack-at-ntt-resonant-lessons-learned-in-web-infrastructure
One (very) simple command: glance.images.list()Size: 316K # messages:1499 # DEBUG: 1068 # INFO: 23 # Others: 408
DOCOMO has shown
(100GB and 80M lines)/day for
100 nodes
42
Activity Monitoring2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUGdogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.553 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004acd50> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2808:05:30.555 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.565 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2808:05:30.566 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.567 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.573 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0049b090> acquired _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.576 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.581 23842 DEBUG oslo.middleware.request_id [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] <BE_ID> (backward<<<) new x-openstack-request-id: <BE_ID><iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181>__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:95 <<keystone-main.service>>2016-04-28 08:05:30.585 11693 DEBUG eventlet.wsgi.server [-] (11693) accepted ('192.168.4.231', 51774) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 <<glance-api.service>>2016-04-28 08:05:30.588 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1>__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<glance-api.service>>2016-04-28 08:05:30.589 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<glance-api.service>>2016-04-28 08:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Determining version of request: GET /v1/images/detail Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:47 <<glance-api.service>>2016-04-28 08:05:30.590 11693DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Matched version: v1 process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:72 <<glance-api.service>>2016-04-2808:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Using url versioning process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:60 <<glance-api.service>>2016-04-28 08:05:30.591 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] new path /v1/images/detail process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:73 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.auth.identity.v2 [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Making authentication request tohttp://192.168.4.231:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:86 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.session [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - -REQ_DOMA] <BE_ID> (bridge>>>) Call from keystoneclient/session.py to Keystone <BE_ID> from request /usr/lib/python2.7/site-packages/keystoneclient/session.py:373 <<glance-api.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<keystone-admin.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__/usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<keystone-admin.service>>2016-04-28 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 <<keystone-admin.service>>2016-04-28 08:05:30.598 23841 INFOkeystone.common.wsgi [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] POST http://192.168.4.231:35357/v2.0/tokens <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException_enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0740f610> acquired _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.600 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.603 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.618 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148<<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f002c5ed0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.628 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-
43
Activity Monitoring2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUGdogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.553 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004acd50> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2808:05:30.555 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.565 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2808:05:30.566 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.567 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.573 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0049b090> acquired _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.576 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.581 23842 DEBUG oslo.middleware.request_id [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] <BE_ID> (backward<<<) new x-openstack-request-id: <BE_ID><iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181>__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:95 <<keystone-main.service>>2016-04-28 08:05:30.585 11693 DEBUG eventlet.wsgi.server [-] (11693) accepted ('192.168.4.231', 51774) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 <<glance-api.service>>2016-04-28 08:05:30.588 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1>__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<glance-api.service>>2016-04-28 08:05:30.589 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<glance-api.service>>2016-04-28 08:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Determining version of request: GET /v1/images/detail Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:47 <<glance-api.service>>2016-04-28 08:05:30.590 11693DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Matched version: v1 process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:72 <<glance-api.service>>2016-04-2808:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Using url versioning process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:60 <<glance-api.service>>2016-04-28 08:05:30.591 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] new path /v1/images/detail process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:73 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.auth.identity.v2 [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Making authentication request tohttp://192.168.4.231:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:86 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.session [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - -REQ_DOMA] <BE_ID> (bridge>>>) Call from keystoneclient/session.py to Keystone <BE_ID> from request /usr/lib/python2.7/site-packages/keystoneclient/session.py:373 <<glance-api.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<keystone-admin.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__/usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<keystone-admin.service>>2016-04-28 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 <<keystone-admin.service>>2016-04-28 08:05:30.598 23841 INFOkeystone.common.wsgi [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] POST http://192.168.4.231:35357/v2.0/tokens <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException_enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0740f610> acquired _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.600 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.603 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.618 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148<<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f002c5ed0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.628 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-
44
Activity Monitoring2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUGdogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.553 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004acd50> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2808:05:30.555 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.565 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-2808:05:30.566 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.567 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.573 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0049b090> acquired _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.576 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.581 23842 DEBUG oslo.middleware.request_id [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] <BE_ID> (backward<<<) new x-openstack-request-id: <BE_ID><iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181>__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:95 <<keystone-main.service>>2016-04-28 08:05:30.585 11693 DEBUG eventlet.wsgi.server [-] (11693) accepted ('192.168.4.231', 51774) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 <<glance-api.service>>2016-04-28 08:05:30.588 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1>__call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<glance-api.service>>2016-04-28 08:05:30.589 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<glance-api.service>>2016-04-28 08:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Determining version of request: GET /v1/images/detail Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:47 <<glance-api.service>>2016-04-28 08:05:30.590 11693DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Matched version: v1 process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:72 <<glance-api.service>>2016-04-2808:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Using url versioning process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:60 <<glance-api.service>>2016-04-28 08:05:30.591 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] new path /v1/images/detail process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:73 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.auth.identity.v2 [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Making authentication request tohttp://192.168.4.231:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:86 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.session [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - -REQ_DOMA] <BE_ID> (bridge>>>) Call from keystoneclient/session.py to Keystone <BE_ID> from request /usr/lib/python2.7/site-packages/keystoneclient/session.py:373 <<glance-api.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<keystone-admin.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-83c5225b-1d19-44d2-a338-24c8ee994bd2> __call__/usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<keystone-admin.service>>2016-04-28 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] 08:05:30.597 23841 DEBUG keystone.middleware.core [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/site-packages/keystone/middleware/core.py:310 <<keystone-admin.service>>2016-04-28 08:05:30.598 23841 INFOkeystone.common.wsgi [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] POST http://192.168.4.231:35357/v2.0/tokens <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException_enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.599 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0740f610> acquired _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.600 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.603 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.618 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.619 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.622 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148<<keystone-admin.service>>2016-04-28 08:05:30.623 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0043b6d0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-admin.service>>2016-04-28 08:05:30.625 23841DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-admin.service>>2016-04-28 08:05:30.626 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f002c5ed0> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-admin.service>>2016-04-28 08:05:30.628 23841 DEBUG dogpile.core.dogpile [iii-83c5225b-1d19-44d2-a338-
Activity Monitoring2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Releasedcreation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.549 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.550 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.552 23842 DEBUG dogpile.core.dogpile[iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value,waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.553 23842 DEBUG dogpile.core.dogpile[iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004acd50> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.555 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Releasedcreation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value,waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.562 23842 DEBUG dogpile.core.dogpile[iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -]NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile[iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.565 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] valuecreation lock <dogpile.cache.region._LockWrapper object at 0x7f8f004ac810> acquired _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.566 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.567 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.573 23842DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:94<<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Calling creation function _enter_create/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:148 <<keystone-main.service>>2016-04-28 08:05:30.574 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:127 <<keystone-main.service>>2016-04-28 08:05:30.57423842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f8f0049b090> acquired_enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:131 <<keystone-main.service>>2016-04-28 08:05:30.576 23842 DEBUG dogpile.core.dogpile [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 <<keystone-main.service>>2016-04-28 08:05:30.58123842 DEBUG oslo.middleware.request_id [iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181 - - - - -] <BE_ID> (backward<<<) new x-openstack-request-id: <BE_ID><iii-e3cf60c9-ba7f-4ee2-9eaf-d070f93f2181> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:95 <<keystone-main.service>>2016-04-28 08:05:30.585 11693 DEBUG eventlet.wsgi.server [-](11693) accepted ('192.168.4.231', 51774) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:826 <<glance-api.service>>2016-04-28 08:05:30.588 11693 DEBUGoslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) openstack.request_id: Correlation<BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:88 <<glance-api.service>>2016-04-2808:05:30.589 11693 DEBUG oslo.middleware.request_id [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>) x-openstack-request-id: Correlation <BE_ID><NONE><iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1> __call__ /usr/lib/python2.7/site-packages/oslo_middleware/request_id.py:89 <<glance-api.service>>2016-04-28 08:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Determining version ofrequest: GET /v1/images/detail Accept: */* process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:47 <<glance-api.service>>2016-04-2808:05:30.590 11693 DEBUG glance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Matched version: v1process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:72 <<glance-api.service>>2016-04-28 08:05:30.590 11693 DEBUGglance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Using url versioning process_request/usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:60 <<glance-api.service>>2016-04-28 08:05:30.591 11693 DEBUGglance.api.middleware.version_negotiation [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] new path /v1/images/detail process_request/usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:73 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.auth.identity.v2 [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] Making authentication request to http://192.168.4.231:35357/v2.0/tokens get_auth_ref/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:86 <<glance-api.service>>2016-04-28 08:05:30.592 11693 DEBUG keystoneclient.session [iii-9a7ba455-6a15-422d-bd1a-a41adbb0afc1 REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (bridge>>>) Call from keystoneclient/session.py to Keystone <BE_ID> from request /usr/lib/python2.7/site-packages/keystoneclient/session.py:373 <<glance-api.service>>2016-04-28 08:05:30.596 23841 DEBUG oslo.middleware.request_id [iii-83c5225b-1d19-44d2-a338-24c8ee994bd2REQxxUSER REQxTENAN - - REQ_DOMA] <BE_ID> (forward>>>
Logi
cal t
ime
0
10
20
30
40
50
60
30%70%
Variability
Warning Y
Warning A
Warning B
Warning C
Error Z
Distributed Execution Graph
Google's Tracing System (see also X-Trace and Magpie) Originally created to understand the system behavior from a search request
Today Google's production clusters generate >1 TB/day of sampled trace data
Dapper: a Large-Scale Distributed Systems Tracing Infrastructure
Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, Chandan Shanbhag, Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Google, Inc. (2010).
// Java:Tracer t = Tracer.getCurrentTracer();String request = ...;if (hitCache())
t.record("cache hit for " + request);else
t.record("cache miss for " + request);
Annotation 41 Java and 68 C++ applications have
custom annotations to better understand
intra-span activity
Overall approach When a thread handles a traced control path, Dapper attaches a trace
context to thread-local storage.
When the control flow library (threading, control flow, RPC) is used to
schedule callbacks, Dapper attached a trace context.
Performance Basic instrumentation as small as possible and record only a fraction
of all traces using sampling (1/1000).
Daemon uses < 0.3% CPU during collection. Small memory footprint
47
Dapper
Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, Chandan Shanbhag, Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Google, Inc. (2010).
FingerprintLo
gica
l tim
e
0
10
20
30
40
50
60
30%70%
Variability
Warning Y
Warning A
Warning B
Warning C
Error Z
Distributed Execution Graph
http://acumendatasystems.com/how-fingerprint-biometric-time-clocks-work/
failureOK
Fingerprint Analytics
Can you identify your failures?
Key Challenges Analysis New matching algorithms Real-time fingerprint detection Fingerprint prediction Alignments and nested FP
M8
M1
Service a Service kService b
$ openstack image list Fault 15aOK
51
Application Fields
Failure Mode and Effect Analysis (FMEA)
Security and Intrusion detection Performance Analysis
Bottleneck
Cloud Accountability
52
OpenStack Engineers positions
Rapid prototyping of cool ideas: propose it today, code it, and show it running in
3 months…
Internship positions for MSc students
Fault injection, fault models, fault libraries, fault plans, brake and rebuild systems
all day long, …
Innovative PoCs
Solving difficult challenges of real problems using quick and dirty prototyping
Join the Cause!
53
Industry-Academia Workshop on
Cloud Reliability and Resilience
This workshop intends to bring
together industry and academia
to identify the most relevant
requirements in the field of cloud
reliability and resilience, on one
hand, and existing state-of-the-art
solutions, on the other.
We invite engineers, scientists,
regulators, and experts to discuss
and contribute to the creation of a
new generation of highly reliable
cloud platforms.
November Event
BERLIN 7-8 NOVEMBER 2016 6, Ernst-Reuter-Platz 710587 BerlinGermany
Copyright©2016 Huawei Technologies Co., Ltd. All Rights Reserved.
The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product
portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive
statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time
without notice.
HUAWEI ENTERPRISE ICT SOLUTIONS A BETTER WAY
Thank you!