Lucas Jellema (AMIS)NLJUG JFall 2013
6th November 2013, Nijkerk, The Netherlands
On the integrity of data in Java Applications
Agenda
• What is integrity?• Enforcing data constraints
– throughout the application architecture• Transactions• Exclusive Access to …• The Distributed World
3
Definition of Integrity
• Truth– Nothing but the truth
• The Only Truth• [Degree of] success or
completeness ofactions is known
4
Sufficient Integrity
IntegrityIntegrity
CorruptibleCorrupted
π48,23
7,0
“five”
4233,0000002
Uncorrupted
CompleteConsistentReliable
Correct
5
Conference Application
6
Conference Application
Client(HTML 5 & Java Script)
Web TierJavaServer Faces
Business TierJPA
RDBMS
EJBPOJO
Domain Model
7
Validation at entry time
8
Validation at entry timeClient and View
9
Validation at entry timeClient and View
10
More validation at entry time – bean Validation
11
Validation at entry timeBean Validation in View
13
Record (Type) level rules
• Program should be Kids when age < 18; and Management or Developer when age > 18
• Using JavaScript – when either field changes
(handle nulls)– on submit of the entire
record• Using Bean Validation:
custom type validator– in either web-tier or JPA
14
Type Level Constraints with Bean Validation
15
Type Level Bean Validation: Custom Validator
16
Validation Implementation options & considerations
Client(JSF based HTML 5 & Java Script)
Web TierJavaServer Faces
Business TierJPA
RDBMS
EJB
Mobile ClientClient
(pure HTML 5 & Java Script)
RESTful Services
POJO Domain Model
Native HTML 5; JavaScript
Native
Custom;JSF Validator;
Bean Validation
Custom;Bean Validation
Custom;Bean Validation
Native HTML 5; JavaScript
17
But wait – there is more!
• More User Interfaces• More Attendee
Instances• More Entities
& More types of Constraints
• More Users, Sessions,and Transactions
• More Nodes in the Middle Tier Cluster
• More Data Stores
18
Domain model
• Attendee• Speaker• Session• Room• Slot• Attendance
– Booked– Realized
19
Multiple-Instances-of-Single-Entity constraints
• Constraints that cover multiple same type objects/instances– Attendee’s Registration Id is unique– No more than 5 conference attendees from the same company– Not more than two sessions by the same speaker– At most one session scheduled per room per slot– Only one keynote session in a slot– Sessions from up to a maximum of three tracks can be scheduled in the same room
20
Inter entity constraints
• Attendees can only attend one hands-on session during the conference• A person cannot attend another session in a slot in which the session
(s)he is speaker of is scheduled• No more planned session attendances are allowed than the capacity of
the room in which the session is scheduled to take place• If the room capacity is smaller than 100, then no more than 2 people from
the same company may sign up for it• Attendees from Amsterdam cannot attend sessions in room 010
• Common challenge:– Many data change events
can lead to constraint violation
21
Event Analysis for Inter Entity Constraint
• No more planned session attendances are allowed than the capacity of the room in which the session is scheduled to take place
Create, Update (session reference)
Update (capacity [decrease])
Update (room reference)
22
Constraint classification
• Based on event-analysis (when can the constraint get violated) we discern these categories of contraints– Attribute– Tuple– Entity– Inter Entity
• Each category has its own implementation methods,options and considerations– E.g. Multi record instance rules cannot
meaningfully be enforced in client/web-tier
23
Nous ne sommes pas ‘Sans Famille’
24
Nous ne sommes pas ‘Sans Famille’
RDBMS
Client(JSF based HTML 5 & Java Script)
Web TierJavaServer Faces
Business TierJPA
EJB
Mobile ClientClient
(pure HTML 5 & Java Script)
RESTful Services
POJO Domain Model
25
Multiple clients forData Source
RDBMS
Client(JSF based HTML 5 & Java Script)
Web TierJavaServer Faces
Business TierJPA
EJB
Mobile ClientClient
(pure HTML 5 & Java Script)
RESTful Services
POJO Domain Model
ESB.NET
BatchDBA/
Application Admin
Client(JSF based HTML 5 & Java Script)
Web TierJavaServer Faces
Business TierJPA
EJB
Mobile Client Client(pure HTML 5 & Java Script)
RESTful Services
POJO Domain Model
26
Integrity Enforcement in the Persistent Store
• All data is available• Persistent store is the final stop: the buck stops here
– Any alternative data manipulation (channel) has to go to the persistent store– Mobile, Batch, DBA, ESB
• Built-in (native) mechanisms for constraint enforcement– Productive development, proven robustness, scalable performance– For example:
Column Type, PK/UK, FK, Check; trigger• Transactions• Enforcing integrity is integral part of persisting data
– Without final validation, persistent store cannot take responsibility for integrity
27
Multiple-Instances-of-Single-Entity constraints
• No more than 5 conference attendees from the same company
28
Implementation Consideration for Multiple-Entity-Instance rule
• Implementation – how and where?– Is the entire set of data available– Is all associated info available– Is the data set stable?– Can the constraint elegantly be implemented (natively? good framework support?)– Are all data access paths covered?
29
Implementing Multi-Instance constraint ‘5 max per company’
Business TierJPA
Register New Attendee – method A- Ensure L2 Cache is up to date in terms of Attendees (fetch all attendees into cache)- Inspect the collection of attendees for same company- Persist Attendee if collection does not hold 5 (or more)
POJO Domain Model
Attendees
Register New Attendee – method B- Select count of attendees in same company from the Data Store- Inspect the long value- Persist Attendee if long is < 5
L2 CacheAttendees
30
Max 5 per companyJPA Facade enforcement
31
Max 5 per Company – Flaws in JPA Enforcement
• Persist does not [always] ‘post to database’– When more than one attendee is added in a transaction, prior ones are not counted
when the latter are validated
Business TierJPA
Attendees
Facade
POJO Domain Model
Thread 1select countpersistselect countpersistcommit
32
Web Tier
ClientHTML 5 & Java Script
Session A
JPA Facade enforcement in a multi-threaded world
Business TierJPA
Attendees
Facade
POJO Domain Model
Thread 1 Thread 2select countpersistcommit
select countpersistcommit
ClientHTML 5 & Java Script
Session B
33
Transactions
• Logically consistent set of data manipulations– Atomic units of work– Succeed or fail together– Any changes inside a transaction are invisible to other sessions/transactions until the
transaction completes (commits)– Note: during a transaction, constraints may be violated; the only thing that matters:
commit [time]– Transaction ends with succesful commit or rollback –
In both cases, transaction-owned locks are released• ACID (in RDBMS)
– vs BASE (in NoSQL: soft state, eventual consistency - hopefully)• Note: post vs. commit with RDBMS
– Post means do [all] data manipulation (insert, update, delete) but do not commit [yet]– Only upon commit are changes persisted and published
34
Perfect Integrity
35
Fine grained locking
Attendees
Unique Key UK1 on (FirstName, LastName)
Transaction 1 Transaction 2
insert … ('John','Doe',…)
update <JANE> set firstname ='John'
commit
insert … ('Jane','Doe',…)
Lock onUK1_JOHN_
DOE
36
Web Tier
ClientHTML 5 & Java Script
Session A
JPA Facade enforcementExclusive Constraint Checking
Business TierJPA
Attendees
Facade
POJO Domain Model
Thread 1 Thread 2take lockselect countpersistcommit
take lock…select countrollback
ClientHTML 5 & Java Script
Session B
LockMgrATT_MAX
37
Distributed or Global Transaction
• One logical unit of work - involving data manipulations in multiple resources (global transaction composed of local transactions)
Client(JSF based HTML 5 & Java Script)
Web TierJavaServer Faces
Business Tier
RDBMS
EJB
Mobile ClientClient
(pure HTML 5 & Java Script)
RESTful Services
POJO Domain Model
RDBMS JMS
ERPJCA
38
Implementation for Distributed Transaction
• Typical approach: two-phase commit– Each resource locks and validates – then reports OK or NOK back to the transaction
overseeer– When all resources have indicated OK
then phase two:all resources commit and release locks
– When one or more resources signal NOK, then phase two: all resources roll back/undo changes and release locks
• With regards to integrity:– With a distributed transaction,
the integrity for each participant is handled as before; this will result in ‘constraint-locks’ in multiple separate resources
39
Distributed (aka global) transaction within a JVM
• Java EE containers (and various non-EE JTA implementations) support global (distributed) transactions within a JVM– JTA (JSR-907) – based on X/Open XA architecture
• Key element is Transaction Monitor (the container) and Resource Managers (JDBC, EJB, JMS, JCA)
• One non-XA resource can participate (file system, email, …) in a global transaction:– All XA-resources perform Phase One – The non-XA resource does its thing– Upon success of the non-XA resource: others perform Phase two by comitting– Upon failure of the non-XA resource: others roll back
40
Distributed transactions across/outside containers
Step 2:Payment
RDBMS
Client(JSF based HTML 5 & Java Script)
Web TierJavaServer Faces
Business TierJPA
EJB
Mobile ClientClient
(pure HTML 5 & Java Script)
RESTful Services
POJO Domain Model
41
Container
Distributed transactions across/outside containers
• Transaction involving remote containers, Web Services, File System or any stateless transaction participant
• There is no actual common, shared vehicle (like a global XA transaction)– There is not really a coordinated two-phase commit
• Transaction consists of – Any resource does its thing – lock, validate, commit (or rollback), report back– If all resources report succes: great, done– If one resource reports failure the all other resources should perform ‘compensation’
– i.e. rollback/undo effects of a committed transaction
Remote/Stateless Enterprise Resource
Remote/Stateless Enterprise ResourceLocal
Enterprise Resource
Transaction
commit
compensate
commit
42
Compensation
• How to implement a compensation mechanism?• How long after the commit can compensation be requested?• What is the state of the enterprise resource between commit and the
compensation expiry time?• Should the invoker notify the resource that compensation is no longer
required (so the ‘logical locks’/’temporary state’ can be updated)– i.e. the global distributed transaction has succussfully completed
Enterprise Resource
commit
compensate
43
RESTful “transaction” is a distributed transaction
Resource A Resource B Resource C
Client
PUT
PO
ST
DELETE
Domain Model/JPA Cache
44
RESTful “transaction” is a distributed transaction
Resource A Resource B Resource C
Client
PUT
PO
ST
DELETE
Domain Model/JPA
45
Distributed Constraints
• Constraints that involve data collections in multiple enterprise resources
RDBMSRDBMSJMS
Client(JSF based HTML 5 & Java Script)
Web TierJavaServer Faces
Business TierEJB
Mobile Client Client(pure HTML 5 & JS)
RESTful ServicesPOJO
Domain Model
ERPJCA
Table XTable Y
46
Distributed Constraints
• Not more than three attendees (resource A) from the same company may attend a session (resource B)– Insert/Update Attendance requires validation – as does update of Attendee.company
Client
Web Tier
Java EE Business Tier
Client Client
ATTENDANCESATTENDEES
Distributed Lock Manager
Web Tier
Java EE Business Tier
MAX_3_COMP_ATT
ESB
48
Java global (distributed) lock managers
• Within JVM: SynchronousQueue• Across JVMs: Apache ZooKeeper, HazelCast, Oracle Coherence, …
JVM
JVM
JVM
49
Summary
• Which level of integrity is required?• Change of data potentially undermines integrity
– Data change is trigger for constraint validation• Exclusive lock on multi-record validation
– released when transaction commits• Ensure that all data access paths are covered
– Not all data manipulations may come through the Java middle tier• Transactions may include multiple enterprise resources
– That may not be able to participate in a distributed transaction and have to support a compensation mechanism
• True integrity and real robustness are very hard to achieve– Much harder than is commonly assumed
51
Handling Integrity Really Well...
Lucas Jellema (AMIS)
Email: [email protected]: @lucasjellema
Blog: http://technology.amis.nlWebsite: http://www.amis.nl