#ILTAG50
Artificial Intelligence in Law:
AI in Action(Part 2 of 3)
Stephen Allen
Head of Legal
Service Delivery
Hogan Lovells
International LLP
Katie DeBord
Partner and Chief
Innovation Officer
Bryan Cave
Neil Cameron
Chief Consulting
Officer
NCCG
MODERATOR: CEO & Co-
Founder
ROSS
Intelligence
SPEAKERS:
Andrew Arruda
Steve Harmon
VP and Deputy
General Counsel,
Legal
Cisco
Ron Friedmann
Partner
Fireman and
Company
Artificial Intelligence in Law:
Top Products in Action
Why we’re here and where we’re going.
CEO & Co-Founder
ROSS Intelligence
Andrew Arruda
Artificial Intelligence in Law:
Top Products in Action
What the Surveys Tell Us about AI
Ron Friedmann
Partner
Fireman and
Company
AI IN THE GLOBAL ECONOMYMCKINSEY 2017
“The current AI wave is poised to finally break through”
“Five elements of successful AI transformations”
“Six characteristics of early AI adopters”
ADOPTION OF AI IN LEGAL:SURVEY RESULTS
Altman Weil, Law Firms in Transition, 2017
Altman Weil, Law Firms in Transition, 2017
Survey, June 2017, ILTA KM Virtual Roundtable (37 responses)
Altman Weil
Law Firms in
Transition, 2015
AI IMPACT IN LEGAL
Aderant
Innovation
in Global
Legal
Business,
2017
That’s the high level view
Now for the AI in practice
Artificial Intelligence in Law:
Top Products in Action
Katie DeBord
Partner and Chief
Innovation Officer
Bryan Cave
Artificial Intelligence: A New Force Of Nature?
BRYAN CAVE’S AI APPROACH: UNDERSTAND, EXPERIMENT, AND APPLY
ACT I: UNDERSTAND
CT and PEG
• Combination of statisticians and technologists tasked to figure out how we could use leverage data better and use technology to do so.
TechX:
• Group of attorneys and allied professionals worldwide that experiment with advanced legal tech based on their practice areas.
ACT II: EXPERIMENT
Two primary areas of focus for BC’s machine learning/AI experimentation:
•Internal proofs of concept using high volume of unstructured data that coalesces under a common theme
•External proofs of concept using off the shelf research, document review, litigation analytics, and decision-making support tools
EXPERIMENTATION WITH OFF THE SHELF TOOLS
•Due Diligence (Kira, eBrevia)
•KM (RAVN)
•Legal Research (ROSS)
•Decision-making Support (Neota Logic)
•Patent/Litigation Analytics (Lex Machina)
Through TechX, we have a process to
regularly pilot and obtain
feedback on AI-Based tools
for:
•Interoperability with other tools
•Benefit of value add vs. risk of tech fatigue
•Security
Considerations Include:
ACT III: APPLY
USE AI/Machine Learning To
Categorize, Harness, and Understand
Unstructured Data Within Own Systems.
APPLY MACHINE LEARNING/AI BASED TECH TO PRACTICE
WHERE APPROPRIATE
MACHINE LEARNING APPLIED TO UNSTRUCTURED DATA TO CREATE “TASKER TOOL”
Method: Applied Bayesian Classifier to unstructured time entry narratives.
Goal: Enhance AFA pricing capabilities by more accurately understanding the levels of effort associated with tasks/phases within matters.
TASKER TOOL TRAINING INTERFACE
EXTENSION OF TASKER TOOL: USING WORD CLOUDS TO DRIVE BETTER BEHAVIOR
MACHINE LEARNING APPLIED TO SERIAL LITIGATION
Problem Statement: 5% of matters
account for over 50% of overall portfolio cost. Can these
matters be identified and resolved earlier
in their lifecycle?
IDENTIFIED VARIABLES IN PORTFOLIO
1. Variance from 2.5 kids
2. Body Mass Index
3. GMAT Scores
4. Cholesterol level
5. Zillow Zestimate
6. Sleep Number
7. LinkedIn Profile Completion %
8. United Miles
9. Daily Average Steps Walked
10.# Cats Owned
11.Etc.
Linear equation explaining some of the variability in total cost
Statistician
Dozens of Variables Minitab
RESULTS
Client’s litigation costs reduced by 2/3
•Outcomes at least constant if not improved
•Generated numerous best practices around knowledge management, staffing models, settlement strategies, etc. that now get leveraged broadly
• Identified and managed emerging legal theories and adopted common strategies for repeat claims types and counsel
AI-BASED “OFF THE SHELF” TOOLS
Opportunity Drivers:
•High Volume Low Margin Work
•Contract Management Applications
•Highly Regulated
•TECHX Attorney-Sponsored
ROI:
•Escalating benefit to end-user
•More consistent delivery of specific layers of legal advice and service
•Attorney participation and thought leadership
•Serendipitous Ideation
How and When
We Make a
Decision to
Deploy
ISSUE SOLUTION
Legal department of national retail brand
bogged down with requests for term
sheets
Application created allowing client to
generate standard, pre-approved term
sheet after answering a series of
questions
Client requires high volume of NDA’s,
which caused delays in business line
Business line can now self-generate
NDA’s, saving time and creating better
work flow
Need to streamline process of updating
franchise disclosure documents
Questionnaire makes process easy,
lowers risk and increases efficiency
DECISION-MAKING SUPPORT USE CASES
COMMON THEMES
Change is Hard (but it’s worth it):
✓ Need to carve out time to focus
✓ Invest in process
✓ Invest in content
✓ Capture and analyze data
✓ Be entrepreneurial – take risks and fail quickly
✓ Don’t let the perfect be the enemy of the good
Artificial Intelligence in Law:
Top Products in Action
Making the complex simple
and the simple
effective……..and dodging
silver bullets…Stephen Allen
Global Head of
Legal Service
Delivery
Hogan Lovells
MACHINE ASSISTED DOCUMENT REVIEW - AS A
SERVICE
MACHINE LEARNING AND UNDERSTANDING OUR
BUSINESS
MACHINE LEARNING AND UNDERSTANDING OUR
BUSINESS
MACHINE LEARNING AND UNDERSTANDING OUR
BUSINESS
MACHINE LEARNING AND UNDERSTANDING OUR
BUSINESS
Artificial Intelligence in Law:
Top Products in Action
Case Study: PoC for ‘AI Matters’ in Bird & Bird
Chief Consulting
Officer
NCCG
Neil Cameron
‘AI MATTERS’
The Business Problem
• Matter fee forecasts – budgets – are increasingly being
demanded by clients and courts
• Attorneys are not good at this – they suffer from all the
’normal’ planning cognitive biases – and some new ones
– In particular, they conflate ‘costing’ and ‘pricing’
– they do not ‘learn’ to improve their estimates
• The results are reductions in matter realisation, firm
profitability, and client satisfaction
Write-Offs
• If fee estimate is exceeded attorneys tend to write off the
difference, or ask the client for more money, or both
• Unless they have managed the client’s expectations
throughout, they will usually write off much of it
• Before the GFC many law firms used to operate on 80%-85%
realisation – now this tends to be nearer 70%
• A firm turning over $1b pa, will have written off $428m
• A 2% improvement in realisation on that would be $28m
Initiatives to Improve Forecasting
• Attorney Training
• Matter Breakdown
• Matter Templates
• ‘Reference Class Forecasting’
Reference Class Forecasting - Wikipedia
• “Human judgment is generally optimistic due to overconfidence and
insufficient consideration of distributional information about outcomes.
Therefore, people tend to underestimate the costs, completion times,
and risks of planned actions.
• “Reference class forecasting predicts the outcome of a planned action
based on actual outcomes in a reference class of similar actions to that
being forecast.
• “The theories behind reference class forecasting were developed by
Daniel Kahneman, the theoretical work helped Kahneman win the Nobel
Prize in Economics.”
‘AI Matters’
• Take a ‘class’ of matters – e.g. M&A
• Identify the key ‘complexity factors’
• Run algorithms against source data and those factors against previous
matter fees
• Train the system on the relationship between them
• Use the system to forecast new matter fee estimates based on known
and assumed complexity factor
• Do this in partnership with the client
‘AI Matters’
• Take a ‘class’ of matters – e.g. M&A
• Identify the key ‘complexity factors’
• Run algorithms against source data and those factors against previous
matter fees
• Train the system on the relationship between them
• Use the system to forecast new matter fee estimates based on known
and assumed complexity factor
• Do this in partnership with the client
Bird & Bird PoC
• Started with M&A
• Worked with lawyers to identify the most likely complexity factors
• Ran models, refined and re-ran models, and again
• Attorneys predict accuracy of fees to a correlation factor of 0.25
• Our current model predicts fees to an accuracy of 0.75
• Phase 2 will work on refining current complexity factors, working with
attorneys to identify new ones, and building a user interface
Humans
(Attorneys)
• 0.25
AI Matters
• 0.75
Conclusion
• Next steps:
– Enhance and develop further accuracy
– Move on to other types of work
– Use it to ‘fingerprint’ matters for knowledge management and quality assurance
purposes
• Before we start using AI to replace what attorneys do; let’s start using it
to undertake what attorneys can’t, or won’t, do
Artificial Intelligence in Law:
Let’s Discuss our AI in Action
Stephen Allen
Head of Legal
Service Delivery
Hogan Lovells
International LLP
Katie DeBord
Partner and Chief
Innovation Officer
Bryan Cave
Neil Cameron
Chief Consulting
Officer
NCCG
Steve Harmon
VP and Deputy
General Counsel,
Legal
Cisco
Ron Friedmann
Partner
Fireman and
Company