1216 - Smooth Migration from IBM

Domino to IBM Verse : The Vital

Metrics You Do Need

Jérôme Deniau, move4ideas / September, 16th 2016

About Jérôme Deniau

• Co-founder & CTO at

• Member of Penumbra – an international consortium of selected Business Partners

pooling their talents and resources

• Designer of for Domino in 2006

• IBM Design Partner member

• Certified IBM/Lotus Instructor since 1993

• Certified Advanced Domino Administrator and Developer since Lotus-Notes R3

• Jerome.deniau@move4ideas.com

Abstract• IBM Domino server is having an exceptional functionality and rich

features which fit perfectly for customer and their very own business

needs.

• IBM Verse defines a #NewWayToWork that comes with Social

powerful features you can use and activate without compromising your

IBM Domino investment.

• How to migrate your IBM Domino server mail system to IBM Verse

simply without no hassle downtime and make your end-users happy

with the migration process.

Agenda

• Why Are We Here

• Pre-requisites

• W4

• Metrics

Why Are We Here

• Audience

Main target: IT Department and C-level but it is not limited

• Architecture change

• Migration headaches

• Where is my starting point

• #NewWayToWork

• Keep It Simple (but NOT) Stupid

• Keep in mind the 4 V’s (Value/Variety/Velocity/Volume)

Prerequisites

• Before Rush

Knowledge

Hard work

Attitude

Prerequisites - Knowledge

• This is the very first step

Can’t do without!

• Source

Have a good knowledge of your Domino environment and

architecture

• Target

Know your target and Procedures

On-premise

Hybrid

Pure-cloud

Prerequisites – Hard Work

• Prepare your team and your system

Complete checklist of your system

No rush

The simpler the better

Prerequisites - Attitude

• Last but not least

Be positive

Be invested

Be inventive

Be an evangelist

• Make it a success

W4

• The W power

The “Why We Should Move” is really far behind us by now

• The four vital points:

Who

What

Where

When

W4

W4 – What For?

• You will be able to define a set of population to migrate them altogether

When

WhatWho

Where

W4 – The Who Dilemma• Active vs inactive users

What defines active people vs inactive people

One can be active without answering mails

Database Activity

Mail rules

• Relations: people who collaborate, work together

• Domino applications and mail agents

• External usersWho

W4 – The What Dilemma

• Smartphones vs IBM Notes addicts

• Attachments: Enterprise data

• Huge databases

• Sensitive data

What

W4 – The Where Dilemma• Limited with IBM Verse for the cloud version

Asia Pacific

Europe

United States

• For this session metrics will take in account these targets instead:

Hybrid

On-premise

Pure cloud (private or public) Where

W4 – The When Dilemma• Last step at last

Once populations are defined

Deployment plan to run the migration at the right time for the population to

migrate

• Off-hours

• Off-peak

• Avoid Monday morning burstWhen

W4 – The Big Picture

W4 – The Big Picture

WhenWhatWho

Where

W4 – The Big Picture

WhenWhatWho

Where

Metrics For The Who Dilemma

Use Case

• We want to migrate only active users!

How to get this information from Domino ?

Active vs Inactive Users

Active vs Inactive Users• You do need to distinguish between false active and false inactive

Comparing databases sizes

Domino built-in tools for accurate information

Database properties: user activity detail

Domino log.nsf ‘Database Usage’ view

Domino activity logging (activity.nsf)

License Tracking (userlicenses.nsf)

Active vs Inactive Users• Quick and dirty metrics based on database files sizes

Export the mail files view from IBM Administrator on regular basis

Too simple and quite confusing

CTRL-C + Paste into a spreadsheet or better use IBM Watson Analytics

Active vs Inactive Users• Less dirty: log.nsf using the Usage views

Statlog should run regularly

Active vs Inactive Users• Quick and dirty (cont.)

Database sizes metrics are insufficients

View indexes sizes are missing (you can get rid of them with

purge or compact –D)

Take care of:

o Updall –c (in clustered or non clustered environment)

Need a tool to analyze correctly the information

Active vs Inactive Users

• Database user detail

No_Force_Activity_Logging=0 and Statlog task should run

Pros:

• simple to use

• C api calls can be used to grab the information

Cons:

• Results are on a database basis

• Erratic problems

• Limited use due to the 32kb limit (Wait for the RNext 16MB)

Solution: use scanEZ from Ytria

Active vs Inactive Users• Database user detail samples

• IBM Notes

• Ytria DatabaseEZ

Active vs Inactive Users

• Domino log.nsf

Information is set by the Statlog task

Add Log_Sessions=1 or 2 in notes.ini

Remove Log_Disable_Session_Info=1 if exists in notes.ini

Active vs Inactive Users• Domino activity logging

Simple configuration

Domino.Notes.Session

Domino.Notes.Database

Validation: show server

IBM Technote for retention period

#1230016 (from 2010) default is 7 days so:

o Log=logfiles/log.nsf,1,0,7,40000,[1-6]

Active vs Inactive Users

• Domino activity logging (continued)

Run the Activity analysis tool

Active vs Inactive Users• License Tracking

• Simple configuration

• Adminp is required to aggregate (so take care of that)

• All views under ‘Active Users’ can be a good starting point

Active vs Inactive Users

• Conclusions

A user who does not send mail can be active

Message Tracking Collector?

Automatic or manual reports in reports.nsf:

o Top 25 xxx

o Messages size

o Message volume

Tips• Mail Tracking Collector retention period

Default is 30 days

Accommodate this value according to your needs

Use Case

• We want to migrate people involved in ‘communities’.

Relationship

Relationship• How to define groups of people working together

Do we have metrics out of the box?

Technically:

Complicated if we talk about Communities detection for Domino

Complicated if talking about Dynamic Graphs

Complicated to implement Louvain method

• So what is available and simple to use?

Relationship• Message Tracking Collector

Yes but what for? Better compare manually the information

Inefficient for relationship but fields can be useful

Relationship• Increasing Domino logging for mail:

Log_MailRouting=30

Result:

Once again limited: but

Data is stored in log.nsf in the “Mail Routing” view

o If LOG_MailRoutingToMiscEvent=1 is not specified

Data is stored in text file(s) console log(s) in <Domino data Folder>/IBM_TECHNICAL_SUPPORT

Relationship• Domino Mail Journaling

High volume on some sites

Needs replication between servers

Caution due to high volumes and data retention delays

Needs development to build a map for user relationship

Do not encrypt $TUA nor $MessageID

$TUA field can help you group mail exchanges on one subject amongst a group of user

o Beware: you can have multiple $TUA fields

Relationship• Activity Logging

Domino.Mail entry type

As-is needs a bit of design to make it work the way you need it

Use the activity logging analysis tool from IBM Administrator

Deposited view

Processed view

Relationship• What the analysis tool does is:

Get the documents from log.nsf

Form=“Mail Routing”

Relationship

• Organizational Units can help

Group people according to their OU’s

• Domino mail server

Group people according to their home server

Relationship• Conclusions

Not really an out of the box functionality

Need a specific development

Or a third party tool

Use Case

• We want to distinguish between ‘Human’ and ‘non Human’

users

Domino Applications & Mail Agents

Domino Applications & Mail Agents• In a wonderful world

Designers are using a ‘fake’ or non user mail account when the application or the agent sends a memo

Designers can use two users information

Return-path: in case of error (Field Principal or From)

User reply address: for a normal reply (field ReplyTo)

(optionally) define a fake mail address in agent security property field “Run on behalf of”

Designers have used NotesLog or Log classes and an agentLog database

Domino Applications & Mail Agents

• Activity logging

Insufficient, lacks of information on agent type and

Does not generate information for the following agents types:

o Web agent

o Manually run agent from the IBM Notes client

o Scheduled agents that run locally on a client

Domino Applications & Mail Agents

• ICAA application supported

• Keep the agents and applications on premise or Hybrid Verse with SoftLayer

• Give a distinguished email address to agents that send mail

With Designer

With correct fields (for MailSend or agents that creates document directly into mailboxes)

This way you can use Domino logs to distinguish between agents and end-users.

Domino Applications & Mail Agents• Conclusion

Ask the developpers

To give mail agents a mail account that can be tracked easily

To use From/Principal and ReplyTo fields correctly into the code

To user agentlog or any log facility when sending mail

o NOT in log.nsf nor Domino Server Console please!

Manage Domino scheduled agents and log the information

Use Case

• We want to take care of external users

External Users

External Users

• Concern is: external users to your company that connect to the

Domino server(s) and use Domino mail functionality

• How many external users use the Domino mail service internally?

• Domino Passthru

Activity Logging option

Other considerations

• Are my HTTP passwords strong enough to protect current and future

information?

Tips• Check HTTP password weakness

Use Domino HTTP stack

Use DIIOP (Domino Corba program)

Develop a JAVA agent with NotesFactory

Loop with Domino users

Use default passwords

Use a dictionary

Store results in an IBM Notes database

Metrics For The What Dilemma

Use Case

• How many mails are sent from my mobile users?

Smartphones vs IBM Notes Addicts

Smartphones vs IBM Notes Addicts

• Concern is: Get the Traveler users addicts

• Metrics:

Number of active users

Number of mails from the handhelds

Smartphones vs IBM Notes Addicts

• Traveler Users

Tell traveler users enabled

Tell traveler users disabled

• Deeper statistics

What is activated

Increase the Log level first

Smartphones vs IBM Notes Addicts• What is activated

Tell traveler log show

Tell traveler log handler

What is my level

Which handler can I check?

Handlers

Tell traveler log handler logNsf level finest

Tell traveler log handler audit level ALL

Smartphones vs IBM Notes Addicts• How to activate user info trace:

Rename the NTSLoggingTemplate.properties in NTSLogging.properties

Located in <DominoDataDir>/traveler/cfg

Define LogLevel=ALL

Define Privacy according to your needs, default is 0x001E (shows everything except BODY and Attachments) everything is 0x8000

OR Tell traveler log fields *

And without any parameter to stop -> “Tell Traveler log fields”

Restart task Traveler for activating or stopping the logging

Smartphones vs IBM Notes Addicts• info trace results are located:

<DominoDataDirectory>/IBM_TECHNICAL_SUPPORT/traveler/logs/xml

mmdd-hhmm-XXXXX_Sync_<CollectionId>_<SyncKey>_out_200.xml

mmdd-hhmm-XXXXX_SmartReply_<NOTEUNID>_in.xml

mmdd-hhmm-XXXXX_SmartForward_<NOTEUNID>_in.xml

mmdd-hh-mm_Ping_in.xml

mmdd-hh-mm_Ping_out_200.xml

Smartphones vs IBM Notes Addicts

Smartphones vs IBM Notes Addicts• The simpler the better

How to track Traveler users

The $Mailer field contains the “Traveler” string

Mail journaling:

define the correct rule!

o Internal users and all documents

o Beware of the size of the resulting database!

Do not encrypt the fields you do need for analysis

o $Mailer

Smartphones vs IBM Notes Addicts• Create a view in mail journaling template

From an existing view (Person)

Add a column for $Mailer field

Add totals

Smartphones vs IBM Notes Addicts• Conclusions

Number of mails sent from a device

Nothing out of the box with Traveler logs

o Need to analyze the log files in xml directory for IBM Lotus Traveler

Mail Journaling

o Need really few design enhancements

Need third party software

o Access to the database (Non-HA and HA)

o Live and simple information: audit4mail from move4ideas

Use Case

• We want to get rid of non enterprise data, mainly based on

attachments not on content.

Attachments

Attachments• Concern: an excellent starting point to start using IBM Files

Remove files from mail files to the repository they deserve

Company information has to be at the company level

• DAOS is not a concern for metrics

DAOS obfuscate file location (in or out NSF files)

Quota apply to logical size NOT the physical size

After migration, DAOS will need to be pruned

Attachments

• A basic and efficient approach for a global picture

Reset: set stat <stat name>

Show stat mail.* (or IBM Administrator interface)

TotalKBDelivered

TotalKBTransferred

TotalKBTransferred.SMTP

DeliveredSize.*

Attachments• Mail journaling

Define the rule for internal users only

Already available in IBM Notes mail journaling databases

Beware of mail journaling databases size

You can customize the view for more information

Extensions of attachments

Attachments

• Use Event task

Create an event generator for:

Message transferred to

Message delivered to

Attachments• (Mis)Use DAOSEST

Download version from IBM web site

Currently 1,7

Use –v option

Beware: on really huge architecture it can never end.

Solution DAOSEST_BUCKETS in notes.ini

o DAOSEST_BUCKETS=16,64,128,256,512,768,1024,2048,3087,4096

o Any value of 10k can be set

Use Case

• We want to know how many huge databases we do have and who

owns such databases.

Huge Databases

Huge Databases• Concern: not a target concern but the source concern

One mail file of 51k documents (31Gb)

16 hours to migrate to SmartCloud in the past (non DAOS)

• At which size do we consider a database is huge?

Time to replicate

Time to compact (even with option –#)

Views indexes size (Welcome Domino 9.2!)

Daosified or non daosified NSF databases

Huge Databases• The main problems are

Internal network bandwith

Slow disks (yes it still exists!)

Overloaded Domino servers

No or bad scheduled FUC tasks

o Fixup

o Updall

o Compact

The notesenv folder for the Onboarding Manager requires space

Huge Databases• Main problems (continued)

Calendar

No ‘Calendar cleanup’ execution on a daily/weekly basis

Orphan calendar entries with attachments in iNotes

o Cancelled reservations are kept within mail files in iNotes

Track attachments

Enterprise or personal

Track view indexes

Track databases size

Huge Databases• Create replica databases with cluster_admin_on

Faster than any other solution (DAOS or NOT)

cl copy srv!!database.nsf TransfertServer!!database.nsf REPLICA

• Archive or not Archive?

A real question

Archive Pro

Get rid of inefficient data

Archive Con

Aggregate them back

Huge Databases

• How to track the huge database

IBM Administrator: copy and paste the Files view into a spreadheet Software

Log.nsf: View Usage/By Size

Use Case

• We want to check for sensitive data and take a decision to migrate that

information or not.

Sensitive Data

Sensitive Data• Attachments vs content

Attachments extensions can help BUT can be obfuscated

Unions vs VIPs vs C-Levels

• Forwards of sensitive data

• Blind copy of sensitive data

• Encryption

Are my users do use encryption for sensitive data when mailing the information to internal or external users?

Sensitive Data

• If forwards matter

Use the Event task

Message forwarded to xxx

Message forwarded to xxx from yyy

• Mail Journaling

Activate the mail journaling function

Other Considerations

• I want more information on Traveler data than the current one I do

have

• What about user data

Archives

User mail rules

Number of contacts

Tips• Traveler (for the non HA server)

Just try: te traveler sql select * from sys.systables

• User mail file content (LotusScript or Java agent or …)

Loop through all mailfiles

Get number of archives and their names

Get number of mail rules

Get number of contacts (private or not)

Export information to a notes database

Export information to a CSV file

Metrics For The Where Dilemma

Use Case

• We want to migrate people from one region or ‘site’ together.

The Where Dilemma• Concern:

Think of collaboration teams

Think of really bad networks

New-Caledonia is an excellent example

• What to catch:

IP addresses from the sender

Cross check with relationships

IP Address• IP Address can help define people location

Log_Sessions=2 in Domino notes.ini

23/11/2015 10:01:09 Opened session for 'CN=Jérôme Deniau/O=Move4Ideas' (Release 9.0) SessId 520221FB Ip '109.8.206.35' 'Auth' 'C'

Inbound SMTP only: Mail Journaling (again)

Do not encrypt the “Received” field

Create a view that displays the “Received” field(s)

Analyze the field content

IP Address and Cross Checking• Cross check log sessions and mail statistics

Mail statistics: Transferred vs Delivered

TotalRouted: means number of recipients

Transferred: means number of physical message transferred

Delivered: means number of physical messages delivered

Use Event task

Architecture

• Use of Domino Server task

MAPS

Can help in discovering the mail routing between servers

No interest if only one Domino cluster manages all users

Metrics For The When Dilemma

Use Case

• We want to define the best time range to migrate a population.

When

When

• Concern:

Avoid user dissatisfaction

• Need to define the best time for switching users

• Communication is essential

• Metrics

Time when user is not sending mail

Problem with smartphone users

o Always on line

Europeans policy on mail during the week-end

When• Metrics can help but

Solution is more enterprise policy defined

Once population are defined

You can force the switch time and date

You can inform the end user for the correct procedures.

You can start replication and onboarding process during off-peak hours

• So what about metrics then

When

• Determine number of mails exchanged during your off-hours

Problematic with worldwide companies

Track internal sender and define the best time range for switching

Depends on replication

Depends on mail availability on the other side.

Use Domino mail journaling features and to an analyze on off hours/off-

days for internal senders

Session Conclusion

• Metrics are the starting point

• Metrics help you understand how people work within your company

• This session gives you the main axis to start

• Do not forget to compare the metrics from the source platform to the

targeted system after the migration.

Thank you for listening!

jerome.deniau@move4ideas.com