LavaCon 2011 Austin, TX

November 16, 2011

Why I left my CMSWhy I left my CMS… and how I did it!… and how I did it!

Mary Connor

Presenter...........................................................1

Summary...........................................................1

Why Agile? Waterfall was drowning us..............2

Which flavor of “Agile” did we pick?...................2

How do we collaborate?.....................................2

How does Agile affect documentation?..............3

People Reasons................................................3

Process Reasons...............................................3

Tool Reasons.....................................................4

Why Doc-To-Help?............................................4

Migrating from a CCMS.....................................5

Our Doc-To-Help Configuration.........................5

Automation Strategy..........................................6

Why not straight to wiki?....................................7

Presenter

Mary ConnorMary Connor (www.cleverhamster.com), a past president of STC Austin, has an M.A. in English and has been teaching and performing technical communication for over 20 years. Currently the Documentation Architect for Advanced Solutions International, she specializes in software documentation and training materials, single-sourcing and content management, usability and universal design, and information system design. She manages wiki content, is researching how to develop content and deliver training using social technologies, and is

cross-training in programming and automated testing within her Agile scrum team.Email mconnor@advsol.com

Why I left my CMS

Why I left my CMS...................................................................................................................................1

SummaryIn this session, you’ll find out why—and how—an experienced software documentation group walked away from a fabulously successful, stable, and well-automated CMS implementation. Two forces drove us off this cliff: our shop’s moves [1] to Agile development, and [2] to .NET-based web technology. The switch to Agile broke open our authoring silo and our department, and the switch to new web technology required documentation we couldn’t source in our CMS. Doc-To-Help made it possible to unify all of the new documentation with the legacy content in Agile-friendly ways, but first we had to extract that content. Getting into a CMS from well-styled content is easy, but find out what it takes to get your content out again for management in a new system.

Why Agile? Waterfall was drowning usASI’s switch to Agile two years ago was not about being fashionable, any more than when we virtualized our environment and embraced telecommuting: it was pure economics. We simply could not afford another software release such as we had just completed, since it was excessively long in development, consequently expensive, and — for all of that — indifferently received by our customers. The biggest risk we faced was not being bold enough in embracing change before the money ran out. To those of you still gazing over the cliff edge of Agile, I will say that the process went dizzyingly fast. A consultant was selected, he worked directly with management, and then all of Production shut down for a solid week of on-site Agile training by this consultant. We grouped into teams, squeezed into make-shift team rooms, selected our Scrum Masters, and were handed over to Product Owners, who started us working from a list of stories they had only just learned how to write. We were live!

Which flavor of “Agile” did we pick?Saying you’re doing “Agile” clarifies little because there is so much variation in how organizations choose to implement their version of it. Here are the Agile and SCRUM fundamentals we embraced and committed to: We belong to collocated teams, meaning the local telecommuters all had to

return to the office (we retained our remote developers and testers, but new hires are local).

Team rosters tend to include a product owner, a scrum master, three developers, one or two QA analysts, one business analyst, and a tech writer. (Over time, the scrum masters and tech writers became shared resources, and BAs moved to a UX team.)

We work in bullpens, where all team members have desks facing into the center, so that we see each other instead of walls. Most bullpens lack telephones.

We answer to product owners, rather than departmental managers, whose roles have largely been reduced to performance reviews and budgeting.

Why I left my CMS...................................................................................................................................2

We work in 2-week sprints, which we stagger by product lines: if those on product A start a sprint this week, then those on product B will start next week. We also start and end sprints mid-week to accommodate New Zealand developers being a day ahead of the teams in the U.S. (our Friday is their Saturday).

Each sprint starts with Planning from the Product Backlog, from which the Product Owner presents her prioritized stories. The team adds up its capacity, estimates the points (size) of each story, nails down the Acceptance Criteria, tasks out the work items, and commits to a set of stories, whose progress we track in TFS.

Each day of the sprint includes team standup (SCRUM) meetings plus a standup with all teams represented, to share status, raise concerns, and collaborate across teams.

Each team’s sprint ends with a webinar Review, where the team presents its finished work to actual customers and all stakeholders, for questions and feedback. A private Retrospective always follows, for the team to give kudos and record what worked, what didn’t, and what to try differently next time, so that the process is always improving.

How do we collaborate?Despite being largely collocated, we make extensive use of collaboration tools: Ventrilo and Skype, for VoIP voice conferencing Mitel, for on-demand desktop sharing and video conferencing Yahoo IM, for instant messaging and status broadcasts (our availability) Etherpad, Google Docs, and Google Wave, for group brainstorming and

live editing of sprint artifacts Campfire, for team discussion threads (to lessen email burden) Drupal, for cross-team blogs, wikis, and discussion forums BBFlashback, for creating informal demos, tutorials, and braindumps Cacoo, for collaborative creation of flowcharts and diagrams, which can

be hyperlinked linked into wiki pages and documentation source files

How does Agile affect documentation? We don’t lag a sprint behind, as some shops do; we must be shippable at

Review. We focus on continuous, automated builds, in support of being ready-to-

ship. Docs are part of Definition of Done; incomplete doc tasks block the entire

team. Integration and overall review of release documentation happens during

final regression sprints.

Why I left my CMS...................................................................................................................................3

People ReasonsChanging our documentation system was driven, ultimately, by personnel reasons that resulted from our switch to Agile:1. House Rule: Teams must be cross-functional generalists

Job titles and formal duties were dropped, so that we are all considered “Product Developers”, all responsible for designing, coding, testing, and documenting. While we all bring strengths and expertise, we are no longer dedicated role specialists. Underlying this is the belief that everyone can and should write, which is key to the participatory model of social media: “Here comes everyone!”

2. Info Dev lost half its writersOur Agile consultant warned us to expect to lose a third to a half of our talent due to unhappiness with the new method. Ironically, all of our writers loved Agile: far from being dinosaurs who could not adapt with the times, they were lured away to other Agile shops and opportunities, such as consulting (Anne Gentle). One writer left over the loss of his telecommuting privileges, going to a shop that was Agile but allowed remote work. So, losing half our resources, those remaining became bottlenecks unless we could open the doors to wider participation.

3. No new writer hires likelyA documentation impact that we did not see coming was our Agile practice of allowing teams to specify how vacancies are to be filled, to best help them reach Definition of Done. This means that a team can skip back-filling a writer in order to get another developer (which is more likely to increase team velocity). The new norm is to fill vacancies with junior programmers, who focus first on automated testing before being pulled into new code creation. The resulting negative impact on documentation volume and quality will have to be heard from customers by Product Owners, and that will take time to become clear.

Process ReasonsWhile process drivers from our development environment would likely not, alone, have forced our changes, they were substantial and complex:1. New documentation outside of AuthorIT

Very worrisome for content management, our newest products and technology were being documented entirely outside of the legacy AuthorIT system, out of necessity. Our new work centers on a Service-Oriented Architecture (SOA) product, which needs developer-written documentation (now done in Drupal) and an API reference (built manually using Sandcastle); our new user interface is done entirely in web parts, which each have to be documented in an HTML file that resides inside of the code package. We couldn’t find a way to unify this fractured source within our AuthorIT system. And not only is the source fractured: users are forced to look in multiple locations for information. Nor could we single-source this new content for reuse (such as training materials).

2. Continuous Integration builds

Why I left my CMS...................................................................................................................................4

A huge change in our development practice was re-engineering to support continuous integration: it’s no longer good enough to have a feature pass tests on a local build – the code check-in now triggers a “commit” build immediately so everyone can get email notifications if the change breaks the build or any other component. This change speeds up the testing cycle immensely, helping teams finish more stories per sprint. Our challenge was how to set up docs to participate in continuous integration as well, updating all deliverables as soon as source file changes are committed.

3. All source files managed in Visual StudioBeyond continuous integration, we saw that having 100% of source material — across all deliverables — managed, branched, and built inside of Visual Studio (Team Foundation Server) streamlines all processes immensely: one-and-only-one way to check out source, associate it with sprint backlog items, and track progress. Everyone in Production is licensed for TFS and can participate, with no change in process. Having documentation source on equal footing with code source would do more to elevate the importance and visibility of documentation than anything else we’d tried.

Tool ReasonsIssues with our current CMS were just as significant but proved difficult to gain traction with management, as none were immediately apparent as emergencies:1. AuthorIT licensing

We use AuthorIT on a terminal server, with floating licenses to support all writers working concurrently. It would cost thousands to buy extra licenses/support to cover concurrent work by all the teams, and the enterprise webserver version of AuthorIT costs in the six figures – quite out of our reach.

2. AuthorIT suitabilityWorse, the library structure we’d developed was too complex to just open to non-writers: the nested variables, embedded topic objects, template inheritance, object-oriented structure (such as hyperlink objects), and other features of the CCMS would be nearly impossible to simplify for casual users. Reimplementing our content library for safe and easy authoring would be a huge project in its own right.

3. Forced migration loomingAt the same time, we faced a migration within AuthorIT, from version 4 to version 5, a project outside the scope of our active sprint work. To get to version 5, we faced reimplementing our publishing templates and macros, and we had several blocking issues on our test migration database. Before too long, we could face having our production library on an unsupported version of the CMS, an unsupported version of Word, and an unsupported version of Windows Server. (This, in fact, was the situation we were in back when we converted to AuthorIT!) A good time to jump.

Why I left my CMS...................................................................................................................................5

Why Doc-To-Help?These are the features that made us proceed with a proof-of-concept with Doc-To-Help:1. Single-sourcing across build targets: compiled Help, HTML, manuals,

training2. Single-sourcing across source types: HTML and Word3. Merging generated API and developer conceptual docs into blended

deliverables4. Unifying documentation strategy across user and developer docs5. Support for team authoring in Team Foundation Server (and SharePoint

repositories)6. Command-line builds, for automation and continuous integration7. Familiar and handy editors for source files: Microsoft Office, Notepad++,

etc.8. Able to handle the scale of our builds (similar build time to AuthorIT v4)9. Affordability: new licenses for the cost of our AuthorIT maintenance

Pre-CMS: Merging documentation and trainingBefore we ever got our content into a CMS, we had to battle with strategies for reuse and explore topic-based authoring. While I was away working for other companies, this organization had grown, and the single-sourcing I had set up was dying under the tonnage of new sources. The training writer expanded on our procedures and wrote new ones when she couldn't find what she wanted in the docs. She used tables extensively as her way to keep screenshots and steps aligned; she didn't know how to use styles, but she applied local formatting masterfully. (She also pasted her content into PowerPoint to create the class slides.) These two information streams — documentation and training — were diverging significantly with every release, in content, style, and formatting standards. The president feared this trend, this worsening duplication of content and effort, so he decided it had to stop: he combined the two groups (we became "Information Development"), and he challenged us to bring it all together again.Desperately Seeking Reuse. Mandated to merge our documentation and training source materials, I began researching training content approaches with a fury. I studied instructional design and came across Reusable Learning Objects (RLOs), a standard about creating content units to be technology-neutral (any platform, any context, any display) and fully reusable. While I understood the importance of the technology neutrality for interoperability, I realized that it was the idea of working with units of reusability that could rescue me.Good Enough RLO. I focused on its core idea of defining the structure, the most practical unit of reusability. That is, what is the largest chunk of generic structure that could be recombined to create all needed doc and training deliverables? That RLO unit turned out to be something like a "section" that contained these content elements:

Objectives *        [ * training only ]

Why I left my CMS...................................................................................................................................6

Feature at-a-glance $      [ $ share with marketing ]Concepts +         [ + documentation only ]TasksReference +Hands-on exercises *

The symbols indicate how the elements are filtered by output type: elements marked with asterisks appear in training outputs only, and those marked with plus signs might appear in documentation only. The dollar sign marks content shared with marketing outputs, to keep them from copying and pasting content haphazardly to build the product catalog.So, this standardized structure, this “unit of learning content”, is what enabled us to maintain multi-purpose, single-sourced content in Word, using only styles and macros to generate the many different publishing outputs that we needed for documentation and training.

Just enough granularity: Score! This highly structured Word source proved very easy to migrate into and implement in our CMS, AuthorIT. It turned out that our “reusable learning object” translated perfectly into usage as reusable “sub-books” (books that are used within larger books that publish).All authoring occurred within these sub-books, and I arranged all sub-books into publishing books as needed. When consulting requested a new training course, for example, they would submit the syllabus of topics they wanted and I would build a new publishing book for them containing the sub-books they want in the order they want. If doing so brought in too much material, I’d sub-divide the sub-books into more granular topics to support how training needed to cover them. That way, we kept the granularity as course as possible to meet our needs, which minimized the complexity of maintenance and difficulty of finding sub-books and topics.

Why I left my CMS...................................................................................................................................7

Each sub-book embodies our metadata, through its naming convention and its underlying object template: Audience License level Purpose (e.g., marketing) Target (e.g., manual only) Subject Release state

You heard me: Don‘t reuse topics! And so this is where the heresy entered our implementation: Migrating to the CMS, I effectively outlawed “topic-based authoring” in favor of “sub-books” (RLOs). Rule 1: Reuse books across the library, but avoid reusing topics across

books Rule 2: If you must, flag those reused topics (asterisk in topic description)

to warn others

Why I left my CMS...................................................................................................................................8

So, writers were able to author each topic exclusively within sub-books, each of which is filled with a hierarchical structure of topics that we could count on staying stable across all publications. That vastly simplified the problem of knowing how to write, because you know what type of user is being targeted in the sub-book (an administrator, for example), you know her licensing level and screen options, you know what you know what level of terminology to use and what level of existing knowledge you can reasonably expect her to have. You can know that certain procedures and explanation will be included with your topic, so you can link to them knowing that they will not fall out of scope in any output.

Migrating from a CCMSWe had no roadmap for this migration project, as I’ve yet to meet anyone who worked to take content out of a component content management system, having gotten it there! And just as with moving houses, the longer you’ve lived there, the harder it is to pack it all up. Here are some of the tasks it required, to get back “upstream” of the CCMS:1. Unraveled the reuse of objects so that they each occur only once in the export

file setThis is not as horrifying and counter-productive as it sounds, for much of the reuse would simply be reimplemented in the new system; the trick is to bring over each bit of content only once. Reuse spanned the entire library, of course: books within books, topics within topics, graphics across topics. What saved us was the discipline of reserving most reuse to books within books (modules) and rewriting content to support that.

2. Exposed all hidden notes/references/commentary/history (designed to never publish) so that it’s included in the export files and not lost

3. Optimized the object templates (such as link formatting and style assignments) for how D2H interprets files on import

4. Wrote Word macros to achieve reproducible post-processing: resizing graphics, substituting styles, converting to Word XML (DOCX), etc.

5. Created export projects and batch files to automate the migration so we could iterate This proved to be a crucial milestone, because speed of conversion R&D increased dramatically once it was possible to kick off a new export/import/build to test each new theory of how to solve a problem or transform a content feature.

Our Doc-To-Help ConfigurationHere are key features of our implementation:1. Doc-To-Help (D2H) is installed on test and virtual build machines

Both machines run Windows 7 and Visual Studio. As these are the only machines running the application, we own only two D2H licenses so far.

2. Source files are editable on any machine behind the firewall that runs TFS

Why I left my CMS...................................................................................................................................9

The workaround that makes this possible is that Word source files are on a non-D2H template on the network, so authors cannot access the D2H special toolbar functionality (for linking, inserting, creating pop-ups, using variables, etc.). Instead, we have authors apply just a few special styles manually, such as “CIH Jump” to tag instances of literal topic title references that will become hyperlinks on publication.

3. The Doc-To-Help project resides inside the TFS working folder structure, so that its source files are the same as the working files that are refreshed (Get Latest Version).

4. A single project (Access database) generates all of our outputs, across productsWe have a growing body of content that is shared between our old and new flagship products, so keeping all of the content in a single project lets us reuse it most easily. We defined a separate “Target” for each deliverable, which controls the publication settings and filters the content conditionally by Attribute using an Expression (example: include Beta AND Commerce content only).

5. Each product has a pair of build targets, one for HTML and one for Word/PDF.

Automation StrategyWe developed a set of batch files not only to automate scheduled publishing but also to tie that publishing into a larger process:1. Downloading the latest source files from TFS and building from that

working folder2. Capturing builds to date-stamped log files (for troubleshooting)3. Completing whatever post-processing and renaming we need

For example, to provide editable/printable Word versions of the live website content, we build those separately and then merge them into the Helpsite build area, so the download links can resolve (as relative URLs) when deployed.

4. Copying filesets to archive and deploy our content Because of the huge numbers and size of files involved, we use a command-line compression utility (PACL) to zip and unzip the Helpsites. To improve performance, we use remote desktop session files to launch batch files to do the unzipping locally on the webserver, rather than over the network.

5. Set up Word “containers” in the project to dynamically merge standalone HTML files into the Helpsite

6. Separated out copies of the target builds to be the ones published internally via IIS, and only update them when the build is good. (Prevents loss of service.)

We also had our IT department set up alerts on the URLs of our website builds, to notify us if the build failed overnight (which results in the error 404: Page Not Found).

Why I left my CMS.................................................................................................................................10

Why not straight to wiki?Being avid followers of the sea change of social media, our first instinct was to turn to a wiki solution to support real-time, team-wide, distributed authoring. We researched the field, narrowed it to the three most promising candidates, but found no solutions among them.1. Confluence

Being the one wiki designed expressly for technical documentation, we had to consider Confluence. However, its licensing costs and its relative complexity were real barriers. It also conflicted with our technology: ours is a Windows (ASPX) product, yet Confluence hails from the Java world and does not work well in Internet Explorer, which is our primary browser. I’d also heard from an adopter that Confluence would choke on the size of our builds to hardcopy formats, as it was only designed to export small subsets of wiki content.

2. MediawikiBeing the open-source wiki powering Wikipedia, Mediawiki made it onto our short list. While many aspects of it proved friendlier and attractive, we soon ruled it out because we couldn’t publish out of the wiki format into our other needed outputs, and (more importantly) we could not secure content and access with anything like the granularity we’d need for production work.

3. Drupal

Why I left my CMS.................................................................................................................................11

Drupal was a finalist not only because of its immense capability (as optional modules) but also because we were long-time users: we’d used Drupal for our product’s community site for years, offering blogging and forums as well as wikis of reference information. We could secure it, extend it, completely customize its theme, and probably publish (export) outputs from it, even use complex taxonomies to create a myriad of dynamic content experiences. However, we were on version 4 and would need to migrate to 6 or 7, which would be a serious project. Worse, we knew it’d take significant expertise to program Drupal for our complex documentation requirements — expertise we lacked in-house and probably couldn’t afford to contract for.

Why I left my CMS.................................................................................................................................12