Read the announcement. Check out the geeky details.

An unavoidable part of moving to the Darwin Information Typing Architecture (DITA), or any other structured authoring system, is converting your existing content into the new format. Most organizations that make the move to structured writing have to make the change while still continuing to meet their regular delivery schedules. This means you need to convert content and get it up and running correctly in the new system between two product cycles, and usually without much in the way of added staff or resources.

Content conversion is a key part of your migration strategy and the quality and completeness of that conversion is essential to getting the migration done within the constraints of your schedule. However, content conversion is a bit of a black box for many people. It is hard for writers and managers to anticipate how difficult the conversion process is going to be, how long it is going to take, how much it is going to cost, and how much cleanup of the output is going to be required after the conversion is done. The purpose of this article is to lift the lid off that black box. In particular, understanding how automated conversion works will help you form a more reasonable expectation about how your own conversion project is going to go, and what you can do to make it go more smoothly.

No automated conversion is ever 100% clean, but the difference between an 80% clean conversion and one that is 95% clean is huge – it means a fourfold difference in cleanup costs. What makes the difference between 80% clean and 95% clean? Between 95% clean and 98% clean? Such outcomes certainly depend upon how well managed the conversion process is. However, having the right approach to content conversion is of critical importance.

Knowledge is the key to intelligent content conversion

There are three essential mechanisms that content conversion technology may leverage. They are:

• patterns
• context
• guided conversion

When encoding content in a semantically rich format such as DITA, it is important to understand the meaning of the content in order to apply the correct tags. While people can understand the full meaning of the text they are reading, a computer does not, at least not very deeply. What a computer is exceedingly good at is recognizing patterns in the content. But patterns don’t provide the full solution. Patterns, when found in a given context, carry much more insight as to the meaning of the text in question. A sequence of 5 digits, for example, may represent a zip code, in the context of a US postal address, or an ICD-9 diagnostic code, in the health care sector. Guided conversion is supported by the provision of high-level mapping rules that hint at the current context so that patterns are interpreted correctly by the automated conversion tool. Compiling these hints depends on having an intimate familiarity with the document set destined for conversion. It is the content owners, armed with this content knowledge, who are best positioned to specify the mapping rules.

Patterns

Patterns are everywhere in content. Patterns occur both in the content itself, and in the file format that contains the content. The foundation of all content conversion tools is the ability to recognize patterns.People also use patterns to recognize things in content.

For instance, a reader will immediately recognize what these numbers mean based on their pattern:

• 9/30/12
• +1 (613) 745-4242
• $65.12

Software can recognize these patterns as well, so if your target format requires semantic markup for date, telephone numbers, or monetary amounts, a simple-pattern matching algorithm can find them and supply the markup. For example, a conversion program could recognize the sequence:

• “+” numbers space “(“ number*3 “)” space number*3 “-” number*4

It can then capture each number sequence in this pattern and write it out using whatever XML format you choose for phone numbers, for instance:

• <phone-number country=”1” area=”613” exchange=”745” number=”4242”/>

Of course, recognizing phone numbers is a bit more complicated than this. For one thing, people do not always include the country code when they write a phone number. People often omit the parentheses and the dash from the number, especially when the country code is used. This is one place where local knowledge of your content comes in – if you have a corporate style for phone numbers, you can tell your conversion software exactly what to look for. Otherwise, the conversion program can use multiple patterns to detect phone numbers in different formats.

Also, this pattern only works for North American phone numbers. Many other countries write their phone numbers differently. This is a case where we can use a context clue to improve our detection of phone numbers. For instance we can use the country code to determine which pattern to expect. The following pattern detects a UK phone number:

• “+44” numbers-and-spaces

UK phone numbers use a different format from North America, so our original pattern will not detect them correctly. A conversion program can detect phone numbers as a two-step process. First you detect the country code to determine which country the number belongs to, then you select a pattern appropriate to the chosen country to fully analyse the number.

You can expect support for matching common patterns, such as phone numbers, to be built in to conversion software. However, it should be easy to extend the system with new patterns specific to the vocabulary of a particular domain.

Context

Patterns, though they are an indispensable part of automated conversion, cannot on their own address the challenge of imparting to the content the depth of meaning, or understanding, required for the intelligent application of semantic markup. This is where context comes in.

For example, consider a list in an Adobe FrameMaker document. In FrameMaker, while a table is a distinct type of object, a list is not. In FrameMaker, you create a list simply by adding a bullet or number style to a set of paragraphs. The result is something that looks like a list in the output. However the FrameMaker file format does not record the fact that the content is a list. The human eye can see the list in the output, but it is a little more challenging for a conversion program to figure out where a list begins and ends and what belongs to each item in a list.

Why does the conversion program have to figure out where the list begins and ends? Because most XML formats treat lists as distinct objects. When an XML document is styled, the style is generally applied to the list as a whole, rather than to the individual paragraphs in the list. This is usually the only way that an XML-based system provides for styling lists, so if the conversion software does not recognize the list in the source and create a proper XML list element in the output, chances are that the list will not be styled properly in the final output.

Example: a nested list

Quick-drop cookies

1. Prepare the dough.
   a. Beat the egg in a large bowl.
   b. Add flour.
   c. Stir in milk.
2. Prepare the topping.
   a. Mix brown sugar and cinnamon in another bowl.
3. Form 1-inch round balls of dough.
   It is helpful to use a spoon when forming these balls.
4. Roll each ball in the topping.
5. Place each ball on an ungreased cookie sheet.

Bake at 425 °F for 12 to 15 minutes.

This is the kind of construct that often occurs in complex procedures in technical documentation, the conversion program has to deal with multiple paragraphs within a single list item, as well as nested lists.

In this example, a paragraph that begins with a numeral indicates a first level list item, while a paragraph beginning with a letter indicates a nested, second level list item. An automated conversion should leverage this pattern to determine the logical nesting level of each item. Alternatively, it should identify nesting level by the indentation or styles that were used. Regardless, the conversion needs to track the current nesting level in order to ensure that the lists are properly opened and closed, and that each list item belongs to the correct list. For our example, this means emitting an opening <ol> each time we transition from an outer list item to a more deeply nested list item, and emitting a closing </ol>  when transitioning in the other direction. The correct output is:

<p>Quick-drop cookies</p>
 <ol>

<li>Prepare the dough.</li>

 <ol>

<li>Beat the egg in a large bowl.</li>

<li>Add flour.</li>

<li>Stir in milk.</li>

</ol>

<li>Prepare the topping.</li>

<ol>

<li>Mix brown sugar and cinnamon in another bowl.</li>

</ol>

<li><p>Form 1-inch round balls of dough.</p>

<p>It is helpful to use a spoon when forming these balls.</p>

</li>

<li>Use a spoon to make 1-inch round balls of dough.</li>

<li>Roll each ball in the topping.</li>

<li>Place each ball on an ungreased cookie sheet.</li>

 </ol>

<p>Bake at 425 degrees Fahrenheit for 12 to 15 minutes.</p>

Note that the list markers (1., 2., a., etc.) have been removed by the conversion.

Guided Conversion

So, how can we establish the appropriate context of a given piece of content? The most reliable authority on this is the content owner who is familiar with the content.  A mechanism is required which enables the content owner to easily express what the correct context is for any document content. This must be a high-level interface that does not require the user to be a programmer or technical expert.

Example: task steps

Upon further reflection, the markup provided by the previous example is not ideal. An improved DITA markup of these instructions for preparing the quick-drop cookies would use steps within a task topic. But, to target a semantically rich content model such as a DITA task, a conversion tool requires guidance. Such guidance may be provided by means of annotations attached to portions of the content, as illustrated in the table below.

The task title annotation can be based on the formatting properties of bold and underline. The annotation of step level 1 or 2 can be based on the presence of the list markers or the indentation level of the text. The tip might be recognized by the paragraph styling. The conversion should be smart enough to try to fit the last sentence into a task in a way that makes sense, in a way that is permitted by the DITA task content model. The elements <result>, <example> and <postreq> are good candidates. A preference can be set for the documentation set, and in this case <postreq> is the best choice.

Guided by these annotations, the conversion software should produce the following output:

<task>

<title>Quick-drop cookies</title>
<taskbody>

<steps>

<step>

<cmd>Prepare the dough.</cmd>

<substeps>

<substep><cmd>Beat the egg in a large bowl.</cmd>

</substep>

<substep><cmd>Add flour.</cmd></substep>

<substep><cmd>Stir in milk.</cmd></substep>

</substeps>

</step>

<step>

<cmd>Prepare the topping.</cmd>

<substeps>

<substep>

<cmd>Mix brown sugar and cinnamon in another bowl.

</cmd>

</substep>

</substeps>

</step>

<step>

<cmd>Form 1-inch round balls of dough.</cmd>

<info><note type="tip">It is helpful to use a spoon when forming these balls.</note></info>

</step>

<step><cmd>Roll each ball in the topping.</cmd></step>

<step><cmd>Place each ball on an ungreased cookie sheet.</cmd>

</step>

</steps>

<postreq>Bake at 425 degrees Fahrenheit for 12 to 15 minutes.

</postreq>

</taskbody>

</task>

Typical problems to look out for 

Here are some examples of the types of conversion issues that cause problems for conversion solutions that do not make full and integrated use of patterns, context, and guided conversion.

Multiple sets of steps within a task topic

A DITA task topic must contain only one procedure. However, many existing user guides are not written that way, and may have more than one procedure in a section. If you are converting sections into topics, and a section has more than one procedure, the conversion software needs to do something to produce valid output that includes both procedures.

Some control of context is required even to recognize that this problem exists. A conversion that depended solely on pattern matching would not even notice that it was creating an illegal second procedure. For a conversion tool to avoid this error, it has to be aware of the context of the procedure, not only in the input it is reading, but in the output it is creating.

Though the content cannot be automatically re-authored, the conversion software can insert an empty task <title> based on context, effectively breaking the topic into two tasks. This allows the conversion software to apply the semantically correct <step> and <cmd> markup to the content of the second procedure. The user still needs to provide the proper text for the title of the second procedure, post-conversion, but this is much quicker and easier, and less error-prone, than re-authoring the topic, either in the input or in DITA.

Procedures authored as a table

A number of organizations use tables to lay out the steps of a procedure. For a generic conversion program, this structure is going to look like a table, not a procedure, and the result will be that the content will come out as a table rather than a task in the DITA XML, which is not what you want.

Guided conversion can identify such tables based on, for example, the content of the first column (Step 1, Step 2 etc) or the header row, or possibly the table style.  The identified tables can be stripped of their table markup, and their contents automatically mapped into step commands, info, examples etc.  Again, the paragraphs can be identified based on the fact that they were contained in such a table, so there is no need to rely on styles.

Tables that contain definition lists, advisories, or any other content, can be similarly identified and stripped of their table markup.

Conditional text

Some conversion tools have trouble working with files that contain conditional text. Sometimes the tool requires that all conditions be turned on before conversion, and then they lose the conditions in the output.

Guided conversion should be used to specify a rule which indicates how different conditions in the source content map to XML. The conversion rule can target the DITA otherprops attribute, or a specialization of the props filtering attribute, for the capture of the conditional information. A guided conversion rule could also cause conditions of a specified type to lead to the creation of entries in the relationship table of the DITA map.

Constructing book and map files

While the aim of a conversion to DITA is to be able to reuse topics in many places, the first place you are probably going to want to use your converted topics is in the same book they came from. That means you will need a ditamap and/or bookmap that reproduces the structure of the converted book. Your conversion tool should be able to produce the required ditamap and bookmap for you.

Discerning the hierarchy is not always as simple as matching heading levels. Not every heading marks a change in hierarchy, and authors do not always use headings in strict hierarchical sequence. Additionally, different topic divisions may be indicated by the use of different heading types. Managing all of these issues requires sophisticated management of context informed by a detailed knowledge of the content and the style conventions that were used to create it.

Another important issue is discovering the book information such as publication date, document number, etc. For some organizations, this may involve the creation of a customized bookmap, if the standard DITA bookmap does not capture all of the publication information the organization uses.

This information in not always easy to find in the source files. No generic conversion software can ever accurately detect, extract, and preserve this publication information, since its format and location is always specific to an individual organization. However, with guided conversion pinpointing the location, pattern, and context of this information, a conversion tool can build the correct map.

In some cases, important metadata is found in the headers and footers rather than the main text flow of the document. Once again, guided conversion can pinpoint the data of interest and relate it correctly to ditamap and bookmap files you are building.

Choosing your conversion strategy

Knowledge, as defined by Joe Gollner:

“Knowledge is the meaningful organization of information, expressing an evolving understanding of a subject and establishing a basis for judgment and the potential for action.”

The level of success that an automated conversion technology can hope to achieve is bounded by the depth of knowledge it can attain of the content to be converted. Context, supported by guided conversion, provides for the meaningful organization of the information revealed by patterns. The conversion software can act on this evolved understanding of your content to produce the richest XML possible. Knowledge is the key to intelligent content conversion.

Because intimate familiarity with the content is so important to specifying the patterns and the context that will produce a high quality conversion that requires little cleanup, you probably don’t want to simply send your files away to be converted. Without your specialist knowledge to supply the patterns and context clues, the conversion you get back is going to be pretty generic, and that is going to mean you will have to do a lot of manual cleanup before the content is really usable.

On the other hand, the people with this knowledge are writers and editors in your organization, and they generally don’t know how to express these kinds of context clues in a programming language. Trying to learn to do conversion programming, so that you can write your own conversions that exploit your knowledge of the content, is going to be even more time consuming than cleaning up all the problems left by a generic conversion.

To get the best of both worlds, you need to work with a conversion service provider who understands the importance of patterns, context, and knowledge of the content in the conversion process, and who will work with you to define the conversion rules that will greatly improve the quality of your conversion output, and thus save you weeks or months of cleanup effort. You need a conversion service provider that possesses the intelligent conversion tools that allow you to capture and express all the context recognition rules in a high-level human-readable way, without the need for programming or technical expertise.

Here’s the problem. Marketing and technical communication teams are overloaded.

Faced with:

• Multiple delivery formats (print, web, online help, and now, myriad mobile formats)
• Increasing demand for translation
• Increasing demand for personalization and localization of content
• Shortening product release cycles with more product variants means more duplicate content need to be kept up-to-date
• Completely new channels and content sources to tackle  (social media, syndication, user-generated content)

It seems we’re the camels, and the market has no shortage of straw to pile on our backs.  The solution can’t be “work harder”, because increases in communication budgets have not been proportionate to increasing demand on us. Because we can’t allocate additional resources to tackle these challenges, we must work smarter.

This means leveraging automation to create all these personalised deliverables with as little manual effort as possible. We just can’t keep up using outdated methods where we convert from format to format. Our users don’t want content on a website, or a microsite, or a portal, or PDF, or a mobile app – they want it on them all.

By dynamically assembling content on request, and automatically publishing it out in the format of the user’s preference, we can focus our precious human resource on the challenges that need humans, like information design, architecture and of course writing. For the grunt work of content assembly, layout, formatting, and publishing, use can leverage cheap, dynamic, automatic processes. This doesn’t just require new software or technology, but a new process and attitudes towards publishing so that it can be dynamically processed while still maintaining quality of output.

Going outside the comfort zone of some readers, I am going to illustrate some simple lessons that can be applied across various verticals and industries by using an ‘extreme case’ of advanced technical communications as an example. For nearly two decades, this field has been pioneering dynamic methodologies to address diverse customers faster and across various formats.

Why Should We Look at TechComm

We usually think of technical communication as ‘the manuals’ (which no one wants to read), but it is so much more. Smart techcomm focuses on facilitating the customer’s experience of the product or service that the published content supports. Techcomm publishers design personas, key messages and establish content strategies and workflows like any other type of publishing. Also like all publishers techcomm leverages social media and dynamic systems to give users what they want, when and where they want it.

What Customers Want

What customers want is the knowledge that is trapped in the heads (and on the hard drives) of product experts like engineers, technical communicators, trainers, and especially, their peers. Websites, reviews, manuals — even ‘content’ itself — are all a means to an end. Your customers want to be able to do something with your products. They may want to buy, evaluate, install, clean, use, repair, or decommission them. To do what they desire, they need you to  transfer the knowledge you have locked inside your walled gardens and give it to them — quickly, so they can get back to whatever it was they were trying to do before they needed your help.

The social media explosion of the past decade – which shows no signs of ebbing – has shown us that communities can produce a lot of content.  The questions remain: can we get them to produce and distribute usable, useful product knowledge, and if so, how can we leverage that knowledge?

* Or government body, or association.

What Can The Crowd Do For You?

They can write for you

Usability and web specialist Jakob Nielsen has detailed the fact that the vast minority of consumers will never become content producers.  However, when dealing with large numbers of users with large numbers of demands, the contributions they make do not need to be substantial, they need only to be enough.  And users know it.

“Official” product content — that content created or published by the enterprise — sometimes has a reputation for being incomplete, difficult to navigate or otherwise unhelpful. In all my research and field experience, I’ve found most consumers don’t care who creates the content, as long as it helps them accomplish their goal.  As we’ll see below, the nature of the user contributions is also different, making them disproportionately valuable.

The lesson: Make it easier for users to extend and add value to your content.  When publishing, your platform needs to allow users to easily contribute content. That content will also need to be structured and wrapped in as much metadata as possible. Your platform will need to help you sort and curate user contributions to find both good contributions and top contributors.

You find these by filtering the content based on metadata (a taxonomy).

Users will build your taxonomy (and therefore links, filters and navigation) for you

A taxonomy is a labeling and categorization system.  It defines data about your content (metadata), and lets you filter, search and relate content.

Users will tag their own content if properly encouraged (again, see Nielsen’s advice on making things easy to encourage participation), building up a ‘folksonomy’ (a taxonomy built by ‘the folks’ that use the content).

Often we think we must choose between a taxonomy or folksonomy.  The truth is that the two play very well together.  Look at Amazon:

As we know, Amazon’s taxonomy of things like ‘Home electronics’ vs. ‘Books’ vs. ‘Home, Garden & Tools’ is an integral and vital way of navigating a vast amount of content.  However, they know that their customers are always right, and that no classification and navigation system they provide will ever (on its own) equal what they can do if they leverage the power of the crowd. Amazon lets users add their own tags. And, they allow their users to search and navigate the site using those tags, in combination with the classification system the company provides.

Leveraging user-generated tags is necessary because it helps solve several problems. First, tags created by users provide a wealth of information about our content. In order to build content classification systems of value, we must know what terms our customers use to describe our products, services and content. Second, because those who work for us creating official content are often resistant to adding metadata tags, leveraging the crowd to build folksonomies can help us provide the right content to the right people at the right time and in the right language. Search engines need this additional information to help customers find what they are looking for and dynamic publishing engines require tags to deliver content to those who need it, when and where they need it.

The lesson: Make it possible for your users to build a folksonomy in all your online channels. Curate those terms and glean business-critical information from them. This means enabling user tagging on their own content — and on yours — and having clear guidelines indicating what user-generated terms may ‘graduate’ from the folksonomy into the official taxonomy.

This will enable analytics and reporting, which allow you to monitor what’s happening to and around your content.  With this metadata in your arsenal, you will be able to sort by products and subjects, locate top contributing users and top rated content.

Users can know things you can’t 

User-generated content is created by users specifically for what users want and/or need.  Technical communicators try to do this as well, but users are in the privileged position of having the enterprise’s published content as a starting point.  They also have real-world field experience with your products before they start writing, an opportunity many enterprise staff will never have.

For general publishing, often we find that when content ‘goes live’ we discover that it was not properly designed for users’ needs. Similarly, in techcomm, it is only when a user first puts the product and all its supporting content into use that the gaps are found.

In a recent content strategy audit, we looked at product content generated outside of the organization, specifically, at common user search queries coming in through the website, and questions and answers posted to forums. We found that the users were getting lots of facts, but not the task-based and conceptual overview information they were looking for.

By often being derived from engineering specifications and product management documents, technical communication content is often excessively reference-based, giving people more of a ‘dictionary of product data’, rather than true ‘user guides’ or ‘how to manuals’.

The lesson:  When dynamic methods are not in place, staff get overloaded just trying to hit deadlines and prepare reviewed, nicely formatted deliverables.

All publishers need to plan to learn from users, and promote their work.  Time is money. Any financial adviser will tell you to always put away 10% at the beginning of the month, no matter how little you have left.  You’ll find a way to make do with the 90%, and be far better off in the long term.  Plan a percentage of time at the beginning of every project to research the community’s needs by watching their output.

You can — and should — have a process in place for finding, capturing, validating, and reworking users’ knowledge until it fits alongside ‘official’ content.

Pulling it all together

By enabling user creation and tagging of content, and building analysis and curation of user content into your process, you will be able to leverage more content than you could ever have the capacity to produce yourself.

XML (oftentimes of the DITA flavor) will be required to make all of this a reality. XML authoring environments make it possible to create semantically-enabled content that can go anywhere, and take its metadata with it. As such, the platform you choose will likely be an XML-based system and require a rethink of information design and editorial processes to enable dynamic publishing of the end deliverables.

By leveraging your folksonomy and taxonomy, you can dynamically create related links from your content to user-generated content and back again. To illustrate, think about how when Reuter’s sells news on “Middle East oil fields” to a 3^rd party news site, that 3^rd party site can automatically link that content to its other “Middle East” and “Oil” content not sourced from Reuters – simply by matching up the metadata tags. It’s a simple feature of dynamic systems, but highly impactful.

If the users have knowledge and the will to share it, saving us time and making them happier in the process, then it seems only fitting that we help them do so.

In my previous column, Understanding Content Conversion: Unfortunately, There’s No ‘Easy’ Button, I examined the various areas that trip up the eBook conversion process –special characters, tables, hyphenations, and so on.  This column follows on that theme, based on a question from a consultant with a major consulting firm:

“…I am researching the time it takes on average to convert from PDF to EPUB, including the time it takes to edit the ‘rough’ EPUB file created by the automated conversion to clean up the formatting errors, resulting in a ‘clean’ EPUB suitable for display on an eReader device. I am having trouble locating such a statistic.”

Why is it so difficult to find this information, you ask?  That’s easy. Because there isn’t one specific answer. The time it takes to correct an automated conversion (cleanup errors) depends on many factors:

1. complexity of content in the source document
2. type of source file
3. completeness of the automated converter
4. post-conversion, manual cleanup
5. skill level of the people doing the work
6. how clean do you want it

Complexity

Most novels are easier to clean up than multi-column textbooks or complex technical content containing formatted tables, images, poetry, sidebars, and footnotes. Complexity increases as the number of complex elements in need of being converted are reconfigured to fit the often much smaller areas of screen real estate common on smartphones, tablets and eBook readers.  Even novels are not that simple to convert, as discussed in Devorah Bloom’s webinar, What to Expect from Automated Conversion to eBook, but they are usually quicker to do than complex scientific articles with math, chemistry, and all that stuff.

Source file

The format of your source content will impact how long it will take to prepare your materials for conversion. It’s important to carefully consider all sources available. It may be obvious that converting from paper will be the most costly, as proofreading will be necessary to make sure everything is correct, but even with electronic files, problems can be expected. PDFs, for instance, are the most common type of source file. They come in many variations, and while some are much better sources than others, common problems introduced in PDF files include:

• word spacing
• paragraph delineation
• hyphens
• emphasis
• special characters.

Word processing files, proprietary software formats, standard file types (XML, HTML, etc), and pretty much every other type of file, will also introduce challenges.

Conversion software

Results from automated conversion software (and automated conversion scripts) vary widely, but in most cases if the software you opt to use isn’t tuned to your content, the conversion will be rough. Rough conversions require serious clean-up. It’s almost always best to invest the effort up front to tune the software to the content being converted.

Sometimes, tuning isn’t enough. Some commercial content conversion software (this includes freeware) is overly strict, lacking in the flexibility department. While conversion rules are necessary, you’ll want to use tools that provide the flexibility needed to handle all the situations you’re likely to encounter when converting source content that isn’t as clean as you’d like.

While writers don’t maliciously introduce problems into the documents they create, they are the source of many conversion challenges. It’s not their fault, actually. They lack an understanding of how their actions create conversion problems and have never been equipped with the knowledge (or the tools) needed to produce easy-to-convert documents. But, knowing this fact will make it easier for you to select the approach that works best for your organization.

One approach is to take whatever content you have been provided and work to clean it up, manually.

Another approach is to use commercial conversion software to make a first pass. When problems crop up (and they will), go back and modify the original documents so that they fit the software’s expectations. While this approach is workable, it’s time-consuming and expensive.

A third approach (that is particularly useful on large document sets) is to work with a firm that specializes in content conversion. Look for a company that has developed conversion software which is designed to be continually adjusted (tuned) to meet new needs. This approach will allow you to continually leverage the power of the conversion software to do as much of the work as possible. At DCL, we use this approach and we do so because every tiny accuracy improvement we make pays tremendous dividends in the clean-up phase.

This is where the big variances enter the equation. If the conversion process worked effectively, this phase would just a review phase and would go very quickly. However, if the conversion is rough, having left behind a lot of debris, it takes longer since you have to find and fix things, and some of those things are difficult and time-consuming to fix by hand. If you’re still intent on doing all this yourself, you should test the results of the conversion on a small, representative sample of your content to better understand what’s involved.

After cleanup, everything has to be reviewed. It’s a necessary step that far too many people skip, leading to content quality problems. Additional, device-specific review and testing will need to be conducted if you’re outputting your content to multiple device types. This step is not intended to clean up errors, but rather to ensure everything worked well. It’s an important task, not to be relegated to clerical staff. Instead, it’s best to be conducted by folks who understand the content, the audience , and the devices on which the content will be displayed.

Review is more than just comparing the original copy to the final result. Because changes are introduced in the eBook versions that were not part of the source file (text is repositioned to support device screen size and orientation, for example), it’s important that reviewers are equipped with the knowledge necessary to know what to check for and what to ignore.

How clean do you want it?

In the traditional book publishing world, perfection was the standard, but that seems to have changed with the rush to get eBooks to market – especially with short run books that need to get out quickly. While a medical text requires checking, double-checking, and triple-checking, other kinds of books might be acceptable with the occasional extraneous hyphen and bullets that don’t wrap exactly right. I’m a little old-fashioned on this, and prefer the perfection approach, but I do recognize that there are short-cuts that some may feel comfortable taking.

Conclusion

So the short answer to the question of how long it should take to produce clean eBook content is based on a number of factors. Each of these variables contributes to the total amount of time you’ll need to spend on correcting an automated PDF to EPUB conversion; it may be 3-4 hours, but it can take also 3-4 days — or longer. It all depends…

We all know that XML is a good thing. The pundits and the vendors all tell us so. XML opens up many possibilities for content. But exactly how (and why) it does so is not always made clear. Getting your content into XML is not, by itself, enough to deliver any or all of the things that are promised for it. To understand why, we need to look at what XML is really about.

I’m going to assume your know the basic mechanics of XML and that you recognize elements, attributes and the tags that define them. I’m assuming therefore that you recognize that the sample below, which is an excerpt from an XHTML document, contains a <p> element with two <i> elements inside it, and that these elements are defined using tags, which are the things inside the angle brackets:

<p><i>War and Peace</i> is a <i>very</i> long book.</p>

These tags are metadata. When people talk about metadata, they often think of it only as a label attached to a piece of content, usually by a content management system, and used to track the document and make it easier to find. That is certainly one important use of metadata, but metadata is much broader than that. Metadata means data that describes other data. There are many useful forms of metadata, including XML tags. (For more on this, see The Meaning of Metadata).

The reason that XML is so useful is that it allows us to assign metadata to content at any level of granularity, not just to a document as a whole, but to an individual sentence or an individual word or phrase. In the sample above, the <i> tag tells us that the string “War and Peace” should be rendered in italics. (This is what <i> means in XHTML. An <i> tag could mean something completely different in another tagging language.)

Of course, “render this in italics” is not the most sophisticated piece of metadata in the world. Also, it seems to violate the oft-cited rule that XML separates content from formatting. Actually, that oft-cited rule is wrong. Some XML tagging languages, such as XHTML and XSL-FO are designed specifically to apply formatting to content. Some XML languages are designed to separate content from formatting. Others have nothing to do with content at all. A much better rule would be to say that XML allows you to apply metadata to content, and that the metadata you apply can express just about anything you want it to.

So, XHMTL does attach formatting to content, and an XHTML processing application (like, say, a web browser) would render our sample content like this:

War and Peace is a very long book.

That’s fine, as far as it goes, but XML can enable us to do a lot more.

Separating text from formatting

If we do want to separate content from formatting, we need to add some metadata to create the separation. XHTML provides a way for us to take the first step along that road by using the <em> (emphasis) tag rather than the <i> tag:

<p><em>War and Peace</em> is a <em>very</em> long book.</p>

Rather than specifying that “War and Peace” and “very” be printed in italics, this markup simply says that they are to be emphasised. It leaves it up to the processing application to decide how to emphasized them. For instance, it could do this:

War and Peace is a very long book.

But there is a problem here. The processing application has every right to print the emphasized content in bold red text, because making text bold and red certainly does emphasize it. But there are conventions for how you show the title of a book, and that convention is that it should be rendered in italics.

If we are going to separate content from formatting, therefore, we had better do it properly. If we just use <em> as a synonym for <i>, then we are not actually separating the content from the formatting and we would be better served to stick to <i> since it is actually a better way of saying what we mean.

If we truly want to separate content from formatting, we had better find a more discriminating way to go about it than simply replacing <i> with <em> everywhere. If we are not going to format the text directly, then we need to give the processing application enough metadata that it can distinguish things that ought to be formatted differently.

Now we have provided enough information for an XML processor to render the sentence appropriately:

War and Peace is a very long book.

About the author

The place to begin is with the pain points.

Introducing database publishing

In one organization, we found a tiny group putting together product catalogs and price books for their dealers with a slow, expensive, and painstaking process. Basic product descriptions and benefits lists were combined with photographs of each product and tables of data copied manually from the price data in a corporate financial database. After the copy was assembled and approved, a graphic designer took six months using Adobe InDesign to carefully lay out each page. By the time the catalog was completed, the pricing data was already out of date. Each catalog was a chaotic mix of page layouts with stars, bars, explosions, and other “decorative” items strewn through hundreds of unique pages. The design chaos made it difficult to update the content with last minute changes. And, the customers had difficulty finding the precise products and prices they needed.

Figure 1: Existing catalog development process

Dynamic development process

So, how you overcome old school product catalog challenges? First, identify the common elements in the catalog copy and assist the catalog team in developing a standard. Overcoming allegiance to layout chaos means stressing the potential for cost cutting. The promise of faster turnaround and lower development costs can persuade even the most recalcitrant sales team of the benefits of standardized copy.

Next comes the design of a simplified authoring environment. The standard copy consists of a product name, a short description, and a benefits list. Each product is illustrated with one or more photographs. The result is a thoroughly simple design that could easily be developed by a single author.

The base for authoring becomes an XML/DITA editing tool that simplifies entering content. Every product description follows the exact same structure, a feature that increases information access and readability for the customers.

[Editors note: DITA refers to the Darwin Information Typing Architecture, an XML-based architecture for authoring, producing, and delivering technical information. XML stands for Extensible Markup Language, a set of rules for encoding documents in a format that is both human- and machine-readable.]

The most complex part of the new process is the design of a process to produce the pricing tables by drawing the data directly from the financial database. Once the required transforms are in place, each table is built automatically and appended to the content in the DITA topic.

For this project, we created transforms to develop custom indexes that make the information easier for dealers to find. Once the XSL-FO stylesheet was in place, a design and publishing process that once took months to complete was completed in minutes and eliminated the graphic-design costs. Once the copy is reviewed and approved, the printing takes about two weeks. However, this company is now moving to on-demand printing triggered by a specific dealer request.

Figure 2: New Catalog Publishing Process

The marketing and sales managers, who had been skeptical at first, saw the potential for additional catalogs. The second project was a non-US catalog with different prices, dealer discounts, and a more limited product set. Metadata allowed for easy selection of products for more limited markets, replacing the one-size-fits-all approach of the previous design process. The automated publishing process also enables the company to produce catalogs much more frequently than they have in the past.

The company plans to link the DITA topics and the publishing process to their e-commerce site and automated online purchasing process.

Taking automated publishing to the next level

But automating the publishing process and integrating XML structured text with data from a relational database is only the first step in catalog publishing. In another organization, we identified the potential for customized catalogs produced on-demand for a sales representative.

Once again, the basic information about a product is authored in DITA topics and stored in a component content management system along with the drawings and photographs required for the catalog.

The analysis of the catalog in this project included not only descriptive information but tables and graphs of data. The original presentation of the information was as chaotic as the first project. Every product section in the print catalog was unique and every page was individually designed. In some cases, product benefits lists and critical engineering data were truncated in order to fit the page layout.

Once again, the first step was standardization. Working with the marketing communications management, we identified a standard set of content for each product and designed a consistent layout to be produced through the DITA Open Toolkit and an XSL-FO stylesheet.

But the key question was how to get the sales representatives to use the source information to produce custom catalogs. We knew that sales representatives wanted to create unique catalogs for individual customers, rather than carrying around the huge catalog of the entire product line. They also wanted nothing to do with DITA topics and XML.

In response, we created a SharePoint interface that allows a sales person to select from a variety of options, including product name, industry type, and other key characteristics of the information. By simply clicking check boxes, each sales person creates a unique catalog that is fully formatted and immediately available for download and printing.

In a similar project some years earlier, we enabled the sales representative to use a simple, clickable interface on a website to select specific functionality for a product set and choose a cover page with the customer’s business name. Because the sales force was widely distributed, we even gave them the ability to select a printing facility nearby. All that was required of them was to drive to the printer and pick up the final catalogs for their next sales call.

Looking for opportunities

In most organizations, development of product catalogs and price books are cost-intensive, time-consuming, and tedious. They represent the ultimate static publishing. But the arguments for dynamic publishing of catalog content are easily persuasive to those watching out for the bottom line:

• Significantly reduce the time-to-market for catalog content from months to days
• Significantly reduce the cost of developing final catalog layout to the initial development of XSL-FO stylesheets
• Enable the development of custom catalogs from the same source content
• Enable on-demand printing of custom catalogs
• Ensure that pricing data is accurate and up-to-date, coming directly from purchasing databases
• Enable on-demand development of custom catalogs by individual sales representatives
• Link catalog content to e-commerce websites

No doubt there are many other opportunities for dynamic publishing in your organization. Your starting point is to develop a single source of all materials (text, images, and pricing data), free of formatting (XML/DITA), and presented to staff and customers through an intuitive and easily available user interface.

Mark Gross, President/CEO Data Conversion Laboratory

Data Conversion Laboratory, the company I founded, has been doing document conversion for thirty years and every once in a while I still get asked from someone I haven’t seen in a while “are you still doing that?” or “isn’t there software that does all that?”

The truth is that if it was easy, it would indeed be all automated, which is already the case for news feeds, financial transactions, and other standardized data flows. But when it comes to documents and books, creativity will not be bound by rules and style sheets, especially at deadline when one wants a certain look, and MS-word chooses not to cooperate. The truth is that a document can contain anything, and computer software doesn’t work well with ‘randomness’.

Even what to the human eye looks like a simple book – a book with simple text, no tables, and no links – still contains complications that will thwart software not meant to deal with it. In a recent test of three free software packages, not one book came out perfectly. Each and every one of them had problems. To complicate matters, each book had different problems (recorded webinar includes examples).

Let’s get flexible

In order to deliver the high quality, customized information that consumers expect – and in some cases, demand – we’re going to have to start thinking seriously about creating automated streams of standardized content – something most organizations don’t have today.

For organizations with large collections of information to distribute and monetize their content, the clearest solution seems to be adopting a flexible, standardized content model and maintaining that content in an appropriate type of content management system. In our mobile-connected, always-on world, this means creating and delivering content able to customers on the device of their choosing. It also means future-proofing that content so it will be ready to be quickly and efficiently prepared and delivered to the many devices that have yet to hit the market.

There are many content standards to choose from: HTML, EPUB, XML, PDF, and DITA. But which is one is right for your purposes? For your audience? For the audience of the future? Will you need to support more than one?

For now, let’s assume that some robust form of XML will be the right thing to store your information – the specific form best for your content will need some more discussion. However, it seems that for most large collections, moving to one of the simpler formats like HTML or EPUB can be a risky investment due to the lack of flexibility they offer.

EPUB, for example, now on version 3.0, is specialized to the needs of electronic books as they currently exist. It’s very possible that all the features of your content are not easily definable within the EPUB standard, and are not displayable on current devices. If you limit yourself to converting to the current version of EPUB you may be limiting your content as new capabilities, not currently envisioned, are introduced. The same is true of HTML, which is designed for display of information.

To preserve your investment in converting to a standardized format the safer approach is to convert and store the content in a more robust version of XML, such as DITA, DocBook, NLM, TEI, S1000D, and various other XML standards created for specific purposes. If properly designed, you would then be able to automatically convert your content to EPUB, HTML, PDF, and other final formats in the future.

OK, why is it so difficult to convert?

If everyone wrote their documents with the intent that they be standardized and converted, life would be easy (and we wouldn’t have that much to do). But the reality is most content is not easily extractable, and lacks the details needed for a full conversion. Much needs to be corrected, and much needs to be inferred based on the content.

As an example, let’s look at the difficulties in extracting content from PDF files. Since PDF is a print format, PDF documents are typically less-structured versions of their word-processor originals. While PDF content is laid out to look good, it includes very little structure—that is, it contains few clues as to the function of text elements (e.g., paragraphs, spaces, line breaks) or how they ought to be displayed in a different context (for instance, an e-book). While converting thoroughly structured content to XML is straightforward, PDF doesn’t contain explicit structuring. But an even more basic problem has to do with properly extracting the content from the PDF to begin with.

Examples of problems you are likely to encounter with commercial packages include the following:

Incorrect Word Spaces

While spacing is usually extracted correctly, since PDF documents create spaces visually (i.e., they are not really labeled as “one standard space” or “two standard spaces”), spacing between words is sometimes misinterpreted by conversion software, causing spaces to be added or deleted incorrectly during PDF-to-Word extraction. That’s why ebooks that have not been fully reviewed will have words coming together, or otherwise incorrectly spaced.

Paragraph Delineation

In most cases, PDF documents contain no explicit information to indicate where a paragraph begins or ends, so this too must be guessed at by conversion software, based on “visual” interpretation of the appearance of chunks of text. While conversion software frequently does guess correctly, paragraph delineation can be a source of extraction errors, particularly when paragraphs are very short or span pages, or images and table get in the way.

Hyphens

Hyphens pose a problem because they serve various purposes among which an automated system cannot distinguish. While the hyphen joining a term such as “half-life” should appear no matter where the words are placed within a document, a hyphen that appears halfway through a word because of a line break (e.g., hyphen-ated) becomes an ugly error once the word is moved to the middle of a line. This is also something you’ll see often in ebooks you download.

Emphasis

Depending on how a document is rendered in PDF, extracting the correct emphasis from a PDF document can sometimes pose problems for conversion software. Again, this is because PDF structure is nothing more than a visual representation; while text may appear emphasized, the PDF does not tag it as “emphasized”—conversion software must make its best guess based on what it can glean from the text’s appearance.

Superscripting and Subscripting

Since PDF documents’ treatment of super and subscripts is limited to the way they appear when laid out in the PDF (rather than by some kind of “superscript” or “subscript” tag), extraction software tends to run into problems with determining the vertical alignment of text. As a result, super and subscripts are frequently misinterpreted by extraction software.

Special Characters

In PDF documents, special characters like foreign or mathematical symbols are frequently represented by unusual or proprietary fonts. In order to extract them to a word processor, these characters first need to be converted to a more standard character representation (e.g., ISO or Unicode). While many conversion software suites build conversion tables to handle such characters, it is impossible to keep up with the vast variety of atypical and proprietary fonts in use, and so many special characters fail to extract properly.

Sub-fonting

PDF’s approach to font embedding is another obstacle to proper extraction. Sometimes when PDFs are created, the PDF document does not store the information for the entire font, but rather stores only the parts of the font, which are used in a given document. The characters within this “sub-font” are accessed via an indirect table within the PDF document itself, making correct interpretation and extraction of sub-fonted characters difficult. Many conversion tools cannot extract these characters at all, and produce “garbage” text instead of accurately extracted content.

Tables

Tables are among the trickiest document elements to extract. This is because the appearance of even a simple table is determined by numerous attributes, including but not limited to column and row delineation, header and body delineation, vertical and horizontal cell spanning, cell separators, and vertical and horizontal cell alignment. With none of this information included in the source PDF, it is nearly impossible for an automated tool to reproduce a table exactly as it appeared in the original document.

While some short or simple documents may be able to undergo a PDF-to-Word (and subsequent PDF-to-EPUB) conversion with minimal difficulty, any long or complex document set will encounter several of these obstacles. The obstacles inherent in any PDF text extraction should underscore, first, the utility of retaining original versions of source documents in word processor format, if possible; and second, the critical importance of a good quality assurance strategy in any conversion process.

So what do I do now?

Obviously many millions of pages get converted; we convert millions of pages ourselves. There are solutions to all of the above and approaches to dealing with all the above, and more, which will be discussed in future columns.

Questions about conversion?

If you have specific questions about conversion that you’d like me to answer, use the comment feature of this website to submit them. I’ll do my best to find an answer for you.

Scott Abel, Editor, TheDynamicPublisher.com

As it turns out, there are subtle variations in the definitions industry leaders and veteran consultants use to explain what they mean when they talk about dynamic publishing. In fact, some of the most widely cited experts don’t offer up a clear-cut definition for the term, opting instead to define related terms like “dynamic content” or “dynamic delivery”.

According to Ann Rockley, author of “Managing Enterprise Content: A Unified Content Strategy” (Second Edition) [New Riders, 2012], what differentiates dynamic content from its static cousin is that “dynamic content does not exist in or as a document; it is information that is assembled only when it is requested. It exists as a series of information objects that are assembled in response to the user’s requests or [other] requirements.”

Rockley’s explanation of dynamic content is straightforward, but isn’t sufficient to answer the question, what is dynamic publishing?

JoAnn Hackos, in her 2002 work, “Content Management Strategies for Dynamic Web Delivery” (Wiley) discusses some of the benefits of “presenting content dynamically”, including the “great potential to make web-based content-rich resources more valuable to users.”

“Users appear eager to work with resources that are ‘customized’ to their needs and respond to their queries effectively,” Hackos writes. They also appear, “to prefer personalizing information resources that they use frequently.”

The founding sponsors of TheDynamicPublisher.com, Quark Software, say they believe dynamic publishing is “based on two fundamental principles: Using structured, reusable XML content and automating the delivery of this content to any media type.”

Quark emphasizes automation of all processes involved in an end-to-end dynamic publishing solution, including, the automation of:

• Content reuse
• Layout
• Workflow
• Formatting and multi-channel publishing
• Custom and personalized content

And while all of this information is interesting and thought-provoking, I am still left without a clear and unambiguous definition for dynamic publishing. Shouldn’t there be a Wikipedia page dedicated to this topic by now? As it turns out, there is. But, unfortunately, while the popular online user-generated encyclopedia is often useful at solving semantic challenges, in this instance it isn’t much help.

So What’s An Editor To Do?

After struggling with this challenge for several days, I realized that perhaps the best way to come up with a solid definition was to ask the community for help. That’s where you come in.

What is your definition of dynamic publishing? Please use the commenting feature of this blog to share your views on the subject. Next week I’ll summarize our findings and attempt to craft a definition that encompasses much of our thinking.

Scott Abel, Editor, TheDynamicPublisher.com

It’s hard to deliver the right information to the right people at the right time in the right format and language, increasingly, on a menagerie of mobile devices that seem to pop up faster than daisies after a good rain — especially if you rely on traditional publishing techniques. If your organization is like most others, no matter how hard you try, or how much money you throw at the problem, your reliance on print-centric processes and outdated, labor-intensive workflows fail you. But, you’re not alone. There are people who understand your problem and are willing to share what they’ve learned with you in order to help you begin to think outside the traditional publishing box.

Enter TheDynamicPublisher.com. Originally a website dedicated to information about dynamic publishing as envisioned by Quark Software, today TheDynamicPublisher.com has been both rebooted as a vendor-neutral, one-stop shop for information about dynamic publishing and related topics. I’m your host, Scott Abel, The Content Wrangler. My job is to “wrangle” relevant, informative, and useful content about dynamic publishing from the world’s best and brightest experts and present it to you.

Dynamic publishing makes possible the efficient delivery of content to mobile devices, on demand

That’s not all. I’m also responsible for developing this site into a vibrant community of content professionals interested in promoting the methods, standards, and software tools required to create, assemble, and deliver relevant content to those who need it, dynamically, on demand.

In order to reach my goal, I’ll be seeking assistance from the crowd to help me build a body of knowledge and a community that will be a resource to those who want to move away from inefficient, old school ways of creating, managing and delivering content. Won’t you join me?

How can you get involved?

If you’ve got questions about dynamic content and related topics, send them to me via email and I’ll find an expert or two to answer them for you.

If you’re a subject matter expert and would like to contribute an article, let me know what you’d like to write about and how it relates to dynamic publishing.

If you work for a company that produces software products or related services designed to help organizations produce dynamic content and you’d like to become a sponsor, send me am email and I’ll let you know how you can get involved.

And, if you’re interested in the topics presented on these digital pages, then read, comment and question. Become part of the community!

TheDynamicPublisher.com is a work in progress. We’ll be updating and improving the site as we go. If you have ideas about how we can make the site better, please let me know.

“The popularization of smartphones and the introduction of tablets and other Web-enabled devices — collectively termed ‘connected devices’ — have contributed to an explosion in digital media consumption. As these devices gain adoption, we have also seen the rise of the ‘digital omnivores’ — consumers who access content through several touchpoints during the course of their daily digital lives. In order to meet the needs of these consumers, advertisers and publishers must learn to navigate this new landscape so they develop cross-platform strategies to effectively engage their audiences.¹”

Cross-media and dynamic publishing are Quark’s focus. But Quark also sees a significant need for our customers to create an engaging customer experience — whether it’s in print, via the Web, or on a digital device such as the iPad, an Android tablet, or other tablets.

In my last blog, I spoke about cross-media design; now I’d like to focus on workflow collaboration. As rate of content consumption changes, so do the requirements for publishing systems to support new types of content — namely digital — and for bringing new content specialists into content creation workflows.

Digital Assets and Workflows

Once you go digital, you’ll need to embrace the need for rich media content, whether it’s for your Web site or your iPad app; therefore, the first step in preparing to adopt a cross-media publishing strategy is to include support for creating and managing your digital assets and publications. This includes videos, slideshows, podcasts, any other interactive or multimedia assets, and iPad apps and issues.

If your team isn’t big enough or if you don’t have enough resources to create these new type of assets, no need to fret; there are services available (such as iStockphoto®) who saw the need to expand their portfolio beyond photos and now offer video, audio, and Flash content.

On the other hand, if you do have in-house multi-media staff, you want to make sure you include this staff in your workflows. It will be important that your digital issue designer has the ability to collaborate with, let’s say, your video editor to ensure that the video you include in your iPad issue delivers the best possible user experience (i.e. full-screen display, etc.). When working together, these two creative minds will surely deliver an immersive, brand-worthy experience.

On another level, if you’re trying to manage the creation of assets and publish content across documents, media, and platforms, you’ll also want to be able to set up specific workflows to support each type of digital asset and publication required. These workflows are vastly different from what may be required for print, in that the dependencies and timeline or workflow automation triggers may change.

iPad Continues to Dominate

“iPads dominate among tablets in driving digital traffic. In August 2011, iPads delivered 97.2 percent of all tablet traffic in the U.S. iPads have also begun to account for a higher share of Internet traffic than iPhones (46.8 percent vs. 42.6 percent of all iOS device traffic).¹” As the iPad continues to dominate, the need to publish to the iPad is gaining ground. Thus, the ability to create an iPad app and publish an issue to that app is something that should be available in the cross-media publishing system.

With that aim in mind, as you evaluate the best cross-platform strategy for you, it’s imperative that you understand what’s involved and leverage best practices shared by the industry experts. The next step for starting your exploration, if you haven’t already, is to take the time to check out The World of Dynamic Publishing, a five-part webinar series brought to you by Quark Software Inc with leading industry experts.

¹comScore, (2011, October 10) Smartphones and Tablets Drive Nearly 7 Percent of Total U.S. Digital Traffic [Press Release]. Retrieved from http://www.comscore.com/Press_Events/Press_Releases/2011/10/Smartphones_and_Tablets_Drive_Nearly_7_
Percent_of_Total_U.S._Digital_Traffic.