Kindle Fixed Layout Tutorial - Part 12

CONVERSION & TESTING










Once you have created at least one page of content, along with all the necessary support files (OPF, NCX/NAV, CSS, images, etc.), you are ready to convert your first test Kindle file. This will almost never goes as planned, and very often the resulting file simply will not work, or will be broken in some unexpected way.





However, if you have created each of your component files carefully, and listed them correctly in the manifest, the chances are your ebook will be work correctly on the first try and be fairly close to what you’re after.





This is why it is imperative to take your time and examine every line of code as you enter it, and then again before you save and close each file. It can save you endless hours searching for a missing semicolon or an unclosed tag.



Compiling The File







If you have created your content as an epub archive, either by using one of the templates to add and edit your own content, or by changing the extension of a .zip archive to .epub, then you are already set to go. Skip ahead to the section on Converting The File.





If, on the other hand, what you now have is a lot of files and folders on your hard drive, you have a bit of work to do yet in order to prepare the files for conversion.





All of the necessary files must be added to a zipped archive in the manner described in the earlier section on Opening the Archive, and as seen in the templates. That is, the top-most level - or “root” level - of the archive structure must contain, at the very least, the OPF and any folders into which you have added support content.





All of your files can, in fact, be dumped together into the root level of the archive if you so choose, or you can order them in any way you like, so long as the location or every file is recorded correctly in the manifest section of the OPF so that the reading system knows where to find them. But only the OPF itself is required to be at the root level.





At this point you can either organize your file structure on your hard drive and then add it all at once to a zip archive, or you can create the compressed archive first with just one file inside and then add the others as they’re ready. Do not create a .rar or .7z archive, or use any other compression format than a standard zip. Also be sure the archive uses relative paths rather than full paths, as the latter will use your current hard drive for the root level, whereas you want the archive itself to be the top level, with all file paths relative to that location.





Once you have created a zip archive containing all of your required ebook files, change the .zip extension to .epub and you’re good to go. If you can’t see the file extensions you’ll need to change your system settings so that they’re not hidden (they’re hidden by default on both operating systems).





In Windows this is done in the Folder Options pop-up on the View tab, where you’ll need to uncheck “Hide extensions for known file types”.





On the Mac you do this in the Finder under Preferences > Advanced > Show all file extensions; but you can also show or hide extensions for individual files by opening the file’s info window (“Get Info”), where you’ll find a “Hide extension” checkbox at the bottom of the Name & Extension section.




Now you can simply rename the extension and you’ve got an epub file ready for conversion.






Converting The File





Amazon provides three different ways to convert your compiled epub archive into a Kindle ebook file, but there is really only one choice that you need consider, and that is Kindlegen.



The other two options are the Kindle Previewer, and the KDP upload portal, both of which have several major shortcomings that make them difficult and inefficient to work with, although this will depend to some degree on what you want and now complex your project is. To some degree it is also a matter of preference, so we’ll look at all three.








KINDLE PREVIEWER





The Kindle Previewer serves two separate functions, with varying degrees of success. First, it provides a graphic user interface for Kindlegen, which is the program that performs the actual conversion behind the scenes. Kindlegen itself is a command-line program that at first may seem somewhat daunting to use, since we are all more or less accustomed to point and click interactivity. In many ways, Previewer is easier to work with for this reason, although it has some drawbacks as well.





Secondly, as its name suggests, Previewer allows you to view your converted Kindle ebook as it would appear on a wide range of devices. A drop-down menu on the home screen allows you to choose your preferred device as the default, and lists the models currently supported by the emulator:





The obvious benefit to this is that it lets you see how you book will look on devices that you do not own. The problem is that many features (such as Panel View or landscape mode) do not work in Previewer the way they do on the actual device, and some of the devices listed don’t work at all. The iOS apps, for example, are not available for testing fixed layout files - the option is grayed out and inactive - which is one of the places Previewer would be most helpful, since it is not possible to test these on an actual iPad or iPhone.




On top of this, Previewer tends to be laggy and unresponsive even on the best desktops, testing the user’s patience as much as it does the ebook. It’s difficult to tell, for example, how many clicks are required to active a feature (such as a tap target or a hyperlink), since sometimes one click seems to work while sometimes two are needed - the delay in response is so great you almost always want to click again!





This makes using Previewer a slow and tedious process at best, although it is still useful to some degree, particularly if you have no other means of testing. But I don’t recommend you rely on it solely for your testing, since it is neither accurate nor fast. I rarely use it at all, as I do all my testing on the actual Kindle apps and devices.





One of the primary benefits to Previewer is that not only can you convert EPUB files, but it also accepts HTML, MOBI, and OPFs as well, and you can you can convert them simply by dragging and dropping the file onto the home page of the interface. Because Previewer accepts OPF files as a valid input, this allows you to skip the zip compression step and create your file structure directly on your hard drive. You then just drag the OPF into Previewer and this tells the conversion program where to look for all the necessary files.




Once the file has been converted it will open in the emulation mode you’ve set as your default. Again, there is a lengthy delay as the file is opened, and some uncomfortable sputtering as it’s content is sorted out. Finally, after a seeming lifetime, you can move around in your book (in fits and starts). This means that Previewer cannot be used as a conversion interface only, since it also automatically opens the converted file in an emulator. For this reason, among the many others given already, I recommend you go directly to the source and use Kindlegen itself without the Previewer interface. But again, this is a matter of personal preference, and the choice is up to you.












KINDLE DIRECT PUBLISHING





When you set up your title on KDP you will be given the opportunity to upload your unconverted source file, upon which it will be converted into a Kindle ebook. KDP accepts many source formats, including EPUB, HTML, DOC, RTF, TXT, and PDF, but the only viable format for fixed layouts is ePub. Only with this format can you control your layouts and metadata as this book has shown. The other formats are only useful for reflowable ebooks, and so do not concern us here.





Once the file has been ingested successfully into the KDP system, you will be provided with a link to download the converted file for external testing, or you can preview it online. Unfortunately, the Online Previewer is virtually worthless, as it has no interactivity and does not display fixed layouts correctly. It is also tediously slow in rendering the pages, and does not show two-page layouts, even on devices that support it.





It does, however, accurately convert your file, which is its main purpose, since this is the file that will be downloaded by your customers. Consequently, there is no better file to test than this. The downside is that it takes forever and a day to do what Kindlegen can accomplish on your own system in less than fifteen seconds. Therefore, I only recommend you use the KDP conversion system for the final round of testing before your book goes live.






And even better option is to do all the testing and conversion before you upload to KDP, and upload your converted mobi rather than the epub. Not only will this will be considerably faster, but it will also allow you to double-check your own testing and verify your process works.



Of course, one drawback to using KDP is that it requires you
to create a title listing for your book, which requires that you have a KDP
account, even if you never plan to sell the book on Amazon. Chances are, of
course, that you will, but it is a hindrance if you just want to convert your
file.









KINDLEGEN





The workhorse of the Kindle conversion pipeline, Kindlegen will become your best friend and ally in the process of turning hard work into literary art. While at first this command-line interface might seem foreboding, and even clunky, it is, in fact, highly efficient and simple to use, providing all the information you will need to repair any errors and determine exactly what is happening to your files during the conversion process.





Once you download and install Kindlegen you will want to put the primary program file (kindlegen.exe) in an easy to access location. I suggest you place it in the main C: drive root directory, and use this for all your conversion activities.





The reason for this is that you cannot simply double-click on the Kindlegen executable to run the program. Instead, you must run it from the Command Prompt in Windows (Windows / System32 / cmd.exe) or in Terminal on the Mac (Applications / Utilities / Terminal.app). In Windows this can also be accessed via the Start menu > All Programs > Accessories menu. I highly recommend you add a shortcut to this either on your main Start menu or your desktop for ease of access, as you’ll be using it a lot.





When you open the Command Prompt what you’ll get is an old-school MS-DOS style terminal where you type in the “commands” you want to run:





As you can see, you can change the default drive and file location using the cd (change directory) command. Here I’ve changed it to the main C:/ drive where my kindlegen.exe program is living. You can, of course, run this from anywhere. You may, for example, want to create a dedicated folder just for your conversions, and run Kindlegen from there instead.




To convert an epub file using Kindlegen you simply type in the command - which is just the word “kindlegen” followed by a space - as I have done in the screen cap above. You then enter the full name of the file you want to convert - including its extension - and press enter. The easiest way to do this is simply to copy and paste the file name.




Note that there must be a space between the command and the file to be converted, and that the file name itself must not contain any spaces. You can either remove any spaces between words in the file name or replace them with dashes or underlines. It is helpful to add a sequential number to your test files rather than simply overwriting the previous version, just in case something goes amiss. That way you always know where you are and can go back to earlier conversions if something didn’t quite work as planned.






Kindlegen supports a number of variables that you can append to the command line in order to achieve various outputs. You do this by adding any variables to the end of the command line string, such as:

Kindlegen filename.epub [-tag] [-tag <option>]

For example, you can change the language of the output log, using the -locale tag along with a two-character country identifier, such as:

Kindlegen filename.epub -locale jp

which will output the log in Japanese! Other language options currently supported are French [fr], German [de], Italian [it], Spanish [es], Chinese [zh], and of course, English [en].




You can also turn off  the file compression, or increase it, using the -c0 variable for none and -c2 for heavy (Huffman) compression. The standard amount applied by default is -c1.




Another option that you may want to use is the -verbose tag, which outputs an additional amount of data in the conversion log, although it does not provide any particularly useful information.




Other variables allow you to change the output file name (useful for adding a version number or time stamp), view the release notes (also available in the Kindlegen extraction folder), or convert all your images to gifs - but I recommend you do this last one in Photoshop instead where you have more control over the image quality! See Section 2.2.2.3 of the Kindle Publishing Guidelines for a complete list of variables.





Once you press enter Kindlegen will leap into action, spewing out a log of what it’s doing, along with some useful statistics on the final file size and compression applied. The conversion will progress sequentially through a series of steps that are each given a line item number for reference, all in about ten to fifteen seconds, depending on the length and complexity of your file.









Using Kindlegen’s log can be among the most important tools in your conversion arsenal, since it is the only way to know what is really going on inside the end product. This, for example, is where you will see what actual metadata values are being applied (which in the case of region-mag has nothing to do with what you enter in yourself), or how much compression is being applied to your file.




Previewer, by the way, also offers you this log, but it’s hidden by default under a drop-down window, and is double-spaced so that you have to scroll several screens to view it all, making it somewhat more cumbersome to access. Previewer will also automatically append a version number to the output file with a date stamp that is helpful.




If there are any non-critical errors (such as non-closed tags or broken links) Kindlegen will attempt to fix them or apply a work-around (for example, it will forcefully close tags), and give you a warning noting the error location so that you can go back and manually correct it. It will complete the conversion successfully, but “with warnings.” You will need to fix these before you upload the file to KDP.




If the error encountered is critical, Kindlegen will abort the conversion at that point and tell you why so you can fix the problem. This can be anything from images that are too large to meet the file size restriction to non-existent support files or incorrectly entered information that is required for the ebook to function properly (such as missing navigation data or manifest entries).

Notice that the first portion of the log extract above is given to your metadata values (I1047 entries), although the final one - region-mag - does not get added until after the files are parsed (I1002); only then does Kindlegen determine whether or not your ebook contains region magnification, and it sets the metadata value accordingly, regardless of the value you have entered.



It then proceeds to build the PRC file (Palm Resource Code, the forerunner of the Mobi format on which Kindle files are based), resolves any hyperlinks, as well as determining the “start reading” location (based either on your Guide entries, or NAV data).




Page Maps





After this you will notice a line (I8000) that states “No Page map found in the book”. This is because the Kindle fixed layout format does not support Page maps, although you can add one if you like. Page maps are used to align an ebook’s pages with the print edition of that title, and it’s how Amazon adds “Real Page Numbers” in the reading progress indicator at the bottom of the Kindle.




A page map is simply an xml file that provides this page numbering information as a “map” - much like the Spine in the OPF, but with its own terminology:






      <?xml version="1.0" encoding="UTF-8"?>

      <page-map xmlns="http://www.idpf.org/2007/opf">

            <page name="1" href="page1.xhtml#page1" />

            <page name="2" href="page2.xhtml#page2" />

            <page name="3" href="page3.xhtml#page3" />

            <page name="4" href="page4.xhtml#page4" />

            ...

      </page-map>





The anchors (#page1, etc.) are used to mark specific line locations within a given file as the start of a new page, since in reflowable ebooks this could be anywhere. In fixed layouts, of course, each page in the ebook equals one page in the print edition as well, so the anchors are unnecessary.





Also note that you can use Roman numerals for Front Matter, and even leave the name value empty for pages with no numbering, such as the Table of Contents.




If you do include a page map, also be sure to add a reference to it OPF manifest:

<item id="pagemap" media-type="application/oebps-page-map+xml" href="pagemap.xml" />

And in the top-level Spine entry you need to add the following page-map portion along with your toc=“ncx” tag:

<spine toc="ncx" page-map="pagemap">

where the page-map value is equal to the id value you entered in the manifest.




However, there is currently no real reason to do this in the Kindle fixed layout format, since these page numbers do not actually show up, even when a page map is included. Only a “Location” number is given on the progress indicator, as well as in the “Go to Page of Location” menu, where the “Page” button is inactive.




In essence, page maps are irrelevant in fixed layouts, since each page is given it’s own numeric location number, making this the de facto page number. That said, some day Amazon might start to include page maps in fixed layouts as they do in reflowable ebooks, so that real page numbers and Roman numerals can be added. If so, you now know how to use them.




As a final note on this point, I should mention that the lack of a page map is not considered an error in Kindlegen, and thus does not receive a warning. You can simply ignore this information.




Mobi7 vs. KF8




Finally, you will see that there are two sections near the bottom that each create a PRC file (I1017). The first of these creates a basic mobi version of your ebook (V6):




Notice that the record count is only five entries, whereas there are 18 pages in the book. This is because mobi7 is an older version of the Kindle format that does not support fixed layouts, and is used to produce a version of the ebook that can be read on older Kindles. This is only relevant to reflowable titles, but this part of the conversion process still takes place regardless. Here it has converted all the individual pages into a single reflowable HTML file compatible with older devices, as well as removing all the CSS files, so that only the NCX, OPF, and images remain.




This is pointless for the most part where fixed layouts are concerned, as this version of the file will never be available for download on Amazon, since Kindles that do not support fixed layouts present a message telling users that this ebook is not compatible with that device or app. Why Kindlegen does not bypass this portion of the process when it reads the fixed-layout=“true” metadata entry is beyond me. The mobi7 data can simply be ignored when building fixed layout titles.






Below this is where the real conversion gets underway:





Here Kindlegen starts building the “enhanced” PRC file, which is file format V8 (i.e. KF8). Note that there are a few additional lines where the various media id links are resolved (these are the links to images and embedded fonts), and the guide items are sorted out. The record count now shows that all 18 pages have been parsed and compiled.



And finally we get a statistic that shows how much the text has been compressed. This has nothing to do with how much compression has been applied to the images during conversion, but applies only to the text content. And since fixed layout files tend to have far less text than standard reflowable ebooks, this will be  a fairly unimportant statistic in most cases.




What is important are the two lines near the end which give the final deliverable file size for both the standard mobi7 version and the KF8 fixed layout file that has just been created. Both of these are packed inside the converted file, along with a copy of the original epub source content, which makes the output file much larger than the one we started with. What these last line entries tell us is how large the actual files are that will be delivered to the buyer.




Here we have a Mobi file that is 4272 kB in size (4.27 Mb), while the KF8 version is just slightly larger at 4445 kB. The file that has been created, however, is 17,080 kB, as it also contains the 12.6 Mb epub archive. Amazon keeps a copy of the source files so that new conversions can be done when updates are available without having to re-upload the file every time.



And, of course, finally, the last line is the one you ultimately want to see, telling you that your ebook file has been built successfully. If you get an appendage to this message that says “but with Warnings” - or the dreaded abort message - you will need to go back through the log and find the entries that are marked as such and fix the errors.




Mobi Unpack





While the Kindlegen log shows the percentage of compression applied to the text content, the only way to see how much your images have been compressed is to unpack a converted file and look at the image file sizes.



Mobi Unpack is a python script that allows you to do just that. You simply double-click the script to launch the graphic interface, then enter the location of the mobi file to be unpacked and the output location where you want the file’s contents to be extracted. You do not need to worry about any of the checkboxes unless you’re doing more advanced analysis procedures. Click “Start” and in a matter of a few short moments the ebook will be successfully unpacked.





What you’ll find when you look at the extracted contents is a zip file named “kindlegensrc.zip” (which is your source epub content), a log file containing a record of the conversion process (which also appear in the interface window during the conversion), and one folder each for the mobi7 and mobi8 content.




      kindlegensrc.zip

      kindlegenbuild.log

            /mobi7/

            /mobi8/


Inside the mobi7 folder will be the compiled HTML containing all the content in one file, along with the OPF and NCX, plus a folder for your compressed images.




The mobi8 folder, however, now contains some extra files that were not in the source epub. These are actually required for an epub to function correctly, but Kindlegen creates them for you automatically. These are a mimetype file and a META-INF folder containing a container.xml file. Unlike when you build an iBooks file from scratch, here you do not need to mess with these at all.




There will also be an actual epub file that has been created from the converted and compressed content, which will be much smaller than the source file that you fed into Kindlegen, and of roughly the same size as the mobi7 file.




A second folder labeled OEBPS holds all the ebook content you created, with a few exceptions. First, if you have included a Nav document or page map, these will inexplicably be gone. Clearly, Amazon are now supporting the ePub3 Nav Doc for basic functionality, so it must still exist in the compiled Kindle file, but in the extracted folder it will be absent, It is, however, still present in the kindlegensrc zip file, so this is something Mobi Unpack is doing, and not Kindlegen.




The other thing you’ll notice right away is that the folder names have changed, along with the names of all the files they contain. The /html/ folder will be renamed /text/ and the /css/ folder will be called /styles/. But these are minor in comparison to what has taken place to your content files.




Every file in each of the folders will be renamed with a numeric string, based on the order it appears in the publication. So, for example, you images will be called “image00001.jpeg”, etc., and your pages labeled “part0001.xhtml” and so on. This makes it a little more awkward to find out which file is which, but of course for images it’s easy enough to see by using thumbnail view.



By looking at the images in the extracted folder you can compare their file sizes to the ones you added to your source epub, and get a good idea of how much compression is being used. The amount will vary greatly depending on the color depth and complexity of each image, as well as how large the image was to begin with. Kindlegen will reduce the image quality by up to 40% to get them within its limits, so the difference can be fairly significant, both in terms of file size and image quality.





One more thing that may be of interest is to look into the OPF. I warn you upfront, however, that it will not be a pretty sight. All you hard work organizing and creating clean, easily readable code will have turned into an endless stream of seeming doggerel. And not only are all your lines run together, but a whole load of new data will have been added as well.




In a section  prefaced by the comment “The following meta tags are just for information and will be ignored by mobigen/kindlegen” you will find a number of extraneous entries regarding the conversion format and build version. And once your ebook has been uploaded to KDP and a purchased copy is downloaded, there will be even more.








Testing The File






Now that you have a converted Kindle ebook you will need to test it thoroughly to be sure it working properly. You can, of course, look at the converted file in the Kindle Previewer, using its various device emulation modes to see how it looks on different readers. I have already talked about Previewer’s shortcomings, but you may find it works just fine for you, and provides as much testing as you need.



The obvious drawback to this approach is that none of your readers will be viewing your book that way, and so at best it’s only an approximation of what the end user will see. To be certain that your product is functioning correctly and at its optimal best you really need to test it on at least one actual Kindle app or device.


The quickest and easiest (as well as cheapest) way to do this is to install the free Kindle app on your computer. Once installed all you have to do is double-click on the converted Kindle file and it will open in the Kindle desktop application, where you can scan through the pages, test the zoom functions and links, and see how your page layout actually appears when rendered. Be sure to click on every link and test out every feature that you’ve added so there are no harsh surprises down the line when customers can’t get something to work.






If you have an Android phone or tablet you can also download the free Kindle for Android app and “side-load” your ebook onto it. Side-loading is the process of manually transferring a file to a mobile device. You do this by connecting the device to your computer using a USB cable (one of which usually comes with the device), and then dragging the file from your computer to the appropriate folder on the device. On Android devices this will generally just be the Kindle folder, but it may have a “books” subfolder as well. Also be sure to eject the device before you unplug it and open the Kindle app, otherwise you’ll find there are no ebooks in your library!




Of course, even better is to test your files on at least one physical Kindle device as well, if not more. Many features only work - or work differently - on a device rather than an app. Refer to the table in Appendix A for a list of functionality variables among the Kindle readers. A lot of unnecessary frustration can be avoided by testing features only on devices that support them!



I highly recommend that you invest in one of the latest Kindle models to do your testing. It is a small outlay for something that will provide virtually unlimited usefulness, particularly if you plan to sell the ebooks you produce.







Above is a shot of my current ebook testing setup. It is
admittedly a bit excessive, but I share it to make the point that you can never
test a file too much, only too little. It is also very helpful to have multiple
devices side-by-side so that you can see exactly where things are working
properly and where they are not. But so long as you can get your book to work
on at least one Kindle device you should be more or less good to go.




A few additional points should be made here regarding the
Kindle testing process.




First, when transferring (side-loading) files to the Kindle
devices you will find them on the Docs tab rather than in Books. This is due to
a shortcoming in Kindlegen (intentional or otherwise), which does not allow for
the addition of the EBOK metadata value that creates this distinction:

<meta
name="CDE Type" content="EBOK" />

This meta entry will be added when the file is uploaded to
KDP, so if you download that file and load it onto your Kindle it will appear
in the Books tab. But any other ebooks that you side-load are considered
personal documents by Kindle and will appear in the Docs tab instead.




As a side note, Calibre will also add the “CDE Type” metadata
entry when it converts files. Unfortunately, it cannot convert fixed layouts as
of yet. Also, if you do go ahead and enter this value in your OPF, Kindlegen
will not only ignore it, but will, in fact, actually delete the line!




Lastly, there are some important differences between the
older Kindles and the newer models that come into play here. While the older
models actually delete a file from the device when you tap on the book and hit
“Delete from Device,” the newer HD models and the 2nd generation Fire retain
the file on the device and only delete it from the library.




This has repercussions if you’re testing features that are
only activated on first opening the book, such as the start location. You will
also notice that when you drag a newer version of the file to the device in
Explorer or Finder that it gives you the Copy/Replace pop-up window.




So if you want to add a clean instance of your ebook to a
device you’ll first need to manually delete the old one from the Books folder
on the Kindle, along with two other items that you might find there, the first
being a .luci folder with the same name as your ebook, and the second a .embp
file that you’ll find in the Sidecars folder. These store location and other usage
data associated with that title.



While this is sometimes a nuisance, there is a
major benefit that has been added to the new devices that offsets it amply. And
this is that you no longer need to eject the device from your computer before
you can view your ebook on it, as you do with the older Kindle models. This
saves you a great deal of time when testing, as you’ll be transferring the file
many times and often.





Once you have an ebook file that is working correctly and
looks the way you want it to, upload it to KDP and then download the preview
file. You now have a fully functional Kindle fixed layout ebook that you are
free to do with as you like. Congratulations! Here’s to many more!