Need advice - Calibre for converting and managing fanfiction

[valiha]

I need some advice on how to use Calibre to convert and manage my vast fanfiction library. I have tried using this program in the past, installing and uninstalling several times. I cannot stand the fact that it creates a copy of every file I add to it. I understand the reasons for it now though I didn't in the past, but just knowing that it leaves a mess of folders and subfolders and files on my computer drives me batty. It says it is "designed around the concept of the logical book, i.e., a single entry in your library that may correspond to actual e-book files in several formats", but I only keep multiple formats for fanfic, so that seems reduntant to me. I have tried to find answers on the Calibre forum over at MobileRead, but most of the conversations are too technical for me. I hope some of you can at least give me some starting points.

So, I've been saving fanfiction for years, in various fandoms, because when I first started reading it, I didn't have internet at home, and net cafes charged ridiculous amounts for sitting there reading. I would bring a bunch of floppies and save favorites as txt files because they would take up the least amount of space and I could cram a lot of them on a single floppy. Those fics are still on my computer, in folders according to fandom and author. I don't bother with genres, because of the popularity of AUs in my main fandom. An author may write stories set in an Old West setting, WWII, present time, SF AU etc.; they all get saved in that author's folder. I number the series, and leave stand-alones with just the titles. I also don't bother with pairings because I'm mostly a gen reader.

I now save most of my favorites as htmls or download as epubs from AO3 to "My Fandoms > Fandom > Author" folders on my computer, and copy to my e-reader. I've been thinking about converting the old fics to htmls too, but it would be a time consuming process to do it one by one (I'm talking thousands of files, which often wrap after 65 characters). I know Calibre can do a batch convert of files from one format to another, but like I said I have issues with the way it behaves. I am wondering if I can set it up so that I convert the files I need, add tags and covers, move them to the location of my choice and then have Calibre delete the files and folders in its library created in the process.

Or if this is not possible with Calibre, could you recommend an alternative program that could do a batch convert from txt to clean, bare bones html/epub, and allow me to add tags to the resulting files? I'd really appreciate any help you can give me.

Comments

[jumpuphigh]

I am wondering if I can set it up so that I convert the files I need, add tags and covers, move them to the location of my choice and then have Calibre delete the files and folders in its library created in the process.

I don't see why you couldn't do that. You'd have to manually copy the converted files to the location that you wanted one-by-one though as Calibre sets up a new file for each book. Deleting from Calibre afterwards would be as easy as selecting the books and clicking the delete button in Calibre.

[valiha]

Would the "Send to disk" work for transferring the files to my reader? I haven't installed it yet, and would have to re-read the FAQ-s and help files. Otherwise I can use Toucan to synchronize the files.

One thing I remember about a book someone had converted for me via Calibre on the MobileRead forums was that it seems to add unnecessary styling to individual paragraphs - every

was styled and there were numerous ids in the css I thought shouldn't be there, all numbered successively. I had opened the epub to see how it was constructed, because I was curious about its structure. I don't understand why this is needed.

[jumpuphigh]

elf can talk better about the formatting of files than I can. I like how my epub files turn out from Calibre although I did go in and futz about with the default settings in order to make them work for me. Since I didn't know what I was doing, it was all trial and error.

Once you've started Calibre, plug in your device and an option will pop up to send files to the device itself in place of the heart on the toolbar.

[stormcloude]

There's also the option of "save single format to disk" which you can use to save all your selected items of one particular format to your hard drive.

I think you'd have to maunually add the tags and covers, unless you had one cover and set of tags that you wanted to bulk add to all of them.

[valiha]

I'd like to see some examples of naming templates for save to disk - there are some codes for that on the calibre manual pages, but they don't give you the result of those codes except in one case, I think (with Isaac Asimov's Foundation, I think).

About the cover, how can I replace the default calibre one? I was thinking of perhaps making a cover for each of my fandoms, but I'm not sure where I would need to make that change.

[baggyeyes]

To send to your ereader, there's a button 'Send to Device'. Just make sure to connect and turn on your ereader before opening Calibre. Calibre takes a moment to see the reader, but it isn't that long.

[nonniemous]

Calibre does better with RTF than TXT files, so I started saving all my fic that way and then converting it. I don't worry about stuff that I've converted, though; the epub files stay in my calibre folders and the fics stay in my fandom stuff, because sometimes I read on my computer and sometimes on my nook. Everything else just gets dumped into the calibre library. Older fics I've opened as whatever they were, resaved as rtf, and then pulled into Caliber for an epub conversion. It's worked pretty well--though not well enough that if the fic turns up on AO3 I don't run over and download their epub version to replace my homemade one.

The latest versions of Calibre are pretty spiffy, though not quite yet "Point and play" for formatting, but I seem to remember that you have some control over whether or not Calibre creates that second copy or where it puts it. It would definitely be worth trying again.

[valiha]

I have thousands of fics in several fandoms, some of them several years old, and not a single one of them rtf. I dislike the way Word scrolls. With txt I could resize the window and set the font so that it was nice and readable without giving me much eye strain, and with page down I'd scroll down a full "page". The trouble is that when I transfer those old fics to my reader, the lines are broken up which doesn't make for a smooth re-read (they wrap after 60 characters or so). It would be a really massive job re-saving all those fics as rtf.

"Everything else just gets dumped into the calibre library."

See, that's my problem. I simply cannot get over that. I get physically twitchy if the files on my computer are not arranged just so. Just the awareness that I have no control over that part of Calibre makes me break out in a cold sweat. It is completely irrational, but I cannot get over it. Not only am I a hoarder, but I'm an OCD hoarder.

[trouble]

One of my friends wrote a guide to using Calibre for a class she took. She's away right now so I'm not sure if she'll see this post, but when she gets back I'll ask her if she has any suggestions that would be helpful.

[elf]

This is why I don't use Calibre.

It's supposed to be a "black-box" arrangement--you set a folder marked CALIBRE DO NOT TOUCH, and don't look inside. Makes me twitchy, but I could handle that. But it doesn't do that--it makes ugly little author folders ALL OVER MY DOCUMENTS FOLDER AAARGH. (Can't stand iTunes either.)

I gather there's a way to change this, a way to make the imports all go into one folder (which I'd then remember to just never look inside), but I haven't teased out the settings for it. Plus the constant upgrades drive me bonkers; I regularly swap between 3 different computers and can't be bothered to keep up.

I want to learn how to make ePubs manually and design (or pester nice code-speaking friends to design) a simple program to batch-convert txt or rtf files.

There's Sigil, but I don't know if it does batch conversions.

[valiha]

Makes me twitchy, but I could handle that.
You have much better control than I do. I just looked inside - the author I imported has 40 stories, but some of the files were split into parts (story 01, story 02 etc) so there are 118 files in total. Naturally, because of the way calibre is set up, I now have extra 118 folders in there. :pulls hair:

Sigil is very easy to use if you have a good, clean starting file. You can either copy&paste the text into it, then use a button (2nd to last in middle row, looks like ch ligature) to divide the file into chapter, or import a file, edit things if you want/need, and save as epub. It does everything else for you. If you turn on split view, you can see the coding behind the epub. I have it open at the moment, and this is what you have in a blank epub:

<?xml version="1.0"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title></title> </head> <body> <p> </p> </body> </html>


It does have problems when you want to import large files (couple of MB). And no batch conversion, unfortunately.

[kaiz]

I really like Calibre a lot and use it to manage my fanfic collection to read on my Kindle. In fact, besides converted academic PDFs, fanfic is pretty much the only thing I read on my Kindle.

Like you, I had a big stash of old fanfic on my hard drive from decades gone by; mostly in HTML format, but some PDF, RTF, and TXT. I used Calibre to batch convert them all to MOBI for the Kindle. Calibre's batch metadata editing feature came in handy as well, since I batch converted old fic by fandom and then updated the tags in a big batch. (I agree with nonniemous that TXT isn't the best format to convert from, but it works well enough...you just lose some formatting like italics and bold.) I also like the Kindle collections plug-in that's available so I can group fanfic into specific fantoms and genres (e.g. Crossovers) on my Kindle so I don't have to sort though the fic alphabetically by title or author which is a huge win.

Anyway, yeah, you're right, Calibre maintains its library in directories/sub directories on a per author/story basis. But I guess this doesn't bother me because:
1) even binary format fic files are small in comparison to other media types (e.g. video files) and I have multiple huge hard drives so I won't ever run out of space, and
2) I keep backups of the old files and any new files in my original directory layout anyway. That way, if I ever stop using Calibre, I'll still have all my files stored in my original directory hierarchy and I'll be able to just delete the Calibre library. Additionally, since I use Calibre to upload stuff to my Kindle, the Calibre library and its files have to stay as they are for that functionality to work.

If you only want to use Calibre to convert files and not manage your e-library, then unless you have the skill to write a script in Python/Java/Perl/awk/whatever, I'm not sure there is any way to batch-copy the files out of the Calibre directory structure and move them to the directory you want.

Writing shell scripts isn't really difficult, though, so you might want to try that anyway just for fun. Simply type "writing shell scripts in [Python/Perl/bash/etc.]" into google and you'll find a lot of good suggestions and tutorials.

Two last points: Calibre does allow you to move your library to a different location than its default. And you can export subsets of your library collection to your hard drive in a user-defined directory hierarchy (which might be a way to do what you want without writing scripts). Information about those features and others that might be useful are located here and here


Hope this helps!

[valiha]

I on the other hand have such an intense dislike of this program. :( I don't usually feel this way about a program. :headdesk:

You just described the contents of my Pocketbook. :) (It's an Android tablet.)

(I agree with nonniemous that TXT isn't the best format to convert from, but it works well enough...you just lose some formatting like italics and bold.)

But txt has no formatting so I wouldn't be losing anything - many old fics I have denoted bold *like this* and italics _like this_, and used *** or --- and such as chapter breaks.

I have zero knowledge of those languages, and right now I have very little free time to try and learn them. I would eventually like to give it a hand, but right now I'd just like to convert the files I have and transfer them to my reader. I will have to spend a couple of days re-acquainting myself with the program before I go on to scripts and regexes. Some members of MobileRead took the time to explain them to me the last time I seriously tried to use the program.

[kaiz]

I on the other hand have such an intense dislike of this program.

Hee, I totally hear you. My antipathy toward Microsoft Office knows no bounds! Also, it just occurred to me...if iTunes drives you (the generic you) nuts, then there's a fair bet Calibre will too, since they've got such a similar user interface and directory layout for files.

I would eventually like to give it a hand, but right now I'd just like to convert the files I have and transfer them to my reader.

Maybe give Calibre's "export library subsets" feature a shot, then; that might work for you. I just tried it out on my library and I was able to save a selection of titles (that I selected using the mouse it the user interface) to an arbitrary directory on my computer.

So...

If you were to use: Add books -> Add books from a single directory to import a bunch of your fic files.

Followed up by: Convert books -> Bulk convert and Save to disk -> Save to disk in a single directory that might just do the trick! Then, you could point your ereader at the directory you've chosen and upload those converted files. Worth a shot at least.

I'm no expert with the program, but I'm happy to try things out on my own library if you have questions and would rather not dive into the Mobileforums groups!

[kaiz]

But txt has no formatting so I wouldn't be losing anything

Oh yeah, but what I meant was that it doesn't seem to convert asterisks like *text* to actual italics or bold during the conversion process (at least not in MOBI); so you lose that aspect of formatting even those the *text* remains in the converted file. Maybe EPUB translates that correctly?

[baggyeyes]

When I tried loading plain old txt on Kobo, it turned out like a massive wall of text. Maybe that was a fluke for me, and it might be better for you. But that is the reason I convert to rich text, then load that. Check your reader's capabilities. It's possible you don't even need to convert anything.

[valiha]

I've been tweaking the settings for conversion to txt and the results are now better, though still not perfect. A weird thing I noticed is that all quotes and apostrophes are shown as a square character in the internal viewer. They show up fine in FB reader on my computer, and I'm about to transfer them to my reader to see how they look in CoolReader.

[nonniemous]

You could also try Sigil; there's a lot more user control as to where your files go and how they're stored, and a good tutorial for conversion here: http://code.google.com/p/sigil/wiki/BasicTutorial

I just used it to combine a series of shorts from one of my favorite AUs into one epub book for my nook.

[valiha]

I'm used to Sigil and use it often (a bit of trivia: Valloric and I are from the same ex-state and speak practically the same language, and have on one occasion lamented the difficulties in getting Central European diacritics to show properly in epub), and once I do my conversion it's quite possible that I will need to fix some of the fanfics.

The problem is the sheer number of fics I have. If I had the time, I could tinker with them indefinitely; unfortunately, I don't so I'll have to use Calibre. But thank you for the reminder, I too might combine some drabbles that way.

[nonniemous]

Yeah, the number does get daunting. I've been slowly converting things over, and I nearly danced in glee when I found ereader versions of just a few favorite fics from decades back. Sigil honestly scares me, and I'm just starting to play around with it. But as I work my way back into the older files, I may very well have to get more familiar with it.

[elf]

Is this where I rant about copyright laws that prevent/discourage a useful thing like "place archives of fanfic online in various formats and people can throw time to convert a few at a time to ebook formats, like Project Gutenberg's distributed proofreaders only for fanfic?"

Because gah. Would love to have groups of fans working on archiving fic & updating to modern formats.

[amalthia]

I don't think you're the first person that's tried to find another program that could batch convert due to how Calibre organizes it's files. So far as I know Calibre is the only program that can do that.

I use two separate folders. One that's the Calibre Library (black box) and the other is my Sony Reader folder. The Sony Reader folder is where I save my converted epub files, my working HTML files (what I use to upload to Calibre for conversion).

I haven't looked in my Calibre Library folder in months. I have complete control over my Sony Reader folder which does give me some measure of peace and it helps me ignore the Calibre Library folder.

I'm not sure if the two folder method would help you while you're converting stories? But I know for myself it's made a big difference. (that and hiding the Calibry Library somewhere I wouldn't run across it on accident...)

I'm also particular about how my files are organized so I can see how difficult the calibre library folder system can be difficult to deal with.

[valiha]

No, I'm not, judging by the number of users who complain about its file structure on MobileRead. You might be able to ignore calibre's multiple subfolders, I am not; I tried repeatedly.

After the conversion, do you transfer the fics by hand or do you use the Send to disc option? If so, do you use any naming templates? I have some AU fics that have a fairly complicated naming scheme.

Right now, converted fics end up together with the imported fics, inside their subfolders. After I transfer them, I can use Remove selected books to delete those folders. Would that still be true if I had a separate folder for converted fics?

[amalthia]

I use the "Save to Disk" option to save the stories to another folder on my computer.

I'm a bit confused by your question, "Would that still be true if I had a separate folder for converted fics?" If you delete the file from Calibre all conversions and the file you imported to calibre are deleted and sent to the recycle bin. Your originals (not converted file) and anything you saved to disk should be okay, assuming they have their own folder not related to or connected to the calibre library.

If you delete it from the Calibre library it's gone. You may want to go to the Calibre forums to get more information on how deletions work or read the manual? Not long ago a guy accidentally deleted his entire desktop folder using Calibre. So on mass deletions I recommend caution.

Your converted fics are supposed to be in the same folders as your imported fics because Calibre creates a folder when an item is imported. Then it saves all conversions of that file into the same folder. It's actually a rather neat piece of programing. It's probably not as useful if you're only making one conversion but I'm making 3 files types for each story I import into Calibre. However, I don't actually want two copies of the mobi or LRF on my computer so I only save the epub to a completely separate folder not anywhere near my calibre library.

I don't delete anything from Calibre because I don't fancy re-converting if something happened to the epubs/lrf/mobi versions of the stories I've saved to my computer. Plus now that I can 'tweak' the epubs I've found it much faster to edit ebooks on Calibre than in any other program. So for me it manages my fan fiction and pro ebook collection.

Now as for hiding your Calibre Library have you tried adding it about 7 directories deep on your computer??? Because I think if you moved that library really deep you'll never run across it by accident and you may actually forget where it's at if you find a weird enough place to hide it. You may even want to add it to hidden directory? Then re-hide your folders. Then only use Calibre program to access the files in that folder.

I mean do you really know what's happening in your Windows folder or what happens when you install new programs? You don't organize your operating system files or your program directory files. There are hundreds of folders that the operating system creates that you don't organize yourself because if you did you'd break your computer. It may not hurt to try thinking of the calibre library directory as an operating system folder and treat it as such.

I think that's why I'm able to handle the fact that there is a folder on my computer that I don't touch beyond creating a backup periodically. I don't touch many folders on my system because if I go in there and move things around it'll fuck up my operating system. I don't mess with my System32 folder, I rarely look in my Program Files folder, and I sure as hell don't mess around with my drivers folders. You have thousands of folders on your computer that you ignore every time you use your computer. The Calibre Library is just one more of those types of folders.

I know with Windows 7 you can hide folders on your computer and pretty much turn them invisible. This may not be a bad option to explore if you have that capability. At least until you're done using Calibre to make your conversions.

Calibre FanfictionDownloader Plugin

[(Anonymous)]

Check out this new modification to the FanFictionDownloader python scripts.

http://code.google.com/p/fanficdownloader/wiki/FanFictionDownloaderCalibrePlugin