Need advice - Calibre for converting and managing fanfiction

[valiha]

I need some advice on how to use Calibre to convert and manage my vast fanfiction library. I have tried using this program in the past, installing and uninstalling several times. I cannot stand the fact that it creates a copy of every file I add to it. I understand the reasons for it now though I didn't in the past, but just knowing that it leaves a mess of folders and subfolders and files on my computer drives me batty. It says it is "designed around the concept of the logical book, i.e., a single entry in your library that may correspond to actual e-book files in several formats", but I only keep multiple formats for fanfic, so that seems reduntant to me. I have tried to find answers on the Calibre forum over at MobileRead, but most of the conversations are too technical for me. I hope some of you can at least give me some starting points.

So, I've been saving fanfiction for years, in various fandoms, because when I first started reading it, I didn't have internet at home, and net cafes charged ridiculous amounts for sitting there reading. I would bring a bunch of floppies and save favorites as txt files because they would take up the least amount of space and I could cram a lot of them on a single floppy. Those fics are still on my computer, in folders according to fandom and author. I don't bother with genres, because of the popularity of AUs in my main fandom. An author may write stories set in an Old West setting, WWII, present time, SF AU etc.; they all get saved in that author's folder. I number the series, and leave stand-alones with just the titles. I also don't bother with pairings because I'm mostly a gen reader.

I now save most of my favorites as htmls or download as epubs from AO3 to "My Fandoms > Fandom > Author" folders on my computer, and copy to my e-reader. I've been thinking about converting the old fics to htmls too, but it would be a time consuming process to do it one by one (I'm talking thousands of files, which often wrap after 65 characters). I know Calibre can do a batch convert of files from one format to another, but like I said I have issues with the way it behaves. I am wondering if I can set it up so that I convert the files I need, add tags and covers, move them to the location of my choice and then have Calibre delete the files and folders in its library created in the process.

Or if this is not possible with Calibre, could you recommend an alternative program that could do a batch convert from txt to clean, bare bones html/epub, and allow me to add tags to the resulting files? I'd really appreciate any help you can give me.

Comments

[jumpuphigh]

I am wondering if I can set it up so that I convert the files I need, add tags and covers, move them to the location of my choice and then have Calibre delete the files and folders in its library created in the process.

I don't see why you couldn't do that. You'd have to manually copy the converted files to the location that you wanted one-by-one though as Calibre sets up a new file for each book. Deleting from Calibre afterwards would be as easy as selecting the books and clicking the delete button in Calibre.

[valiha]

Would the "Send to disk" work for transferring the files to my reader? I haven't installed it yet, and would have to re-read the FAQ-s and help files. Otherwise I can use Toucan to synchronize the files.

One thing I remember about a book someone had converted for me via Calibre on the MobileRead forums was that it seems to add unnecessary styling to individual paragraphs - every

was styled and there were numerous ids in the css I thought shouldn't be there, all numbered successively. I had opened the epub to see how it was constructed, because I was curious about its structure. I don't understand why this is needed.

[jumpuphigh]

elf can talk better about the formatting of files than I can. I like how my epub files turn out from Calibre although I did go in and futz about with the default settings in order to make them work for me. Since I didn't know what I was doing, it was all trial and error.

Once you've started Calibre, plug in your device and an option will pop up to send files to the device itself in place of the heart on the toolbar.

[nonniemous]

Calibre does better with RTF than TXT files, so I started saving all my fic that way and then converting it. I don't worry about stuff that I've converted, though; the epub files stay in my calibre folders and the fics stay in my fandom stuff, because sometimes I read on my computer and sometimes on my nook. Everything else just gets dumped into the calibre library. Older fics I've opened as whatever they were, resaved as rtf, and then pulled into Caliber for an epub conversion. It's worked pretty well--though not well enough that if the fic turns up on AO3 I don't run over and download their epub version to replace my homemade one.

The latest versions of Calibre are pretty spiffy, though not quite yet "Point and play" for formatting, but I seem to remember that you have some control over whether or not Calibre creates that second copy or where it puts it. It would definitely be worth trying again.

[valiha]

I have thousands of fics in several fandoms, some of them several years old, and not a single one of them rtf. I dislike the way Word scrolls. With txt I could resize the window and set the font so that it was nice and readable without giving me much eye strain, and with page down I'd scroll down a full "page". The trouble is that when I transfer those old fics to my reader, the lines are broken up which doesn't make for a smooth re-read (they wrap after 60 characters or so). It would be a really massive job re-saving all those fics as rtf.

"Everything else just gets dumped into the calibre library."

See, that's my problem. I simply cannot get over that. I get physically twitchy if the files on my computer are not arranged just so. Just the awareness that I have no control over that part of Calibre makes me break out in a cold sweat. It is completely irrational, but I cannot get over it. Not only am I a hoarder, but I'm an OCD hoarder.

[trouble]

One of my friends wrote a guide to using Calibre for a class she took. She's away right now so I'm not sure if she'll see this post, but when she gets back I'll ask her if she has any suggestions that would be helpful.

[kaiz]

I really like Calibre a lot and use it to manage my fanfic collection to read on my Kindle. In fact, besides converted academic PDFs, fanfic is pretty much the only thing I read on my Kindle.

Like you, I had a big stash of old fanfic on my hard drive from decades gone by; mostly in HTML format, but some PDF, RTF, and TXT. I used Calibre to batch convert them all to MOBI for the Kindle. Calibre's batch metadata editing feature came in handy as well, since I batch converted old fic by fandom and then updated the tags in a big batch. (I agree with nonniemous that TXT isn't the best format to convert from, but it works well enough...you just lose some formatting like italics and bold.) I also like the Kindle collections plug-in that's available so I can group fanfic into specific fantoms and genres (e.g. Crossovers) on my Kindle so I don't have to sort though the fic alphabetically by title or author which is a huge win.

Anyway, yeah, you're right, Calibre maintains its library in directories/sub directories on a per author/story basis. But I guess this doesn't bother me because:
1) even binary format fic files are small in comparison to other media types (e.g. video files) and I have multiple huge hard drives so I won't ever run out of space, and
2) I keep backups of the old files and any new files in my original directory layout anyway. That way, if I ever stop using Calibre, I'll still have all my files stored in my original directory hierarchy and I'll be able to just delete the Calibre library. Additionally, since I use Calibre to upload stuff to my Kindle, the Calibre library and its files have to stay as they are for that functionality to work.

If you only want to use Calibre to convert files and not manage your e-library, then unless you have the skill to write a script in Python/Java/Perl/awk/whatever, I'm not sure there is any way to batch-copy the files out of the Calibre directory structure and move them to the directory you want.

Writing shell scripts isn't really difficult, though, so you might want to try that anyway just for fun. Simply type "writing shell scripts in [Python/Perl/bash/etc.]" into google and you'll find a lot of good suggestions and tutorials.

Two last points: Calibre does allow you to move your library to a different location than its default. And you can export subsets of your library collection to your hard drive in a user-defined directory hierarchy (which might be a way to do what you want without writing scripts). Information about those features and others that might be useful are located here and here


Hope this helps!

[valiha]

I on the other hand have such an intense dislike of this program. :( I don't usually feel this way about a program. :headdesk:

You just described the contents of my Pocketbook. :) (It's an Android tablet.)

(I agree with nonniemous that TXT isn't the best format to convert from, but it works well enough...you just lose some formatting like italics and bold.)

But txt has no formatting so I wouldn't be losing anything - many old fics I have denoted bold *like this* and italics _like this_, and used *** or --- and such as chapter breaks.

I have zero knowledge of those languages, and right now I have very little free time to try and learn them. I would eventually like to give it a hand, but right now I'd just like to convert the files I have and transfer them to my reader. I will have to spend a couple of days re-acquainting myself with the program before I go on to scripts and regexes. Some members of MobileRead took the time to explain them to me the last time I seriously tried to use the program.

[kaiz]

I on the other hand have such an intense dislike of this program.

Hee, I totally hear you. My antipathy toward Microsoft Office knows no bounds! Also, it just occurred to me...if iTunes drives you (the generic you) nuts, then there's a fair bet Calibre will too, since they've got such a similar user interface and directory layout for files.

I would eventually like to give it a hand, but right now I'd just like to convert the files I have and transfer them to my reader.

Maybe give Calibre's "export library subsets" feature a shot, then; that might work for you. I just tried it out on my library and I was able to save a selection of titles (that I selected using the mouse it the user interface) to an arbitrary directory on my computer.

So...

If you were to use: Add books -> Add books from a single directory to import a bunch of your fic files.

Followed up by: Convert books -> Bulk convert and Save to disk -> Save to disk in a single directory that might just do the trick! Then, you could point your ereader at the directory you've chosen and upload those converted files. Worth a shot at least.

I'm no expert with the program, but I'm happy to try things out on my own library if you have questions and would rather not dive into the Mobileforums groups!

[valiha]

OK, I could try it with an author who has written about 40 fics and see how it works out.

Remind me, once those books are imported, Calibre places them into subfolders in its library - one subfolder per book, so I'd get an author folder and 40 subfolders, correct? :shudders: Would I then be able to remove/delete those books/folders from Calibre's GUI because I wouldn't need them any more? Because that's what drives me crazy with Calibre, it leaving duplicates all over the place.

[kaiz]

I'd get an author folder and 40 subfolders, correct?

Yes, that's correct. *g*

Would I then be able to remove/delete those books/folders from Calibre's GUI because I wouldn't need them any more?

I don't think that you can delete the book listings from the user interface without also deleting them from Calibre's folder/subfolder hierarchy. From what I can tell, Calibre reads the contents of its directory structure to build the book listings in the user interface.

However, you can easily delete bunches of books from the library, and Calibre gives you a bunch of options for doing so, including:

Remove selected books
Remove files of a specific format
Remove all formats from selected books except…
Remove all formats from selected books
Remove covers from all selected books
Remove books from selected device

So you can select the books you want to delete by clicking on them in the user interface, convert them in a big batch, then export the converted files to your chosen directory, then choose one of those deletion options to get rid of the book files.

[kaiz]

But txt has no formatting so I wouldn't be losing anything

Oh yeah, but what I meant was that it doesn't seem to convert asterisks like *text* to actual italics or bold during the conversion process (at least not in MOBI); so you lose that aspect of formatting even those the *text* remains in the converted file. Maybe EPUB translates that correctly?

[valiha]

Oh, got that, thanks. I have no idea if the result is different with epub, I am now downloading the latest version of Calibre and I'll find out once I try it.

[valiha]

I don't think that you can delete the book listings from the user interface without also deleting them from Calibre's folder/subfolder hierarchy.

Yes! That's what I want. I want the subfolders gone from my computer once I'm done with my conversion.

[kaiz]

Okay, just tried this out. Using the Remove selected books option will also delete the directory from your computer. So win! :-)

[valiha]

Thank you! I just installed the program, and will be starting my own experiment in a few minutes. I didn't use the Advanced option; should I have? But I didn't feel confident doing that. In any case, Calibre is now installed, I made a folder for it, and I'll try to get the same results you did.

[kaiz]

I'm pretty sure that I used the basic install options, so I think you'll be fine. Good luck on the experiments. I'll be around off and on all day so if you run into something weird, I can test things out on my library, too, and eventually I think we can figure it out so you get some useful results! :-)

[nonniemous]

You could also try Sigil; there's a lot more user control as to where your files go and how they're stored, and a good tutorial for conversion here: http://code.google.com/p/sigil/wiki/BasicTutorial

I just used it to combine a series of shorts from one of my favorite AUs into one epub book for my nook.

[valiha]

I'm used to Sigil and use it often (a bit of trivia: Valloric and I are from the same ex-state and speak practically the same language, and have on one occasion lamented the difficulties in getting Central European diacritics to show properly in epub), and once I do my conversion it's quite possible that I will need to fix some of the fanfics.

The problem is the sheer number of fics I have. If I had the time, I could tinker with them indefinitely; unfortunately, I don't so I'll have to use Calibre. But thank you for the reminder, I too might combine some drabbles that way.

[nonniemous]

Yeah, the number does get daunting. I've been slowly converting things over, and I nearly danced in glee when I found ereader versions of just a few favorite fics from decades back. Sigil honestly scares me, and I'm just starting to play around with it. But as I work my way back into the older files, I may very well have to get more familiar with it.

[valiha]

Well, I've imported the fics, and had to rename some of them because calibre didn't recognize the names. Doesn't it take the titles as names of books?

Oh and a complaint: when I first started it, the interface was a strange mix of one of our local languages and English even though I chose English during setup. I had to choose English (UK) and restart to get a clean English menu. ???

[elf]

This is why I don't use Calibre.

It's supposed to be a "black-box" arrangement--you set a folder marked CALIBRE DO NOT TOUCH, and don't look inside. Makes me twitchy, but I could handle that. But it doesn't do that--it makes ugly little author folders ALL OVER MY DOCUMENTS FOLDER AAARGH. (Can't stand iTunes either.)

I gather there's a way to change this, a way to make the imports all go into one folder (which I'd then remember to just never look inside), but I haven't teased out the settings for it. Plus the constant upgrades drive me bonkers; I regularly swap between 3 different computers and can't be bothered to keep up.

I want to learn how to make ePubs manually and design (or pester nice code-speaking friends to design) a simple program to batch-convert txt or rtf files.

There's Sigil, but I don't know if it does batch conversions.

[kaiz]

I'm pretty sure Calibre guesses at the title of the book based upon the name of the file. So if you've got file named "Fic1001020.txt" you're going to end up with a title of either "Fic1001020", or for PDF files or cases where I have a filename with odd characters I noticed that I sometimes get a title that is the first sentence in the document.

the interface was a strange mix of one of our local languages and English

You might want to send them a bug report on that. I didn't experience that problem with the Mac version 0.8.8; if you're using the Windows version, perhaps it's a bug that is Windows only.

[elf]

It does not convert *text* and _text_ to text and text; I don't think any program does that with imported text. (Word does it while you're typing but I haven't found a way to make it do so to copy/pasted text.)

[valiha]

The epubs look horrible - everything is centered, the wrapping wasn't removed, there's an extra line because of it (wrpping was probably achieved by hard break) and quotation marks are converted into squares. :headdesk: I'll have to remove them and try heuristic conversion to see if the results will be any better.