ebooks | Another calibre question

Current Mood: frustrated

Entry tags:

calibre

Another calibre question

HI, I have a calibre conversion question fandom people might understand better than MobileRead forumers.

I had the fanfiction downloader program called Graffer installed on my comp. I was happy with it because it would download fics from multiple sources and produced clean html files. Unfortunately it's creator gave up on the program and it is no longer updated, and it has stopped working for several major archives.

My work on converting my fics via calibre is slow, ad I often want to chuck my comp out the window, that's how frustrated calibre makes me feel. I was expecting import and conversion to be fairly straightforward, but calibre kept messing up the authors. I finally loked at the html coding and discovered that Graffer added a line in the metadata section which calibre would read as author name, but was actually the name of the programmer: <meta name='author' content='Grzegorz Hordynski' />

I wet to MobileRead to see if I can find a way to change this line in the html file automatically through bulk convert, but couldn't figure out the instructions or the regexes. I've been having a back and forth conversation with a member who doesn't understand what I'm after, so does anyone here know how to set up calibre so that it changes the programmer's name into the correct author name for selected ebooks in the actual html file, not just in the metadata?

Flat | Top-Level Comments Only

I can't help with the older files, but I have a suggestion which might be useful for future downloads! I just started using this fanfiction downloaded which is super awesome. It's called flag

http://www.flagfic.com/

It supports

fanfiction.net
fictionpress.com
twilighted.net
adifferentforest.com
thewriterscoffeeshop.com
twiwrite.net
ficwad.com
adultfanfiction.net
fictionalley.org
harrypotterfanfiction.com
mediaminer.org

And will soon support

tthfanfic.org
wraithbait.com
hpfandom.net
archive.skyehawke.com
archiveofourown.org

And allows downloads in
EPUB
MobiPocket
PDF
HTML

Edited 2011-10-05 20:39 (UTC)

Hi, I actually started using it couple of weeks ago in addition to the Fanfiction Downloader program when it there's an archive it doesn't support yet, and I think it's great. (Haven't looked under the hood of the files it downloads to see whether it adds metadata such as Graffer did, but I will).

The problem is I have a massive amount of fics I saved from LJ and DW as single html or as full we pages for fics posted for BigBangs and fest exchanges because they had accompanying artwork. I don't have the time to edit all those files individually, and was hoping to do it with calibre.

I wish I totally had a solution, because I have a ton of HTML files too! I hope someone else knows, because it'd be handy information to have. :)

I have never been so frustrated by a program in my entire life. (ˇ_ˇ)

Oh, and archiveofourown has its own downloader, so I use that for fics in my fandom that are posted there. Most of my main fandom is still based on LJ, so unless there's a downloader built specifically for it, it won't be of much help. :(

Yeah, I actually have no idea why he's building a download for AO3. Maybe he likes building downloads? That's my only guess!

I'm building a downloader for AO3 because my users keep asking for one - I'm aware that AO3 already has its own, but either people are having trouble finding it, or they don't like it, I'm not sure which.

I don't claim to understand the strange desires of FLAG's users, but I do try to keep them happy :-).

Haha, okay! Honestly, I can understand users having trouble finding AO3's downloader. I took me about three months of using the site to realize that it had one. XD

"I don't claim to understand the strange desires of FLAG's users, but I do try to keep them happy :-)."

Sounds like a plan. A psychological approach would probably only yield further unanswerable questions. :)

"A psychological approach would probably only yield further unanswerable questions."

That was my thinking - happy users that confuse me are infinitely better than annoyed users who feel over-analysed!

Try Squeebook for LJ. http://www.squeebook.net/ I haven't tested it out myself, but I hear excellent things about it for LJ/DW fanfics.

I have used squeebook before, and yes, it's great. But it is only useful for the times you have a couple of fics to convert. It would take a massive amount of time ad work to get all my old files copy/pasted. Some of them are over five years old, and some authors have been really prolific, with 40-50 fics in a single fandom. Others are fics written years ago, just as numerous, but recently downloaded, with the problem of additional metadata inserted by the downloader.

Ooh, that will work for me! I will try that. :D

If you can point me to a place to download the program (or if you can get it to me), I can take a quick look and let you know if it's possible to fix that. Also, links to the MR discussion(s) would help shorten that process.

calibre? Or Graffer? There is no longer a website for Graffer, and I didn't keep the exe file after installing the program. calibre can be found at http://calibre-ebook.com/

But you probably wouldn't need Graffer anyway; this is the coding it inserts into the downloaded html file, and the thing I'm trying to replace is the programmer's name with the name of the author:

The MobleRead thread in question is Importing and converting html files.

Ok, I think I actually misunderstood you originally, since I just skimmed the post. I thought you were going to continue to use Graffer, but wanted to fix it so it didn't put the program author's name in there. I've reread it, and what you're trying to do is fix the files well after the fact, using Calibre (you hope). Some questions for you:

1) Do all of the html files you're trying to convert have the author information in them somewhere? In the text, as part of the filename, somewhere?

2) Is the location of the author's name consistent for all of the files?

3) Is it the information actually marked as author, or is it just in the text somewhere, like the example you gave above?

Edited (clarification) 2011-10-06 01:19 (UTC)

No, I'm switching to Fanfiction Downloader, FLAG ans squeeBook depending on what my source is.

OK, here are the answers:

1) I name my files differently depending on whether it's just a single story from an author I liked and saved, in which case I put the name of the author in the file name (Author - Name of fic.html). If I have several fics from the same author, I create a folder for that author and keep all their fics there. In that case, I don't include their name in the file (Name of fic.html, or if it's a series, Series Name 01 Name of fic.html etc). There's usually boththe author name and fic title in the file itself, before the text. i.e. Fic name by Author, often followed by additional information about the fic itself (disclaimers, warning and such - the usual descriptions authors post with their stories).

2) I mostly use the Author name - Fic title format in naming my files, especialy those I saved as html, web page, or copy/pasted from LJ or whatever other source, though there were several that were named Fic name by Author before I changed the configuration of Fanfic Downloader. I still need to rename the rest of those.

3) I'm unclear on what you mean by this? Marked, as in marked in the html coding, or marked as in, Author: So and so, Title: Such and such? in the fic header?

I was hoping all this renaming and correcting would be something I would be able to do automatically and in bulk via calibre because I really don't have the time to do it manually. I wanted to convert files of various formats to a single format, and have them contain metadata about the files, maybe even a cover for each fandom that would show up on my reader. But the more I use calibre, the more I hate it. I dislike it's duplication of imported files, its several levels deep subfolders. Now it seems that files have to be perfect before they can be imported and converted.

Let me know if there are other things I need to clarify.

Just trying to clarify what you're trying to do. There are usually better tools that do just one job than expecting one program to do them. :) (I don't use calibre, so can't really comment on what it will or won't do well.)

To remove the programmer's name from the Graffer-grabbed files: grab a good text editor (like NoteTab) and use it to do a multi-file search and replace on the meta tag. That'll clear that out so you don't have to deal with it when you find a library program you do like to use and want to import your files.

To mass rename batch files without having to figure out the regexes, use AF5 Rename your files. It's a fairly logical interface for doing just one thing: renaming files using patterns. :)

(Oh hi Keen Eddie! <3) Yes, I'm realizing that. :( There have been some breakthroughs on the MobileRead forum, so hopefully I'll solve the calibre related issues there.

I just recently discovered that I can open several files in Notepad++ ad do a search and replace. I was hoping search&replace would be something I could do on a larger number of files without actually having to open them (hence thinking I could use calibre for that), but it seems I won't be able to do that (not with my level of technical knowledge).

I've got two free renaming programs I installed (um, Ant Renamer I think, and Bulk Rename Utility), and so far I've figured out how to prefix the name of the author folder in which I keep my fics written by that one author; what I can't figure out is how to rename a files like

Author - 01 Title.html
Author - 02 Title.html
Author - 03 Title.html

to

Author - 01 - Title.html
Author - 02 - Title.html
Author - 03 - Title.html

Do you see what I mean? How do I tell the renamer to add a dash and a space without touching the number?

I ♥ Keen Eddie (Obviously, since Eddie's been my default icon for eight years. *g*)

It's pretty easy to open multiple files at once in NoteTab and do the search and replace. Depending on how old or new your machine is, you can open files in bunches so it doesn't overwhelm computer's memory, but really, that's the fastest way to do that sort of thing without having to learn or deal with any of the scripting languages that would do what you need. And honestly, even though you have to open all the files, it doesn't take any longer. :) Plus, it sounds like it's much more your speed, and would cause the least amount of headache for you. :)

With the file renaming, sure, I know exactly what you mean. Download the program I gave you the link to (AF5 Rename your files) and read the help files--it easily does that sort of thing without making it complicated. (I've used that program for years--it's a very straightforward tool.)

Edited 2011-10-08 08:09 (UTC)

My system lacks the necessary files to open its help. Stupid Windows. Is there web based help anywhere, with examples? I basically need to see how people rename files, what strings and expressions they use - I am very much a "follow the example" type of learner.

There seems to be a download for Graffer on Softpedia.

OK, this is about html files you have already downloaded, right? I don't know if there's a way to do this in Calibre, but it would only be a few lines of Python script or Shell script to go through all your files and remove that line.

What Operating System are you on? Dou you already have Python installed, by any chance?

Yes, it's files I have already downloaded and saved in the appropriate fandom folder.

I'm on Windows 7, and I have zero knowledge of Python and Shell script. I am under such tight schedule at work that I have no time to learn either script, and any time I can set aside for reading is precious. I am only doing this because my reader would present me with wrong author information when I transferred the files. I wanted to know what was causing it, and looked at the html code. It then kind of blew up from there. :)

Darn, posting html quote is messing up my formatting. Trying again...

If you don't find another solution to your problem and if you are willing to install Python, I'd write that script for you.

OTOH, I just looked at the bulk conversion thing of Calibre, and the Search & Replace looks exactly like what you need. Your regular expression would most likely just be the offensive tag as is if it stays exactly the same in all files:

<meta name='author' content='Grzegorz Hordynski' />

Leave the replacement string empty. If that doesn't work, try

<meta name='author' content='Grzegorz Hordynski' \/>

(With a backslash before the slash to tell calibre that you mean the character slash. The slash might have a special meaning in regular expressions which you don't want here.)

Thing is, there are several locations of Search&Replace: there's one you get when you mark several books and choose Metadata > Edit metadata in bulk, then there's one in Preferences, Common options, Search&Replace, then there's one under Convert books > Bulk convert, Search & Replace. Which one am I supposed to use? I tried them all, and none seem to work. They all leave everything exactly the way it is. (And I want to replace Grzegorz Hordynski with the correct author name, not just remove it from the file. But let me try using your edit first.)

:back a while later:

It populates the Author(s) column correctly, but it does nothing to the file itself. When I look into the zip file created by calibre, that bloody line is still there. It is supposed to be able to change the content of the file. I know, because I read tutorials on MR that show how to remove repeating headers and footers. Why am I not able to change this one bloody line in the html file???

And it doesn't convert all books I selected, just the first one, and renames the fic in an insane fashion - I start with a file in an author folder numbered by the order in which events happen in these six connected fics, and calibre rearranges the order the way it sees fit. When I convert the fic, it renames it to Title by Author - Author.epub. WTF is that?

Just to make my workflow a little clearer, here's the setup of my fandom folders (I'll use Harry Potter as the example):

My fandoms > Harry Potter > Fics > Individual authors > Author folder > Title. html (or number Title.html if they make up a series)

My fandoms > Harry Potter > Fics > Various authors > Author name - Title.html

So I imported the books from the very first author folder, I clicked on Convert books, Bulk convert, went to Search & Replace, entered the offending line, entered text I wanted to replace it with, hit OK and the conversion was done only on first book. The rest were disregarded, and the resulting epub was named Title by Author - Author.epub (Title by Author is the first line of the fic, fllowed by a short synopsis and story word count, and the original link. The fic was downloaded from fanfiction.net)

Yes, I meant the bulk import -> search and replace. But I've never bulk imported myself, so I can't really help you if it doesn't work.

Re the "Title by author - Author.epub" -- is maybe the title tag in the meta info of the html set to "Title by author"? You might want to try removing that with the bulk import search and replace, too. It's going to fallback on filename for title, then, I think. If you convert one author folder at a time, you can just edit the author back in via bulk editing of meta tags.

No, look, I have to step outside for a while, I have an appointment to go to and will be back in an hour, and I'll try to explain better, but here's the html code for the beginning of the file so you can see what it looks like:

and from the file when opened normally:

Darling by agelade
A canon-compliant retelling of book 7, mostly from Draco's point of view. There's a good reason the Malfoys weren't immediately arrested after V was defeated, and it's not because his parents suddenly got lovey-dovey. Canon compliant, canon ships.
Story word count: 33355
Original link

Yes, that's what I meant: the title tag is set to "Darling by agelade" (right below the offending author name.) So Calibre, quite rightfully, assumes that that's the title.

The regular expression to get rid of the whole title tag would be

<title>.*?</title>

I don't want to get rid of it, I want to replace the content inside the title tag. How do I do that? No, that wouldn't work, right? Because I want to do a bulk edit, and each story of course has a different title.

So tell me at least how do I replace the author tag with the correct author name? That at least should be the same for each selected story.

And how do I get it to take into account the correct order of the series? What is the naming convention I have to apply to the original file so that it is correctly recognized upon import?

OK, no, it did convert all books, but my God, the names! And why doesn't it take into account how the series was numbered?

So let's go with actual examples:

I start with Harry Potter > Fics > Individual authors >agelade

The stories are:

01 Darling.htm
02 Daring Coda The Same.htm
03 Darling Coda The Portrait.htm
04 Darling Coda 12th August, 2004.htm
05 Darling Coda The Notion of Bluffing.htm
06 Darling Coda Stars

Upon importing end conversion, I end up with

Calibre Library > agelade > Darling by agelade (272) > Darling by agelade - agelade.epub
Calibre Library > agelade > Darling Coda_ 12th August, 2004 by agela (275) > Darling Coda_ 12th August, 2004 by agela - agelade.epub
Calibre Library > agelade > Darling Coda_ Stars by agelade (277) > Darling Coda_ Stars by agelade - agelade.epub
Calibre Library > agelade > Darling Coda_ The Notion of Bluffing by (276) > Darling Coda_ The Notion of Bluffing by - agelade.epub
Calibre Library > agelade > Darling Coda_ The Portrait by agelade (274) > Darling Coda_ The Portrait by agelade - agelade.epub
Calibre Library > agelade > Darling Coda_ The Same by agelade (273) > Darling Coda_ The Same by agelade - agelade.epub

So I have to rename all my fics BEFORE importing to calibre following a certain scheme (which is what, exactly? How do I organize my files so that calibre populates authors and series in a correct way? And that is doubling the work; it was supposed to automate things for me, not make me work twice as hard to fix it before conversion. What if I had imported my entire Fandoms flder???)

Amazing write-up! This could aid plenty of people find out more about this particular issue. Are you keen to integrate video clips coupled with these? It would absolutely help out. Your conclusion was spot on and thanks to you; I probably won’t have to describe everything to my pals. I can simply direct them here!

I hate this program so much. My latest post on the MB thread, after trying out their advice to change the data and connecting my device to see how my app would read the converted epub:

OH MY GOD, it just deleted everything on my SD card!!!

I had three fandoms saved there in my fiction folder, and notes from my postgraduate lectures, word documents and powerpoint presentations - you've got to be kidding me!!

I clicked Save to disk, Save only epub format to disk, it recognized Pocketbook's internal card, but when I clicked on Removable disk (G it asked if I wanted to format the card. Of course I don't, I have important documents there. So I click Cancel, and I get

G:\ is not accessible.
Data error (cyclic redindancy check).

What the hell is that??? :headdesk: All those files gone. I will have to re-download them from our university's server. I really did not need this.

And it's not just those files; I had all my apps backed up to that card. I had information for my contacts stored there. I had appointments saved in my diary app. How the hell am I going to rebuild those?

So what the hell is it about this program that's got everyone singing praises to it? Honestly, I have had nothing but trouble with it from the first time I tried this, but this is really the lowest point so far.

Everybody sings its praises because it handles many formats and devices, is frequently updated to support newer conventions and formats and is free. I'm sorry you're having trouble with it, but a) your frustrations are clearly not universal, and b) I personally find it much easier to want to help people who are not prone to fits of panic and entitlement.

I would appreciate it if you signed your post if you don't have a Dreamwidth account/OpenID, or logged in if you do.

I think you would panic too if you thought that your 4GB card which contains favorite stories, postgraduate notes, music and video files was deleted and didn't know the reason.

And please do explain to me how complaining about a feature that doesn't work for you even though it works for others is entitlement? Even if a program is free, there's no law saying you cannot complain about it because you haven't paid for it. If nobody complained about a problem they had, there would be no improvement.

I am fully prepared to be told I missed something blindingly obvious, and be called an idiot and a fool. It might be that there's something wrong with my installation. If that does happen, I will post an apology here and on MobileRead. But please, tell me your name so that I can apologize to you personally as well.

Flat | Top-Level Comments Only

Another calibre question

no subject

no subject

no subject

no subject

no subject

no subject

FLAG / AO3 DOwnloader

Re: FLAG / AO3 DOwnloader

Re: FLAG / AO3 DOwnloader

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

herve leger

no subject

no subject

no subject