Thursday, September 25, 2008

The Economist in Kindle Format

As loyal Fat Knowledge readers are aware of from my posts, I love me my The Economist. A few weeks back, for reasons unbeknownst to me, the post office decided not to deliver my weekly edition. Then, also unbeknownst to me, that copy was returned to The Economist and they decided to put my account on hold. After missing a 3rd week, I contacted The Economist and all the unbeknownsts became knownst to me. But, I was told that it would take another 2 weeks before I would get my next one in the mail.

Unable to live without my The Economist for multiple weeks, I created a way to download the contents of the weekly issue off of their website and then put it into a file that can be read on a Kindle. First, I created this .php file which downloads the entire content of the weekly edition from The Economist's website to a directory of files on my machine. Next, I used Mobipocket Creator to convert that directory into a Kindle friendly file. One advantage of this digital version is that it is available on Thursday, a day or two sooner than the physical version arrives in the mail. I also prefer the way I setup the table of contents to what I have seen on other Kindle newspapers and magazines.

Thinking that some others might also like to read The Economist on their Kindles as well, I share the results of this experiment with you: today's new 8/27/08 edition of The Economist in Kindle format. Just download the file and either email to your Kindle, or copy it over using the USB cable.

Truth be told though, as big of a Kindle and e-book fan as I am, I still prefer the dead tree edition of the magazine. Magazines have color and are easy to skim neither of which can be said for e-books. Magazines are great for reading on the go, easy to dog ear pages to read, and worry free if you spill your coffee on them. Unlike books which are heavy and don't stay open on their own, magazines are light weight and stay put. Unlike newspapers which are unwieldy to read on a bus, require a ton of paper and energy to print, and leak ink on your hands, magazines come in a convenient size that doesn't waste nearly as much paper and leave your hands ink free. While I am a full convert of e-books over paper books and newspapers, the magazine is still my one dead tree holdout.

Update: Commenter Jon gives instructions on how to set this up on a Windows machine to get the latest Economist each week. If you are fairly computer literate and are willing to spend a little time, I think you will be able to get it to work (see also Update 3 for another solution).

1. Download PHP. Visit http://us3.php.net/get/php-5.2.8-win32-installer.msi/from/a/mirror and click the yellow highlighted link near the top and run the file whose download should then start.

2. When asked what Web Server you would like to set up choose "Do not setup a web server" (the last option). Aside from this, just do the usual hitting next and install without changing anything. You'll get what you need.

3. Click on this link and save the file.

4. Navigate to where you saved the file, right click on it, and hit "Open with > Choose Program" then in that hit browse and navigate to C:\Program Files\PHP\php.exe and select this. Then hit the checkbox next to "Always use the selected program..." and hit OK.

5. Double click the file. A command prompt window should come up. Wait until it closes. Do not do anything to help it along, it will finish, but it might take 5 minutes or longer to download all the pages from the web.

6. Download Mobipocket Creator and install. Don't change anything during install. After installation open mobipocket creator. Then hit open at the top and navigate to the folder C:\Economist_[date] open this folder and open the file with a .opf extension. Then hit build at the top. Whew, that's it, there should now be a .prc file in that folder that you can transfer to and read off your kindle. Ignore build errors in making the .prc, they shouldn't matter.

7. Email the Economist_[date].prc file to your Kindle, or transfer it via the USB cable.
Customization: This requires touching the code which could be a bit scary for the less computer literate. Open the economist.php file that you created in step 3 with Wordpad. At the top there is a section called "Variables". 3 customizations are possible:

A) To save files to a different directtory change the $base_dir to another location like "C:\\Kindle Files\\".
B) Enter your username and password to access the site at $GLOBALS['loginEmail'] and $GLOBALS['loginPass'].
C) To create a version without images (much smaller file) set $with_images = false;
D) Get a back issue by setting $get_back_issue = false; and then setting the $back_date and $back_date2 variables to the date of the issue you want.

Update 2: The Economist website changed their formatting of URLs on 2/19/09. I have updated the .php file to fix this. Just redownload and overwrite the old version (step 3 above).

Update 3: This MobileRead thread alerts me to the existence of Calibre. This open source software will also allow you to download the latest Economist (as well as many other magazines and newspapers) in Kindle format. It has a graphical user interface, so it is much easier to use. Just make sure to set the output type to MOBI. Personally I lke my format and Table of Contents better, but then again I should as I designed it for an audience of one (though this guy gives my formatting props as well). :)

Update 4: The Economist has stopped the free access to the weekly addition. You now need a username/password to access it. Fortunately Nic has updated his script to allow you to enter that information. I have updated my script as well with this new code. Thanks Nic!

47 comments:

Anonymous said...

thanks for the setup, i'll try it when i get kindle.

my physical economist leaks ink on my hands all the time :-(

Fat Knowledge said...

Yikes. I have never gotten ink on my hands from The Economist. But, once again they are choosing not to deliver my magazine and aren't notifying me of it.

Anonymous said...

I also like to read the Economist on my kindle and the kindle file you created is pretty neat. I am not computer geek though and I have no clue about php file that you are talking about.I don't even know how to use that file. Could you possibly give me non-technical step-by-step setup ?

Fat Knowledge said...

Hi anon,

Glad you like the file. Unfortunately in order to use the .php file you would need some serious computer skills, as you would need to install PHP on your machine in order to run this script. Having installed this myself, I can't say it was an easy thing to do and wouldn't suggest it to someone who isn't a computer geek.

If I find a way to get the latest version of The Economist in Kindle version built automatically so people like yourself could just download it each week, I will leave a comment here.

Jon said...

Fat Knowledge, I'm not entirely sure why you cite the act of setting up php as difficult. I'd say the hardest part of the process was finding the code(good stuff by the way, thanks) did not automatically get the latest issue. Which was about a minute for me after finding that the script didn't work from the get go, but for someone that doesn't know code I imagine it would be more. The process is a bit tedious though I suppose.

Anyway, the process for those less computer literate (I'm assuming you're on Windows, if Mac I don't have one so I'm not going to try and figure it out, if Linux and you can't figure this out, get off Linux, you have no business running it):

1. Visit http://us3.php.net/get/php-5.2.8-win32-installer.msi/from/a/mirror and click the yellow highlighted link near the top and run the file whose download should then start.

2. When asked what Web Server you would like to set up choose "Do not setup a web server" (the last option). Aside from this, just do the usual hitting next and install without changing anything. You'll get what you need.

3. Click on Fat Knowledge's link and highlight everything from '[openbracket]?php' to '?[closebracket]' at the bottom. Hit ctrl-c open wordpad (start > program files > accessories) and hit ctrl-v. Then hit ctrl-s and navigate to where you would like to save this file so you can access it easily later. In the drop down menu change the format from "Rich Text Format" to "Text Document" and save the file as "economist.php" (the quotes are needed!!). Close wordpad.

4. Navigate to where you stored the file, right click on it, and hit "Open with > Choose Program" then in that hit browse and navigate to C:\Program Files\PHP\php.exe and select this. Then hit the checkbox next to "Always use the selected program..." and hit OK.

5. Ok, we're almost there, we have everything basically setup. Here's the scary part for the less literate, touching code. You should still be in the folder with the file, if not, navigate back. Right click it and under "Open with" select wordpad. If this is not there yet hit "Choose program" and it should already be in that list. Do NOT set it as the default. Now, go to the economist website and see what issue is the latest. As of me writing this it is January 10, 2009.

On line eleven you'll see 20080913 in between quotes. Replace that with the date you just got off the website using the same format. For me this is 20090110. Then on line twelve change what is in between quotes in a similar manner. Save and close. You will need to make this change whenever you want to get a new issue.

6. Double click the file. A command prompt window should come up. Wait until it closes. Do not do anything to help it along, it will finish.

7. Go to http://www.mobipocket.com/en/downloadSoft/DownloadCreator.asp and download and install. Don't change anything during install. After installation open mobipocket creator. Then hit open at the top and navigate to the folder C:\Economist_[date] open this folder and open the file with a .opf extension. Then hit build at the top. Whew, that's it, there should now be a .prc file in that folder that you can transfer to and read off your kindle. Ignore build errors in making the .prc, they shouldn't matter.

Fat Knowledge said...

Jon,

Great writeup, thanks. Since you went to the trouble of writing everything up, I am going to add it to the body of the text.

You are right that I did not make it obvious that you needed to set the date. I figured that anyone who could handle setting up PHP and getting everything else could figure that out as well. Looks like I was right. :)

I redid the code in light of your suggestions. Now it will automatically get the latest issue. If you want a back issue you need to specify. Also added ability to not get images.

Anonymous said...

I'm thinking of buying a Kindle solely to read the Economist. My paper edition usually takes ~1 week to arrive. I'd really like to read Thursday's issue that weekend, but I hate reading on my computer.

How is the reading experience using this method?

Fat Knowledge said...

Anon,

I definitely prefer reading on the Kindle to reading on a computer as my eyes don't get tired, and you can read in bed or on the run.

But, as a wrote, I still prefer the paper version because of the color, the ability to easily skim articles, the magazine is light weight & easy to bend, and I don't care if I spill food on it.

Overall though, reading on the Kindle is pretty good especially since you can start reading on Thursday and you can carry multiple back issues with you.

If you are wondering about the e-ink technology and how it compares to paper, there are lots of reviews on that on the web, but I personally think it is good enough for lots of reading, but the contrast still needs some work (the background is gray rather than white).

Anonymous said...

Thanks for your quick response. I was worried about the presentation quality and style of the Website -> PHP Spider -> MobiPocket transformations. But seeing how it didn't even enter your mind, I assume it's a non-issue. :)

Fat Knowledge said...

Well, it is a non-issue for me, but I designed it to my tastes. :) It is mainly text, so there isn't a whole lot to screw up. The images get increased in size on the Kindle, and don't look as good on the web (because of the lack of color), but I find it tolerable (and images can be removed from the file if you like). There is a table of contents in the beginning to select articles and I have formatted the title and subtitles in articles to look good on the Kindle. Personally I think it is as good as the NY Times or WSJ on the Kindle, but then again I am biased.

Maybe someone else who has used the script can leave a comment as to how well they like it and you will be able to get a non-partial opinion on the matter. :)

Anonymous said...

Getting a lot of "wrong parameter count" error messages, and it's only grabbin the ToC and the cartoon. Hmm.

Anonymous said...

More details: the HTTP requests are failing.

I've got 'net access, not sure what the problem is.

Fat Knowledge said...

Anon,

Ok, I modified the .php code with a two line change in lines 205 and 206 of the create_html function that will fix the problem.

Just redownload that file and replace the one on your machine (step 3 in the instructions).

Apparently they changed the way they are formatting the urls.

Anonymous said...

That did it! Thanks.

I get an Undefined variable: end in C:\foo\economist.php on line 134. But that doesn't seem to be harming anything.

Thanks again!

George said...

Thank you so much for this script! I was able to get it working on OSX and using the MobiPerl module, I've automated the process of retrieving from the website, converting to the mobi format, and emailing to my Kindle. I hope the Economist becomes a legitimate Kindle provider since I know code like this is a pain to maintain, especially when they can change the format of the website on a whim. But in the meantime, this will do. Thanks again!

Fat Knowledge said...

Cool George.

I hope the Economist becomes a legit provider as well (although they better release a version that is formatted as well as mine :)). I was also hoping that someone like Kindle Feeder would take this code and integrate it into their site so it would be easily accessible by a lot of people. So far no takers on that though.

Anonymous said...

Anyone else get a short output file when they run it on this week's issue?

Anonymous said...

Disregard -- I re-opened the file again and it was full-length. Not sure what happened.

Anonymous said...

How do I get it to come up to the ToC? It always comes up on the first article, and I have to hit Prev page several times to get to the ToC.

Thanks for this!!!!!

Fat Knowledge said...

Anon,

You're welcome.

I don't know how to get it to come up to the ToC. I run into the same issue that you do.

Unfortunately, it acts like a normal book, where you want to start on page 1 of the text. If I knew a way to get it to start on the ToC I would definitely make the change.

vasanth said...

can you please tell the process for pdf format ?

Anonymous said...

Has anyone figured out how to get the TOCs to be formatted in a manner similar to NYTImes? It is a far more intuitive style to present a TOC.

Johnny Ace said...

I have been collecting the weekly issues, but missed an issue, and am having problems getting it to generate.

How do I set the parameters within the .php file to get the content I desire to generate?

Help! (but thanks in advance!)

Fat Knowledge said...

Vasanth,

There is no process for .pdf that I am aware of. Sorry.

Anon,

I don't know how to change the TOCs to be more like NYTimes. Personally, I like the way the TOC works here, but I guess everybody likes it setup slightly different.

Johnny Ace,

If you are looking to get a back issue, you need to set 3 variables at the top of the file. First, set $get_back_issue = true; (instead of false). Then set $back_date and $back_date2 to the appropriate dates in the correct format. That should be it. Then set $get_back_issue=false; once you are done so that next week you will get the current edition.

Nic said...

Thanks for making this available. I made some small updates to have it automatically email my Kindle on Thursday morning:

http://nicj.net/2009/04/13/the-economist-and-the-kindle

Fat Knowledge said...

You are welcome Nic. I like the improvements you have made.

Jordan said...

Just want to thank you for making this available. I did some adjustments as well to automate the conversion and email process, but I had to use Blat and stunnel to get it to send out. Now if there was only a way to utilize whispernet directly to kindle for free!

Fat Knowledge said...

Jordan, glad to hear you got it working and were able to make some improvements. Yeah, it would be great if Whispernet was free, but 10¢ isn't too bad. Of course, I am a cheapskate that uses the USB cable to save that fee. :)

Anonymous said...

Just want to thank you for making this available. Greatly appreciate your help.

Is there a possibility for you to generalize this program so that one can get information from any web site on to a Kindle. Say For example, from any foreign news papers or magazines.

Thanks
Net

grill said...

It's out as of today!

Fat Knowledge said...

grill, on the 7/04/09 edition I ran into an error because The Economist's link to "Mexico’s mid-term election: Tilting to the PRI" isn't valid. But, the rest of it worked and I was able to transfer it to the Kindle.

Anonymous said...

Did you modify the code specially for Kindle DX ? Secondly, how would you get even the image of the cover page of the Economist?

Anonymous said...

How would I change the parameters to if want to have economist for all through the last year - Say July 5th 2008 to July 4th, 2009 issues at once

Thanks

Fat Knowledge said...

Anon,

No, I didn't modify the code for the DX, as I have no DX to test it on. If you are willing to donating one to me so that I can test it out, I would be more than happy to make it DX compatible. :)

I'm not quite sure what you are asking as to the cover page image, but if you go to this page: http://www.economist.com/printedition/cover_index.cfm there is a link to a large image of each cover available.

To get all the issues from the last year, you would need to modify the .php code to add a loop that would go through the whole thing for each different issue. The tough part would be figuring out what the dates for each different issue would be. Although I guess they are typically 7 days apart so it wouldn't be that tough.

Anonymous said...

Hello

Do you have a similar code for Harvard Business Review?

Thanks

admin said...

even its to late to know about this but its really great, have tried it and its working great.

Knowledge Management

Anonymous said...

The Economist's "This week" page is now gated. Is there any way for those of use with accounts to modify the script so we can still fetch this content?

Anonymous said...

hi, apologies, but I have just got my kindle, followed the steps, but seem to be getting a blank file when i try to download? Appreciate any help you can provide

Nic said...

Looks like they've blocked this week's print edition to only subscribed users.

Nic said...

An updated version that uses your Economist.com's login credentials: http://nicj.net/2010/01/03/the-economist-and-the-kindle-take-2

Fat Knowledge said...

Great stuff Nic. I always knew that if I was lazy enough someone else would make it work again. :)

Anonymous said...

amazon kindle 2 coupon

Pariuri Sportive said...

I appreciate you posting the your adjustments. After I read about the PHP on the Fat Knowledge site, I delved into automating the conversion and emailing process as well.

I wasn’t able get to the the PHP mail() function to work in windows, so I used a bit of a workaround. I had to utilize Blat and stunnel to use my gmail account as the smtp server. I wasn’t familiar with the PHP mail() function, but I didn’t see any settings for outbound mail on yours. Just wondering where/how you set up the mail servers with the mail function.

Anonymous said...

Thank you, thank you, thank you!

I'm one of those people who believes that, if you pay for it, you should be able to do what you want if you are adamant about not sharing. I love my Economist just like you, and just knowing that I cannot (up until now) use it on my new Nook hurts me. Physically. Cannot live without Economist. Spasms out the butt until I can see a political cartoon.

Anonymous said...

I don't get the script to work...

Anonymous said...

Links seem to not work properly.

Nevertheless, thank you for going through this. I have a subscription myself, but since I dont live in the US or UK I cannot access the e-book version on amazon. Stupid DRM

Anonymous said...

If you have a Kindle Fire there is an easier way. The economist android app actually works if you side load it to the Kindle Fire. It also works on Android 4- Ice Cream Sandwich, but not on Android 3.x Honeycomb.

I'm actually kind of surprise this has not been publicized more. Definitely not something Amazon wanst people to know but worth the effort.

Probably works on the nook color as well and you still get your print subscription- that is when it shows up on time.

Post a Comment

Note: Only a member of this blog may post a comment.