Mike's Page - Tech

Printer held Hostage by Ink Cartridge

nospam@example.com (Mike Boone) — Mon, 07 Apr 2008 20:57:03 -0400

I've had a Canon Pixma MX700 multifunction printer since the beginning of the year. I've been pretty happy with it. It's worked well for scanning, copying, faxing and printing. Until today. Today I needed to scan something, and was greeted with the following message when I turned it on:

U150 The following ink tank cannot be recognized.

The yellow cartridge was indicated. I though that it was an odd problem, but I didn't care for the moment, because I didn't want to print, just scan. But no button on the printer would bypass this error and let me scan.

I ended up calling Canon, and happily received a domestic phone rep. He had me try a couple things, but eventually decided that the yellow ink would need to be replaced. They would send me one free. I asked if there was any way to bypass the error so I could just scan, and he said there was not. If I wanted to use it today I would have to go out and buy a yellow ink cartridge. So after driving all the way across town and spending $16, I have my printer back...for now.

It seems like questionable design to cripple all the functions of the printer just because one color cartridge is faulty. A better failure mode would be to only prevent color printing. Black and white printing, and the other printer functions should still be usable. Hey Canon, how about updated firmware for this? Please?

Determining Image File Types in Ruby

nospam@example.com (Mike Boone) — Mon, 10 Mar 2008 22:22:03 -0400

Today I came across a PNG file that had been uploaded from a browser with a .JPG extension and image/jpeg MIME type. It's too bad that MIME types are apparently unreliable when it comes to file uploads. I went looking for a way to determine the file type by actually reading the file. This is probably a solved problem, but I was unsuccessful Googling for the answer. I came up with the following Ruby method which decides the image file type using up to the first 10 bytes:

def image_type(file)

  case IO.read(file, 10)

    when /^GIF8/: 'gif'

    when /^\x89PNG/: 'png'

    when /^\xff\xd8\xff\xe0\x00\x10JFIF/: 'jpg'

    when /^\xff\xd8\xff\xe1(.*){2}Exif/: 'jpg'

  else 'unknown'

  end

end

This works well on a small set of test files (400+ from a browser temp files directory). Let me know if there's a case where this code doesn't work, or if there's a better solution in general.

I'm in Rails

nospam@example.com (Mike Boone) — Sat, 15 Dec 2007 07:55:47 -0500

I've made a code contribution to Ruby on Rails. It was minor, but I'm glad I could help. See ticket 10435 and changeset 8386.

Supposed to get some rain today...keeping my fingers crossed.

Update 2007-12-17: Looks like my patch made it into Rails 2.0.2. And we did get that rain, but more would be good.

Update 2008-03-17: I managed to get another small patch committed. Go open source!

Formatting a Javascript Date for MySQL

nospam@example.com (Mike Boone) — Sat, 08 Dec 2007 08:59:50 -0500

I don't like to code in JavaScript, but that's the Ajax-y wave of the future. So I write it when I have to. This week I needed to format a Javascript date as a MySQL-style date like 2007-12-08.

JavaScript apparently doesn't have a nice date formatting function or even a generalized sprintf. Another Javascript quirk is that the Date.getMonth() function returns 0-11, while the getDate() function returns 1-31. I'm not sure what the designer was thinking on that one.

I came up with this function...feel free to use it in your code with or without credit:

function formatDate(date1) {

  return date1.getFullYear() + '-' +

    (date1.getMonth() < 9 ? '0' : '') + (date1.getMonth()+1) + '-' +

    (date1.getDate() < 10 ? '0' : '') + date1.getDate();

}

Improving upon Rails' validates_inclusion_of

nospam@example.com (Mike Boone) — Tue, 28 Aug 2007 16:36:34 -0400

I'm fairly new to the Ruby language and the Ruby on Rails framework, but it worked nicely for quickly building my website BirdSite.

I was using validates_inclusion_of fairly regularly to make sure the data in my models was valid. But then, watching the development log, I noticed extra SQL queries being made when I accessed my main model. It seems that validates_inclusion_of was forcing the model to make queries to the related models even when I didn't need data from them.

Some Googling led to a blog post that did the validation a different way. It worked well, until I added the code to a new model today and it broke. The code was translating the model field name into a string from which to call the model itself. But the code assumed that the model would be a single word. I added a change to strip the spaces out, and it worked again.

I contacted the company that hosts the blog and they confirmed that it was a problem, and they didn't know of a nicer Ruby way to correct it short of dropping the spaces with search/replace. They updated their code.

I'm glad I could help out. It really works better than the standard validates_inclusion_of.

Rails acts_as_ferret without DRb

nospam@example.com (Mike Boone) — Tue, 14 Aug 2007 17:17:38 -0400

I wanted to add a search feature to my Ruby on Rails application BirdSite. There is a plugin called "acts_as_ferret" which allows Rails applications to use the Ferret search engine (based on Lucene).

Using this tutorial, I was able to get search up an running. It worked great on my development system. But there was a warning that you needed to run the indexer using a DRb server instead of directly from the Rails app. This is because the index cannot handle multiple processes writing to it simultaneously. (I also ran into this problem using PHP with another engine called Xapian).

I was hoping my new site was low traffic enough to avoid problems. Since I'm hosting it on a shared server, I can't run a DRb server there anyway. But I got my first indexing exceptions less than 24 hours after I loaded my stuff onto the live server.

So I decided to try periodic indexing by a cron job. This would allow me to update the index once an hour, from a single process. The downside is that the search index is only updated hourly, but I decided I could live with that.

The first step is telling your Rails app to not index the content when updating. My model already had a before_save method, so I added this code:

# in the model

def before_save

  # other stuff goes here

  # disable automatic ferret indexing...move it to a cron job

  self.disable_ferret(:always)

end

Then I had to create a rake task which would build the index:

# ferret_index.rake

desc "Updates the ferret index for the application."

task :ferret_index => [ :environment ] do | t |

  MyModel.rebuild_index

  # here I could add other model index rebuilds

  puts "Completed Ferret Index Rebuild"

end

This task is simplified: I'm telling it to rebuild the entire index each hour. I'm guessing when my dataset gets big enough, this will be really slow. In that case I'll need to track all the model instances that got updated in the past hour and just index those.

Finally, I needed a cron job to run the rake task, making sure to set the environment to "production":

cd /rails_app && rake ferret_index RAILS_ENV=production

So far this is working well, and I haven't received any indexing exceptions since.

Check out the BirdSite!

nospam@example.com (Mike Boone) — Fri, 06 Jul 2007 14:50:51 -0400

I recently published a new website: birdsite.org. This site was inspired by BugGuide, where I've been a participant for several years, both as a photographer and a developer. I'd been thinking about doing a similar site for other fauna that I photograph. This forum thread made it clear that others were interested in such a site for birds. I decided to build it. For my own programming amusement I created this site as my first project with Ruby on Rails. I still have lots of features to add but I wanted to get the bare bones up and running.

Drupal Anti-spam Registration Question

nospam@example.com (Mike Boone) — Wed, 30 May 2007 14:27:06 -0400

Last year I had to add an anti-spam question to my PHPBB forum to prevent spam bots from signing up and wreaking havoc. Lately the same bot problem has crept up on a site I have running Drupal 4.7.x. So I resorted to the same solution, though this one is implemented as a Drupal module. It is very simple...the only thing you configure is the question and answer you want to use. Click here to see it, and follow the instructions in the comments. I hope it helps you.

PHPBB Anti-spam Registration Question

nospam@example.com (Mike Boone) — Sat, 21 Apr 2007 13:45:19 -0400

If you are not comfortable installing this on your own, I'll do it for you. If you're using PHPBB 2.0.23 and the SubSilver template, I will install it for $25 via Paypal. (If you're using a different template or need an older site updated to the latest PHPBB, I can do that too, but it will take longer and cost extra). Contact me.

I've been meaning to document this for a while now, and I finally made myself write it up. The simple measure I will describe has been working for my PHPBB website since last December. It has kept spam accounts from registering and posting anything on my site.

This idea involves asking a question during registration that spam bots do not know how to answer. You can choose any question and answer that you want. The website that I run PHPBB on is a sailing website, so I asked a question that sailors could answer. For this example I chose another question.

You have to change two files, templates/(your template)/profile_add_body.tpl and includes/usercp_register.php.

In templates/(your template)/profile_add_body.tpl, add this after the  line (about line number 68):

<!-- BEGIN switch_add_profile -->

<tr> 

  <td class="row1"><span class="gen">Anti-Spam Question:</span></td>

  <td class="row2">

    <span style="font-size: small;">Enter the name Luke Skywalker's

    father. Check your spelling!

    Requiring this question to be answered will hopefully limit

    spammers who try to sign up.</span><br />

    <input type="text" class="post" style="width: 200px"

      name="bonusq" size="25" maxlength="255" value="" />

  </td>

</tr>

<!-- END switch_add_profile -->

In includes/usercp_register.php, add this after else if ( $mode == 'register' ) { (about line number 275):

// mod by MB to require human data to prevent spam bots

if(trim(strtolower($_POST['bonusq']))!='darth vader') {

    $error = TRUE;

    $error_msg .= ( ( isset($error_msg) ) ? '<br />' : '' ) .

      'You did not answer the Anti-Spam question correctly...' .

      'please try again.';

}

// end mod

So there you have it. I'm sure there are some sophisticated PHPBB mods out there that will do the same and more for you, but this simple change has saved me a lot of trouble. I'm keeping my fingers crossed that it will last.

Update 2007-05-23: I omitted one other mod to make this work. You also need to edit your includes/usercp_register.php file. Find the line if ( $mode == 'editprofile' ) (about line number 941) and modify that block to look like this:

if ( $mode == 'editprofile' )

{

  $template->assign_block_vars('switch_edit_profile', array());

} else {

  // Else block is Mod by MB 2006-08-11

  $template->assign_block_vars('switch_add_profile', array());

}

Update 2007-06-24: Several folks wrote in expressing confusion about where these blocks of code belonged. I apologize, and I amended the article to give approximate line numbers where the mods belong. I used PHPBB 2.0.22 as my reference. Note that if you have made other modifications to your PHPBB files, the line numbers might not be quite right.

Update 2007-06-27: Bug fix. See comment #7 below.

Update 2007-09-29: A lot of people have written in having trouble with this modification when it comes to users editing their profile. To avoid problems, I have applied the modification to a new copy of PHPBB 2.0.22 and tested it. I made copies of the modified files and they are available to download. The modification as it appears in this blog post is what I used, and it works fine for me. I did not make any of the other changes from the comments (except the bug fix from June which has already been incorporated into this post).

Update 2008-02-27: The update works just fine for me in PHPBB 2.0.23 as well. Neither profile_add_body.tpl nor usercp_register.php were changed between 2.0.22 and 2.0.23.

Java Runtime Environment Space Eater

nospam@example.com (Mike Boone) — Wed, 18 Apr 2007 17:03:28 -0400

If you're looking for some spare disk space, check out your C:\Program Files\Java folder. Mine was full of various folders like jre1.5.0_7, jre1.5.0_11, etc. I'm not sure how they all got there, but I'm guessing they came from installing several versions of OpenOffice over the years. Looking at the Add/Remove programs in the Control Panel, I had like 8 versions of the Java Runtime Environment, collectively eating up several hundred megabytes. So I cleared out all but the latest and things are nice and tidy for the moment.

Creating a Date Range Array with PHP

nospam@example.com (Mike Boone) — Mon, 08 Jan 2007 15:16:37 -0500

I recently needed to come up with an array of dates between two given dates. I typically work in the MySQL-compatible 'YYYY-MM-DD' date format, so I wanted it to use dates like that. So I came up with the following function:

function createDateRangeArray($strDateFrom,$strDateTo) {
  // takes two dates formatted as YYYY-MM-DD and creates an
  // inclusive array of the dates between the from and to dates.

  // could test validity of dates here but I'm already doing
  // that in the main script

  $aryRange=array();

  $iDateFrom=mktime(1,0,0,substr($strDateFrom,5,2),     substr($strDateFrom,8,2),substr($strDateFrom,0,4));
  $iDateTo=mktime(1,0,0,substr($strDateTo,5,2),     substr($strDateTo,8,2),substr($strDateTo,0,4));

  if ($iDateTo>=$iDateFrom) {
    array_push($aryRange,date('Y-m-d',$iDateFrom)); // first entry

    while ($iDateFrom<$iDateTo) {
      $iDateFrom+=86400; // add 24 hours
      array_push($aryRange,date('Y-m-d',$iDateFrom));
    }
  }
  return $aryRange;
}

Note that the first parameter of my mktime function calls is a 1 due to problems I experienced before with Daylight Savings Time.

PHPBB Inactive Member Removal Cron Job

nospam@example.com (Mike Boone) — Tue, 05 Dec 2006 09:25:36 -0500

A commenter to a previous article asked exactly how I delete inactive members from a PHPBB forum that I run. So I'll try to explain. This solution runs on Linux/Unix systems...I'm sure it could be done for Windows, but I'll leave the particulars to you.

It's really two separate steps. First, you need a script which will handle the deletion of inactive members. I called mine cron.php. It deletes all inactive PHPBB users who don't activate within 48 hours. It looks like this:

#!/usr/bin/php -q
<?php
  // cron job to delete inactive users older than 48 hours
  $db=mysql_connect('server','user','password');
  mysql_select_db('your_phpbb_database_here',$db);
  $strSQL="DELETE phpbb_users u, phpbb_user_group ug, " .
    "phpbb_groups AS g FROM phpbb_users u, " .
    "phpbb_user_group ug, phpbb_groups g WHERE " .
    "u.user_active=0 AND u.user_id>0 AND " .
    "u.user_id=ug.user_id AND ug.group_id=g.group_id " .
    "AND g.group_single_user=1 AND " .
    "FROM_UNIXTIME(u.user_regdate)<" .
    "DATE_SUB(NOW(),INTERVAL 2 DAY);";
  mysql_query($strSQL,$db) or die(mysql_error());
  mysql_close($db);
?>

You'll need to make sure the /usr/bin/php points to the location of PHP on your system, and replace the MySQL server name, user, and password with yours.

Now that you have a script, you need to tell the system to run it daily. You can do this with a cron job. If you have command line access to your website, you might be able to do this with "crontab -e". But my webhost has an administrative panel that lets you set up cron jobs on the web. If you can't set up a cron job, you could put the script into a web-accessible folder and periodically call its URL, either manually or through an automated process on your local PC.

This idea works great if the majority of your spam registrations don't activate their account. Usually they just want their spam links in your member list. But I'm finding that more and more spammers are activating and posting, so it remains that we want to stop spammers from registering in the first place. I'm experimenting with another method, which I'll post about when I see some results.

Update 2007-11-28: I replaced my original SQL statement with the SQL in comment #1 below, which I finally tested and it seems to work well.

Splitting a large XML file on Linux 2.4

nospam@example.com (Mike Boone) — Wed, 06 Sep 2006 21:01:50 -0400

A client recently had a problem processing an XML with his PHP script. "File too large" was the error, and the data file was over 2 gigabytes in size.

It turns out you can recompile PHP to deal with large files (see Requirements section here). But this blog entry made recompiling sound problematic...you get access to the large file but certain file functions break due to integer overflows. So I wanted to avoid this option.

The Linux command line tools seemed to be able to deal with it OK: stuff like less, head and tail were working on it. So I decided to try to break the XML file into parts. The split command worked but indiscriminately cut right through the middle of a data record. I considered using csplit, but as this file contained over 700,000 data records, I didn't want to deal with that many individual XML files.

I decided to write a Perl script to split the file into blocks of 100,000 records each. It didn't take long to put together and Perl's regular expression matching made handling the records easy on the small test data. For some reason I thought Perl would be OK with the large data file, but when I went to run the script, it too choked on the large file. I would have to recompile Perl to get around it. As the client's box is using a Red Hat package for Perl, I didn't want to mess with it.

Then I had an idea. Since the Linux command line tools were handling the file OK, I wondered if I could trick my script by feeding it one line at a time. Instead of looping an open file in the Perl script, I used a loop like this:

while( $line = <STDIN> ) {
# do stuff
}

Then I called the script like this:

cat bigfile.xml | ./split.pl

And it worked!

Drupal: An image module that uses filemanager module

nospam@example.com (Mike Boone) — Sun, 27 Aug 2006 18:31:35 -0400

I'm working on a personal project using Drupal. The image module allows you to let users post images to the site. However, it puts all of its files into a single directory, which is a performance concern if you want to have a lot of images on your site. The filemanager module has support for lots of files, but the image module was not written to use it.

So I decided to try my hand at Drupal coding. I started with image.module and overhauled it to work with filemanager.module. Along the way I found that Drupal 4.7 doesn't offer a good way to maintain certain data through the form preview, so I filed a bug. The workaround I came up with, and also used by a prominent Drupal coder, was to use the $_SESSION variable to maintain that data.

So anyway, here is the new image.module code. Please feel free to try it and let me know what I should fix.

Some notes: the current version 0.1, requires you to use only the original, preview, and thumbnail sizes, and also requires a database table called image_fm. I also could not come up with an elegant way to handle the case of a single user posting multiple images at the same time, sicne the $_SESSION variable used only allows for one per user at a time. This may be easy to fix, but I don't fully understand the Drupal form API yet.

PHPBB Fake Members

nospam@example.com (Mike Boone) — Thu, 17 Aug 2006 10:14:55 -0400

It's becoming frustrating to be a PHPBB administrator, at least if you want to keep your memberlist clean. Form bots out there create fake users on your site in the hopes that your memberlist will show their spam URL. It's been an ongoing, and losing battle, to keep them out.

First I tried to change the account creation form slightly.
Next, I instructed search engines to not index the memberlist and user profile pages.
Several months ago I turned on PHPBB's "captcha" feature that was included with a more recent version
Last week, I removed the URL field completely from the signup form, and any bot that tries to submit it will fail. This might have stopped some of the bots, but there are still fake users being created. Their profiles list no websites, but you can still tell they are bogus when they list their interest as "pills" on a sailing website!

I think my next move will be to create a cron job to delete user accounts that are not activated within 24 hours, as many spam accounts never activate.

Update 2006-08-22: The fake users keep coming. So I came up with a cron job that runs this query once per day. It will remove inactive PHPBB users older than 48 hours. This gives time for the new users to properly activate.

DELETE FROM phpbb_users WHERE user_active=0 AND user_id>0 AND FROM_UNIXTIME(user_regdate)<DATE_SUB(NOW(),INTERVAL 2 DAY);

The user_id>0 part is to avoid deleting the Anonymous user, which has a user ID of -1 on my installation.