Encoding problems

First dump initial, non-utf database:

mysqldump -uUSER -pPASSWORD --default-character-set=latin1 --skip-set-charset dbname > dumpfile.sql

then (linux) change all latin1 in dumpfile.sql to utf8

sed 's/latin1/utf8/g' dumpfile.sql > dumpfile_collate_utf8.sql

Optionally gzip and send to localpc
gzip -d dumpfile_collate_utf8.sql.gz


Recreate same database with utf8 collation and import dumpfile_collate_utf8:

mysql -uUSER -pPASSWORD --execute="DROP DATABASE dbname;
    CREATE DATABASE dbname CHARACTER SET utf8 COLLATE utf8_general_ci;"
mysql -uUSER dbname < dumpfile_collate_utf8.sql

Now the next part, get on Rails

I've set RAILS_GEM_VERSION = '1.1.6' to be sure we are in non unicode.
1. Create controller with:

require 'iconv'
require 'singleton'
class ConverterController < ApplicationController
end

Inside create two actions:

  def convertme
    rows = Model.find(:all)
    for row in rows
      row.title = iconvert(row.title, "latin1", "utf8")
      # replace latin1 with appropriate encoding
      row.save!
    end
  end

  def iconvert(str, encoding_from, encoding_to = "utf8")
    i = Iconv.new encoding_to, encoding_from
    utf_str = ""
    begin
      utf_str << i.iconv(str)
    rescue Exception => e
      utf_str << e.success
      ch, str = e.failed.split(//, 2)
      utf_str << "?"
      logger.info "FAIL !! char:#{ch}"
      retry
    end
    return utf_str
  end

Point browser/console to /converter !

Thats IT !
What doing iconvert: its taking string, trying to convert it to utf-8, if conversion fails (on some japanese/korean) it put "?" instead. Thats because some non-utf encoding cannot be converted to utf-8(needs to be utf-16 or utf-32).

Also in case of large database conversion, i suggest to add column to database table, like "utf_converted" and do conditionally find(:all) and then row.utf_converted = true before save, just to be safe if something fail, to do not convert twice.