Couch speeding

By Taylor Luk on November 12, 2009 @ 02:08 AM

I gave a talk "Couch Speeding" yesterday at November 09 Roro ruby group meetup. There are a few common performance issues when dealing with couchdb database and i have shown a few tips and solutions that i have found over the past months.

Play FPS with real gun

By Taylor Luk on November 11, 2009 @ 03:27 PM

This is something me and my buddies talk about a lot in the past, today i have found one on youtube actually did it.

Enjoy...

Setup Xapian and Ruby binding on Ubuntu server

By Taylor Luk on August 17, 2009 @ 10:00 AM

Today, I need to get my Xapian setup with Ruby 1.9 on my Ubuntu 8.04 vps and there are very little documentation on setup Xapian and ruby, so I decide to share and cover some information on the setup process.

In the past I have introduced Xapit in a post Fulltext search your CouchDB in Ruby, It is a Xapian fulltext search toolkit for ruby and it's easy and extensible, also i have added a CouchDB adapter to Index your CouchDB database using Xapit.

Here is a list of documentation about installing Xapian,

• Official documentation
• acts_as_xapian installation wiki

Xapian

Latest ubuntu package for Xapian i can find is 1.0.4, so i decide to download the latest package of xapian-core and xapian-bindings from the official website.

xapian-core

wget http://oligarchy.co.uk/xapian/1.0.14/xapian-core-1.0.14.tar.gz
tar zxf xapian-core-1.0.14.tar.gz
cd xapian-core-1.0.14
./configure && make && sudo make install

xapian-ruby binding

xaipan bindings contains language binding to many languages including ruby, python and php.

wget http://oligarchy.co.uk/xapian/1.0.14/xapian-bindings-1.0.14.tar.gz
tar zxf xapian-bindings-1.0.14.tar.gz
cd xapian-bindings-1.0.14

./configure && make && sudo make install 

If you have a alternative ruby interpreter such as (ruby 1.9 or REE) then you need to pass in the location of your ruby interpreter during setup.

./configure RUBY=/opt/ruby1.9/bin/ruby1.9
make && sudo make isntall

Quick Test

A quick irb session will reveal if everything has gone smoothly.

irb1.9
irb(main:001:0>require 'xpaian'
=> true

Congratulations

Now you have successfully installed xapian and it's ruby binding. Checkout acts_as_xapian or xapit if you are looking for a fulltext search solution in your Rails application.

Testing Rails on Ruby 1.9 using Rspec + Remarkable

By Taylor Luk on August 18, 2009 @ 10:00 AM

Background

When I started developing with Ruby on Rails last year and one of the remarkable thing I have found is the test-first cultural of the ruby community. There is a exhausted list of TDD/BDD/stubing/mocking libraries, sometime it's frustrating to get a good setup that is simple enough to maintain.

I consider myself as a newbie in TDD and instantly loved the syntax of Rspec and it's expressiveness, but dislike it's complexity. Finally, there it comes the Shoulda and find all the validation/association testing macros are really time savers.

As you can see, I stepped into Ruby 1.9 and find little information about a nice setup testing your ruby application in Ruby 1.9. Shoulda didn't work for me and found it wasn't Ruby 1.9 ready at that time (couple months ago) and things may(or not) have got better.

It forced me to look for alternative, and i discovered...

Remarkable

Remarkable is a framework for Rspec matchers that supports macro and internationalization. Remarkable-rails allowed Shoulda style testing in Rspec.

Those macros enables simple validation/association testing on your ActiveRecord models and It ran straight out of the box for Ruby 1.9.

It has remarkable taste

# It has your style. You can choose between:
it { should validate_numericality_of(:age).greater_than(18).only_integer }
it { should validate_numericality_of(:age, :greater_than => 18, :only_integer => true) }

should_validate_numericality_of :age, :greater_than => 18, :only_integer => true

should_validate_numericality_of :age do |m|
  m.only_integer
  m.greater_than 18
  # Or: m.greater_than = 18
end

Rspec

In general Rspec works on Ruby 1.9 but it depends on a test library (test-unit) that isn't part of ruby 1.9 standard library anymore.

Stay with good old test-unit with this particular version that works, you need to remove any newer version you have. (hint: gem update)

gem install test-unit --VERSION=1.2.3

machinist

It's description speak for itself.

Fixtures aren't fun. Machinist is.

Remarkable setup for Rails

Finally I would like to share this remarkable setup that i used for many rails project. All gems are included as gem dependencies for test environment.

config/environments/test.rb

Testing stack

config.gem "rspec", :lib => false
config.gem "rspec-rails", :lib => false
config.gem "remarkable_activerecord", :lib => false
config.gem "remarkable_rails",  :lib => false
config.gem "notahat-machinist", :lib => "machinist", :source =>"http://gems.github.com"

Install

RAILS_ENV=test rake gems:install

Conclusion

I find the experience testing in Ruby 1.9 is faster and more responsive. As a side note, however, I was actually planning to replace test-unit completely with minitest and gotten to a point it works perfectly for non-rails project, please let me know if you have solved it or have similar experience.

I hope you enjoy this post as part of Ruby 1.9 series i am writing.

Ruby 1.9 encoding gotcha, Retreat to ASCII-8BIT

By Taylor Luk on August 14, 2009 @ 02:52 AM

(Note: this is not a rant)

Why should i care?

Never thought i would actually write about character encoding in Ruby1.9, it never interests me as long i can get my code to work doing whatever i need to do. Lately, I have been running into corners for a few time like when you need to parse a large amount of xml feed without knowing it's character encoding. I keep asking myself why should i even bother, but the fact is

You can't trust anyone to encode their files properly, I actually ran into cases like a xml file tells you (ContentType UTF8) and the feed itself is actually encoded in UTF16-LE

One of a very nice Ruby 1.9 feature is encoding aware character string, It includes many good stuff features such as M17N, character encoding conversion and detection (*limited). I have learn more about Ruby 1.9 and how character encoding works and i recommend to go through James Edwards II's series of posts about character encoding.

This is a error (and other similar ones) which ruby just came up on you, either you aren't doing things properly or the library you are doing aren't in a great shape.

incompatible character encodings: ASCII-8 BIT and UTF-8

Most of the time things are good and peaceful, and there are time things are dreadful, but easily fixable.

1. Set your source file encoding

   # encoding: utf-8

2. Make sure your Regex are Unicode friendly

   /#{some_patter}/u

Encoding fail

There are some cases ruby's character detection (limited) failed to get correct file encoding, Well okay, I open file with correct encoding

File.open(file_path, 'r:utf-8')

Again what about the time Ruby's encoding detection fails and where you have no knowledge about the external file (again think external feeds).

Then you are simply out of luck.

How about character encoding libraries ?

I came a cross rCharDet gem, It is a port of python's UniversalDetector and which is also a port of Mozilla Charset Detectors which is a library taken intelligent and uses statistical approach to detect the correct encoding.

Sounds good and scientific but then i find out,

It is not Ruby 1.9 compatible

How about i fix it?

when I tried to fixed rCharDet, It looks and smells like python code and yet again.

something epic fails about "encoding: ASCII-8 BIT" i couldn't remember. 

Ruby fails because it doesn't know the encoding of input string, and isn't that exactly why i am use this library to start with (hair pulling)

I start to realize

there are cases where you HAVE to work with unknown encoding (just don't know) and that is precisely the reason this non-standard encoding ASCII-8BIT is invented.

Give me back to good old byte stream.

if "1.9".respond_to?(:encoding)
  source.encode!('ASCII-8BIT')
end

Fortunately, I have successfully ported rCharDet to Ruby 1.9 and residing on my github account, it's also how i fixed the issues when i port my h2o-template to ruby, it's also how Rails core team fixed this in ActionView erb template.

So please Rack can you do the same (just accept the ASCII-8BIT patch) and my app can run again, because these Chinese characters(華人醫學) in the URL aren't letting me to go to sleep tonight.

Hey, PAC man. Sup, subdomains !

By Taylor Luk on August 12, 2009 @ 11:56 PM

Updated: link to rake task app:sync

Today, i would like to discuss the idea of subdomain in web apps, then share some tips and code to allow you to development and enable subdomain in your rails app, easily.

Enabling subdomain for your user account can be a great feature for your web application, there has been lot of debate about whether this is a good idea.

The bad

I still remember reading this article "Subdomains + Development = Sucks" from couple years ago, it basically highlights a lot of legitimate downside when you want to use subdomain to differentiate user accounts for your web application.

• SSL certificate for a Wildcard subdomain is way more expansive than a standard SSL cert for a domain.
• As for development, it's painful to setup a local development environment (try to get local wildcard subdomain), difficulties with local host names
• definitely more complex to develop and test.

Enough said, Subdomain is pain in the ass.

The good

Despite all that I love subdomains since users get greater isolation and DNS CNAME is one of the best thing happen to internet architecture.

• technically they are actually separate website if you ask the trust worthy Firefox or even Google the search engine.
• customer can point one of their domain name to a separate host, external system and services (ie: Amazon s3). Yet preserves their brand identity.
• Browser allows cookies to be shared between root domain and a subdomain.

And My recipe

Today, I'd like to share couple goodies that i have discovered and some code i have developed to help you enable subdomains in your Rails application.

1. Convention

Use .local on your local environment and .tld for production, Main thing is stick to a pattern that you are comfortable with.

username.myawesomeapp.local
username.myawesomeapp.com

2. subdomain-fu

a rails plugin to help your application understand subdomains, includes nice addition to routing and helpers.

3. Introducing Rack::CustomDomain

It's a rack middleware that i have developed (get it here) to switch on custom domains for your application, So your customer can enter a CNAME entry to support their own domains

username.com  =>  username.myawesomeapp.com

Rack is a beautiful thing and it provides Layered Abstraction and transparent among each rack component, using this ruby class all the subdomain-fu features will still work as usually.

For example: taylorluk.siterest.com, www.taylorluk.com are effectively the same account

Setup it's easy, drop it to app/middlewares/custom_domain.rb and add following to your environment.rb or a environment specific config file.

config.load_paths += %W( #{RAILS_ROOT}/app/middlewares )
config.middleware.use "CustomDomain", ".myawesomeapp.com" 

4. "Proxy PAC"

editing /etc/hosts is waste of my time

This is the magic sauce if you ask me, I used to use a ruby gem "ghost" to add subdomains to local. It's easy to use but the problem is it's too much work. Why there isn't a easy solution to bring wildcard subdomain to your local workstation without installing a fr**king DNS server and luckily i find the solution in this article.

Proxy PAC is a javascript proxy configuration file that you can install to your browser

Save it /Users/USERNAME/.proxy.pac

function FindProxyForURL(url, host) {
  if (shExpMatch(host, "*.myawesomeapp.local")) {
    return "PROXY myawesomeapp.local";
  }
  return "DIRECT";
}

• Firefox - Preferences/Advanced/Network/Connection Settings and enter the file name file:///Users/USERNAME/.proxy.pac
• Safari - Preferences/Advanced/Proxy settings select Use a PAC file and enter file://localhost/Users/USERNAME/.proxy.pac

Congrats

Now you have got wildcard subdomain right on your local environment. All of these have helped me when I am building my side project "SiteRest". The development is a lot easier, now i use a rack task to synchronize data between local and production environment and bang accountname.myawesomeapp.local, I can diagnose problem with a specific account easily.

Enjoy !

Ruby 1.9 and Passenger

By Taylor Luk on August 12, 2009 @ 04:41 PM

I have been developing on ruby 1.9 for couple months on my workstation a install through macport seems easy enough and switching between leopard's ruby and macports 1.9 is done via some shell script.

sudo port install ruby19

Two days ago, I am in the process of preparing a fresh ruby 1.9 stack to deploy a Rails prototype. Things are smooth by grabbing a fresh tarbal from http://ruby-lang.org then install it.

wget ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.1-p243.tar.gz  
tar zxf ruby-1.9.1-p243.tar.gz && cd ruby-1.9.1-p243
./configure --prefix=/opt/ruby1.9 && make && make install

Passenger doesn't start

I just can't get passenger to behave probably and ran into issues described here, here and here. Apparently, All recent patchlevel of Ruby 1.9 contains a backward-incompatible change that breaks many Rack compliant server such as Passenger

Solution

At the moment the only fix is to patch your Tempfile class in Ruby's standard library by remove those 4 lines added in one of the problem commit.

--- lib/tempfile.rb     23 Jul 2003 16:37:35 -0000      1.19
+++ lib/tempfile.rb     5 May 2004 23:33:57 -0000
@@ -106,7 +106,10 @@ class Tempfile < SimpleDelegator
   # file.
   def unlink
     # keep this order for thread safeness
-    File.unlink(@tmpname) if File.exist?(@tmpname)
+    if File.exist?(@tmpname)
+      closed? or close
+      File.unlink(@tmpname)
+    end
     @@cleanlist.delete(@tmpname) if @@cleanlist
   end
   alias delete unlink

Mysql

Then Next hiccup will be mysql gem won't compile, according to this blog post you need to download the latest version from rubyforge and build it yourself.

wget http://rubyforge.org/frs/download.php/51087/mysql-ruby-2.8.1.tar.gz
tar zcf mysql-ruby-2.8.1.tar.gz && cd mysql-ruby-2.8.1.tar.gz
/opt/ruby1.9/bin/ruby extconf.rb --with-mysql-config
make && make install

Conclusion

Luckily, isitruby19 is a excellent community effort to document and fix issues on incompatible gems, It provides a lot of useful tips if you run into issues get other ruby gem to work.

Migration to Ruby 1.9 may not be effortless, but i have been reworded with faster application response time.

Fulltext search your CouchDB in Ruby

By Daniel Tintner on June 18, 2009 @ 01:00 PM

Background

CouchDB is a awesome schema-free document-oriented database which now a official Apache project, Rubists quickly sit & relax with CouchDB viewing it as a different way to solve the storage problem.

Between me and my buddy Nathan, we have quickly decided to use CouchDB as the secondary tier of data storage in the application we are building. There are many CouchDB library in Ruby from CouchObject, ActiveCouch, CouchRest, RelaxDB and CouchPotato. Well, that is a lot of libraries to research and it took me quite a while to settle with CouchRest.

Things are good... Until one day, I that realize there is no obvious ways perform Fulltext index/search in Couchdb.

Pick A Search Frameworks

There are many open source and commercial fulltext search engine, Sphinx is a fast and reliable indexer that supports two popular open source database, thinking-sphinix is best Ruby plugin at the moment to enable Fulltext searching to your Rails application. However, Sphinx has a strict indexing requirement of 32bit or 64bit primary key for target document, so It doesn't work for CouchDB.

CouchDB-lucene is the defacto fulltext search plugin for CouchDB, building CouchDB-lucene is quiet straightforward given if you know how to handle a java stack on your server then compiled, configure and restart. There are two reasons i decided to keep looking for alternative (I am not anti-java).

• Dependent on Java stack and building couchdb-lucene itself requires a dozen dependent libraries.
• Extending couchdb-lucene and customization needs to be done in Java.

Interaction with Ruby
Libraries like Ferret and Xapian has native ruby binding and which can be customizable in Ruby. It happens to be extremely important for my project which requires interfacing with the search index and Interact with the search toolkit in almost every aspect such as Xapian Database, Indexer and QueryParser in Ruby. This ability has significant importance to my project to fine tune and perform additional analysis.

Xapian trying to Xap-it

Xapian
Xapian is a highly adaptable toolkit which allows developers to easily add advanced indexing and search >facilities to their own applications. It supports the Probabilistic Information Retrieval model and also supports a rich set of boolean query operators. It includes binding for many programming languages (Ruby, Python, PHP, Perl), Documentation on the Ruby binding is definitely lacking, Luckily, It provides pretty much all core C++ class and methods wrapped in Ruby.

Xapit
Xapit( pronounce as "zap it" ) is a new high level search library developed by RyanB, High level Ruby library for interacting with Xapian, a full text search engine that is in Ruby and most importantly ORM agnostic.

It includes a AbstractAdapter class ready to be extended for any other kind of ORM, Consequently, that is exactly what i did, I forked the project and 30 minutes later, It is indexing and searching my couchdb objects.

Introducing Xapit with CouchRest support

Please refer to this documentation on setting up Xapian and installation of Xapit. To enable Xapit in your model you need to include Xapit::Membership to enable Xapit search. Xapit's indexing is processed in Ruby so it supports virtual attributes be adding a ruby methods and this is exactly how i intend to index CouchDB's nested attributes.

item.rb

class Item < CouchRest::ExtendedDocument
  use_database COUCHDB_SERVER

  # Enable xapit for this model
  include Xapit::Membership

  xapit do |index|
     index.text :title, :weight => 2
     index.text :description, :weight => 2

     # Index nested document property by a method
     index.text :feature_names
  end

  property :owner_id
  property :title
  property :description
  property :features

  view_by :owner_id

  def feature_names
    self.features.map{|m| m['name'] if m }.compact
  end
end

play with it in your script/console

>>Xapit.index_all
>>Item.search('Nintendo Wii')

Conclusion

Xapian is feature rich and it's got a lot more to offer, Looks like I will be settling with Xapian and a Ruby oriented way of fulltext indexing for my CouchDB needs.

Limitation

• Xapit is still under active development
• You need to trigger Index update manually
• It doesn't Incremental index update at the moment

They say the first post is the hardest

By Daniel Tintner on June 18, 2009 @ 02:44 AM

Hello

My name is Taylor (Tai Loi) Luk and welcome to my personal blog, Here you will find more about projects that i am working on and things That i care.

About

I am residing in Sydney, Australia and starting a small company "Idealian" in late 2008, Since 2 years ago I started working on couple side projects, for the days to come, I going to take off the "side-project" label and they will be my primary focus.

Start a personal blog is something i wanted to do for a while, at the same time I am building a hosted content service. One day this idea strikes to me

why not host my own blog on the system that i am building.

Topics

I will try to maintain a balance between technical and personal writing covering topics

• Programming related topics ie: Ruby, Rails, PHP and Javascript
• Experiments and
• Project related topics such as SiteRest, H2o, Shop2

Until next time friends...