Spinlock - The ramblings of a geeky type

Simple Servo actuation in ROS

2012-02-16T06:55:00.000-08:00

With the turmoil from Christmas, New Year, Australia day and children going to 'Big School' behind me my project needed momentum. After installing fresh copies of Ubuntu and ROS on both desktop and netbook (nice and easy http://www.ros.org/wiki/ROS/Installation) I set about trawling through the ROS documentation and tutorials and other peoples source code in an attempt to figure out how it all fits together.

Of course nothing rolling up the sleeves and wiggling the front wheels seemed like the perfect 'Hello World' style verification that everything was working well.

Using the Pololu board to drive the cars servo's proved refreshingly easy. While windows required some basic FTDI drivers, plugging the Polulo into Ubuntu automagically loads a standard Serial over USB driver.

[12425.073822] usb 3-3: USB disconnect, device number 4
[12426.996370] usb 3-3: new full speed USB device number 5 using ohci_hcd
[12427.173388] cdc_acm 3-3:1.0: This device cannot do calls on its own. It is not a modem.
[12427.173451] cdc_acm 3-3:1.0: ttyACM0: USB ACM device
[12427.178486] cdc_acm 3-3:1.2: This device cannot do calls on its own. It is not a modem.
[12427.178552] cdc_acm 3-3:1.2: ttyACM1: USB ACM device

Pololu have a compact serial protocol for the maestro range of serial controllers (http://www.pololu.com/docs/0J40/5.e) making control as simple as writing a command to the serial device /dev/ttyACM0. Initially nothing happened, but reading further in the documentation revealed that I had to use the Pololu control centre to set an appropriate serial mode (http://www.pololu.com/docs/0J40/5.a) after that the following code got the wheels to wiggle back and forth.

#!/usr/bin/python
import math, time, serial

def get_command(channel, target):
        target = target * 4
        serialBytes = chr(0x84)+chr(channel)+chr(target & 0x7F)+chr((target >> 7) & 0x7F)
        return serialBytes

ser = serial.Serial('/dev/ttyACM0')
ser.write(chr(0xAA))
ser.flush()

i=0.0
while(i<2*math.pi):
        i = i+0.01
        print math.sin(i)
        ser.write(get_command(1,int((math.sin(i)*300)+1500)))
        ser.write(get_command(0,int((math.sin(i)*300)+1500)))

        time.sleep(0.002)

ser.write(get_command(1,0))
ser.write(get_command(0,0))

ser.close()

There are two writes in each loop as the RC car I'm using has two steering servo's.

After browsing through a variety of other peoples ROS stacks, (PI Bot, Turtlebot, ART) I found that the Austin Robotics Technology stack written for their Marvin DARPA autonomous car entry was structured well, and included a more sophisticated set of driving messages for driving a car style robot. By basing my efforts around the way ART laid out their stack I get the benefit of a good node structure, but also exposing me to issues ART have had to consider relating to larger cars and I want my project to be expandable to ATV (or larger) sized vehicles.

Using my very rusty Python and my still peripheral understanding of ROS I was able to convert the demo above into a ros node capable of consuming the ART steering message which meant;

I could use ART's test_steering node from the art_servo package to generate test steering messages for me.
I can run the pololu driver on my netbook and use the ROS_MASTER_URI environment variable to run roscore and the steering test node on my desktop.

While it doesn't make much sense to run these nodes across multiple computers mode, it will come in very handy when teleoperating the robot, or processing complex data from the kinect.

With all that grooviness baked in I'm feeling much better about leaving behind the comfortable world of C# and the mindbogglingly complex world of writing Microsoft Robotics Services manifests.

#!/usr/bin/env python
import roslib; roslib.load_manifest('wombot_servo')
import rospy
import serial

from art_msgs.msg import SteeringCommand
from art_msgs.msg import SteeringState


class PololuDriver(object):
    def __init__(self):
        rospy.init_node('pololu_driver')
        rospy.on_shutdown(self.ShutdownCallback)
        rospy.Subscriber("steering/cmd",SteeringCommand, self.SteeringCallback)

        port = '/dev/ttyACM0'

        try:
            self.ser = serial.Serial(port)
            self.ser.open()
            self.ser.write(chr(0xAA))
            self.ser.flush()
        except serial.serialutil.SerialException as e:
            rospy.logerr(rospy.get_name()+": Error opening or initialising port "+port)
            exit(1)

    def DegToTicks(self,degrees):
        #quick and dirty,   assume 30 is near-extent
        ticks = (max(-30,min(30,degrees))/30.0)*480
        return int(ticks + 1500)

    def run(self):
        rospy.spin()

    def get_command(self, channel, target):
        target = target * 4
        serialBytes = chr(0x84)+chr(channel)+chr(target & 0x7F)+chr((target >> 7) & 0x7F)
        return serialBytes

    def ShutdownCallback(self):
        rospy.loginfo("Shutting down")
        if hasattr(self, 'ser'):
            self.ser.write(self.get_command(0,0))
            self.ser.write(self.get_command(1,0))
            self.ser.write(chr(0xA2))
            self.ser.close()

    def SteeringCallback(self,data):
        ticks = self.DegToTicks(data.angle)
        rospy.loginfo(rospy.get_name()+": Request Type="+str(data.request)+" Angle="+str(data.angle)+" Ticks="+str(ticks) )
        ticks = self.DegToTicks(data.angle)
        self.ser.write(self.get_command(0,ticks))
        self.ser.write(self.get_command(1,ticks))


#Init and run
PololuDriver().run()

GPGPU approaches to filtering operations

2011-10-16T15:04:00.000-07:00

In fiddling around with the Kinect it has become more obvious that I will need to improve performance.

The Microsoft Reserach KinectFusion video shows capabilities that would be perfect for my honours project, creation of a smoothed 3D surface in real-time with decent reaction to transients in the scene, if only I had a few Microsoft Research and UWash prof's in my pocket.

Kinect Z Buffer noise
Borglabs - Creating Pointclouds (.ply) files - Attempt to remove noise

While an optimal filter hasn't emerged from either my own testing or others I will need to be filtering the depth data by evaluating neighbors in the space, time or both domains (the KinectFusion is at least operating over the time domain as shown by the delayed influence of changes in the scene.

The KinectFusion algorithm, as well as probabilistic techniques such as those based on Poisson and Bayesian methods require surface normals for each point, which the Kinect does not provide (so far as I know). So these will also have to be calculated based upon the neighbors for each point.

So just to create a surface we are already performing multiple operations per 'point' each requiring multiple calculations, including perhaps a median filter, which requires a sort on each point. With the 76,800 points in each 240x320 depth frame this is a problem begging for parallelism. Which, based on comments from the KinectFusion video, is exactly what they did.

I would prefer to keep my application in C# if at all possible partially because I'd like to demonstrate the idea that a CLR program is quite capable of sophisticated graphical programming, but also because C++ feels like a legacy from a past life for me now (Line-Of-Business apps being my bread and butter), which means that I have a few options for increasing my parallelism.

Standard C# multi-threading
.NET Parallel libraries (with functions like parallel.for() )
Microsoft Research Accelerator Library (allows x64 multicore, or gpu acceleration)
High Level Shader Language (HLSL), which the XNA framework can leverage
Linking to an unmanaged library to user pixel shaders (.pix) files
Various CUDA libraries, include CUDA.NET

I have a chunky GeForce GTX560Ti graphics card with 384 CUDA (general purpouse gpu) cores, and graphics cards are all heading in this direction, so I would like to use the gpu.

The MSR Accelerator library looks to be the easiest method to implement, however GPU's are all about how to get data into and out of the graphics cores, ensuring the right data is in the right place is the key, Accelerator is probably very clever about how it handles this task, but in using the library I can't manage that myself.

I will have to create an implementation of the an algorithm, probably the median filter function (unless more details on the filter in KinectFusion get released) in

Normal Code
Microsoft Research Accelerator
HLSL Pixel shader

And test run time over a large number of calculations.

"Robot Platform" (air-quotes intended)

2011-10-13T10:31:00.000-07:00

I bit the bullet and purchased a robot platform, a Traxxas Summit.

The Summit has a number of advantages over the cheaper similar sized rc vehicles such as remotely lockable center and front center diffs, waterproofing, independent suspension (easier to simulate and closer to ATV's, a potential future research platform), has a wide stance (useful to shield the wide Kinect sensor), and should be large enough to carry a variety of control and sensor hardware.

It's also pretty cool :)

Stored procedure for importing data from Oracle to Sql Server

2011-08-30T16:36:00.000-07:00

The approach to ETL we have taken for our Data Warehouse involves staging a lot of tables before transforming into our star schema. There are a lot of fancy tools in the SQL Server and Oracle worlds for capturing change within databases, however sometimes as a lowly programmer / DW designer you just have to do it yourself.

Many of our tables don't change from day-to-day, so it doesn't make any sense to import them if they're not needed. creating a package based on the dtswizard is easy enough, but harder to change once saved. I ended up creating a little stored procedure to determine whether a table has changed and import it if it hasn't. Using this procedure assumes you have the oracle client installed, are using TNS, and assumes a more recent version of Oracle (I think 9+) for the lastscn functionality.

The stored proc will create two tables in your database.

once the sp is in the db you can simply create a file with lines like the following



EXEC procImportIfChanged 'PROD_CRD','PA_INF_ADMISSION'

EXEC procImportIfChanged 'PROD_CRD','PA_INF_AGEGROUP'

EXEC procImportIfChanged 'PROD_CRD','PA_INF_ASCED'

EXEC procImportIfChanged 'PROD_CRD','PA_INF_ASGC'

EXEC procImportIfChanged 'PROD_CRD','PA_INF_ATAR'

EXEC procImportIfChanged 'PROD_CRD','PA_INF_ATTENDANCE_MODE'

EXEC procImportIfChanged 'PROD_CRD','PA_INF_ATTENDANCE_TYPE'

EXEC procImportIfChanged 'PROD_CRD','PA_INF_ATAR'

EXEC procImportIfChanged 'PROD_CRD','PA_INF_ATSI'

Remember, I never said this was good, just that it works :)

--------------------------------------------------

-- procImportIfChanged

--

--    Check the number of rows in the table and

--    the 'last changed row scn' to establish

--    whether the table may be stale.

IF EXISTS (SELECT name

           FROM   sysobjects

           WHERE  name = 'procImportIfChanged')

  DROP PROCEDURE procimportifchanged;

GO

CREATE PROCEDURE Procimportifchanged @db_name  VARCHAR(128),

                                     @tbl_name VARCHAR(128)

AS

  BEGIN

      DECLARE @table_stale BIT

      DECLARE @local_table_rows INT

      DECLARE @local_table_scn BIGINT

      DECLARE @remote_table_rows INT

      DECLARE @remote_table_scn BIGINT

      -- consider table stale until proven otherwise

      SET @table_stale = 1

      -- Create table change data tracker table thingoe if it doesn't exist

      IF NOT EXISTS (SELECT name

                     FROM   sysobjects

                     WHERE  name = 'procImportIfChanged_TableData')

        BEGIN

            CREATE TABLE procimportifchanged_tabledata

              (

                 database_name        VARCHAR(128),

                 table_name           VARCHAR(128),

                 local_table_crdate   DATETIME,

                 table_rows           INT,

                 table_lastscn        BIGINT,

                 latest_table_rows    INT,

                 latest_table_lastscn BIGINT,

              );

        END

      -- Create table change data tracker table thingoe if it doesn't exist

      IF NOT EXISTS (SELECT name

                     FROM   sysobjects

                     WHERE  name = 'procImportIfChanged_ImportLog')

        BEGIN

            CREATE TABLE procimportifchanged_importlog

              (

                 database_name   VARCHAR(128),

                 table_name      VARCHAR(128),

                 import_datetime DATETIME

              );

        END

      -- get the remote table rows and scn

      DECLARE @qryTableInfo NVARCHAR(4000)

      SELECT @qryTableInfo = N'select table_rows,lastchange_scn from OPENQUERY('

                             +

                             @db_name +

      ',''select count(*) table_rows,max(ora_rowscn) lastchange_scn from ' +

                      @tbl_name + ''')';

      DECLARE @remote_table TABLE (

        tbl_rows    INT,

        tbl_lastscn BIGINT)

      INSERT INTO @remote_table

                  (tbl_rows,

                   tbl_lastscn)

      EXEC Sp_executesql @qryTableInfo

      SELECT @remote_table_rows = tbl_rows,

             @remote_table_scn = tbl_lastscn

      FROM   @remote_table;

      -- If there's no row for this table in the config table,  create one.

      IF NOT EXISTS (SELECT 1

                     FROM   procimportifchanged_tabledata

                     WHERE  database_name = @db_name

                            AND table_name = @tbl_name)

        BEGIN

            PRINT @db_name + '.' + @tbl_name +

                  ' doesnt exists in info cache, creating row'

            INSERT INTO procimportifchanged_tabledata

                        (database_name,

                         table_name,

                         local_table_crdate,

                         table_rows,

                         table_lastscn,

                         latest_table_rows,

                         latest_table_lastscn)

            VALUES      (@db_name,

                         @tbl_name,

                         CONVERT(DATETIME, '20000101', 103),

                         0,

                         0,

                         @remote_table_rows,

                         @remote_table_scn);

        END

      ELSE

        BEGIN

            IF EXISTS (SELECT name

                       FROM   sysobjects

                       WHERE  name = @tbl_name)

              BEGIN

                  PRINT @db_name + '.' + @tbl_name +

                        ' exists in database and info cache table'

                  DECLARE @table_crdate DATETIME

                  SELECT @table_crdate = crdate

                  FROM   sysobjects

                  WHERE  name = @tbl_name;

                  IF EXISTS (SELECT 1

                             FROM   procimportifchanged_tabledata

                             WHERE  database_name = @db_name

                                    AND table_name = @tbl_name

                                    AND local_table_crdate = @table_crdate)

                    BEGIN

                        PRINT @db_name + '.' + @tbl_name +

                              ' creation date matches recorded information'

                        UPDATE procimportifchanged_tabledata

                        SET    latest_table_rows = @remote_table_rows,

                               latest_table_lastscn = @remote_table_scn

                        WHERE  database_name = @db_name

                               AND table_name = @tbl_name;

                        -- 'un-stale' the table if the number of rows, and last changed date are the same as recorded

                        IF EXISTS (SELECT 1

                                   FROM   procimportifchanged_tabledata

                                   WHERE  database_name = @db_name

                                          AND table_name = @tbl_name

                                          AND table_rows = latest_table_rows

                                          AND table_lastscn =

                                              latest_table_lastscn

                                  )

  BEGIN

      PRINT @db_name + '.' + @tbl_name +

            ' appears unchanged from local table, not re-importing table'

      SET @table_stale = 0

  END

        END

  END

        END

      IF EXISTS (SELECT name

                 FROM   sysobjects

                 WHERE  name = @tbl_name)

        BEGIN

            IF ( @table_stale = 1 )

              BEGIN

                  PRINT @tbl_name + ' is stale, dropping'

                  DECLARE @drop_sql NVARCHAR(4000)

                  SELECT @drop_sql = N'drop table ' + @tbl_name;

                  EXEC Sp_executesql @drop_sql;

              END

        END

      IF NOT EXISTS (SELECT name

                     FROM   sysobjects

                     WHERE  name = @tbl_name)

        BEGIN

            PRINT 'Loading table ' + @tbl_name

            DECLARE @load_qry NVARCHAR(4000)

            SELECT @load_qry = N'select * INTO ' + @tbl_name +

                               ' from OPENQUERY('

                               +

                               @db_name +

                                                  ',''select * from ' +

                               @tbl_name

                               +

                               ''')';

            EXECUTE Sp_executesql @load_qry

            UPDATE procimportifchanged_tabledata

            SET    local_table_crdate = crdate

            FROM   sysobjects

            WHERE  name = @tbl_name

                   AND database_name = @db_name

                   AND table_name = @tbl_name;

            UPDATE procimportifchanged_tabledata

            SET    table_rows = latest_table_rows,

                   table_lastscn = latest_table_lastscn

            WHERE  database_name = @db_name

                   AND table_name = @tbl_name;

            INSERT INTO procimportifchanged_importlog

                        (database_name,

                         table_name,

                         import_datetime)

            VALUES      (@db_name,

                         @tbl_name,

                         Getdate());

        END

  END

GO

Blood from a stone, or, JSON from ASMX Web Service

2011-07-11T22:32:00.000-07:00

Just a quick note for posterity.

If you've got an older ASP.NET website that you're just now trying to graft JSON Web Services into, and can only get XML out of it no matter how hard you try, there's a whole stack of stuff that needs to go into the web.config file.

The easiest way to get it into yours is to create a new web-site (just a file-system based one) and snip all the missing sections out of there.

Windows Phone 7 - It's biggest problems are it's little ones

2011-06-08T02:50:00.001-07:00

I've been living with the Windows Phone for a while now and I've watched a lot of the discussion about the phone's shortcomings yet it's the little things that no-one talks about that really prevent me from really enjoying the device.

When you Live outside Australia

Microsoft have always professed that Copy-Paste is coming, but that you don't really need it anyway, as browser recognises phone numbers and addresses and makes them clickable so that you don't have to copy these into the phone or maps app…

…unless you live outside the USA

…which a lot of us do.

And that's the point really. Unless you live in the US you don't get the premium Windows Phone experience. Andrew Birch took the time to make up a table which shows the tattered mess of services and feature around the world http://andrewtechhelp.com/andrews-tech-opinions/115-windows-phone-7-feature-availability-matrix

For example in Australia we can buy Apps (subject to regional pricing, lets not go there). There's no music, especially the awesome Zune-pass. You can't rent TV shows, but you can purchase and rent movies. The pattern is different for each country.

The story continues across the phone. Only the US has Bing Voice Search and Bing Traffic, only the US and UK have Bing Local Search (though both Local search and Voice search work if you perform locale gymnastics http://www.istartedsomething.com/20101021/workaround-local-listings-voice-search-wp7-australia/) The USA is also the only country to have clickable phone numbers and addresses.

In the online world, people who don't live in America are used to getting left out (Amazon music, Netflix or Hulu for example) so it's already a sore spot. But turning off local search when the listings are obviously there just rubs salt in the wound. And as a side note, because of the 'clever' stuff Bing does with tie-in's to all sorts of services, it doesn't work nearly as well when you aren't in the USA, and you can't change the default search engine.

Hubs

Hubs just don't live up to the promise. They're designed to stop the in-and-out of other phones, aggregating all your 'stuff' into task centric areas. So there's a photo hub which has all my photos. And it has my photos loaded from Zune, and those taken on the phone, and can integrate Flickr and I think one of the windows cloud services. But there is no Picasa integration, and that's where I keep my online albums, along with a few quadrillion other people. I remembered the rhetoric in the publicity about hubs being dynamic and modular and wonderful, so being a good windows programmer I thought "I'll write the hub add-in myself" The google API's are good but I searched the Windows Phone documentation and discovered that there is no ability for third party apps to integrate seamlessly into those hubs. Sorry to break it to you MS, without extensibility its not a hub, it's a photo app, and frankly I've seen better photo apps on iPhone.

Volume

Volume management is very rudimentary. It's totally non-contextual. When you start a noisy game and turn the volume down it stays down, and I miss the next couple of phone calls. Even more inexcusably if I plug in my headphones start some music I'll turn the volume down to save my poor eardrums. When I finish listening and pull the headphones out, the volume stays at the headphone volume, and I miss the next couple of calls. The keyboard is really loud. I can't type a full message without my cubicle-colleagues shooting angry looks so I have to mute the volume and occasionally forget to un-mute, and I miss the next couple of calls. I could turn key press sounds off altogether, but I just want it quieter, by a fair bit.

Clocks

Upon release the phone had no timers, stopwatches or world clocks, a few weeks after release some Microsoft first party apps appeared on the store to add these features, but they're separate apps. They are pretty but have inconsistent UI's, they generally sit over on the loooong apps list, and provide a more in-and-out experience than you'd find on... say, iPhone.

FM Radio

I like being able to listen to FM radio, I don't commute on public transport anymore, so I can dig all the ABC Radio National I need on my car stereo, but I would like this feature. I can bookmark radio stations so that I don't always have to remember the frequency and dial them up except... I do. Because I can't actually name a bookmarked station.

Data Counters

I haven't found any. Only way to know if you're usage is crazy (for example, if you have a Yahoo email account and it's churning through data like crazy) is when you get the bill at the end of the month. Given the nature of mobile data plans it may be a very large bill.

Personalisation

I can move the tiles around a bit, but I've only got a couple of colours to choose from. And I don't really like any of them. Who would it hurt to allow the use of whatever colour I like. It's my own fault if I choose Poo Brown, actually I think Poo Brown is one of the ten colours.

Ringtones. A smartphone without custom ringtones. Really ?
The supplied tones (ring - message - alarm) are all roughly the same 'theme'. I'm not sure about android, but the iPhone provides a good collection of distinctive ringtones (which you then hear every-where due to the sheer number of the things around)

The Bucket list

The home screen is genuinely an attempt to do things a little differently, but real-estate is at a premium and only a few things can fit on there. Everything else goes onto an alphabetical list on the next screen, and it gets very long and ungainly and most of the icons are in your theme colour.

A touch of Flair

Microsoft has some good graphics people. People love the Bing backgrounds, and that make it onto the phone (though I've come to use the google search app), the Zune app is quite pretty, and when the stars align and you see one of those lovely large artist photo's in the background it's a nice experience, and the Panorama controls in the 'hubs' look great.

Photos Hub is where your personal flair is really supposed to come into it's own, I do get the background picture but there appears to be some kind of Bayesian intelligence ensuring that with all the awesome photos on my phone it only ever chooses the most boring. I'd love to point it at a folder and say 'there, I've got some pretty pictures in there choose from that lot', but that's not an option.

People tiles on the home screen are great. I call and text my wife a lot and not having to find her on the list each time is good. (There is some kind of cognitive dissonance here as I still find myself going into the list half the time, I'm a programmer and gamer and a geek first class, so I should be able to adapt). But with the funky animation on the tiles, the persons name is only displayed half the time. The rest of the time it uses the contact picture, which is updated from one of the integrated online services like hotmail or facebook (and I love how it grabs contact details and piccies from facebook). However when the contact behind one of those smart tiles changes their profile picture to their cat, or child's painting, or Master Chief you have to scroll, stop, wait for the live tile to reveal a name, repeat.

Updates

Feb update in April, The update that was supposed to be in December became the March update which I got in May. they promise lots of goodies in mango, but I just think "When will I acutally see Mango"

API's

There are no apps that use the camera for cool stuff like augmented reality or bar-code scanners because there are no API's to access the camera, or the gyro's. Apps like Skype cannot work because programmers can't use sockets (raw network communication). Many of these things are slated for Mango, but that’s quite a wait.

Calendars

I can still only link one, I have a personal calendar a work calendar and a calendar I share with my wife. When Apple got Google calendar sync working well this was an awesome and well implemented feature.

And On

I could go on (and on). Windows Phone is a novelty to most people, so I'm often asked what it's like and I that's a question I can't answer without qualification. Using the thing day-to-day it's fine, the phone works well enough it's not as well designed in many ways as an iPhone, but it's biggest problem are it's little ones. There are so many little rough edges that need polishing, and they not only haven't been polished, they're not even in the language for the Mango release.

When I got an iPhone 3G there where rough edges but once every two or three works an update would appear (a great big ungainly update with ambiguous or non-existent patch notes) I knew that the rough edges where getting looked at and improved. 6 months later I had a phone where a lot of the niggles where gone, 12 months later it was a very polished experience. In comparison, 6 months into owning a windows phone I was wondering why I couldn't get the only released update for my phone, and why there where some new features, but no polish on the old ones.

Lets play with Robots Instead

2011-03-04T17:53:00.000-08:00

I had a constructive talk to Dr A again today. I related to him my disappointment regarding finding an appropriate honours topic in the DW/DM space. And came around to talking about my enthusiasm for all things Robotics. Which was fortunate as he's one of the coordinators for the (previously unknown to me) Newcrest Mining Research Laboratory, where they currently have students undertaking PhD's in Ground Penetrating Radar, Machine Learning approaches to Robot navigation and Image Segmentation algorithms.

Taking a good couple of seconds to think about it, I abandoned all thoughts of a dry, wordy honours project on the use of strategic information in a modern corporation and decided to play with Robots. :)

Dr A has directed me to Feng Lu who is working with the Robots in the lab, hopefully talking to him will give me some ideas on where I can help.

Searching for an Honours Topic

2010-11-23T17:34:00.000-08:00

My day job is building Data Warehouses for the Universities strategic information. I have a degree of latitude in how I shape the Data Warehouse and figured that doing my research in Data Warehousing / Data Mining would be a good way to tie my work to my research. So I set about calling a number of the Academics who had DW / DM on their research interests and was left a little underwhelmed.

Dr Zahid Islam spoke most authoritatively on the subject but is focused on a lower level than I am really interested in. If I where to perform Missing Value Imputation, or finding Data Clusters I would use one of the excellent and easy to use Microsoft Data Mining extensions for SSAS. I am more interested in new methods for ETL, methods for making the most of things that Computers can offer up in spades (Memory, Storage size) while living within other constraints (CPU, IO throughput). Ways to automatically adjust which attributes from a large set one should throw at the Data Mining algorithm, and better ways to integrate DW information into a large modern corporate institution.

I have emailed someone who lists Corporate Information as a research interest, but have not heard back as yet.

Contemplating research

2010-11-20T17:09:00.000-08:00

I've been planning on doing something more Academic for quite a while, assuming a topic would drop from the sky, like mana from... well as an Atheist gamer I suppose I get mana from killing monsters, rather than heaven, and I haven't killed many actual monsters recently.

In short, I've had a chat with some of the Academics and decided an Honours would be an appropriate sized chunk for me. I can do some research, slog through the paperwork and hopefully come out the other end with a few more letters I'll never remember to put after my name, and a feel for whether I can make it through a post-graduate research degree.

September 15th

2010-09-12T16:07:00.000-07:00

I'm looking forward to two might-be-announcements on Sep 15th. NikonRumours is suggesting that will be an announcement of the Nikon D7000 on Sep 15 and other rumours suggest that a HTC public meetup will be the venue for announcing a new Windows Phone series 7 device, personally I'm hoping a HD2 replacement, variously rumored to be the HD7 will be announced.

The problem with electronics today

2008-11-11T06:35:00.000-08:00

So I splashed out and purchased some of the hardware I need to get this show on the road. An MCU, a gyro (at $100/pop, I can buy the rest later) an accelerometer a brushless motor, and an ESC to drive it.

Being accustomed to software development, I am quite used to wierd little problems bloating even the simplest tasks out to more elephantine proportions. I was very surprised when this hardware that was supposed to keep me busy for months, just worked.

It's due, by no means, to any electrical prowess on my behalf. I can rightly attribute this quick success to the arduino and it's well written libraries.

When programming microcontrollers, I'm used to having to look up the hardware adress for a port, manually set the bitmask indicating the direction for all the pins on the port, make several mistakes with endian-ness and conversion between hex, decimal and binary, and then manually handle the shuffling of data in and out of the appropriate registers. And then once your code is right you've gotta put together the right combination of compilers, header and startup files, and cross your fingers that it all goes together like it should.

The Arduino in contrast makes all this stuff remarkable easy. If you want to interact with an I2C device (a form of two wire connection to other chips in the circuit) simply import the wire library and use the simple function calls, Voila, instant accelerometer. Want to read analog values from the gyro, hook it up and call AnalogueRead(pin_number).

I don't know if the ATMega will have the grunt for the final project (It should, it seems equivalent to the PIC from the mikrokopter and the AVR from the 1st gen uavp), and the ide is built for simplicity, it doesn't appear to handle multi file projects very well, if at all. But as a prototyping unit, it's almost like writing pseudocode. It's like childs play.

Which is probably the point of the post. If you're interested in playing with microcontrollers, or need something to form the brains of a simple robotics project. You can't go far wrong getting hold of a recent Arduino.

The other point being, now I've gotta go and buy more expensive sensors, a long time before I thought I would have to.

Gyro's and Linears and MCU's oh my!

2008-11-03T17:40:00.000-08:00

Gyroscopes

Gyroscopes measure angular velocity, that is to say, how fast they are being rotated around their axis. The quadcopter needs three of these to measure rotation around the roll, pitch and yaw axes. The development of gyro's is an interesting story, probably best left to Wikipedia. The punchline is that once cumberous heavy spinning masses, gyros are now available in a small semi-conductor package using a tiny vibrating element to determine rotational forces.

While gyroscopes do a good job of indicating very small rotations very quickly there is a problem. Gyroscopes don't indicate the difference between flat-level and the current position, nor even the angle rotated since the previous sample, they tell us the speed of rotation at a point in time. While this can be done very quickly, and velocity multiplied by time gives us distance (or angle) some factors limit the accuracy with which angle can be measured by a gyro over the long term

a) the velocity is not static, sampling the velocity very fast will give a small error because the system was not rotating at the same speed for the whole sample period

b) Gyro's have a limit of resolution, they may be able to tell you something is rotating at 0.458 rad/sec, but not 0.458793424957 rad/sec. In that example the reading will be out by 0.000793... rad/sec. While the effect of this is quite small for any particular reading when taking 300 readings a second, 18,000 a minute, small errors can add up.

c) Gyro's in general have a degree of drift over time, and due to temperature variation.

So gyroscopes drift over time. The quadcopter might fly perfectly for a little while then slowly start to tip and eventually flip over. Which brings us to

Accelerometer

Accelerometers measure the forces of acceleration acting on the meter. Used a lot in aircraft and racing to measure 'g-force' around corners and during acceleration and braking, also in the Wii-mote to great commercial success. While the standard use for an accelerometer is to measure, well acceleration, in autopilot type systems they have a very usefull property. Gravity always excerts a constant 9.8m/ss acceleration directly down. Accelerometers read this force like any other, and when the chip is relatively stationary does a great job of telling the system what angle they're at. In combination with gyroscopes you can make a pretty good approximation of which way the platform is leaning.

Wot' I Got

Looking around: the UAVP project, both new and old use the ADXRS300 from Analog Devices. Mikrokopter uses ENC-03J's from muRata. The other options around are the IDG-300 which seems like a great device as it provides two axes in one chip, however the UAVP people feel that it's has much worse accuracy than the ADXRS300. The final option MLX90609 from Melexis. It has a digital output, which is nice, but it's SPI, a 4 wire interface, and the accelerometers, and eventual motor speed controllers use I2C, a similar, but 2 wire protocol. If the MLX output in I2C it would probaby be a hands down winner as it looks to be in the ADXRS quality range, it's not, so I may as well go with something the UAVP people have experience with. The MLX also wasn't available from the Australian Sparkfun distributor.

The Accelerometer was pretty easy. Both project use a LIS3L variant, and I simply chose the one I could easily obtain. 3 axis with I2C interface.

I threw an Arduino Deicimila development board into the mix so I could experiment with the components, rather than spend all my time trying to wrestle with development environments and the microcontroller. The libraries that come with the Arduino are excellent. On a side note, if anyone has children keen on robotics or electronic hardware, I can heartily recommend the Arduino as a very cheap, yet extensible way to get involved with programming for microcontrollers.

I ordered the Arduino, a LIS3LV02DQ (article on interfacing with Arduino), a single ADXRS300 gyro from Ocean Controls and waited.

The gear arrived from Ocean Controls way faster than I was expecting. So I could start to play.

What's needed

2008-11-03T14:51:00.001-08:00

While a sensible person might get hold of one of the excellent quadcopter platforms mentioned in the previous post, you don't learn a whole lot by starting halfway there.

After a considerable amount of research (many others would consider it procrastination) I made my move and bought enough gear to start playing with some of the hardware required.

The Quadcopter platform requires motion sensing. 3 axis of gyroscope and 3 axis of accelerometer are needed as a base. Extra sensors can be added to improve stability and functionality, commonly these are a compass for solid long-term yaw stability, a pressure sensor for altitude measurement, gps for position hold, come-home and flight mission tasks, along with a whole variety of other sensors for a variety of purposes.

It requires motors, counter rotating propellers and brushless motor speed controllers for those motors. ...Times 4

It requires a frame. there are some great looking frames out there, but most people end up using aluminium box section tubing initially.

And finally it requires the brains to tie it all together. There are a whole variety of excellent microcontrollers easily available with development boards on the market. This wasn't so much the case untill recently, but the emergence of highly affordable microcontrollers and semiconductor devices has allowed the emergence of a healthy hobbyist community.

I have decided to get hold of a microcontroller, a single gyro, and a chip which contains all three axis of accelerometer. This will allow me to exeriment with connecting these devices and providing telemetry back to a computer. But more on that in the next post.

(Picture from Sparkfun, http://www.sparkfun.com/commerce/product_info.php?products_id=395)

Quadcopters

2008-10-31T01:21:00.000-07:00

(Picture from Amir - blog at http://amir.ch/weblog/)

Quadcopters are cool, geeky cool. If anything epitomises the opportunities created by cheap, powerfull computing it's the ability for hobbyists to launch these gangly looking things into the air and have them stay there on sheer computational grunt.

Quadcopters are different to other RC aircraft like helicopters because the quadcopter is naturally unstable, without computer control any imbalance between the power of the motors would flip and crash the 'copter much more quickly than a person could respond.

This inherent instability means that you need a sophisticated control system, and because the computer does most of the work they are very simple to fly. And given more smarts, fly themselves.

http://www.uavp.ch/
http://www.mikrokopter.de/

iPhone 3G, July 11

2008-06-29T18:19:00.000-07:00

So the iPhone 3G was announced at WWDC and we're (Australia) getting it. The keynote was pretty impressive containing a great showcase of a wide variety of upcoming apps. I like the information-at-your-figertips aspect of the medical imaging app shown, but frankly I geek'ed out more over super monkeyball. For those that didn't see the demonstration, super monkeyball is a game in which you tilt the game world around to guide a monkey in a rolling ball through a series of puzzles, the clincher with the iPhone implementation being that to tilt the world, you tilt the iPhone.

It certainly seems to be gunning for enterprise. It has a variety of collaboration features such as push email and active-sync. I didn't foresee that Apple would be gunning for some of RIM's Blackberry customers. It has been commented upon that Blackberries aren't all about the features but the flexibility of deployment in an 'Enterprise environment' encrypted storage, roles based activation of phone features etc. However alot of Blackberry users just like the ability to get email on the fly, and may find the large screen and streamlined user experience 'UX' quite compelling.

Also in the firing line is the obvious contender, Windows mobile phones. The SDK, the app store (simplified commercialisation especially) and the huge installed userbase with a common platform make the iPhone an attractive development target. I think there will be a plethora of high quality apps developed for the platform (and of course no small amount of really horrible software aswell). Despite the great promise of iPhone development I am still conscious of a problem with the whole 'developing for the iPhone' deal. Though it is less a 'problem' per se' as it is the result of a conscious decision.

Everyone who has ever had to reinstall a fresh version of window knows that the worst enemy of a smoothly running OS is a whole bunch of installed software, bad software that poo's all over the registry, puts files into bits of the OS it shouldn't, and doesn't uninstall itself properly. The same problems afflict Windows mobile phone, software you stick on there for a giggle slowly clogs up the poor little thing and reduces it's performance more and more, untill you drop it on the ground, it hard-resets, and after a resync, things are good again for a while.

iPhone's app store gets around this by only letting you install decent software, vetted and proven to not wreak havoc and destruction. A great idea, except that you can bet some of the grooviest software probably wont be on there.

An example would be software that lets you tether your phone so you can use it as a modem for your laptop. Apple don't provide that feature on purpouse. It's easier to convince mobile carriers to offer large data cap plans / unlimited plans, when they know that you'll only be downloading songs and web-pages, and not 5GB Age of Conan install files. The amount of software that would be censored by Apple is probably small, but the fact is that unless you jailbreak your iPhone (another warranty-voiding kettle of fish altogether) Apple will be telling you exactly what you can and cannot put on your iPhone. It's called a walled garden, it's pretty and safe inside, but you're subject to the whims of an unapologetically commercial company.

It also potentially makes in-house development of custom applications difficult, though I can't speak with more authority because the latest releases of the SDK need OSX 10.5 and I've only got 10.4 on the trusty G4.

My final gripe of the day is price. Yes the iPhone is a premium piece of hardware, but lets face it, Steve made a pretty big thing out of price at the keynote. I find it difficult to swallow that the iPhone is 'cheap' when most of the plan details from around the world so far show that the monthly spend will be close to half the up-front cost. If the iPhone is more expensive than it was before, and your hiding the price in a subsidy paid through the term of the contract, that's okay, but you shouldn't then get up on stage and brag about how 'cheap' you've made the iPhone because you havn't. It's just a different kind of expensive. I wanna buy one outright for $600-$700, toss it on a $15 voice cap, and pay another $15-$20 for enough data that I'm not worried about doing a bit of web-browsing, email check and map searching. And while there are no details on Optus and Vodafone plans, I'm pretty sure there wont be one of those.

I do actually like the thing. I'm considering making it my phone for the next 2 or 3 years. I like it's big, clear, relatively tough screen, it's fluid and responsive UI, and there fact there will be a programmable mobile device with a meaningfully large userbase. But I figure someone has to have a poke at it.

Animoto video

2008-05-06T21:36:00.001-07:00

Animoto is a funky service that someone referred to on one of Leo Laporte's TWiT sites.

It's pretty groovy, it makes pro looking slideshows from your piccies. It's simple too, toss in 15 piccies and choose some music, either your own, or some surprisingly good options of theirs and it does the rest.

There's a longer option you can make for $3, and it's billed through google checkout. Seems like a funky idea with a good business model

iPhone - just add smart

2008-04-15T05:36:00.000-07:00

(image stolen from http://www.apple.com/iphone/)

It's good that there's a company that does things the way Apple does. I like nice design I like polished interfaces. I like when it's a good product like the Titanium / Aluminium Powerbook, like the mac-mini. I don't like when they try to make up for lack of features with marketing.

The iPhone was marketed heavily as a smart-phone, more a competitor to PocketPC's than Nokia's and I told everyone I could that this was just garbage. Yes it was pretty but if you can't write custom apps for it it's not a smart-phone, it's not even a mildly-clever-phone.

However (<- that's a big however) now Apple has released the SDK and from a really high-level perusal it looks quite reasonable. Which means iPhone has gone from 'sit in the dunce corner pretty boy' to much more respectable piece of serious-geek-equipment. I'm not talking Leo Laporte geek here, I'm talking Don Box geek (I know he's a Microsoft guy, but I can only talk from my perspective).

It still feels like Apple want alot more involvement in your application development than, say Microsoft, Java or Symbian desire, there seems to be a little 'ET phone home' about their distribution process. I'll be interested to see if you can distribute apps without hosting them with Apple, but since you can now get DRM free music from iTunes I'll give it the benefit of the doubt

This blog wasn't going when the iPhone came out, but just imagining it was, consider this a retraction. The iPhone is allowed to join the lunchtime chess-club. Let's just hope it can convince Windows Mobile to buy afew bits of Hugo Boss gear.

Weekend of raiding success

2008-04-14T23:56:00.000-07:00

Saturday was raiding day and we got 25 people together.

We started by trying to repeat our success the previous week on Magtheridon. It's alot easier now that the elementals don't decimate our group before the main game even starts. Cube clicking is much easier now that you don't have to synchronise multiple groups and manage backup clickers on 5 cubes all while trying to avoid cave-ins and dish out damage.

We one-shotted Magtheridon and, drunk with our success moved onto the Lurker Below (in Serpent Shrine Cavern). Despite half the group knowing how to deal with the fight, the rest of us flailed around like lost puppies. I spent most of one attempt just trying to get back up on my platform. Lurker Below: 3, us: 0. We'll try again later.

Sunday:

Up into Karazhan for the 4th time. Big success we one-shotted every boss except Opera (how humiliating). This was the first time the guilds new Aussie run has taken down all the bosses, so we expect to be getting much faster now.

I finally upgraded my Auction house special Valanos' Longbow. Kara wasn't kind enough to give me Sunfury or Wolfslayer, but the Steelhawk crossbow is good enough until I get 150 badges.

A succesfull weekends raiding.

That's some engineering for your gold

2008-04-14T23:29:00.000-07:00

So I was wondering what it 'costs' to level the engineering profession in WoW. Sure, lootables will lay it all out there for you, but I'm not the type to studiously auction off everything I create to minimize the cost drain. So I got rid of all but 100 gold from my character, ported to Iron Forge and went at it.

I followed Lootables suggestions where they made sense, but adjusted on the way. Some of the Lootables recipes are chosen they sell well on the AH. I didn't want to wait for stuff to sell so I vendored and trashed my way along.

The answer is around 170-180, that's where you get to with 100G on Uther if you're lazy. I bought a set of goblin jumber cables (7G out of the 100G total, I didn't say this was scientific) just in time for a quick Heroic Shattered Halls half run where I got to use them, and they worked, and... no-one realised that the humble hunter had saved the day (or at least the run back from Honor Hold). Never mind, I'll soldier away on my skill untill I can use the Goblin Jumper Cable XL and I can save the day, and people will notice, and cheddar will rain from the sky.

And next time I grace the shattered halls with my presence we will be taking one more toy.