Elastic Applications in Erlang

How realistic or useful are elastic applications in general, or specifically in Erlang? To demonstrate, I start with Joe Armstrong’s favorite Erlang program – the universal server:

universal_server() ->
    receive
        {become, F} ->
            F()
    end.

Once spawned, it sits and waits until it gets an instruction to become something. Then it becomes that thing it’s told to be.

First, this is really cool. Second, we should expand a bit:

  • make the wrapping process loop
  • optionally make the internal function loop
  • if internal function receives messages, make it respect an “abort” message which would make it stop executing and return the control back to its parent

So, blah blah – all of the above is only few lines of code in Erlang. I’ll call this module gen_node and the gen_node.erl file is as follows:

-module(gen_node).
-export([start/0]).

start() ->
    receive
        {become, F} ->
            F(),
            start();
        reset ->
            start();
        stop ->
            ok
    end.

and then the way we invoke it would be:

1> c(gen_node).
{ok,gen_node}
2> N = spawn(gen_node, start, []).
<0.40.0>

OK, the process is spawned and waiting, let’s give it something to do. Let’s define 2 functions: one that doubles the given number, and another that squares it. They both communicate via messages:

3> Double = fun Double() ->
    receive
      reset -> ok;
      {From, Args} -> From ! Args+Args, Double()
    end
   end.
#Fun<erl_eval.44.106461118>
4> Square = fun Square() ->
    receive
      reset -> ok;
      {From, Args} -> From ! Args*Args, Square()
    end
   end.
#Fun<erl_eval.44.106461118>

Now, let’s tell the spawned process to become “Double”, then reset it and tell it to become “Square”:

5> N ! {become, Double}.
{become,#Fun<erl_eval.44.106461118>}
6> N ! {self(), 16}.
{<0.33.0>,16}
7> flush().
Shell got 32
ok
8> N ! reset.
reset
9> N ! {become, Square}.
{become,#Fun<erl_eval.44.106461118>}
10> N ! {self(), 16}.
{<0.33.0>,16}
11> flush().
Shell got 256
ok

So, this is cool because now we have a generic computing node that we can tell to transform into any arbitrary processor and communicate to it via messages.

Now, finally, let’s wrap this up in an OTP application using gen_server so the worker processes are also supervised:

https://github.com/unix1/gen_node

Now the interaction becomes:

1> Double = fun Double() ->
    receive
      {_, reset} -> ok;
      {From, Args} -> From ! Args+Args, Double()
    end
   end.
#Fun<erl_eval.44.106461118>
2> Square = fun Square() ->
    receive
      {_, reset} -> ok;
      {From, Args} -> From ! Args*Args, Square()
    end
   end.
#Fun<erl_eval.44.106461118>
3> application:start(gen_node).
ok
4> {ok, N, _} = gen_node:start_server().
{ok,<0.42.0>,#Ref<0.0.0.45>}
5> gen_node:become(N, Double).
ok
6> gen_node:send(N, 16).
32
7> gen_node:reset(N).
ok
8> gen_node:become(N, Square).
ok
9> gen_node:send(N, 16).
256

It worked!

So, what’s next?

This is obviously just a concept code. To be usable something needs to track the states and types of nodes, and something else to intelligently control what they are doing and how the behavior should adapt. Possibilities could include:

  • predicting application-specific demand and adjusting resources: e.g. queue X is filling up, while queue Y processes are waiting too long – so make an adjustment in real-time
  • defining capacities of systems by giving different weights to each type of operation, or calculating resources needed to make certain computations
  • more fully using existing resources, or distributing load where the right resources are available

Questions: is this applicable and interesting? Boring? Already been done? Do you have any ideas which direction this should take?

Any constructive feedback is fully welcome.

Posted in Erlang | Tagged | Leave a comment

PHP Sessions in Erlang Mnesia

Motivation

  • create a very simple first Erlang/OTP application
  • link to conventional web development

I decided one of the simplest things to do would be to create a session storage for PHP in Mnesia. But that, in doing so, I would also create an extremely simple key value store wrapper around Mnesia which would track access times.

Disclaimer

Do NOT use this in production, or anywhere where it’s important. If you do, you’re crazy because:

  • it has not been tested in production or production-like environment
  • as I mentioned, this is my first Erlang application
  • connection is made to a single Erlang node, and there is no failover mechanism if that fails
  • session garbage collection is not implemented

Good Stuff

OK, the disclaimer is out of the way, let’s get to the good stuff! Here’s what I did and how.

Tools used

  • Erlang runtime and development headers
  • PHP >=5.3 (>=5.4 preferred) binary and development headers
  • Apache
  • mypeb
  • kvstore
  • lib360

Architecture

[ PHP ] <–> [ mypeb ] <–> [ Erlang ] <–> [ kvstore ] <–> [ Mnesia ]

Instructions

  • download and install prerequisites as best done in your environment
    • Erlang runtime and development headers
    • Apache
    • PHP >=5.3 binary and development headers
  • download and compile mypeb
    • git clone git://github.com/videlalvaro/mypeb.git
    • cd mypeb
    • phpize
    • ./configure
    • make
    • sudo make install
    • php -m | grep peb
    • Note 1: the last command is a test to verify peb module loads successfully
    • Note 2: you might need to specify --with-erllib and --with-erlinc options with ./configure command if they are not automatically found
  • download, compile and run kvstore
  • download lib360
    • git clone git@github.com:unix1/spoof.git
  • add sample PHP script to your web server and test everything
    • download sample gist
    • replace the path in require_once() to wherever you downloaded lib360
    • replace the your-erlang-cookie-here with the contents of ~/.erlang.cookie file
    • access the file through the web server and have fun!

Missing

These are things that I thought of but didn’t bother implementing that I might someday add:

  • make kvstore more configurable and easy to install/start/stop
    • kvstore currently fails to stop (you have to abort)
    • it could use rebar and more automated scripts instead of manual commands
    • support for configurable multiple table names
    • make access time tracking optional
    • implement other storage backends
  • account for failover when primary node connection fails from PHP
  • PHP session garbage collection
  • implement kvstore access through lib360 database layer

Suggestions/Contributions/Feedback

If you have any feedback, feel free to give it via github, blog comments, etc. as appropriate. Enjoy!

Posted in Erlang, PHP, programming | Tagged , , , , | Leave a comment

How To Add QML Module with Plugins for Qt Creator

I decided to take a plunge into a KDE Plasma development with a simple plasmoid with Qt Creator. There’s a very good overview and a guide

However, you will notice that when you do this in Qt Creator

import org.kde.plasma.core 0.1 as PlasmaCore

you will get an underline and the following error

QML module not found

followed by import paths and some vague tip about QML_IMPORT_PATH and importPaths property. What this means is that you need to explicitly declare the import path for KDE components in your *.qmlproject file. Specifically, directly in/under the Project container add the following property

importPaths: [ "/usr/lib64/kde4/imports" ]

or whatever the case is for your specific environment. Now switching back to your .qml file you will notice that the red underline is still there but the message is different when you hover over it

QML module does not contain information contained in plugins

First, the QML can be run at this point. Second, this is because KDE has not included plugins.qmltypes files with their Plasma components. To generate them yourself you can run a command

qmlplugindump org.kde.plasma.core 0.1 /usr/lib64/kde4/imports > /usr/lib64/kde4/imports/org/kde/plasma/plugins.qmltypes

substituting your paths, if different. You’ll note that your user probably doesn’t have privileges to write to that directory – so either run the command as root or generate the file in another directory and then copy it.

After refreshing/reloading the project in Qt Creator you will see the red underline has disappeared and Qt Creator is in a happy state. You can use this method with other components with missing plugins.qmltypes files; and use this with your custom C++ components as well.

Posted in programming, Qt | Tagged , , , , , | Leave a comment

Simple PHP Object Oriented Framework (spoof) documentation

I got some time to nearly fully document and run doxygen against spoof. The documentation is located at docs.codefly.org and is also linked from http://codefly.org/lib360/

Few select links to documentation:

Doxygen is awesome. Enjoy.

Posted in programming | Tagged , , , | Leave a comment

Simple PHP Object Oriented Framework

This marks the initial release of Simple PHP Object Oriented Framework – spoof on codefly.org. Well, I intend for this to be a framework, but I’m starting with the extensible core – the main class library. To that end, the first crack is at:

  • autoload
  • data access components

Autoload is a simple implementation for spl_autoload_register:

  • one class per file
  • \namespace1\subnamespace\MyClass automatically turns into [root_dir]/namespace1/subnamespace/MyClass.php
  • you never have to use require_once again

Database components are a more significant undertaking, as they are designed to be language, connection and execution agnostic.

Finally, currently both PHP 5.2 and 5.3+ are supported. However, I am making a full use of PHP 5.3 namespaces which are not backward compatible. I would expect to continue work on 5.3+ version only.

Please check it out and feedback is more than welcome.

Posted in programming | Tagged , , , , , | Leave a comment

PHP array_merge vs plus union Operator

The purpose of this post is simple: I’ve seen way too many uses of array_merge function with associative arrays in PHP. In most cases the “+” union operator should have been used and here’s why. Given:

<?php
$a1 = array('abcd' => 'value 1');
$a2 = array(1234 => 'value 2');

var_export(array_merge($a1, $a2));
var_export($a1 + $a2);
?>

The result is as follows:

array (
  'abcd' => 'value 1',
  0 => 'value 2',
)
array (
  'abcd' => 'value 1',
  1234 => 'value 2',
)

I bet the latter is what you would need most of the time. Let’s list the behavior of the two methods:

  • non-matching string keys: simplest case – both methods append all elements and produce merged result
  • matching string keys: array_merge overrides from latter array; union operator keeps element from the former
  • non-matching integer keys: array_merge re-keys latter array(s) as integer keys are available, starting from 0; union operator leaves original keys
  • matching integer keys: array_merge re-keys and appends; union operator keeps element from the former array

Also consider the following:

<?php

$key = "1234";
$a = array();

$a[$key] = "abc";
$a["$key"] = "abc";
$a["" . $key] = "abc";
$a[(string)$key] = "abc";

?>

All of the above will result in an integer key! PHP will force anything that looks like an integer into an integer even if you explicitly specify a string. From PHP array documentation:

If a key is the standard representation of an integer, it will be interpreted as such (i.e. “8″ will be interpreted as 8, while “08″ will be interpreted as “08″).

This means all integer rules will apply when using these elements with array_merge – i.e. all of them will be re-keyed. Conclusion: if you are using associative arrays, you most likely need the “+” union operator to merge arrays and not the array_merge function. Just be mindful of the order in which they are added.

Posted in programming | Tagged , , , | Leave a comment

Amazon Finally Releases Android Kindle

It wasn’t a tough call or anything but back in March I did guess that Amazon was planning a Kindle running on Android. Why else would they start their own app store? Now that their strategy is actually taking shape it unsurprisingly presents Amazon-skinned Kindle Fire Android device geared towards its users consuming more Amazon content, services and products. For this purpose, the tablet is very competitively priced – at the time of writing a full $50 less than Nook Color ($199 and $249 respectively).

One can find many reviews of Kindle Fire online. It has also been a frequent comparison subject against Nook Color. Because of the price difference, it compares favorably. However, if I had to pick between the two I would likely go with the Nook Color because of better specs (double RAM, SD card slot, etc.), better support for standards, and just my general discontent for Amazon and its business practices.

Posted in android | Tagged , , , | Leave a comment