tag:blogger.com,1999:blog-47460826957560187422023-11-15T18:49:13.612+01:00Exercises in Restful Integration and Continuous DeliveryAnonymoushttp://www.blogger.com/profile/00999479804876258493noreply@blogger.comBlogger10125tag:blogger.com,1999:blog-4746082695756018742.post-48019781349409804162016-01-02T19:21:00.000+01:002016-01-02T19:21:59.643+01:00Moved to a New AddressThis blog moved to a new address.<br />
<br />
<a href="http://www.koivunalho.org/blogs/exercises-in-integration-and-delivery">Exercises in Restful Integration and Continuous Delivery</a><br />
<br />
Welcome!
Anonymoushttp://www.blogger.com/profile/00999479804876258493noreply@blogger.com0tag:blogger.com,1999:blog-4746082695756018742.post-22514275496761603482014-09-30T10:11:00.000+02:002014-10-05T10:36:29.923+02:00<DIV>
<H1>IO::Iron Policies - No Typing Errors to Iron.io Services!</H1>
<P>Policies is a way to limit the names of message queues, code packages,
caches and items (item keys)
to a predefined group of possible strings. This can limit the chances
for typos and enforce an enterprise policy. The policies are loaded from
a JSON file which is specified either when creating a
IO::Iron::Iron*::Client object, or
in the config file .iron.json (or equivalent).
<H2>Policies in Config file</H2>
<P>Add the item <I>policies</I> to the config file. The value of the item is the
file name of the policies file.
<P>Example config file:
<pre style="width:1000px;overflow:auto">
{
"project_id":"51bdf5fb2267d84ced002c99",
"token":"-Q9OEHZPhdZtd0KHBzzdUJIqV_E",
"host":"cache-aws-us-east-1.iron.io",
"policies":"iron_policies.json"
}
</pre>
<h3>Policies file specified when creating the client</h3>
<pre style="width:1000px;overflow:auto">
my $policies_filename = '/etc/ironmq/global_policies.json';
my $client = IO::Iron::IronCache::Client->new('policies' => $policies_filename);
</pre>
<h4>Examples of Policies File and Explanation of Configuration</h3>
<P>The 'default' policies JSON file:
<pre style="width:1000px;overflow:auto">
(
'definition' => {
'character_group' => {
},
'no_limitation' => 1, # There is an unlimited number of alternatives.
},
'queue' => { 'name' => [ '[:alnum:]{1,}' ], },
'cache' => {
'name' => [ '[:alnum:]{1,}' ],
'item_key' => [ '[:alnum:]{1,}' ]
},
'worker' => { 'name' => [ '[:alnum:]{1,}' ], },
);
</pre>
<P>The above file would set an open policy for IronMQ, IronCache and IronWorker alike.
The file is divided into four parts: definition for defining meta options, and
queue|cache|worker parts for defining the changing strings
(queue|cache|worker names and item keys). The character group <I>alnum</I> covers
all ascii alphabetic characters (both lower and upper case) and digits (0-9).
<P>N.B. The option <I>no_limitation</I> controls the open/closed policy.
If <I>no_limitation</I> is set (1=set), the policy control is
turned off.
<P>An example of policies file
<pre style="width:1000px;overflow:auto">
{
"__comment1":"Use normal regexp. [:digit:] = number:0-9, [:alpha:] = alphabetic character, [:alnum:] = character or number.",
"__comment2":"Do not use end/begin limitators '^' and '\$'. They are added automatically.",
"__comment3":"Note that character groups are closed inside '[::]', not '[[:]]' as normal POSIX groups.",
'definition' => {
'character_group' => {
"[:lim_uchar:]":"ABC",
"[:low_digit:]":"0123"
},
},
"cache":{
"name":[
"cache_01_main",
"cache_[:alpha:]{1}[:digit:]{2}"
],
"item_key":[
"item.01_[:digit:]{2}",
"item.02_[:lim_uchar:]{1,2}"
]
}
}
</pre>
<P>This policies file sets policies for cache names and item keys. Both have two
templates. Template "cache_01_main" is without wildcards: the template list
can also only contain predefined names or keys. Sometimes this could be
exactly the wanted behaviour, especially in regard to cache and
message queue names.
<P>Items beginning with '__' are considered comments. Comments can not be
inserted into lists, such as I<character_group>.
<P>The <I>definition</I> part contains the list <I>character_group</I> for user-defined
groups. The following groups are predefined:
<dl>
<dt>[:alpha:]<dt>
<dd>ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz</dd>
<dt>[:alnum:]<dt>
<dd>ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789</dd>
<dt>[:digit:]<dt>
<dd>0123456789</dd>
<dt>[:lower:]<dt>
<dd>abcdefghijklmnopqrstuvwxyz</dd>
<dt>[:upper:]<dt>
<dd>ABCDEFGHIJKLMNOPQRSTUVWXYZ</dd>
<dt>[:word:]<dt>
<dd>ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_</dd>
</dl>
<P>All lower ASCII (7-bit) characters are allowed in names and in character
groups, except for the reserved characters (RFC 3986):
<B><pre>!$&'()*+,;=:/?#[]@</pre></B>
<P>A character group definition is closed inside characters '[::]',
not '[[:]]' as normal POSIX groups. Only the equivalents of the POSIX groups
mentioned above can be used; e.g. POSIX group <B>[[:graph:]]</B> is not available.
<P>When using the character groups in a name or key, only two markings are allowed:
<B>[:group:]{n}</B> and <B>[:group:]{n,n}</B>, where 'n' is an integer.
This limitation (not being able to use any regular expression) is due to the
double functionality of the policy: a) it acts as a filter when creating
and naming new message queues, code packages, caches and cache items; 2) it
can be used to list all possible names, for example when quering for
cache items.
</DIV>Anonymoushttp://www.blogger.com/profile/00999479804876258493noreply@blogger.com0Stockholm, Sweden59.329323499999987 18.06858080000006359.07010549999999 17.423133800000063 59.588541499999984 18.714027800000064tag:blogger.com,1999:blog-4746082695756018742.post-62909054279625504492014-08-31T23:39:00.001+02:002014-08-31T23:46:28.556+02:00<DIV>
<H1>IO::Iron::Applications - Command line tools for Iron.io services</H1>
<P><a href="https://metacpan.org/release/IO-Iron-Applications">IO::Iron::Applications</a> is an
auxiliary package for <a href="https://metacpan.org/release/IO-Iron">IO::Iron</a>. IO::Iron contains
the library for using
the <a href="http://www.iron.io/">Iron.io</a> cloud services in Perl programs. IO::Iron::Applications
contains command line programs to operate those services.
<P>IO::Iron::Applications is my addition to the IO::Iron interphase library package which I wrote
earlier.
The Iron.io WWW interface <a href="https://hud.iron.io/dashboard">hud.iron.io Dashboard</a> is great
but a bit slow to use when you only need to
quickly change some values in IronCache, send a message to IronMQ, erase or empty cache for debugging
purposes or other similar activities. With these command line utilities the same functions can be
performed fast from a normal shell and not using a web browser.
<H2>Policies</H2>
<P>The programs take use of the IO::Iron package feature policies so wildcard characters can be used in
cache names, item keys, etc.
<P>For example, if iron_cache_policies.json:
<pre style="width:1000px;overflow:auto">
{
"definition":{
"character_group":{
"[:lim_uchar:]":"ABC",
"[:low_digit:]":"01"
}
},
"cache":{
"name":[
"cache_[:lim_uchar:]{1}0[:digit:]{1}"
],
"item_key":[
"item.02_[:lim_uchar:]{1,2}[:low_digit:]{1}"
]
}
}
</pre>
<P>then
<pre style="width:1000px;overflow:auto">
ironcache list items .* --cache cache_A01 --policies iron_cache_policies.json
</pre>
<p>would print out:
<pre style="width:1000px;overflow:auto">
Cache Item expires
cache_A01 item.02_A0 Key not exists.
cache_A01 item.02_A1 Key not exists.
cache_A01 item.02_AA0 Key not exists.
cache_A01 item.02_AA1 Key not exists.
cache_A01 item.02_AB0 Key not exists.
cache_A01 item.02_AB1 Key not exists.
cache_A01 item.02_AC0 Key not exists.
cache_A01 item.02_AC1 Key not exists.
cache_A01 item.02_B0 Key not exists.
cache_A01 item.02_B1 Key not exists.
cache_A01 item.02_BA0 Key not exists.
cache_A01 item.02_BA1 Key not exists.
cache_A01 item.02_BB0 Key not exists.
cache_A01 item.02_BB1 Key not exists.
cache_A01 item.02_BC0 Key not exists.
cache_A01 item.02_BC1 Key not exists.
cache_A01 item.02_C0 Key not exists.
cache_A01 item.02_C1 Key not exists.
cache_A01 item.02_CA0 Key not exists.
cache_A01 item.02_CA1 Key not exists.
cache_A01 item.02_CB0 Key not exists.
cache_A01 item.02_CB1 Key not exists.
cache_A01 item.02_CC0 Key not exists.
cache_A01 item.02_CC1 Key not exists.
</pre>
<p class="body_text">On the command line, all normal regular expression are allowed. E.g.
<pre>
item.02_A.{1}0
</pre>
<p class="body_text">would return
<pre style="width:1000px;overflow:auto">
Cache Item expires
cache_A01 item.02_AA0 Key not exists.
cache_A01 item.02_AB0 Key not exists.
cache_A01 item.02_AC0 Key not exists.
</pre>
<h2>Following command line programs are available:</h2>
<h3>ironcache</h3>
<dl>
<dt>clear: Clear a cache.</dt>
<dd>E.g. <span class="brush:bash;">ironcache clear cache_main</span></dd>
<dt>delete: Delete a cache.</dt>
<dd>E.g. <span class="brush:bash;">ironcache delete cache_main</span></dd>
<dt>delete: Delete item from cache.</dt>
<dd>E.g. <span class="brush:bash;">ironcache delete item item.01_AB1</span></dd>
<dt>get: Get item/items from cache/caches.</dt>
<dd>E.g. <span class="brush:bash;">ironcache get item item.02_A.{2} --cache cache_A01 --config
iron_cache.json --policies iron_cache_policies_test_01.json --warn</span></dd>
<dt>increment: Increment an item/items in cache/caches.</dt>
<dd>E.g. <span class="brush:bash;">ironcache increment item item.02_AC1,item.02_BC1 --cache cache_A01
--value 225</span></dd>
<dt>list: List caches or items in a cache/caches.</dt>
<dd>E.g. <span class="brush:bash;">ironcache list items .* --cache cache_A01</span></dd>
<dd>E.g. <span class="brush:bash;">ironcache list caches</span></dd>
<dt>put: Put or replace item/items to a cache/caches.</dt>
<dd>E.g. <span class="brush:bash;">ironcache put item item.02_CC1,item.02_CC2 --cache cache_A01 --
value 123</span></dd>
<dt>show: Show the properties of a cache/caches.</dt>
<dd>E.g. <span class="brush:bash;">ironcache show cache cache_A01</span></dd>
</dl>
</DIV>Anonymoushttp://www.blogger.com/profile/00999479804876258493noreply@blogger.com0Stockholm, Sweden59.329323499999987 18.06858080000006359.07010549999999 17.423133800000063 59.588541499999984 18.714027800000064tag:blogger.com,1999:blog-4746082695756018742.post-22699077052668774022014-06-30T18:20:00.000+02:002014-07-01T23:10:33.271+02:00Revision Control and Project Culture<DIV>
<P>Version control, or simply repository control, is one of the most important parts of a software project. After all, it is in many cases used daily. No wonder, then, that version control is not only part of the project structure, but also part of its culture.</P>
<P>This blog entry is partly based on a report,
<A HREF="http://www.koivunalho.org/mikko/published/Rapport-svngit.pdf">Jämförelse: Subversion och Git</A>,
written for Init Ab, a consulting company headquartered in Stockholm.</P>
<H1>Centralized and Distributed Version Control</H1>
<P>A <I>repository</I> is the place where the source code of a program is kept.
The control to a repository is organized with
<I><A HREF="http://en.wikipedia.org/wiki/Revision_control_software">revision control software</A></I>.
This software maintains a monopoly on read and write access to
the repository.</P>
<P>Two recently popular programs in this area are <A HREF="http://subversion.apache.org">Subversion</A>
and <A HREF="http://git-scm.com">Git</A>.
They represent very different views on version control.</P>
<P>Subversion is the leading program among centralized version control software.
A centrally controlled repository is the "classic" way to
arrange control over source code. In this system every user
first copies the needed parts of the software to his or her
local disk and, when done with making changes to it, commits
the changed files to the central repository. For every operation,
access to the repository is required.</P>
<P>In a decentralized (i.e. distributed) revision control software
there is no absolute central repository. Instead, a new user copies
the whole repository from any other existing user. Together with the
current code also the history of changes is copied. Every user
maintains a complete copy of the repository and therefore there is
also no need for centralized backups. In practise, it is customary
for a project to keep a "dummy user" account which is used for release testing,
<I>nightly builds</I> or linked to a <I><A HREF="http://en.wikipedia.org/wiki/Continuous_integration">continuous integration</A></I> system,
for example <A HREF="http://en.wikipedia.org/wiki/Hudson_(software)">Hudson</A>.
<H1>Growing Popularity</H1>
<P>According to recent studies by Eclipse Community Survey<SUP>1</SUP> and ITJobsWatch<SUP>2</SUP> in the last few years Git has become as popular as Subversion also in business world. Among Open Source hobbyist developers Git has been popular already for some time. However, as the statistics show us, Subversion hasn't actually been losing ground to Git. Subversion is the direct descendant of once hugely popular <A HREF="en.wikipedia.org/wiki/Concurrent_Versions_System">CVS</A>, Concurrent Version System, and there is still a great number of enterprises who are running CVS and will only consider changing to Subversion.</P>
<TABLE><TR><TD>
<TABLE>
<TR><TD>Year</TD><TD>Git</TD><TD>Subversion</TD></TR>
<TR><TD>2009</TD><TD>2.4%</TD><TD>57.5%</TD></TR>
<TR><TD>2010</TD><TD>6.8%</TD><TD>58.3%</TD></TR>
<TR><TD>2011</TD><TD>12.8%</TD><TD>51.3%</TD></TR>
<TR><TD>2012</TD><TD>27.6%</TD><TD>46.0%</TD></TR>
<TR><TD>2013</TD><TD>36.3%</TD><TD>37.8%</TD></TR>
<CAPTION>Results of the Eclipse Community Survey
regarding SVN and Git usage.</CAPTION>
</TABLE>
</TD>
<TD WIDTH="50"></TD><!-- between -->
<TD>
<TABLE>
<TR><TD>Year</TD><TD>Permanent positions:</TD><TD></TD><TD>Rank:</TD></TR>
<TR><TD></TD> <TD>Git</TD> <TD>Subversion</TD><TD>Git</TD><TD>Subversion</TD></TR>
<TR><TD>2012</TD><TD>1167</TD><TD>3354</TD><TD>263</TD><TD>91</TD></TR>
<TR><TD>2013</TD><TD>2049</TD><TD>2836</TD><TD>157</TD><TD>107</TD></TR>
<TR><TD>2014</TD><TD>3605</TD><TD>3265</TD><TD>90</TD><TD>99</TD></TR>
<caption>ITJobsWatch: Git & Subversion.</caption>
</TABLE>
</TD></TR></TABLE>
<H1>(De)centralized Culture</H1>
<P>I will not concentrate on technical side of revision control but rather on the cultural aspects that these two very different solutions foster.</P>
<P>Version control, or simply <I>repository control</I> is one of the most important parts handling a project or participating in one. After all, we use it daily. The program which we use to access the repository is one of our most often used tools. Therefore, when it feels like it refuses to co-operate with us, it immediately becomes a major irritation. So it must be simple, reliable and fast.</P>
<P>But more than a tool for programmers, version control is also a link between project leadership (maybe even middle-level management, depending on company structure) and developers and architects. It provides us with (inflexible?) boundaries to how we shape our work.</P>
<P>Ben Collins-Sussman, one of Subversions designers, claims that decentralized version control works badly for teams which don't consist of equally competent people. He quotes some requests<SUP>3</SUP> he got when developing Subversion:
<BLOCKQUOTE><I>Can you guys please give Subversion on Google Code the ability to hide specific branches?</I><BR>
<I>Can you guys make it possible to create open source projects
that start out hidden to the world, then get revealed when they're ready?</I><BR>
<I>Hi, I want to rewrite all my code from scratch, can you please wipe all the history?</I>
</BLOCKQUOTE>
Developers are humans and they have a tendency to <I>want to work privately, in a cave, then
spring "perfect" code on their community, as if no mistakes had ever been made</I>. In a decentralized version control environment it can be too easy to "slip" into isolation, thinking that committing into your own repository has the same purpose as committing to the central repository. But this is not the case. The local copy of the repository is for the developers hourly or daily use for local backups; but the central repository is "public" so the project manager and others can see where the developer is going. The project policy could be to commit every day before finishing work, and if the central repository is connected to a continuous integration system with unit tests, errors and bad solutions will be discovered earlier. Collins-Sussman quotes Google's culture och mantra: <I>don't run from failure - fail often, fail quickly, and learn</I>.</P>
<P>On the other hand, if the team is small and every developer about at the same level, decentralized version control can foster <A HREF="en.wikipedia.org/wiki/Meritocracy">meritocracy</A> and friendly competing spirit. In a true decentralized version control environment (without a "centralized dummy user") changes are copied directly from one user to another so trusting the other's code becomes a necessity.</P>
<P>A decentralized environment is not the only way to foster meritocracy, however. The <A HREF="http://www.apache.org">Apache Software Foundation</A> is also known for its meritocratic structure in open source projects. They use Subversion exclusively. Project participants are divided into three groups: users who can make suggestions and bug reports, developers who submit their code but cannot commit, and committers who have write access to the repository. Anyone can become user and being a developer only requires to checkout the freely available source code from the Subversion repository. The committers' group replenishes itself from the developers' group by selecting with a common decision the ones whose submitted source code has the best quality. <A HREF="http://www.gnome.org/foundation/">The GNOME Foundation</A>, Apache Software Foundation, <A HREF="https://www.mozilla.org/">Mozilla Foundation</A>, and <A HREF="http://www.documentfoundation.org/">The Document Foundation</A> officially claim to be meritocracies.</P>
<P>Centralized version control favours a more structured organization, whereas decentralized can suit a self-forming or self-governing team, or hobbyist group. On the hand other, the technical know-how must be somewhat higher, especially when using Git. Git is powerful but somewhat complicated to use, more error-prone (or gives that appearance) in daily usage than its main decentralized competitors <A HREF="bazaar.canonical.com">Bazaar</A> or <A HREF="mercurial.selenic.com">Mercurial</A>, not to mention centralized Subversion.</P>
<P>Naturally decentralized version control can suit a well structured organization or a company, as well, but it requires stricter guidelines and processes to guide its usage which in part may nullify its benefits.</P>
<H1>Conclusion</H1>
<P>The question of team and organization's culture is the most important. As mentioned above, version control is a daily tool, and its users' culture will influence the way it is being used; but also the opposite: the version control tool will influence the users by favouring certain work flows and usage patterns over others.</P>
</DIV>
<DIV>
<H1>References</H1>
1. <A HREF="http://www.slideshare.net/IanSkerrett/eclipse-survey-2013-report-final/14">Eclipse Community Survey Report 2013</A>, Retrieved 2014-06-13.<BR>
2. <A HREF="http://www.itjobswatch.co.uk">ItJobsWatch</A>, Retrieved 2014-06-13.<BR>
3. Brian W. Fitzpatrick and Ben Collins-Sussman, <I>Team Geek, A Software Developer's Guide to Working Well with Others</I>, 2012, First Edition, O'Reilly Media.<BR>
</DIV>Anonymoushttp://www.blogger.com/profile/00999479804876258493noreply@blogger.com0Stockholm, Sweden59.329323499999987 18.06858080000006359.329323499999987 18.068580800000063 59.329323499999987 18.068580800000063tag:blogger.com,1999:blog-4746082695756018742.post-27086610041586922432014-05-31T21:56:00.001+02:002014-05-31T21:56:14.793+02:00HtmlUnit - For Integration Testing and WebcrawlingTo put it in just a few words: <it>HtmlUnit is a web browser without a window</it>.<br />
<br />
Intended for integration testing, <a href="http://htmlunit.sourceforge.net/">HtmlUnit</a> allows user programmatically to manipulate a webpage on a high level, i.e. as if doing it with a normal web browser. The calling program can fill and submit forms, click on buttons, imagemaps and hyperlinks, or activate JavaScript created object. JavaScript, cookies and AJAX are supported. So are proxies and immediate redirection.<br />
<br />
<h2>GUI integration testing</h2><br />
This kind of testing is about as close to human testing we can get with automated testing. Testing static webpages is always easy because the content only gets loaded once from the remote server but nowadays webpages have more often <a href="http://en.wikipedia.org/wiki/DHTML">dynamic content</a> than not. Once the page is loaded not only the outward appearance but also the content itself is changed with the help of JavaScript, CSS (Cascading Style Sheets), AJAX and Adobe Flash (although flash - being a self contained "applet" or videoplayer - is outside the scope of HtmlUnit.<br />
<br />
With HtmlUnit the test program can "crawl" through the HTML code section by section confirming that content is correct. Or it can jump straight to a certain part identified by id or name tag. It can "hover" the mouse pointer (emulated, of course) over parts of text or a button on a form, or e.g. select an item from a select (list) button which is wired with JavaScript, and then confirm that the page or form content changes as planned.<br />
<br />
HtmlUnit does UI testing for webpages, or more precisely integration testing for HTML elements' and JavaScript's integration.<br />
<br />
<h2>Webcrawling</h2><br />
Because HtmlUnit is a headless (i.e. windowless) web browser, it can also be used to programmatically browse websites and extract information. On many webpages JavaScript is intimately linked to the processing of forms so that a form cannot be submitted properly without JavaScript's help. These kind of pages are of course examples of poor webform design (separation of concerns is not completed; business logic is mixed with the program flow) - but ours being an imperfect world, even they must be accepted. And that's where HtmlUnit shows what it's made of.<br />
<br />
There is plenty of pages where user only needs to log in through the front page, and immediate the sought after information is available, or maybe via a simple form, like logging to your telephone company's website only to see how much saldo or network quota you still have left for the current month. Many simple hardware devices, such as home routers, only provide a Web interface, no SOAP or REST API. HtmlUnit to the rescue! Earlier it was impossible or close to it to get to this content.<br />
<div><br />
</div>Let's see an example in Java:<br />
<pre class="prettyprint" style="background-color: #eeeeee; font-family: Monaco, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console', monospace; font-size: 12px; max-width: 70em; overflow: auto; padding: 0.5em;"><span class="tag" style="color: #000088;"><pre span=""></pre>import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.html.HtmlAnchor;
import com.gargoylesoftware.htmlunit.html.HtmlForm;
import com.gargoylesoftware.htmlunit.html.HtmlInput;
final WebClient webClient = new WebClient(BrowserVersion.CHROME, proxyIP, proxyPort);
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getCookieManager().setCookiesEnabled(true);
</span></pre></div><div><br />
</div>We have imported some HtmlUnit element classes. We create a new WebClient instance by tell it which browser it should spoof and which server to use as a proxy. Both of these are optional. Sometimes an HTTP server or the client side JavaScript changes layout of the page depending on the requesting browser. We also enable redirection, JavaScript support and cookies support. Another way:<br />
<br />
<pre class="prettyprint" style="background-color: #eeeeee; font-family: Monaco, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console', monospace; font-size: 12px; max-width: 70em; overflow: auto; padding: 0.5em;"><span class="tag" style="color: #000088;"><pre span=""><span style="font-family: Monaco, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console', monospace;"> final WebClient webClient = new WebClient();</span></pre></span></pre><br />
Let's continue. We want to find the submit button and input fields for userid and password. Once we get them, we can finish logging in by clicking the submit button and loading a new page in the bargain.<br />
<br />
<pre class="prettyprint" style="background-color: #eeeeee; font-family: Monaco, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console', monospace; font-size: 12px; max-width: 70em; overflow: auto; padding: 0.5em;"><span class="tag" style="color: #000088;">HtmlInput submitButton = null;
HtmlPage titlePage = null;
try {
titlePage = webClient.getPage(hostname);
} catch (IOException e) {
e.printStackTrace();
}
final List<htmlform> forms = titlePage.getForms();</htmlform>
// iterate through the list to find what we need.
<span class="Apple-tab-span" style="white-space: pre;"> </span>[...]
submitButton = loginForm.getInputByName("login");
final HtmlTextInput usernameTextField = loginForm.getInputByName("login_id");
final HtmlPasswordInput passwordTextField = loginForm.getInputByName("login_password");
usernameTextField.setValueAttribute(userId);
passwordTextField.setValueAttribute(password);
try {
entryPage = submitButton.click();
} catch (IOException e) {
e.printStackTrace();
}
List<htmlanchor> links = entryPage.getAnchors();</htmlanchor>
for (HtmlAnchor link : links) {
logger.debug("Entry Page link: " + link.asXml());
if (link.asXml().contains("create_new_entry.new")) {
linkToJobAdPage = link;
}
}</span></pre><br />
<div><br />
</div><h2>HtmlUnit for Perl</h2><br />
HtmlUnit is not a Java monopoli just because it was developed on Java. It's also available for other programming languages.<br />
<br />
<a href="http://celerity.rubyforge.org/">Celerity</a> is a JRuby wrapper around HtmlUnit – a headless Java browser with JavaScript support.<br />
<br />
<a href="https://metacpan.org/release/WWW-HtmlUnit">WWW::HtmlUnit</a> is the Perl equivalent, an Inline::Java based wrapper of the HtmlUnit v2.14 libraryAnonymoushttp://www.blogger.com/profile/00999479804876258493noreply@blogger.com0tag:blogger.com,1999:blog-4746082695756018742.post-18396628277484421382014-04-30T01:52:00.000+02:002014-05-31T23:04:55.005+02:00IO::Iron gets command line toolsNow that all the functions of <a href="http://www.iron.io/">Iron.io</a>'s IronMQ, IronCache and IronWorker services are turned into Perl client libraries, it is time to think about not only the programmer but also the tester and application supporter's needs. They require an easy and quick access to the services: command line tools.<br />
<br />
Perl has several possible frameworks for creating command line utilities: e.g. <a href="https://metacpan.org/release/CLI-Framework">CLI::Framework</a>, <a href="https://metacpan.org/release/App-Cmd">App::Cmd</a>, <a href="http://badgerpower.com/">Badger</a>, <a href="https://metacpan.org/release/CLI-Application">CLI::Application</a> and <a href="https://metacpan.org/release/CLI-Dispatch">CLI::Dispatch</a>. From these I picked App::Cmd mostly because of its decentralized nature.<br />
<br />
<h3>Command Line Tool Design for Continuous Integration</h3><br />
One of the principles of <a href="http://en.wikipedia.org/wiki/Continuous_Integration">Continuous Integration</a> states that "Everyone commits to the baseline every day". To make the programmers' load lighter every change should be in as much isolation as possible. Centrally located code which refers to individual parts of the system should always be generated automatically so the programmer does not need to remember and bother to keep up-to-date any kind of central index / reference table / central documentation / user reference or any other kind of central keeping place for things. Automatic code generation not only ensures that the "keeping place" is always up-to-date but also avoids typing errors. In Continuous Integration repository it also prevents or at least limits the possibility of merge conflicts in same files.<br />
<br />
Equally important is to follow the practise of "Generate User Documentation from Program Code". App::Cmd is a good example of this. Actually, App::Cmd uses <a href="https://metacpan.org/release/Getopt-Long-Descriptive">Getopt::Long::Descriptive</a> which is its own small system (or framework) for processing command line options and parameters. The options and parameters are defined in a meta language (in a Perl hash data structure) and the Getopt::Long::Descriptive package uses this data structure to present the same options to user when needed, e.g. when user mistypes a parameter name.<br />
<br />
<blockquote class="tr_bq">my ($opt, $usage) = describe_options(<br />
'my-program %o <some-arg>',<br />
[ 'server|s=s', "the server to connect to", { required => 1 } ],<br />
[ 'port|p=i', "the port to connect to", { default => 79 } ],<br />
[],<br />
[ 'verbose|v', "print extra stuff" ],<br />
[ 'help', "print usage message and exit" ],<br />
);</blockquote>becomes on the text terminal:<br />
<br />
<blockquote class="tr_bq"> my-program [-psv] [long options...] <some-arg><br />
-s --server the server to connect to<br />
-p --port the port to connect to<br />
-v --verbose print extra stuff<br />
--help print usage message and exit</blockquote><br />
<h3>Loose Coupling at Runtime on Application Level</h3><br />
However, App::Cmd goes even further. When several distinctively different command functions are combined into one application, they are completely separated into individual files. This is a fine example of loose coupling inside one application. The commands do not know of each others' existence and neither do the programmers need to know of it. In IronMQ's case, the executable ironmq contains individual commands like 'add', 'delete', 'show' and 'list' but their existence is not documented permanently (statically) anywhere. One programmer works on one command and completes his or her work regardless of whether the other programmers have finished with the other commands. When user executes ironmq, App::Cmd framework discovers at runtime which commands and parameters are available.<br />
<br />
This fastens the application development by making interconnected parts not reliant of each other. It speeds Continuous Integration and time to deployment.Anonymoushttp://www.blogger.com/profile/00999479804876258493noreply@blogger.com0tag:blogger.com,1999:blog-4746082695756018742.post-62163164248000674532014-03-30T15:09:00.000+02:002014-03-30T15:17:03.301+02:00Dist::Zilla as a Continuous Delivery ToolI just recently converted my <a href="https://metacpan.org/release/IO-Iron">IO::Iron</a> distribution to using <a href="http://dzil.org/index.html">Dist::Dilla</a> as a <a href="http://en.wikipedia.org/wiki/Release_engineering">release</a> and <a href="http://en.wikipedia.org/wiki/Build_automation">build automation</a> tool. Dist::Dilla is mainly targeted at people writing free software Perl packages for releasing into <a href="http://www.cpan.org/">CPAN</a> (Perl free software archive) but if used properly it can make easier the releasing of any software.<br />
<br />
<h3>
Before</h3>
<br />
When I started to build the IO::Iron distribution, I already knew of Dist::Zilla but two things kept me from adapting it. Firstly, I considered it too difficult to learn for such a small project (which later grew), and, secondly, being bloated and suffering from featuritis. Instead, I went with the classic solution of using Module::Starter to begin, and continued with manually editing the Makefile.PL and every other file, including MANIFEST, README and Changes. I used my private Subversion repository. I uploaded to CPAN via the <a href="http://pause.perl.org/">CPAN Author page</a> page.<br />
<br />
After I had forgotten to update the Changes file a few times, I started to reconsider Dist::Zilla. The more I read about it, e.g. Dave Rolsky's excellent blog entry <a href="http://blog.urth.org/2010/06/02/walking-through-a-real-distini/">Walking Through a Real dist.ini</a>, the more it seemed to make sense. About two weeks ago I decided to take the time required, a day or two, and go through the setting up of Dist::Zilla and converting IO::Iron.<br />
<br />
<h3>
After</h3>
<br />
It was worth the effort. Dist::Zilla does not replace the Makefile.PL which is used when user takes a distribution into use. Makefile.PL builds, tests and installs at user's end. But Dist::Zilla prepares the distribution for uploading. It automates almost all the repeating steps involved when releasing: determines prerequisites, manages version numbers and Changes file, checks that the changes have been committed, and - above all - builds the Makefile.PL.<br />
<br />
Dist::Zilla streamlines the code-test-commit-release -cycle and defines a workflow, thus rising release quality.<br />
<br />
<h3>
Inner and Outer Workings</h3>
<br />
Using Dist::Zilla is done with the command line tool <i>dzil</i>. It is very similar to M<a href="http://en.wikipedia.org/wiki/Make_(software)">ake</a> in outward appearance. Dist::Zilla itself is actually a frame for defining workflow stages. All functionality is executed by plugins. Building a release is divided into stages or roles similiar to what Makefile.PL uses: build, test, install, release, etc. The plugins are attached into separate stages. For example, gathering the distribution files and reading them into memory (from which they will later be written into a new build directory) is a stage and the equivalent roles are FileGatherer and FileInjector. All required plugins which fill these roles will be executed at this stage, and a plugin can read an existing file from disk, or create a file dynamically.<br />
<br />
<h3>
Creating Distributions</h3>
<br />
When creating a CPAN distribution, such as IO::Iron, whose <a href="https://github.com/mikkoi/io-iron">"distribution source code"</a> is now located publicly at Github, the last action (i.e. plugin) when executing <i>dzil release</i> is normally <a href="https://metacpan.org/pod/Dist::Zilla::Plugin::UploadToCPAN">"UploadToCPAN"</a>, but this can be changed by editing the <i>dist.ini</i> file. CPAN distribution format is convenient also for other code releases than just CPAN packages. Instead of uploading, the last action in the chain could be committing code to the repository, or making a direct installation.<br />
<br />
<h3>
Continuous Delivery</h3>
<br />
Dist::Zilla's modular structure makes it adaptable to new situations, even to different programming languages. It is not limited to Perl, not even to programming. With a different set of plugins it could just as well serve as a blog authoring (automatic spell checking, abbreviation expanding, date/version managing) and uploading tool. It becomes a competitor to e.g. Maven [http://maven.apache.org], which is best known in conjunction with Java (although it is more of a project management tool than software authoring tool).<br />
<br />
In the field of continuous delivery and continuous integration Dist::Zilla contributes to lightening the strain of programmers from remembering often repeated actions, codifying workflow and rising the quality of releases, especially when releasing often. The plugins are reusable pieces of code easily shared among developers. This in turn reduces the time and effort of rebuilding the build system when updating old projects or creating new ones.Anonymoushttp://www.blogger.com/profile/00999479804876258493noreply@blogger.com0tag:blogger.com,1999:blog-4746082695756018742.post-57868193874507299252013-12-27T12:12:00.000+01:002013-12-27T12:12:42.139+01:00Cloud Architectural Challenges<h2>
Building applications for cloud.</h2>
<br />
How is creating software for cloud different from previous architecture? Not that much really. There is a great deal of hype which makes it seem that cloud software architecture is a huge improvement compared to "traditional" software architecture - whatever that is: mainframe architecture, thin/fat-client architecture, server farm architecture, any other architecture...<br />
<br />
The differences between other architectural styles and cloud architecture rise not from the offerings of the cloud but rather from the unique challenges posed by operating in the cloud. The challenges are two, and they are mainly a question of uncertainty...<br />
<br />
<h3>
Challenge: Out of Sync</h3>
<br />
The cloud is - by its very nature - a distributed environment. A distributed application consists of multiple parts which communicate with each other but don't necessarily run in the same system. When an application has functionality which is used at different times or different frequencies, or can be run parallel to other parts, then it often would make sense to remove it outside the normal execution process and perhaps even set it into a different system to be connected only when required.<br />
<br />
This kind of functionality might for example be the billing or archiving functions of a website. After making an order in a web-shop, the website will not wait until the customer's credit card is actually billed; the web-shop returns control to the user immediately and a different subsystem of the application handles the credit card billing. When it is finished, the user will get a confirmation email and user's profile in the web-shop will be updated.<br />
<h4>
<br /></h4>
<h4>
Workers</h4>
<br />
The example above is very trivial. The subsystem handling the billing is a "worker", a unit which is activated only when the application requires its service. The unit might exist outside the application's system, maybe even on a different server. The important thing is that the application does not wait for it to return anything. It runs alone according to the parameters the application provides to it, and after running it closes itself down automatically without need for any interaction. Therefore, it runs out of sync with the main application.<br />
<h4>
<br /></h4>
<h4>
Being Out of Sync</h4>
<br />
When programs run inside one server and one operating system, the communication is instantaneous or - in practice - real time. But communication in the cloud is not in real time, sometimes not even stable. The connection might take a long time to establish, or it might break, or simply be slow. The vendor might have an unscheduled maintenance break or the service might have been relocated physically to a different server. In general, subsystems often use IP addressing to connect to each other. The two main architectural choices are RPC (remote procedure call) interface or REST (representational state transfer) interface. The main difference between these is less in the implementation and more in their philosophy.<br />
<br />
RPC is an interface for a tightly coupled application where the subsystems are in fact subroutines and the main program waits for the completion of the subroutine before continuing. REST on the other hand is an API which can function both synchronously and asynchronously. REST is best used when subsystems of the application are autonomous services which can - in principle - be offered to any application. REST encourages the design of the API into the form of a (public) service. REST is stateless API so no client context is stored on the server between requests.<br />
<br />
Because of the possible or (pessimistically) likely problems in connection between the application and its subsystems in the cloud, it is generally better to go for REST style architectural design. Properly implemented it provides a robust loosely coupled system fit for cloud.<br />
<h4>
<br /></h4>
<h4>
Messaging</h4>
<br />
Of course, the distributed parts (workers and others) need to communicate with each other. There are many ways to do it but generally the best is an out-of-sync way: a message queue. A message queue is an external application, "messaging middle-ware", to whose care the application gives a message and then "forgets it". Another part of the application polls the message queue at preset intervals and reads the message when it is available. The message queue guarantees that a message will never get lost but it doesn't know how quickly the other subsystem will read it or act upon it. It does not wait for a return message. It will wait, however, for a receipt from so it knows the message was handled. When using a message queue to link subsystems together, an API (REST or other) is not necessary.<br />
<h3>
<br /></h3>
<h3>
Challenge: Unreliability</h3>
<br />
As mentioned above, cloud is a volatile environment, also in the sense that vendor companies may come and go, new services promoted and old ones canceled. Cloud architecture is also about preparing for the eventuality of migration to new services or platforms. It is the natural additional price to pay when seeking "affordable" cloud services as most companies always do.<br />
<h4>
<br /></h4>
<h4>
Design and prepare for eventual platform or vendor change.</h4>
<br />
The platforms that cloud vendors and service providers offer include not only real or virtual servers but also "platforms" that are more like services, such as databases, messaging middle-ware, worker platforms, and of course varied special services like log collectors (IT operation oriented) or daily currency rate providers (business oriented). All purchased services, not to mention free services, have a tendency to change APIs or even disappear as time goes by.<br />
<br />
Cloud architecture has ways to prepare for this eventuality. Most of these are coding practices that can be forced for instance if the implementation is done using a framework. Connection to external APIs can be isolated, database connection abstracted into an ORM (object relational mapper), message queue connection as well. Unfortunately this also means that the risk for complicating the implementation rises.<br />
<br />
Another isolation layer could be a proxy server for REST or RPC calls. A proxy server can provide additional security as it could also keep the remote services' passwords and other connection details hidden from the service users.<br />
<h3>
<br /></h3>
<h3>
Change is always pending</h3>
<br />
With cloud architecture, preparing for trouble and change is always paramount because in the cloud an application can have very little control over its environment. The cloud creates a new kind of approach into dealing with vendors: pay-as-you-go. If the costs of vendor service are billed accurately according to the actual usage of resources (memory, CPU cycles, bandwidth, tech support requests, ...), this will prompt the design of applications better optimized for cloud environment.Anonymoushttp://www.blogger.com/profile/00999479804876258493noreply@blogger.com0tag:blogger.com,1999:blog-4746082695756018742.post-14036467277464619442013-11-04T02:01:00.001+01:002013-11-04T02:05:10.206+01:00IronMQ - Message Queue in the cloudWhy didn't anybody think of it before!<br />
<br />
The advent of cloud services are breaking apart the server-centered thinking: with the cloud - or in the cloud - all Internet services are close to each other. The trunk line connections even between separate clouds provide fast enough access speed to actually start "picking" the services. Paas (Platform as a service) will give way to Saas (Software as a service), or maybe even "Service as a service". Selecting any Internet service will be possible if services are compatible enough and "close" enough.<br />
<br />
Cloud makes the services close enough but it's not enough by itself. Distributed computing and application integration requires a reliable way for the applications to talk to each other, preferably without syncronization because in a real word (cloud/Internet) services and applications don't necessarily go at the same speed. One of the best ways to balance the sending and receiving is to use a message queue.<br />
<br />
Until now message queues have been limited inside one server, with a few exceptions, such as IBM's Websphere MQ. And even then, the interface to the message queues has been via linkable system libraries, which binds them to platforms or even specific programming languages. And of course the message queue must have an available node, port or other connection point accessible from within the server.<br />
<br />
<a href="http://iron.io/">Iron.io</a> has changed that! If cloud makes services available to all applications, then there should be a message queue inside cloud - but outside servers. IronMQ is that messages queue; and its API is in line with most cloud services because it is a REST compatible API.<br />
<br />
IronMQ is "Message queue as a service", the first of its kind. Customer may pay on a per-message basis which goes perfectly with the idea of Saas. For a hobbyist it's a heaven since the payments only start running after the first 10 million requests (REST calls).<br />
<br />
Iron.io uses OAuth for user authentication, and access protocol is of course HTTPS. REST interface for a message queue is not the big innovation here; there is other message queues which also provide a REST interface to supplement their normal socket interface, or linkable library interface. What is an innovation is how well IronMQ is interacting in cloud/Internet environment: from a passive party (what a message queue by nature is) it turns into an active party via its "push queues". Push queue is a queue which "knows" who is going to read the messages. It simply means that the message is relayed to another HTTP (or HTTPS) endpoint. The subscriber does not need to keep polling the queue for new messages; it simply sets up an HTTP(S) server/reader and waits for the messages. Besides remote HTTP endpoints, messages can also be pushed to different queues or IronWorker, Iron.io's worker system.<br />
<br />
IronMQ pushes the concept of push queues even further: it accepts messages pushed to it by the REST compatible method of Webhooks, user-defined HTTP callbacks. They are usually triggered by some event, such as pushing code to a repository or a comment being posted to a blog. When that event occurs the source site makes an HTTP request to the URI configured for the webhook.<br />
<br />
Iron.io has three cloud services: IronMQ, IronWorker and IronCache (a key-value storage). All of them have a REST interface and excellent cloud-usability. Cloud applications are often parts of integration systems. But the integration itself has been difficult because most integration tools, such as message queues, are running "inside" servers and are good at providing "internal" services. Iron.io's services are "between" servers and they are accessible by the most widely used REST protocol, HTTP.<br />
<br />Anonymoushttp://www.blogger.com/profile/00999479804876258493noreply@blogger.com0tag:blogger.com,1999:blog-4746082695756018742.post-89327270013527757912013-10-26T23:38:00.001+02:002013-10-26T23:38:42.345+02:00Exercises in Restful Integration and Continuous DeliveryHaving participated in several application integration projects, and seen both<br />
great success and horrible blunders, this blog is a web diary and collection<br />
of notes on things that I've seen work or fail; either tested by myself or simply witnessed working or failing.<br />
<br />
RestChess is an attempt to integrate cloud/Internet services some of which are of old technology and some very new. My modest attempt is to "do things right", both in actual application integration and in delivery. <a href="http://en.wikipedia.org/wiki/Continuous_delivery">Continuous Delivery</a> may be a buzzword but also a good goal.Anonymoushttp://www.blogger.com/profile/00999479804876258493noreply@blogger.com0