Artificial Intelligence ZA Meetup – June 2016

AI Meetup

The second AI ZA meetup was held on 21 June 2016 at Entelect’s headquarters in Melrose Arch, Johannesburg. The event was packed with over 50 attendees eager to learn about artificial intelligence and machine learning.

The goal of this meetup was to continue with a more in depth classification problem as well as run through the introductory concepts for all those that were new to the meetup.

The Talk

Rishal Hurbans and Hennie Brink gave a short talk on classification algorithms. The uses, advantages, and disadvantages of classification algorithms were covered.

The first concept covered was how to determine if a classification algorithm can be used. The diagram below illustrates the decision tree for deciding if a classification algorithm is right for the problem being tackled and dataset at hand.

Deciding on Classification

Two families of algorithms were explored; the Linear Support Vector Machines(SVC) algorithm, and the Naive Bayes algorithm. These were selected to illustrate the differences in classification algorithms, and where each are most effective.

Linear SVC is used for supervised learning for high dimensional spaces where the number of features being analyzed is large, however, it can perform poorly when the number of dimensions is far greater than the number of samples.

Naive Bayes is best when each feature is meant to be analyzed independently without any assumptions of relationships between features. This makes it a good classification algorithm for the problem tackled in the hackathon, which was text analysis.

The Hackathon

The goal of the hackathon was to analyze the Amazon fine foods reviews dataset and determine if a review score could be assigned based on a person’s textual review. Most of the meetup group teamed up and tried to solve this problem together, whilst some got up and running with the introductory hackathon on password strength.

After 2 hours of hacking away on the text sentiment problem, some got close, but realized that sentiment analysis is difficult and is often inaccurate. Many were looking forward to expanding on their code after the meetup to improve the accuracy of their implementation.

By simply querying the dataset, some interesting trends came to light. Many 5 star reviews have the word “love” in it, about a third of those reviews also have the word “hate” in it! The dataset also contained quite a number of 1 star reviews with the word “love” or “like” in them. This goes to show that simply using keywords for textual analysis can be extremely inaccurate.

The meetup ended with some interesting casual chats about artificial intelligence, philosophy, ethics, virtual reality, and our future as humans.

To tinker with this problem, visit the GitHub repository

The “Machine Learning for Beginners” documentation and exercises are available on GitHub. If you’re interested in AI, join us on Meetup.

Artificial Intelligence ZA Meetup Kickoff

AI Meetup

Thursday 19 May 2016 marked the kickoff of the first AI hackathon meetup in Johannesburg, South Africa. The meetup is planned to happen monthly with a focus on growing a community around learning and practicing concepts in artificial intelligence.

The evening started with drinks and pizza where everyone got to socialize and network with their peers. Some interesting conversations around AI and innovation sprouted.

The Talk

Rishal Hurbans and Gail Shaw gave a short talk introducing AI concepts and algorithms. The talk was light and useful to anyone without a background in AI.

The first concept that the meetup group will be tackling is Machine Learning. The Machine Learning for Beginners GitHub repository included a getting started guide for Python and R. The repository demonstrates a simple example for learning the difference between an apple and orange based on it’s weight and texture. It also includes an example for learning classifications of the Iris flower species – the Iris dataset is a popular dataset used for testing and learning in the area of data science. The two examples serve as a quick getting started guide for beginners in machine learning and people not familiar with the Scikit Learn library for Python. The Scikit Learn library provides a number of built-in classifiers and prediction algorithms for machine learning which makes getting up and running simple and easy.

The Hackathon

The hackathon session included an exercise and dataset where the group were challenged to create an algorithm to classify password strength. The dataset consists of 50000 randomly generated passwords for use as training data and 25000 randomly generated passwords for use as testing data. The password strength is measured by detecting the use of uppercase characters, numbers, special characters, and length. More information on the exercise can be found here.

The key aspect of the hackathon was to think about the properties of the password that are most likely to be useful for machine learning, and thereafter, preparing the data such that it can be consumed by a machine learning classifier. Often data, as it exists, is not suitable for machine learning. Removing redundant features, removing unnecessary features, and choosing the correct types of features are an important part of the process.

The group split up into small teams of two or three where they discussed solutions and hacked away at some code. The outcome was interesting as different teams had different approaches to the problem, and conducted various experiments to learn more about the performance of the machine learning classifier.

Some teams chose Boolean features for the occurrence of numbers, special characters, and upper case characters; whilst other teams counted the occurrences of the mentioned features. Some teams represented the output as a percentage of accuracy, and other teams visualized the output in good looking charts. One of the most interesting outcomes were the experiments that teams conducted to test the threshold for learning by reducing the training set and testing the outcomes of the machine learning.

The Outcomes

Here’s some insight from a few of the participants:

The appproach I used was to extract the data into a list of inputs and targets, then mapping a feature extraction function to each password to get a list of feature sets that map nicely with the target array. The feature extraction function extracted four features from the data:

An integer value representing the length of the password subtracting 8,

Three bit/boolean value representing if the password contains an uppercase letter, a digit and a special character.

Three learning algorithms were trained using the input data set and then these classifiers were used to predict the strengths of the password in the testing set. The results of the predictions were compared to the target values, with the error being calculated as the difference between the prediction and the target. The accuracy (error = 0) of each method used is as follows:

Stochastic Gradient Descent: 62.552% accuracy (15638 / 25000)

K-nearest neighbours: 99.998% accuracy (24997 / 25000)

Random Forests: 100.000% accuracy (25000 / 25000)
-Kevin Gray

We decided on the following stack: Jumpy, Pandas, re and obviously sklearn.

Since we needed to generate features, we decided that the apply function of a pandas series would work out really well. For each series you can apply a function which is “applied” to each element in the series.

Given this, we wrote two functions. The first function parses the file by just opening it and iterating over each line. Each line is split using regex to strip out the password and the score. The second function, called count_chars, counts the number of characters in the password. It takes a password and a set of characters from the string module and returns the count of characters. We can then apply the function using the syntax below:
training.password.apply(lambda x: count_chars(x, string.ascii_uppercase))
So from the latter function and the len function we created the features below.
upper_case_count
lower_case_count
punc_case_count
password_len

We then used the cross_val_score to evaluate how well the model generalizes and got an average score of 99%. Checking this against the test set we got approximately the same accuracy. We ran some performance checks on the code and found the following:
Parsing either file takes about 5.72 µs
Creating all the features takes about 6.91 µs

-Bradley, Stuart, Kirton

The “Machine Learning for Beginners” documentation and exercises are available on GitHub. If you’re interested in AI, join us on Meetup.

UX South Africa Conference Johannesburg 2016

The UX South Africa conference was held in Johannesburg on 11 and 12 May. We were there, and this is what went down.

The conference was held at the The Focus Rooms in Sunninghill, Johannesburg. The venue was great – filled with quirky signage, and fun material for the guests.

Focus Room 1

Focus Room 2

The event consisted of ±200 attendees and 17 speakers from across the globe.

Notable Talks

Here are some of the talks that caught our attention and left a lasting memory.

Designing for Cultural Diversity

By Chui Chui Tan, UX Editor at Smashing Magazine

This talk explored the diversity of cultures in different regions around the globe. It highlighted the need to carefully research and craft products that fit the target market whilst taking into consideration the cultural and language differences for the context that it will be used in.

Designing for Cultural Diversity

Designing for the Future: Principles of smart interfaces and interaction design

By Basak Haznedaroglu, UX Designer at Microsoft Berlin

To build for the future, we must first understand the past. This talk explored the evolution of interfaces, from command line terminals, our “beloved” Clippy assistant, to today’s bots, and digital assistants. With new technology emerging, and new ways to interact with technology, there are more opportunities for designers to build great experiences. Basak urges us all to “Stay relevant!”.

Designing for the Future

Developing a Human Centered UX Strategy

By Jason Hobbs, Director at Firma

This talk aimed to show the difference in UX process when designing and evaluating a product. Jason took the audience through some real-world examples of how a human centered design framework can be used, and what value it can add to a product as well as to the end-users’ experience.

Developing a Human Centered UX Strategy

Design Thinking and Agile

By Rishal Hurbans, Solution Designer at Entelect, founder of Prolific Idea

Design thinking is an emerging concept that seems to be a buzzword at the moment. This talk started off explaining the meaning of design thinking and how it’s being used by giants such as Apple, Tesla, Space X, and Google. The talk took us back in time to illustrate the exponential progression in technology and why the sci-fi tech being announced is a soon to be reality. The talk dived into how design thinking concepts fit so well with the principles of the Agile Manifesto with a case study on a real-world local project. Rishal challenged the audience to define goals and strive towards them rather than being consumed by the problem space.

Design Thinking and Agile

Done is Better than Perfect

By Anita Iles, Operations Director / UX Architect at Isoflow

This talk works through a real-world case study to highlight the importance of getting something in the hands of people instead of trying to perfect a product before launch. Perfection is impossible, stop trying to achieve it. Anita explains that UX processes and practices should not be done just for the sake of it. The use of a few sensible processes can add more value to a project than trying to squeeze them all in.

Combining Empathy and Data in User Experience

By Richard Omolio, UX Designer at Booking.com

Data analysis, experiments, and empathy can improve your website’s user experience and help achieve business goals. This talk took the audience through the journey of tweaking and optimizing Booking.com for a better user experience. Richard explains that the immense traffic to the site helps their UX team gather and analyze data that can be used to make design decisions for the product. He also emphasizes the use of design experiments and A/B testing for deciding on design decisions based on the feedback from Booking.com data.

Combining Empathy and Data in User Experience

The UX South Africa conference was insightful, and filled with interesting and fun people. It was great to see local and international talent showcasing their experience and sharing their knowledge. We’re looking forward to the next one!

UX SA 2016

The REST Architectural Style

By Rishal Hurbans & Matt Van Der Westhuizen

In this era of digital technology, we find ourselves developing software that is required to be shared and utilised as far as possible in a manner that is as concise as possible. This is not a new concept, however, the need to refine and better these integrations have become more and more prevalent due to high demand for interoperability between different pieces of software.

An emerging trend is the use of RESTful APIs for both exposing data resources as well as consuming data resources from other sources. Within the development community, there exists some misunderstandings around what REST, and RESTful actually is. It is important to understand that REST is not a protocol, and it’s not a standard. REST is simply an architectural style achieved by combining a number of different well-known architectures.

Architectures

There exist commonly used architectures in most distributed web-based software. It is important to understand the purpose of these patterns and where it is applicable. This will help in understanding the principles of a RESTful architecture.

Client-Server

Architectures - Client-Server

The client-server architecture is a fairly well-known and widely used architecture. Most network based applications work in this manner. The architecture consists of a server that acts as a central point that one or many clients may interact with. The idea behind a client-server architecture is that the server does the heavy lifting and orchestration required, whilst the client consumes the rich data and functionality by interacting with the server. When these computers communicate with each other, there’s a clear need for a common protocol that is understood and supported by both parties. The commonality required is the message format. Imagine the protocol being a channel of communication between two people such as speech which is common between the two, and the message format being the language that is understood between them.

Layered

Architectures - Layered

The layered architecture is utilised for a number of different factors. A layered approach allows developers to separate concerns and loosely couple components that interact with each other. The advantage of this approach when implemented optimally is that layers may change independently without impacting the rest of the application. There are also cases where a layered architecture may consist of a layer that is shared across multiple other layers. An example of this is a shared domain that is utilised throughout the application. The layered architecture implies that the various layers reside within the same application container.

Tiered

Architectures - Tiered

The tiered architecture is very similar to the layered architecture. It consists of the same pattern and characteristics. The terminology “tier” is mapped to the terminology “layer”. The key difference between a layered architecture and a tiered architecture is that each tier resides on a different physical or virtual environment. This implies that tiers that interact with each other consist of different deployments and environments.

Stateless

Architectures - Stateless

A vast majority of web applications make use of a server-side managed session to keep track of a specific client. These sessions are often used for auth (authentication and authorization), keeping track of context, and storing meta data in memory that may be useful for managing the user’s activity. This makes scaling difficult as additional technologies and development is required to create session management servers in a clustered environment. The stateless architecture removes the need for the server to create and hold sessions. In a stateless architecture, the state is managed by the client via some mechanism such as tokens, header data, etc. that are passed back and forth between the client and server.

Cacheable

Architectures - Cacheable

The cacheable architecture is the concept of cleverly caching data that is used often, and changed infrequently. Caching may occur in places like the client browser, and a caching server. Caching mechanisms are expected to be smart enough to serve cached data or resources until that piece of data or resource changes for the client requesting it. One of the advantages of caching is a reduced load on the network, this translates to reduction of unnecessary requests, lower data usage, and a more optimal performing application.

Uniform Interface

Architectures - Uniform Interface

The concept of a uniform interface includes creating a standard method of interacting with a system. The uniform interface architecture also promotes abstraction of the implementation from the interface definition. This allows for clients to interact with all services in the same way, using a standard protocol and message format with a well defined set of operations.

REST Architectural Style

The REST architectural style was documented by Roy Fielding in his dissertation in the year 2000 in an effort to provide a documented architecture that could be used to guide the development and design of the modern web. The first part of this paper characterised a large number of existing architectural constraints of which the ones covered above are only a subset.

In the second part of the paper REST is defined by composing a number of these architectural constraints into a cohesive whole. The patterns used are:
1. Client Server: This constraint provides separation of concerns, improves scalability and improves the portability of the interface.
2. Stateless: This constraint requires that each request contains sufficient state for the server to understand what to do… this provides visibility, readability and scalability.
3. Cacheable: This constraint requires that all responses from the service to its clients are explicitly labelled as cacheable or non-cacheable so that intermediary servers or components can cache responses. This improves efficiency, scalability and user-perceived performance.
4. Uniform Interface: This constraint defines a simple standard for working with a REST service. It includes identification of resources by URI, manipulation of resources through representation, self-descriptive message and HATEOAS.
5. Layered System: This constraint requires that a client should not be able to tell if it is connected to the end server or to an intermediary. Layered systems cause a reduction in performance, but this should be counteracted by caching.
6. Code On Demand (optional): This optional constraint allows client functionality to be extended by downloading and executing code from the server.

Fundamental Concepts

Resources

A logical resource is any concept that can be addressed and referenced using a global identifier, for example a car, dog, etc. URIs are used as the global identifier when implementing REST over HTTP.

Server

A logical server where resources are located, together with any corresponding data storage features.

Client

Logical clients make requests to logical servers to perform operations on their resources.

Request & Response

The client sends a request consisting of headers and optionally a representation. The server responds with a status code, headers and optionally a representation.

Representation

The representation contains some or all information about a resources and allows modification of the resource on the server by modifying the representation and sending it back to the server.

Richardson Maturity Model

The Richardson Maturity Model was first introduced by Leonard Richardson in a 2008 conference talk as a way to judge the quality of a RESTful web API. It assumes that most of the RESTful constraints are met by using web technologies and looks at how an API uses HTTP, URIs and HyperMedia to determine its quality.

Richardson Maturity Model

Level 0 – The swamp of POX

The swamp of Plain Old XML describes an API that is not RESTful at all. An example would be a SOAP-like web service using HTTP as transport, but not using any of the web mechanisms to its advantage. In the example below the API user will always POST a request to the same URI. What action is performed is controlled by the contents of the request.

The scenario used in the code examples in this section references the idea of having a conference where attendees have questions for presenters on a specific talk. Given this; talk and question, are resources that we would like to interact with.

Create a question
POST /createQuestionService HTTP/1.1
<createQuestion talk="Robust REST Architectures" question="Do you guys even REST?"  name="HelloPieter" />

HTTP/1.1 200 OK
<talk>
    <questions>
        <question talk="Robust REST Architectures" question="Do you guys even REST?" username="HelloPieter" />
        <question talk="Robust REST Architectures" question="What is HATEOAS?" username="CodeWarrior" />
    </questions>
</talk>

What’s bad about the above example is that the service is essentially a black box. Without documentation it’s pretty much impossible to figure out what you can do with the service.

Level 1 – Resources

The first level of RESTfulness begins to get rid of the black box by introducing the concept of resources. Instead of having a single URI for everything our API does we have a URI for each resource that can be manipulated with the API. This simplifies interaction with the API to working with a number of smaller, simpler boxes.

List talks
# Request
POST /talks HTTP/1.1

HTTP/1.1
<talks>
</talks>
Create a talk
POST /talks/create HTTP/1.1
<talk name="Robust REST Architectures" />

HTTP/1.1 200 OK
<talk id="1" name="Robust REST Architectures">
    <questions>
    </questions>
</talk>
Create a question
POST /talks/1/questions/create HTTP/1.1
<question question="Do you guys even REST?" username="HelloPieter" />

HTTP/1.1 200 OK
<talk id="1" name="Robust REST Architectures">
    <questions>
        <question id="1" question="Do you guys even REST?" username="HelloPieter" />
    </questions>
</talk>

Level 2 – HTTP Verbs

The second level of RESTfulness is attained by using HTTP verbs to implement the uniform interface constraint. That is instead of using URIs to specify what action is being taken as in the previous example, URIs only specify the resource that is being acted on and the verb specifies the operation.

List talks

GET /talks HTTP/1.1

HTTP/1.1 200 OK

<talks>
</talks>

Create a talk

POST /talks HTTP/1.1
<talk name="Robust REST Architectures" />

HTTP/1.1 200 OK
<talk id="1" name="Robust REST Architectures">
    <questions>
        <question id="1" question="Do you guys even REST?" username="HelloPieter" />
    </questions>
</talk>

Add a question

POST /talks/1/questions HTTP/1.1
<question question="Do you guys even REST?" username="HelloPieter" />

HTTP/1.1 201 Created
Location: /talks/1/questions/1

Update the question

PUT /talks/1/questions/1 HTTP/1.1
<question username="PieterMan" />

HTTP/1.1 200 OK
<question id="1" talk="1" question="Do you guys even REST?" username="PieterMan" />

View the talk

GET /talks/1 HTTP/1.1

HTTP/1.1 200 OK
<talk id="1" name="Robust REST Architectures">
    <questions>
        <question id="1" question="Do you guys even REST?" username="PieterMan" />
    </questions>
</talk>

Delete the question

DELETE /talks/1/questions/1 HTTP/1.1

HTTP/1.1 200 OK
<talk id="1" name="Robust REST Architectures">
    <questions>
        <question id="1" question="Do you guys even REST?" username="PieterMan" />
    </questions>
</talk>

Level 3 – HATEOAS

The acronym HATEAOS means Hypermedia As The Engine Of Application State. In essence it involves adding links to your representations to make your API explorable, similar to how links in HTML documents make web sites explorable.

List available API resources

GET /api HTTP/1.1

HTTP/1.1 200 OK
<api>
    <links>
        <link>
            <rel>talks</rel>
            <href>/api/talks</href>
        </link>
        <link>
            <rel>questions</rel>
            <href>/api/questions</href>
        </link>
    </links>
</api>

List all talk resources

GET /api/talks HTTP/1.1

HTTP/1.1 200 OK

<talks>
    <talk>
        <name>Robust REST Architectures</name>
        <links>
            <link>
                <rel>self</rel>
                <href>/api/talks/1</href>
            </link>
            <link>
                <rel>questions</rel>
                <href>/api/talks/1/questions</href>
            </link>
        </links>
    </talk>
</talks>

The Glory Of Rest

So in summary the ideal REST API is a self-documented interactive service:
* GET /api gives a list of all resources with their links
* Following those links allow you to explore all of the resources
* Using HTTP verbs on those resource URIs allow you to perform CRUD operations

Overview of HTTP

The most popular protocol used for RESTful implementations is HTTP. HTTP (Hypertext Transfer Protocol) is an application protocol for distributed hypermedia information systems. HTTP is widely used in networked computing, it is hugely prevalent in how the internet works. This section describes the basic concepts of the protocol.

URLs (Universal Resource Locators)

Architectures - HTTP URLs

HTTP URLs consist of several components including:
* Protocol: The protocol that the request is being made on. Typically HTTP or HTTPS. HTTPS is a secured version of HTTP. It requires a certificate to validate the authenticity of the server in relation to the domain.
* Domain: The domain includes the address of the server. Typically this is represented by a sub-domain, a domain name, and a top level domain. In the example www.prolificidea.com, www is the sub-domain, this is the default sub-domain for the world wide web. prolificidea.com is the registered domain name.
* Port: Different applications can communicate via different ports on a computer. Typically, port 80 is the default port for HTTP web application servers.
* Resource: The resource being requested by the client. This could be any media including virtual objects. An example is the /user resource gives the client data about users.
* Query: Queries are used to provide parameters for the resource being requested. An example is the /user?name=bob specifies that the client wants a user with the name “bob”.

Verbs

HTTP provides several well known, widely used verbs for operations on resources, including:
* GET: Read data or resources. This verb should never be used to modify data.
* POST: Create data or resource. This verb should be used to create new data.
* PUT: Update data or resources. This verb should work on existing resources. The same PUT request should be able to be repeated with the same end result.
* DELETE: Delete data or resources. Similarly to PUT, a DELETE request should be able to be repeated with the same end result.

HTTP provides operations that are less known, but still useful:
* PATCH: Patch is used to update data or resources similarly to PUT. The difference is that PATCH requests should be used to update parts of a resource instead of updating a resource in its entirety. PATCH reduces the need to update complete objects by updating only the properties that require an update.
* HEAD: Head is similar to GET requests, however, the request should not contain a body. This request is useful to validate data or resources.
* TRACE: Trace invokes a remote, application-layer, loop-back. The TRACE request reflects the message that the final recipient server receives. This is useful to determine the final message after a request is proxied along a chain of servers. The TRACE request is useful for diagnostics on applications.

Status Codes

  • 1xx: Information. The request was received by the server, continue processing.
  • 2xx: Successful. The request was received, understood, accepted, and processed by the server.
  • 3xx: Redirection. Additional action is required to complete the request.
  • 4xx: Client Error. There was a problem with the request received by the server.
  • 5xx: Server Error. The request was received, understood, and accepted by the server, however, a server error occurred in processing the request.

Many applications tend to use the 200 OK status code for all responses to the client. In the case of errors, additional information is stored in the response message body. This should be avoided. The HTTP status codes should be leveraged and used correctly in the given context. An example being, a POST request should yield a 201 Created status code in the response rather than just a 200 OK.

HTTP/2

Architectures - Layered

HTTP/2 is a new faster version of HTTP. The aim of HTTP/2 is to make applications quicker and more robust. HTTP/2 was spawned from Google’s SPDY protocol (SPDY is not an acronym, Google just thought is sounded cool). The biggest change is the method of transporting data. Data is being transported using binary rather than using text as HTTP/1 does. Binary data is smaller than text and therefore makes it more efficiently transported over a network. Nothing changes in terms of the semantics of the protocol. The same operations and standards exist when using it. This change shouldn’t affect the average web application developer. Developers that build browsers, application servers, or applications that work directly with the network layer will be impacted by the changes of HTTP/2.

Message Formats

REST is not tied to a specific message format. Initially XML was used, but in recent years JSON has become more popular as many clients are easily able to interpret JSON messages. In theory there’s nothing preventing you from creating a RESTful service that uses images as representations.

Message Formats: XML

Extensible Markup Language was introduced in 1996 and forms the basis for SOAP-WS standards. Here’s an example of a valid XML document:

<?xml version="1.0" encoding="UTF-8"?>

<Person>
    <Name>Matt</Name>
    <Surname>Van Der Westhuizen</Surname>
    <Hobbies>
        <Hobby>Boardgames</Hobby>
        <Hobby>GameDev</Hobby>
        <Hobby>Reading</Hobby>
    </Hobbies>
    <PhoneNumbers>
        <PhoneNumber>
            <Type>home</Type>
            <Number>1234567890</Number>
        </PhoneNumber>
        <PhoneNumber>
            <Type>cell</Type>
            <Number>0821234567</Number>
        </PhoneNumber>
    </PhoneNumbers>
</Person>

As you can see the syntax is quite verbose and perhaps a bit ugly. Some of the good features of XML are that you can create customised mark-up and there is versatile standards and tooling around XML (e.g. XSD, XSLT, XPath / XQuery, etc). The main downsides are the verbosity and the complexity of the standards leading to slower more complex parsing code.

Message Formats: JSON

JavaScript Object Notation was first publicised in 2002 (based on the date the JSON.org website was launched). It supports only three primitive types:
* Scalar values
* Arrays
* Objects

Here is an example of a valid JSON document:

{
    "name": "Matt",
    "surname": "Van Der Westhuizen",
    "hobbies": [
        "Boardgames",
        "GameDev",
        "Reading"
    ],
    "phoneNumbers": [
        {"type": "home", "number": "1234567890"},
        {"type": "cell", "number": "0821234567"}
    ]
}

JSON is very easy to consume on web clients thanks to being a subset of JavaScript. The limited number of concepts make parsing the data on non-web platforms quite easy to implement. JSON has less data overhead thanks to the lack of closing tags enclosing each item and no envelopes as are typically the case with SOAP services built on XML. Thanks to this simplicity there is little to no need for complex tooling.

The one downside of JSON is that it is not as rigorous as XML, but there are some efforts underway such as JSON-HAL which defines a conventional REST message format and JSON Schema which is similar to XSD for XML and can be used to define the structure of JSON resources.

Message Formats: HAL

Hypertext Application Language defines a structure for Resource representations. You still have your resource state as the root of the object as you used to, but it requires you to add a links section and specifies how to embed resources inside another resource. This diagram illustrates the concept quite nicely:

HAL Resource

Here is an example showing a collection of users containing some embedded user resources:

{
    "_links": {
        "self": { "href": "/users" },
        "next": { "href": "/users?page=2" },
        "find": { "href": "/users{?surname}", "templated": true }
    },
    "_embedded": {
        "users": [
            {
                "_links": { "self": "/users/1" },
                "id": 1,
                "name": "Rishal",
                "surname": "Hurbans",
                ...
            },
            ...
        ]
    },
    "page": 1,
    "pages": 4
}

One of the benefits of HAL is that it can work with either JSON or XML, but the downside is that there is no agreed upon way to specify validation rules.

Message Formats: JSON Schema

JSON Schema is similar to XSD for XML, but only works for JSON resources. It provides a format for providing schema meta-data about resources that can be used for validation. It also has an extension for defining resource hypermedia which works similarly to HAL’s _links.

Here’s an example of JSON Schema for a person which would allow us to validate that person’s name is a string and that they should have at least 1 hobby:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Person Schema",
    "type": "object",
    "properties": {
        "name": {
            "type": "string"
        },
        "hobbies": {
            "type": "array",
            "minItems": 1,
            "items": { "type": "string"}
        }
    },
    "required": ["name", "hobbies"]
}

Conclusion

The REST Architectural style is comprised of a combination of well known architectures. It leverages the benefits of each architecture to provide a consistent, yet flexible pattern for building interactive services. The Richardson maturity model is useful as a yardstick to determine ease of use and extensibility of your RESTful implementation. Although typical RESTful implementations operate over HTTP using JSON as a message format; one could potentially use any protocol and message format. In theory one could create an API that conforms to the principals of the REST architectural style that uses the FTP protocol with images as the message format – it’s just questionable if that would be useful? Bottom line, is that the REST architectural style is a guideline, it’s open to creativity (and abuse). A truly elegant RESTful API is a combination of following these guidelines as well as designing resources and interactions that solve real-world problems in a sensible way.

References

Building a HTML5 Space Invaders Visualiser

game

Space Invaders is a classic game that most of us have enjoyed as a child. I needed to build a basic “Space Invaders” visualiser for the Entelect R100K Challenge. The challenge this year prompts contestants to develop a bot program to play the game versus another bot program. Each player has a turn to move their space ship, shoot, or build a booster building. To find out more details about the challenge and the details behind the mechanics, check out the rules. In essence, each iteration (or turn) of the game is recorded in a text based map. This map is pretty readable but doesn’t really embrace the atmosphere of the game. The aim of this visualiser is to give the contestants a tangible way to visualize their matches and provide some inspiration to contestants that are making their own visualisers for the Gooey Challenge. You can download the code on GitHub.

Overview

The visualiser needs to be accessible to everyone, so it made sense that it should run in a modern browser. This means that it needed to be developed using JavaScript. Plain old JavaScript would be a mission, and since time wasn’t on my side, making use of a rending engine or game engine made sense. So from past experiences playing with HTML5 game engines, I decided to use Phaser. It’s easy to use, provides the required features and is open source.

Since each game state is recorded in a different text file, it’s pretty obvious that the minimum requirement of the visualiser is to read each game state in sequence and play them back. To make it visually appealing, each character that represents a game item should be rendered as a sprite that’s a bit more interesting.

Making Some Artwork

The first thing I decided to do is create some artwork that would represent the game. I didn’t have much time, so making low detailed sprites was the only way to go, furthermore, the low detailed sprites gives the visualiser a retro feel to it which fits in really well with the spirit of the challenge.

The game objects are represented below. Most objects are 1×1 characters in size, while some are 3×1 – this includes space ships and buildings only.

AAA: Your Ship
VVV: Opponent's Ship
XXX: Alien Factory
MMM: Missile Controller
#: Wall
-: Shield
x: Alien
|: Alien Bullet
!: Your Missiles
i: Opponent's Missiles

So here’s the artwork for each of these objects:

AAA: Your Ship

ship-player

VVV: Opponent’s Ship

ship-opponent

XXX: Alien Factory

factory-alien

MMM: Missile Factory

factory-missile

#: Wall

wall

: Shield

shield

x: Alien

alien

|: Alien Bullet

missile-alien

!: Your Missile

missile-player

i: Opponent Missile

missile-opponent

Developing the Visualiser

Now that there’s some artwork to use as sprites in the visualiser, it’s time to put everything together. Again, we have a sequence of text files that represent a game state, and some images that will represent each character in a specific game state. So the following image depicts the goal for each game state.

examplemap

The Phaser game engine allows for initialisation with a canvas width, canvas height, the type of renderer, and a name for the object. Furthermore, Phaser allows for a preload function and an update function. This is useful since the assets and game states can be loaded in the preload function, and the update function is just responsible for rendering each text-based game state.

The Phaser Game is initialised with the following. It’s telling Phaser to make a new canvas named “phaser” with the correct width and height. It’s also specifying the functions that should be used for preloading and updating.

var game = new Phaser.Game(SPRITE_SIZE * COLUMNS, SPRITE_SIZE * ROWS, Phaser.CANVAS, 'phaser', {
 preload: preload,
 update: update
});

The preload function needs to load the game state text files, as well as the images for the sprites. Phaser conveniently handles this by allowing various types of assets to be loaded, this includes images and strings – which works out perfectly for this purpose.

In the preload function, a helper function named loadMaps(replayPath, iterations) is used to load all the maps, and the game.load.image(name, filepath) function is used to load all the images into the game.

function preload() {

 loadMaps('example-replay', 52);
 
 game.load.image('alien', 'images/alien.png');
 game.load.image('factory-missile', 'images/factory-missile.png');
 game.load.image('factory-alien', 'images/factory-alien.png');
 game.load.image('missile-alien', 'images/missile-alien.png');
 game.load.image('missile-opponent', 'images/missile-opponent.png');
 game.load.image('missile-player', 'images/missile-player.png');
 game.load.image('shield', 'images/shield.png');
 game.load.image('ship-opponent', 'images/ship-opponent.png');
 game.load.image('ship-player', 'images/ship-player.png');
 game.load.image('wall', 'images/wall.png');
}

Here’s the loadMaps function. It simply iterates for the number of game state files and loads them into the game as text strings.

function loadMaps(replayPath, iterations) {

 for (var i = 0; i < iterations; i++) {
  game.load.text('maptext' + i, replayPath + '/' + zeroFill(i, 3) + '/map.txt');
 }

}

You may notice the zeroFill() function. There exists a directory for each iteration of the game state, it is named with the iteration number and padded with zeros to make up a three character name. E.g. 1 becomes 001. The zeroFill() function is required to correctly pad the zeros for the correct game state path.

Now that the game sprites are loaded and the game states are loaded, they need to be replayed. The Phaser update function is important to accomplish this. Since Phaser handles the update rate, I needed to do some clever checks to achieve the frame rate that I wanted.

function update() {
 
 if (game.time.now - timeCheck > FRAME_RATE_DELAY_MS) {
  game.world.removeAll();
  var map = game.cache.getText('maptext' + iteration);
  generateMap(map);
  iteration += 1;
  timeCheck = game.time.now;
 }
}

The initial condition determines if the right amount of time has passed, if it has, then we can update the game and objects being rendered. Phaser doesn’t allow for user defined ticks, so this is necessary to delay the iterations of the game state.

Next, all the elements of the game’s world are remove, this is to prevent old objects from sticking around and obscuring the current game state.

A map string then gets populated with the current iteration’s game state from the cache that was populated in the preload function.

The generateMap(map) function uses the game state text, iterates over the string and adds the correct sprite based on the respective character. Some logic is needed for when 3 block objects and new line characters. To find out more about this, have a look at the code on GitHub.

Finally, the iteration number is incremented and the time of rendering is recorded for use in the next iteration.

And it’s that easy to create a simple but effective visualiser using Phaser and JavaScript.

Adding Juice

Part of what sets a game or visualiser apart from the rest is the subtle detail and effort put into it. Here are some ideas for improving the above visualiser:

  • Tweening transitions between each game state: As you can see, the visualiser is very jittery, this fits for the retro style of this visualiser but a differently styled visualiser will be greatly improved by simply tweening the iterations.
  • Particle effects for collisions.
  • Animations when destroying and rendering interactions between sprites.
  • Sound effects for shooting, collisions, and spawning buildings.
  • A menu system for loading different matches.

Ludum Dare 32

This is my first time competing in a game jam in almost a decade. Don’t get me wrong, I’ve been prototyping and thinking of game concepts, however, due to time limitations, I’ve never really got something out there. I’m probably amongst many that are in the same predicament and I think LDGJ is a perfect platform to get guys like us creating something tangible.

The game I’ve made for this challenge is called “Beat”. When I thought unconventional weapon, I thought, how about music? In this game, you play as a speaker – yes, that thing that produces sounds. The aim is to gain as many points by beating as many monsters as possible. Each type of monster dies by only one chord, other monsters get more angry and run at you faster. It costs one point to play a chord and when a monster gets to you, well, that’s game over.

I have many other mechanics that I would like to add to this game and I welcome any of your ideas.

I hope you enjoy playing this game – I enjoyed making it and I hope to improve it in the future.

Made with Unity, Garage Band, and Photoshop.

You can find OSX and Windows links here.

8ca8cca730892451147a0b3123cd90bc3db1648e

8ca8cca730892451147a0b3123cd90bc3db1648e

3D Programming with JavaScript

Overview

This article is based on talks I gave at JSinSA 2014 and Entelect DevDay 2014.

8ca8cca730892451147a0b3123cd90bc3db1648e

I needed to develop a 3D interface for simulating a two player version of PacMan. I also enjoy experimenting with different tools for game development, and so I decided to look into developing 3D applications for the web.

This article explains some of the concepts that will be useful when learning 3D Programming with JavaScript. It also contains some information about popular JavaScript based 3D and physics APIs that can ease the development process and let the developer work on ideas and not technical plumbing.

Modern Browsers

57d9dbe522b76251856e0d9f12a1850c448969e5

Modern browsers have evolved drastically. Browsers were initially only capable of rendering HTML and submitting forms, they were later enhanced to enable developers to load parts of the page asynchronously. The demand for more interactive applications on the web grew and led to the development and use of browser plugins like Flash. With the demand for interactive applications and compatibility on different platforms like mobile devices, browsers needed something native to allow developers the freedom to develop applications across many platforms. This is the creation of <canvas>.

The canvas element in HTML is simply an element that allows drawing of shapes on the page.

To accomodate different drawing needs, the canvas provides two different contexts:

2D Context

The 2D context allows for drawing simple shapes in a step by step manner. Drawing shapes involves setting the brush properties before drawing a specific shape. The 2D context makes it easy to draw text, lines, arcs, rectangles, etc.

3D Context

The 3D context is in the form of the WebGL API. WebGL allows for drawing any 3D geometry from any perspective or point of view. This allows for more complex shapes and geometries to be drawn.

Web Browsers

Most modern browsers as well as their mobile version counterparts support WebGL. These include, Google Chrome, Safari, Firefox, Opera, and even latest versions of Internet Explorer. This is great for diverse distribution of your application.

I don’t know about you, but I like my browser like i do the rims on my car…Chrome.

WebGL

WebGL means “Web Graphics Language”. WebGL uses the canvas element to render drawings, just like the 2D context. WebGL is a JavaScript exclusive API, this is great because it works on a variety of different operating systems and browsers. WebGL is based on the OpenGL Embedded Systems 2.0 specification, due to this, it has support for general mobile device hardware. Best of all, WebGL is royalty free, there is no licence required to use the API.

Hardware Acceleration

WebGL leverages off the GPU (Graphics Processing Unit) for hardware acceleration. GPUs are geared towards graphics calculations and rendering. The aim is to calculate and render complex graphics as efficiently as possible – this is not a job for the CPU.

What’s in the box?

3D rendering consists of two main concepts, the Vertex Shader, and the Fragment Shader. This might sound complicated, but in essence:

  • The Vertex Shader is a position calculator. It handles the mathematics and calculations for converting points so that they are positioned correctly.
  • The Fragment Shader is a colour chooser. It determines what colour different elements in the 3D space should be.

Coding with Raw WebGL

WebGL is a API and has brought amazing capabilities to modern browsers. As with any raw 3D development, there are some pains.

  • WebGL has many settings and configurations. One needs to understand what settings are required, how they are used, and what the correct setting should be. These are typically settings and configurations for the Vertex Shader and Fragment Shader.
  • Rendering simple 3D shapes with WebGL can be cumbersome due to the amount of code required. WebGL expects the developer to provide all the vertices of geometries (Creating a cube involves approximately 112 lines of code). Developing and debugging this can potentially be a nightmare.
  • Too much plumbing and mathematics, not enough fun. We all want to see our ideas come to life, and not be bogged down by details.

These pains can be avoided by using a 3D library like three.js, but before jumping straight into it, there should be a way to create and maintain a standard JavaScript project.

Standardised JavaScript Projects

27191161f7130c4da60024b4a3727cd77fcddd2e

When starting out with using a new technology, learning a new language, or building any project in general, it’s good to have a basic template to work from. Usually the template will take care of any settings that the project requires to run, managing dependencies, managing builds, etc.

When working with JavaScript, there are an abundance of tools that will be useful in developing a project.

Yeoman

Yeoman is a great tool for generating a standard JavaScript project. Yeoman includes generators that will provide you with a boilerplate project for most popular JavaScript frameworks like Node.js, Angular.js, and many more. Yeoman generates directory and file structures with default Grunt and Bower configurations for the respective project.

Grunt

Grunt is used to build and check your project for correctness in terms of syntax and the general semantics of JavaScript. Grunt will check your JavaScript files in realtime and notify you of any issues with your code as they happen. Grunt also has a nifty lightweight HTTP server that can be used to live deploy projects and test them during development.

Bower

Bower is a dependency management tool, it allows for dependencies to be imported and included in a project automatically. With bower, there is no need to download 3rd party dependencies manually.

Setting Up a Three.js Project

The following commands will work on any UNIX system.

Install Yeoman

npm install -g yo

Install the Three.js generator

npm install -g generator-threejs

Make a new directory for your project

mkdir threejs-project

Navigate to your new project directory

cd threejs-project

Generate a Three.js project with Yeoman

yo threejs

Deploy your project with Grunt

grunt serve

3D – THREE.JS

Three.js is a cross-browser JavaScript API for 3D programming. It allows developers to create 3D scenes and applications with ease. Three.js simplifies 3D programming by providing simple operations for common tasks. Three.js handles all the mathematics and basic 3D setup and configurations for the developer.

With raw WebGL, it takes around 112 lines of code to create a simple cube. Three.js allows for this to be done in a single line of code.

There are a few concepts related to Three.js and 3D Programming in general that should be understood before embarking on a project.

Scene

4f7c1d1020f0132295cff141e54a9becadc6ee45

The scene is a container for 3D objects. The scene will hold everything that is in the 3D world. This includes any object in the 3D world.

Camera

0bac45c1487f2255fd81b69ae266fddbee728905

The camera is an object that cannot be seen be the user. The scene is displayed to the user from the perspective of the camera.

Controls

74eed69f87fbb50aa1c8290d8738c502a2837840

Controls could be mouse, keyboard, touch events, gamepads, etc. Controls can be used to move and manipulate the camera or any other object in the scene.

Objects

e200afa27e4e506f8e09076f08ab796c041da123

Objects are 3D entities in the scene. These can be something as simple as a cube, to something as complex as 3D humanoids, vehicles, buildings, etc. It’s great that we can create objects, but what are they made up of?

734a24c8f6aec898cb998cd4793c0679808439e8

  • Object Geometry: The geometry is the shape of the object. This includes all the points that make up the object as well as their positions relative to each other.
  • Object Texture: The texture is typically an image that that is overlaid over the geometry. It gives the shape any aesthetics and any effects that is required. It is the skin for the geometry.

Renderer

The renderer is responsible for assembling the scene and it’s objects, and displaying it to the user from the perspective of the camera.

Physics – Cannon.js

When developing 3D applications, more often than not, there is a requirement for physics, gravity, or collision detection.

It is tedious and difficult to write custom collision detection and physics code, again, we want more fun and less mathematics and plumbing.

Cannon.js is a useful API that is compatible with Three.js for physics. Cannon allows developers to bind to existing Three.js 3D objects and perform physics calculations and manipulations on them.This is useful for simulating gravity and creating worlds where collisions actually result in an effect instead of objects overlaying each other.

USES & ADVANTAGES

3D Programming with JavaScript has many uses, here’s just a few.

  • Great for game development
  • Interactive applications for marketing
  • Simulations
  • Write once, run almost anywhere

DEMO

The demo is a 3D world that renders a Twitter hashtag or search term. The application listens for tweets using the mentioned hashtag and populates the 3D world with the Twitter handles of the users that made the tweet. The aim is to display the 3D world at events and watch as the world gets more populated as the event goes by and more people tweet about it.

You can check it out here: http://www.prolificidea.com/tweetd.html

It works best on desktop as WSAD and mouse are used to control the camera within the world.

References

@RishalHurbans

Prototyping & Unity3D

Overview

I’ve been involved in quite a number of projects that required the development of prototypes in a very short period of time. These prototypes were meant to be developed to evaluate the viability of the ideas and sometimes to serve as a proof that the application can be developed given the constraints. I recently gave a talk on rapid prototyping and Unity 3D at the Entelect DevDay. The talk was comprised of some basic theory of prototyping, as well as an overview on Unity 3D. The twist in my talk was that the presentation was a prototype that I built with Unity 3D and the content of the prototype illustrated the topics on prototyping that I spoke about. This article will highlight some of the theory and concepts of prototyping, as well as showcase the Unity 3D game that illustrated the said concepts.

ScreenshotPrototyping

In this section of the talk, I cover the concepts around software prototyping, and some general theory and strategies in tackling the development of a prototype.

What is Software Prototyping?

Prototyping is the process of creating incomplete versions of the proposed software. That sounds bad, it’s incomplete! Well, the aim of prototyping is to simulate only a few key aspects of the solution to evaluate the viability of ideas in terms of cost, and complexity. Often projects are taken on without completely understanding the effort behind achieving the requirements – prototyping can assist in the analysis of these requirements and assist in learning more about the problem domain.

Why Prototype?

  • Early user acceptance testing: Users get a chance to use and experience the product early in development. This will result in early feedback from the user base and thus allow for changes to be implemented earlier rather than later. It’s a known fact that the cost for a change in a project increases significantly in later phases of development.
  • Realise requirements and constraints that were not previously considered: By simulating some of the functionality for a product, the developer may realise side effects, constraints, or additional requirements that were not thought about. This assists in achieving a more complete and robust solution.
  • Better cost, time, and complexity estimates: By realising additional requirements and constraints early, as well as receiving user feedback early, one can make better complexity and time estimates – this overall results in better costing estimates.
  • Slaying the dragon: In software development, we speak about slaying the dragon – where a single team of heroes attempt to slay a large project/dragon. With software prototyping, we try to make slaying the dragon more like shooting fish in a barrel by tackling smaller or more complex features first.

The Process of Prototyping

  1. Identify Core Requirements: These are the requirements for the product.
  2. Develop Initial Prototype: Prototype the features that are important depending on the goal of the prototype. If complicated features with unknown possibilities exists, then tackle these first. If there are many simple features, try simulate an experience across all these features without delving into the complexity in each.
  3. Evaluate and Review the Prototype: The developed prototype should be reviewed with the target user group. The performance of the features and usability should be evaluated and measured.
  4. Revise and Enhance the Prototype: Given the feedback from reviewing the prototype, enhancements and changes can be made.
  5. Repeat: If time permits, or there still exists unknowns, the above process should be repeated.

Dimensions of Prototyping

Horizontal Prototyping

  • The aim is to provide a broad view of the entire system.
  • There will be little complexity in individual features.
  • This approach is good for websites and instances where a general feel for the product needs to be achieved- Typically applications that are targeted to the public, or applications that require intensive usability testing.

Vertical Prototyping

  • The prototype will focus on a small set of features, even one or two features.
  • The chosen features are explored and researched completely.
  • This approach is good for products where an obscure algorithm is used or something unusual or unorthodox is attempted. This is useful for applications where complex logic and processing is required.

Types of Prototyping

  • Throwaway Prototyping: This is also known as close-ended prototyping. Rapid Prototyping involves creating a working model of various parts of the system at a very early stage of development, after a relatively short investigation. This kind of prototyping is useful to show users what the feature will look like, however, the code base or project is not necessarily used for the production version of the application.
  • Evolutionary Prototyping: The main goal when using Evolutionary Prototyping is to build a very robust prototype in a structured manner and constantly refine it. The prototype forms the heart of the production application and additional features are added to it.
  • Incremental Prototyping: In incremental prototyping, parts of the system are developed as separate prototypes and plugged together to form a complete application. It is important to develop the interfaces for the separate components early, as integration may turn out to be a nightmare.
  • Extreme Prototyping: Extreme Prototyping is employed mainly for web applications. Usually in 3 phases.
  1. Static HTML is created – this gives users an instant tangible feel for the product.
  2. Thereafter, the service layer is simulated – this includes business rules and logic.
  3. Lastly, the actual service layer is developed – this involves creating a data layer as well as plugging into the front end HTML views.

This gives users an early view of the application without having actual functionality behind it as the backend will gradually come together as the process moves along.

Advantages of Prototyping

  • Reduced time and costs: By exploring the requirements and constraints, effort is better estimated.
  • Improved and increased user involvement: User involvement is important and prototypes clear up misconceptions, expectations, as well assists in gathering user feedback from early stages of development.
  • Realise oversights, additional requirements, and constraints.

Disadvantages of Prototyping

  • Insufficient analysis: The confidence in a prototype could result in further analysis of features to be abandoned. This could result in part of the system being well defined whilst the remaining parts are vague and incomplete. This can be controlled through correct processes in requirements analysis.
  • User confusion between the prototype and the finished system: If the final system is completely different to the prototype, users may be confused in how the application operates. This can be avoided by following the correct prototyping principles.
  • Expenses of implementing prototyping: Although prototyping saves cost in the actual development phase, there will be costs involved in implementing a prototyping phase. A prototyping phase should only be included in projects where it makes sense.

Unity 3D

Unity 3D is a cross-platform game engine with a built in IDE and designer. Unity 3D allows for the development of games and interactive multimedia-rich applications. The approach is write once, deploy everywhere! Unity 3D allows for applications to be deployed to platforms such as iOS, Android, Windows, OSX, and even consoles. Unity 3D also has great support for running your application within a browser using it’s custom plugin.

Concepts

Unity employs typical object oriented concepts when building applications. Objects have a representation by some image/sprite, 3D model, or sound. Objects also have components with different behaviour attached to them – by plugging components to objects, we can achieve any behaviour we want. Unity 3D comes bundled with predefined logic for typical game related functionality such as physics, player controls and effects – but what about custom functionality?

Code

Unity allows you to write functionality with and imperative language and scripting language, namely C#, and JavaScript; using the MonoDevelop Environment. MonoDevelop is a cross platform IDE. This means that you can write Unity applications on a range of platforms such as Windows, Linux, and OSX.

What’s the point if we don’t develop games?

Unity can be useful in prototyping a basic idea and testing viability across different platforms, e.g. mobile devices. Screens can be easily mocked up and small pieces of page navigation etc. can be added to receive early user feedback. Although unity is a game engine, you can develop almost any kind of interactive application with it, fast…

…and of-coarse, Unity 3D can be used to develop games! Almost everyone that’s a dev now had a dream to develop the next Super Mario or Duke Nukem in their youth.

Demo

During my talk, I then delved into the practical aspects of creating a new Unity 3D project, scenes, and objects. I also explained the use of components, assets, and creating custom components using C# and JavaScript.

Credit