Spark

Deploying standalone Spark distributed computer

http://spark.apache.org/docs/latest/spark-standalone.html

Hadoop - Distributed Operating System

Distributed Operating System

When considering a cluster computer, the main architectural solution would be to use a distributed software model. Distributed programming takes a Big Data processing task and breaks it into smaller-size processes that can run on separate CPU cores. The advantage of correctly designed distributed approach is that if any process fails, it can be re-run on different processor without data loss and dos not effect the final outcome.

If there are more processes to be run then available CPU cores, in well designed distributed computer, adding more computing units, or nodes, will scale up the architecture making extra CPU cores readily available.

Another benefit of distributed computing is that each process takes a small chunk of data, e.g. 64-124MB instead of 3GB as it is often the case with whole human DNA sequences. Small data processes can be easily run on inexpensive hardware such as Pi board featuring 512 MB of RAM.

In summary, Pi boards are a good candidate for distributed computing as can be rapidly scaled up to required processing needs by simply adding a couple of additional boards at the cost of $16-$30 per 4 CPU unit.

Hadoop

Hadoop is a set of open source Java software tools designed to be used as operating system for distributed computer.

The two major components of Hadoop are "Distributed File System", or HDFS and Yarn. The HDFS is used to store Big Data in small chunks across distributed set of inexpensive disks, e.g. SD cards in Pi boards. YARN is a manager for distributed processes.

There are multiple other software components that are part of Hadoop, but HDFS and YARN are essential for running the distributed computing software.

I am maintaining this article on:
https://github.com/UkiDLucas/book-genomics-easy-as-pi/blob/master/en/05a-Hadoop.md

Android Wear

/Users/ukilucas/Library/Android/sdk/platform-tools

> platform-tools $ ./adb devicesList of devices attached17625eb8 device
> platform-tools $ ./adb forward tcp:4444 localabstract:/adb-hub

> platform-tools $ ./adb connect 127.0.0.1:4444

connected to 127.0.0.1:4444

I have started on the Android Wear for my Moto 360 to START/STOP the Warrior Fit app exercises and collect the gyro data to detect the reps and exercise correctness.

The styling of the app (buttons, etc.) will come soon.

Detecting exercise type and repetitions using mobile phone gyroscope

Today, I have been experimenting to see if I can detect the EXERCISE pattern and number of REPETITIONS using Android gyroscope:

Exercise: 

leaning forward - literally bowing forward, 3 repetitions, phone centered on sternum, upright.

You can see by: 

- X axis (blue) I am clearly doing 3 exercises

- Y axis (green) is not changing, I am not leaning to the side

- Z axis (yellow) is not changing, I am not moving up or down

- Acceleration (orange) I was going faster at first and then slowing down 

It it THAT SIMPLE!

Now, I just need to ROUND OFF the numbers to DISCRETE VALUES and DIVIDE the exercise by the reps :)

Android: using zipalign command line tool to align APK for Prod Deployment

Aligning APK with Zipalign command line tool



usage:

zipalign : you can find that tool in your Android/sdk/build-tools/23.0.3/ folder

-f : replaces existing file

-v : verbose output

4 : required by Android alignment bytes

input apk : already signed APK as it comes out from Android Studio

output APK : file you will upload to Google Play Developer Console



> 23.0.3 $ pwd

/Users/ukilucas/Library/Android/sdk/build-tools/23.0.3



> 23.0.3 $ 

./zipalign -f -v 4 ~/CyberFit-release-23.apk ~/CyberFit-release-23_aligned.apk

Clarity of Code over Performance

It happens very often developers and architects get into very academic (entertaining and useless) discussions about a virtue of using one approach over another. Often, the academically right solution may increase future maintenance overhead costs, or cause poor user experience.

When choosing the solutions you should always prefer the clarity of user experience and readability of the code over small speed optimizations and kludgy (i.e. "clever") shortcuts.

It is always cheaper to scale up to stronger CPUs and more memory than re-write the application, in case of the mobile by the time you rewrite the app users will move from quad to octal-CPU phones and double the memory and speed.

Design and User Experience

Development for mobile is a wonderful opportunity for beautiful and functional design that complements device’s technical abilities and screen properties. Most people underestimate the amount of work that goes into graphical design and its implementation.

Just as a food for thought, display is one of the most expensive parts of the mobile device today, did I spend extra several hundred dollars for the highest quality large screen to view your app stretched up from the poor-resolution, tiny screen?

I don’t want to re-iterate the existing and very good documentation available for both Apple iOS and Android devices, but included are few tips that may not be voiced in official websites.

UX and UI standards

Style Guide Naming Conventions

As soon as the first "visual assets" are created it is essential that the Style Guide Naming Conventions are created. It is very important that the whole team understands what we mean when we talk about "HEADER" and "PARAGRAPH". The team should have a dictionary of about couple dozen, or so style name conventions that are clearly defined and consistently used.

A note about fonts

It is important that Android, iOS, etc. developers have common naming standards even if one platform's "PARAGRAPH" font is "Roboto" and the other's is "San Francisco".

Keep in mind that particular style details will differ depending of the device size (iPhone 5, 6, 6+, Android medium DPI, extra high DPI, etc.) as well on the user's preferred language selection (i.e. English vs German). The PARAGRAPH font may may have 15 different sizes across different devices. San Francisco font has 2 families, each having 6 variants and it is different on iPhone and Apple Watch. It really takes some training to choose it correctly. If the UI designer is not aware of this the developers may, or may not choose the best option.

Color Palette for app flavors

The color palette should be well named, the apps rarely use more than a dozen colors, in fact they should not use more as the app becomes a carnival eye-sore. In most cases the names should NOT suggest the color, especially is you are creating flavors of the app in different color schemes.

More about naming

The names should NOT imply the implementation details of the object, nor their location, for example LOGIN_BUTTON_BLUE violates both of the rules.

The names should be always GENERIC_TO_SPECIFIC

Examples of acceptable style names:

• COLOR_PRIMARY
• COLOR_SECONDARY
• COLOR_BACKGROUND
• COLOR_HYPERLINK
• COLOR_CALL_TO_ACTION
• COLOR_INACTIVE
• COLOR_TEXT
• BUTTON_DIALOG_POSITIVE
• BUTTON_DIALOG_NEGATIVE

Style centralization and hierarchy

The base styles should be implemented in a central location for each platform, so the changes to whole applications can be made instantly.

The styles can be "inherited" when it makes sense, so the desired font family is (i.e. "Roboto") is mentioned only once in the code. The font variants and sizes should be indicated once per device size (i.e. Apple Watch, iPhone 5,6,6+, tablet sizes, etc). It is essential to maintain strict hierarchy with minimum duplication of code.

Common Design Deliverables

There should be clear expectations about type and timing of deliverables.

Typical materials to be delivered before work begins include:

• marketing standards that define acceptable usage of logos, fonts, colors, etc.
• style naming standards
• style guide per platform (Android, iOS, etc)
• style considerations for different screen sizes, fonts, margins, stretching, etc.
• special needs considerations for color blind and need of increased font sizes
• UX wireframe flow of the application (using Balsamique, etc.)
• UI mocks for the widget library (pickers, headers, pop-ups, buttons)
• UI mockups per feature (actually does not have to be pixel perfect)
• UI graphical asset delivery per platform (e.g. done via GIT enabled tool like SparkleShare)

Assets to match the screen

Web developers have one screen density (traditionally 72 DPI), that is all, the assets can be cut ("Photo-chopped"), or drawn as vector graphics. In adaptive Web design you can use different size images depending on the screen size. It is relatively simple.

Mobile devices have different screen sizes and different pixel density ranging from 72 DPI to high 400 DPI. Image on high DPI screen coupled be 6 times smaller then intended when 72 DPI image is simply copied to newest phone.

Mobile developers can use vector graphics to draw most (material style) widgets, even use vector graphics for iconography. Android have ability to use 9patch stretchable images as well.

However for the images that do not stretch they have to be provided in variety of sizes, usually at least three. It is especially important that icons as nice and crisp on every major device size.

Stretching of the images is a very bad idea that ends up in a poor quality product.

Apple is not Android

Many companies say incorrectly: “We want a consistent look across all platforms” which usually is a bad approach. All platforms have their own ways of doing things, navigating and users are used to certain interaction.

Blackberry cannot support the richness of iPhone and iPhone does not support features of Android. Some devices are too small to support side-by-side fragments. Some devices have physical menu buttons that should be supported. Some phones do not support Touch ID, or NFC. Some platforms use and expect desktop widgets.

Users of each platform are used to different type of experience, forcing them otherwise makes for unhappy customers. Each platform should be considered a different media channel with its own rules to follow.

Who is responsible for providing creative assets?

The designers should be able provide the implementation of their art which might include the following assets: HTML & CSS, mobile layouts, stretchable (9patch) images, etc.

Designer should be an integral part of the development team, the teams where there is much of "throwing over the fence" are usually dysfunctional and the end product suffers. I am not a big fan of one-off design orders over the Internet. You get the asset, but really you need a family off assets following a theme. You need assets that thinly integrate with the layouts.

If the designer provide only wireframes and Photoshop images that may work, but expect a lot of back and forth and frustration, sometimes even a major re-doing.

It is sometimes worthwhile to hire a design firm to provide an initial idea and creative direction, but each product development team will need an artist sitting together with developers and working together. If the company considers the mobile product as a long term strategy then the artists should become the core part of the team.

The designer should also be able to run emulators for all platforms and major screen sizes and know how to scale them to be able to predict the final user experience.

Pixel Density

When thinking about pixel density (DPI) imagine a balloon.

Think about a balloon that you blow to 50% capacity (phone size) and draw 1 inch square on it, than blow it to 70% capacity (7 inch tablet), at this point your original square is much bigger. Draw another 1 inch square, you use much less of balloon rubber to do that (less pixels). Blow up the balloon to 100% of the holding capacity (tablet), the original square got much bigger and fuzzy.

You do not want to use the phone graphics from small phone on large devices.

Fonts

You should not use more than 6 fonts: family, size, color combinations:

• H1

• H2
• Paragraph
• Hyperlink
• Subscript
• Inactive

Text sizes should be set using SP units on Android, EM on the Web. They are fairly consistentbetween different devices (scale-independent) and take in considerationuser settings.

Being lazy on tablets

It is a cardinal sin, and most of us are guilty of it, to make your phone apps stretch on tablets!

There is nothing more upsetting to people spending obscene amounts of money (up to $1,400) for their Pro line of tablets and see fuzzy phone app with paragraph font size equivalent to 64 px.

Web mobile app developers should at least allow full Website access if they do not design the touch interface in full HD.

Case study

A while ago I compared Gmail applications between 4, 7, and 10 inch screens. The 10 inch was done in nice 2 column layout with small fonts which made easy to view my email content. The 7 inch tablet app was nothing, but blown up small phone app, with huge fonts as if Google was assuming all Galaxy Tab 7 users were vision impaired. The small screen app has wasted a lot of screen with big buttons in editing mode, buttons that should have been in menu with better iconography. Even the Blogger app I am using now has several buttons that take screen space. I assume that Blogger app is used very frequently, so it should be designed for frequent users.

Since then Google has improved their UX dramatically.

Making symbolic link named ".m2" to Gradle JAR repo

After many years of using Maven, I am used to that my JARs are in ~/.m2/ directory, so I created myself a link as follows:

$ ln -s ~/.gradle/caches/modules-2/files-2.1 ~/.m2 

$ ls -alt ~/.m2/

total 0

drwxr-xr-x  65 ukilucas  staff  2210 Apr 28 23:18 .

drwxr-xr-x   3 ukilucas  staff   102 Apr 28 23:18 junit

drwxr-xr-x   4 ukilucas  staff   136 Apr 28 23:18 org.hamcrest

drwxr-xr-x   3 ukilucas  staff   102 Apr 28 14:15 com.android.tools.external.lombok

drwxr-xr-x   3 ukilucas  staff   102 Apr 28 14:15 org.abego.treelayout

drwxr-xr-x   3 ukilucas  staff   102 Apr 28 14:15 com.intellij

..

Running Groovy with multiple command line parameters

Source for ArgumentsTest.groovy

#!/bin/bash
//usr/bin/env groovy  -cp extra.jar:spring.jar:etc.jar -d -Dlog4j.configuration=file:/etc/myapp/log4j.xml "$0" $@; exit $?


def params = ""
args.each() {
    if (it) {
        params +=   it + "! "
    }
}

println "Hello World " + params

Execute permission

$ sudo chmod +x ArgumentsTest.groovy
Password:

Output

$ ./ArgumentsTest.groovy Uki Natalia Zoe
Hello World Uki! Natalia! Zoe!

Mac Groovy installation with brew

$ brew install groovy

==> Downloading https://dl.bintray.com/groovy/maven/apache-groovy-binary-2.4.6.zip

######################################################################## 100.0%

==> Caveats

You should set GROOVY_HOME:

  export GROOVY_HOME=/usr/local/opt/groovy/libexec

==> Summary

🍺  /usr/local/Cellar/groovy/2.4.6: 63 files, 27.6M, built in 6 seconds

$ groovy -v

Groovy Version: 2.4.6 JVM: 1.8.0_77 Vendor: Oracle Corporation OS: Mac OS X

$ 

If you like this post, please donate 2 cents ($0.02 litterally) to show token of appreciation and encourage me to write more:

Donate Bitcoins