Monday, February 12, 2024

Rebooting Shell Scripting and this Blog

 This blog is now the official blog of Bashpile: The Bash Transpiler!


A transpiler is a compiler that produces a different language.  This means that you can write in a modern, clean syntax and still have code that runs on any Linux or Unix platform that supports Bash scripts!


More details are at the main page: https://github.com/michael-amiethyst/homebrew-bashpile

Thursday, December 7, 2017

Studying and passing the AWS Developer Associate Exam

Amazon Web Services has been around for over a decade now and has offerings that continue to grow. They also have over a 35% market share.  Lastly the Developer Associate exam seems to come up on "best programmer certs" type of posts (e.g. here).

So if you're convinced that it's a good idea to get a cert, how do you do it?  What should you study?  No one wants to waste a day and over a hundred dollars to take a test and then fail it.  This post will go over some of the best resources to use to study.

When I studied for the exam and passed it, these were the most useful resources, in order:

  1. ACloudGuru has a fantastic series of videos that covers everything on the test is broad strokes, but is detailed enough to give you a real advantage on the test.  The training isn't free but it is quite reasonable and you literally get hours of content.
  2. The AWS Documentation will have the details you need, but usually in a format geared for your day-to-day developer work instead of in a Q/A format.
  3. On-the-job experience is great too.  However experience will usually have you do a deep-dive on one or two technologies (e.g. migrate something to EC2 and S3) instead of a broad overview of the entire platform.  This overview is what the test is going to be about.
  4. The AWS FAQs have some information and are in a good Q/A format, but rarely mention anything on the test.
  5. The AWS Whitepapers are recommended on Amazon's page of test tips, but was not really useful at all.  You would probably be better prepared by reading the Wikipedia page's list of AWS services.
The test itself was in a bit of an odd location.  When I took it the facility was primarily used for aviation testing.  Needless to say, when you're waiting in line with people needing pilot's licenses you wonder if you got the address wrong.  So don't worry if you get directed to an airport, or near one.  Also, at my location only two other people were taking the test and I was expecting more of a classroom like situation (e.g. 15+ people taking the test at the same time).  Overall, the experience wasn't that bad.  The questions mostly matched up with the preparation questions from ACloudGuru but not entirely.

Lastly, some companies will pay for your tests and training.  If you're really lucky you'll be with a company that needs a certain headcount of certified developers in order to qualify to be some kind of "premium partner" with AWS.

All in all, it required a fair chunk of time to study but you end up with a really good overview of the (now large) AWS ecosystem of services as well as a boost to your resume.

Thursday, July 28, 2016

Writing Effective Exception Descriptions

Introduction

Writing exception descriptions are not really taught in programming classes but are vitally important in the field.  Namely if you need to refine your descriptions to find a problem it can lead to multiple pushes to a production environment before you actually fix the problem.  Depending on the issue this could waste valuable time.

Between reading about how to write good exception descriptions from various companies and my personal experience I've accumulated the following tips to share.  Note that most of the time exception text bubbles into your log files so some references to log files will be mentioned as well.

Be Layer-Centric

If you catch an exception and re-throw, describe the problem in the terms of the current (throwing) layer.  As an example, if you are writing TCP/IP library code and catch an IP exception, state your TCP problems.  If you are (ab)using TCP for Inter-Process Communication (IPC), state the process you are trying to reach.  I have mistakenly thought there was a networking error when a program couldn't connect to localhost before.

Not a Singleton?  Include your object ID

Imagine a case where there is a complex object with invariants.   With two instances of the class logging at the same it could easily appear to violate it's invariants or demonstrate impossible behavior.  Imagine your TCP class logging that it closed the connection and then that it read data successfully - this could be very confusing.  With object IDs you would log that client 1 closed the connection and client 2 read successfully - this makes much more sense.

Multi-threaded?  Include your Thread ID

Some times you need to trace the execution flow in a particular thread.  If you do not include your Thread IDs this is an impossible task to do via the logs.

Don't Trust Library Exceptions

Frequently exceptions generated by library, module or component code is not very descriptive.  Continuing with the networking examples I've had many HTTP exceptions with no more description than "Bad Request".  Note the lack of IP address, host name, or port.  Given that exceptions turn into logs that then can go into various applications (e.g. Splunk) I think the descriptions are generic due to security concerns.

Include Some Variables

Try to include one or two other things to help you track down a problem, knowing WHAT from following the recommendations above won't necessarily help you figure out WHY.

Make it easy

When you write your subclass consider including a String.format style constructor.  This would let you avoid having to do a String.format call on every exception you need to throw.  Alternatively you could make an ExecptionDescription builder that includes the object ID and Thread ID automatically.

Conclusion

In conclusion if you plan ahead and write descriptive exception messages that let you track the instance, the thread and some internal state you will be able to diagnose (possibly critical) issues much faster.

Tuesday, July 12, 2016

Overview of JSON Libraries and How to Choose a Library

Welcome

It's good to be back.  The blog had an expected hiatus while I moved to a new city, also around Seattle.

Intro

Many of my posts are about overcoming unexpected difficulties when using some software library or module or another.  I decided to take a step back and try to select a quality module in the first place.  This is my analysis of the JSON libraries I was able to find and can count as a how-to to select a library in general.

The short result is that if a Library is in Maven, you can look at Maven Central to see how popular a library is and how frequently it is updated.  Then you can use one of the most used and updated libraries out there.  You can also consult StackOverflow.  Here are some examples.

The List

  • JSR 353 - We have javax.json-api and org.glassfish/javax.json.  Out of these two the
    This could be you!  Image from http://blog.smartbear.com

    org.glassfish one has more users and more updates (version 1.0.4 vs 1.0), so it is the winner so far.
  • Fast-XML Json Parsing / Jackson has many more updates (version 2.8), is far more used and the updates are fresher.  That makes it the winner so far.
  • JSON-Simple is comparable to the JSR 353 libraries.  If not for the Jackson library you could choose based on how well prior Google libraries have worked vs Oracle libraries.
  • org.json falls between Jackson and the JSR 353 libraries with about 700 users and a release in Feb of 2016.
  • Google GSON has about 2000 users and was updated last month.  This puts it on par with Jackson.
  • quick-json has 0 users!  This would make it almost unqualified even if there were no alternatives.
  • JsonPath - Lastly, JsonPath has a couple of hundred users.

Conclusion

Now, you may be skeptical of the Wisdom of the Crowd (e.g. appeal to the people or groupthink) but more users means that there is a higher chance someone found the same problems that you have.  Even in this day and age Googling an error string can still produce 0 results.
Based off of this my project is using Jackson and hasn't had any problems with it for months.

Next Steps

Once you have your JSON library picked out, how do you manipulate it?  I've had good luck using  http://www.jsonschema2pojo.org/, although in the options you have have it add annotations... which makes it a non-POJO technically...

Wednesday, May 18, 2016

Java Multithreading in Practice: Part 3

Introduction - The Saga Continues

In Part 1 we discussed presenting a cleaner interface to Java 6's ExecutorService and ScheduledExecutorService.  In Part 2 we discussed ensuring that the system doesn't eat / ignore exceptions in threads.  In Part 3 we will go over reusing the thread pool... because that's the point of a thread pool.

Background

We don't read ALL of the documentation on a given class, we read parts and whatever the IDE gives
Thread Pool, not to be confused with Deadpool
us on a mouse-over.  This habit led me to reading the method-level documentation of ExecutorService but not the class-level documentation.  This is important later.

Hypothesis

The ExecutorService will have an equivalent of Thread.join(), and then be re-usable since that is the whole point of a thread pool.  A quick skim of shutdown() says to use awaitTermination().  Upon reading the full docs of awaitTermination() it appears that this is what I am looking for.

Results

Running threads after a while mysteriously fail with a ThreadPool size of 0.  I never set the size to 0, why would it do that?  After struggling with this and creating more unit tests to narrow the problem down for a full day (full day) I find that awaitTermination() terminates the WHOLE POOL instead of terminating the currently running threads.  What's the point of having a terminated pool?  Isn't that what close() is for?  Does this have a close()?  Why not?  Maybe because it is in the java.io package instead of java.concurrent?  Again, the entire point of having a thread pool is to reuse the threads!

Conclusion

In addition to needing to keep a List<Callable<Void>> I need to keep a List<Future<Void>> and loop through all of the Futures to cancel() each one.  WHAT?  Again, nothing in the documentation of awaitTermination() says that it terminates the thread pool itself.  That is buried in the class level documentation.  I end up adding a cancelThreads() method to our custom ThreadPool object leaving us with a final API like the following:
public class ThreadPool {

    ThreadPool(int size) {...}

    void add(Runnable) {...}

    void add(Callable<Void>){...}

    void clear(){...}

    boolean isEmpty(){...}

    void runSynchronously(){...}

    void runAsynchronously(){...}

    Callable&ltVoid> toSafeCallable(Callable<Void&gt){...}

    Runnable toSafeRunnable(Runnable){...}

    void cancelThreads(){...}

    void close(){...}

Tuesday, May 10, 2016

Technology Gone Terribly... Right

I'm not an Amazon fan-boy.  I worked there as a contractor  for 8 months and still didn't start using Amazon in my personal life.  I may be forced to subscribe to Amazon Prime to get Dr. Who, but that will be very begrudgingly.

However, Amazon Web Services is a true gem in an otherwise frustrating field.  I worked with AWS for a 3 month stint about two years ago and had to use some AWS services at my current job (p.s. my Google+ job history is outdated) and getting both the Simple Notification Service and the Simple Storage Service only took me a day.  No real hassles, surprises or gotchas.

I usually write about how to deal with things going wrong, but for once I'm writing about something going right.

It isn't nearly as interesting, is it?

Tuesday, May 3, 2016

You're Killing Me Spring: "Singleton" Scope

Introduction

I generally try to avoid traditional Gang of Four Design Pattern Singletons.  In Java they are only guaranteed to be singular per running Java Virtual Machine, don't scale to a clustered environment, can be serialized / deserialized to create a duplicate even in the same JVM and are generally the OOP version of global variables.
However, in a given project I had an exception to this rule.  I had an object that was using a single TCP port going into an environment where only one instance would be deployed per machine so I thought to myself that I would use the Singleton pattern.

Background

Created at imgflip, original image copyright Dos Equis.
I have had problems with Spring before.  Specifically I was working with Spring Web Flow 3 years ago and had some configuration (including setters) in my beans.xml file, in my Spring Web Flow xml file I imported the (Singleton) beans defined in the beans.xml file.  However, instead of importing the bean it created a new instance of it, and on top of that it didn't call any of the initialization configuration in the beans.xml file leading to null pointer exceptions.  It took me three weeks to track this down.  I was working for myself (e.g. that time was on my own dime) and I ragequit Spring in favor of Google Guice.  For this project I was forced into using Spring so I had the following Hypothesis.

Hypothesis

"Verify that Spring is treating Singletons correctly because it's been unreliable, surprising and buggy in the past."  In code-form it was a standard Spring @Service in the default Singleton scope with a check in the constructor (see Further Reading for other reasons to do this) to blow up if the constructor is called twice.  In addition, Spring makes you have a public constructor even for your Singletons so blowing up can avoid programmer error of someone later trying to just call new ExampleSingleton.
@Service
// implied Spring Singleton Scope
public class ExampleSingleton {

    // should only be called once by Spring
    public ExampleSingleton() 
    {
        if (instance != null) {
            throw new IllegalStateException("Singleton constructor called twice!");
        }
    }
}    

Results

As you may have guessed from the image, Spring was attempting to create the "Singleton" repeatedly and blowing up with the IllegalStateException.  From doing some research I found that @ComponentScan was behaving strangely.  It turns out that if you have multiple Spring @Configuration classes and if they overlap on a package (e.g. @ComponentScan(basePackages = {"com.yourcompany"}) and another with @ComponentScan(basePackages = {"com.yourcompany.utils"}), Spring will (re)create all of the Singletons in the overlap (in this example com.yourcompany.utils) twice.  I found this surprising and very strange.  In addition, the same link mentions that Spring only promises that a Singleton will happen once per ApplicationContext, and Servlets usually have more than one.

Conclusion

I concluded that you can't trust Spring to honor the @Singleton contract and to manage my instances myself.  After doing more research I found that in Java the best way is to have a SingletonFactory.  I ended up having a base class called SpringSingleton, which has the documentation on why all of this is necessary and a protected constructor that keeps track of which sub-classes have been instantiated with a Set<SpringSingleton> that is accessed in a synchronized block.  In addition the @Configuration classes have an @Autowired SingletonFactory and @Bean definitions that get the beans from the SingletonFactory.  All in all it still isn't foolproof for Serializable Singletons (see below) but does a great job otherwise.  There is more complexity (4 classes for the simple case) but new Singletons end up being easy to implement and work correctly.  I would post the code but I made it at work so the company has copyright.

Too bad I couldn't just have @Component on a class with the default scope and have it work right.

Further Reading

You can abuse an enum in Java to ensure a "serializable" Singleton can't be easily duplicated but that has some drawbacks.  Given that you can invoke even a private constructor via the Java reflections API you will still want to verify that your Singletons are only being called once (or at least post a warning in the logs).

PS

There was an issue with the site CSS where black lines would appear over an image. This has been fixed.