adam bien's blog

Lukasz, please introduce yourself.

Hello, I am Łukasz Matuszczak, a software architect at ATREM S.A, company from Poznań, Poland. I have a 12 years experience in Java related technologies. It’s dangerous role to be an architect, because it’s easy to fall into the trap of enforcing sophisticated solutions that just don’t pay off, for example many abstraction layers only for the sake of layers, or patterns that you learned many years ago, even when they don’t fit new technologies. The funny cartoon about it: A good architect leaves a footprint. You tried to convince us during airhacks that developers with deep domain knowledge are much more desirable than architects. Fortunately I know a lot about our customers’ business area, I’m still coding a lot, and I have more fun with that than with the role of architect.

ATREM is not an IT company - we deliver many engineering services mainly to Polish energy industry. We have great experience in industrial automation, telecommunication, corrossion protection. We were always strongly connected with natural gas sector. But nowadays we do many other things for example we were involved in expanding sewage treatments, constructing wind farms, building power and safety installations for Euro 2012 stadiums, or making IT and automation systems for airports. I was fortunate to be at the birth of IT department in ATREM, with only a few friends from Poznań University of Technology including Jarek Drobny who also attended airhacks. It's nice to see how it evolved to more than 50 developers, implementation consultants, administrators...

You attended airhacks.com in MUC. Have you learnt anything useful? Did you had the time to chat with the other attendees?

Well, your workshops totally blew me away. What surprised me most, during airhacks you could not only get a lot of technical hacks, but also many valuable comments on the organization of teams, how to talk with customers and even recruitment tips.

It was also very interesting to hear the problems and questions of the other attendees. I was surprised that many of them migrated or considered to migrate from Spring to Java EE standards.

For me especially valuable were examples connected with constructing REST interface - that you should always think in terms of resources with CRUD operations even if at the beginning it doesn't fit. For example an on-site equipment installation, or search query can be quite good REST resources.

We have a strong OSGi knowledge and another interesting thing was your ideas on how to make pluggable application using Java EE. For example our clients have our product installed on-premises, and they need to collect data from devices using some unusual protocol. I think deploying another "microservice-like" WAR with webhooks in main application is an interesting alternative to OSGi plugins deployed "inside" the application.

The workshops were very intensive - you should do more breaks :). I think most attendees were at the same time delighted and exhausted. We had very little time to chat with each other.

What are you building?

Since 2004 we were building basically two products:

• Telexus - initially mainly SCADA software, but it evolved into a data collector from different devices (for example gas flow meters) using different media (TCP, UDP, SMSes, Serial ports, analog modems), and then analysis of this data, with extensive reporting, visualization, aggregation, data validation, event notification, forecasting and integration with billing systems
• Ewista - asset management system - initially created for gas transmission and distribution networks, but it has become so general and configurable that it can be used in different areas. It has also an expanded GIS module based on ESRI solutions, but we also integrate Ewista with cheaper products. We can plan and record many events related to the infrastructure for example inspections, failures.

Soon it became clear that our customers often describe use cases that need some features from both products. So we were integrating both applications more and more, and since 2012 we're building one integrated platform called Entia. The detail features are listed here.

The old version of Entia (without Java EE) is very pluggable and configurable, based on OSGi plugins, and our own components called "bricks". We have strong implementation team who adapt the application to the specific needs of different customers. Product development is driven by projects for different customers but we almost always try to transform the requirements to something more general. This generalization has pros and cons. It's nice because our clients often have unclear/fuzzy needs, and we can often extract some features that we can sell to other clients. On the other hand the system has so many parameters that it's hard to maintain, configure and understand.

In 2014 we decided that after 10 years of product development it's time to make a little revolution - drop unused features and ideas, refresh technologies, implement fully many things we have learned during the years about clean code, testing, code reviews, organization of teams. Another thing was that our solutions were mainly desktop-based (using Swing and JIDE), and we wanted to focus on Web and mobile interfaces. We also rethinked our highly configurable way of making very general platform and turned into thinking about more dedicated solutions which can better answer user needs. We still want to base our solutions on some common core, but we are much more cautious in making everything very customizable and extensible. After all it's not so easy to build a big SAP-like platform :).

At this point we considered a variety of different technologies, but we finally decided to use Java 8 + Java EE 7.

You migrated your code base to Java EE. How much code could you delete?

Well, we’re still migrating. We have over 2 million lines of code in the old product so it's not easy to migrate them in one day. For now we often make some hybrids, where some features are provided by the legacy product, and some by the new Java EE application. We communicate through JMS, Websockets, REST and SOAP depending on the needs, and capabilities of our old system. For example we provide a new gas transport forecasts module based on the data collected by the old system, or we interpret SMS messages from devices in new system pushing measurement data to the old system with JMS for further processing. For the end user it's hard to notice that there are two cooperating applications running in different technologies - sometimes the only difference is the new Web interface.

We can remove a huge amount of code due to the fact, that our old system is built on the top of framework we developed ourselves. For example in the past we created our own solutions for dependency injection, configuration management, naming, remote event bus with lease renewal, security, asynchronous RMI calls, thread pooling for handling remote requests. We even created simplistic ORM to read many time series data when Hibernate was to slow.

It was fun to develop our own framework, but it was hard to maintain it, we had to manage many external dependencies, and it was simply pointless from the business point of view. Moving to the Java EE allowed us to wipe most of the framework code. In places when Java EE standards are not enough, we still don't want to code the solution on our own, so for example we use PicketLink/Keycloak

After airhacks.com I think we will delete even more, because we fell few times into the trap of making unnecessary interfaces for all our EJBs or wrapping EntityManager in DAO-like class to which we delegated all calls from service with almost the same API as DAO. It was nonsense.

You mentioned OSGi as a main technology in your legacy product. Can you live without it in Java EE?

I have mixed feelings about OSGi. OSGi in our legacy product was a big improvement when we introduced it in 2009 - we could work on the same product by many teams, even with an external remote team. Supported by Spring Dynamic Modules we could make system very extensible, based on Whiteboard pattern in many places. We haven't used many features of OSGi however, as dynamic replacement of binaries.

On the other hand I don't miss all these NoClassDefFoundErrors, Eclipse Buddy-Policies, dynamic imports, fragments, reexports and other hacks. And of course many libraries including JDK (e.g. RMI) are not prepared for this classloader hell (maybe Java 9 and Project Jigsaw will solve some of OSGi drawbacks). We had also many performance problems - for example try to serialize something using XMLEncoder when the bean classes are visible only by buddy policy. We even tried to integrate OSGi in our Java EE applications using JBoss OSGi but it is not very popular solution, and in fact nobody knows how to mix such concepts as EJBs and OSGi services.

For now, we use some CDI Instance injection for pluggable behavior, for example to extend available options in application menu, and we abandoned OSGi completely. For more dynamic extensions to application we will use microservices style. It is anyway a better solution because when the services run in different JVMs they are more robust. For example generating alarms in SCADA should not be disrupted by the generation of a large report for billing system as it happened sometimes in our old system.

Is Java EE productive?

I would say that at the beginning we even felt a little uncomfortable, that we could start our work instantly and move fast, without having read tons of documentation and specifications. We especially liked that everything was based on annotations, and even these were optional in many cases, because the defaults are very reasonable. Of course we still have some doubts for example in the proper use of CDI, we have to rethink many our previous patterns. But it’s great that we can install application server, add one dependency to our WAR, and then we can focus on business logic. Many of our previous concerns are simply solved by the server -- dependency injection, transactions, concurrency, thread pools, timers, ORM, web server, monitoring

What about performance, is Java EE fast enough?

We haven't had many problems yet. The startup time of application servers is very short. The nice thing is that we have much more control over thread usage (managed thread pools) and memory usage (we can tune number of beans in object pools).

When we were making our own framework in our legacy product it wasn’t so easy to wrap everything in proxies for example to start all the business components lazily. The consequence was that for simplicity we initialized almost everything at startup. Sometimes the startup process lasted about half an hour. Also all objects - for example related with every device or variable had to be loaded into memory during the initialization of server. At one time before some optimizations we were dangerously close to limit 32GB heap size, above which compressed references cannot be used. Therefore many Java EE solutions especially related to EJBs, but also CDI (with its Instance) are a big improvement for us.

Another important thing is that we have access to many built-in metrics, so we can measure the performance in many parts of application without additional effort. As I recall how many MBeans and tools we added in our legacy product to measure performance, I am really grateful to solutions provided by application servers. In Wildfly however we lack convenient access to metrics in html format, as it is possible in Glassfish Monitoring and Management.

Java EE introduces many indirections, proxies, interceptors, aspects. It seems however that the performance cost of all this proxying around EJBs and CDI beans is surprisingly low. We never found that this was a bottleneck in our application. By the way, in my opinion the only cost of all these indirections inherent to Java EE are … these humongous stacktraces :). They are very hard to read especially for beginners.

How many developers are working in your project?

We have about 20 core developers employed by ATREM and we also hire external teams and build new teams at universities in Poznań. We also cooperate with some remote workers.

How big is your WAR?

It’s about 25MB mostly because it contains backend and frontend made with Vaadin.

Are there many runtime dependencies in your pom.xml?

We try to reduce them as much as possible. Our dependencies are Vaadin, Gson (we had some problem with serialization of byte arrays. In other places we use javax.json), Guava. We also use SLF4J, PicketLink provided by Wildfly. We’re fighting with our old habits and try to use new possibilities of Java 8, for example streams together with java.nio instead of commons-io. Probably in the future we will add some specialized libraries – commercial and open source, for example ArcObjects for GIS, SNMP4J, SMSLib, SerialIO for device communication, JasperReports for reporting, Batik for SVG manipulation, POI for Excel files manipulation.

How important is Java EE to you? Is your code dependent on any proprietary application server features?

Standard APIs are very important to us, because we want to minimize our efforts to put our technology stack together. I know what I'm talking about because our legacy product had 830 bundles occupying 440 MB, and 300MB are external dependencies. This is a real mess to manage all the versions and conflicts.

In new application we run everything on Wildfly, but we tried to run part of our app on Glassfish in the past, and we had only minor, easy to solve problems. I checked now and when I search word “jboss” in our codebase I can see it only in arquillian dependency and we have also jboss-deployment-structure.xml with dependency to org.slf4j jboss module. So dependency to javax:javaee-api:7.0 rules !!! :)

Which servers, tools and IDEs are you using?

We deploy on Wildfly. However we also consider Payara for future projects considering its support options.

Our favorite IDE is Eclipse Mars with JBoss Tools. We build, test and deploy to our staging servers using Atlassian Bamboo. Jobs are based on Maven. We monitor code quality with Sonar Qube. We still use Jenkins a little, and it’s somewhat strange that we use Gradle (newer build tool) for our older legacy product - that’s because the building process of the product is so complicated that it was easier for us to control it with groovy script. For Java EE, I would recommend Maven.

We used to organize our development process with IBM Rational Team Concert, but recently we migrated everything to Atlassian stack - JIRA Agile, Confluence, BitBucket, Bamboo.

Our developers work on Windows, but we deploy our products to Windows, Linux and AIX servers. We didn’t try however to install Java EE 7 server on AIX, only our legacy product, but I hope we will manage to deploy our new application on AIX without obstacles.

Is your company hiring Java EE developers?

Yes, definitely we are looking for new employees but also experienced independent teams. As for individual candidates, deep Java EE knowledge is not required, as we think you don’t need to learn it for long before writing your first line of code. The more important thing is if you are passionate about IT, your logical thinking skills, creative ideas, and so on.

You can read about our team on:

• IT @ ATREM (in Polish)
• It systems in ATREM (in English)

Please feel free to contact me at l.matuszczak@atrem.pl if you have any questions.

Lukasz, thank you for the interview!