153 items found: Search results for "change" in all categories x
September 21, 2015 | DevOps
The drive for change comes from, everywhere. Within IT departments, developers are asking for more control, and want faster response times than can be provided by traditional gatekeeper IT services. Management hears about new technologies allowing more rapid deployments, better ROI, reduction in costs, increased efficiency, and increased scalability, and they want it for their customers. On top of that, systems age, technologies lose their lustre, and technical debt builds up, while it simultaneously gets harder to recruit people from a shrinking talent pool, all of which creates a case for change from within the technology itself.
Learn to create your first Kubernetes operator by checking out our Senior Consultant Michal Tusnio’s latest blog, “Kubernetes Operators – Whys, Hows and Whats” where he takes you on a journey from zero to operator.
In this blog we share the latest developments in our efforts to create a more sustainable business as part of The 2023 Mayor’s Business Climate Challenge.
Check out John Sharpe and Will May’s latest blog where they give suggestions for Terraform Provider authors who are thinking about upgrading from SDKv2 to Framework
Check out Matthew Revell-Gordon’s latest blog as he explores building a local Kubernetes test cluster to better mimic cloud-based deployments, using Colima, Kind, and MetalLB.
Check out Mateus Pimenta’s TL;DR video to learn how federated computational governance could be implemented using Open Policy Agent (OPA) and policy-as-code to support a successful Data Mesh architecture.
Check out Part 2 of Ebru Cucen and Fahran Wallace’s blog series, in which they discuss their experience ingesting 400 million nodes and a billion relationships into Neo4j and what they discovered along the way.
As data becomes ubiquitous and deeply interconnected, tracing where who or which system that data comes from – its lineage – will create bigger problems and opportunities for us on the horizon. Watch the recording of James Bowkett talk from NODES 2022 – Neo4j Online Developer Education Summit 202 on ‘Tracing Your Data’s DNA.’
OpenCredo is thrilled to be a part of the 2022/2023 Mayor’s Business Climate Challenge (BCC). Read this blog to learn more about our initiatives to lower carbon emissions as part of the voluntary energy efficiency programme!
Watch Simon Copsey’s talk from the Lean-Agile Delivery & Coaching Network and Digital Transformation Meetup on “Seeing Clearly in Complexity” where he explores the Current Reality Tree tool.
Read Matt Farrow’s blog as he explores the potential for using Open Policy Agent to filter and mask data being sent to and read from Apache Kafka.
Many businesses advocate for efficiency, but this is not always the right goal.
In part one of this article, we explored how product teams can balance two important considerations – efficiency and effectiveness.
In this second part we will introduce the – often unexpected – implications of turning to technology to bring about efficiency and wider change, and the deeper considerations that must be addressed first.
Many businesses advocate for efficiency, but this is not always the right goal. Part one of this article explores how product teams can balance two important considerations – efficiency and effectiveness. Part two builds on this idea, uncovering the non-obvious implications of using technology to bring about efficiency and wider change.
As data becomes ubiquitous and deeply interconnected, tracing where who or which system that data comes from – its lineage – will create bigger problems and opportunities for us on the horizon. Watch the recording of James Bowkett’s talk from Devoxx UK on ‘Tracing Your Data’s DNA’
May 24, 2022 | Software Consultancy
Watch Simon Copsey’s talk from the Agile Ex Meetup on “Coaching at the Organisational Level” where he explores some of the common downward spirals in IT and the larger organisation, as well as the tools that coaches and individuals can use to surface and discuss them in a way that allows for systemic change, even without formal authority.
Serverless functions are easy to install and upload, but we can’t ignore the basics. This article looks at different strategies related to testing serverless functions.
There are two camps of Graph database, one side is RDF, where they are strict with their format, and somewhat limited for their extensibility. The other side is LPG, where they can define labels to the relationships. With its recent extension, RDF now allows users to add properties, thus becoming RDF*. In this blog, Ebru explores the structural and performance differences between LPG and RDF*.
Kubernetes’ second release in 2021, version 1.22, has been out for a little while now and with 1.23 on its way, we thought we’d take a look back. Kubernetes 1.22 was a highly comprehensive release with 53 enhancements in all three graduation levels: 13 features have graduated to stable, 24 enhancements reached beta status, and 16 new features have been accepted into the alpha stage.
The latest version has some noteworthy security features such as running Kubelet without root access, pod security policies, and seccomp. There are also a couple of deprecated and removed APIs. In this blog, we’ll discuss the significant changes in v1.22, as well as how to handle the removed APIs.
November 4, 2021 | Kubernetes
We always read that ‘security is everyone’s responsibility’. For any organisation, big or small, security should always be the primary concern—not a mere afterthought. In terms of Kubernetes, securing a cluster is challenging because it has so many moving parts and, apart from securing our Kubernetes environment, we also want to control what an end-user can do in our cluster.
To achieve these goals, we can start with the built-in features provided by Kubernetes like Role-Based Access Control (RBAC), Network Policies, Secrets Management, and Pod Security Policies (PSP). But we know these features are not enough. For example, we may want specific policies like ‘all pods must have specific labels’. And even if we have the policies in place, the next big question is how to enforce them on our Kubernetes cluster in an easy and repeatable manner.
In this blog post, we’ll address this challenge and other questions pertaining to OPA and how it can integrate into Kubernetes.
Message and event-driven systems provide an array of benefits to organisations of all shapes and sizes. At their core, they help decouple producers and consumers so that each can work at their own pace without having to wait for the other – asynchronous processing at its best.
In fact, such systems enable a whole range of messaging patterns, offering varying levels of guarantees surrounding the processing and consumption options for clients. Take for example the publish/subscribe pattern, which enables one message to be broadcast and consumed by multiple consumers; or the competing consumer pattern, which enables a message to be processed once but with multiple concurrent consumers vying for the honour—essentially providing a way to distribute the load. The manner in which these patterns are actually realised however, depends a lot on the technology used, as each has its own approach and unique tradeoffs.
In this article we will explore how this all applies to RabbitMQ and Apache Kafka, and how these two technologies differ, specifically from a message consumer’s perspective.
Developing a Terraform provider is a great thing for a company to do as it allows customers to quickly integrate a product with their existing systems with very little friction. During development, occasionally there might be bugs and issues to fix, and it can be quite difficult to work out what is causing them. In this post, I outline how you can attach a debugger such as Delve to a Terraform provider to save time when solving these issues.
Multi-cloud is rapidly becoming the cloud strategy of choice for enterprises looking to modernise their applications.
And the reason is simple – it gives them much more flexibility to host their workloads and data where it suits them best.
In this post, we focus on Google’s application modernisation solution Google Anthos and the role it can play in your cloud transformation strategy.
While working with a client recently, we experienced some issues when attempting to make use of NLB external load balancer services when using AWS EKS. I wanted to investigate whether these issues had been fixed in the upstream GitHub Kubernetes repository, or if I could fix it myself, contributing back to the community in the process.
October 15, 2020
Continuous Verification is a term that is starting to pop up from time-to-time… but what does it mean? Well… according to Nora Jones and Casey Rosenthal, authors of O’Reilly’s Chaos Engineering books,
“Continuous verification (CV) is a discipline of proactive experimentation, implemented as tooling that verifies system behaviors. This stands in contrast to prior common practices in software quality assurance, which favor reactive testing, implemented as methodologies that validate known properties of software. This isn’t to say that prior common practices are invalid or should be deprecated. Alerting, testing, code reviews, monitoring, SRE practices, and the like—these are all great practices and should be encouraged”
Over the course of this post, we will unpack this statement: to understand what is behind it and what it might mean for your development process.
With the upcoming Cassandra 4.0 release, there is a lot to look forward to. Most excitingly, and following a refreshing realignment of the Open Source community around Cassandra, the next release promises to focus on fundamentals: stability, repair, observability, performance and scaling.
We must set this against the fact that Cassandra ranks pretty highly in the Stack Overflow most dreaded databases list and the reality that Cassandra is expensive to configure, operate and maintain. Finding people who have the prerequisite skills to do so is challenging.
At the time of this post, the UK is making steps to exit from an unprecedented lockdown measures for the Coronavirus. Much of the UK workforce are still making efforts to work-from-home with mainly key workers operating – at risk – in public. Many industries have shut down completely. Consequently, many businesses are reflecting on what happens next and how do we better mitigate future pandemic events?
April 2, 2020 | Machine Learning
Recent years have seen many companies consolidate all their data into a data lake/warehouse of some sort. Once it’s all consolidated, what next?
Many companies consolidate data with a field of dreams mindset – “build it and they will come”, however a comprehensive data strategy is needed if the ultimate goals of an organisation are to be realised: monetisation through Machine Learning and AI is an oft-cited goal. Unfortunately, before one rushes into the enticing world of machine learning, one should lay more mundane foundations. Indeed, in data science, estimates vary between 50% to 80% of the time taken is devoted to so-called data-wrangling. Further, Google estimates ML projects produce 5% ML code and 95% “glue code”. If this is the reality we face, what foundations are required before one can dive headlong into ML?
March 20, 2020
Traditionally, Usability and Security have been set in opposition to each other: with tight security, we end up with painful user experience. In this blog, Guy focuses on financial services as an exemplar of how we can introduce usability into a vertical with challenging security and compliance requirements.
November 13, 2019 | Software Consultancy
Pioneering and pushing technology boundaries – pretty much a given nowadays for the software-driven startup. Here are some insights we’ve observed working with a number of venture capital (VC) companies who have managed to navigate the choppy waters and successfully grow their business including winning further investment along the way.
With our deep hands-on technical expertise and pragmatic focus, OpenCredo has become a natural software acceleration partner for VC funded organisations who are looking to deliver tangible value as effectively as possible. We’ve been brought in to work alongside these innovators at various stages of their journey. As such we’ve gained an appreciation for and acquired, first-hand insight into some of the pressures and challenges faced. From getting and securing that next round of funding, to grappling with the technical decisions and challenges inherent in sensibly evolving offerings to accommodate future growth and scaling.
This blog is written exclusively by the OpenCredo team. We do not accept external contributions.
The Cloud Native landscape can be bewildering, and not only for newcomers. As a traveller on the Cloud-native journey, I have sometimes been overwhelmed by the number of products and projects. This is why I took hold of the opportunity to go to Day 3 of the Cloud Native London Conference last month hosted by Skills Matter.
Here are my top highlights from Day 3 of the CloudNative London 2019.
Terraform 0.12 in recent years has emerged as the de facto standard with regards to defining and managing cloud infrastructure. It is one of four primary tools offered by HashiCorp, (Terraform, Vault, Consul and Nomad) and underpins the workflows that make up their Cloud Operating Model.
Since its first release in 2014, the wider Terraform community has embraced frequent releases and this past year has been no exception. HashiCorp announced the release of Terraform 0.12 in May 2019 and as of writing this post the official release is 0.12.9.
As a technology leader, you’ll be aware that competitive pressures and shifting business requirements are driving changes in the technical architectures of many organisations. This means you need a new strategic approach based on the ability to continually evolve elements of your systems and architectures.
Here, in our little nook of the internet, we usually write about our experiences of emerging technologies, the tricky coding problems we’ve solved and how we have enhanced our clients’ businesses. We do this because we are very proud of this work. It truly matters.
Today is a little different. Today I’d like to share something a little more personal. Something that brings loneliness, kindness and technology together in an oxytocin-generating, slightly awkward embrace. Because hugs matter too.
Writing your own Kafka source connectors with Kafka Connect. In this blog, Rufus takes you on a code walk, through the Gold Verified Venafi Connector while pointing out the common pitfalls
Creating and managing a Public Key Infrastructure (PKI) could be a very straightforward task if you use appropriate tools. In this blog post, I’ll cover the steps to easily set up a PKI with Vault from HashiCorp, and use it to secure a Kafka Cluster.
August 29, 2018 | Software Consultancy
My last year or so here at OpenCredo has involved a very well-supported first foray into tech consultancy. Different engagements pose both unique, as well as familiar challenges for me as a consultant, all of which played a part in shaping and moulding the way I understand and approach problems. This blog is a brief collation of wisdom that’s helped me the most during this adventure; gained by learning the hard way, as well as that acquired from mentoring and colleagues who have gone before. The shared wisdom has made me a much more effective consultant, and kept me sane in the process, for which I’m very thankful.
July 31, 2018 | Machine Learning
Machine Learning, alongside a mature Data Science, will help to bring IT and business closer together. By leveraging data for actionable insights, IT will increasingly drive business value. Agile and DevOps practices enable the continuous delivery of business value through productionised machine learning models and software delivery.
July 13, 2018 | Software Consultancy
As a consultant I often find myself in situations that require tricky problem solving, typically of a technical nature. Yet although it is common to approach a consultancy engagement in terms of its technical context, not all problems have a purely technical solution.
May 31, 2018 | DevOps
As traditional operations has embraced the concept of code, it has benefited from ideas already prevalent in developer circles such as version control. Version control brings the benefit that not only can you see what the infrastructure was, but you can also get reviews of changes by your peers before the change is made live; known to most developers as Pull Request (PR) reviews.
May 16, 2018 | Microservices
To identify service boundaries, it is not enough to consider (business) domains only. Other forces like organisational communication structures, and – very important – time, strongly suggest that we should include several other criteria in our considerations.
April 18, 2018 | Microservices
Quite a few of the anti-patterns we observe today on microservices projects are strongly related to how people approach the problem. Given their nature, these anti-patterns tend to be deeply ingrained and self-sustaining. Addressing them starts with increased awareness and by changing ways of approaching the problem, rather than by the introduction of yet another technical tool or framework.
Google Cloud Functions is the Google Cloud Platform (GCP) function-as-a-service offering. It allows you to execute your code in response to event triggers – HTTP, PubSub and Storage. While it currently only supports Node.js code for execution, it has proved very useful for running low-frequency operational tasks and other batch jobs in GCP.
Events are obviously the fundamental building block of event-sourced systems. Commands are equally a common concept in such systems although the distinction between events and commands, if any, is not always clear. There are certainly varying views on what role each one should play.
February 14, 2018 | Cloud
AWS Announced a few new products for use with containers at RE:Invent 2017 and of particular interest to me was a new Elastic Container Service(ECS) Launch type, called Fargate
Prior to Fargate, when it came to creating a continuous delivery pipeline in AWS, the use of containers through ECS in its standard form, was the closest you could get to an always up, hands off, managed style of setup. Traditionally ECS has allowed you to create a configured pool of “worker” instances, with it then acting as a scheduler, provisioning containers on those instances.
Machine Learning is a hot topic these days, as can be seen from search trends. It was the success of Deepmind and AlphaGo in 2016 that really brought machine learning to the attention of the wider community and the world at large.
The recent 0.10.0 release of HashiCorp Terraform, saw a significant change to the way Providers are managed. Specifically, the single open source code repository for Terraform has been divided into core and multiple provider repositories.
August 8, 2017 | Cassandra
Recently, the sad news has emerged that Basho, which developed the Riak distributed database, has gone into receivership. This would appear to present a problem for those who have adopted the commercial version of the Riak database (Riak KV) supported by Basho.
This blog is written exclusively by the OpenCredo team. We do not accept external contributions.
May 9, 2017 | Cassandra
Data analytics isn’t a field commonly associated with testing, but there’s no reason we can’t treat it like any other application. Data analytics services are often deployed in production, and production services should be properly tested. This post covers some basic approaches for the testing of Cassandra/Spark code. There will be some code examples, but the focus is on how to structure your code to ensure it is testable!
This blog is written exclusively by the OpenCredo team. We do not accept external contributions.
April 13, 2017 | Terraform Provider
Recently I’ve been doing a lot with Terraform; having briefly flirted with it in the past, it’s only now with v0.8.x that I’ve been seriously stepping out with it (and Azure, since you asked). In the main I think it’s great, especially as it means I don’t have to yak-shave with the AWS and Azure CLIs. However, I have started to bang my head against some of Terraform’s limitations, specifically around HCL (Hashicorp Configuration Language) – used to define infrastructure in the Terraform .tf files.
In recent years, Cassandra has become one of the most widely used NoSQL databases: many of our clients use Cassandra for a variety of different purposes. This is no accident as it is a great datastore with nice scalability and performance characteristics.
However, adopting Cassandra as a single, one size fits all database has several downsides. The partitioned/distributed data storage model makes it difficult (and often very inefficient) to do certain types of queries or data analytics that are much more straightforward in a relational database.
On previous blog posts we have provided examples of different types of acceptance tests coverage, UI, API and Performance. One area where automation is often lacking is around validating the security of the application under test. This has been discussed in the post on non functional testing You Are Ignoring Non-functional Testing. With this post we will enhance the automation framework to quickly check for some common security flaws.
March 20, 2017 | DevOps
DevOps has swept the tech landscape. Now, many are discovering the benefits of programmable infrastructure. I have been lucky to work on many projects where we’ve taken advantage of tools such as Terraform, Ansible, or Chef.
Google has recently made its internal Spanner database available to the wider public, as a hosted solution on Google Cloud. This is a distributed relational/transactional database used inside for various Google projects (including F1, the advertising backend), promising high throughput, low latency and 99.999% availability. As such it is an interesting alternative to many open source or other hosted solutions. This whitepaper gives a good theoretical introduction into Spanner.
February 28, 2017 | Software Consultancy
As Kotlin’s 1.1 release draws closer, I’ve been looking at some of the new language features it supports. Type aliases may seem like a relatively minor feature next to coroutines, but as I will show in this blog post, they can open up a new programming idiom, particularly when combined with extension functions.
February 16, 2017 | Cassandra
One of the default Cassandra strategies to deal with more sophisticated queries is to create CQL tables that contain the data in a structure that matches the query itself (denormalization). Cassandra 3.0 introduces a new CQL feature, Materialized Views which captures this concept as a first-class construct.
February 13, 2017 | Data Engineering
One of the stated intentions behind the design of Java 8’s Streams API was to take better advantage of the multi-core processing power of modern computers. Operations that could be performed on a single, linear stream of values could also be run in parallel by splitting that stream into multiple sub-streams, and combining the results from processing each sub-stream as they became available.
January 25, 2017 | Cassandra
One of the simplest and best-understood models of computation is the Finite State Machine (FSM). An FSM has fixed range of states it can be in, and is always in one of these states. When an input arrives, this triggers a transition in the FSM from its current state to the next state. There may be several possible transitions to several different states, and which transition is chosen depends on the input.
January 24, 2017 | Cloud
This blog aims to provide an end to end example of how you can automatically request, generate and install a free HTTPS/TLS/SSL certificate from Let’s Encrypt using Terraform. Let’s Encrypt is a free, automated, and open certificate authority (CA) aiming to make it super easy (and free – did I say free!) for people to obtain HTTPS (SSL/TLS) certificates for their websites and infrastructure. Under the hood, Let’s Encrypt implements and leverages an emerging protocol called ACME to make all this magic happen, and it is this ACME protocol that powers the Terraform provider we will be using. For more information on how Let’s Encrypt and the ACME protocol actually work, please see how Let’s Encrypt works.
January 23, 2017 | Data Analysis
More often than not, people who write Go have some sort of opinion on its error handling model. Depending on your experience with other languages, you may be used to different approaches. That’s why I’ve decided to write this article, as despite being relatively opinionated, I think drawing on my experiences can be useful in the debate. The main issues I wanted to cover are that it is difficult to force good error handling practice, that errors don’t have stack traces, and that error handling itself is too verbose.
October 4, 2016 | Software Consultancy
As many of you know, OpenCredo are part of the global Trifork family, and as such have access to the combined knowledge and experience of many technology and business leaders throughout the group. Getting public access to all of this expertise and technical leadership can be tricky – until now. GOTO Accelerate is a one-day business focused conference that has emerged from the very successful GOTO technology events.
If there is one thing to understand about Cassandra, it is the fact that it is optimised for writes. In Cassandra everything is a write including logical deletion of data which results in tombstones – special deletion records. We have noticed that lack of understanding of tombstones is often the root cause of production issues our clients experience with Cassandra. We have decided to share a compilation of the most common problems with Cassandra tombstones and some practical advice on solving them.
September 21, 2016 | DevOps
Sometimes, it can be difficult to write automated tests for parts of your application due to complexities introduced by an external dependency. It may be flaky or have some sort of rate limiting, or require sensitive information which we don’t want to expose outside of our production environment. To get around this, teams might take the approach of manually stubbing the service or using mocks – but the former is tedious and error prone, whereas the latter doesn’t test collaboration at all.
September 15, 2016 | Cassandra
Cassandra isn’t a relational database management system, but it has some features that make it look a bit like one. Chief among these is CQL, a query language with an SQL-like syntax. CQL isn’t a bad thing in itself – in fact it’s very convenient – but it can be misleading since it gives developers the illusion that they are working with a familiar data model, when things are really very different under the hood.
September 6, 2016 | Cassandra
A growing number of clients are asking OpenCredo for help with using Apache Cassandra and solving specific problems they encounter. Clients have different use cases, requirements, implementation and teams but experience similar issues. We have noticed that Cassandra data modelling problems are the most consistent cause of Cassandra failing to meet their expectations. Data modelling is one of the most complex areas of using Cassandra and has many considerations.
August 26, 2016 | Kubernetes
This post is the second of a series of three tutorial articles introducing a sample, tutorial project, demonstrating how to provision Kubernetes on AWS from scratch, using Terraform and Ansible. To understand the goal of the project, you’d better start from the first part.
August 26, 2016 | Kubernetes
This post is the first of a series of three tutorial articles introducing a sample, tutorial project, demonstrating how to provision Kubernetes on AWS from scratch, using Terraform and Ansible.
August 24, 2016 | Cassandra
At OpenCredo we are seeing an increase in adoption of Apache Cassandra as a leading NoSQL database for managing large data volumes, but we have also seen many clients experiencing difficulty converting their high expectations into operational Cassandra performance. Here we present a high-level technical overview of the major strengths and limitations of Cassandra that we have observed over the last few years while helping our clients resolve the real-world issues that they have experienced.
July 3, 2016 | DevOps
Several of us from the OpenCredo team were in attendance at the inaugural EU edition of the DevOps Enterprise Summit conference. We have been big fans of the two previous US versions, and have watched the video recordings of talks (2014, 2015) with keen interest as many of our DevOps transformation clients are very much operating in the ‘enterprise’ space.
June 24, 2016 | Software Consultancy
Akka has been designed with a Java API from the very first version. Though widely adopted, as a Java developer I think Akka has been mainly a Scala thing… until recently. Things are changing and Akka is moving to a proper Java 8 support.
June 15, 2016 | Software Consultancy
It’s as simple as that – and as a consultant, it’s a problem I see all the time. Testing is always focused on functional testing. Non-functional testing, by comparison, is treated like a second class citizen. This means that functional requirements get refined, and non-functional requirements are ignored until the very end.
June 7, 2016 | Software Consultancy
In my previous blog post I explained to you the Seven Deadly Sins of Project Managers. With that post in mind, I would like to share with you my views on what a project manager needs to overcome these sins and become a valuable member of the team, and ultimately helping to achieve success on the project.
May 31, 2016 | Kubernetes
Do you ever wake up and think to yourself: oh geez, Kubernetes is awesome, but I wish I could browse and edit my services and replication controllers using the file system? No? Well, in any case, this is now possible.
May 27, 2016 | Software Consultancy
People make mistakes. People can improve. People need to be told of their mistakes. Surprisingly project managers are people too… I know… in some cases it is really hard to believe it; and then they can improve too! Shocking! I have more than 15 years working in IT, occupying different roles from software development to project management in three different countries. I’ve seen really bad project managers and some really good project managers; Yes, they DO exist!
April 28, 2016 | Software Consultancy
In a conventional RDBMS-with-ORM system, we are used to thinking of domain objects as mapped to rows in database tables, and of the database as a repository where the current state of every object exists simultaneously, so that what we get when we query for an object is the state that object was in at the time the query was issued. To perform an update, we can start a transaction, retrieve the current state of the object, modify it, save it back again and commit. Transactions move the global state of the system from one consistent state to another, so that the database transaction log represents a single, linear history of updates. We are therefore able to have a very stable, intuitive sense of what it means to talk about the “current state” of any domain object.
April 27, 2016 | Software Consultancy
Concursus is an open source Java 8 framework for building distributed systems using CQRS and event sourcing patterns. One of its major differences from other such frameworks (such as Jdon, Axon and ES4J) is that it eschews a programming model where each event type is represented by a separate Java class, instead mapping event types to methods on interfaces.
April 5, 2016 | Software Consultancy
This post is part of a series which introduce key concepts in successful test automation. Each post contains sample code using the test-automation-quickstart project, a sample Java test automation framework available from Github.
April 2, 2016 | Terraform Provider
When it comes to automating the creation of infrastructure in cloud providers, Terraform (version at time of writing 0.6.14) has become one of my core go to tools in this space. It provides a fantastic declarative approach to describing the resources you want, and then takes care of making it so for you, keeping track of the state in either a local file or a remote store of some sort. Various bits of sensitive data is often provided as input to terraform.
March 29, 2016 | Software Consultancy
Acceptance test suites generally are used for UI and API testing, and we have covered both these approaches in our Test Automation Quickstart project. However, an application may, for example, send registration or expiration warning emails. Often, tests related to this are left to manual testing, instead of putting them into an automated test suite.
However, there’s no need to check emails manually: it suffers from all the same problems as other manual testing. It’s slow, expensive, and inconsistent. There are many libraries available to interact with email through code – this post will focus on how to use them within an automated test suite.
March 17, 2016 | Software Consultancy
In order to be able to regularly release an application, your automated tests must be set up to give you fast and reliable feedback loop. If bugs are only found during a long and expensive multi-service or end-to-end test run, it can be a hinderance to fast delivery. Unfortunately I have often seen this problem in development environments: a huge suite of clunky, flaky and slow end-to-end tests which test the full functionality of the application as opposed to being more lightweight and reflecting basic user journeys. This produces the “ice cream cone” anti-pattern of test coverage, where unit tests aren’t providing the kind of coverage and feedback they need to.
March 14, 2016 | Software Consultancy
Test automation provides fast feedback on regressions. In order to achieve this tests need to execute quickly, something which becomes more of a problem as test suites grow. This is especially true of tests which exercise a user interface where the interaction with the system is slower.
A good way to address this is to have your tests execute in parallel rather than consecutively. Given sufficient resources this allows your execution time to remain low almost indefinitely as more scenarios are added to the suite.
March 7, 2016 | DevOps
Businesses exist to make money; their purpose isn’t just to generate revenue, but to create profits, now and in the future. Generating profits means delivering products or services that people want to buy. The creation of what people want is the entire purpose of delivery pipelines. (NB: The rest of this article will use ‘product’ to refer to both products and services.)
March 3, 2016 | Software Consultancy
In this post, I’ll be sharing some React/Redux boilerplate code that Vince Martinez and I have been developing recently. It’s primarily aimed at developers who are familiar with the React ecosystem, so if you are new to React and/or Redux, you might like to have a look at Getting Started with React and Getting Started with Redux.
Many of our clients are currently implementing applications using a ‘microservice’-based architecture. Increasingly we are hearing from organisations that are part way through a migration to microservices, and they want our help with validating and improving their current solution. These ‘microservices checkup’ projects have revealed some interesting patterns, and because we have experience of working in a wide-range of industries (and also have ‘fresh eyes’ when looking at a project), we are often able to work alongside teams to make significant improvements and create a strategic roadmap for future improvements.
March 2, 2016 | Microservices
Microservice-style software architectures have many benefits: loose coupling, independent scalability, localised failures, facilitating the usage of polyglot data persistence tools or multiple programming languages.
However, they also introduce other challenges. A major one is the fact that the end-user functionality of the system will ultimately emerge as a composition of multiple services. This significantly increases the complexity of deploying the system. In addition, because we lose the concept of “versions” of the system, it becomes harder to answer questions like “what capabilities are in production?” and “when is a new feature considered ‘done’?”.
January 30, 2016 | Microservices
The OpenCredo team will be presenting two sessions on our recent learnings with implementing microservices at the OOP Conference, which will be running from the 1st – 5th February. We will also be running a booth, and so if you are interested in learning more about our recent projects, are keen to see if we can help you with your latest technical or organisational challenges, or want to join our team, then stop by and say hello!
January 29, 2016 | DevOps
DevOps is 2016’s tech holy grail – unified development and operations, both working to deliver what the business needs, quickly, reliably, and adaptably. Done well, DevOps transforms the way organisations work; it helps break down barriers between tech teams, and between technology and the rest of the business. Good DevOps is the antidote to increasing segmentation and specialisation within companies. With the promised benefits, is it any wonder that senior managers are pushing for it in organisations spanning all sizes and industries?
January 18, 2016 | Software Consultancy
Last time in this series I summarised all the Akka Persistence related improvements in Akka 2.4. Since then Akka 2.4.1 has been released with some additional bug fixes and improvements so perhaps now is a perfect time to pick up this mini-series and introduce some other new features included in Akka 2.4.x.
January 8, 2016 | Microservices
Many of our clients are in the process of investigating or implementing ‘microservices’, and a popular question we often get asked is “what’s the most common mistake you see when moving towards a microservice architecture?”. We’ve seen plenty of good things with this architectural pattern, but we have also seen a few recurring issues and anti-patterns, which I’m keen to share here.
Good consulting is, by its nature, an act of collaboration. We recently helped a company with a variety of challenges – some architecture, some coding, some systems, some people, some process (normal consultancy challenges) – unique to this client. During the project, we formalised some things we had thought before, but which had never crystallised – all the work we did was transformative. Whether it’s a code review, process review, DevOps implementation, or outright transformation, the primary goal is the same – improving flow. Flow (sometimes known as throughput) is the movement of raw materials through a system to become finished goods. It’s analogy in the service industry is the movement of customer requirements through to usable solution. And we help improve it.
In the rush to embrace DevOps, many organisations seek out tools to help them achieve DevOps nirvana; the magical tools that will unify Development and Operations, stop the infighting, and ensure collaboration. This search for tools to solve problems exists in many domains, but seems particularly prevalent in IT (it may be real, or a reflection of my exposure to IT). The temptation to embrace new tools as a panacea is high, because the problems in IT seem so pervasive and persistent.
December 1, 2015 | Software Consultancy
This post introduce some of the basic features of Hazelcast, some of its limitations, how to embed it in a Spring Boot application and write integration testings. This post is intended to be the first of a series about Hazelcast and its integration with Spring (Boot). Let’s start from the basics.
November 25, 2015 | Microservices
In May a 1.0 release of RAML (RESTful API Markup Language) has been announced delivering a few much welcome additions in the RAML 1.0 specification. This major release marks an important milestone in the evolution of RAML and indicates the team behind the specification is confident this release delivers the comprehensive set of tools for developing RESTful APIs. I’ve been using RAML 0.8 for several months now and have enjoyed the simplicity and productivity it offers for designing and documenting APIs. I must say I’m quite pleased with the changes introduced in the new release and would like to review those I consider particularly useful.
It was once again a privilege to present at the annual ‘muCon 2015‘ microservices conference held in London (at the shiny new Skillsmatter CodeNode venue). Based on feedback fro talks I gave earlier in the year, I presented a completely new version of my ‘The Business Behind Microservices‘ talk, which focuses on the organisational and people side of implementing a microservice-based application.
November 4, 2015 | Software Consultancy
Writing reusable roles for Ansible is not an easy task but one that’s worth doing. This post should walk you through the basics of writing reusable roles with dependencies backed by public and private git repositories.
November 3, 2015 | Software Consultancy
My JavaOne experience was rather busy this year, what with three talks presented in a single day! The first of these talks “Debugging Java Apps in Containers: No Heavy Welding Gear Required” was delivered with my regular co-presenter Steve Poole, from IBM, and we shared our combined experiences of working with Java and Docker over the past year.
November 1, 2015 | Microservices
To use or not to use hypermedia (HATEOAS) in a REST API, to attain the Level 3 of the famous Richardson Maturity Model. This is one of the most discussed subjects about API design.
The many objections make sense (“Why I hate HATEOAS“, “More objections to HATEOAS“…). The goal of having fully dynamic, auto-discovering clients is still unrealistic (…waiting for AI client libraries).
However, there are good examples of successful HATEOAS API. Among them, PayPal.
October 31, 2015 | Microservices
Over the past few weeks I’ve been writing an OpenCredo blog series on the topic of “Building a Microservice Development Ecosystem”, but my JavaOne talk of the same title crept up on me before I managed to finish the remaining posts. I’m still planning to finish the full blog series, but in the meantime I thought it would be beneficial to share the video and slides associated with the talk, alongside some of my related thinking. I’ve been fortunate to work on several interesting microservice projects at OpenCredo, and we’re always keen to share our knowledge or offer advice, and so please do get in touch if we can help you or your organisation.
In some companies, the inevitable rapidly became accepted as the way to do things, and both development and IT operations worked together to figure out how to collaborate on building systems that satisfied development’s desire for change, and operations desire for stability. Outsourcing infrastructure, and all it implied, gave rise to Devops – the unification of business needs, developer delivery, and operational capacity – but it also gave rise to something else, in companies where the operations teams weren’t quite as quick to move – Shadow IT.
October 28, 2015 | Software Consultancy
Let’s have a quick look at the most interesting changes and new features that are now available to Akka users. As there are many new features to highlight in the new Akka release I will focus on those related to Akka Persistence first and cover other areas in a separate post.
Last week Steve Poole and I were once again back at the always informative JAX London conference talking about DevOps and the Cloud. This presentation built upon our previous DevOps talk that was presented last year, and focused on the experiences that Steve and I had encountered over the last year (the slides for our 2014 “Moving to a DevOps” mode talk can be found on SlideShare, and the video on Parleys).
October 16, 2015 | Software Consultancy
OpenCredo is helping Skillsmatter with the organisation of the inaugural ContainerSched conference, and we were last night in attendance at CodeNode, working our way to finalising the program alongside the Skillsmatter team. I’m pleased to say that the provisional lineup looks great (speaker acceptance emails are being sent out over the next few days), and so I wanted to share the details of some of the great content we have confirmed already.
October 12, 2015 | DevOps
DevOps is transformative. This (hopefully) won’t be true forever, but it is for now. While the modern management practices of separating development and operations (and to a lesser extent, everyone else) prevail, the tearing down of the walls that separate them will remain transformative. In company after company, management and front-line staff are coming to realise that keeping functions separate, which are inherently interdependent, is a model for blame, shifted responsibility, and acrimony. It’s easy to divvy-up a company up based on function. To many people, it seems the most logical way to do it. Ops does operations, Dev does development, Marketing markets, etc. It seems much harder to do it any other way. So why do it?
September 24, 2015 | Software Consultancy
You’ve implemented a change in how things work, and people aren’t happy. You spent time investigating the problem, and putting serious thought into what the issue was, and you’ve put a fix in place that you were sure people would be happy with. They aren’t. Why not?
September 24, 2015 | Microservices
Unless you’ve been living under a (COBOL-based) rock for the last few years, you will have no doubt heard of the emerging trend of microservices. This approach to developing ‘loosely coupled service-oriented architecture with bounded contexts’ has captured the hearts and minds of many developers. The promise of easier enforcement of good architectural and design principles, such as encapsulation and interface segregation, combined with the availability to experiment with different languages and platforms for each service, is a (developer) match made in heaven.
September 18, 2015 | Microservices
We’re pleased to begin our series of OpenCredo webinars with “The Business Behind Microservices”, which takes a look at the some of the business and organisational challenges that come along with the decision to implement microservices.
Recently I was working on a project that was using SaltStack for configuration management and Consul for service discovery. It occurred to me that using Consul’s key/value store would be great place to store data needed for my Salt runs, but unfortunately Consul was not supported in SaltStack as an official data store at that point in time. Being an open source project however, this provided an excellent opportunity to contribute back and this blog post looks to provide some details on how this works, as well as a practical demo on how you can take advantage of Consul as an external data store.
If you are operating in the programmable infrastructure space, you will hopefully have come across Terraform, a tool from HashiCorp which is primarily used to manage infrastructure resources such as virtual machines, DNS names and firewall settings across a number of public and private providers (AWS, GCP, Azure, …).
August 12, 2015 | Microservices
Over the last few months one of my main projects at OpenCredo has involved creating various microservices which are interacted with via REST. We’ve been working with a relatively rich domain model, which in turn has presented a lot of challenges in how to design our various resources. This blog post aims to summarise various techniques and practices which I’ve found helpful in overcoming these challenges.
August 11, 2015 | DevOps
For years, OpenCredo has been working with organisations to help them introduce new technologies, and more effective development practices, to their IT teams. This has met with a great deal of success, and we have worked with a variety of companies of various sizes. During these projects, we have consistently noticed that the changes we make reach beyond IT in their impact and effects.
As a company, we at OpenCredo are heavily involved in automation and devOps based work, with a keen focus on making this a seamless experience, especially in cloud based environments. We are currently working within HMRC, a UK government department to help make this a reality as part of a broader cloud broker ecosystem project. In this blog post, I look to provide some initial insight into some of the tools and techniques employed to achieve this for one particular use case namely:
With pretty much zero human intervention, bar initiating a process and providing some inputs, a development team from any location, should be able to run “something”, which, in the end, results in an isolated, secure set of fully configured VM’s being provisioned within a cloud provider (or providers) of choice.
A few weeks ago, we thought about building a Google analytics dashboard to give us easy access to certain elements of our Google Analytics web traffic. We saw some custom dashboards for bloggers, but nothing quite right for our goal, since we wanted the data on a big screen for everyone in the office to view.
Why OpenCredo partnered with Google
Recently OpenCredo chose to partner with Google in order to share knowledge and resources around the Google Cloud Platform offerings. Our clients come in many shapes and sizes, but typically all of them realise three disruptive truths of the modern IT industry: the (economic) value of cloud; the competitive advantage of continuous delivery; and the potential of hypothesis and data-driven product development to increase innovation (as popularised by the Lean Startup / Lean Enterprise motto of ‘build, measure, learn’).
July 28, 2015 | Microservices
Recently I co-presented a talk at Goto Amsterdam on lessons learnt whilst developing with a Microservices architecture; one being the importance of defining and documenting your API contracts as early as possible in the development cycle. During the talk I mentioned a few API documentation tools that I’d used and, based on feedback and questions from attendees, I realised that this topic merited a blog post. So, the purpose of this is to introduce 5 tools which help with designing, testing and documenting APIs.
July 8, 2015 | Software Consultancy
January 30, 2015 | Software Consultancy
When I first started programming in Scala a few years ago, Traits was the language feature I was most excited about. Indeed, Traits give you the ability to compose and share behaviour in a clean and reusable way. In Java, we tend deal with the same concerns by grouping crosscutting behaviour in abstract base classes that are subsequently extended every time we need to access shared behaviour in part or in total.
November 21, 2014 | White Paper
In our latest white paper we cover the high level fundamentals of test automation, discuss the primary problems test automation solves and take a look at the steps required to implement this within your development process – all aimed at improving software delivery.
November 19, 2014 | Microservices
Undeniably, there is a growing interest in microservices as we see more organisations, big and small, evaluating and implementing this emerging approach. Despite its apparent novelty, most concepts and principles underpinning microservices are not exactly new – they are simply proven and commonsense software development practices that now need to be applied holistically and at a wider scale, rather than at the scale of a single program or machine. These principles include separation of concerns, loose coupling, scalability and fault-tolerance.
Last year some of us attended the London Spring eXchange where we encountered a new and interesting tool that Pivotal was working on: Spring Boot. Since then we had the opportunity to see what it’s capable of in a live project and we were deeply impressed.
February 17, 2014 | Cassandra
In our previous posts we gave an overview of Cassandra’s new compare-and-set (lightweight transaction) commands and a more detailed look into the API for using them when inserting new rows into the database.
In this third post, we are going to cover update statements. We recommend reading the previous posts, as there are some details which are the same for inserts and updates which are not repeated here.
January 28, 2014 | Software Consultancy
A while ago I published this blog post about writing tests for mobile applications using Appium and cucumber-jvm.
In this post, I will look at an alternative approach to testing an Android native application using Cucumber-Android.
Throughout the post I will draw comparisons between Appium and Cucumber-Android, the goal being to determine the best approach for testing an android application using Cucumber. I will focus on the ease of configuration and use, speed of test runs and quality of reporting.
January 6, 2014 | Cassandra
The team over at Cucumber Pro recently posted a sneak peek on their blog, demonstrating some key features of their offering.
As more of a technical user of Cucumber, there isn’t much that’s new or ground-breaking for me – almost every feature is already available through your preferred IDE combined with a few plugins.
December 2, 2013 | Cassandra
Perhaps the most important of Cassandra’s selling points is its completely distributed architecture and its ability to easily extend the cluster with virtually any number of nodes. Implementing a classical RDBMS-style transaction consisting of “put locks on the database, modify the data, then commit the transaction”-style operations are simply not feasible in such an architecture (i.e. that doesn’t scale well).
November 14, 2013 | Cassandra
Cassandra 2.0 was released in early September this year and came with some interesting new features, including “lightweight transactions” and triggers.
Despite the rising interest in the various non-relational databases in recent years, there are still numerous use-cases for which a relational database system is a better choice. The latest major release of Cassandra (version 2.0) provides some interesting features that aim to close this gap, and offers its fast and distributed storage engine enhanced with new options that will make users’ lives easier.
November 6, 2013 | Software Consultancy
In many organisations, development and test teams have a ready answer for this, and that answer is usually wrong. Commonly, teams use test counts and code coverage statistics, which alone are not enough to validate a test approach and run the risk of giving a false sense of security to stakeholders. In practice, we are not able to fully prove the efficacy of our test strategy until after a release. Once software is in use, new defects highlight where our tests are failing to validate the software and where we need to invest effort to improve coverage. This is where many teams fail to learn and improve.
September 19, 2013 | Software Consultancy
This post will give an overview of mobile testing using Appium. We will integrate tests for a native Android application into an existing Cucumber-JVM based set of acceptance tests and demonstrate multi platform testing from a single set of BDD scenarios. The sample code for this can be found here.
February 19, 2013 | Software Consultancy
This blog post continues on from Part 1 which discussed types of tests and how to create robust tests. Part 2 will examine techniques to help whip a test suite in to shape and resolve common issues that slow everything down. The approaches in this post will focus on spring based applications, but the concepts can be applied to other frameworks too.
February 4, 2013 | Software Consultancy
This API will in future be used by a mobile client and by third parties, making it important to verify that it is functionally correct as well as clearly documented.
An additional requirement in our case is for the tests to form a specification for the API to allow front and back end developers agree on the format in advance. This is something that BDD excels at, making it natural to continue to use Cucumber. This post will focus on the difficulties of attaining the appropriate level of abstraction with Cucumber while retaining the technical detail required for specification.
January 10, 2013 | DevOps
Recently I have started looking into SaltStack as a solution that does both config management and orchestration. It is a relatively new project started in 2011, but it has a growing fanbase among Sys Admins and DevOps Engineers. In this blog post I will look into Salt as a promising alternative, and comparing it to Puppet as a way of exploring its basic set of features.
December 18, 2012 | Software Consultancy
The first thing most people think of when they start a project with the good intentions of test driven development is: write a test first. That’s great, and something I would fully encourage. However, diving in to writing tests without forethought, especially on large projects with a lot of developers can lead to new problems that TDD is not going to solve. With some upfront thinking (but not big upfront design!) a large team can avoid problems later down the line by considering some important and desirable traits of a large and rapidly changing test suite.
August 16, 2012 | Neo4j
It’s been more than a year now since I rolled out Neo4j Graph Database Server image in Amazon EC2.
In May 2011 the version of Neo4j was 1.3 and just recently guys at Neo Technology published version 1.7.2 so I thought now is the time to revisit this exercise and make fresh AMIs available.
Last year I created Neo4j AMI manually in one region then copied it across to the remaining AWS regions. Due to the size of the AMI and the latency between regions this process was slow.
March 21, 2012 | Software Consultancy
Event processing Language (EPL) enables us to write complex queries to get the most out of our event stream in real time, using SQL-like syntax.
EPL allows us to use full power of aggregation of the high volume event stream to get required results with the minimal latency. In this blog we are going to explore some aspects of numerical aggregation of data with high precision BigDecimal values. We will also demonstrate how you can add you own aggregation function to Esper engine and use them in EPL statements.
February 3, 2012 | Software Consultancy
November 4, 2011 | Neo4j
In the previous post we compared the performance of fetching relationships from densely populated nodes using Neo4j native store and using lucene index.
We’ve seen that we can fetch the small subset of relationships from a super-node (containing ~1M relationships in total) directly from the Lucene index, the performance of the first run (cold-caches) is better then using the Neo store directly. The subsequent runs with caches warmed up show comparable performance, slightly in favor of direct Neo store fetching, sue to low level cache optimizations.
June 3, 2011 | Neo4j
Neo4J is one of the first graph databases to appear on the global market. Being open source, in addition to its power and simplicity in supporting graph data model it represents good choice for production-ready graph database.
However, there has been one area I have struggled to get good-enough performance from Neo4j recently – super nodes.