83 items found: Search results for "infrastructure" in all categories x
May 31, 2018 | DevOps
As traditional operations has embraced the concept of code, it has benefited from ideas already prevalent in developer circles such as version control. Version control brings the benefit that not only can you see what the infrastructure was, but you can also get reviews of changes by your peers before the change is made live; known to most developers as Pull Request (PR) reviews.
March 20, 2017 | DevOps
DevOps has swept the tech landscape. Now, many are discovering the benefits of programmable infrastructure. I have been lucky to work on many projects where we’ve taken advantage of tools such as Terraform, Ansible, or Chef.
January 24, 2017 | Cloud
This blog aims to provide an end to end example of how you can automatically request, generate and install a free HTTPS/TLS/SSL certificate from Let’s Encrypt using Terraform. Let’s Encrypt is a free, automated, and open certificate authority (CA) aiming to make it super easy (and free – did I say free!) for people to obtain HTTPS (SSL/TLS) certificates for their websites and infrastructure. Under the hood, Let’s Encrypt implements and leverages an emerging protocol called ACME to make all this magic happen, and it is this ACME protocol that powers the Terraform provider we will be using. For more information on how Let’s Encrypt and the ACME protocol actually work, please see how Let’s Encrypt works.
If you are operating in the programmable infrastructure space, you will hopefully have come across Terraform, a tool from HashiCorp which is primarily used to manage infrastructure resources such as virtual machines, DNS names and firewall settings across a number of public and private providers (AWS, GCP, Azure, …).
January 7, 2013 | Software Consultancy
The practice of continuous integration in which build servers are used to build and perform testing of code is now widespread and mainstream.
While not all teams have adopted continuous integration effectively, its increasing adoption has led many to start to look for additional opportunities to improve the cost, quality and speed of delivery with which software targeted to meet business needs can be released into production environments.
Traditionally Continuous Integration addresses the question of “does the software build and pass our unit and integration test suites?”. This is often insufficient.
Check out John Sharpe and Will May’s latest blog where they give suggestions for Terraform Provider authors who are thinking about upgrading from SDKv2 to Framework
Check out Matthew Revell-Gordon’s latest blog as he explores building a local Kubernetes test cluster to better mimic cloud-based deployments, using Colima, Kind, and MetalLB.
Read Matt Farrow’s blog as he explores the potential for using Open Policy Agent to filter and mask data being sent to and read from Apache Kafka.
Watch our Lunch & Learn by Hieu Doan and Alberto Faedda as they share how engineers and security teams can secure their software development processes with the Secure Pipelines application.
Serverless functions are easy to install and upload, but we can’t ignore the basics. This article looks at different strategies related to testing serverless functions.
November 4, 2021 | Kubernetes
We always read that ‘security is everyone’s responsibility’. For any organisation, big or small, security should always be the primary concern—not a mere afterthought. In terms of Kubernetes, securing a cluster is challenging because it has so many moving parts and, apart from securing our Kubernetes environment, we also want to control what an end-user can do in our cluster.
To achieve these goals, we can start with the built-in features provided by Kubernetes like Role-Based Access Control (RBAC), Network Policies, Secrets Management, and Pod Security Policies (PSP). But we know these features are not enough. For example, we may want specific policies like ‘all pods must have specific labels’. And even if we have the policies in place, the next big question is how to enforce them on our Kubernetes cluster in an easy and repeatable manner.
In this blog post, we’ll address this challenge and other questions pertaining to OPA and how it can integrate into Kubernetes.
Our recent client was a Fintech who had ambitions to build a Machine Learning platform for real-time decision making. The client had significant Kubernetes proficiency, ran on the cloud, and had a strong preference for using free, open-source software over cloud-native offerings that come with lock-in. Several components were spiked with success (feature preparation with Apache Beam and Seldon for model serving performed particularly strongly). Kubeflow was one of the next technologies on our list of spikes, showing significant promise at the research stage and seemingly a good match for our client’s priorities and skills.
That platform slipped down the client’s priority list before completing the research for Kubeflow, so I wanted to see how that project might have turned out. Would Kubeflow have made the cut?
Multi-cloud is rapidly becoming the cloud strategy of choice for enterprises looking to modernise their applications.
And the reason is simple – it gives them much more flexibility to host their workloads and data where it suits them best.
In this post, we focus on Google’s application modernisation solution Google Anthos and the role it can play in your cloud transformation strategy.
While working with a client recently, we experienced some issues when attempting to make use of NLB external load balancer services when using AWS EKS. I wanted to investigate whether these issues had been fixed in the upstream GitHub Kubernetes repository, or if I could fix it myself, contributing back to the community in the process.
October 15, 2020
Continuous Verification is a term that is starting to pop up from time-to-time… but what does it mean? Well… according to Nora Jones and Casey Rosenthal, authors of O’Reilly’s Chaos Engineering books,
“Continuous verification (CV) is a discipline of proactive experimentation, implemented as tooling that verifies system behaviors. This stands in contrast to prior common practices in software quality assurance, which favor reactive testing, implemented as methodologies that validate known properties of software. This isn’t to say that prior common practices are invalid or should be deprecated. Alerting, testing, code reviews, monitoring, SRE practices, and the like—these are all great practices and should be encouraged”
Over the course of this post, we will unpack this statement: to understand what is behind it and what it might mean for your development process.
With the upcoming Cassandra 4.0 release, there is a lot to look forward to. Most excitingly, and following a refreshing realignment of the Open Source community around Cassandra, the next release promises to focus on fundamentals: stability, repair, observability, performance and scaling.
We must set this against the fact that Cassandra ranks pretty highly in the Stack Overflow most dreaded databases list and the reality that Cassandra is expensive to configure, operate and maintain. Finding people who have the prerequisite skills to do so is challenging.
At the time of this post, the UK is making steps to exit from an unprecedented lockdown measures for the Coronavirus. Much of the UK workforce are still making efforts to work-from-home with mainly key workers operating – at risk – in public. Many industries have shut down completely. Consequently, many businesses are reflecting on what happens next and how do we better mitigate future pandemic events?
March 20, 2020
Traditionally, Usability and Security have been set in opposition to each other: with tight security, we end up with painful user experience. In this blog, Guy focuses on financial services as an exemplar of how we can introduce usability into a vertical with challenging security and compliance requirements.
Terraform 0.12 in recent years has emerged as the de facto standard with regards to defining and managing cloud infrastructure. It is one of four primary tools offered by HashiCorp, (Terraform, Vault, Consul and Nomad) and underpins the workflows that make up their Cloud Operating Model.
Since its first release in 2014, the wider Terraform community has embraced frequent releases and this past year has been no exception. HashiCorp announced the release of Terraform 0.12 in May 2019 and as of writing this post the official release is 0.12.9.
One of the benefits we have working at OpenCredo (OC) is the opportunity to both attend and speak (although not on this occasion) at conferences. For some of you, this may be pretty common, but OC was actually the first to offer me this as part of a broader learning and development plan.
Cloud-native development and delivery is a core area of expertise for OC and we are always looking for what’s new and interesting in this space. So when I was offered the chance to go to CloudNative London it seemed like a good place to start. With its diversity in topics and technologies, the conference provided a perfect opportunity to collaborate and hear from others in the industry and what they are doing in this space.
As a technology leader, you’ll be aware that competitive pressures and shifting business requirements are driving changes in the technical architectures of many organisations. This means you need a new strategic approach based on the ability to continually evolve elements of your systems and architectures.
Creating and managing a Public Key Infrastructure (PKI) could be a very straightforward task if you use appropriate tools. In this blog post, I’ll cover the steps to easily set up a PKI with Vault from HashiCorp, and use it to secure a Kafka Cluster.
While Prometheus has fast become the standard for monitoring in the cloud, making Prometheus highly available can be tricky. This blog post will walk you through how to do this using the open source tool Thanos.
May 16, 2018 | Microservices
To identify service boundaries, it is not enough to consider (business) domains only. Other forces like organisational communication structures, and – very important – time, strongly suggest that we should include several other criteria in our considerations.
April 18, 2018 | Microservices
Quite a few of the anti-patterns we observe today on microservices projects are strongly related to how people approach the problem. Given their nature, these anti-patterns tend to be deeply ingrained and self-sustaining. Addressing them starts with increased awareness and by changing ways of approaching the problem, rather than by the introduction of yet another technical tool or framework.
Google Cloud Functions is the Google Cloud Platform (GCP) function-as-a-service offering. It allows you to execute your code in response to event triggers – HTTP, PubSub and Storage. While it currently only supports Node.js code for execution, it has proved very useful for running low-frequency operational tasks and other batch jobs in GCP.
February 14, 2018 | Cloud
AWS Announced a few new products for use with containers at RE:Invent 2017 and of particular interest to me was a new Elastic Container Service(ECS) Launch type, called Fargate
Prior to Fargate, when it came to creating a continuous delivery pipeline in AWS, the use of containers through ECS in its standard form, was the closest you could get to an always up, hands off, managed style of setup. Traditionally ECS has allowed you to create a configured pool of “worker” instances, with it then acting as a scheduler, provisioning containers on those instances.
February 6, 2018 | Cloud
Among the many announcements made at Re:Invent 2017 was the release of AWS Privatelink for Customer and Partner services. We believe that the opportunity signalled by this modest announcement may have an impact far broader than first impressions suggest.
January 11, 2018 | Data Engineering
The last few years have seen Python emerge as a lingua franca for data scientists. Alongside Python we have also witnessed the rise of Jupyter Notebooks, which are now considered a de facto data science productivity tool, especially in the Python community. Jupyter Notebooks started as a university side-project known as iPython in circa 2001 at UC Berkeley.
The recent 0.10.0 release of HashiCorp Terraform, saw a significant change to the way Providers are managed. Specifically, the single open source code repository for Terraform has been divided into core and multiple provider repositories.
August 8, 2017 | Cassandra
Recently, the sad news has emerged that Basho, which developed the Riak distributed database, has gone into receivership. This would appear to present a problem for those who have adopted the commercial version of the Riak database (Riak KV) supported by Basho.
This blog is written exclusively by the OpenCredo team. We do not accept external contributions.
Over the years, OpenCredo’s projects have become increasingly tied to the public cloud. Our skills in delivering cloud infrastructure and cloud native applications have deepened and the range of cloud projects we are able to take on has grown. From enterprise cloud brokers to cloud platform migration in restricted compliance environments, our ability to deliver on the cloud is now a core component of our value proposition.
June 15, 2017 | Data Engineering
CockroachDB is a distributed SQL (“NewSQL”) database developed by Cockroach Labs and has recently reached a major milestone: the first production-ready, 1.0 release. We at OpenCredo have been following the progress of CockroachDB for a while, and we think it’s a technology of great potential to become the go-to solution for a having a general-purpose database in the cloud.
May 22, 2017 | Data Analysis
As a final piece of our recent blog series about Apache Spark on 16 May we have presented details of a use-case about using Spark Structured Streaming to generate real-time alerts of suspicious activity in an AWS-based infrastructure.
This blog is written exclusively by the OpenCredo team. We do not accept external contributions.
My recent blogpost I explored a few cases where using Cassandra and Spark together can be useful. My focus was on the functional behaviour of such a stack and what you need to do as a developer to interact with it. However, it did not describe any details about the infrastructure setup that is capable of running such Spark code or any deployment considerations. In this post, I will explore this in more detail and show some practical advice in how to deploy Spark and Apache Cassandra.
Apache Spark is a powerful open source processing engine which is fast becoming our technology of choice for data analytic projects here at OpenCredo. For many years now we have been helping our clients to practically implement and take advantage of various big data technologies including the like of Apache Cassandra amongst others.
April 13, 2017 | Terraform Provider
Recently I’ve been doing a lot with Terraform; having briefly flirted with it in the past, it’s only now with v0.8.x that I’ve been seriously stepping out with it (and Azure, since you asked). In the main I think it’s great, especially as it means I don’t have to yak-shave with the AWS and Azure CLIs. However, I have started to bang my head against some of Terraform’s limitations, specifically around HCL (Hashicorp Configuration Language) – used to define infrastructure in the Terraform .tf files.
Google has recently made its internal Spanner database available to the wider public, as a hosted solution on Google Cloud. This is a distributed relational/transactional database used inside for various Google projects (including F1, the advertising backend), promising high throughput, low latency and 99.999% availability. As such it is an interesting alternative to many open source or other hosted solutions. This whitepaper gives a good theoretical introduction into Spanner.
August 26, 2016 | Kubernetes
This post is the last of a series of three tutorial articles introducing a sample, tutorial project, demonstrating how to provision Kubernetes on AWS from scratch, using Terraform and Ansible. To understand the goal of the project, you’d better start from the first part.
August 26, 2016 | Kubernetes
This post is the second of a series of three tutorial articles introducing a sample, tutorial project, demonstrating how to provision Kubernetes on AWS from scratch, using Terraform and Ansible. To understand the goal of the project, you’d better start from the first part.
August 26, 2016 | Kubernetes
This post is the first of a series of three tutorial articles introducing a sample, tutorial project, demonstrating how to provision Kubernetes on AWS from scratch, using Terraform and Ansible.
July 3, 2016 | DevOps
Several of us from the OpenCredo team were in attendance at the inaugural EU edition of the DevOps Enterprise Summit conference. We have been big fans of the two previous US versions, and have watched the video recordings of talks (2014, 2015) with keen interest as many of our DevOps transformation clients are very much operating in the ‘enterprise’ space.
June 7, 2016 | Software Consultancy
In my previous blog post I explained to you the Seven Deadly Sins of Project Managers. With that post in mind, I would like to share with you my views on what a project manager needs to overcome these sins and become a valuable member of the team, and ultimately helping to achieve success on the project.
April 5, 2016 | Software Consultancy
This post is part of a series which introduce key concepts in successful test automation. Each post contains sample code using the test-automation-quickstart project, a sample Java test automation framework available from Github.
April 2, 2016 | Terraform Provider
When it comes to automating the creation of infrastructure in cloud providers, Terraform (version at time of writing 0.6.14) has become one of my core go to tools in this space. It provides a fantastic declarative approach to describing the resources you want, and then takes care of making it so for you, keeping track of the state in either a local file or a remote store of some sort. Various bits of sensitive data is often provided as input to terraform.
Many of our clients are currently implementing applications using a ‘microservice’-based architecture. Increasingly we are hearing from organisations that are part way through a migration to microservices, and they want our help with validating and improving their current solution. These ‘microservices checkup’ projects have revealed some interesting patterns, and because we have experience of working in a wide-range of industries (and also have ‘fresh eyes’ when looking at a project), we are often able to work alongside teams to make significant improvements and create a strategic roadmap for future improvements.
March 2, 2016 | Microservices
Microservice-style software architectures have many benefits: loose coupling, independent scalability, localised failures, facilitating the usage of polyglot data persistence tools or multiple programming languages.
However, they also introduce other challenges. A major one is the fact that the end-user functionality of the system will ultimately emerge as a composition of multiple services. This significantly increases the complexity of deploying the system. In addition, because we lose the concept of “versions” of the system, it becomes harder to answer questions like “what capabilities are in production?” and “when is a new feature considered ‘done’?”.
January 29, 2016 | DevOps
DevOps is 2016’s tech holy grail – unified development and operations, both working to deliver what the business needs, quickly, reliably, and adaptably. Done well, DevOps transforms the way organisations work; it helps break down barriers between tech teams, and between technology and the rest of the business. Good DevOps is the antidote to increasing segmentation and specialisation within companies. With the promised benefits, is it any wonder that senior managers are pushing for it in organisations spanning all sizes and industries?
October 31, 2015 | Microservices
Over the past few weeks I’ve been writing an OpenCredo blog series on the topic of “Building a Microservice Development Ecosystem”, but my JavaOne talk of the same title crept up on me before I managed to finish the remaining posts. I’m still planning to finish the full blog series, but in the meantime I thought it would be beneficial to share the video and slides associated with the talk, alongside some of my related thinking. I’ve been fortunate to work on several interesting microservice projects at OpenCredo, and we’re always keen to share our knowledge or offer advice, and so please do get in touch if we can help you or your organisation.
In some companies, the inevitable rapidly became accepted as the way to do things, and both development and IT operations worked together to figure out how to collaborate on building systems that satisfied development’s desire for change, and operations desire for stability. Outsourcing infrastructure, and all it implied, gave rise to Devops – the unification of business needs, developer delivery, and operational capacity – but it also gave rise to something else, in companies where the operations teams weren’t quite as quick to move – Shadow IT.
Last week Steve Poole and I were once again back at the always informative JAX London conference talking about DevOps and the Cloud. This presentation built upon our previous DevOps talk that was presented last year, and focused on the experiences that Steve and I had encountered over the last year (the slides for our 2014 “Moving to a DevOps” mode talk can be found on SlideShare, and the video on Parleys).
October 16, 2015 | Software Consultancy
OpenCredo is helping Skillsmatter with the organisation of the inaugural ContainerSched conference, and we were last night in attendance at CodeNode, working our way to finalising the program alongside the Skillsmatter team. I’m pleased to say that the provisional lineup looks great (speaker acceptance emails are being sent out over the next few days), and so I wanted to share the details of some of the great content we have confirmed already.
October 12, 2015 | DevOps
DevOps is transformative. This (hopefully) won’t be true forever, but it is for now. While the modern management practices of separating development and operations (and to a lesser extent, everyone else) prevail, the tearing down of the walls that separate them will remain transformative. In company after company, management and front-line staff are coming to realise that keeping functions separate, which are inherently interdependent, is a model for blame, shifted responsibility, and acrimony. It’s easy to divvy-up a company up based on function. To many people, it seems the most logical way to do it. Ops does operations, Dev does development, Marketing markets, etc. It seems much harder to do it any other way. So why do it?
October 1, 2015 | Data Engineering
Akka Streams, the new experimental module under Akka project has been finally released in July after some months of development and several milestone and RC versions. In this series I hope to gently introduce concepts from the library and demonstrate how it can be used to address real-life stream processing challenges.
Akka Streams is an implementation of the Reactive Streams specification on top of Akka toolkit that uses actor based concurrency model. Reactive Streams specification has been created by the number of companies interested in asynchronous, non-blocking, event based data processing that can span across system boundaries and technology stacks.
September 24, 2015 | Microservices
Unless you’ve been living under a (COBOL-based) rock for the last few years, you will have no doubt heard of the emerging trend of microservices. This approach to developing ‘loosely coupled service-oriented architecture with bounded contexts’ has captured the hearts and minds of many developers. The promise of easier enforcement of good architectural and design principles, such as encapsulation and interface segregation, combined with the availability to experiment with different languages and platforms for each service, is a (developer) match made in heaven.
September 20, 2015 | Microservices
Over the past five years I have worked within several projects that used a ‘microservice’-based architecture, and one constant issue I have encountered is the absence of standardised patterns for local development and ‘off the shelf’ development tooling that support this. When working with monoliths we have become quite adept at streamlining the development, build, test and deploy cycles. Development tooling to help with these processes is also readily available (and often integrated with our IDEs). For example, many platforms provide ‘hot reloading’ for viewing the effects of code changes in near-real time, automated execution of tests, regular local feedback from continuous integration servers, and tooling to enable the creation of a local environment that mimics the production stack.
Recently I was working on a project that was using SaltStack for configuration management and Consul for service discovery. It occurred to me that using Consul’s key/value store would be great place to store data needed for my Salt runs, but unfortunately Consul was not supported in SaltStack as an official data store at that point in time. Being an open source project however, this provided an excellent opportunity to contribute back and this blog post looks to provide some details on how this works, as well as a practical demo on how you can take advantage of Consul as an external data store.
August 26, 2015 | Cloud
Unless you’ve been living under a rock for the last year, you’ll undoubtedly know that microservices are the new hotness. An emerging trend that I’ve observed is that the people who are actually using microservices in production tend to be the larger well-funded companies, such as Netflix, Gilt, Yelp, Hailo etc., and each organisation has their own way of developing, building and deploying.
August 11, 2015 | DevOps
For years, OpenCredo has been working with organisations to help them introduce new technologies, and more effective development practices, to their IT teams. This has met with a great deal of success, and we have worked with a variety of companies of various sizes. During these projects, we have consistently noticed that the changes we make reach beyond IT in their impact and effects.
As a company, we at OpenCredo are heavily involved in automation and devOps based work, with a keen focus on making this a seamless experience, especially in cloud based environments. We are currently working within HMRC, a UK government department to help make this a reality as part of a broader cloud broker ecosystem project. In this blog post, I look to provide some initial insight into some of the tools and techniques employed to achieve this for one particular use case namely:
With pretty much zero human intervention, bar initiating a process and providing some inputs, a development team from any location, should be able to run “something”, which, in the end, results in an isolated, secure set of fully configured VM’s being provisioned within a cloud provider (or providers) of choice.
Why OpenCredo partnered with Google
Recently OpenCredo chose to partner with Google in order to share knowledge and resources around the Google Cloud Platform offerings. Our clients come in many shapes and sizes, but typically all of them realise three disruptive truths of the modern IT industry: the (economic) value of cloud; the competitive advantage of continuous delivery; and the potential of hypothesis and data-driven product development to increase innovation (as popularised by the Lean Startup / Lean Enterprise motto of ‘build, measure, learn’).
Working with OpenCredo clients, I’ve noticed that even if you are one of the few organisations that can boast ‘Infrastructure as Code’, perhaps it’s only true of your VMs, and likely you have ‘bootstrap problems’. What I mean by this, is that you require some cloud-infrastructure to already be in place before your VM automation can go to work.
August 2, 2013 | Software Consultancy
Configuration management was born in the pre-cloud era. Remember the days when acquiring a super powerful multi core server felt like winning the jackpot? Infrastructure was a slightly different place back then. Yet for all the recent developments in DevOps, its legacy is still with us.
January 10, 2013 | DevOps
Recently I have started looking into SaltStack as a solution that does both config management and orchestration. It is a relatively new project started in 2011, but it has a growing fanbase among Sys Admins and DevOps Engineers. In this blog post I will look into Salt as a promising alternative, and comparing it to Puppet as a way of exploring its basic set of features.
February 13, 2012 | Software Consultancy
Recently we were approached by a client to do some performance testing of the web application they had written. The budget allowed two days for this task. Ok. No problem. Yes, we can. Naturally I had one or another question though…