71 items found: Search results for "operations" in all categories x
January 29, 2016 | DevOps
DevOps is 2016’s tech holy grail – unified development and operations, both working to deliver what the business needs, quickly, reliably, and adaptably. Done well, DevOps transforms the way organisations work; it helps break down barriers between tech teams, and between technology and the rest of the business. Good DevOps is the antidote to increasing segmentation and specialisation within companies. With the promised benefits, is it any wonder that senior managers are pushing for it in organisations spanning all sizes and industries?
September 27, 2023 | Blog, Kubernetes
Learn to create your first Kubernetes operator by checking out our Senior Consultant Michal Tusnio’s latest blog, “Kubernetes Operators – Whys, Hows and Whats” where he takes you on a journey from zero to operator.
Check out Greg Nuttall’s latest blog where he looks at the challenges posed by GDPR’s “Right to be Forgotten” in the context of Apache Kafka, and delves deeper into three strategies for overcoming them.
March 9, 2023 | Blog, Data Analysis, Neo4j
Check out the last part of Ebru Cucen and Fahran Wallace’s blog series, in which they discuss their experience ingesting 400 million nodes and a billion relationships into Neo4j and what they discovered along the way.
February 16, 2023 | Blog, Data Analysis, Neo4j
Check out Part 2 of Ebru Cucen and Fahran Wallace’s blog series, in which they discuss their experience ingesting 400 million nodes and a billion relationships into Neo4j and what they discovered along the way.
February 1, 2023 | Blog, Data Analysis, Neo4j
As data becomes ubiquitous and deeply interconnected, tracing where who or which system that data comes from – its lineage – will create bigger problems and opportunities for us on the horizon. Watch the recording of James Bowkett talk from NODES 2022 – Neo4j Online Developer Education Summit 202 on ‘Tracing Your Data’s DNA.’
January 26, 2023 | Blog, Data Analysis, Neo4j
Fahran Wallace and Ebru Cucen’s most recent blog post is part 1 of a three-part series. They investigate how OpenCredo ingested 400 million nodes with a billion relationships into Neo4j.
March 10, 2022 | Data Engineering, Open Source
In this lunch & learn session, Ebru Cucen and Alberto Faedda explore the historical background of GraphQL with case examples and a demo of how it can be used.
April 20, 2021 | Data Engineering, Machine Learning, Software Consultancy
Our recent client was a Fintech who had ambitions to build a Machine Learning platform for real-time decision making. The client had significant Kubernetes proficiency, ran on the cloud, and had a strong preference for using free, open-source software over cloud-native offerings that come with lock-in. Several components were spiked with success (feature preparation with Apache Beam and Seldon for model serving performed particularly strongly). Kubeflow was one of the next technologies on our list of spikes, showing significant promise at the research stage and seemingly a good match for our client’s priorities and skills.
That platform slipped down the client’s priority list before completing the research for Kubeflow, so I wanted to see how that project might have turned out. Would Kubeflow have made the cut?
February 17, 2021 | Blog, Cloud, Cloud Native, GCP, Open Source
Multi-cloud is rapidly becoming the cloud strategy of choice for enterprises looking to modernise their applications.
And the reason is simple – it gives them much more flexibility to host their workloads and data where it suits them best.
In this post, we focus on Google’s application modernisation solution Google Anthos and the role it can play in your cloud transformation strategy.
December 11, 2020 | Cloud, Cloud Native, Kubernetes, Microservices
“WebAssembly is a safe, portable, low-level code format designed for efficient execution and compact representation.” – W3C
In this blog, I’ll cover the different applications of Wasm and WASI, some of the projects that are making headway, and the implications for modern architectures and distributed systems.
September 22, 2020 | AWS, Blog, Cassandra, Cloud, DevOps, Open Source
With the upcoming Cassandra 4.0 release, there is a lot to look forward to. Most excitingly, and following a refreshing realignment of the Open Source community around Cassandra, the next release promises to focus on fundamentals: stability, repair, observability, performance and scaling.
We must set this against the fact that Cassandra ranks pretty highly in the Stack Overflow most dreaded databases list and the reality that Cassandra is expensive to configure, operate and maintain. Finding people who have the prerequisite skills to do so is challenging.
April 2, 2020 | Machine Learning
Recent years have seen many companies consolidate all their data into a data lake/warehouse of some sort. Once it’s all consolidated, what next?
Many companies consolidate data with a field of dreams mindset – “build it and they will come”, however a comprehensive data strategy is needed if the ultimate goals of an organisation are to be realised: monetisation through Machine Learning and AI is an oft-cited goal. Unfortunately, before one rushes into the enticing world of machine learning, one should lay more mundane foundations. Indeed, in data science, estimates vary between 50% to 80% of the time taken is devoted to so-called data-wrangling. Further, Google estimates ML projects produce 5% ML code and 95% “glue code”. If this is the reality we face, what foundations are required before one can dive headlong into ML?
October 8, 2019 | Cloud, Cloud Native, Culture
Following on from the last two blogs by Stuart (who shared highlights for day 1) and Trent (who shared highlights from day 2), I will conclude with mine on CloudNative London 2019 Day 3.
The Cloud Native landscape can be bewildering, and not only for newcomers. As a traveller on the Cloud-native journey, I have sometimes been overwhelmed by the number of products and projects. This is why I took hold of the opportunity to go to Day 3 of the Cloud Native London Conference last month hosted by Skills Matter.
Here are my top highlights from Day 3 of the CloudNative London 2019.
September 12, 2019 | Cloud Native, Microservices, Software Consultancy
As a technology leader, you’ll be aware that competitive pressures and shifting business requirements are driving changes in the technical architectures of many organisations. This means you need a new strategic approach based on the ability to continually evolve elements of your systems and architectures.
February 20, 2019 | DevOps, Hashicorp, Kafka, Open Source
Creating and managing a Public Key Infrastructure (PKI) could be a very straightforward task if you use appropriate tools. In this blog post, I’ll cover the steps to easily set up a PKI with Vault from HashiCorp, and use it to secure a Kafka Cluster.
July 31, 2018 | Machine Learning
Machine Learning, alongside a mature Data Science, will help to bring IT and business closer together. By leveraging data for actionable insights, IT will increasingly drive business value. Agile and DevOps practices enable the continuous delivery of business value through productionised machine learning models and software delivery.
July 13, 2018 | Software Consultancy
As a consultant I often find myself in situations that require tricky problem solving, typically of a technical nature. Yet although it is common to approach a consultancy engagement in terms of its technical context, not all problems have a purely technical solution.
May 31, 2018 | DevOps
As traditional operations has embraced the concept of code, it has benefited from ideas already prevalent in developer circles such as version control. Version control brings the benefit that not only can you see what the infrastructure was, but you can also get reviews of changes by your peers before the change is made live; known to most developers as Pull Request (PR) reviews.
May 16, 2018 | Microservices
To identify service boundaries, it is not enough to consider (business) domains only. Other forces like organisational communication structures, and – very important – time, strongly suggest that we should include several other criteria in our considerations.
February 14, 2018 | Cloud
AWS Announced a few new products for use with containers at RE:Invent 2017 and of particular interest to me was a new Elastic Container Service(ECS) Launch type, called Fargate
Prior to Fargate, when it came to creating a continuous delivery pipeline in AWS, the use of containers through ECS in its standard form, was the closest you could get to an always up, hands off, managed style of setup. Traditionally ECS has allowed you to create a configured pool of “worker” instances, with it then acting as a scheduler, provisioning containers on those instances.
February 6, 2018 | Cloud
Among the many announcements made at Re:Invent 2017 was the release of AWS Privatelink for Customer and Partner services. We believe that the opportunity signalled by this modest announcement may have an impact far broader than first impressions suggest.
July 11, 2017 | Cloud, Cloud Native
Over the years, OpenCredo’s projects have become increasingly tied to the public cloud. Our skills in delivering cloud infrastructure and cloud native applications have deepened and the range of cloud projects we are able to take on has grown. From enterprise cloud brokers to cloud platform migration in restricted compliance environments, our ability to deliver on the cloud is now a core component of our value proposition.
June 15, 2017 | Data Engineering
CockroachDB is a distributed SQL (“NewSQL”) database developed by Cockroach Labs and has recently reached a major milestone: the first production-ready, 1.0 release. We at OpenCredo have been following the progress of CockroachDB for a while, and we think it’s a technology of great potential to become the go-to solution for a having a general-purpose database in the cloud.
May 9, 2017 | Cassandra
Data analytics isn’t a field commonly associated with testing, but there’s no reason we can’t treat it like any other application. Data analytics services are often deployed in production, and production services should be properly tested. This post covers some basic approaches for the testing of Cassandra/Spark code. There will be some code examples, but the focus is on how to structure your code to ensure it is testable!
This blog is written exclusively by the OpenCredo team. We do not accept external contributions.
May 2, 2017 | Cassandra, Data Engineering
My recent blogpost I explored a few cases where using Cassandra and Spark together can be useful. My focus was on the functional behaviour of such a stack and what you need to do as a developer to interact with it. However, it did not describe any details about the infrastructure setup that is capable of running such Spark code or any deployment considerations. In this post, I will explore this in more detail and show some practical advice in how to deploy Spark and Apache Cassandra.
March 23, 2017 | Cassandra, Data Analysis, Data Engineering
In recent years, Cassandra has become one of the most widely used NoSQL databases: many of our clients use Cassandra for a variety of different purposes. This is no accident as it is a great datastore with nice scalability and performance characteristics.
However, adopting Cassandra as a single, one size fits all database has several downsides. The partitioned/distributed data storage model makes it difficult (and often very inefficient) to do certain types of queries or data analytics that are much more straightforward in a relational database.
March 7, 2017 | Data Analysis, GCP
Google has recently made its internal Spanner database available to the wider public, as a hosted solution on Google Cloud. This is a distributed relational/transactional database used inside for various Google projects (including F1, the advertising backend), promising high throughput, low latency and 99.999% availability. As such it is an interesting alternative to many open source or other hosted solutions. This whitepaper gives a good theoretical introduction into Spanner.
February 28, 2017 | Software Consultancy
As Kotlin’s 1.1 release draws closer, I’ve been looking at some of the new language features it supports. Type aliases may seem like a relatively minor feature next to coroutines, but as I will show in this blog post, they can open up a new programming idiom, particularly when combined with extension functions.
February 16, 2017 | Cassandra
One of the default Cassandra strategies to deal with more sophisticated queries is to create CQL tables that contain the data in a structure that matches the query itself (denormalization). Cassandra 3.0 introduces a new CQL feature, Materialized Views which captures this concept as a first-class construct.
February 13, 2017 | Data Engineering
One of the stated intentions behind the design of Java 8’s Streams API was to take better advantage of the multi-core processing power of modern computers. Operations that could be performed on a single, linear stream of values could also be run in parallel by splitting that stream into multiple sub-streams, and combining the results from processing each sub-stream as they became available.
January 26, 2017 | Data Engineering
Suppose you are given the task of writing code that fulfils the following contract:
This blog is written exclusively by the OpenCredo team. We do not accept external contributions.
January 25, 2017 | Cassandra
One of the simplest and best-understood models of computation is the Finite State Machine (FSM). An FSM has fixed range of states it can be in, and is always in one of these states. When an input arrives, this triggers a transition in the FSM from its current state to the next state. There may be several possible transitions to several different states, and which transition is chosen depends on the input.
October 13, 2016 | Data Analysis
In Lisp, you don’t just write your program down toward the language, you also build the language up toward your program. As you’re writing a program you may think “I wish Lisp had such-and-such an operator.” So you go and write it. Afterward you realize that using the new operator would simplify the design of another part of the program, and so on. Language and program evolve together…In the end your program will look as if the language had been designed for it. And when language and program fit one another well, you end up with code which is clear, small, and efficient – Paul Graham, Programming Bottom-Up
October 10, 2016 | Cassandra
In the culmination of our blog series on the topic, on October 6th 2016 OpenCredo Consultants Dominic Fox, Alla Babkina and Guy Richardson, and hosted by Marco Cullen, presented the common design and implementation issues that they have come across in real-world Apache Cassandra deployments.
September 27, 2016 | Cassandra, Data Engineering
If there is one thing to understand about Cassandra, it is the fact that it is optimised for writes. In Cassandra everything is a write including logical deletion of data which results in tombstones – special deletion records. We have noticed that lack of understanding of tombstones is often the root cause of production issues our clients experience with Cassandra. We have decided to share a compilation of the most common problems with Cassandra tombstones and some practical advice on solving them.
September 6, 2016 | Cassandra
A growing number of clients are asking OpenCredo for help with using Apache Cassandra and solving specific problems they encounter. Clients have different use cases, requirements, implementation and teams but experience similar issues. We have noticed that Cassandra data modelling problems are the most consistent cause of Cassandra failing to meet their expectations. Data modelling is one of the most complex areas of using Cassandra and has many considerations.
August 26, 2016 | Kubernetes
This post is the second of a series of three tutorial articles introducing a sample, tutorial project, demonstrating how to provision Kubernetes on AWS from scratch, using Terraform and Ansible. To understand the goal of the project, you’d better start from the first part.
August 24, 2016 | Cassandra
At OpenCredo we are seeing an increase in adoption of Apache Cassandra as a leading NoSQL database for managing large data volumes, but we have also seen many clients experiencing difficulty converting their high expectations into operational Cassandra performance. Here we present a high-level technical overview of the major strengths and limitations of Cassandra that we have observed over the last few years while helping our clients resolve the real-world issues that they have experienced.
July 3, 2016 | DevOps
Several of us from the OpenCredo team were in attendance at the inaugural EU edition of the DevOps Enterprise Summit conference. We have been big fans of the two previous US versions, and have watched the video recordings of talks (2014, 2015) with keen interest as many of our DevOps transformation clients are very much operating in the ‘enterprise’ space.
April 5, 2016 | Software Consultancy
This post is part of a series which introduce key concepts in successful test automation. Each post contains sample code using the test-automation-quickstart project, a sample Java test automation framework available from Github.
April 2, 2016 | Terraform Provider
When it comes to automating the creation of infrastructure in cloud providers, Terraform (version at time of writing 0.6.14) has become one of my core go to tools in this space. It provides a fantastic declarative approach to describing the resources you want, and then takes care of making it so for you, keeping track of the state in either a local file or a remote store of some sort. Various bits of sensitive data is often provided as input to terraform.
March 2, 2016 | DevOps, Microservices
Many of our clients are currently implementing applications using a ‘microservice’-based architecture. Increasingly we are hearing from organisations that are part way through a migration to microservices, and they want our help with validating and improving their current solution. These ‘microservices checkup’ projects have revealed some interesting patterns, and because we have experience of working in a wide-range of industries (and also have ‘fresh eyes’ when looking at a project), we are often able to work alongside teams to make significant improvements and create a strategic roadmap for future improvements.
January 8, 2016 | Microservices
Many of our clients are in the process of investigating or implementing ‘microservices’, and a popular question we often get asked is “what’s the most common mistake you see when moving towards a microservice architecture?”. We’ve seen plenty of good things with this architectural pattern, but we have also seen a few recurring issues and anti-patterns, which I’m keen to share here.
December 16, 2015 | Cloud, DevOps
In the rush to embrace DevOps, many organisations seek out tools to help them achieve DevOps nirvana; the magical tools that will unify Development and Operations, stop the infighting, and ensure collaboration. This search for tools to solve problems exists in many domains, but seems particularly prevalent in IT (it may be real, or a reflection of my exposure to IT). The temptation to embrace new tools as a panacea is high, because the problems in IT seem so pervasive and persistent.
November 24, 2015 | DevOps, Microservices
It was once again a privilege to present at the annual ‘muCon 2015‘ microservices conference held in London (at the shiny new Skillsmatter CodeNode venue). Based on feedback fro talks I gave earlier in the year, I presented a completely new version of my ‘The Business Behind Microservices‘ talk, which focuses on the organisational and people side of implementing a microservice-based application.
November 1, 2015 | Microservices
To use or not to use hypermedia (HATEOAS) in a REST API, to attain the Level 3 of the famous Richardson Maturity Model. This is one of the most discussed subjects about API design.
The many objections make sense (“Why I hate HATEOAS“, “More objections to HATEOAS“…). The goal of having fully dynamic, auto-discovering clients is still unrealistic (…waiting for AI client libraries).
However, there are good examples of successful HATEOAS API. Among them, PayPal.
October 31, 2015 | Microservices
Over the past few weeks I’ve been writing an OpenCredo blog series on the topic of “Building a Microservice Development Ecosystem”, but my JavaOne talk of the same title crept up on me before I managed to finish the remaining posts. I’m still planning to finish the full blog series, but in the meantime I thought it would be beneficial to share the video and slides associated with the talk, alongside some of my related thinking. I’ve been fortunate to work on several interesting microservice projects at OpenCredo, and we’re always keen to share our knowledge or offer advice, and so please do get in touch if we can help you or your organisation.
October 30, 2015 | Cloud, DevOps
In some companies, the inevitable rapidly became accepted as the way to do things, and both development and IT operations worked together to figure out how to collaborate on building systems that satisfied development’s desire for change, and operations desire for stability. Outsourcing infrastructure, and all it implied, gave rise to Devops – the unification of business needs, developer delivery, and operational capacity – but it also gave rise to something else, in companies where the operations teams weren’t quite as quick to move – Shadow IT.
October 18, 2015 | Cloud, DevOps
Last week Steve Poole and I were once again back at the always informative JAX London conference talking about DevOps and the Cloud. This presentation built upon our previous DevOps talk that was presented last year, and focused on the experiences that Steve and I had encountered over the last year (the slides for our 2014 “Moving to a DevOps” mode talk can be found on SlideShare, and the video on Parleys).
October 12, 2015 | DevOps
DevOps is transformative. This (hopefully) won’t be true forever, but it is for now. While the modern management practices of separating development and operations (and to a lesser extent, everyone else) prevail, the tearing down of the walls that separate them will remain transformative. In company after company, management and front-line staff are coming to realise that keeping functions separate, which are inherently interdependent, is a model for blame, shifted responsibility, and acrimony. It’s easy to divvy-up a company up based on function. To many people, it seems the most logical way to do it. Ops does operations, Dev does development, Marketing markets, etc. It seems much harder to do it any other way. So why do it?
September 24, 2015 | Microservices
Unless you’ve been living under a (COBOL-based) rock for the last few years, you will have no doubt heard of the emerging trend of microservices. This approach to developing ‘loosely coupled service-oriented architecture with bounded contexts’ has captured the hearts and minds of many developers. The promise of easier enforcement of good architectural and design principles, such as encapsulation and interface segregation, combined with the availability to experiment with different languages and platforms for each service, is a (developer) match made in heaven.
August 12, 2015 | Microservices
Over the last few months one of my main projects at OpenCredo has involved creating various microservices which are interacted with via REST. We’ve been working with a relatively rich domain model, which in turn has presented a lot of challenges in how to design our various resources. This blog post aims to summarise various techniques and practices which I’ve found helpful in overcoming these challenges.
August 11, 2015 | DevOps
For years, OpenCredo has been working with organisations to help them introduce new technologies, and more effective development practices, to their IT teams. This has met with a great deal of success, and we have worked with a variety of companies of various sizes. During these projects, we have consistently noticed that the changes we make reach beyond IT in their impact and effects.
August 10, 2015 | Cloud, Software Consultancy
As a company, we at OpenCredo are heavily involved in automation and devOps based work, with a keen focus on making this a seamless experience, especially in cloud based environments. We are currently working within HMRC, a UK government department to help make this a reality as part of a broader cloud broker ecosystem project. In this blog post, I look to provide some initial insight into some of the tools and techniques employed to achieve this for one particular use case namely:
With pretty much zero human intervention, bar initiating a process and providing some inputs, a development team from any location, should be able to run “something”, which, in the end, results in an isolated, secure set of fully configured VM’s being provisioned within a cloud provider (or providers) of choice.
August 5, 2015 | Cloud, GCP, Kubernetes
Why OpenCredo partnered with Google
Recently OpenCredo chose to partner with Google in order to share knowledge and resources around the Google Cloud Platform offerings. Our clients come in many shapes and sizes, but typically all of them realise three disruptive truths of the modern IT industry: the (economic) value of cloud; the competitive advantage of continuous delivery; and the potential of hypothesis and data-driven product development to increase innovation (as popularised by the Lean Startup / Lean Enterprise motto of ‘build, measure, learn’).
July 8, 2015 | Software Consultancy
I was quite excited around autumn last year when Google started to work on a new version of Angular (Angular 2.0) which promised to revolutionise web development. There were rumours that Angular 2.0 wouldn’t be backward compatible with its predecessor, and would be written in Google’s AtScript which is a JavaScript based language on top of Microsoft’s TypeScript. The lack of backwards compatibility raised some concerns, especially for the clients that we had used Angular at. But, lets not get ahead of ourselves here….
May 13, 2015 | Software Consultancy
Listen to Brenden Matthews discuss Elastic Analytics with Spark, Mesos and Docker as filmed at the most recent London Mesos User Group Meetup.
In this talk, Brenden Matthews discusses how he provided elastic analytics to Airbnb and how the Mesosphere DCOS can easily bring the same type of infrastructure to your own environments.
November 19, 2014 | Microservices
Undeniably, there is a growing interest in microservices as we see more organisations, big and small, evaluating and implementing this emerging approach. Despite its apparent novelty, most concepts and principles underpinning microservices are not exactly new – they are simply proven and commonsense software development practices that now need to be applied holistically and at a wider scale, rather than at the scale of a single program or machine. These principles include separation of concerns, loose coupling, scalability and fault-tolerance.
October 23, 2014 | Cassandra
Spring Data Cassandra (SDC) is a community project under the Spring Data (SD) umbrella that provides convenient and familiar APIs to work with Apache Cassandra.
February 17, 2014 | Cassandra
In our previous posts we gave an overview of Cassandra’s new compare-and-set (lightweight transaction) commands and a more detailed look into the API for using them when inserting new rows into the database.
In this third post, we are going to cover update statements. We recommend reading the previous posts, as there are some details which are the same for inserts and updates which are not repeated here.
January 6, 2014 | Cassandra
The team over at Cucumber Pro recently posted a sneak peek on their blog, demonstrating some key features of their offering.
As more of a technical user of Cucumber, there isn’t much that’s new or ground-breaking for me – almost every feature is already available through your preferred IDE combined with a few plugins.
December 2, 2013 | Cassandra
Perhaps the most important of Cassandra’s selling points is its completely distributed architecture and its ability to easily extend the cluster with virtually any number of nodes. Implementing a classical RDBMS-style transaction consisting of “put locks on the database, modify the data, then commit the transaction”-style operations are simply not feasible in such an architecture (i.e. that doesn’t scale well).
November 14, 2013 | Cassandra
Cassandra 2.0 was released in early September this year and came with some interesting new features, including “lightweight transactions” and triggers.
Despite the rising interest in the various non-relational databases in recent years, there are still numerous use-cases for which a relational database system is a better choice. The latest major release of Cassandra (version 2.0) provides some interesting features that aim to close this gap, and offers its fast and distributed storage engine enhanced with new options that will make users’ lives easier.
July 4, 2013 | Software Consultancy
In which situations Spring Data Hadoop (SDH) can add value, and in which situations would it be a poor choice? This article follows on from an objective summary of the features of SDH.
January 7, 2013 | Software Consultancy
The practice of continuous integration in which build servers are used to build and perform testing of code is now widespread and mainstream.
While not all teams have adopted continuous integration effectively, its increasing adoption has led many to start to look for additional opportunities to improve the cost, quality and speed of delivery with which software targeted to meet business needs can be released into production environments.
Traditionally Continuous Integration addresses the question of “does the software build and pass our unit and integration test suites?”. This is often insufficient.
June 3, 2011 | Neo4j
Neo4J is one of the first graph databases to appear on the global market. Being open source, in addition to its power and simplicity in supporting graph data model it represents good choice for production-ready graph database.
However, there has been one area I have struggled to get good-enough performance from Neo4j recently – super nodes.
