Search for a command to run...
Quick highlights of AWS Step function and Argo Workflows
A Software Product Engineer, Cloud enthusiast | Blogger | DevOps | SRE | Python Developer. I usually automate my day-to-day stuff and Blog my experience on challenging items.
No comments yet. Be the first to comment.
The way we consume tech content has changed - my blog is changing with it. Introducing the Prompts series.
Deep metrics, log shipping, and a ready dashboard before your performance test starts - plus the war story of why it matters.
One prompt that lets an LLM read, draft, and publish on your Hashnode blog - token stays safe in an env var.
Hello All, If you've been running Argo CD on EKS, you know the drill — install the helm chart, manage the controllers, handle upgrades, configure SSO, worry about HA, and repeat for every cluster. It'
Introduction If you've spent any time managing data pipelines in the real world, you know the pain of deploying changes manually — copy-pasting notebooks, praying nothing breaks in prod, and maintaini
This blog quickly shows the comparison between AWS Step functions and ARGO workflow.
AWS Step Functions is a serverless orchestration service that lets you integrate with AWS Lambda functions and other AWS services for any specific use cases.
Step functions help to execute stateful and stateless jobs in a sequential way to achieve a solution.
Assume 10-STEP operational workflow; in each step, it does a specific operation that is interrelated to all other jobs, like the success of STEP A triggers STEP B.
Each step can be in the different executor,
STEP-1: python, STEP-2: bash etc..,
And above is a single example and there can be n different use cases for such, like sequential Jobs, parallel jobs and at scale etc.
AWS Step functions are AWS agonistic. Yes, we can communicate with any of the other AWS services via API/Cloud SDK.
One of the practical AWS Step function use cases.
Restore a database from the backup.
Sanitize the sensitive data.
update the connection strings in the secrets manager.
ARGO workflow is an open-source container-native workflow engine for orchestrating jobs on Kubernetes.
container native - Yes, bring out your imagination of use cases here to leverage the power of ARGO-Workflow at scale.
Some of the most common use cases are,
CICD pipelines
analytics operation
high volume data processing
sequential, parallel and more
directed-acyclic graph (DAG)
How it is different from AWS Step functions?
AWS Step functions are AWS agonistic whereas ARGO Workflows are cloud-agnostic.
Can be provisioned using cloud formation templates and AWS CLI. (submitting step functions via CLI)
job submission can be in JSON/YAML.
support GUI, API, CLI and through supported AWS services. (Pls refer to the aws doc linked at the bottom.)
support to create lambda functions at each stage.
ARGO Workflows can be used across cloud providers.
Workflows can be templated/bootstrapped.
ARGO jobs can be submitted using GUI and CLI.
Workflows can be templated and YAML syntax
AWS STEP Functions.
It is easy to pay as you go, However, if we decode below are the price points to be considered while using step fn.
State transition charges + based on resource usage(CPU RAM utilized for each step)
Lambda backend charges, EC2, ECS, ECR(if required)
ARGO
Cloud hosting charges in the Kubernetes environment. (can be a part of existing cluster) or Independent hosting in any vm.
Who knows maybe ARGO can come up with their native cloud-hosted environment in the future, and I am counting on it.
We will discuss ARGO workflow with an example in detail in our upcoming blog post
Thank you!
References:
https://docs.aws.amazon.com/step-functions/latest/dg/development-options.html